This example is for a web site (e-business).
Dynamic maintainability testing is achieved by testing the maintenance procedures for a particular application. A selection of maintenance scenarios are used as test cases to ensure the required service levels are attainable.
This form of testing is particularly relevant where the underlying infrastructure is complex, and corresponding support procedures may involve multiple departments/organisations. This form of testing may take place as part of operational acceptance testing.
The maintainability of a system can be measured in terms of the time taken to diagnose and fix problems identified within that system. A simple measure can be the mean time taken to diagnose and fix an identified fault.
For example, one may specify that no more than 2 hours should be spent in fixing any identified fault. This information can usually be obtained through metrics collected during the course of system testing, and metrics obtained from reliability testing are particularly suitable for this purpose. It may therefore not be necessary to run separate tests to evaluate this.
The maintainability of a system can also be measured in terms of the effort taken to make required enhancements to that system. This can be tested by recording the time taken to achieve a new piece of identifiable functionality such as a change to the database, etc. A number of similar tests should be run and an average time calculated. The outcome will be that it is possible to give an average effort required to implement specified functionality (this may be measured in terms of an additional function point, or lines of code, etc.). This can be compared against a target effort and an assessment made as to whether requirements are met.
The maintainability of a system can also be measured in terms on the effort required to make required adaptations to that system. This can be measured in the way described above for perfective maintainability testing.
For example: a record is made of the effort required to make a change to the live environment, e.g. to update the version of a database management system "A" and regression test the impacted applications. This can be compared against the relevant maintenance objectives, e.g. that it must be possible to export data, update DBMS and import data in less than 3 hours; and that it must be possible to regression test and check database management system "A" and impacted applications in less than 7 hours. These actions can be undertaken and the execution of them forms the test. The test is passed if the objectives are met.
This refers to actions to reduce future maintenance costs. It is extremely unlikely that specific dynamic testing would be carried out for this type of maintenance. Where it is carried out reference should be made to the method employed in the adaptive maintenance section above.
Zappo Software is in the process of testing its new website. It has completed 150 hours of reliability testing and 200 hours of “functional system testing” and three faults have been identified.
Zappo Software has specified the following maintainability requirements:
1. Not more than 1 hour spent correcting any identified fault (corrective maintenance)
2. Each new function point (system enhancement) shall be implementable within 2 working days (perfective maintenance).
3. Each new change required to adapt to changes in environment shall be implementable within 2 working days (adaptive maintenance).
4. Each change in the system designed to reduce future maintenance costs should be undertaken within 1.5 working days (preventive maintenance).
The above targets for maintainability constitute the expected test results.
At the end of Zappo’s testing the following results have been recorded:
§ Six faults during 350 hours of testing have been recorded. The average time to fix each fault is 30 minutes.
§ Zappo have implemented new areas of functionality during the course of development corresponding to 8 function points. The new functionality took a total of 25 days to implement.
§ A new version of the database was installed and changes to the system to allow for this took 1 day.
§ A change was made to parameterise a previously hardcoded value. This will reduce future maintenance effort. The change took 1 day to implement.
The system passes maintainability tests for corrective, adaptive and preventative maintenance but does not meet the expected criteria in the case of perfective maintenance where the total time spent on implementing new functionality exceeded the target time.