When using this section of the website, refer to the procedure testing and usability testing sections for techniques which may used for testing the installation procedures and the usability of the installation process:
§ The procedure testing section discusses audit and reconciliation techniques which may be adapted for use in Installability testing, for example to see if a new version of a library has been overwritten by the installation with an old version with no warning messages displayed to advise the user that this would happen.
§ The usability testing section discusses techniques which can be employed for measuring the usability of the installation process and for identifying the contexts of use, including the profile of the target installation team.
It may be necessary to test that applications that hitherto worked successfully still function correctly following the installation of other applications. Regression testing of this form is not covered by this section. The compatibility testing section discusses techniques for measuring the affect on existing systems and software of an installation, for example use of "before and after" software audits.
Installability testing uses a model of the Installability requirements, specified in terms of the Installation process (as described in the installation manual or guidelines), the people who will carry out the installation, the target platform(s) and the software/system to be installed.
Ideally, the installability requirement will be defined as part of the overall requirements. If the defined installability requirements are incomplete or missing, the installability tester should consider and document possible attributes of the installation which are more or less desirable. From these attributes it is then possible to derive and validate the installation requirements, in order that tests may be built. Success/acceptance criteria may also include a requirement to complete a subset of the functional tests to demonstrate, or measure, that the system has installed.
The installability requirements are specified in terms of acceptance or success criteria based on consideration of:
· the installation process – for example for a particular installation process one or more of the following may be specified:
· the contents of the installation manual, for example that it should state any particular requirements such as minimum memory or processor speed.
· the maximum number of installation steps
· time that the installation process is expected to take (including maximum acceptable / performance test)
· resources the installation process is expected to take (including minimum, maximum and optimum resources)
· any variations in the entry criteria, process and exit criteria for the installation process (for example for platform and regional variations)
· ability to back out of the install, including definition of the system configuration on completion of a back out, steps to back out
· ability to uninstall, including definition of the system configuration on completion, steps to uninstall
· installation media and whether remote or local installation
· tools and techniques required by the installer
· standards that apply to the installation
· dependencies in the process
· the people who will carry out the installation for example:
· their expectations
· the skills required by the installer(s), (may require some usability testing) e.g. experienced technician, technician - not trained in technology, end user, customer
· installation staff numbers
· installation guide requirements for example, suitability for audience, help for troubleshooting, media, understanding and meeting the installation team's expectations
· the configuration of the target platform(s), for example:
· existing software and hardware with version and functionality defined
· the state of the existing software and hardware (any applicable settings)
· the data, files, records and other stored information needed
· any security issues that apply to cover data integrity and system integrity such as virus checking, firewalls, safety, confidentiality, irreversible actions
· any potential conflicts (hardware and software)
· levels of system access e.g. root, administrator, restricted use
· system configurations, for example single workstation, network installation, internal, business to business, external, client, server, mainframe, single or multiple platforms
· maximum and minimum configuration of the target system (for example memory)
· configuration of target platform pre installation: including system configuration files, for a new installation on clean machine, new installation on machine containing other applications, reinstall or upgrade, de-install
· configuration of target platform post installation, including system configuration files, a new installation on clean machine, new installation on machine containing other applications, reinstall or upgrade, de-install, back-out
· dependencies in the configuration
· permitted upgrade paths, including new installation, upgrade from last version, upgrade from older versions, downgrade
· the type of software/system to be installed, for example:
· bespoke - internally developed application
· bespoke - third party developed application
· product - standardised installation
· product - customised installation
Test coverage is measured as the percentage of Installability requirements exercised during the testing. As a subsidiary measure, if formal test case design techniques are used, test coverage can be measured using the measurement techniques defined in BS7925-2.
A decision will have been made when writing the test strategy as to what determines the acceptance criteria for the installation process. This enables the tester to evaluate the results of testing to see whether the software / system has passed the installation test or not, and whether the testing is completed. The completion criteria of the installation testing could be measured in terms of the following:
§ Were the acceptance criteria defined earlier measured satisfactorily?
§ Were the results of the post-installation audit checked satisfactorily?
§ After installation, is the installed software / system resident and run-able on all targeted hardware platforms?
§ Were all the tests defined in test design executed and passed?
The standard requires that for each test case, the following shall be specified:
§ The start/entry state
§ Which Installability requirement is being tested
§ The actions and inputs
§ The expected outcome
The installation tester may wish to document additional material in the test cases:
|
Test case parts: |
May include: |
|
The start/entry state |
Starting condition Entry criteria Pre-requisites |
|
Which Installability requirement is being tested |
The test object The test objective(s) The scope of the test (what is to be included and what excluded in the testing work) |
|
The actions and inputs |
The steps to be used described in the description, text or form to be used by the installer |
|
The expected outcome |
Closing condition Exit criteria Acceptance criteria Completion measurement The method that is to be used to check that the exit criteria can be judged to have been met and the installation met installation requirements The expected status of the installed system covering both hardware and software |
The standard requires that test cases are designed to exercise the Installability of the system under specified conditions.
Techniques used to design the Installability tests include the techniques described in the procedure testing section and usability sections of this standard, together with formal test design techniques described in BS7925-2.
Boundary value analysis as described in BS7925-2 may be used to test the Installability. For example, if one of the installation requirements is that the installation must complete within a certain time, the tester can push this time to its boundary by setting the test environment conditions:
Requirement for installation time: to be complete in less than 60 minutes
To design the test, the tester looks at the memory and processor requirements for the system, and selects a test platform with the minimum memory and processor requirements. The tester also considers the traffic on the system during the installation and arranges for the test installation to take place with maximum traffic. If the test installation completes within the specified 60 minutes in these conditions it has met the installation time requirement.
Test case: |
|
|
The start/entry state |
Minimum memory (x Mb) Minimum processor speed (xyz processor) Maximum traffic (z transactions per minute) |
|
Which installability requirement is being tested |
Requirement for installation time: to be complete in less than 60 minutes |
|
The actions and inputs |
Install as described in the installation guide steps 1 to 16 |
|
The expected outcome |
Install completes within 60 minutes |
Branch testing as described in BS7925-2 may be used to test the Installability process. The example in BS7925-2 uses code, but the method may used to analyse a procedural or process flow. This technique may be used for test coverage and test case design where different options are available to the installer during the installation. For example, one of the installation requirements may be to allow the installer to select a country/location which will affect the remainder of the installation.
Requirement for installation location: to be able to select a country and currency. Countries are USA, UK, Japan. Currencies are US Dollar (ISO code USD), Euro (ISO code EUR), UK Sterling (ISO code GBP), Japanese Yen (ISO code JPY). The countries and codes are related as follows
Country |
Valid Currency settings |
USA |
USD |
UK |
GBP or EUR |
Japan |
JPY |
The tester draws up a control flow diagram from the installation guide, to describe the choices within the installation. In the figure below, the steps are numbered 1 to 7:
1. Initial installation processes
2. Select USA
3. Select Japan
4. Select UK
5. Select EUR
6. Select GBP
7. Complete installation
Figure: Installation choices:
 |
Tests will be drawn up for routes through the installation. To give a minimum coverage of branches according to BS7925-2, 4 tests would be required. Note that while this does cover the branches, it does not cover all the tests which the tester may need, to address risk or uncover possible faults. A set of test cases to give minimum coverage are shown below:
Test case 1: |
|
The start/entry state |
System is ready for install |
Which Installability requirement is being tested |
USA with USD |
Branch |
1, 2, 7 |
The actions and inputs |
Select USA as the country |
The expected outcome |
USD is auto-selected as the currency and language is auto-selected as US English. Location map on start up shows USA. |
Test case 2: |
|
The start/entry state |
System is ready for install |
Which Installability requirement is being tested |
Japan with JPY |
Branch |
1, 3, 7 |
The actions and inputs |
Select Japan as the country |
The expected outcome |
JPY is auto-selected as the currency and language is auto-selected as Japanese. The limited character set is used. Location map on start up shows Japan. |
Test case 3: |
|
The start/entry state |
System is ready for install |
Which Installability requirement is being tested |
UK with Euro |
Branch |
1, 4, 5, 7 |
The actions and inputs 1 |
Select UK as country |
The expected outcome 1 |
Offered chance to select currency - EUR or GBP |
The actions and inputs 2 |
Select EUR |
The expected outcome 2 |
EUR is set as the currency, language is set as UK English. Location map on start up shows Europe including UK |
Test case 4: |
|
The start/entry state |
System is ready for install |
Which Installability requirement is being tested |
UK with GBP |
Branch |
1, 4, 6, 7 |
The actions and inputs 1 |
Select UK as country |
The expected outcome 1 |
Offered chance to select currency - EUR or GBP |
The actions and inputs 2 |
Select GBP |
The expected outcome 2 |
GBP is set as the currency, language is set as UK English. Location map on start up shows UK |
Tests may also be derived using scenarios. In this case the tests are designed not by using one of the formal techniques in BS7925-2 but by considering how the installation process might be used by different installers or on different platforms.