Overview

This chapter describes several approaches to testing software. Software testing must be planned carefully to avoid wasting development time and resources. Testing begins "in the small" and progresses "to the large". Initially individual components are tested using white box and black box techniques. After the individual components have been tested and added to the system, integration testing takes place. Once the full software product is completed, system testing is performed. The Test Specification document should be reviewed like all other software engineering work products. A sample Test Specification document appears on the SEPA web site.

Strategic Approach to Software Testing

• Testing begins at the component level and works outward toward the integration of the entire computer-based system.
• Different testing techniques are appropriate at different points in time.
• The developer of the software conducts testing and may be assisted by independent test groups for large projects.
• The role of the independent tester is to remove the conflict of interest inherent when the builder is testing his or her own product.
• Testing and debugging are different activities.
• Debugging must be accommodated in any testing strategy.
• Make a distinction between verification (are we building the product right?) and validation (are we building the right product?)

Strategic Testing Issues

• Specify product requirements in a quantifiable manner before testing starts.
• Specify testing objectives explicitly.
• Identify the user classes of the software and develop a profile for each.
• Develop a test plan that emphasizes rapid cycle testing.
• Build robust software that is designed to test itself (e.g. uses anitbugging).
• Use effective formal reviews as a filter prior to testing.
• Conduct formal technical reviews to assess the test strategy and test cases.

Unit Testing

• Black box and white box testing.
• Module interfaces are tested for proper information flow.
• Local data are examined to ensure that integrity is maintained.
• Boundary conditions are tested.
• Basis path testing should be used.
• All error handling paths should be tested.
• Drivers and/or stubs need to be developed to test incomplete software.

Integration Testing

• Top-down integration testing

1. Main control module used as a test driver and stubs are substitutes for components directly subordinate to it.
2. Subordinate stubs are replaced one at a time with real components (following the depth-first or breadth-first approach).
3. Tests are conducted as each component is integrated.
4. On completion of each set of tests and other stub is replaced with a real component.
5. Regression testing may be used to ensure that new errors not introduced.

• Bottom-up integration testing

1. Low level components are combined in clusters that perform a specific software function.
2. A driver (control program) is written to coordinate test case input and output.
3. The cluster is tested.
4. Drivers are removed and clusters are combined moving upward in the program structure.

• Regression testing (check for defects propagated to other modules by changes made to existing program)

1. Representative sample of existing test cases is used to exercise all software functions.
2. Additional test cases focusing software functions likely to be affected by the change.
3. Tests cases that focus on the changed software components.

• Smoke testing

1. Software components already translated into code are integrated into a build.
2. A series of tests designed to expose errors that will keep the build from performing its functions are created.
3. The build is integrated with the other builds and the entire product is smoke tested daily (either top-down or bottom integration may be used).

General Software Test Criteria

• Interface integrity (internal and external module interfaces are tested as each module or cluster is added to the software)
• Functional validity (test to uncover functional defects in the software)
• Information content (test for errors in local or global data structures)
• Performance (verify specified performance bounds are tested)

Validation Testing

• Ensure that each function or performance characteristic conforms to its specification.
• Deviations (deficiencies) must be negotiated with the customer to establish a means for resolving the errors.
• Configuration review or audit is used to ensure that all elements of the software configuration have been properly developed, cataloged, and documented to allow its support during its maintenance phase.

Acceptance Testing

• Making sure the software works correctly for intended user in his or her normal work environment.
• Alpha test (version of the complete software is tested by customer under the supervision of the developer at the developer’s site)
• Beta test (version of the complete software is tested by customer at his or her own site without the developer being present)

System Testing

• Recovery testing (checks the system’s ability to recover from failures)
• Security testing (verifies that system protection mechanism prevent improper penetration or data alteration)
• Stress testing (program is checked to see how well it deals with abnormal resource demands – quantity, frequency, or volume)
• Performance testing (designed to test the run-time performance of software, especially real-time software)

Debugging

• Debugging (removal of a defect) occurs as a consequence of successful testing.
• Some people are better at debugging than others.
• Common approaches:

1. Brute force (memory dumps and run-time traces are examined for clues to error causes)
2. Backtracking (source code is examined by looking backwards from symptom to potential causes of errors)
3. Cause elimination (uses binary partitioning to reduce the number of locations potential where errors can exist)

Bug Removal Considerations

• Is the cause of the bug reproduced in another part of the program?
• What "next bug" might be introduced by the fix that is being proposed?
• What could have been done to prevent this bug in the first place?