Overview

The importance of software testing to software quality can not be overemphasized. Once source code has been generated, software must be tested to allow errors to be identified and removed before delivery to the customer. While it is not possible to remove every error in a large software package, the software engineer’s goal is to remove as many as possible early in the software development cycle. It is important to remember that testing can only find errors, it cannot prove that a program is bug free. Two basic test techniques involve testing module input/output (black-box) and exercising internal logic of software components (white-box). Formal technical reviews by themselves can not find allow software defects, test data must also be used. For large software projects, separate test teams may be used to develop and execute the set of test cases used in testing. Testing must be planned and designed. The SEPA web site contains the template for a generic test plan.

Software Testing Objectives

• Testing is the process of executing a program with the intent of finding errors.
• A good test case is one with a high probability of finding an as-yet undiscovered error.
• A successful test is one that discovers an as-yet-undiscovered error.

Software Testing Principles

• All tests should be traceable to customer requirements.
• Tests should be planned long before testing begins.
• The Pareto principle (80% of all errors will likely be found in 20% of the code) applies to software testing.
• Testing should begin in the small and progress to the large.
• Exhaustive testing is not possible.
• To be most effective, testing should be conducted by an independent third party.

Software Testability Checklist

• Operability (the better it works the more efficiently it can be tested)
• Observabilty (what you see is what you test)
• Controllability (the better software can be controlled the more testing can be automated and optimized)
• Decomposability (by controlling the scope of testing, the more quickly problems can be isolated and retested intelligently)
• Simplicity (the less there is to test, the more quickly we can test)
• Stability (the fewer the changes, the fewer the disruptions to testing)
• Understandability (the more information known, the smarter the testing)

Good Test Attributes

• A good test has a high probability of finding an error.
• A good test is not redundant.
• A good test should be best of breed.
• A good test should not be too simple or too complex.

Test Case Design Strategies

• Black-box or behavioral testing (knowing the specified function a product is to perform and demonstrating correct operation based solely on its specification without regard for its internal logic)
• White-box or glass-box testing (knowing the internal workings of a product, tests are performed to check the workings of all insdependent logic paths)

Basis Path Testing

• White-box technique usually based on the program flow graph
• The cyclomatic complexity of the program computed from its flow graph using the formula V(G) = E – N + 2 or by counting the conditional statements in the PDL representation and adding 1
• Determine the basis set of linearly independent paths (the cardinality of this set id the program cyclomatic complexity)
• Prepare test cases that will force the execution of each path in the basis set.

Control Structure Testing

• White-box techniques focusing on control structures present in the software
• Condition testing (e.g. branch testing) focuses on testing each decision statement in a software module, it is important to ensure coverage of all logical combinations of data that may be processed by the module (a truth table may be helpful)
• Data flow testing selects test paths based according to the locations of variable definitions and uses in the program (e.g. definition use chains)
• Loop testing focuses on the validity of the program loop constructs (i.e. simple loops, concatenated loops, nested loops, unstructured loops), involves checking to ensure loops start and stop when they are supposed to (unstructured loops should be redesigned whenever possible)

Graph-based Testing Methods

• Black-box methods based on the nature of the relationships (links) among the program objects (nodes), test cases are designed to traverse the entire graph
• Transaction flow testing (nodes represent steps in some transaction and links represent logical connections between steps that need to be validated)
• Finite state modeling (nodes represent user observable states of the software and links represent transitions between states)
• Data flow modeling (nodes are data objects and links are transformations from one data object to another)
• Timing modeling (nodes are program objects and links are sequential connections between these objects, link weights are required execution times)

Equivalence Partitioning

• Black-box technique that divides the input domain into classes of data from which test cases can be derived
• An ideal test case uncovers a class of errors that might require many arbitrary test cases to be executed before a general error is observed
• Equivalence class guidelines:

1. If input condition specifies a range, one valid and two invalid equivalence classes are defined
2. If an input condition requires a specific value, one valid and two invalid equivalence classes are defined
3. If an input condition specifies a member of a set, one valid and one invalid equivalence class is defined
4. If an input condition is Boolean, one valid and one invalid equivalence class is defined

Boundary Value Analysis

• Black-box technique that focuses on the boundaries of the input domain rather than its center

• BVA guidelines:

1. If input condition specifies a range bounded by values a and b, test cases should include a and b, values just above and just below a and b
2. If an input condition specifies and number of values, test cases should be exercise the minimum and maximum numbers, as well as values just above and just below the minimum and maximum values
3. Apply guidelines 1 and 2 to output conditions, test cases should be designed to produce the minimum and maxim output reports
4. If internal program data structures have boundaries (e.g. size limitations), be certain to test the boundaries

Comparison Testing

• Black-box testing for safety critical systems in which independently developed implementations of redundant systems are tested for conformance to specifications
• Often equivalence class partitioning is used to develop a common set of test cases for each implementation

Orthogonal Array Testing

• Black-box technique that enables the design of a reasonably small set of test cases that provide maximum test coverage
• Focus is on categories of faulty logic likely to be present in the software component (without examining the code)
• Priorities for assessing tests using an orthogonal array

1. Detect and isolate all single mode faults
2. Detect all double mode faults
3. Mutimode faults

Specialized Testing

• Graphical user interfaces (see Chapter 31 and the SEPA web checklist)
• Client/server architectures (see Chapter 28)
• Documentation and help facilities (see Chapter 8 and Chapter 15)
• Real-time systems

1. Task testing (test each time dependent task independently)
2. Behavioral testing (simulate system response to external events)
3. Intertask testing (check communications errors among tasks)
4. System testing (check interaction of integrated system software and hardware)