Software Testing Strategies

• Software Testing
  • must be planned carefully to avoid wasting development time and resources
  • the Test Specification document should be reviewed like all other software engineering work products
  • begins at the component level and works outward toward the integration of the entire system
    
  • different testing techniques are appropriate at different points in time
  • the developer of the software conducts testing and may be assisted by independent test groups for large projects
    
  • testing and debugging are different activities
    

• Verification and Validation
  • make a distinction between them
    • verification - are we building the product right?
    • validation - are we building the right product?
      
  • software testing is only one element of Software Quality Assurance (SQA)
    
  • quality must be built in to the development process, you can't use testing to add quality after the fact

  
  
  
  

  • depends on the size of the project
  • Independent Test Group (ITG)
    • removes the conflict of interest inherent when the developer is testing her own product
      
    • misconceptions regarding the use of independent testing teams
      
      • the developer should do no testing at all
        
      • software is tossed "over the wall" to people to test it mercilessly
        
      • testers are not involved with the project until it is time for it to be tested
        
    • developers and the ITG must work together throughout the software project to ensure that thorough tests will be conducted
      

• Software Testing Strategy for Traditional Software Architectures
  • Unit Testing
    • makes heavy use of testing techniques that exercise specific control paths to detect errors in each software component individually
      
  • Integration Testing
    • focuses on issues associated with verification and program construction as components begin interacting with one another
      
  • Validation Testing
    • provides assurance that the software validation criteria (established during requirements analysis) meets all functional, behavioral, and performance requirements
      
  • System Testing
    • verifies that all system elements mesh properly and that overall system function and performance has been achieved
      
      
• Software Testing Strategy for Object-Oriented Architectures
  • Unit Testing
    • components being tested are classes not modules
      
  • Integration Testing
    • as classes are integrated into the architecture regression tests are run to uncover communication and collaboration errors between objects
      
  • Systems Testing
    • the system as a whole is tested to uncover requirement errors
      
      
• Strategic Testing Issues
  • Specify product requirements in a quantifiable manner before testing starts.
    
  • Specify testing objectives explicitly.
    
  • Identify categories of users for the software and develop a profile for each.
    
  • Develop a test plan that emphasizes rapid cycle testing.
    
  • Build robust software that is designed to test itself.
    
  • Use effective formal reviews as a filter prior to testing.
    
  • Conduct formal technical reviews to assess the test strategy and test cases.
  • Develop a continuous improvement approach for the testing process.
    
    
    
• Unit Testing
  
  
  
  
  • Module interfaces are tested for proper information flow.
    
  • Local data are examined to ensure that integrity is maintained.
    
  • Boundary conditions are tested.
    
  • Basis (independent) paths are tested.
    
  • All error handling paths should be tested.
    
  • Drivers and/or stubs need to be developed to test incomplete software

    
    
    

    
    

• Integration Testing
  

• Top-down integration testing
  
  
  
  
1. Main control module used as a test driver and stubs are substitutes for components directly subordinate to it.
   
2. Subordinate stubs are replaced one at a time with real components (following the depth-first or breadth-first approach).
   
3. Tests are conducted as each component is integrated.
   
4. On completion of each set of tests and other stub is replaced with a real component.
   
5. Regression testing may be used to ensure that new errors not introduced.
   
   
• Bottom-up integration testing
  
  
  
  
1. Low level components are combined into clusters that perform a specific software function.
   
2. A driver (control program) is written to coordinate test case input and output.
   
3. The cluster is tested.
   
4. Drivers are removed and clusters are combined moving upward in the program structure.
   
   
• Combined Approach
  
  
  
  
  • reduces problems due to lack of data from bottom levels when testing upper levels with the top down approach
  • minimizes the need to implement drivers in order to test bottom levels with the bottom up approach
    
    
• Regression testing - used to check for defects propagated to other modules by changes made to existing program
  
1. Representative sample of existing test cases is used to exercise all software functions.
   
2. Additional test cases focusing software functions likely to be affected by the change.
   
3. Tests cases that focus on the changed software components.
   
   
• Smoke testing
  
  1. Software components already translated into code are integrated into a build.
     
  2. A series of tests designed to expose errors that will keep the build from performing its functions are created.
  3. The build is integrated with the other builds and the entire product is smoke tested daily (either top-down or bottom integration may be used).
     
     
• General Software Test Criteria
  • Interface integrity
    • internal and external module interfaces are tested as each module or cluster is added to the software
      
  • Functional validity
    • test to uncover functional defects in the software
      
  • Information content
    • test for errors in local or global data structures
      
  • Performance
    • verify specified performance bounds are tested

• Analysis and Design Models
  
  
  
  
• Object-Oriented Testing Strategy
  
  
  
  
• Object-Oriented Unit Testing
  • smallest testable unit is the encapsulated class or object
    
  • similar to system testing of conventional software
    
  • do not test operations in isolation from one another
    
  • driven by class operations and state behavior, not algorithmic detail and data flow across module interface
    
    
• Object-Oriented Integration Testing
  • focuses on groups of classes that collaborate or communicate in some manner
    
  • integration of operations one at a time into classes is often meaningless
  • regression testing is important as each thread, cluster, or subsystem is added to the system

 

Debugging

• Debugging (removal of a defect) occurs as a consequence of successful testing.
  
• Some people are better at debugging than others.
  
  
  
  
  
  
  
  
• Common approaches:
  
• Brute force - memory dumps and run-time traces are examined for clues to error causes
  
• Backtracking - source code is examined by looking backwards from symptom to potential causes of errors
  
• Cause elimination - uses binary partitioning to reduce the number of locations where error could be generated

Notes and figures from:

• Software Engineering: A Practitioner's Approach, 6th Edition, by Roger S Pressman, R.S. Pressman & Associates, McGraw-Hill, 2005