Determining test deliverables begins with the creation of an overall software testing strategy. In an earlier article on software testing deliverables, I addressed test plans, test cases, defects/faults and status reports -- here I'll update and expand that information based on reader feedback and lessons learned from applying these practices over the last few years. Performance assurance and testing practices will be dealt with in a later article -- any organization moving into this space should deploy a separate performance assurance testing strategy and set of deliverables.
Any approach the test team takes toward testing will depend on its role within the overall system development lifecycle (SDLC). The application landscape and the test team's role within that landscape directly affect which testing approach is most appropriate. I'll review four application landscapes:
- Traditional back-office systems
- Field support systems
- Customer-facing systems
- Enterprise resource planning (ERP) applications
Recommendations for each application landscape are presented on which test phases the test organization will be involved in and why. These test phases are broken down into distinct phases and their relationship to the software's code maturity explained:
- Responsible for design review phase
- Participate in or review unit test phase/code instrumentation
- Responsible for function test phase
- Responsible for system test phase
- Responsible for performance test phase
- Participate in or review user acceptance phase
As the test strategy is presented I will be using these terms:
- Responsible -- an activity the test team or teams should own.
- Participate -- an activity the test team or teams would be involved in but not own.
The test organization is only responsible for three of these test phases, depending on the application landscape. These consist of:
- Design review phase -- review process
- Function test phase -- testing process
- System test phase -- testing process
The design review phase involves the implementation of a formal review process and does not have any specific test types associated with it. The function test phase and system test phase each have associated test types. Not all testing types are appropriate for all test phases -- I recommend the implementation of specific test types to fit the needs of each test phase based on industry best practices. The following test types are considered:
- Error handling testing
- Smoke testing
- Functionality testing
- User interface testing
- Usability testing
- Report and chart testing
- Conversion testing
- Environment testing
- Documentation ...
To continue reading for free, register below or login
To read more you must become a member of SearchSoftwareQuality.com
testing
- Security testing
- Performance testing (responsibility of performance assurance testing group)
- Volume testing
- Stress testing
- Installation testing
I have associated the most appropriate test techniques to each test type. Not all test techniques are appropriate to each test type. The following test techniques are considered:
- White-box tests or glass-box tests
- Black-box tests
- Equivalence partitioning
- Boundary-value analysis
- Error guessing
- Output forcing
Software maturity and test phases
As software moves from concept to product, it goes through several levels of maturity. This is not a linear series of events. It is a progression, as the software progresses through phases of testing toward production-appropriate levels. The following table presents the relationship between code maturity, software state and test phases for back-office applications:
Code maturity
Software exists as…
Test phase
Conceptual (no code)
Requirements & specifications
Design review phase
Under construction
Code units
Unit test phase & instrumentation
Construction complete
Functional software
Function test phase
Functionality confirmed
Package or release
System test phase
Package or release confirmed
Package or release
User acceptance test phase
Back-office applications
Back-office applications tend to be the most mature and best understood applications within the corporate application landscape. The risks associated with updates to the applications are usually minimal -- unless your organization is deploying new back-office applications or converting from one back-office solution to another. Within this context of "software maturity" the test teams are responsible for the system test phase and also support the user acceptance phase. If the level of risk increases or there are a significant number of production incidents then a more formal application of the other test phases must be applied.
Field support systems
Field support systems have become a common factor within the corporate landscape. These applications enable your virtual and direct customer support groups to interact efficiently with your customer base. These applications are not used directly by your customers but do have a direct impact on the customer experience. The test organization will usually require supplemental assistance from other teams including the network team, infrastructure team and developers to perform testing. In order to address the ongoing challenge of rapidly deployed field support systems, the test team will need to ensure appropriate rigor has been applied to the construction and testing of the code before it reaches system test.
Code maturity
Software exists as…
Test phase
Conceptual (no code)
Requirements & specifications
Design review phase
Under construction
Discrete code units
Unit test phase
Construction complete
Functional software
Function test phase
Functionality confirmed
Package or release
System test phase
Package or release confirmed
Package or release
User acceptance test phase
Taking responsibility for the design review phase, at least from an oversight perspective, enables the team to verify the content, testability and accuracy of requirements before construction begins. This brings testing rigor and discipline into the development process early, mitigating the risks presented by less disciplined or inexperienced development teams.
Taking responsibility for the function test phase enables the team to validate the scope and maturity of any function testing. The system team is able to size the system test phase effort appropriately and determine the appropriate level of rigor -- this may involve the team actually performing some or all aspects of the function test phase.
The test organization takes responsibility for the system test phase, including all required planning, design, construction, execution, and reporting deliverables and processes.
Customer-facing systems and ERP
Testing customer-facing systems (millions of users) and ERP applications (corporate-wide impact) presents substantive risks. Development within these types of application landscapes tends to be more accelerated than traditional back-office applications. There is a significant increase in both production risks and rewards here.
The test teams need to get ahead of the quality curve to ensure the product is ready when it enters the system test phase. The teams and their partners need to recognize the risks inherent in both the application landscape and the impact of several development groups (vendors, outsource, client and in-house) being involved. Test teams function as auditor (Does it function as expected?) and safety net (Did the changes impact other aspects of the system architecture?). Testing is required for the design review phase, functional testing phase and system testing phase.
Code maturity
Software exists as…
Test phase
Conceptual (no code)
Requirements & specifications
Design review phase
Under construction
Discrete code units
Unit test phase
Construction complete
Functional software
Function test phase
Functionality confirmed
Package or release
System test phase
Package or release confirmed
Package or release
User acceptance test phase
Summary
Taking responsibility for the design review phase enables the team to verify the content, testability and accuracy of requirements before construction begins. This brings testing rigor and discipline into the development process early, mitigating the risks presented by less disciplined or inexperienced development teams.
Taking responsibility for the function test phase enables the team to validate the functionality, content and readiness before the system test phase begins. The system team is then able to size the system test phase effort appropriately and determine the appropriate level of rigor -- this may involve the team actually re-executing some (or all) aspects of the function test phase.
The test teams also own the system test phase, including all required planning, design, construction, execution, and reporting deliverables and processes.
Test phases and associated test types and techniques
Certain test phases lend themselves to the application of specific test types. As the software release moves through the SDLC, the application, test environment and test data become more production-like. The test team must consider the application landscape when deciding which test types to use -- the more mature and stable the application landscape, the less cost-benefit is derived from performing more sophisticated test types.
Application landscape
The following table represents the relationship between test types, test phases and the application landscape. The function and system test phase columns represent the test types that are applied if strict adherence to all aspects of testing is required -- an example is a medical device that can cause the death of the patient if it fails.
There are three application landscape columns:
- Back-office -- Mature, low-risk applications with minimal customer exposure/risk.
- Field support -- Less mature, medium-risk applications with minimal customer exposure/risk.
- Customer-facing and ERP -- High-risk applications with significant customer exposure/risk.
This table represents a typical release for each application landscape. It can be modified depending on the nature of the immediate testing engagement. For example, conversion testing is not included for any of the application landscapes, but if any significant data transformations or conversions are part of the release under test, then conversion testing has to be performed.
Test Phase
Application Landscape
Test Type
Function
System
Back
Office
Field
Support
Customer
Facing
& ERP
Smoke Testing
X
X
X
X
Regression Testing
X
X
X
X
Functionality Testing
X
X
X
X
X
Error Handling Testing
X
X
User Interface Testing
X
X
X
X
Usability Testing
X
X
X
Report & Chart Testing
X
X
X
X
X
Conversion Testing
Environment Testing
X
X
Documentation Testing
X
X
Security Testing
X
X
Performance Testing*
X
X
X
Volume Testing
X
X
Stress Testing
X
X
Installation Testing
X
X
X
*Performance testing is the responsibility of the performance assurance testing group.
Test type and test techniques
The following table represents the relationship between test types and test techniques. The application of any particular technique is always the decision of the designer, but the following table represents which techniques are usually applied to a particular test type.
Test Type
White
Box
Black
Box
Equivalence
Partitioning
Boundary
Value
Error
Guessing
Output
Forcing
Smoke Testing
X
X
X
X
Regression Testing
X
X
X
X
Functionality Testing
X
X
X
X
Error Handling Testing
X
X
X
X
User Interface Testing
X
X
Usability Testing
X
X
Report & Chart Testing
X
X
X
X
Conversion Testing
X
X
X
X
Environment Testing
X
X
X
X
Documentation Testing
X
Security Testing
X
X
X
Performance Testing
X
X
Volume Testing
X
X
Stress Testing
X
X
Installation Testing
X
X
X
X
To continue reading about elements of software testing deliverables including test plans, cases, defects and status reports, see "Software testing deliverables: From test plans to status reports."
