Tip

Software testing deliverables: Developing a software testing strategy

Determining test deliverables begins with the creation of an overall software testing strategy. In an earlier article on software testing deliverables, I addressed test

Requires Free Membership to View

plans, test cases, defects/faults and status reports -- here I'll update and expand that information based on reader feedback and lessons learned from applying these practices over the last few years. Performance assurance and testing practices will be dealt with in a later article -- any organization moving into this space should deploy a separate performance assurance testing strategy and set of deliverables.

Any approach the test team takes toward testing will depend on its role within the overall system development lifecycle (SDLC). The application landscape and the test team's role within that landscape directly affect which testing approach is most appropriate. I'll review four application landscapes:

  • Traditional back-office systems
  • Field support systems
  • Customer-facing systems
  • Enterprise resource planning (ERP) applications

Recommendations for each application landscape are presented on which test phases the test organization will be involved in and why. These test phases are broken down into distinct phases and their relationship to the software's code maturity explained:

  • Responsible for design review phase
  • Participate in or review unit test phase/code instrumentation
  • Responsible for function test phase
  • Responsible for system test phase
  • Responsible for performance test phase
  • Participate in or review user acceptance phase

As the test strategy is presented I will be using these terms:

  • Responsible -- an activity the test team or teams should own.
  • Participate -- an activity the test team or teams would be involved in but not own.

The test organization is only responsible for three of these test phases, depending on the application landscape. These consist of:

  • Design review phase -- review process
  • Function test phase -- testing process
  • System test phase -- testing process

The design review phase involves the implementation of a formal review process and does not have any specific test types associated with it. The function test phase and system test phase each have associated test types. Not all testing types are appropriate for all test phases -- I recommend the implementation of specific test types to fit the needs of each test phase based on industry best practices. The following test types are considered:

  • Error handling testing
  • Smoke testing
  • Functionality testing
  • User interface testing
  • Usability testing
  • Report and chart testing
  • Conversion testing
  • Environment testing
  • Documentation testing
  • Security testing
  • Performance testing (responsibility of performance assurance testing group)
  • Volume testing
  • Stress testing
  • Installation testing

I have associated the most appropriate test techniques to each test type. Not all test techniques are appropriate to each test type. The following test techniques are considered:

  • White-box tests or glass-box tests
  • Black-box tests
  • Equivalence partitioning
  • Boundary-value analysis
  • Error guessing
  • Output forcing

Software maturity and test phases

As software moves from concept to product, it goes through several levels of maturity. This is not a linear series of events. It is a progression, as the software progresses through phases of testing toward production-appropriate levels. The following table presents the relationship between code maturity, software state and test phases for back-office applications:

Code maturity

Software exists as…

Test phase

Conceptual (no code)

Requirements & specifications

Design review phase

Under construction

Code units

Unit test phase & instrumentation

Construction complete

Functional software

Function test phase

Functionality confirmed

Package or release

System test phase

Package or release confirmed

Package or release

User acceptance test phase

Back-office applications

Back-office applications tend to be the most mature and best understood applications within the corporate application landscape. The risks associated with updates to the applications are usually minimal -- unless your organization is deploying new back-office applications or converting from one back-office solution to another. Within this context of "software maturity" the test teams are responsible for the system test phase and also support the user acceptance phase. If the level of risk increases or there are a significant number of production incidents then a more formal application of the other test phases must be applied.

Field support systems

Field support systems have become a common factor within the corporate landscape. These applications enable your virtual and direct customer support groups to interact efficiently with your customer base. These applications are not used directly by your customers but do have a direct impact on the customer experience. The test organization will usually require supplemental assistance from other teams including the network team, infrastructure team and developers to perform testing. In order to address the ongoing challenge of rapidly deployed field support systems, the test team will need to ensure appropriate rigor has been applied to the construction and testing of the code before it reaches system test.

Code maturity

Software exists as…

Test phase

Conceptual (no code)

Requirements & specifications

Design review phase

Under construction

Discrete code units

Unit test phase

Construction complete

Functional software

Function test phase

Functionality confirmed

Package or release

System test phase

Package or release confirmed

Package or release

User acceptance test phase

Taking responsibility for the design review phase, at least from an oversight perspective, enables the team to verify the content, testability and accuracy of requirements before construction begins. This brings testing rigor and discipline into the development process early, mitigating the risks presented by less disciplined or inexperienced development teams.

Taking responsibility for the function test phase enables the team to validate the scope and maturity of any function testing. The system team is able to size the system test phase effort appropriately and determine the appropriate level of rigor -- this may involve the team actually performing some or all aspects of the function test phase.

The test organization takes responsibility for the system test phase, including all required planning, design, construction, execution, and reporting deliverables and processes.

Customer-facing systems and ERP

Testing customer-facing systems (millions of users) and ERP applications (corporate-wide impact) presents substantive risks. Development within these types of application landscapes tends to be more accelerated than traditional back-office applications. There is a significant increase in both production risks and rewards here.

The test teams need to get ahead of the quality curve to ensure the product is ready when it enters the system test phase. The teams and their partners need to recognize the risks inherent in both the application landscape and the impact of several development groups (vendors, outsource, client and in-house) being involved. Test teams function as auditor (Does it function as expected?) and safety net (Did the changes impact other aspects of the system architecture?). Testing is required for the design review phase, functional testing phase and system testing phase.

Code maturity

Software exists as…

Test phase

Conceptual (no code)

Requirements & specifications

Design review phase

Under construction

Discrete code units

Unit test phase

Construction complete

Functional software

Function test phase

Functionality confirmed

Package or release

System test phase

Package or release confirmed

Package or release

User acceptance test phase

Summary

Taking responsibility for the design review phase enables the team to verify the content, testability and accuracy of requirements before construction begins. This brings testing rigor and discipline into the development process early, mitigating the risks presented by less disciplined or inexperienced development teams.

Taking responsibility for the function test phase enables the team to validate the functionality, content and readiness before the system test phase begins. The system team is then able to size the system test phase effort appropriately and determine the appropriate level of rigor -- this may involve the team actually re-executing some (or all) aspects of the function test phase.

The test teams also own the system test phase, including all required planning, design, construction, execution, and reporting deliverables and processes.

Test phases and associated test types and techniques

Certain test phases lend themselves to the application of specific test types. As the software release moves through the SDLC, the application, test environment and test data become more production-like. The test team must consider the application landscape when deciding which test types to use -- the more mature and stable the application landscape, the less cost-benefit is derived from performing more sophisticated test types.

Application landscape

The following table represents the relationship between test types, test phases and the application landscape. The function and system test phase columns represent the test types that are applied if strict adherence to all aspects of testing is required -- an example is a medical device that can cause the death of the patient if it fails.

There are three application landscape columns:

  • Back-office -- Mature, low-risk applications with minimal customer exposure/risk.
  • Field support -- Less mature, medium-risk applications with minimal customer exposure/risk.
  • Customer-facing and ERP -- High-risk applications with significant customer exposure/risk.

This table represents a typical release for each application landscape. It can be modified depending on the nature of the immediate testing engagement. For example, conversion testing is not included for any of the application landscapes, but if any significant data transformations or conversions are part of the release under test, then conversion testing has to be performed.

 

Test Phase

Application Landscape

Test Type

Function

System

Back
Office

Field
Support

Customer
Facing
& ERP

Smoke Testing

X

 

X

X

X

Regression Testing

X

 

X

X

X

Functionality Testing

X

X

X

X

X

Error Handling Testing

X

 

 

 

X

User Interface Testing

X

X

 

X

X

Usability Testing

 

X

 

X

X

Report & Chart Testing

X

X

X

X

X

Conversion Testing

 

 

 

 

 

Environment Testing

X

 

 

 

X

Documentation Testing

X

 

 

 

X

Security Testing

 

X

 

 

X

Performance Testing*

 

X

 

X

X

Volume Testing

 

X

 

 

X

Stress Testing

 

X

 

 

X

Installation Testing

 

X

 

X

X

*Performance testing is the responsibility of the performance assurance testing group.

Test type and test techniques

The following table represents the relationship between test types and test techniques. The application of any particular technique is always the decision of the designer, but the following table represents which techniques are usually applied to a particular test type.

Test Type

White
Box

Black
Box

Equivalence
Partitioning

Boundary
Value

Error
Guessing

Output
Forcing

Smoke Testing

 

X

X

X

X

 

Regression Testing

 

X

X

X

X

 

Functionality Testing

 

X

X

X

 

X

Error Handling Testing

 

X

 

X

X

X

User Interface Testing

 

X

 

 

 

X

Usability Testing

 

X

 

 

 

X

Report & Chart Testing

 

X

X

 

X

X

Conversion Testing

X

X

 

 

X

X

Environment Testing

X

X

 

 

X

X

Documentation Testing

 

X

 

 

 

 

Security Testing

 

X

 

 

X

X

Performance Testing

 

X

 

X

 

 

Volume Testing

 

X

 

X

 

 

Stress Testing

 

X

 

X

 

 

Installation Testing

X

X

 

X

X

 

To continue reading about elements of software testing deliverables including test plans, cases, defects and status reports, see "Software testing deliverables: From test plans to status reports."


This was first published in March 2009

There are Comments. Add yours.

 
TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.