Professional software testers and QA/test consultants are often asked to address the challenge of compliance. The introduction of compliance standards into any organization
Often there are several misconceptions about the goals of compliance. In most cases, especially government-mandated compliance, compliance is about meeting the minimum standards required to legally operate in that country. In other cases, a governance body is formed to standardize the minimum standards under which a member can operate. Once again, those standards usually exist to set a minimum set of legal bounds or responsibilities. Compliance is not about adding value to development, testing or implementation of any product; it is about being able to operate within the confines of any particular standard.
From a testing perspective, compliancy adds additional cost to any given testing effort. It does not add value, though it can trigger process improvements in the testing space. Compliancy can add significant burden on the testing organization because testing for compliancy is not about mitigating risk -- it is about proving compliance. This requires the testing organization to create, maintain and govern testing artifacts that prove compliance -- in other words, an audit trail. This is not a matter of simply trusting that the appropriate level of testing has occurred (even if it has). This is about proving that the appropriate level of testing has occurred. Faith and trust in your testers is not compliancy; only an auditable trail of evidence can illustrate compliance.
This article provides a starting point for meeting the compliancy challenge. In this example, an internal corporate audit compliancy group has determined that current testing practices do not meet compliancy standards -- even though the current testing practices have supported the delivery of product within acceptable risk parameters.
The testing challenge
The Corporate Audit Compliancy group has identified two immediate challenges that must be addressed by the testing organization:
- Product is not consistently tested and implemented by authorized individuals in a segregated,
- Testing evidence must be supplied with any implementation request, including the following:
- Testing evidence: Summary of testing that includes testing details, test criteria/requirements,
expected results, actual results and test conclusion/report out
- Signoff of testing results by business owners, IT and internal audit
- Testing evidence: Summary of testing that includes testing details, test criteria/requirements, expected results, actual results and test conclusion/report out
Test deliverables (test compliancy)
The testing organization has determined that the following testing products must be institutionalized to meet the new Corporate Audit requirements:
- Test planning -- detailed test plan
- Test requirements -- detailed description of what is being implemented
- Test cases/scripts -- detailed test cases/scripts that test the requirements
- Test results -- capture the actual testing results
- Test defects -- capture any issues discovered during testing and their resolution
- Test conclusion -- status of the product based on the test results (to date)
Test planning (test details)
Test planning should address the communication of what will be tested and why it will be tested. It should also communicate what will not be tested and why it will not be tested. Understanding what will not be tested can be as important, if not more important, than what will be tested.
Test requirements (test criteria)
The most efficient way to identify the details of any system is to capture these details as requirements. The requirements should be specific enough to support both business and IT signoff against a specific implementation request. These requirements, over time, should provide both IT and the business a picture of the system(s) as a whole.
Note that this list of requirements should include both the changes/enhancements to the system and those aspects of the system that will be affected by the new implementation. Compliancy standards will often expand the scope of testing and testing requirements beyond what would be normally addressed to mitigate business risk.
Test cases/scripts (test details)
The most efficient way to identify, design and maintain tests is to capture these details as test cases. These cases then become a repository or catalogue of tests that can be applied when testing the requirement or requirements that they are associated with.
The tests should be specific enough to support both business and IT signoff against specific functionality. These tests, over time, should provide both IT and the business a picture of the system(s) as a whole. This list of tests should be associated with the appropriate requirements; the test cases/scripts should exist to cover (test) only one or more requirement.
Test case design (expected results)
Any test design studio or test design template should provide "one-stop shopping" for any test design task. To be compliant, a test case/script requires step descriptions, inputs and expected outputs. An effective design template or test design studio allows you to present a reusable template (in terms of content requirements) for all future test case/script design. The test case design must present any internal or external auditor with a clear test case that contains auditable pass/fail criteria.
Test execution (actual results)
The results of execution must be captured inside the test run with appropriate levels of security and data integrity. The capture of results cannot consist of a simple statement of pass or fail; there must be proof of success or failure. This can usually be supplied with screenshots or data dumps taken at the time of execution and associated with the validation step. With appropriate test design (see above) and execution results, most of the compliancy challenge will be met. Note that any defects encountered during test execution need to be captured and tracked.
Test defects (actual results)
A defect management process supported by an auditable defect management system will be required to meet the new compliancy standards. This tool should be leveraged as the primary mechanism for capturing, addressing and communicating defects. The standard content of a defect needs to present a clear audit trail that includes defect description, defect severity, defect status and defect resolution. Whenever possible, defect reports will include screenshots of the before "fix" and after "fix" testing results with a clear description of the defect resolution.
Test report (test conclusion)
The test conclusion/final test report should include defect reporting and defect summary reporting, execution reporting and execution summary reporting, coverage reporting and coverage summary reporting (requirements coverage), and report conclusions that address the initial test plan's in-scope and out-of-scope assertions.
Understanding the compliancy requirements and applying them to a well-defined and controlled testing process will help testing address the compliancy challenge.
The mechanics of conforming to a compliancy standard should result in the implementation of "good" testing practices with a particular attention to creating auditable results and enforcement of test artifact integrity. It is important for testers and especially test managers to understand both their legal and moral responsibilities when working within an industry that must conform to a certain compliancy standard. Become familiar with the compliancy standards, your companies interpretation of those standards, and understand your legal responsibilities under those standards.
About the author: David W. Johnson is a senior computer systems analyst with over 20 years of experience in information technology across several industries. He has played key roles in business needs analysis, software design, software development, testing, training, implementation, organizational assessments and support of business solutions. David has developed specific expertise over the past 12 years on implementing "Testware," including test strategies, test planning, test automation and test management solutions. You may contact David at DavidWJohnson@Eastlink.ca.
This was first published in March 2007