Health care applications in the near future will have to conform to stringent and complex FDA requirements. It's only a matter of time before the majority of health care applications involved in patient care and outcomes fall under FDA regulations in some way. Regulatory requirements and Agile don't go hand in hand; in fact, they move in totally opposite directions. We think of Agile as fast and limber, while FDA regulations are complex...
However, the basic aspects of Agile, which include ongoing planning, coordination and communication, fit nicely, with a few additions, to accommodate documentation required for FDA compliance. It's not an impossible fit to merge the two together, while it may be awkward. As a test team manager, how do you handle the additional documentation and process orientated work associated with even basic FDA regulations without it affecting your Agile productivity? How do you keep going forward with Agile and maintain your speed as a development team when you'll be spending considerable time adding to your process?
Start early, and incorporate the pieces that affect you slowly. In this way, the changes become part of the Agile process. Keep in mind, this article is about a few aspects related to quality assurance that you can integrate into your Agile Scrum team early. It's not meant to cover the full breadth of the possible FDA regulations your application is required to fulfill.
Here we review three quality assurance requirements and how you can mingle them with your Agile process early to introduce the concepts to your team so they become part of the process rather than a separate effort. In this way, your test and Scrum team will adapt readily without the added duress of accommodating complex regulations all at once, and negatively impacting Agile productivity and testing effectiveness.
Test cases: What needs to be in there?
As a test team, your first step is reviewing your existing test case creation method. Are you including both verification and validation tests? For example, verification tests must provide evidence that your software adheres to the requirements. Tests must cover not only the specific requirement, but also ensure the new code is consistent with existing functionality and is complete -- no missing or "TBD" pieces that affect the functionality.
Your test cases also need specific validation test steps, or you can opt to create separate verification and validation test suites. It's basically the same test, only validation includes some type of acceptance testing that confirms the requirement meets the design's intent and users' needs. If you do customer acceptance testing already, then those tests need to be formalized and the execution status tracked.
For both types of testing, all executions must be tracked with pass or fail states and any issues identified from the tests noted with specific explanations. Each issue must indicate when they will be fixed. For issues not fixed, customers must be given a detailed explanation of the defect or issue and its impact on the application's functionality.
You'll also need to add time to review test cases with your Scrum team and evaluate test results. Depending on your Agile process, the Scrum team must also incorporate code reviews, formal and documented unit testing, and formal and documented integration testing. It's feasible to combine your regression, integration and system-level testing into one effort. However you choose to do it, you'll be adding more testing time as well as time to document results and issues.
Once you've covered the functionality, you also need to cover usability testing. Can a user understand the application and use it successfully? In other words, are the requirements you've added useful to your end user? It's no longer enough to assume that if the test team can figure it out, the customer can too. Your test and Scrum teams will need to work together in a separate test effort to ensure customer usability. I've actually tested in places where the customer usability/acceptance testing was done simultaneously with the functional testing. Agile = yes, ideal = no. It ends up being a race for your test team to find any obvious defects before customers see them. The potential exists to undermine your customers trust if they notice multiple, obvious issues during the acceptance testing phase.
As a test team, you'll need to ensure your test cases include a clear objective. What is the point of the test? What exactly and specifically are you verifying? For each execution of the test case, the pass/fail criteria must also be documented. Any corrections to the test case must be noted with the date and time of the change and who performed it and why. All changes must be re-reviewed with the Scrum team.
Adding the test summary report
As a test manager, you'll need to start producing a test summary report for each execution of the test case suite, including regression, integration, system and functional. Additionally, you'll need a report for each type of test if you need to test against different operating systems, third-party builds or platforms. To start out, pick one or two and produce those and add more in as the team gets used to the extra workload.
The test summary report should include details about the test case objectives, all prerequisite conditions, pass and fail percentages, and descriptions of all issues found and their impact to the application functionality. The report should also specify the build, version, platform and operating system details in general, as well as any software configurations.
Your test execution results must include the time, date and who executed the test. Before you are under regulation, spend the time to develop or create an automated report that executes against your test case repository or tool. It's valuable time spent if you can set up the automatic report up to generate accurate numbers without the duress of actually being under regulation.
All defects must be included with specific detail and what impact they have on the applications functionality. The report should also include resolution or fix details and if the issue is not getting fixed, you'll need to explain why.
The traceability matrix
Ah, the traceability matrix. The mention of the word conjures up images of a complex web of requirements mapped to test cases, executed by who, when, where, why and with what result. Sound familiar? If your test team has not yet experienced the joy of the traceability matrix, you're in for a real treat. Despite its often confusing complexity, the traceability matrix is extremely useful and required for meeting FDA regulations.
In the traceability matrix you basically map your test cases to the appropriate requirement or requirements they test. You can use any number of tools, or create your own spreadsheet. The world is your oyster. When you create your matrix, make sure it covers all requirements, including interface and infrastructure. Include and document both the white and black box test cases.
Adding the traceability matrix gives you at least two possible advantages. You're setting up for meeting FDA regulations, and you're performing a double check that your team has a test case for each and every type of requirement. It's a valuable tool in any form it takes to ensure no requirements go untested.
As a test team leader and Agile process champion, it's important to make the move to add regulatory processes as early as possible. You're going to do more documentation, a lot more. You're going to need to get approvals and physical signatures. You're going to have to write a test case for every requirement. Getting your Agile process ready for FDA regulations is essential for most health care applications. By adding in these processes early, you can better massage them to fit into your Agile process with less interruption to your productive schedule. Additionally, you'll gain insight into your test coverage and likely improve the quality of your test cases. Finally, working within a Scrum team allows the test team to use information from engineers, analysts and product managers, resulting in a reliable, well-tested application that ultimately improves patient care and outcomes.