This testing tip is in response to a reader question about regression testing of flight critical software. Asks the reader:
"I'm involved in software safety evaluation of flight critical software. I try to ensure that whenever any fix/update is added to the software, all test cases related to safety critical software are also run. To this end, I want to enforce developing a basic test case consisting of all safety-related test cases, to be modified as needed with additional test cases addressing the particular changes. I have been challenged as being too restrictive, as with use of partitioning and other modern software development methods, this represents testing 'overkill.' I'd appreciate your opinion on whether I am being overly cautious and restrictive."
If the question is, "Is it 'overkill' to develop a regression process around safety-related features?" I would say certainly not. The safety features would be the bare minimum of what you would want to include in regression testing on a go-forward basis. All it takes is a single accident to suddenly bring the testing process into foreground. Here are just a few safety-related areas that might be identified for regression:
- critical computations, control and logic related to:
- engine management
- fuel management
- aircraft control surfaces
- navigation algorithms
- flight instrumentation/sensors
- weapons systems
- screen navigation and user interface
- system redundancy and failover
- system health and diagnostics
The categories themselves indicate the seriousness of a potential defect. If you continue to experience pressure regarding a regression process, here are a few strategies you could employ:
- Industry data -- Gather industry data around software-related accidents.
- Defect data -- Gather defect data specific to your avionics systems.
- Change frequency -- How many changes are being introduced on a monthly or yearly basis?
- Safety board -- Leverage a software safety review board (if available) to help sponsor / guide testing and quality initiatives.
- employ automation -- May provide for some schedule gains and allay timeline concerns. There may even be a way to leverage test data from other related test activities past or present.
Above all, do not attempt to do this all at once. An incremental plan can be developed to gradually grow this regression process. This would also lend credibility to your proposal as it acknowledges real world schedule and resource constraints. With the right partnerships, a regression plan and process can be incrementally developed to improve system safety.
-----------------------------------------
About the author: Baher Malek is currently a principal quality and test engineer for a Fortune 100 company where he helps traditional project teams adopt agile approaches to software testing. Baher is also a frequent attendee and speaker at the Indianapolis Workshops on Software Testing and the Workshop on Open Certification for Software Testers. You can contact Baher via email at baher.malek@gmail.com.
