This testing tip is in response to a reader question about regression testing of flight critical software. Asks the reader:
"I'm involved in software safety evaluation of flight critical software. I try to ensure that whenever any fix/update is added to the software, all test cases related to safety critical software are also run. To this end, I want to enforce developing a basic test case consisting of all safety-related test cases, to be modified as needed with additional test cases addressing the particular changes. I have been challenged as being too restrictive, as with use of partitioning and other modern software development methods, this represents testing 'overkill.' I'd appreciate your opinion on whether I am being overly cautious and restrictive."
If the question is, "Is it 'overkill' to develop a regression process around safety-related features?" I would say certainly not. The safety features would be the bare minimum of what you would want to include in regression testing on a go-forward basis. All it takes is a single accident to suddenly bring the testing process into foreground. Here are just a few safety-related areas that might be identified for regression:
- critical computations, control and logic related to:
- engine management
- fuel management
- aircraft control surfaces
- navigation algorithms
- flight instrumentation/sensors
- weapons systems
- screen navigation and user interface
- system redundancy and failover
- system health and diagnostics
The categories themselves indicate the seriousness of a potential defect. If you continue to experience pressure regarding a regression process, here are a few strategies you could employ:
- Industry data -- Gather industry data around software-related accidents.
- Defect data -- Gather defect data specific to your avionics systems.
- Change frequency -- How many changes are being introduced on a monthly or yearly basis?
- Safety board -- Leverage a software safety review board (if available) to help sponsor / guide testing and quality initiatives.
- employ automation -- May provide for some schedule gains and allay timeline concerns. There may even be a way to leverage test data from other related test activities past or present.
About the author: Baher Malek is currently a principal quality and test engineer for a Fortune 100 company where he helps traditional project teams adopt agile approaches to software testing. Baher is also a frequent attendee and speaker at the Indianapolis Workshops on Software Testing and the Workshop on Open Certification for Software Testers. You can contact Baher via email at firstname.lastname@example.org.