With limited test time, how do testers focus on finding the highest risk defects? Mary LeMieux-Ruibal from Cognizant and Mirkeya Capellán from Sogeti USA are leading a concurrent session at STAREAST 2012 titled, “Creating a Risk-based Testing Strategy.” LeMieux-Ruibal and Capellán discuss risk-based testing in general and on Agile software teams.
What is risk-based testing and can it be used in Agile development?
Risk-based testing is an approach in which the test efforts are focused on the highest risk areas. In other words, if a test team had a limited amount of time (and, of course, this is always the case) where would they be best off focusing their test efforts? This approach allows for that focus.
However, given Agile’s flexibility when it comes to requirements, and its iterative nature, one might wonder if this approach will work in Agile environments. Is it necessary to have a fully fleshed-out requirements spec in order to determine the highest risk features on which to focus test efforts? According to our experts, a risk-based test strategy can most definitely be used in Agile environments.
“Agile development environments do not mean that testing is not planned. In fact, as sprints are defined and tasks are assigned, the risk-based strategy is also being developed. A good Scrum master knows to place the highest-risk and critical framework tasks down in the initial sprints in order to establish a strong foundation for execution and design. The test team works within this framework to ensure the high-risk and low-risk tasks and modules are executed and validated in a cost-effective manner. The risk-based test strategy prioritizes and outlines the best approach to ensure effective regression and integration testing for each sprint release.”
“The risk-based analysis can work in Agile environments and can be adapted depending on the project needs. In the case of Agile, where little planning is done up-front, you still have to assess and select the stories that will take place on that iteration and also provide an estimation time for completion. The stories are broken into tasks with the customer feedback and prioritized by its level of importance; therefore choosing the high value features for the next release or iteration. Having the stories at hand, testers can start creating a risk-based analysis by using the stories given for that iteration. The objective is the same – identify the high risk stories that can be tested on that iteration and prioritize them depending on their critically to the business.
What about the costs associated with testing?
One basic calculation for risk is
Risk = (Probability of Failure) x (Cost of Failure)
From this calculation, it’s clear that the cost that would be incurred in the case of a failure is a major factor in determining how risky something is. It would make sense that organizations would want to focus on testing for the riskiest defects – those that were most likely to occur and cost the company the most money, should they fail. But what about the cost of testing for those defects? Often, organizations will spend a lot of money for the tools, environments and resources for testing. Some of the highest-risk defects may also be the most costly to test. For example, performance can make or break an application, but performance test resources and tools can be expensive and require specialized skills. How does this factor into testing decisions?
“The risk assessment helps identify the high-risk functions and the implications that may occur in the event of unforeseen problems. By prioritizing the criticality of the risk, you help to alleviate the cost by not having to spend an inordinate amount of time to come up with a solution. You do not have to wait for the business user to determine the criticality of the problem, since that was already addressed by your risk assessment. When a problem does occur, the cost estimation comes very handy since we don’t want to deal with last minute purchases, resources, approvals, licenses, etc. when in need of an immediate solution. Identifying these costs (in terms of monetarily and resource availability) up-front can save a lot of time and effort, giving the team a heads up of what to expect if something goes wrong. Of course, it all depends on the size of the project and how critical the risks are. It is always important to have a contingency plan in place.”
It sounds, then, that the cost of test, while not factored into the formula for assessing risk of features, is still considered up-front during the project planning phases. By understanding high-risk areas, the team is prepared to create plans, factoring in the costs of preventing failures through test efforts.
What types of testing can risk-based testing be used for?
According to our experts, risk-based testing can be used beyond functional testing to virtually every type of test effort. LeMieux-Ruibal notes, “This strategy does also address the performance, security, availability and disaster recovery aspects of testing and what KIND of tests should be executed within each of these to ensure efficient test coverage.”Capellán reiterates the flexibility of a risk-based strategy saying the same logic can be applied to all types of testing. She says, “During the initial planning, the same technique applies regardless of the test type; identify the functions or features that are most critical to the business and prioritize them according to their risk level.”
For comprehensive conference coverage, see our Software Testing Analysis and Review conference page.
With more than thirteen years of IT experience, Dr. Mirkeya Capellán is a QA manager consultant at Sogeti USA, specializing in Internet development, graphic design, and quality assurance testing.
Mary LeMieux-Ruibal has more than fifteen years of experience in quality management with strength in establishing quality assurance and PMO processes which enhance her clients’ established software development lifecycle.
LeMieux-Ruibal and Capellán will be leading the concurrent session,“Creating a Risk-based Testing Strategy” at STAREAST 2012 Wednesday, April 18, 2012 11:30 AM.