We all know statistical data is collected by organizations to measure the success of programs and to identify areas of continuous improvement. However, many organizations are not successful in addressing these objectives.
While senior managers across many organizations love to see metrics, what they fail to understand is the reason for their teams’ inability to produce meaningful metrics that provide clear actionable steps for continuous improvement. Over the last few years, the number of client requests pertaining to establishing meaningful metrics programs has quadrupled, and it is bound to increase even further. So what makes these testing metric programs so challenging? Tons of test data is collected and presented by organizations every day, so what exactly are they missing? All metrics definition and techniques are available in multiple sources, so why do organizations still struggle to collect, analyze and report these metrics?
Thinking about these questions, we started to look at several metric reports put together by client organizations. What we found was not surprising.
- Most testing metric reports were internally focused -- all geared to articulate how great the testing organization is, without any actionable next steps;
- Every metric report had several months of data, with minimal analysis and no actionable items for stake holders; and
- 99% of the action items identified were related to the testing organization -- indicating a siloed approach to the entire metric reporting program.
Revisiting the golden rules of testing metrics management
Based on our findings, we decided to re-establish the golden rules of testing metrics management. These rules are not new. We all are aware of them; it’s just that we don’t implement them.
Rule #1: Every single metrics program should be driven and reviewed by the executive leadership once every quarter. In spite of hearing repeated escalations from business partners on production defects, performance issues and delayed releases, very few organizations have a CIO dashboard which clearly articulates the reasons for quality and cost issues.
Rule #2: Ensure all participating IT and Business groups are in alignment with the metrics program.
Rule #3: Ensure necessary data sources are in place, and data collection processes functional, to ensure uniform data collection. Not following uniform standards will make data collection challenging.
Rule #4: Ensure all metrics collected are trended and analyzed to identify areas of improvements. Metrics data collected is useless if not trended, analyzed and identified with actionable insights.
While it is not easy to follow all of the four golden rules of testing metrics management, compliance is bound to significantly improve the success of any testing metrics program.
What should you look for in a testing metrics program?
Most testing metrics are internally focused, and this is one of the primary reasons for these programs to fail. Not only are they internally focused, analysis carried out on the testing metrics data is used to identify issues within the testing organization only or to defend the actions of the testing organization. This needs to change. Organizations need to look at the mentioned suggestions as to how they can make better use of the metrics data that they have been collecting.
As depicted in the table above, there are clearly two types of metrics in any testing metrics program:
- Metrics which impact the overall project, and hence are very important to the senior management, and
- Metrics which are internally focused, primarily towards the testing organization.
Testing effectiveness, schedule adherence and requirement stability help in the identification of issues which impact the entire project resulting in delays, defects in production and budget overruns. These are therefore very important metrics and should be a part of the CIO dashboard, along with the reasons for failure. Based on these metrics, as a quality goal keeper and in order to ensure the success of the metrics programs, testing organizations should clearly identify actionable improvements for all involved stakeholders.
Defect severity, defect ageing, number of defects, testing productivity, cost of quality, percent automated and many more are internally focused metrics -- important from a standpoint of maintaining the effectiveness and efficiency of the testing organization itself. Clear steps need to be defined to identify actionable items from the analysis of these metrics. The actionable items will be internally focused and will be used to set internal improvement targets and initiatives.
While most organizations initiate testing metric programs to improve themselves, very few actually achieve this goal. The problem is not with the intention of the program, but with the implementation. Like with any other program, organizations implementing metric programs need to be very clear about what they want to achieve. Without fundamentals in place, no amount of data collection or data analysis or sophisticated dash board can help an organization succeed at these programs.
To ensure success of testing metric programs, organizations have to clearly define both the type of metrics and also need to have a clear roadmap for the program implementation, data gathering, data analysis and reporting mechanisms. Senior management buy-in is most critical for the success of these programs, and so is the onboarding of all major stakeholders. While the testing organization is central to this entire initiative, it also needs to assume the larger role of ensuring tracked metrics are properly collected, analyzed and reported, with actionable insights. This activity of the test organization will ensure senior management visibility to the right se t of metrics and will enable them to take corrective action wherever and whenever required.
About the Author
Vasudeva Naidu handles client relationship management for several Insurance and Healthcare clients. With over 13 years of experience, he has helped companies define and implement testing efficiency improvement programs and has successfully deployed multiple Test Centers of Excellence (TCoE) worldwide. He has also championed several testing thought leadership initiatives over the years. Vasudeva Naidu can be reached at Vasudeva_naidu@infosys.com.
For a comprehensive resource on measuring quality, see Quality metrics: A guide to measuring software quality.