How do test managers best measure the effectiveness of their team?
In a word: “Cautiously.” An awful lot of managers (at every level) will throw their hands up in frustration because these are hard concepts to work with if you are going to try and find meaningful measurements around them. Others will take whatever they are handed by their superior or HR department or some widely published expert or guru speaking at a conference.
I find some of the experts/gurus/hired guns at conferences or “industry training events” to be pretty disingenuous about such things as (first) defining an idea such as “effectiveness” and (second) measuring it. Some will fall back to an “industry standard” argument. Whenever I hear people refer to an “industry standard” of anything involving software, I automatically think (and sometimes say out loud) “What industry?” The context must be considered. When the “authority” you are referencing in turn refers to “industry standard” without regard to context, proceed with extreme caution.
Teams and individual testers will all have strong and weak points about them. Measuring these points may be a challenge in that many times, cookie-cutter metric systems don’t give the manager the whole (or realistic) picture. Likewise, no single approach to address these will look for every single team. Your problems and strengths are unique and so are the solutions.
I start with a basic question: “How do you believe the team in question is operating?” This is not a “go by the numbers” exercise. It is a simple idea: Do you think the team is generally OK? Are there some areas where you feel/believe the team could do better? Do those areas outnumber the areas where you feel/believe the team is pretty good? How about areas where you feel/believe the team simply rocks? What can be done to improve the weaker areas without hampering the areas your team is good at?
If a numeric value is needed, I look to customer satisfaction. This can be internal customers or external. Looking at defects found in production is one aspect of this. I recommend caution here as well. If the testers were overridden in concerns or some other aspect contributed to the defects (e.g., the testers reported a defect and it was decided to not fix it.) True, the overall satisfaction is a reflection of the entire development team’s effort.
The effectiveness of any team is a challenge to measure. The hard part is to look for ways to measure information that helps managers learn and lead your team better.
This was first published in February 2012