My manager wants me to have quality as a key performance indicator, so now I need to measure quality as a number. How can I do that?
Well, there are a couple of ways to look at quality as a KPI measurement.
First, you could say you're screwed, throw up your hands and be a victim, because that terrible boss just doesn't understand.
Second, you could make up some really silly proxy metric for quality -- something like average response time on a one to 10 scale, plus 404's per thousand on a scale, plus three other things, divided by five. Boom. It's quality on a one to 10 scale. For bonus points, make them metrics you are certain you can drive up with a little work.
Or, you could look for a third way.
The third way to a KPI measurement might combine finding out what driving the need is for the measure -- ask -- with some education about consequences. You might, for example, ask what the boss would want to see improve measurably when quality goes up: Would it be more sales? Click-through conversions? Renewals?
In my experience, there are three common answers:
- "Less bugs."
- "You know, quality."
- "Senior management wants metrics."
Each one of those has an interesting possibility.
I probably don't need to explain the terrible problems with measuring defects and using them to evaluate the performance of people. However, you could look at how fast the open bug count is rising. If it is rising fast, that means bugs are being created and not fixed. If you avoid turning that into some sort of goal or target, looking at the numbers occasionally at a high level to understand the health of the process in context, with a full analysis -- not a quick graph plopped into a quarterly report -- might work in certain situations.
You know, quality
There's a lot we can do here. You can reject the KPI measurement premise and state that, just as programmers can't do work without understanding the real need, we can't do work without understanding the need. In other words, "No, I don't know."
Those conversations will tend to come back to the inherent squiffy-ness of this word, quality. It is hard to nail down, and that is by design. The ancient Romans had two root words: qual and quant. And while quant -- for quantify -- meant to number things, qual meant an attribute that defied measurement. Trying to quant qual doesn't really make sense.
You could, however, take surveys of your customers. In my experience, this is usually rejected as "subjective" and "too much work," but it actually is measuring quality in a subjective and actionable way, and actually creates a customer feedback loop.
Or, of course, you can make some stuff up. (See above.)
Senior management wants metrics
Without a tie-in to value or a trigger to decisions, metrics, in my experience, devolve into a vanity exercise. At best, they are a sort of marketing for the department: "We handled 97% of calls, with same-day resolution this month," sounds better than, "We're pretty good."
If I were presented with a request by senior management, I would take that as a good thing, and start talking about objectives and key results, or OKRs. Maybe, instead of measuring schedule conformance -- that is, seeing how well we did against someone's guess before the project even started -- we can measure the value the team or project delivers over time.
Hey, it’s worth a try.
Oh, to be a software tester. Here's what else to look forward to
Want to go lean? Let Matt Heusser walk you through it
The software testing industry's best and brightest sound off