This content is part of the Essential Guide: Key IT metrics: A CIO guide
Problem solve Get help with specific problems with your technologies, process and projects.

My manager wants quality as a KPI measurement -- help!

So, your boss wants quality as a KPI measurement. You could quit or make something up, but expert Matt Heusser has a better idea.

My manager wants me to have quality as a key performance indicator, so now I need to measure quality as a number. How can I do that?

Well, there are a couple of ways to look at quality as a KPI measurement.

First, you could say you're screwed, throw up your hands and be a victim, because that terrible boss just doesn't understand.

Second, you could make up some really silly proxy metric for quality -- something like average response time on a one to 10 scale, plus 404's per thousand on a scale, plus three other things, divided by five. Boom. It's quality on a one to 10 scale. For bonus points, make them metrics you are certain you can drive up with a little work.

Or, you could look for a third way.

The third way to a KPI measurement might combine finding out what driving the need is for the measure -- ask -- with some education about consequences. You might, for example, ask what the boss would want to see improve measurably when quality goes up: Would it be more sales? Click-through conversions? Renewals?

In my experience, there are three common answers:

  1. "Less bugs."
  2. "You know, quality."
  3. "Senior management wants metrics."

Each one of those has an interesting possibility.

Less bugs

I probably don't need to explain the terrible problems with measuring defects and using them to evaluate the performance of people. However, you could look at how fast the open bug count is rising. If it is rising fast, that means bugs are being created and not fixed. If you avoid turning that into some sort of goal or target, looking at the numbers occasionally at a high level to understand the health of the process in context, with a full analysis -- not a quick graph plopped into a quarterly report -- might work in certain situations.

You know, quality

There's a lot we can do here. You can reject the KPI measurement premise and state that, just as programmers can't do work without understanding the real need, we can't do work without understanding the need. In other words, "No, I don't know."

Those conversations will tend to come back to the inherent squiffy-ness of this word, quality. It is hard to nail down, and that is by design. The ancient Romans had two root words:  qual and quant. And while quant -- for quantify -- meant to number things, qual meant an attribute that defied measurement. Trying to quant qual doesn't really make sense.

You could, however, take surveys of your customers. In my experience, this is usually rejected as "subjective" and "too much work," but it actually is measuring quality in a subjective and actionable way, and actually creates a customer feedback loop.

Or, of course, you can make some stuff up. (See above.)

Senior management wants metrics

Without a tie-in to value or a trigger to decisions, metrics, in my experience, devolve into a vanity exercise. At best, they are a sort of marketing for the department: "We handled 97% of calls, with same-day resolution this month," sounds better than, "We're pretty good."

If I were presented with a request by senior management, I would take that as a good thing, and start talking about objectives and key results, or OKRs. Maybe, instead of measuring schedule conformance -- that is, seeing how well we did against someone's guess before the project even started -- we can measure the value the team or project delivers over time.

Hey, it’s worth a try.

Next Steps

Oh, to be a software tester. Here's what else to look forward to

Want to go lean? Let Matt Heusser walk you through it

The software testing industry's best and brightest sound off

Dig Deeper on Topics Archive

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

What is your favorite comeback when the boss wants to treat quality as a key performance indicator?
"Trying to quant qual doesn't really make sense."

I'd like to offer another perspective here.

Have you watched water freezing? Zero by Celsius is the freezing point, right? But it's also a melting point. So how it works?
When an amount of water is chilled to 0 it doesn't instantly become ice. It takes time and it takes something else - it needs to keep losing energy. That amount of energy is measurable:

In physics of our world quantitative change (in this example - loss of energy) may lead to a qualitative change: in this example, liquid becomes solid matter.

It doesn't work so straight forward with such ethereal things like quality or sociology but it still does work: small assorted changes accumulate and eventually produce a systemic effect.
Quality measurements depend on this simple thing: the value of your tests in terms of their ability to prevent "escapes".  Escapes are those things that your customers find that your internal processes - including requirements development, software development, testing and release - should have found.  It requires a relationship with customer support to get access to the support tickets and an innate, unbiased ability to do root cause analysis to answer questions like, 'what was the real cause of the problem and how could we have found it internally?' and the ability to make changes in the process (anywhere within the SDLC) to prevent those types of issues from happening again.  Maybe it's improving documentation.  Maybe it's beefing up your end-to-end business process testing and not just testing individual "modules".  Or perhaps it developing a inline help or phone-home mechanism for your product.  Whatever it is, the organization has to be committed to preventing process escapes.  The good news is that escapes can translate into a metric that's visible and actionable (e.g. SMART).
I give them Goodhart’s law - "When a measure becomes a target, it ceases to be a good measure."
Hmmm, I don't know. Option #2 sounds pretty attractive!

Our answers regarding what exactly should improve is a mixture of responses 2 and 3. I do believe that customer satisfaction is a valid measure of quality, and surveying customers would be a good method of gaining the data. Senior management does not necessarily agree, however. 
In my experience, when someone asks for a metric, they’re really saying that they want to know about something, but they don’t know what they want to know. That means the third option Matt talks about, asking, is a perfectly viable option. Talk to them to help them figure out what they really want to know, and you’ll be much better positioned to provide useful information. Granted, you’ve still got a long way to go to provide an accurate measure, but that’s a different battle.