How to find value in customer experience testing

The year 2016 will be the year of customer experience testing. Expert Jennifer Lent explains why and how we'll stop guessing and instead focus on what customers do with apps.

As 2015 gives way to 2016, a new idea is taking hold: customer experience testing. I think this is a crucial focus for 2016, but it's the latest variant of an old idea that software pros have known all along: Applications won't succeed unless they do what the customer wants.

What does it mean to test customer experience? The 2015 -- 2016 World Quality Report defines it as "a combination of more behavior-driven testing (where the focus is determined by understanding or analyzing the actual end-user usage of different software application features) and more exploratory user scenario-based testing."

But here's the best quick take on customer experience testing: "Testing based on actual customer use, instead of expected customer use." It comes from Christopher Willis, chief marketing officer at mobile test services firm Perfecto Mobile, in Woburn, Mass.

I like his definition because it forces software pros to step outside their comfort zone. Typically, exploratory testing is based on expected use, as defined by software requirements and the product owners and testers who drive that process. If we want to base testing on actual use, well, we have to get close to people who actually use the software. Not exactly a new idea, right? But it's nonetheless a radical departure from how most software testers work today.

In this edition of Quality Time, I look more closely at this process and raise two issues that must be addressed to conduct effective customer experience testing: efficient feedback loops and convincing high-ranking business executives to recognize the pivotal role quality assurance (QA) pros play in delivering a high-quality customer experience.

Testers in the field

The job of the smart tester is to represent the customer and the customer's domain.
Yuval YeritCTO, AgileSparks

I asked Agile coach Yuval Yerit, CTO of consulting firm AgileSparks in Rosh Hayiin, Israel, the best way for test pros to take on customer experience testing if they haven't done it before. His simple answer: Put yourself in the customer's shoes. "When you're working on a mobile banking app [for example], it's easy to relate to the user. You're a client of the bank; you use the app." In other words, to understand the customer experience, you have to immerse yourself in that same experience.

Getting firsthand customer experience is more challenging for apps aimed at narrow or specialized customer segments. If, for example, your target users are specialty engineers who do fieldwork, you have to spend time with them. Work next to them. Find out what they do, what they need from the software under development. It's a radically different role from the one most professional testers play today. "But the job of the smart tester [in a time of increased test automation] is to represent the customer and the customer's domain," Yerit said. This is where QA is headed, he said.

Effective feedback loops

A customer's experience with the software is constantly changing -- no tester can capture the full range of those experiences, and dozens of factors impact them, particularly on mobile devices. Customers need effective channels to convey key information about their experience back to the company behind the application. Most organizations have those channels -- live chat, email, phone, social media or some subset of them -- in place. But how effective are they? And what happens to the feedback an organization receives? If a customer can't complete a purchase on a mobile device, a phone rep will no doubt do it for them. But was that problem captured? What happened, and why? Does that information make its way back to QA? How can we improve the customer experience if we don't understand exactly what that experience is?

Shifting the QA mission, starting at the top

Focusing on customer experience testing involves both retooling the QA process and enhancing the customer feedback channel back to QA. This requires leadership and commitment from the highest levels of management. It also means managers have to act on a new understanding: Software testing plays a pivotal role in assuring high quality customer experiences. "They [senior management] have to realize the tester's role is not to find defects," Yeret said. "It's not an efficient use of skills."

Instead, re-focus QA and testing on customer experience and business assurance, according to the World Quality Report. Observe customer behaviors and identify areas that interfere with the customer experience. Improve customer feedback systems to route information about application issues back to QA for confirmation and retesting to assure smooth business operations. All this requires serious attention and commitment from top management.

Recognition from top management and focus on the customer: Two ideas software pros have been talking about for a long time. Will they finally take hold in 2016? We'll see.

Next Steps

Why customer experience testing is not for sissies

In the mobile market, testing stakes are even higher

Another way to get inside a user's head

Dig Deeper on Topics Archive