Jezper - Fotolia

Manage Learn to apply best practices and optimize your operations.

The secret to asking the right usability testing questions

In the brave new world of software testing, usability can be overwhelming. Expert Gerie Owen guides us through asking the right usability testing questions.

Testers live in a digital world, and in this world customer experience is the key to success. J.D. Power, an organization that studies and rates customer satisfaction across a wide range of products and services, focuses heavily on Web customer experience. Digital customer experience on the Web impacts more than 50% of the customer satisfaction areas included in the overall score. So, how does an organization ensure a high-quality digital interaction with their customers? Usability is the key. Testers, it's a brave new world. Here is how to ask the key usability testing questions.

Although usability testing offers many new challenges and opportunities, the most significant and complex challenge is also the most basic. This is the challenge of understanding the user. Testers must understand the intentions of the customer and their expectations in order to address the vital usability testing questions.

Because usability testing is a core component of mobile testing, it is important to understand the mobile customer and how, where and when the customer will interact with the application. This also includes an understanding of the environmental factors involved in the interaction. In addition to environmental and context considerations, connectivity, including data transfer speed, downloading time and the varying speeds of wireless networks add new challenges.

The scope of mobile usability testing questions expands beyond device/platform/browser combinations. Form factors including variation in screen size, display resolution, and method of navigation, as well as data entry add complexity to usability testing. The logistics of testing, including decisions on actual device versus emulation, field versus on-site, and whether or not to bring in actual potential users present new challenges.

Finally, in our objective world of testing, how do we measure something as inherently subjective as usability? What are the criteria on which our usability tests pass or fail?

How do we, as testers, approach these usability testing questions? First and foremost, we must test early and often. More than any other component of application development, usability is made or broken in design. Poor design choices that become evident during testing are complicated and expensive to fix; therefore, it is critical that testers work closely with the usability designers.

Field testing has the distinct advantage of providing the opportunity to test in context.

Our test strategy and plan for usability testing, like any testing, should include what we will test and how we will execute the testing with a focus on the users and their interaction with the application. Determining what we will test should be based on our understanding of the customer.

So, how do we learn about our customers? Working closely with usability designers, testers can learn about the customer through personas and user value stories. Personas are archetypical users who represent the motivations, goals and expectations for interaction of their respective user groups. User value stories are scenarios in which the user interacts with the application and achieves value or not. Personas and user value stories are powerful tools for developing user-centric test scenarios.

Personas and user value stories provide a basis for our test scenarios; however, we also need criteria for evaluation. D. Zhang, in Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications, suggests nine attributes that are based on ISO 9241 as well as human-computer interaction design standards. They are learnability, memorability, efficiency, errors, effectiveness, simplicity, user satisfaction, comprehensibility and learning performance. Overall, these are measures of how users learn, understand and remember how to use the application, as well as how efficiently and effectively they accomplish their intentions and derive value from the application.

Now that we have our test cases, how, where and when do we execute them? And who executes the test? There are several methods of usability testing. Creating an on-site test lab offers the ability to control the test coverage and it provides measurable results. Using a test lab also provides the option to moderate the test. In this scenario, the test moderator sits with the testers, assists and documents issues as the testing progresses. In an unmoderated scenario, the testers execute the tasks on their own and share their observations to the tester at the end of the test period.

Field testing has the distinct advantage of providing the opportunity to test in context. Jonathan Kohl, founder and principal software consultant at Kohl Concepts Inc., suggests that UX designers should "use the real world as your user interface." Usability testers need to include field testing not only to cover environmental context scenarios but also evaluate their user value stories in action.

What is the most important skill that testers need for effective usability testing? It is the ability to understand the needs, intentions and values of the customer and thereby being able to develop effective usability testing questions. If you are focused on the user throughout the test cycle, you will be testing usability inherently. Testers, enjoy the brave new world of usability.

Next Steps

Does your mobile app cut it? Here's how to find out

Sometimes it takes a village (or a crowd) to test

The case for outsourcing testing

Finding the value in customer experience testing

Dig Deeper on Topics Archive

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

In your organization, what is the trickiest part of designing usability testing questions?
Questions are not "designed". They are to be asked. The software is to be explored for answers.

With regards to usability - always remember that the software is used in the context. Go beyond the device - explore when/where/how the product is used.

Remember about Time. Experienced user vs. new user. Learning (and forgetting!) curve.
What is the most important skill that testers need for effective usability testing? 

In my mind, it's the realization that products designed for the fit, healthy hearty, hale, and all faculties operating perfectly are in reality a minority of users. Most of us have limitations in some form or other, many brought upon by age, others by life circumstance.

Too many products are designed with the usability of the target user as a persona, and that target user is rarely defined by age, temperament or impediment. Much of the usability issues that develop come from overlooking this fact. What works fantastic for a fully sighted person will be painful for someone with astigmatism, or color blindness, or arthritis. Usability to be truly "usable' needs to take those of us that don't sit squarely in the middle into account as well.

Additionally, I highly recommend "Don't Make Me Think, Revisited" and "Rocket Surgery Made Easy' by Steve Krug. These books introduce the broad and often conflicting landscape that Usability is, and how to design, develop and test for it.
In November this year at Toronto Testing Meetup we practiced testing from user perspective using personas and P.E.R.S.O.N.A. guide: Purpose-Errors-Risks-Style-Operations-Needs-Attitude.

A walkthrough with mindmap here:
The trick is to also think outside the box. The big 'what if' comes to my mind. We have a lot of lazy users and people out there. What if they are using a macro to fill data? What if they they put the wrong data in the wrong field? What if they get distracted and their system times out and/or connection drops? What if they do something stupid like try and enter ASCII chars into a text field that are not allowed in the database? This last one was an issue for us. Somebody found the code to put the degree symbol in a description. When the data was transferred to our i-Series, that record became  unusable with invalid data. You could not even fix  it using a DFU.
It's good that the article directly refers UX personas.
What is amusing though - the process description. Some people create personas. Some other people create "usability test cases". Some yet other people "execute" those test cases.

A game of broken phone, really.

Personas should be a starting point in exploration. My recent blog entry:
One of the easiest models that can help with usability is the 5 questions model?  Who is it for, no really what users do you expect to use, what disabilities do they have that could impact them using the product.   What technology do they have to help, and what design considerations can you use to make it right.  When do they tend to use the software, Where are they accessing from (think screen size and location), Why are they coming to your site in the first?  How do they interact with sites which might be competitors.  Each of these questions could help unravel some of the usability issue.