What makes a good objective measure of an application's user experience?
User experience generally would be considered an aspect of usability. Ironically, conventional approaches to measuring usability frequently overlook user experience or only treat it superficially.
Common measures of usability include:
- Time it takes to do the function
- Time to learn the function
- Number of errors doing the function
- Number of complaints about the function
These all may contribute to the user experience but ordinarily are not measures of it. Moreover, measuring usability needs to go beyond the specific usage because other factors can change the overall user experience. For instance, I once had a very negative user experience when I could not install the annual update to my Norton security software until I was won over by an offshore tech who spent several hours taking over my computer remotely and getting the new version to work perfectly.
An objective measure that may signal a poor user experience is abandonments, but simply knowing about them is not helpful without understanding why the user abandoned the site; and finding that out is practically impossible once they are gone. Conversely, good experience may be indicated by users who perform desired behaviors, such as buying the product or recommending the site to someone else; but again, you need to know more than just the externally observable behavior.
Caveats to measuring usability
It is important to learn about the user's experiences and subjective interpretations of them. Unfortunately, actually talking with folks takes more time and skill than many organizations are willing to commit. Instead, it's probably most common to rely on a survey or questionnaire, which are notoriously unreliable and frequently produce invalid findings.
Low survey response rates and unrepresentative user sampling can render results meaningless.
Low survey response rates and unrepresentative user sampling can render results meaningless. The precise wording of questions can shape answers such that they are interpreted differently from the questioner's intent, lack a shared basis for interpretation (what does "excellent" or "met expectations" really mean?) or lead responses to reflect the desires of the questioner ("How much did you love our system?") rather than the true user experience. Consider all the surveys you've taken where none of the choices matched your true answer or where you've scored everything high regardless of your real feelings.
User experience is shaped by a variety of factors. Not long ago, some website builders felt compelled to clutter sites with extraneous graphics and gizmos on the premise that they improved user experience. At a time of lower connection speeds, the extra features probably interfered with the user experience. Distracting whirling doodads are pretty much gone these days, but I've noticed that several recently redone sites feature soothing nature photos that seem to have nothing to do with the page's content. Perhaps all it takes is a kitten video to make a wonderful user experience.
This was first published in December 2013