Test pros who took part in the STAREAST 2013 conference in Orlando April 28 to May 3 are back on the job with a new quality control plan: Get a handle on error messages.
Error messages -- a seemingly mundane aspect of the test process -- gives test pros visibility into how users experience software. They provide a window into the overall quality -- or lack thereof -- of the application under test. What's more, they are a reminder that manual testing will remain important, even at a time when quality assurance (QA) pros are increasingly being asked to conduct automated testing. "No amount of automation will ever catch bad error messages," said Michael Bolton, of the Toronto-based software test consultancy DevelopSense. The audience laughed in agreement as he displayed examples of cryptic error messages on the big screen:
- Contact your satellite service provider. Please include your log file.
- An attempt was made to load a program with an incorrect format.
In this article, software test experts who spoke at STAREAST explain what error messages can reveal about the software under test, and offer advice on fixing them.
Communicate with the software user
Software users cannot decipher error messages that focus on underlying technical issues, Bolton said. "Error messages must be customer-facing."
The error messages we are seeing in software today are absurd -- they are ridiculous, added Thomas McCoy, a test manager for the Department of Families, Housing, Community Services and Indigenous Affairs in Canberra, Australia. "I can't believe we are in this state in 2013."
Quality control plan: Take action on error messages
McCoy noted that in some circumstances -- negative testing, for example -- error messages indicate that the software is working as intended, and that stops testers from further exploring them. He gave an example: To validate that a field in a Web application functions properly, a tester enters a future date -- May 1, 2015 -- in a field designed to accept the user's date of birth. The software returns the following error message:
- Field value or value types detects error in date of birth. Refer to Documentation SBL UIF 00299.
The message is so obscure, it's meaningless. Even the developer couldn't say for sure what "Documentation SBL UIF 00299" was, McCoy said. But in cases like this, testers aren't likely to think further about the content of the message because they are narrowly focused on making sure the field will not accept a future date.
"As testers, we think: 'Okay, good, that worked. The field doesn't accept future dates,'" McCoy said. Instead of dealing with the actual content of the error message, testers move on to the next thing, he said.
So, how should the error message read? "It should say, 'You cannot enter a future date in the date-of-birth field,'" McCoy said. "Cryptic error messages cause a lot of chaos for the users of the application, and as testers we have to deal with that."
Business impact of error messages
An error message doesn't have to be cryptic to sound off alarms. Jon Bach, director of Live Site Quality at eBay, received the following message in the midst of buying a new power cord for his iPhone 5 on eBay Now, the company's same-day delivery service:
- We are sorry, eBay Inc. isn't accepting PayPal right now.
"When you get a message like that, you better find out pretty quickly what's going on," Bach said. "Is this a simple state bug -- it turned out it was -- or is PayPal really down?" Investigating the error message immediately, Bach understood that a PayPal failure could grind eBay's auction site to a halt, and also damage the company's reputation. eBay owns the online payment service PayPal, and buyers and sellers rely on it to transfer funds.
Bach completed his purchase a few minutes later. As a customer, he got his power cord. As a QA professional, he got the message: Error messages can serve as an early warning about software problems that could bring the business down.
Bump it up to the next level
When the Department of Families' McCoy approaches software developers about rewriting cryptic error messages, he often encounters resistance: "We can't change that. It comes from the legacy Seibel application -- it comes from the SAP application." When he can't persuade a developer to redo the message, he "bumps it up to the next level," getting the project manager involved. "You have to take action," he said. "It's our job as testers. You can't just leave these messages as they are."
The good news
The good news on bad error messages is that they keep us employed, DevelopSense's Bolton said. "Testing is so much more than checking. We need to keep our eyes on the app, and make sure we get an appropriate result from functions." No amount of test automation is ever going to address that, he said.