Manage Learn to apply best practices and optimize your operations.

Breaking the bug reporting rules

Software test expert Chris McMahon talks about three rules for bug reporting, but then explains that there are times when breaking these rules are necessary for even better bug reporting. Intermittent problems and other special cases require skill in the way the tester reports the bugs so that the development team can get the best possible information.

There are three rules I recommend to software testers when reporting defects and issues:

  • Report only what you see. Do not report anything that you have not experienced directly yourself.
  • Do not speculate on the cause of the error.
  • Report the simplest consistent reproduction strategy that demonstrates the issue as clearly as possible.
But rules are meant to be broken and sometimes you have to throw the rules out the window. When such a situation comes up, it is good to be able to make an entirely different kind of bug report that still helps to solve the problem.

The classic case of having to throw out these rules is upon encountering an intermittent error, a bug that manifests itself sometimes but not other times. The first thing for a tester to do when this happens is to maximize the chance of encountering the error. This might mean setting certain aspects of the test environment in a certain way, or it might mean choosing test data that share some sort of information that makes it more likely to trigger the error. For example, I was testing a courseware application and encountered an error that was much more common when two test users shared the same academic department.

This breaks the first rule. Because the bug is intermittent, the tester must report something that he does not see: that sometimes the bug is not there and sometimes it is.

Take careful note of any aspect of the situation that might be relevant to the issue at hand. This includes any data, any error messages, any unusual behavior of the user interface (UI) like excessive time to execute a request, unusual display aspects like flickering, anything that might be relevant to the issue at hand.

This breaks the second rule. In the case of an intermittent error, it becomes the tester's responsibility to speculate in a reasonable manner about possible causes. In this situation, any information, speculative or otherwise, might be useful.

Having one intermittent bug often means that subsequent actions following the bug can have dire consequences. Upon triggering an intermittent bug, try hard to find the consequences of such an issue. Such consequences might be corrupt data; or ultimately a crash; or a performance problem. Depending on the nature of the original issue, all sorts of trouble might follow.

This breaks the third rule. In the absence of a reliable reproduction strategy, exercising the application in more baroque ways may expose critical information that a simple reproduction strategy could hide.

Think about your audience
You need to report this information to developers who can address the problem. Whatever information your developers need, it is up to you to provide. In particular, error messages, stack traces, server logs, any sort of code-level information about what happened is critical for the technical investigation that follows the bug report about the intermittent bug.

A real example
Here is a bug report I submitted recently that was praised by several developers:

I've actually encountered a couple of different data errors doing this, but this reproduction strategy is the most consistent. Note that I believe one version of this operation leaves behind corrupt data, so it is likely best to start with a fresh set of instructors in the same department. The error happens more consistently for 2 instructors in the same department.

and click Save.

This operation will either succeed or will cause an exception whose root cause is "Duplicate entry '1111-24680' for key 'barBaz'" where the entry is variable. In the case where the save operation succeeds: and click Save.

This operation will either succeed, or will cause an exception whose root cause is "Column 'fooBar' cannot be null."

If the operation succeeds, the browser state will almost certainly be bad. Find the record again and click the back button and get an exception whose root cause is Null Pointer Error.

This is a good bug report, and it breaks all the rules in interesting ways. Here is why it is a good bug report:

It begins by admitting that there are aspects of the situation that are unknown. Then it explains the data conditions that are most likely to cause the error: "instructors in the same department."

Like a normal bug report, it explains detailed navigation steps. But when it says "will either succeed or will cause an exception" it accurately describes the nature of the possible exception.

From there it continues to list more dire consequences of triggering the intermittent bug. I found that after triggering the intermittent bug there was a logical path through the application that would allow me to execute another save operation having gotten an occasional error on the first save. That action in turn generated a different error that provided even more information about the nature of the original error.

Finally, I reported the ultimate state of the UI, which was in a bad way after triggering two unexpected errors.

When to break the rules
Break the bug reporting rules any time an ambiguous situation exists. Report as much as you can discover, report what you do not see as well as what you see, and speculate on the connections between the important actions involved. Do it with your particular audience in mind, and do it in a responsible way.

This is a skill every good tester should master.

Dig Deeper on Topics Archive

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.