There are three rules I recommend to software testers when reporting defects and issues:
- Report only what you see. Do not report anything that you have not experienced directly yourself.
- Do not speculate on the cause of the error.
- Report the simplest consistent reproduction strategy that demonstrates the issue as clearly as possible.
The classic case of having to throw out these rules is upon encountering an intermittent error, a bug that manifests itself sometimes but not other times. The first thing for a tester to do when this happens is to maximize the chance of encountering the error. This might mean setting certain aspects of the test environment in a certain way, or it might mean choosing test data that share some sort of information that makes it more likely to trigger the error. For example, I was testing a courseware application and encountered an error that was much more common when two test users shared the same academic department.
Take careful note of any aspect of the situation that might be relevant to the issue at hand. This includes any data, any error messages, any unusual behavior of the user interface (UI) like excessive time to execute a request, unusual display aspects like flickering, anything that might be relevant to the issue at hand.
Having one intermittent bug often means that subsequent actions following the bug can have dire consequences. Upon triggering an intermittent bug, try hard to find the consequences of such an issue. Such consequences might be corrupt data; or ultimately a crash; or a performance problem. Depending on the nature of the original issue, all sorts of trouble might follow.
Think about your audience
You need to report this information to developers who can address the problem. Whatever information your developers need, it is up to you to provide. In particular, error messages, stack traces, server logs, any sort of code-level information about what happened is critical for the technical investigation that follows the bug report about the intermittent bug.
A real example
Here is a bug report I submitted recently that was praised by several developers:
I've actually encountered a couple of different data errors doing this, but this reproduction strategy is the most consistent. Note that I believe one version of this operation leaves behind corrupt data, so it is likely best to start with a fresh set of instructors in the same department. The error happens more consistently for 2 instructors in the same department.
This is a good bug report, and it breaks all the rules in interesting ways. Here is why it is a good bug report:
When to break the rules
Break the bug reporting rules any time an ambiguous situation exists. Report as much as you can discover, report what you do not see as well as what you see, and speculate on the connections between the important actions involved. Do it with your particular audience in mind, and do it in a responsible way.