How we got here
There are many different ways to manage software defects. Some teams try to fix bugs as soon as they find them; others convert today's defects into tomorrow's requirements. Some teams "manage" defects with sticky notes, while others, well, mostly just hope the bugs will go away.
This article assumes you are using some sort of system to manage your bugs and triage the ones to fix. I will describe a system I use to report defects, designed to minimize documentation while improving developer/tester relations.
Good versus bad bug reports
Good bug reports are specific. By specific, I mean that they tell the reader exactly what the problem is; not what the general area the problem appears in. To do that, the bug needs to provide a reproduction strategy, telling the end user exactly what steps to take, in what environment, to recreate the problem in the minimal number of steps. The report describes what the customer will see and compares that to what the customer actually expects to see.
Finally, good bug reports are unique. That means we aren't filing three different reports that cover the same bug, nor are we filing multiple reports on different symptoms of the same root cause, or different inputs that cause the same error. We may want to do a little exploring to learn about the defect before filing, to help the developers. One easy way to do this is, and to prevent duplicate reports, is to search the bug tracking system before filing the bug.
That's it. Now, in most cases you will want to provide a one-sentence very high-level description, or bug title. If possible, the title should have all the attributes of the base report – specific, reproduction strategy and what's expected. In most cases, though, you won't be able to fit all of those into the title, so you'll have to choose one. Here are some examples of bug titles, based on a fictional "messaging" website:
IE6: Reply icon is not transparent Type text into field before page is loaded, submit never becomes enabled Reply to deleted signal, see java error Click to Unfollow someone does not unfollow (reload page, see following, again)
You'll notice that I like to pre-load key issues in environment (such as Internet Explorer Six only) at the front of the bug report. Likewise, if the defect is in a particular 'piece' of a very large application suite, I might put that in the title. You could put those in a field in the bug tracking software, and likely should, but putting it in the title gives a good eye-level feel for defects.
There are plenty of other types of information you might want to include beyond this bare-bones report. You might want to include what branch of the code you saw the problem, if the defect is regression (that is to say, if it worked on a previous release but not this one) and some indicators of how severe the defect is, and its priority to be fixed.
Yes, severity and priority can be different -- for example, management might 'schedule' a very complex fix for a future iteration, where a developer might be able to do a quick fix for a less severe bug right now.
In some cases, a particular bug may be hard to describe. In that case, you can click print screen in windows, open MS paint, paste, edit, save, and attach an image of the defective window to the bug report. If the reproduction strategy is complex, or the developers just don't believe you, saying "it works on my machine," you may want to record an actual screen cast that demonstrates the problem. Techsmith makes a tool I highly recommend called Snagit that does an excellent job of making screen recordings and short movies, and it's cheap; about 50 US dollars.
Defective bug reports
It seems pretty obvious then, that bad bug reports are missing some of these elements. Here are a few examples of titles for bugs that might be problematic:
Sometimes the browser 'locks up' Follow a user it totally busted Software does not support internationalization I can't figure out how to reply
The first defect might simply be written that way because the actual reproduction would be too long for the title. But it's likely that the tester simply didn't know exactly what he did to cause the problem. In that case, it's likely to come back with UNABLE TO REPRODUCE stamped on it. So we want to do more than complain; we want to hand our developers bug reports warm, tasty, and ready-to-eat.
If I don't have a solid reproduction but I am able to demonstrate the defect in about five minutes by exploring, I'll title the bug "Screen locks up performing Find user," or whatever operation I am doing to create the lock up. Then I'll note in the bug report itself that a tester can reliably reproduce the problem in five minutes or less, describing the actions that seem to bring on the lock up.
The second and third examples don't actually say what I did or what I expected to see. They might be acceptable if the bug report itself is well-written, but they leave the reader asking questions instead of knowing what do to. The final bad title isn't really a bug at all, or at least, it doesn't look like one. It looks like a usability issue. Instead of filing a formal report, I would suggest getting the developer together with the tester to walk through the reply feature. In some cases, you might loop in a product manager to redesign the feature.
Exceptions to the rule
We started this article with some hard and fast rules about what makes a bug report. You'll recall I said defect reports should be unique. That means that you should never enter the same bug twice, and that each bug report should report just one problem.
Of course there will be times that my overly-simplistic rules do not apply. For example, say you have a particular set of bugs that are extremely similar or trivial. Perhaps the programmers added some new commands to our software, and each command is missing a help entry. I wouldn't create five defect reports. Likewise, I wouldn't create multiple defect reports if the build reports five similar errors, or if I find that that the save, open, and save commands all display the same error. In all these cases, it's likely that the defect is essentially one single root cause, so I would only create one defect report.
Using automation to accelerate software testing in Agile
Can automation improve and speed up software testing in agile development? Yes and no, says test pro Matt Heusser.
Affordable automated testing tools for securing websites
Application security expert recommends best free tools available for automated testing in software applications.
There may be times when you don't have a reproduction strategy, but the problem has been reported by multiple customers. In that case, you may want a 'placeholder' to stick documentation as you pursue a reproduction strategy.
In some cases, the feature may be so defective that even the slightest attempt to use it results in an error. In those situations I like to write the feature is 'busted,' 'borked,' or 'broken' with no explanation. You may disagree (a few of my bosses have), but if the software fails the most trivial of sanity checks on a functional test, I don't feel a need to heavily document the problem. At that point, the feature doesn't need testing; it needs development. Of course, it would be a good idea to determine if the issue might be an environmental issue or a build issue before assuming the code is broken. This might be done by checking whether or not the problems occur in different environments.
So while I started with a list of rules, please keep in mind – they are more like guidelines.
My goal with this tutorial was to provide a firm grasp of bug reporting fundamentals. Applying the fundamentals leads to less time writing documentation and less friction between programmers and software testers, and that means better software released in less time.
If I came close to that goal, then I am pleased. If you want to print out the article and use it to kick off something even better like a brown bag, a discussion, or you own company standards, I'd like to hear about it. Please feel free to email me about it: [email protected].
About the author: Currently a technical staff member at Socialtext, Matt Heusser has been developing, testing or managing software projects for over a decade. He teaches information systems courses at Calvin College and is the original lead organizer of the Great Lakes Software Excellence Conference, now in it's fourth year. You can read more of Matt's writings on his blog, "Testing at the Edge of Chaos".