How do 30,000 software testers get involved in a contest to find bugs in popular software applications? Johnston explains how the uTest Bug Battle competition works, what motivates the community of testers, how the prizes are decided upon and what's in store for the future of bug battles in this interview. Also, check out part one of this interview to find out about the Q3 uTest Bug Battle, which focused on career sites' flaws.
SSQ: When do you plan to announce what the Q4, 2010, bug battle is about?
That would be probably be early November. If you're a member of the uTest community then you can get this information in the uTest forum, and if you're not a member of the uTest community, first of all you should join, but secondly you can always check out our blog at blog.uTest.com. We'll be announcing that normally about a week or two before the bug battle begins and we'll start announcing what's going to be the battle ground, if you will, and we'll start promoting it and getting our community excited and engaged about that particular topic.
SSQ: So do you find there are a lot of people that are part of the uTest community who are not normally testers, participate just in the bug battle?
Yeah, absolutely, that does happen from time to time. So a lot of people might find themselves too busy to participate in our usual paid projects, but something like a bug battle might be a little more interesting and fun. The potential prize money might be a little bigger. Like I mentioned before, we give away about $4,000 per bug battle in prize money. That prize money is allocated across the best bug, which is based on criticality and severity – we give away five prizes based around that, we give away five prizes based around the best user feedback and usability feedback. So we have a lot of fun with it. And we do find people that don't necessarily participate every week in our paid projects for customers, but do participate in bug battles. It's also a great way for us to get exposure for new testers, for people that haven't started to build a reputation within the uTest community yet, that haven't earned a rating yet because they haven't been involved in paid projects. We've found a number of promising testers every bug battle, people that have had their opportunity to shine and get on our radar.
SSQ: And so even if they don't win a prize that can still earn reputation points for submitting valid bugs. Is that correct?
Absolutely! They can get on the radar and they can find themselves in the ensuing weeks invited to other projects because they've made an impression by being in the bug battle.
SSQ: When you mention best bug and best feedback, is there a panel of judges and are they a part of uTest staff? Who decides the winners and what are the criteria?
We have a panel of volunteer testers from the community who will go and test each bug and every bug is reviewed by three testers who give it a thumbs up or a thumbs down for further consideration. Then the uTest staff will take a look at the finalists. We involve the community in the vetting process and then we determine the finalists. Then everyone is allotted a certain number of bugs and then they'll go through and find the ones that they found to be particularly severe or critical or that showed a lot of testing skill in terms of where that bug was hidden and what they had uncovered. We would narrow it down from there and allocate the different prizes – who had the best and second best. Then who had the best user feedback. We give out 10 or 20 honorable mentions for people who did a very good job and they get a smaller cash prize.
SSQ: Does everyone who submitted a bug get some kind of feedback or acknowledgement?
Within the judging process, no, but one thing we've started is in the forum is that everyone who participated will go in and talk about their favorite bug that was submitted. And they will start a debate with other members, some of whom have been testing for 10, 15, 20 years. Some of whom may have been testing for 10, 15, 20 months. A lot of times it turns from the caliber of the bug to the bug reporting itself. What they liked, what they thought was missing from other submissions, so it does become a collaborative learning experience for everyone involved. I can't tell you the number of times we see conversations like, "How did you document this?" "Well, I got that from a log file" or "Why did you use video capture instead of a screen shot?" and then they might answer, "It makes it easier for customers to recreate the bugs I report." You just see the light bulb going on for people as they learn these best practices, not from uTest, but from one another.
SSQ: Right. What about tools for reporting? Are those provided by uTest or does everyone use their own tool to document and report bugs?
uTest has a testing environment that's Web-based so people can use it to report bugs and in there they would name the bug, identify the type, what the severity is, and then what the frequency of that bug is. So all this information gets structured and then they describe the action that they took the result they expected then the results they got. They can also upload screen shots or video capture to help document that bug.
SSQ: Are there benefits to uTest? Why do you put all your energy and time into bug battles?
We have a community of 30,000 people and we're about 30 employees so we're always looking for opportunities where we can engage not only 10 testers at a time but with hundreds or thousands of testers and this is one of the things we spend between 15 and 20 thousand dollars per year in terms of prize money and countless hours. But we think it's a good investment in our community, it's something that they really enjoy and it's something that engages new testers. We've found it's a very good learning and mentoring opportunity within the community itself between experienced testers and novice testers. So at the end of the day, the reason we keep doing them and the reason we do them once per quarter is because it offers an opportunity for engagement and learning within our community of testers.
SSQ: Are bug battles going to continue indefinitely?
We evaluate everything year by year, but we intend to keep doing them every quarter. We intend to keep going forward with them, the community really seems to enjoy them. They're valuable, we get great feedback. They'll go on until the community expresses they're not interested or until we come up with something better. So until we find a better idea, we'll keep hammering these out.