Manage Learn to apply best practices and optimize your operations.

How to make crowdsourced testing actually help the development process

The trouble with paying bug bounties to outsiders for testing help is doing so can create an adversarial relationship between developer and tester. Expert Robin Goldsmith offers advice.

In a famous "Dilbert" comic strip, the Pointy-Haired Boss offers to pay a bounty for each bug detected, whereupon...

Wally responds, "I'm going to code me a minivan." The main message, of course, is the number of bugs the developer -- hopefully unintentionally, other than Wally -- puts in the code is the major determinant of how many bugs are detected.  Testing techniques and skills are secondary. That is, the best tester in the world cannot find a bug that isn't there, and the weakest tester can't help finding bugs in dirty code.

However, the "Dilbert" strip doesn't address several other critical aspects of crowdsourced testing and bug bounties. Many organizations have used a variant, often called pelt counts, where the tester's performance is measured by the number of bugs he detects. While seemingly a brilliant management use of metrics, in fact, it's often misguided and instead can actually cause additional undesirable results.

Part of the issue is pelt counts make testers focus on quantity, rather than quality of bugs.  Such a shifted priority can divert limited test time and effort to detecting lots of often trivial bugs, rather than hunting for those that may be harder to detect, but matter more.

The bigger issue is pelt counts create an adversarial relationship. In order for the tester to succeed, the developer must fail. Instead of concentrating on writing clean code, the developer often pays more attention to preventing the tester from reporting the bugs that are there. The most common symptom is lots of time wasted arguing about whether or not something is a bug.

An effective -- but, unfortunately, seldom-used -- solution is to shift the metric's focus to the user through crowdsourced testing or other methods. In that scenario, both developers and testers are measured by the number of bugs the user encounters. The fewer bugs the user sees, the better both the developer and tester are evaluated. The fewer bugs the user sees, the better both the developer and tester are evaluated. That's a rational win-win-win.

Relatively recently, bug bounties began to be applied in a different way -- to testers outside the development process and, thus, usually with no reflection on developers. Sometimes referred to as crowdsourcing, the benefit is getting lots of eyes testing the application often in lots of unlikely-to-be-tested situations, but paying only for unique bugs that actually are detected.

Many times, those involved in crowdsourced testing invest their time with no reward, which similarly can have unanticipated consequences. Moreover, because they are unlikely to know the business domain, even a large group of crowd testers may still find only superficial bugs, while missing more important ones. So, as usual, "Dilbert" makes a very good point: Bug bounties have to be handled carefully, and crowdsourced testing may be one option.

Next Steps

Bug bounties in the Friendly Skies

Interested in getting started? You might try a third party

Bug bounties are not just for regular bugs anymore

This was last published in July 2016

Dig Deeper on Software Testing Methodologies



Find more PRO+ content and other member only offers, here.

Join the conversation


Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.

How does your company handle the conflicts inherent in a bug bounty program?
Note: I have over 40 years experience as a consulting software engineer for some of the most advanced bleeding-edge s/w projects in the country.

First, the primary purpose of QA is to catch bugs as early as possible in the development life cycle, when it is cheapest, quickest, and easiest to fix with minimal effect on the rest of the code base.  QA should be involved from the beginning in iterative requirements elicitation, analysis, architecture, system design, and coding.

Second, one of several reasons programmers make poor testers of their own code is that they subconsciously want the code to work and believe it will. Unit testing has been shown to find only about 25% of the bugs.  A friend once came up to me after his program crashed in production saying, "Who'd have ever thought a user would put a negative number in the Age field?"

Third, by the time you get to system and beta testing, the goal should not be in making the product better, but in validating that you have already produced a quality product.

Fourth, putting not-fully-tested s/w into production to let users act as beta-testers and catch bugs for bounties will damage your product's and your company's reputation.  Does anybody remember how rushing dBase IV into production early killed what was then the gold standard in PC database management?

Fifth, users are rarely sophisticated enough to properly qualify an anomaly they have discovered.  At the very least, have a website with a form that will guide and prompt them through the myriad details developers can use to reproduce and/or locate the source of the anomaly.  Have developers contact users if necessary to elicit more information.

Don't put out a product that is not robust and of commercially acceptable quality. Your reputation only has to be tarnished once and a dozen competitors are there to take your place.
Beartooth, thank you for all the fine points which very much echo my sentiments. In fact, in some of my seminars I do cite Ashton-Tate’s dBase demise, but of course I have to explain who they were to most attendees who think computers started with iPhones. I think you would find my Proactive Testing™ and Proactive SQA™ methodologies go well beyond conventional awareness of how to participate early enough not only to catch errors but also to prevent them. I fully agree that crowdtesting should be a supplement, not a substitute, for competent internal testing. One important distinction that the main article perhaps didn’t make clearly enough is between paying bug bounties to internal and external testers. It’s only internal testing where it’s possible to reward both testers and developers for NOT delivering defects to their users in the field. With external testers, such as crowdtesters, bug avoidance is not measurable, so bounties have to be based on the value of bugs detected.
Reminds me of the Cobra Effect, when people who were paid to catch cobras were found to be breeding cobras in order to guarantee a supply to catch. Makes me think people might be trying harder to break the program than to actually use it.