News Stay informed about the latest enterprise technology news and product updates.

Testing software in the dark is problematic

Testing late in development and without software artifacts such as the design, architecture and code, makes it difficult to catch and repair bugs and ensure quality software.

ORLANDO -- Imagine you're playing a video game where you're in the dark trying to avoid bad guys and, if possible, kill them before they do any damage. You're running around blind just hoping you'll correctly guess at a target. That's often what software testers do when looking for bugs.

The inability to visualize our products and their intricacies makes testing artificially complex.
James Whittaker
Software architectMicrosoft

Now imagine that you've been given a cheat sheet to see where the bad guys will be "born" so you can eliminate them before they can even think about doing any harm. Such cheat sheets are often possible in video games and are absolutely possible in software testing, said James Whittaker, a software architect at Microsoft.

Testers shouldn't have to run around blind searching for bugs, Whittaker told attendees of the 2008 STAREAST Conference and Testing Expo Wednesday. It would be much better if they had an advanced radar system.

"Why do I have to do black box testing and not be able to see under the covers?" he asked. "Why can't I see this information? It's there. Testers are shackled -- what testers see and what users see is the same."

In order to have quality software, testers need "every cheat sheet imaginable at their beck and call," and that includes the design, architecture and code, Whittaker said.

"The inability to visualize our products and their intricacies makes testing artificially complex," he said. "Testers need all of the artifacts. We should not be Lewis and Clark. We own every bit of what we test, and we should be able to see those parts."

Important to test early
> Another thing that hinders software quality, Whittaker said, is that there's too much distance between bug creation and detection. Consider static analysis and testing, he said, why wait until the program is complete to do testing?

"How about a 'spell checker' for code? It can tell you immediately when there's a flaw and fix it. How about a Right Click and a 'Testmenow' option in the IDE?" Whittaker asked. "We need to move testing into development. We need detection right when a bug is created."

Proponents of development methodologies such as Extreme Programming agree. Elizabeth Hendrickson, the founder of Quality Tree Consulting, said in her keynote address, "Lessons Learned from Extreme Programming" that testers need to be involved in development at the beginning -- in conversations with customers to help elicit requirements and in writing user stories and acceptance tests.

"Testers have an incredible amount of insight that they can bring to an XP team," she said.

However, Whittaker says more emphasis should be on testing during the programming phase. Like spell checkers and grammar checkers that can help a decent writer write very good documents, a bug checker can help a decent programmer writer really good code, he said. And that's who will be writing code in the future -- decent programmers, not excellent ones, he continued.

Software testing resources
Techniques for successful software test teams 

Software testing offers big ROI

How to evaluate testing software and tools

"We need to be able to tell a programmer to write code in two hours and have it be good," Whittaker said. "Testing, like spell checks, will help make that happen. With all of the software that's planned, it's imperative that we get the software right."

Antony Marcano, creator and curator of, brought up the fact that continuous testing is already happening. "I feel that what you described is already happening now," he said to Whittaker.

Whittaker agreed but said it's not enough. "We need to do more. The innovation has to pick up. I still can't see all of the software -- the artifacts," he said.

Dig Deeper on Topics Archive

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.