Gray box testing -- not surprisingly -- is a combination of black box testing and white box testing -- but what,...
exactly, does that mean? In his presentation at the Software Test Professionals Conference next week in San Diego, Howard Deiner will shed light on that question. His talk, titled "Trust, but verify -- How Agile testing practices will affect how your organization tests," is expected to explain why test pros should include Extreme Programming (XP) practices such as gray box testing in their software quality assurance program, and how to do it.
Deiner wants to get in touch with how testers feel about the development and testing process and to assure them that testing is a primary component in high-quality software. "The testers are developers in their own right," Deiner told me, "Maybe not of the code, per se, but they're still developing the end-user experience. It's just as important."
White box vs. black box vs. gray box
There are three types of functional testing, according to Deiner: white box testing, black box testing and gray box testing. "White box" means you get to see all the code. "Black box" means the code is all hidden. "Gray box" is somewhere in between -- testers see some of the underlying code but not the whole thing.
White box testing is the testing that the coders will do on their own code, or preferably in pairs. These are the tests that drive test-driven development. "It's not traditional testing," Deiner said, "it's what they do when they're developing the code. In Agile/Scrum, there may be times when the testers do some white box testing, but it's usually not the QA [quality assurance] folks."
Then there's black box testing, which application testers are likely more familiar with. It's mostly about testing the application interface and/or presentation layer. At this point, the developers should have tested to find bugs in the code, so testers shouldn't have to be distracted with any of that, Deiner said. "You don't want to know how it works; you want to see that you get the results you're expecting," he said. This is always done by the testers.
Gray box testing, then, is a mix of the white and the black. It's behavior-driven and it tends to get the whole team -- business and delivery folks -- working together. It depends on the development skills within the testing organization, Deiner said, but QA folks might be leading business users through the initial tests, but there will also be automated regression tests from the development side. The QA team will have an important role in designing the automated tests, even if they're not actually programming them.
You can get the computer to tell you if the software does this or that, but will it tell you whether or not it had fun doing it?
Nonfunctional testing concerns
Apart from the functional testing, there are also nonfunctional requirements, such as performance, scalability, security requirements, and look and feel, among others. It's up to the QA folks to set up an environment where they can test applications along these lines.
It's a harder job because it's all about the real-world experience. XP best practices say in order to achieve continuous integration, an organization ought to run test-driven development with all the testing automated. But practically, that's not possible, Deiner said. "It's a bit difficult to get the computer to test certain things. You can get the computer to tell you if the software does this or that, but will it tell you whether or not it had fun doing it?"
A video game is a classic example of an application where it's a prime concern to the developer that the user has fun, but more and more organizations are making an enjoyable user experience a must-have for every application. It's a subjective process that really amps up the need for human eyes in the development process. "The take-home is to involve QA in the development process early," Deiner said. "Get them thinking like developers, along with the development team." The changes in Agile are wrapped up in the rise of DevOps, according to Deiner, and "QA is really vital in getting DevOps working."