“We need to learn to tell stories well,” says Michael Bolton of DevelopSense, Inc., who gave a keynote speech titled, “Evaluating Testing: The Qualitative Way,” at STAREAST 2012. He suggests that software testing can be viewed through the lens of qualitative research, and that in fact these two disciplines have several parallels. Testers can apply much from qualitative research, which is, according to Bolton, “about identifying if something is there or not, and the degree to which it is there.”
Though Bolton prefaced his speech by saying he is not necessarily a qualitative research expert, he provided carefully chosen sources for much of his information, including Qualitative Research Design: An Interactive Approach by J.A. Maxwell, TED talk from Wade Davis “Dreams from Endangered Cultures,” and Software Testing as a Social Science by Cem Kaner, among others.
Rather than focus on categorizations or frequencies, qualitative research instead focuses on meaning, on observation, on making distinctions, on description and narration. Social scientists like anthropologists study people, putting human values first and observing participants. Software testers, likewise, can better understand the project they are working on by remembering that quality is the value that the project has to a person or people; it is also ultimately a human study.
Software testing in context
Perhaps one of the most salient parallels between social science and software testing is the fact that like the human subjects being observed and reported on in qualitative research, software testing revolves around a set of variables that can be unpredictable. Furthermore, these variables are not isolated; they live within a context. Bolton broadens the definition of a computer program from simply (and obviously) “a set of instructions for a computer,” to encompass communication among several contributors and as a tool that delivers value to people.
“A program is far more than its code. A software product is far more than the instructions for the device. Quality is far more than the absence of errors in the code. Testing is far more than writing code to assert the other code returns some ‘correct’ result,” Bolton says.
He explains that software testing involves people, computer programs, products and the relationships between all these components. He likens excellent testing to anthropology; excellent testing should be interdisciplinary, systems-focused, investigative and storytelling. Testers have the task of telling three related stories: a story about the product, a story of how they tested it and a story of how effective that testing was, all with the purpose of relaying a “real story about the product.”
Gathering information and examining it critically
Qualitative researchers use a number of methods to convey their findings, such as memos, concept maps, mind maps, self-reflection similar to journaling and personal notes. These documents all inform the research findings and require critical thinking to create.
Bolton explained that the process of investigating a bug is much like writing a news story. The tester begins with identifying the bug’s effect on people, interviews some people, forms a grounded theory, refines the research questions, engages with the product, observes further, conducts a critical literature review and constructs a narrative.
“Testing is not an artifact; it is an activity. It is a performance,” says Bolton.
Like qualitative researchers, news reporters must ask many questions. The questions that news reporters ask are often similar to questions testers must ask: who, what, when, where, why and how? Testers must engage with the product, identify the benefits of it, determine the value that it provides to people and fix bugs using this overall context.
Then testers may find that their story resembles the performance of a news reporter: it has a headline; it involves cyclic elaboration of the story, it has coherent narration, background information and illustrations; and it is essential to consider the sources from which the writer has acquired his or her information.
Because published information is generally a step behind the most innovative ideas that people are currently imagining, Bolton emphasizes the importance of communicating with people at conferences, via blogs and on social sites like Twitter.
A note about exploratory testing
Bolton maintains that exploratory testing is a complex cognitive activity, and that it goes far beyond the misconception of “fooling around” on a computer. It is through qualitative, exploratory research that we find surprises.
Presenters Lanette Creamer and Matt Holcomb also presented on exploratory testing at STAREAST.
Bolton concluded his keynote by drawing one last parallel between social scientists and testers—they both must manage trust. Their respective audiences rely on them for critically examined information. “If testing’s goal is to reveal quality-related information, it would serve us well to sharpen our skills in qualitative research,” he says.
For comprehensive conference coverage, see our Software Testing Analysis and Review conference page.