Thirty years after Glenford Myers published the first edition of The Art of Software Testing, many testers use techniques that directly derive from Myers’ work. It's nice to see a book have such a long impact, but it raises the question -- what's new and exciting in software testing?
James Bach recently addressed the question in his keynote at CAST 2011 in Seattle, titled "New Cool Things."
Bach was quick to point out that these are not all his ideas. Some of them are emergent, some are new, and some are very old, but have rarely been discussed or named.
Here are a few trends and innovations in software test that James Bach is excited about.
Collapse of the factory
The first great trend that Bach spoke about was something he called the "collapse of the factory." He traced the history of "traditional software testing back to Bill Hetzel's book Program Test Methods, which is a collection of essays that were published in the first major software testing conference, held at the University of North Carolina in 1972. After “program test methods,” the term "test factory" emerged in the middle 1970's. James summarized the factory attitude by saying, "Humans are squishy and icky. They're like jellyfish, and we don't want to touch them."
In Bach’s words, "The factory is collapsing. It doesn't scale. It doesn't work." He quoted an article from 1968, called "Testing in the Large," that claimed that the classic test approaches do not scale beyond a project team of 12 members. Meanwhile, today's cell phone might have five million lines of source code on it. He explained this as a trend; projects were failing in the 1990s and only gained a temporary reprieve by outsourcing -- large teams spent so much time maintaining automation that they had little time to test for new features and new uses of the software. Bach predicted that projects run in this way would fail in increasing numbers, thus making the factory “collapse” even as more organizations pile on resources, time and money.
Politics in software testing
The next trend James Bach mentioned was politics in software testing, starting with a new ISO standard, which could lead to a law that all testing conform to the standard and, eventually, all testers would be required to obtain a government license in order to test. Other parts of that trend include Maturity Models and tester certification, which, while not as big in the United States, is increasingly required to find a job in Europe. Bach found this trend worrisome, and suggested that it’s something that all testers pay attention to, regardless of location.
The “intersubjectivity” revolution
Another trend James mentioned was the ”intersubjectivity revolution,” an answer to the idea that objectivity is false -- that what we actually do when we think we are being objective is instead building a useful communication bridge that allows us to cooperate despite our differences. This approach keeps human beings in mind and focuses on communication of aspects that are hard to measure. Instead of counting test cases and failures, for example, James suggests that testers give a score, from zero to "three plus," of the testers confidence on the software. He calls this a "qualitative assessment," saying it is based on personal credibility and trust, and is based on an ethical code; to do it well requires storytelling skills.
In addition to trends, Bach brought up innovations in learning testing.
The major innovation in teaching that James Bach suggested is the use of testing as a learning experience-- that one learns testing by doing it, struggling with challenges and thinking instead of memorizing definitions on a screen. He listed his dice game as a teaching system that is becoming popular, even famous. Several people all over the world have learned the dice game and are developing their own test challenges. James also suggested the online Black Box Software Testing course by The Association for Software Testing (AST), Skype test coaching, and Weekend Testers as new teaching techniques that are experiential instead of driven by memorization.
In addition to ways to learn, James suggested that "Test Framing" is increasingly important as a skill, a skill that encompasses the ability to track tests backwards, to explain why a test is performed, what risks it addresses, why the technique was appropriate and how it reduces risk. James calls this the "story of how the test is framed," and says that having the skill can build credibility in software testing. While testers have been doing this for years, the major innovation here is that the skill of framing is being studied and taught as an explicit skill.
Bach's next topic was on how to teach testing instructors.
Test coaching methods
While James Bach has been a test coach for more than a decade, he pointed out a lack of documentation on coaching methodology. He went on to say, "It's so easy for a consultant in Europe to write down a maturity model, but it is a whole other thing to trace that idea back to an actual case study, a real situation that is closely analyzed. That takes work." He argued that new methodologies must be tested, that methods should be tried as an experiment and, in most cases, they will require a fair amount of change and re-write in order to be successfully adopted by someone trying to read a document.
A second trend within learning is corporate-level testing professionalism. Large companies are developing systems to grow software tester skills instead of forcing testers to follow a defined process.
Rapid testing management, tools and books
According to Bach, Session Based Test Management is crossing the chasm, obtaining broad general adoption and acceptance. He suggested new heuristics, or rules of thumb, are evolving to explain testing in terms of higher-level abstractions. Even regulated testing, which has historically resisted rapid approaches due to audit concerns, is starting to accept rapid approaches, which James cited his own work as well as the recent work of Griffin Jones.
Bach went on to suggest five books that list ideas for software testers: Tacit and Explict Knowledge by H.M. Collins, Dialogue, Skill and Tacit Knowledge edited by Goranzon and Ennals, The Shape of Actions by Harry Collins, Seeing like a State by James Scott, and How to Read Wittgenstein by Monk and Critchley. Notice that none of these books have the word “test” in them, nor are they ostensibly about software testing. Yet the problems these books investigate are central to the problem of software testing: how people think and communicate, the limits of what computers can do, and how process improvement schemes can backfire.
After discussing books, Bach spent a moment or two on cool new tools, including Shmuel Gershon’s Rapid Reporter, Mind Mapping Tools, collaboration tools like Type With Me, using GotoMeeting for collaboration, PDF ReMangler to simplify documentation, or the new free performance test tool Cloud Test Lite.
From the evolution of the field, to new terms, to books and tools, Bach took us on a tour of what’s new and exciting in software testing.