As the ISO, IEC and IEEE prepare for their final vote on the fourth part of ISO/IEC/IEEE 29119, many prominent...
software testers are leading a petition to suspend the standard. Fueled by the #stop29119 hashtag on Twitter, these testing professionals are not only challenging the standards being produced, but the right of the standards bodies to produce them.
ISO 29119-4 is proposed to codify a set of software test design techniques. According to SoftwareTestingStandard.org, "The test design techniques that are presented in this standard can be used to derive test cases that can be used to collect evidence that the requirements of each system under test have been met and/or that defects are present." The standard comprises about 25 techniques which are split into three branches, "Specification-based techniques," "Structure-based techniques" and "Experience-based techniques."
ISO 29119-4 builds on the three parts of ISO 29119 that have already been published. ISO29119-1 is a glossary of concepts and definitions designed to provide a common language with which to discuss software testing. ISO 29119-2 defines a generic process model of the software development lifecycle. ISO ISO 29119-3 defines templates for software documentation that organizations can use to document their software testing.
ISO 29119 is the most commonly used name for the new testing standards which are officially named ISO/IEC/IEEE 29119 and are a collaboration of three separate but cooperative standards bodies.
The group against ISO 29119 feels that any standardization of testing will harm the quality of software produced by organizations that adopt that standard. The group believes that reliance on a standard leads organizations to expect heavy documentation, which in turn drives testers to focus on the production of heavy documentation. This focus on documentation pulls testers away from the real job of software testing, according to Keith Klain, the CEO of Doran Jones.
"Can I show that I've met the standard of what a test case is meant to look like?" asked Klain, "Absolutely. Can I show you I ran thousands of them? Absolutely. Was that testing worthwhile? Was it good testing? Did we get any interesting information? Well, that's a much harder question to answer …" In Klain's view, standards don't actually help answer the good questions. He said standards actually cloud judgment, "because [complying with] a standard, particularly in knowledge work, gives the appearance of being good."
Meanwhile, the standards bodies claim that standards benefit all industries by promoting safety and interoperability. They believe there is more good to be gained from pursuing broad standards than from abandoning them.
The push for standardization has grown out of electrical engineering problems, some of which still haven't been solved today. For example, if someone lives in the U.S. and vacations in Europe, he will probably need to bring an adapter to plug in electronic devices such as a cell phone charger. This is because standards bodies in Europe and the U.S. came up with independent standards that are now too strongly ingrained to be changed. The strength of these national standards makes a unifying international standard impossible. This is a case ISO 29119 is meant to steer software testing around.
Iain McCowattPresident, International Society of Software Testing
Jon Hagar is the owner and lead consultant of Grand Software Testing and the author of Software Test Attacks to Break Mobile and Embedded Devices as well as the lead project editor from IEEE on ISO 29119. He pointed out that standards have to be living, evolving things. "In order to remain active," he said, "any ISO standard has to be reviewed and updated no less than once every five years. For ISO 29119, we're looking at reviewing this more like every two years, at least for the foreseeable future."
A bad standard, the petition signers argue, is worse than no standard at all. "I don't think there is a need for a standard," stated Iain McCowatt, one of the presidents of the International Society of Software Testing and the author of the petition to stop ISO 29119. "There is a need for testers who are dedicated to improving their skills and inventing new ways of testing." McCowatt believes the discipline of software testing is still very young and it's too early to employ standardization.
In Hagar's view, the standards are not meant to be a stopping point for the evolution of testing, but a starting point. He compared the evolution of standards to the scientific method. "This is our current history-based hypothesis," he said. "As we gather evidence about it -- what works fully, what works when tailored, what doesn't work at all -- we'll make better hypotheses in the future." If we don't set down and codify our predictions about the process, Hagar argues, we're not equipping ourselves to get better as a group.
Hagar added that the standards are not meant to define the cutting edge of the testing industry, but the "middle of the road." The standards are guidelines that should allow organizations that adopt them to understand generally what to expect from the testers and/or testing services they employ. The standards are also intended to aid in international contract negotiations where testing processes are concerned.
Hagar stressed that ISO 29119 is not intended to be a guide for individual testers to follow. "This is not the ISTQB," he said, "and it's not the SWEBOK." For testers looking to hone their trade, Hagar recommended becoming somewhat familiar with these standards, but more importantly reading textbooks and such that are tailored to the needs of individual testers.
Protests go beyond the details of the standard itself, and also delve into questions about process.
"What lots of people object to is this attempt by one faction to define itself as being the very embodiment of responsible, professional testing." Said James Christie, whose recent session about the new testing standard at CAST 2014 sparked the current #Stop29119 campaign. "The ISO working group [has] effectively defined those who disagree with them as being irrelevant at best and by implication, […] that those who don't comply are irresponsible and unprofessional."
Hagar sees the current objections as a good thing. "I think there are a lot of good points being made," he said, "and I look forward to putting that feedback to good use in the next revisions." He said he's presented at several conferences over the past few years and that up until now the various testing communities have been rather quiet. "I only wish I had gotten this great feedback two years ago," he said. If so, perhaps the current standards would be less objectionable to their detractors.
James Christie said opposition to the standards is nothing new. "I think it's important to stress that the opposition has been there for a long time, but it's only just recently gotten organized." He said that forming organizations to take action is important because individuals often go unheard or "can be dismissed as irrelevant, disaffected individuals."
Stuart Reid, a major contributor to ISO 29119 has made a few explanatory videos available on YouTube.
Justin Rohrman makes his opinions on ISO 29119 known on the Uncharted Waters blog.
In other news about standards, the UK has settled on ODF as a documents standard.