The software industry has struggled with metrics, and a key reason is lack of automation, said Dr. Bill Curtis, director of the recently formed Consortium for IT Software Quality (CISQ). It has taken the software industry "a long time to do what other areas of engineering do—measure." CISQ is holding its kick-off meeting in Europe next month, and its primary goal is developing standard software quality metrics that are computable.
Today, there are a lot of measurements that can go into how maintainable, reliable, robust, secure and comprehensive software is. The problem is that there's no agreement on which tests are best and no standardization there, so comparisons are not apple to apple, said Curtis, co-author of the Capability Maturity Model (CMM) framework and senior vice president at CAST, a New York, N.Y.-based software development company.
CISQ's intention is to drive toward automation to the lower costs of measurement, and to make quality metrics ubiquitous, with the potential of "building a marketplace of competitive products that people can choose from to measure software," primarily business applications, he said.
A partnership of the Software Engineering Institute (SEI) and the Object Management Group (OMG), the CISQ is a response to requests both organizations had been receiving about taking on the task of defining quality attributes that are computable, which many outsourcing contracts are starting to demand, Curtis said. "Existing standards like the ISO give a general description of some attributes, but not a computable metric," he said.
Curtis said the CISQ is not a competitive effort to the ISO, but rather will try to work closely with the ISO and perhaps speed the process. "Industy efforts move faster; the ISO tends to be more academic and slower." For example, he said, the ISO's 25000 standard for
In addition, he said, "the biggest frustration in standards efforts is the lack of executive follow-through to drive the effort. If we get executive support up front, the higher the probability it will get into commercial use."
Thus far, Curtis said, IBM, Morgan Stanley and Tata Consultancy Services have joined as members, and discussions are under way with other Fortune 200 companies. The first part of the effort will be to hold executive forums once a year in both Europe and the U.S. to focus on industry issues related to quality, which will leverage the strength of the SEI. "The SEI is good at getting different parts of the industry together to discuss common issues," Curtis said.
The second part of the effort will be to get the technology folks from the member companies together to work on the standards. "This is what the OMG is very good at. They have a well-oiled, proven standards development process that works quickly and is industry focused."
The kick-off meetings in Frankfurt next month and in the U.S. in December (which was postponed from October) will set objectives and prioritize what the CISQ will work on, he said. "The OMG already has standards in place for how to represent a software measure; what they don't have is specific measures [of quality]. In the outsourcing world in particular, Curtis said, there is increasing pressure to be able to answer the question: "Are you giving me software that's reliable and that I can afford to maintain, or junk?"
The answer to that question lies in standardization of metrics related to software quality, he said. It raises questions of what to count, how to count it, if the measures would differ by language, how adverse conditions may impact the metrics, etc.
Curtis said CISQ will also leverage the security world's body of work on good coding practices, the Common Weakness Enumeration (CWE), a community-developed dictionary of software weakness types maintained by Mitre Corp. Curtis said CISQ will "look at bad coding for all kinds of coding issues, and we need a common, defined way of how to represent a violation, which is what the OMG will work on."
Ideally, he said, companies will be able to leverage the standards to build technology that looks for and identifies patterns of quality violations in the code. And the CISQ will develop a certification for those providing services to assess the quality of IT application software, based on the standards. The certification would be similar to the SEI's work in the process area, "authorizing people who do a CMMI assessment," he said. "In this [CISQ] world, people would be authorized to use the standard to provide quality diagnostics on the code."
The hope, Curtis said, is to spawn a new market with new tools for this function. "CAST is in that market, security vendors are in this market, all providing different ways to analyze the code base. What the OMG has found, is when you have a standard it generates a lot of entrepreneurial opportunities and spurs growth of the technology market. When you have a standard you give the customer base confidence that smart people have thought this through, and if you pick a vendor you're not locking yourself into a dead end. It does spur the marketplace and competition for providing technology."
Curtis said CISQ's goal is to have an initial draft of a quality standard by the end of next year. What form that will take is still under discussion.