Today's organizations grappling with software quality issues are re-examining process improvement methodologies, development styles, project team size, and the roles of developers and testers throughout the software development lifecycle.
Some organizations are looking to lighter-weight methodologies and more iterative development styles. Some are advocates of Agile development. Some are adopting automated code analysis and testing tools. Some are blurring the traditional lines between development and QA.
The Standish Group, a West Yarmouth, Mass.-based consultancy on IT projects and value performance, recommends new "rules" for quality. These include the following:
- Have a quality focus
- Test early and often; write test scripts prior to code, test often and spell check, and have a formal review process
- Do a better job at the
- Improve existing software through application and code refactoring
- Employ service-level agreements (SLAs) that cover new internal developed code quality
- Have an ongoing investment in skills
- Use industry benchmarks to develop SLAs
- Make use of automated inspection tools
- Use optimization tools throughout the testing process
It's all about building quality into the life cycle, but it's a big challenge, experts agree. Even organizations that have reached Level 5 of the Software Engineering Institute's (SEI) Capability Maturity Model/Capability Maturity Model Integration (CMM/CMMI), and have developers that are trying harder to achieve good quality and are using the best tools, still have defects in their code.
"When you talk to the experts and ask, shouldn't we strive for defect-free code, they'll say that's great but it's impossible. The attitude is that software is inherently defective. My answer is it's no more defective than the process used to produce it," said Watts S. Humphrey, who founded the Software Process Program of the SEI at Carnegie Mellon University and was the director of programming at IBM.
Critics of process improvement methodology say it can be too "heavy" and actually impede the process. "Unless it's well designed or focused on getting the job done, you can slow down the job," Humphrey agreed. "Most problems come from a reactive mode -- you put in place a new procedure, rather than focus on how to do it right and get ownership in the team."
Cem Kaner, professor of software engineering at Florida Institute of Technology and director of Florida Tech's Center for Software Testing Education & Research, agrees.
"We often pretend that a process change is something very simple, that you just decide to do it and exercise discipline and it gets done," he said. "But it's much more. It involves discovering that the process will be beneficial. Each person has got to have some level of discovery for themselves -- ownership is one way to speak of it -- so they gain confidence in [the process]."
However, Kaner said process advocates can alienate programmers by being too heavy-handed. "The adversarial tone of many test organizations and most process advocates who push the notion that you have to baby-sit programmers and project managers becomes a self-fulfilling prophecy -- no one will cooperate with them because they're such jerks."
Joe Niski, an analyst at Midvale, Utah-based Burton Group, also cautions against getting "hung up on quality monitoring and the certification process. That can become an end unto itself. The typical horror story is, you've checked everything off on the list but in practice you're still not beyond Level 1 or 2."
And methodology does not equate to quality, Kaner said. "I don't think quality is about methodology. I think technology groups choose development strategies based on what they believe they need to achieve the mission they've been assigned," he said. "The question of how to get it done is important, but what you're trying to achieve is a big driver. Whether you follow a companywide standard methodology or not, the question is more: Is what I'm doing and my group doing serving the needs of the project and the team, or not?"
Rise of Agile methods
Without a doubt, the emergence of Agile development methods such as Extreme Programming (XP) and scrum are having an impact on development organizations, and agile advocates say better quality software is resulting.
"The Agile movement has contributed a lot of practical things you can do [to improve quality] no matter what methodology you're using," Niski said. Some of these practices are cross-functional and smaller teams, including the customer on the team, fast and iterative development of smaller "chunks, and continuous test.
"In traditional development very often people are fairly undisciplined in tracking the requirements and the code and the size of the software as it grows, and not as good at keeping track of defects during testing. In the Agile world, that's actually more disciplined and not less," said Michael Mah, managing partner at QSM Associates Inc. (Quantitative Software Management) in Pittsfield, Mass., and a senior consultant with Cutter Consortium.
There are a variety of paths to more agile development. A new process SEI's Humphrey advocates is called the Team Software Process (TSP), which "takes the principles of CMM/CMMI and shows people how to do it," Humphrey said. "You measure quality and use data to control how you're doing."
According to the SEI, TSP supports customer needs through the requirements and review process, prototyping and iterative development, and regular customer status meetings. Also, one team member takes on the role of customer interface manager and is responsible for maintaining a focus on the customer's needs. The SEI position TSP as an Agile method.
"Agile, without a doubt in my experience, has been extremely successful," said Jorge Rodriquez, vice president of engineering for the LifeCycle Quality Management (LQM) products at Borland Software Corp. in Cupertino, Calif. "Iterative Agile development is one way we feel the industry will make significant positive gains in terms of quality. The value comes in understanding what the business analyst wants and being able to break down into small bites."
While the industry is starting to recognize and adopt Agile, "Agile is a roots movement; it's team focused. Getting to overlay that onto an existing medium-size [to large] organization can be a challenge," Rodriquez said.
As Borland moves to a more agile, iterative process itself, the company is making customers part of the development process. "We're focused on getting customers what they want," Rodriquez said. "We want to manage our SDLC, optimize it, make it effective, and manage it like a business process. The industry is at the cusp of begging for help to understand how to turn [the SDLC] into managed business process."
Niski also points to the work Ivar Jacobson is doing with Essential Unified Process (EssUP), described as a new "practice"-centric software development process that integrates practices from the unified process camp, the Agile methods camp and the process maturity camp. "He's turning Unified Process on its head," said Niski. "It's admitting failure of a lot of RUP [Rational Unified Process] implementations. Instead of a big process, you start from the bottom up with specific requirements that make sense. You choose documentation that gives you just enough and the accountability you need."
But no method is a panacea. Scrum co-creator and evangelist Ken Schwaber said Scrum in and of itself does not improve quality, but rather exposes poor quality and shows the consequences of it more quickly. "If relationships are wrong, Agile will just produce bad quality results faster. There's nothing inherent in Agile that will cause people to act differently," he said. "It's an opportunity to act differently. If used correctly, Scrum will point out when you're dropping quality or keeping it."
Even when an organization has Agile teams hitting on all cyclinders -- users and developers collaborating and developers building in very fast iteration cycles with continuous testing -- there may be time-to-market pressures that force a deadline, Mah said. "In some cases, there will be an ambitious attempt to build more than what people can reasonably build, but I think that in an Agile environment there's more of a chance of recovering the project at its early critical phases rather than pulling functionality out at the eleventh hour," he said.
Changing role of QA
With iterative or Agile development, the roles of QA and developers are blurring. "The growth in the popularity of unit testing and test-driven development is bringing a lot of QA right into the coding process," said Burton Group's Niski. Traditionally, developers "threw code over the wall" to testers vs. having testers as part of the team with a continuous feedback loop. "I see the blurring of those two approaches," he said. "The tester may be managed through a central test team, but assigned to a specific project."
With scrum, testing is the team's responsibility, Schwaber said. "We ask the team to have an increment of high-quality work done at the end of every iteration. The team consists of designers, testers, etc. We don't care how they do it, he said. QA, he added, is part of the cross-functional team. "The drive of Scrum is to reintegrate [development and QA] so you build quality into the product rather than having people point out you've done it badly," Schwaber said.
Other than early adopters of more Agile approaches, testing and development are still viewed as two camps, Rodriguez said. "What Borland feels needs to happen is the tearing down of walls across the organization. There's a lot of leveraging that can happen with sharing development assets, testing assets, and making sure requirements are done in way so they're testable requirements."
It's about people
No matter what process improvement or development methodology an organization employs, quality improvements depend on everyone being committed to quality within whatever constraints they must work. It also requires executive management buying into a quality focus and creating an environment to enable that.
"The performance of an organization that develops products is fundamentally determined by the performance of the product development group. And the group's performance is determined by teams working on the product. The performance of the team depends on the performance of the team members," Humphrey said.
"A craftsperson determined to do a good job will create something much better than a craftsperson who is not, but both work in a context," Kaner said. "If you give someone four hours to do an eight-hour job, there's a limit on what they can do no matter how motivated they are."