SearchSoftwareQuality.com's recent reader survey found that while some long-standing problems application development and test professional face continue to be a challenge, some newer technology and methodologies are starting to reap benefits, in particular the influence of agile development and virtualization.
Interest in agile is on the rise among respondents, with 42% planning to implement agile processes within the next year vs. 17% in 2009. And Scrum is the dominant methodology (45%), with both XP and XP/Scrum hybrid at 9% and other methods in the single digits.
The top four benefits respondents report from agile are consistent with the prior year's list: increased productivity, faster time to market, fewer software defects and reduced development costs. Of note, 73%of respondents cite productivity improvements vs. 66 percent last year, and the number of respondents reporting reduced development costs also increased, from 28% in 2009 to 41% this year.
"Agile is becoming mainstream to some extent; we're seeing that with respect to the tools, and the way agile is described," said Dave West, a senior analyst at Forrester Research Inc. "At the conferences there are less religious purists. I think agile has now crossed the chasm, and that means agile is probably changing. It's not out-of-the-box agile, but the principles and virtues of agile are being followed."
Ovum principal analyst Michael Azoff said most organizations are adapting agile to their work practices. "Agile is spreading in a big way in that sense. Even if you're doing waterfall, you can take testing upstream and do it a lot earlier."
Mettler Toledo Turing Softwares, in Coimbatore, India, which develops products and system solutions for precision measuring devices, employs agile, but not in a "textbook fashion," said Arunkumar Balakrishnan, senior consultant and a survey respondent. "We believe in short iterations [and] customer interaction, but we also believe in a deep analysis and a consolidated design phase prior to releases. The embedded nature of the software that we develop requires and provides the environment to use this approach."
Another survey respondent, Pekka Marjamäki, a test designer at Logia Software Oy in Finland, said his company has tried agile "in a few projects, and the more we learn about it, the more we can improve our processes and testing. We may not be the pioneer of agility, but we try to learn as we go. This sometimes causes the process to turn 'inagile' and toward waterfall (we call it Chaos-Scrum) and thus increase the workload of testers and developers alike. But we [aren't] born experts, are we?"
But the more things change, the more they stay the same, an adage that some of the survey results support. For example, respondents cite the top challenges of adopting agile processes are some age-old issues: documentation, communication and resistance to change. However, while documentation was the top challenge for agile adopters for two years in a row, the percentage cited it dropped from 67% to 52% this year.
Forrester's West chalks up the documentation issue to resistance to change: "[Developers] don't want to do it and their managers want them to. We find teams have adopted agile, but there is a lot of friction at the middle management/PMO/governance level."
On the other hand, some teams may have taken on agile with the idea that they wouldn't need any documentation, said Azoff, "then they were faced with the reality that you need some documentation. The [agile] manifesto talks about sufficient documentation; the core XP people say the code is the documentation. But I think that's extremely naïve because there are a lot of other stakeholders in a project who won't be digging into source code." Some documentation is also necessary from a maintenance point of view, he added.
Requirements hard to get right
Asked what tools are essential to agile development and what tools are essential to development in general, respondents report the same top three: requirements management, bug tracking and project management.
According to Azoff, "One of the most successful areas of ALM is agile tools that support agile processes, in particular the project management side of things, which includes support for stories, whiteboarding electronically, etc. Although those [survey] answers are the same, the types of tools are different. For traditional development, it's a traditional requirements management tool."
Requirements, no matter the methodology, continues to be a persistent problem for organizations, as does process improvement and testing. When asked which part of the application lifecycle their organizations have the most difficulty with, requirements gathering, followed by process improvement and software testing/QA, topped the list for the past two years.
"Requirements gathering may be the biggest thorn in our foot," Marjamäki said. "We have a specialized group of solution consultants who try to gather requirements from customer needs. This, however, does not a good requirement make. We might have to deal with unreviewed documents that have English and Finnish mixed together. The communication sometimes happens via email, so not every update to requirements is transferred to the document. The test group needs to read through the whole project documentation, sometimes very late in the project lifecycle to gather the most recent requirements, given that we aren't working in an agile environment. The test designer, with hastily broken down requirements, tries to make good test cases that benefit both system testing and acceptance testing."
He continued, "Because the procedures are hard to change in higher parts of the organization, we have tried to manage the lack of good requirements by moving towards exploratory testing."
"Organizations are not willing to invest in proper processes and even tools to help with requirements definition up front, because it takes commitment, and it won't give you a quick ROI hit," said Theresa Lanowitz, founder of analyst firm voke inc. "People are not investing in strategic requirements, and we hear over and over that requirements are the most strategic thing you can do for business."
Requirements represent a core aspect of ALM, Azoff said, so the fact that it's a difficulty suggests ALM isn't fully delivering. The ability to have requirements traceability throughout the lifecycle "is difficult to do when you don't have a fully integrated system, and one of the difficulties in the ALM market is that companies are not going to buy everything from one vendor, so the tools need to interoperate. That's one key challenge in the ALM market."
In terms of process improvement, Lanowitz said it takes a similar commitment to approaching requirements more strategically. "You won't see a quick ROI. There has to be a long-term strategy and you keep modifying it; processes are not static. They're very much like the application lifecycle, dynamic, and [process improvement] requires top-level management support."
Testing is an area where organizations can invest in solutions and get a quick ROI, Lanowitz said, "but testing people have to be given some latitude and time to invest in tools and figure out where they want to make improvements. Keeping the testing organization in a tactical mode holds them back."
Marjamäki echoed Lanowitz: "The reason why requirements gathering and testing are low in priority is possibly the acronym ROI. If you can make something cheaper and with less manual labor, then it is considered better for business and for quality; that is, better cost/quality ratio. In addition, testing, and QA in general, is considered a development-supporting action. Testing and QA should be considered as separate parts of the lifecycle of a product."
When asked which area of software testing their organizations are having the most difficulty with, respondents report integration testing and regression testing as the top two (both at 20%).
Marjamäki said regression testing may be considered the hardest "because it's the first area to be automated. Companies that don't have an automation specialist may confront difficulties from that area because manual regression testing is painstaking and hard. You do [make] mistakes and skip things when testing manually. Regression is usually left undone if not properly planned and automated."
He said integration testing is also an area of difficulty for his company. "The nature of integrating things together is very complex. Be it integration of two components or multiple large complex systems, integration testing still needs to be done to verify that all work together as expected. This requires proper tools and environment to be done correctly, and these are both expensive and require lot of experience and skill."
An area of testing where organizations can invest and get a "big quick hit" is virtual lab management, Lanowitz said. According to survey respondents, 59% test applications in a virtualized environment, up slightly from 57% last year.
With virtual lab testing, "you eliminate the expense of building out physical labs, you see increased code coverage, less defects going into production, and faster time to market. You get a quick ROI hit and it frees up testers from doing tactical work," she said.
Today, about 50 percent of respondents report that their organizations also use virtualization technologies as part of the application development process. Forrester's West said "most organizations put CRM systems in the cloud first, but increasingly we're seeing the use of virtualization and cloud resources to support more intelligent software delivery processes, particularly test. This requires some sophistication around test data management."
Balakrishnan said his company uses virtualization for software testing, and "during our development phase to ensure that the code works on different platforms."
While server virtualization has provided the data center with some fast payback, Lanowitz said it has not yet had the same impact across the application lifecycle, mostly because there has been no large ALM or test player in this market. However, she said the inclusion of virtual lab capability in Microsoft's Visual Studio is significant. "Once people realize they can test as close to the production environment as possible, that it's self-servicing, that you eliminate the back and forth with development, and once they see mainstream [ALM] vendors like Microsoft, this will take off. And Citrix acquired VMLogix, so there's a big opportunity here as well to put the notion of virtualization into the application lifecycle and beyond."