The times they are a-changin' for QA engineers and testers -- or at least they should be if their organizations are developing service-oriented applications.
According to a recent Aberdeen Group report, significant change is afoot in quality assurance, with the focus turning to quality throughout the end-to-end business transaction. On top of that QA needs to add to its responsibilities of unit and functional testing and take into account integration, regression and business process testing, as well as performance and security testing. In addition, service-oriented architecture (SOA), composite applications and Web services require QA to be a continuous process, and testers need to know more about both the technology and the business requirements.
"Folks taking a transaction view -- how the services are interacting across the enterprise -- those guys are by far outperforming everyone else in terms of quality," said Perry Donham, director of enterprise applications research at Boston-based Aberdeen Group Inc. and author of the report "SOA and Web Services Testing: How Different Can It Be?"
Along with a transaction view, testing organizations must also consider the business view, "a theme running through my research," Donham said. As the emphasis has shifted from IT driving business to business driving IT, IT is turning to SOA and Web services to provide the flexibility to respond to those business requests, he said.
"So you can't treat a service or set of services as self-contained; they interact across the business, and you need to understand the business view of the transaction," Donham said. "Functionally it may work and the unit test works, but from the business process it doesn't work. A lot of folks are missing that. At the end of the day, you want to solve some business problems."
According to the Aberdeen report, which surveyed 240 end users, tracking business requirements is a key component of the new way for building software. Eighty-one percent of what Aberdeen categorizes as "best-in-class" companies among those surveyed use automation to manage those requirements across the lifecycle.
Testing gets more complex
SOA, composite applications and Web services also increase the testing complexity. In the past when you'd built an application and tested it, you knew what the inputs were and what the outputs should be. It was fairly simple to do a QA test, and the interaction among the applications was pretty straightforward, Donham said.
"Now, with an individual service you can unit test, but once you put it into the soup of services interacting, defining an application [isn't] clear," he said. "Instead of looking at the services individually and unit and functionality testing, you need a transactional view. That may mean testing the new service, but it also means integration and regression testing. We've always done integration testing, but on a traditional application and how it behaves interacting with other applications. Now it's not only how it interacts with other services, but with other versions of those services."
Validating how all the components of a service-oriented application work, both individually and together, "is the single biggest sea change" for QA, said John Michelsen, chief scientist and co-founder of iTKO Inc., a Dallas-based developer of SOA testing and validation solutions and a sponsor of the Aberdeen study.
Michelsen also said that instead of one large team delivering a solution annually, there are now dozens of small teams collaborating. And because there is no one project owner, the components are iterating on their own lifecycles, which impacts the behavior of the application.
"You may use my services, but then I'm changing the behavior of your application because I'm changing my services," he said. "Testing is now a continuous activity that has to be validated. Continuous validation of tests is one of the main changes with SOA."
Top challenges for SOA testing
The Aberdeen survey found that the top challenges related to SOA and services testing, on average, included incomplete technical specifications, 41%; QA tools not designed for SOA or Web services, 33%; and not enough time allotted for testing, 42%.
Additional pressure for QA testers, Donham said, is that they don't have domain knowledge.
"The traditional QA role is often button pushing -- they're given a set of specs and they test to those specs," he said. "What you really need for quality is to have domain knowledge applied to QA. In many cases, that means the business analyst sitting with the QA tester, providing inputs into the test case."
According to Michelsen, the issues of software quality that have plagued the industry do not change with SOA.
"Software is still the weakest link in technology," Michelson said. "If we built iPhones the way we built software, this thing would never have released. For some reason, we tolerate significant failures in software despite that fact that we know how to do it on the hardware side. There is still plenty of opportunity to get better at this."
Indeed, Aberdeen found that only 7% of respondents have completely redesigned QA; most (65%) are changing their processes incrementally, on a project-by-project basis. And 60% of the best in class have trained key developers, business analysts and testers in SOA and Web services technology, vs. just 35% of the industry average.
Also, on average, Aberdeen said respondents are performing compliance, orchestration and security testing at "alarmingly low rates." Commented Donham, "We're just not there yet."
Getting around the SOA testing curve
There are steps organizations can take to move themselves further along the SOA testing curve. One is the adoption of automated testing tools to help with the complexity and layers of testing, according to Aberdeen.
"Testing and validation is the single biggest way to reduce production outages," Michelson said. "But the big challenge is can we automate enough of the testing to deliver to production with minimal outages or issues?"
The use of automated SOA and Web services testing tools is already differentiating the best-in-class companies, according to Aberdeen. Fifty-seven percent of best in class use them compared with 29% of what Aberdeen deems as "laggards."
The best in class are also implementing process change at the organizational level, incorporating business users at all phases of the lifecycle and taking a lifecycle view of quality. The survey found that 70% of best-in-class firms measure quality throughout the project lifecycle, not just in testing.