Test automation is a discipline and requires an investment in the right people as well as the right tools, according to Marc Nadeau, senior director of QA at Blackboard Inc., a Washington, D.C.-based provider of education technology. After some ups and downs along the path to test automation, in conjunction with a transition to agile development, Blackboard has assembled its arsenal for automation, which includes the skillsets and the tools.
Among the key tools are Borland SilkTest for automated functional, regression and smoke testing; the Selenium open source suite of tools for automating Web app testing; some homegrown tools based on the JUnit framework; and most recently Borland SilkCentral Test Manager, for automated software test case management.
Blackboard began its move to agile about three to four years ago working in small teams, according to Nadeau, but in the past two years they've followed a Scrum-like methodology with regular sprints.
The company's first foray into test automation focused on functional and regression testing, Nadeau said. "But as time went on it became more difficult. We kept expanding our nightly smoke testing, and over time most of the focus was in that area."
Nadeau explained, "It takes 40-plus hours of automated execution to execute one pass on the nightly smoke test; one pass in manual mode took over 4,000 hours. Our application was getting complex, with lots of integration points. Our smoke test is fairly sophisticated and broad, and we rely on it a lot." SilkTest, he said, is the primary tool for the smoke testing.
Automation is key to quality, he said. "Our current nightly processing is critical; we have multiple platforms, and the smoke tests are critical for us to. At any given time we have 10-12 development teams working, and we need to give them instant feedback that they still have a solid code line to work in." Last year Blackboard did more than 100,000 hours of smoke testing with builds, he said.
As part of its automation efforts, Nadeau said Blackboard had looked into test case management systems several years ago but didn't find anything to meet its needs. "We had budget and project concerns," he said. "It takes a fair amount of effort to move into test case management when you have a large legacy of manual testing."
Blackboard, he said, "still has a lot of manual execution, and we have a lot automated in the nightly builds running, so we needed a tool to manage the process better, to facilitate reporting. We're trying to get out of Word docs and Excel. Because the manual tests have used as many 110 manual testers at any given time, a management tool is important. And we outsource a lot of the execution—we write detailed test cases that can be automated and executed in a repeatable fashion."
Now that Borland's SilkCentral Test Manager has open APIs, Nadeau said the transition to test case management will be easier, making the time right for acquiring the product. "We're now using the APIs to help us do more things the product doesn't do yet."
He explained, "When you have a large test suite, mass updates aren't that easy in any of these commercial tools, as applications change rapidly—especially in the agile world."
Dedication to quality is fundamental to Blackboard. Nadeau said they have 50 developers and nearly 50 QA analysts, so the ratio is almost 1-to-1. Then they have 50 additional testers on the test execution team. "With agile teams we pair developers with QA analysts who write the test cases as the developers work on the features. Then we have full functional testing/regression testing with the test execution team as the developers move to the next release and offer support from time to time, so the ratio is close to 2-to-1 ; it's unusual, but it's part of our dedication to quality."
As Blackboard's agile efforts evolve and they adopt more test-driven development (TDD) and more unit testing, Nadeau expects to be able to reduce that ratio.
While automation of functional and regression testing is lagging behind where they'd like to be, Nadeau said they've laid the foundation. "The key to success to automation is you automate the repetitive processing. As we moved to agile with Scrum teams we need constant feedback. The nightly builds help that—it gives the teams a fair amount of confidence."
Blackboard, he said, had to retool its team and enhance its toolsets for automation. "Now with that in place we can really tackle functional and regression testing again."
Users need a toolkit, not just a single tool, because testers do many types of tests, each of which has different goals. Blackboard's SilkTest and SilkCentral are core lines of integrated tools, Nadeau said, and efforts to build toolkits for the automation side are ongoing.
His advice to others on the path to test automation is to make the investment in people as well as tools, particularly in training and developing a methodology to get the proper skillsets in place. "Especially in areas of automation, it's a discipline, and you want the appropriate staff to take it on, otherwise there's a lot of rework," Nadeau said. "You need the right manager to build an automation methodology and a foundation to build on. The tools themselves don't solve the problems."