Test automation: Exploring automation case studies in Agile development
Test automation is certainly an area that more and more organizations are investing in,
executing tests in a fraction of the time and freeing up testers and other team members to focus on
those tasks that a computer can’t replicate. In Experiences with Test Automation, Dorothy
Graham and Mark Fewster have compiled 28 case studies from industry leaders with their experiences
of test automation. Chapter 29 includes a collection of short experience stories and the Appendix
includes a comprehensive table with information about the 79 tools mentioned in the book. The book
starts with a summary of the common themes and lessons learned.
and Mark Fewster discuss test automation and some of what they learned from these case
SSQ: You originally co-authored another book, Software Test Automation. Tell us more
about that book. What types of automation and tools were covered?
Graham and Mark Fewster: The first book focused on test execution automation but did not cover
any specific tools. Rather, the first half of the book covered general principles of test execution
automation (such as why it can be good to do, what can go wrong, what to be aware of and how to
best structure and organize the automation to get the best results). The second half of the book
comprised a number of case studies from other people, using a variety of tools that were around at
SSQ: How has test automation changed since your original book came out?
Graham/Fewster: It has become much more prevalent. There is a much richer range of tools that
support testing. More people use automation (though not necessarily to its best advantage). One of
the major changes is that there are now many open source tools available. Agile development is also
raising the importance of test automation, since test-driven design is by definition
automation-first, at the developer level.
SSQ: Though the book is titled Experiences with Test Automation, it looks like it
covers a huge variety of types of automation and tools. Does it include experiences with automation
of the workflow or the build, or other parts of the lifecycle outside of “test”?
Graham/Fewster: Not specifically, although some of the chapters mention automation as part of
the build (continuous integration tools), and model-based testing is close to requirements
analysis. Some of the chapters also mention performance testing as part of what they did. We are
amazed at the variety and scope of the application of test automation today and are pleased to be
able to share our contributors’ stories in this book.
SSQ: Today’s tools and processes, such as continuous
integration, include automatically executing test suites and then feeding results back in to
development. Do you see such automation strategies eventually removing the need for test or QA
Graham/Fewster: No. The tools execute tests and check results against an expected result. The
test cases and expected results have to be created and although some of this can be automated too,
good testing is still the domain of human creativity, so people will always be needed in testing.
Whether or not there is a test or QA department is more a function of how a company is organized
rather than being a technical issue. Some companies may include automation as part of the
responsibilities of a test or QA department. But tools and automation do not replace people – they
support people. Continuous integration and rapid feedback of results to developers is simply more
efficient practice, whatever department is doing it.
SSQ: It’s very interesting that you, again, include such a variety of experiences,
application domains and tools. Obviously, even if we stick strictly to “test automation” (rather
than extending to automation throughout the lifecycle) there are many different types of test
automation. Can you list some of those?
automation can be done at different levels; for example, test-driven development (TDD) is done
by developers at the unit test level, but can also be done at system/business/acceptance test level
using tools that interact with the Graphical User Interface (GUI) – this is where many people have
used commercial tools to automate regression tests. Test automation can also be very effective in
between these levels, where the tests interact with APIs or directly to a database, for
In additional to automating test execution, tools can also be used to help with various tasks
around the execution, such as setting up pre-requisites for tests (including for manual tests).
This kind of “out of the box” automation is described well in Chapter 19 by Jonathan Kohl.
As well as supporting test execution, tools can also help in other types of testing,
particularly performance testing and security testing.
We continue the conversation with Graham and Fewster in How
IT leaders can boost ROI with test automation.
For comprehensive conference coverage, see our Software
Testing Analysis and Review conference page.