Test automation is certainly an area that more and more organizations are investing in, executing tests in a fraction of the time and freeing up testers and other team members to focus on those tasks that a computer can’t replicate. In Experiences with Test Automation, Dorothy Graham and Mark Fewster have compiled 28 case studies from industry leaders with their experiences of test automation. Chapter 29 includes a collection of short experience stories and the Appendix includes a comprehensive table with information about the 79 tools mentioned in the book. The book starts with a summary of the common themes and lessons learned.
Authors Dorothy Graham and Mark Fewster discuss test automation and some of what they learned from these case studies.
SSQ: You originally co-authored another book, Software Test Automation. Tell us more about that book. What types of automation and tools were covered?
Dorothy Graham and Mark Fewster: The first book focused on test execution automation but did not cover any specific tools. Rather, the first half of the book covered general principles of test execution automation (such as why it can be good to do, what can go wrong, what to be aware of and how to best structure and organize the automation to get the best results). The second half of the book comprised a number of case studies from other people, using a variety of tools that were around at the time.
SSQ: How has test automation changed since your original book came out?
Graham/Fewster: It has become much more prevalent. There is a much richer range of tools that support testing. More people use automation (though not necessarily to its best advantage). One of the major changes is that there are now many open source tools available. Agile development is also raising the importance of test automation, since test-driven design is by definition automation-first, at the developer level.
SSQ: Though the book is titled Experiences with Test Automation, it looks like it covers a huge variety of types of automation and tools. Does it include experiences with automation of the workflow or the build, or other parts of the lifecycle outside of “test”?
Graham/Fewster: Not specifically, although some of the chapters mention automation as part of the build (continuous integration tools), and model-based testing is close to requirements analysis. Some of the chapters also mention performance testing as part of what they did. We are amazed at the variety and scope of the application of test automation today and are pleased to be able to share our contributors’ stories in this book.
SSQ: Today’s tools and processes, such as continuous integration, include automatically executing test suites and then feeding results back in to development. Do you see such automation strategies eventually removing the need for test or QA departments?
Graham/Fewster: No. The tools execute tests and check results against an expected result. The test cases and expected results have to be created and although some of this can be automated too, good testing is still the domain of human creativity, so people will always be needed in testing. Whether or not there is a test or QA department is more a function of how a company is organized rather than being a technical issue. Some companies may include automation as part of the responsibilities of a test or QA department. But tools and automation do not replace people – they support people. Continuous integration and rapid feedback of results to developers is simply more efficient practice, whatever department is doing it.
SSQ: It’s very interesting that you, again, include such a variety of experiences, application domains and tools. Obviously, even if we stick strictly to “test automation” (rather than extending to automation throughout the lifecycle) there are many different types of test automation. Can you list some of those?
Graham/Fewster: Test automation can be done at different levels; for example, test-driven development (TDD) is done by developers at the unit test level, but can also be done at system/business/acceptance test level using tools that interact with the Graphical User Interface (GUI) – this is where many people have used commercial tools to automate regression tests. Test automation can also be very effective in between these levels, where the tests interact with APIs or directly to a database, for example.
In additional to automating test execution, tools can also be used to help with various tasks around the execution, such as setting up pre-requisites for tests (including for manual tests). This kind of “out of the box” automation is described well in Chapter 19 by Jonathan Kohl.
We continue the conversation with Graham and Fewster in How IT leaders can boost ROI with test automation.
For comprehensive conference coverage, see our Software Testing Analysis and Review conference page.