There is no doubt that an automated software testing strategy is imperative to deliver high-quality code quickly....
Agile techniques such as continuous integration depend on a strong automated testing strategy. Does that mean we no longer need manual tests? Although some might argue every test should be automated, that may not be realistic or the best strategy. In this article we'll take a look at two approaches to Agile software testing: the Agile test automation pyramid and Agile testing quadrants to help your organization determine the most effective Agile software testing strategy.
One of the biggest mistakes some software organizations make is to jump right into automated tests without first figuring out a solid test strategy. They may hire consultants who have expertise with a particular automated test tool and start by creating automated tests at the GUI level -- the absolute worst place to start.
Using the Agile test automation pyramid
According to the Agile test automation pyramid, first introduced by Mike Cohn in his book, Succeeding with Agile, the majority of automated tests should be written at the unit test level, preferably using test driven development (TDD). Although it's not necessary to employ TDD techniques to write automated unit tests, by doing so the team ensures the automated tests are written, even before the code. These automated unit tests, written in the same programming language as the system under test, form the basis for regression tests that are executed with each build and are required for continuous integration or continuous deployment.
The second layer of tests checks the business logic and APIs. They are deeper than the unit tests, perhaps hitting the database, but do not require the GUI layer. Separating your automated tests from the GUI allows tests to use expected data inputs and outputs, specified in a spreadsheet, for example, that can be executed without necessitating the user interface.
The GUI tests are the most brittle and require the most maintenance. These should be kept to a minimum and only be used to ensure your user interface is solid, not to test the business logic, which will already be tested with the unit tests and the automated functional tests.
This type of automation ensures the code is functioning as designed. However, not only do you need to ensure the code is working, you need to do testing with the business experts to ensure you are giving them functionality that they need and that's easy to use. Some amount of manual and exploratory tests will be needed to validate work flows and usability.
Using the Agile testing quadrants
To design a complete Agile software testing strategy, your team must consider all types of testing that are appropriate for the software being developed. It's not enough to simply consider tests that should or could be automated. Start first with the big picture by considering what types of test should be done and then determining the best strategies for doing that testing.
The Agile testing quadrants (Figure 1) were first introduced by Brian Marick, one of the original Agile Manifesto signatories, in a blog post in 2003. Over time, the quadrants have been discussed and revamped, but they still provide the basis for Agile organizations to create a test strategy. Lisa Crispin and Janet Gregory describe the quadrants in detail in their popular Agile Testing books.
The left half (Q1 and Q2) of the diagram in Figure 1 describes tests used for the code that is solid. These are described as "Supporting the Team." The right half (Q3 and Q4) are tests that critique the product. The lower half (Q1 and Q4) are technology tests (white-box tests, requiring insights into the code), while the upper half (Q2 and Q3) are business tests (black-box tests, requiring input from customers or users).
Consider big picture when planning a test strategy
The majority of your automation will be regression tests from the Q1 quadrant, written to exercise the code paths of the application. The tests in the Q4 quadrant require automated test tools to carry out performance, security and other specialized testing. Many of the tests in Q2 are automated, perhaps at the story level, and some at the GUI level. The tests in Q3 that critique the product at the business level are manual tests and include exploratory and usability testing.
To ensure quality and a solid test strategy, it's important to look at the big picture. Start by separating the types of testing that must be done for your product into quadrants. Focus your automation efforts first on the tests in Q1 and Q4, with the help of appropriate tools. Plan for some automation for the Q2 efforts. Finally, save some time for the manual test efforts in Q3.
Automation is an important part of any Agile software testing strategy. However, there are different types of testing and different types of automation. Using the Agile test automation pyramid and test quadrants, you'll be on your way to creating a solid strategy to fully test your application.
Managing Agile software testing processes
Discover these four tips for effective software testing