Devising a test automation strategy: Getting started

In this tip by Agile expert Lisa Crispin, learn the important steps to move your organization forward with a well-planned automation strategy.

You’re the QA director or CIO at a growing software organization. You’re sold on the idea that test automation is necessary in order to keep your team on track, provide quick feedback and manage technical debt. But like many software organizations, you have none at all. Where do you get started?

The whole-team approach

The history of test automation is littered with failed “test automation projects.” Test automation sounded like a good idea, so the company bought a vendor GUI test automation tool, gave it to the test team, and waited for the magic to happen. Here’s the news: automating tests is software development. It must be done with the same care and thought that goes into writing production code. And it needs a diversity of people and skills: testers to know the right things to test, customers to provide examples of desired behavior, programmers to design maintainable tests and more.

Test automation succeeds when the whole development team treats it as an integral part of software development. It’s not any harder or easier than writing production code. Learning to do it well takes time, effort and lots of small experiments.

Change is hard, and people are motivated to change when they feel pain. Start by sharing the pain of manual regression testing. Your product can’t be released to production without regression testing, right? Ask the testers to work with customers to identify the most critical areas of the application. They can then write manual regression testing scripts to ensure those areas work.

The last day or two of each iteration, get everyone on the team together to divide up these manual test scripts and execute the manual regression testing. I’ve been on three teams that took this approach. Believe me, having to slog through tedious manual regression test scripts provides major motivation for automating tests!

Overcome barriers to automation

Your cross-functional team has all the skills needed to overcome barriers to test automation. Get together for some time-boxed brainstorming meetings to identify the impediments standing in the way of automating tests. In my experience, the best approach is to identify the biggest obstacle, then try a small experiment to get around it.

For example, when my team wanted to try test-driven development (TDD), which involves automating unit-level tests, we couldn’t figure out how to do that with our nasty legacy code. First we tried brown bag sessions at lunch where the whole team tried writing unit tests. Then we brought in an expert to do training on it. That helped, but finally we just had to budget time to get traction on it.

Should you tackle automating unit tests with test-driven development? Would GUI tests be the easy win? It’s helpful to outline a strategy for automating tests.

Plan your automation strategy

There are so many tests to automate and so little time; it pays to focus on the automation that will provide you with the best return on investment (ROI). Mike Cohn’s Test Automation Pyramid provides a good guideline.

Unit and component tests verify our code design and architecture, and provide the best return on investment. They are quick to write, quick to run and provide a good safety net as part of a continuous integration process. They form the solid base of just about any test automation strategy. If your team has no test automation, learning to automate unit tests is the obvious place to start.

The next-best return on investment is the middle layer of the triangle, at the API or service level. These tests generally take the place of the user interface. They pass test inputs to production code, obtain results, and compare actual and expected results. Writing and maintaining them takes more time than unit tests, and they generally run more slowly, but they provide crucial feedback.

Today’s test libraries and frameworks allow us to automate and maintain GUI tests more cheaply than the old-school record/playback tools. Even so, they run more slowly than unit or API-level tests, and may require more frequent updating.

In my experience, a two-pronged approach to start automating tests on a legacy system works well. While the team masters TDD and grows its unit test library, you can get some quick wins with simple GUI tests covering the most critical functionality.

The test automation pyramid represents our long-term goal, but you won’t get there overnight. Most teams start with an “upside-down” pyramid, more GUI tests than anything else. You can “flip” the pyramid over time.

Choosing tools

We don’t have a wide array of tools at the unit test level; you just use the flavor of xUnit that goes with your production programming language, such as JUnit for Java or NUnit for .Net.

For API and GUI level tests, start by experimenting with how you’d like to specify your tests. Does a given/when/then format work well for your business experts? Perhaps your business domain lends itself to a tabular test format. Or, you might have a domain where specifying inputs and expected outputs in a spreadsheet works better. Some businesses prefer working with a time-based scenario.

Once you’ve found a format to try, look for a test library or framework that supports that. If your team has enough bandwidth, you might be able to “grow your own.” Use retrospectives to evaluate whether your experiment is having good results. If not, start a different experiment. This is a big investment. The right choices now will mean big returns in the long term.


Automating tests is coding. Automated test code deserves the same respect, care and feeding as production code. It makes sense for the people writing production code to also write the test automation code.

Testers are expert at knowing the right tests to automate. Other team members contribute expertise that helps us get timely feedback, with data and scenarios that represent production. It just makes sense to have everyone on the team work together to automate tests.

Get started

Pick one problem to solve, and do a small experiment to overcome it. If automating unit tests seems too daunting, try “defect-driven development,” an idea from Jared Richardson. For each defect that your team fixes, first write a unit test to reproduce the problem, then correct the code. Once the test passes, check in the test and the code.

In December 2003, my team had zero automated tests. It took us eight months just to get traction on automating unit tests via TDD, and to create a suite of GUI regression tests for the most critical parts of the legacy code. Eight years later, we have over 6,000 JUnits, over 500 FitNesse test pages and over 122 GUI test scripts. Each of these contains multiple assertions. We know within 45 minutes of any code check-in whether it broke anything. As our application changes, we must continually look for the most effective ways to automate our tests.

This is a journey, not a destination. Just take that first step.

What are some of the obstacles to beginning test automation in your organization, and what approaches have you tried to address them? Email comments to [email protected]



Next Steps

Get the tips you need to navigate automated API testing

Learn how to prevent automation from ruining your career

Discover the top automated functional testing tools

Dig Deeper on Topics Archive