vali_111 - Fotolia

How do you narrow the options in a complex smoke test suite?

Trying to determine how much ground a smoke test should cover is tricky. Expert Amy Reichert offers step-by-step advice for getting the most out of this technique.

Smoke test or regression test -- they are similar and yet different. A fine line separates what defines a smoke test, or at least a smoke test worth executing, and what defines a valuable regression test. So, as a quality assurance, how do you narrow the tests you need to execute and create a valuable smoke test suite? How do you determine the amount of coverage you need and where?

It helps if the team has performed risk analysis on the functional areas of the application so that information is up to date and available. If not, consider creating a map of all the functional areas within your application and rank them from highest to lowest priority. If you have trouble ranking them directly, try grouping them into critical, high, medium and low. Once you separate them in this manner, rank them within their group. Take the critical group and pull test cases that test some function in each area. Repeat for the high group. How many tests do you have? For a smoke test suite that's primarily executed manually you'll want to stay around 30 to 35 total test cases. The goal is to make it reasonable for a tester to execute in four hours, or half a work day. If your smoke test is automated, you should be able to increase the total number of tests to 50. I suggest keeping it small enough to allow time for failure analysis. The advantage of automated smoke tests is the ability to execute the suite more frequently while still allowing time for failure analysis.

The next consideration is taking the test cases you have ranked so far, and reading through them to gauge complexity. The smoke test suite needs to be relatively superficial but also contain enough depth or bite to actually make running it worthwhile. I am not a fan of smoke tests that only verify whether an application loads or opens. What I really need to know is if basic functions are working properly.

When reviewing your smoke test suite, look for tests on basic functions all the way through but that are not so complex they take an hour or more to execute. Remember, it's a smoke test; it's not going to touch on everything in your application. But it should at least touch on all the functionality within an application. Scope and depth is a tricky business -- use your knowledge of the application to determine how deep the test cases need to go. Keep in mind the smoke test suite is not a fully functional regression. Good luck!

Next Steps

Smoke testing's role in continuous delivery

Why you should automate testing, but carefully

Look into software testing's crystal ball

Dig Deeper on Topics Archive