I know of some groups that are creating user interface automated tests when the requirements are still changing. It seems like it would be better to create the UI tests that require recording after the code is stable. Would you agree?
“Test early and often” is the mantra of savvy analysts, developers, and testers. They know that acting on early feedback is less expensive than doing rework on last-minute feedback. UI automated testing (i.e., record/playback) provides a way to conduct repeatable testing of whether functional and nonfunctional requirements have been met. Yet there are costs to setting up automated testing. And it is very expensive to maintain automated functional tests if the tool you are using fails tests when minor interface layout and navigation changes are made. For example, some automated testing tools will fail a test if the position of a label on a GUI or webpage is even slightly changed.
From the perspective of functional requirements, here are some tips for early automated UI functional testing:
- Use automated UI functional testing when you have stable functional requirements, or when your UI layout and navigation are stable (even if your functional requirements are still volatile).
- If your UI layout and navigation are still volatile, identify ways to isolate the UI from functional testing.
- If the functionality you are testing will be used infrequently, consider whether the benefits of the automated test outweigh the cost of setting it up and maintaining it.
- Conduct automated regression tests as often as possible, and not just as part of the final QA activities or just before executable code from multiple projects is put into production.
- Keep in mind that there are many valuable early forms of manual testing. These include:
- Usability tests, such as card sorting or paper prototypes, that explore navigation flows with no underlying functionality
- Reviews, during which someone with a tester’s mind-set evaluates the requirements
- Having an expert who did not specify the requirements arrive at a solution for a sample set of input data (this is especially useful for complex calculations)
- Manually testing executable code with yet-to-be-added code sections stubbed out
For nonfunctional testing, UI scripts are very powerful for performance, load, and stress testing (e.g., simulating 1000 customers trying to place an order at the same time). For early testing, if some of the requirements are not completely settled or if you haven’t yet implemented communication with other software components and services, then you can stub out portions of the test. Early performance and load testing provides early warnings to capacity planners and infrastructure architects about any problems with the scalability of the architecture.
This was first published in February 2011