Testing the user interface: Automated and exploratory testing

Should user interface (UI) testing be automated or exploratory? In this tip, SSQ contributor Chris McMahon answers that both automated and exploratory testing can be used for UI testing, and what is most effective is using them together. McMahon explains when each type of testing is appropriate and how when they are used together, they can complement one another to provide the most comprehensive UI test coverage.

While software testers do a wide variety of work in their jobs, they are typically responsible for two critical functions: designing and implementing user interface (UI) level test automation and performing manual exploratory testing. Each activity informs the other, and when done well, each provides critical information to the whole software development team.

Designing automation

There are typically three levels of test automation on a well-maintained software project. Unit tests cover the behavior of the smallest bits of code in order to enforce good design. Integration tests validate correct business function and business logic. UI tests ensure that the user is presented with the proper controls for the application.

What this means is that automated UI tests should be very shallow. Such tests should not be concerned so much with business logic or the design of the system, but instead should validate that the interface presented to the user continues to contain the correct information and continues to function in the correct way. UI tests should check two things:

  • First, UI tests should check that the information on the page is correct. These tests are of two forms. One sort of test says, "message X exists under these conditions." The other sort of test says, "message Y does not exist under these conditions." Automated tests like these are important because they confirm that the user continues to have access to information required to manipulate the application correctly. The negative case is even more important: exposing information in the application that should not be exposed could be a security issue, or even cause legal trouble.
  • Second, UI tests should check that elements in the UI that cause a change of state continue to function. If the UI has a clickable link, then that link should take the user to the proper destination. If the UI has a "Save" button, then information entered in the UI should be saved properly.

It is important when designing a UI test that the test takes the most efficient path possible through the UI in order to validate the existence or nonexistence of page elements, and to validate the correct changes of state in the application. UI tests are necessarily slow, which makes them expensive to run. Efficiency is key in a well-designed UI test suite.

Minimize the extent to which the tests exercise individual UI elements as much as possible. When a fault in a single UI element causes many tests to fail, analyzing the failure becomes a painful process.

Implementing automation

The needs of each individual team vary widely, but in general, a UI test suite should be run as often as possible, and results of the tests should be visible upon the conclusion of each test run. Any failures in the UI suite should be analyzed and addressed immediately.

Anecdotal evidence suggests that many UI test efforts quickly become unmanageable due to bad design, or neglect over time, or both. Implementing well-designed UI tests such that they run as frequently as possible prevents this. Maintaining the UI tests with a minimum number of failures at any given time is a critical activity.

Exploratory testing

The great value of having a well-designed and well-implemented automated UI test suite is that it allows testers the freedom to spend more time examining the most interesting aspects of the application. The newest features and the newest code and the most complex areas of the application are the most likely to have issues for exploratory testers to discover. Having an automated UI test suite in place gives testers a significant amount of confidence that the parts of the application that they examined in the past continue to function properly. Being able to devote the energy of the test team solely to the ongoing work brings an enormous advantage in terms of improving quality.

But there is still a close relationship between UI test automation and exploratory testing. That relationship is manifested in two ways:

  • For one thing, exploratory testing informs the design of the automated tests. Without intimate firsthand knowledge of the operation of the UI, it is impossible to design automated tests well for the UI. So in the course of building automated tests, exploratory testing is required to view messages in the UI, to exercise elements in the UI, and to determine the most efficient paths for the automated tests to navigate.
  • Also, every failing automated test should be a charter for some exploratory testing. Automated UI tests are only a very thin layer over a complex application. If an automated UI test fails for some reason, it is often an indication that a deeper problem exists that may only be discovered by exploratory testing from an actual person who knows and understands how the application should behave.

Automated and manual testing together

Having excellent UI test automation in place allows for excellent exploratory testing by intelligent testers. The automation is keeping an eye on the work already done while the testers are keeping an eye on the work in progress. And when an automated UI test fails, an intelligent tester is on the case immediately, analyzing the failure, investigating the possibility that the failed test indicates some deeper problem in the application. This is the best of all possible situations for a mature UI test effort, and it is worth the time and effort invested.

Dig Deeper on Automated and autonomous testing