Most agile practitioners recommend a combination of automated user interface (UI) tests and exploratory testing...
(ET) on agile software projects. But it is a common mistake on agile teams to view UI test automation and ET as entirely separate activities. In fact, each activity should inform the other. In particular, an automated UI test that fails should always be a sign that some ET is required.
How they work
Well-designed automated UI tests are shallow. They should contain very little business logic testing, they should be focused on checking that users have access to page elements that they need, and that users do not have access to page elements that they should not.
Exploratory testing is the process of designing the next test in real time, based on the result of the previous test. ET activity is usually governed by a "charter," some sort of aspect of the system on which to focus testing activities. Examples of charters might be "test the access settings for this feature" or "test that the new feature is consistent with existing features" or "look for inconsistencies in the new behavior."
A failing UI test is an ET charter
Whenever a UI test fails, that means that something in the application has changed in some unexpected way. And where there is an unexpected change, there are often problems to be found. Remember, UI tests are shallow things. They are merely indicators that something in the software may not be correct. It takes human beings to evaluate the extent of any potential problem.
Responsible agile teams will treat a failing UI test as a sign that some exploratory testing is needed in that area of the application.
Appearance and layout
No automated test in the world can validate that the application actually looks good. When a UI test fails, a tester should pay attention to that area of the application looking for things like skewed text, wrong colors, poor layout, even misspellings. Changing one area of the UI often means that nearby areas become distorted or disrupted. When a UI test fails, always make sure that everything near the element that caused the failure is still nicely formed.
User experience and workflow
Most software applications automate some business function that users depend on. If a UI test fails, that may mean that users will be exposed to a new path through the application. That new path may or may not be well-designed or even correct. When a UI test fails, a tester should pay attention to whether the new experience for the user is an improvement, or whether the change to the software has inadvertently reduced the quality of the experience for the user.
A failing test of any sort often means that the software is simply wrong. But again, an automated UI test is a shallow thing. The software may be wrong in very subtle ways, or might be failing only after a particular set of actions. The underlying data for the system might be mishandled in some way, or some aspect of the environment itself may be incorrect. Finding issues like this is where ET is strongest, and a failing UI test is often a pointer to such issues.
Turning the situation around
It is also the case that ET may influence how UI tests are implemented. Exploratory testing may discover issues in a part of the application that should be covered by automated UI tests but is not. The automated UI test suite may get too unwieldy for certain aspects of the application and regular ET activity for those aspects may be required. A productive relationship between automated UI test design and ET flows both ways.
Both automated UI test design and ET are activities that demand extensive discipline, skill and intelligence. Probably very few testers in the world are expert in both activities. Many people who are good at automated UI test design may actually call themselves "developers." Many people who are good at ET may call themselves "Product Owners" or even "system administrators." Regardless of who is responsible for automated UI test design and who is responsible for ET on the project, it is a mistake for them to ignore the other's work. And it is worth pointing out that experts in both disciplines are in high demand. Being an expert automated UI test designer and an expert at ET is a worthy career goal.