Testers must consider functionalities that are constantly updating and improving on mobile devices. What is your advice for being as efficient and effective as possible?
This is a common problem in a variety of environments, not just in mobile or embedded technology applications. Testers working on software that has a drop released to the public every two or three weeks can relate with this question. Likewise, people working on applications that will be used only internally for their company can relate to this if they have a regular enhancement or release rotation.
There are a couple of things I find helpful to remember when working in environments like this. First, there is no way that everything, every possible data combination, every possible environment configuration, hardware and software, can be tested 100%. Second, there is no way that a team of testers can find every variance or anomaly. Some bugs from each iteration or update will not be detected the same cycle they are introduced. The key is to vary your test approach with each cycle.
Some people will say “automate more of your tests.” Creating scripts that can be automated to run a basic checklist can certainly save you time and effort in running the same mundane checks each test cycle. The thing to remember is that these free up your skilled testers to spend more time investigating the application farther and mapping out new routes to follow in the future. You have more time to explore how the application actually works and not simply confirm that it does precisely what you or someone else thinks it should do.
There are some warnings on what to avoid that I feel compelled to give. First, avoid the temptation to execute the same tests, same transactions, same data sets and same variables each and every time you test an update. You may well find yourself with a bug-free path that matches your scripts’ conditions, but lots of bugs around them. Second, as tempting as it may be to test “just what changed” with each update, don’t. The farther away the software gets from its initial state, the more likely it is that there are twists and turns that the developers and you have forgotten about. Test the changes, and test around the changes. Then test functions that touch on what the change did, even if it is indirect.
Beware the trap of relying on your automated script to test for you. Relying on one or two well-written scripts can be a huge help. That is precisely what it is, a help. They can do some of the more mundane things so you can focus on what humans do really well – Think. Just because you have a script that bangs through the application, do not stop thinking.
Finally, when it comes to many releases packed close together, be flexible. Allow yourself and your team some breathing room. Look for new ways to save effort and to challenge the system in ways you have not tried before. These tests may not be run every pass, but perhaps staggering them can keep the job interesting and maybe give you different insights each time through.
This was first published in October 2011