Is it possible to do performance testing on mobile applications? Won’t performance depend on the specific device, carrier and factors that are out of the developer’s control?
All performance testing is based on trends. We can’t really give hard numbers of “proof” except on the precise configuration and scenarios we test.
The items you mention that are “out of the developer’s control” have corresponding items with other environments as well. All performance testing is limited in some way by its host system, the system it is running on, the other applications running that may impact the environment, connectivity and other issues. These questions exist in some form no matter what it is you are testing.
Fortunately for us, there are a huge number of bright people who think “I need a tool for X.” Instead of giving up when they don’t find one on the market, they write them. It strikes me that this is the root of most, if not all, of the open-source tools that are available-- including emulators for mobile devices. Yes, I know, some devices do not have emulators. However, if you can get a set of benchmarks on one device, using a variety of open-source tools and emulators, this gets you well down the road, no?
There are a fair number of such tools we can make use of to improve our testing and productivity.
While these tools cannot prove with any certainty how an application will perform in any given environment (which to me includes device, carrier and the like), you can get a reasonable approximation of how it performs in general. Sometimes, that is the best you can do.
When I need to test applications that will be running in too many environments for me to really test each one, I try and establish some reasonable baselines. For example, “In X environment, with these products running and Y in place, we saw K for average response time for transaction type L and N average response for transaction type N.”
In and of itself, it may not tell you much. However, if the results seem reasonable they may be good enough, no? Further, if this is the first generation of the application, you have a baseline to measure future versions against. At the same time, if there are older versions of the same application, you can compare the newer version’s performance with the older version’s performance.
Likewise, if there are competitor products available to run on the same platform, a simple test may be to compare the same types of transactions through their app. This will give you at least an idea of how you stack up and see if the gut feel of “good enough” or “must be better” is realistic.
Read how Perfecto Mobile CQ Lab can simplify automated app testing
Dig Deeper on Topics Archive
Related Q&A from Peter Walen
Veteran software quality pro Peter Walen explains what software tester skills are really necessary in today's enterprise organizations. Continue Reading
Software testing veteran Peter Walen explains how software testers can write test scripts that others can follow without having to test by rote. Continue Reading
Crowd sourcing can be a key piece of a test strategy for enterprise mobile apps aimed at customers, not employees. Continue Reading