If you want to know what the phrase "software testing in the wild" really means, look no further than mobile commerce...
platform provider Upstream, which serves 80 million customers in 45 different countries, the vast majority of which are emerging markets.
The company needed local testing for its offering in countries like Nigeria, where many consumers are paid daily and never have SIM cards with large balances on them. And then there was the issue of equipment -- iPhones are hugely popular in Mexico and Brazil, but not in Ghana or Nigeria.
"Local testing is a key feature in order to deliver working customer experience and engage the final user," said Upstream's CTO Massimo Del Vecchio. "You cannot underestimate this complexity if we want to deliver our products to our end users." But even if it was feasible to expand Upstream's testing team, there was no possible way it could have testers on the ground in each and every country, not to mention testers with the wide range of the phones, tablets or computers used locally, Del Vecchio said.
The solution was crowdsourced testing, an idea that's gaining ground at a time when user demand for mobile applications is only exceeded by user expectations. Upstream reached out to Applause, a crowd-sourced testing provider with 250,000 testers worldwide and access to a vast array of testing devices. An early joint project was telling. Upstream had designed an app to allow Mexican citizens to interact with a very popular local celebrity. Based on Upstream's "in the lab" unit and exploratory testing, the app worked fine. But when Applause did local testing, there were latency issues Upstream could never have foreseen. The app needed to be reshaped as a result, Del Vecchio said. "Basically we need a company that had the local footprint ability on one side and the wide technical breadth of testing capabilities on the other. There are a lot of different business scenarios, different options and different devices. But the key is localization. We need someone who can test in the local country."
Mark Holland, vice president of delivery at Applause, said local testing puts a spotlight on any issues an app might have. "If you're a company doing testing in the lab you are really limited to what you can replicate," he said. "As we know, out in the real world things change. It's different to conduct a transaction in South Africa than it is another country," but that's something no one knows until you are actually there.
For companies thinking of taking the leap to local testing, Del Vecchio has some advice. First, it's important to have a third-party company that can provide local testing and a lot of devices, of course. But the timing is just as important. "Whatever journey they will take they need to find the third party that enables them to have early feedback during the execution and implementation stages," he said. "What I mean is don't engage them at the end. We use them through the entire software deployment lifecycle and that's key to leveraging a third party."
Why devs will learn to love crowdsourced testing, too
The changing test landscape and what to do about it
How you can use AI in software testing