The growing trend of bring your own device to the enterprise is impacting quality assurance needs and practices.
"We have been talking for quite some time now about how software and your brand are inextricably linked, and mobile just exacerbates that," said Theresa Lanowitz, founder and industry technology analyst at voke inc. "If mobile apps go outside your organization, it's further linked to software. People are very quick to judge
The expectation of consumers that their mobile apps "just work," and work well, is driving into the enterprise, as employees increasingly want access to corporate data and applications from their personal smartphones and tablets.
"If you look at the consumer product space, testing has evolved to work with a variety of devices -- you never know what you're going to get," said Matt Heusser, a consulting software tester in the Grand Rapids, Mich., area. "Testing consumer products now is like a box of chocolates. People I see on the enterprise side are talking about adopting test strategies to be more like that."
Heusser said he sees a few strategies emerging. One, he said, is to declare a list of supported devices allowed to access internal applications so that testing is limited to those devices. Another strategy for large enterprises is to do crowdsource testing -- testing the app on a wide variety of their devices, or hire a third party to do the same.
Another technique, Heusser said, is to get a list of percentage of users by device or platform, and prioritize testing based on those percentages. It's important to involve the stakeholders, he said. For example, he explained, if your survey finds only nine people are using a certain device, but one is the CFO using an accounting application, that needs to be weighed into the decision making. Once a decision is made based on the percentage of users and stakeholder priorities, and you have a list, "As a tester you're supporting things that need to be supported; then it's not an argument about how you're spending your time."
In addition to new testing strategies, Bring your own device (BYOD) in the enterprise is also shifting quality standards and priorities. "For a long time, on the desktop we've told testers they're supposed to focus more on things functionally not working and leave user interface design to others," said Karen Johnson, a software test consultant in the greater Chicago area. "In the mobile space, testers need to reschool themselves on what usability is."
According to the World Quality Report, 2012-2013, a study conducted by Capgemini, Sogeti and HP, the top priority when testing mobile apps is efficiency/performance, which was cited by 64% of respondents. That was followed by functionality (48%), portability (46%), UI/ease of use (36%), compatibility/regression testing (31%), security (18%) and certification of application (14%).
The study, which reached out to more than 1,550 CFOs, CIOs, IT directors and QA directors around the globe, also characterizes organizations as "behind the curve" with mobile testing. According to the report, "Only 31% of respondents currently test mobile applications, and those surveyed readily admit to being ill-equipped for mobile testing. This suggests that QA has fallen behind the mobile curve."
Challenges to mobile testing, according to survey respondents, include: that they do not have the right tools to test (65%); they do not have the devices readily available (52%); they do not have the right testing process/method (34%); and they had no mobile testing experts available (29%).
"It's been a lot of years since we didn't have what we needed in our own test labs," said Johnson. "In the late '90s some of us were fighting for extra PCs to have different operating systems. Now, we're going through the same battle with mobile. I don't know very many companies that have test labs with a multitude of devices. The team usually asks, 'Does anyone have an iPhone with this iOS?' Or, 'Are you on Verizon or AT&T?' The team is usually looking within the team."
However, voke's Lanowitz said there are a variety of mobile testing vendors offering different approaches. For example, she said, some vendors are building out device clouds. "Say these are the top five devices I want to test; realistically, you can't test everything out there, so you build out cloud for these devices. The mobile test vendors are doing a good job putting together a grid of devices and OSes."
Also, she added, "The mobile testing vendors that are coming to the market from the perspective of skill adaptation on the functional testing side are coming to the market with experience in delivering solutions for traditional testing."
Still, Johnson said that unlike with the desktop world, there aren't solid data analytics indicating what customers are really using. And she said that while in North America there is a presumption or bias that everyone is using an iPhone or Android, the overseas market is different, having an array of different devices and carriers.
"It makes for a testing nightmare," she said. "There are so many variables, but you can't even get to some devices [being used overseas] or you can't provision [them]. Now we have tools saying we have that with vendors, but there are still some challenges there."
But, Heusser is optimistic. "In the test community, the people who are real testers are risk management experts. They are very good at taking an infinite combination of things and reducing that to a number of finite things you can test, and coming to some conclusions about quality. If the team has any gray hairs, you have people who have experienced a paradigm shift. Those people are able to adjust."
Johnson echoed this. "I started mobile testing three years ago. I was pretty alone; there wasn't training." But that is changing, she said. "From now to the next three years it will be chaotic, but we'll figure it out."
This was first published in September 2012