Manage Learn to apply best practices and optimize your operations.

Testing in the cloud: Considering the risks

In this tip, SSQ contributor Matt Heusser talks about the risks of cloud computing, giving the tester some strategies for questions that can be asked of cloud vendors and IT.

Cloud computing promises a huge number of advantages for software developers. There’s the ability to rent servers, to scale those servers out, to create new capacity on-demand and possibly even to eliminate the need for a physical data center.

Sadly, eliminating the need for testing isn't one of them.

Instead of a list of common failure points and things to test for, I'll focus on high-level risks. As a tester or IT professional, you'll want to begin a dialogue about elements which are within your role to address. Let's break these down into the following four areas: functionality, scalability, security and other business risks for cloud computing.

Functional testing

Starting with what doesn't change much, functional testing remains functional testing, regardless of where it’s executed.

All the traditional rules still hold. You'll want to perform quick attacks, break down the software into its business logic, execute domain modeling, conduct scenario testing and so on.

What will be different is how you create the environment. Unless you have physical servers you are "porting" to the cloud, you will likely need some sort of test set-up code. This code should be nearly identical to the production code, perhaps with a different domain name, less built-in scalability, or different data.

In other words, someone will need to test the process of building an electric data center, upgrading software within that data center, possibly saving the data locally. Are you that person?


Of course the cloud is supposed to scale up automatically -- but think for a bit about what that might mean. First, you have to notice that your servers are getting loaded, then create new servers for the need, then have a way of balancing requests among your servers, ideally sending traffic from the same client to the same server so that client’s data can be cached.

Then there is the database layer, which will likely have the same scalability issues as the Web server.

Plus you still have the classic problems of requirements for scale. No one will know how much the application will need to scale, how quickly it should scale, or exactly how much that should cost. My suggestion remains to conduct an experiment, and, if you can't, use production monitoring to respond to problems as they emerge.

One way to monitor performance is to use to a tool to physically login as a dummy user and time common operations, reporting that information in a database. Alternatively, you might pull those numbers from the server logs.  Notice that with cloud computing, the pain point shifts: Instead of the app slowing down, the hosting provider should spool up more servers. (Yes, you’ll want to test to make sure that happens.) These new servers likely have an hourly rate, so instead of an increase in system response time, you will see an increase in outgoing cost. 

Security and privacy

All the traditional rules about security testing apply here, all those and more. You'll still want to look into login, quick attacks, the security layer, fuzzing and social engineering issues. In addition to classic security concerns, your cloud-based application is likely outside of your data center. It may make some API calls into your data center (through the corporate firewall), or, perhaps, the entire application may exist outside of the firewall. Your next set of concerns are likely about the privacy of customer data, who might have access to that data "in the cloud," and consequences if that data is lost or stolen. This isn't something to minimize; hosted accounts may have backdoor URLs or may be running older versions of system software with known security issues. If your software in the cloud is using system software (like a Web server, a content management system, or a database), you'll probably want to keep up on security for that product and test upgrades and patches.

Business process risk -- where the real action is

Over and over again I hear that cloud is now a "utility grid" and has no risk; that just isn't true. Where I come from, any given rain or snowstorm might just take out our electric and phone lines, for up to days at a time.

It turns out that this happens on the Web, too. Even major and well-trusted cloud providers have experienced major downtime.

Another consideration is the cost of the system. Sure, the cloud may be able to scale to millions of simultaneous users, but that also involves scaling up a huge number of simultaneous virtual machines, each machine billing at perhaps fifty-cents per compute hour. A single week of 1,000 CPUs at fifty-cents an hour costs eighty-four thousand dollars. How, exactly, will your company predict and manage costs in this sort of world? (And we didn't even include bandwidth costs, which would also skyrocket.)

For a third risk, let's consider what happens after the $84,000 charge, or any large charge. Perhaps someone in accounts payable disputes the charge, perhaps something goes wrong in PayPal, perhaps some account is over-drawn. All of these could lead to your cloud vendor not getting paid. It's worth asking what happens if the monthly payment fails to go through, and the smart money is betting that the servers immediately go down -- possibly without any sort of backup.

Speaking of backup, let's talk about data. Using a public utility for computing means the data is no longer stored on your company’s computers; instead, it is stored in the cloud. Is that data backed up? How often is it backed up? What would a restore even look like? Can you export that backed up data to re-create and debug problems? For that matter, can the developers get full access to those machines (both production and test) to debug issues?

All of these are important questions. They are reasonable questions; they are the kind of questions that we testers are good at answering.

It turns out by answering these questions, we testers can add a lot of value to the development process, and help make sure that good decisions are made, both for the transition and if to undertake a cloud strategy at all.

Parting thoughts

This isn't a test strategy for cloud-based applications as much as a census of possible risks. Again, the next step is a dialogue. Which of these risks is the company concerned about, how are we addressing them, and which fall on our plate in the testing group?

Take these ideas, throw them in a bin, add a bit of organization, and then you'll have the start of a test strategy for cloud computing.

Dig Deeper on Cloud Application Testing

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.