This content is part of the Essential Guide: Prepare and manage enterprise apps for an IaaS model
Manage Learn to apply best practices and optimize your operations.

How to craft a workable cloud test strategy

When the brave new world of testing meets the cloud, get ready to be flexible. Expert Gerie Owen shares the secret of crafting a workable cloud test strategy.

Testers, welcome to the brave new world of cloud testing. Cloud testing offers us two opportunities: testing the...

cloud itself, and testing in the cloud. A cloud test strategy refers to testing the infrastructures, platforms and software packages that are provided through the "as a service" delivery model. A cloud test uses cloud technology in order to test more quickly, efficiently and effectively. Both cloud testing and testing in the cloud offer both challenges and advantages for testers. In the first part of this series, we explored how the cloud works. Now we'll look at both cloud testing and testing in the cloud and develop a framework for that testing.

Testing in the Cloud

Testing in the cloud offers us a whole new approach to testing, especially in the area of environment management. Traditionally, testers deal with contention for QA and test environments. Given the limited availability of infrastructure on premises, environments are often shared among the different test teams. Nonfunctional testing is often executed in the same environment as functional testing. Sharing environments results in schedule delays for all the test teams and projects involved. Often, test environments don't completely mirror production and this can results in missed bugs, especially in load and performance testing.

Testing in the cloud offers a solution to many of our test environment management issues. Cloud test environments can be provisioned quickly and easily in order to eliminate the need for sharing environments among test teams, eliminating environment-related schedule delays. Environments can be scaled to mirror production resulting in more accurate nonfunctional testing. QA environments and the provisioning thereof are owned by the vendor, which ensures standardization. Finally, the as a service model reduces cost.

Testing the Cloud

Although testing the cloud involves testing not only software as a service (SaaS), but also infrastructure as a service (IaaS) and platform as a service (PaaS), we'll concentrate on SaaS testing as it is what most testers will encounter. A SaaS cloud test comes with a unique set of challenges that must be considered before designing the strategy.

As with most testing, the challenges begin with requirements. SaaS products are not developed based on one company's set of requirements; rather, the product is customized or configured to meet the purchasing organization's needs. Some examples of SaaS products include Salesforce, SAP and Workday. Therefore the purchasing organization must engage in a gap analysis to determine how to fit its business processes into the existing software. This creates a challenge not only for business solutions analysts to develop the gap analysis, but for testers to apply it to testing. Since the core product code is the same for all purchasers, the cloud test focus changes from functional testing to business process testing.

It is crucial for testers to understand the difference between customizations and configurations and to test accordingly. Customizations refer to code changes made to the core product whereas configurations are options selected within the product. Both must be tested, but it is important to do regression testing around any customizations. In addition, most SaaS products will require integration with multiple downstream legacy systems. Full end-to-end systems' testing is a critical component of the cloud test strategy.

Dealing with Data

Data testing also poses some unique testing challenges in SaaS applications. Data migration is a key functionality, and testers must understand the data migration process and plan to test both statically and dynamically migrated data. Data privacy and security become critical components of the SaaS data test strategy since all data, including nonpublic information, will be transmitted to and stored in the cloud. Testers must include validating the security of the transmission channel as well as the cloud tenant in the test framework.

Nonfunctional testing for SaaS products is complex. In addition to the usual performance, stress and load testing, availability, reliability and scalability are important considerations for the cloud test plan. Security testing is probably the most complex nonfunctional test as Multi-tenancy requires in-depth testing of roles and permissions. Multi-tenancy is the basis of SaaS architecture where the same code is deployed to multiple customers or tenants. The test must ensure that the purchasing organization's tenant operates in isolation from other tenants, since they are all running the same code.

Finally, since the SaaS vendor controls the frequency of patches and upgrades and the window of time purchasing organizations have to test, the cloud test strategy must include a plan for short iterations.  Agile methodologies tend to work best in this environment.

On the other hand, SaaS testing is not without its advantages. Testers no longer need to include multiversion testing and backwards compatibility in their cloud test plans and there are no client server installations to test.

Although SaaS testing adds complexity in all phases of testing, our current testing skill set is the basis for moving into the cloud. Testers, as long as you are willing to broaden your horizons by thinking outside of the box and embracing the concepts of multi-tenancy and as a service, you will be well on your way to going where no testers have gone before -- into the cloud!

Next Steps

ALM and the cloud: here's whatyou need to know

Why software testers need to get used to change

The brave new world of software testing, part one

Dig Deeper on Topics Archive

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

What has been your biggest challenge when building a cloud test strategy?
Surely cloud based apps need to be working towards Continuous Delivery (CD), which in turn implies some degree of Test Driven Development. The key metric of CD is the time between making a change to a line of code (either application or deployment automation) and making the change available in production. Typical times for largescale applications would be an hour or so, including all data migration and all regression testing.

It *is* possible to do it the way around noted, but the variable cost of running a regression test really slows down the economic rate of change of the developed systems.

Pure SaaS products such as have incorporated tooling to enable test automation.

More of a challenge for complex integration is that while 'gap analysis' is easy, 'overlap analysis' is not (i.e. convincing sets of application components that the world that they understand is consistent from each's point of view).
I would like to know challenges faced in building test strategy for apps moving from on-prem to cloud. Any pointers greatly appreciated.