Testers, welcome to the brave new world of cloud testing. Cloud testing offers us two opportunities: testing the cloud itself, and testing in the cloud. A cloud test strategy refers to testing the infrastructures, platforms and software packages that are provided through the "as a service" delivery model. A cloud test uses cloud technology in order to test more quickly, efficiently and effectively. Both cloud testing and testing in the cloud offer both challenges and advantages for testers. In the first part of this series, we explored how the cloud works. Now we'll look at both cloud testing and testing in the cloud and develop a framework for that testing.
Testing in the Cloud
Testing in the cloud offers us a whole new approach to testing, especially in the area of environment management. Traditionally, testers deal with contention for QA and test environments. Given the limited availability of infrastructure on premises, environments are often shared among the different test teams. Nonfunctional testing is often executed in the same environment as functional testing. Sharing environments results in schedule delays for all the test teams and projects involved. Often, test environments don't completely mirror production and this can results in missed bugs, especially in load and performance testing.
Testing in the cloud offers a solution to many of our test environment management issues. Cloud test environments can be provisioned quickly and easily in order to eliminate the need for sharing environments among test teams, eliminating environment-related schedule delays. Environments can be scaled to mirror production resulting in more accurate nonfunctional testing. QA environments and the provisioning thereof are owned by the vendor, which ensures standardization. Finally, the as a service model reduces cost.
Testing the Cloud
Although testing the cloud involves testing not only software as a service (SaaS), but also infrastructure as a service (IaaS) and platform as a service (PaaS), we'll concentrate on SaaS testing as it is what most testers will encounter. A SaaS cloud test comes with a unique set of challenges that must be considered before designing the strategy.
As with most testing, the challenges begin with requirements. SaaS products are not developed based on one company's set of requirements; rather, the product is customized or configured to meet the purchasing organization's needs. Some examples of SaaS products include Salesforce, SAP and Workday. Therefore the purchasing organization must engage in a gap analysis to determine how to fit its business processes into the existing software. This creates a challenge not only for business solutions analysts to develop the gap analysis, but for testers to apply it to testing. Since the core product code is the same for all purchasers, the cloud test focus changes from functional testing to business process testing.
It is crucial for testers to understand the difference between customizations and configurations and to test accordingly. Customizations refer to code changes made to the core product whereas configurations are options selected within the product. Both must be tested, but it is important to do regression testing around any customizations. In addition, most SaaS products will require integration with multiple downstream legacy systems. Full end-to-end systems' testing is a critical component of the cloud test strategy.
Dealing with Data
Data testing also poses some unique testing challenges in SaaS applications. Data migration is a key functionality, and testers must understand the data migration process and plan to test both statically and dynamically migrated data. Data privacy and security become critical components of the SaaS data test strategy since all data, including nonpublic information, will be transmitted to and stored in the cloud. Testers must include validating the security of the transmission channel as well as the cloud tenant in the test framework.
Nonfunctional testing for SaaS products is complex. In addition to the usual performance, stress and load testing, availability, reliability and scalability are important considerations for the cloud test plan. Security testing is probably the most complex nonfunctional test as multi-tenancy requires in-depth testing of roles and permissions. Multi-tenancy is the basis of SaaS architecture where the same code is deployed to multiple customers or tenants. The test must ensure that the purchasing organization's tenant operates in isolation from other tenants, since they are all running the same code.
Finally, since the SaaS vendor controls the frequency of patches and upgrades and the window of time purchasing organizations have to test, the cloud test strategy must include a plan for short iterations. Agile methodologies tend to work best in this environment.
On the other hand, SaaS testing is not without its advantages. Testers no longer need to include multiversion testing and backwards compatibility in their cloud test plans and there are no client server installations to test.
Although SaaS testing adds complexity in all phases of testing, our current testing skill set is the basis for moving into the cloud. Testers, as long as you are willing to broaden your horizons by thinking outside of the box and embracing the concepts of multi-tenancy and as a service, you will be well on your way to going where no testers have gone before -- into the cloud!
ALM and the cloud: here's whatyou need to know
Why software testers need to get used to change
The brave new world of software testing, part one