Why is SaaS testing harder than traditional testing?

SaaS development cycles are shorter than traditional Web apps, require more test types and result in greater scrutiny from quality assurance.

Testing is testing, or is it? There are several differences between software as a service (SaaS) testing and testing

traditional Web applications.

The differences include the focus or type of testing and the amount of testing needed. SaaS is sold as a faster, more efficient and effective method for getting software products out to customers. It is proven to be successful. However, testing and quality assurance (QA) has become more critical as applications are delivered more rapidly. The amount of testing increases, but the focus of testing also differs between SaaS testing and traditional Web applications. The size of the difference depends on the development methodology used and the types of testing currently done in an organization.

The importance of a QA test plan

SaaS releases occur continuously, or at least more frequently than traditional releases. When a software development team moves rapidly, that means things need to be organized, including testing. The testing needs to be documented, organized and defined in the QA test plan.

SaaS is intended to be a fast and effective manner of getting application releases in the hands of customers.

Regardless of the types of testing used, a detailed QA test plan is essential for SaaS because it defines scope, elements to be tested, test tools used and the level of testing done for each SaaS component and application feature. Additionally, the plan includes test definitions for the automated, unit, manual, performance, security and load tests, as well as who executes them and the process for handling failures. Having a QA test plan to share with users creates confidence in the quality of the release and decreases incidences of confusion within the development team, as well as failed releases.  

Increases testing effort

It's ironic that the compressed development cycle for SaaS actually increases the overall testing effort. This is because a release requires testing, whether it occurs weekly, daily or hourly. Each deployment of code to a test server needs test execution; the same holds for each time the code is deployed to staging or even to production.

Additionally, SaaS testing tends to require executing a greater number of test types. Service-level agreement (SLA) adherence, failover/disaster recovery and deployment are examples of SaaS tests that are typically not part of traditional Web application testing. These may be tested in standard Web applications, but they generally are not deemed critical. In SaaS, SLA adherence is required in order to avoid business disruption.

Failover and disaster recovery are essential in order to verify the SaaS is solid and responds appropriately if a release or server fails. Deployment testing is related to the involvement of operations personnel in an Agile development team as DevOps engineers. In SaaS systems, the rapid deployment schedule forces testing during the deployment process on test and staging servers before code reaches production. This is linked partially to a no-downtime philosophy for SaaS deployments.

Testing changes

What determines if testing types truly change depends on the organization and its actual testing habits. Many organizations run mainly functional and performance testing on Web applications, regardless if they are traditional or SaaS-based. Both the focus and priority of testing types change for organizations that test more than function and performance.

Unlike traditional Web application testing, SaaS testing is not affected by client or server installation, versioning, backwards compatibility or multi-platform testing. However, testing security and access, SLA adherence, deployment, failover and interfaces between components are increasingly critical. With SaaS testing, automation is essential for functional, performance and unit testing. Additional unit testing may be added to cover interface integration points and failover response. Effective automation will simultaneously improve user experience significantly, reduce the impact of additional testing needs on the release cycle, and free up QA resources to focus on customer-centric areas.

In summary, SaaS is intended to be a fast and effective tool for delivering application releases into the hands of customers. In order to support this, development organizations need to alter the focus or type of their testing and use a QA test plan in order to make sure releases happen correctly. A QA test plan is also useful in defining testing so teams and application releases remain organized and on time.

Next Steps

Read more about quality assurance.
Learn about QA testing for mobile.
Read more about QA testing.

This was first published in July 2014

Dig deeper on Cloud Application Testing

Pro+

Features

Enjoy the benefits of Pro+ membership, learn more and join.

Related Discussions

Amy Reichert asks:

What types of applications is your organization offering as a service? What difference has that made for your team?

0  Responses So Far

Join the Discussion

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

SearchSOA

TheServerSide

SearchCloudApplications

SearchAWS

SearchBusinessAnalytics

SearchFinancialApplications

SearchHealthIT

Close