A guide to purchasing the right quality assurance software
A collection of articles that takes you from defining technology needs to purchasing options
Finding the right quality assurance and test management software for your organization can be a daunting process. The market for QA tools is very mature; there is a dazzling array of products from complete application lifecycle management (ALM) to niche software with support for testing various commercial off-the-shelf (COTS) packages. Since all of these products offer different combinations of features, it is critical to understand what an organization actually needs before making a purchasing decision.
The best way to understand an organization's needs is by looking at what led to considering a purchase of QA and test management software. Examining pain points and performing a needs analysis illuminate what specific features are most important to your organization. Understanding the demographics of the company and size of the test team reveal the kind of scope QA software should have. Analyzing the types of testing and the testing methodologies will point out needs for specific types of QA and test management tools, such as supporting agile or integrating with automation tools. Finally, researching the business climate and regulatory requirements will provide information on potential test process changes and reporting requirements that may impact the purchasing decision.
Identifying pain points easily solved by QA and test management software
The business case or use case for QA and test management software is developed from evaluating the organization's pain points and issues. This, along with the needs analysis, will be useful in narrowing down the choice of vendors and their software. There are many specific use cases for purchasing QA and test management software, but there are some common themes that an organization may fit into. If so, that company's specific pain points can be added in to create the business case. The common themes are as follows.
Large organizations often have a need for a standardized, repeatable process. Generally in these situations, testing is decentralized and business analysis, development, and release management teams may be using many different tool stacks. This type of environment indicates a need for a product that has integrations with many other products and could potentially become the basis of an ALM suite. When organizations find a continuing pattern of critical defects in production which jeopardizes their reputation and sales, there is a clear need for more effective testing. This scenario suggests that QA and test management tools that focus on traceability to requirements, test execution tracking and defect management would be of value.
More specialized businesses cases include organizations in regulated industries and organizations undergoing an agile transformation or moving to continuous integration (CI). Regulated industries have a need for test management software that provides smooth reporting features as these organizations must report to regulatory authorities early and often. Businesses that use or are planning to use agile or CI methodologies require QA and test management software that provides collaboration and work management features geared to quick but effective testing. In the sections below, there are some products and vendors that fit various business cases.
QA and test management software with a broad scope
For large organizations with distributed teams that need test process and efficiency, Hewlett Packard Enterprises' (HPE) ALM and Quality Center, Microsoft's Visual Studio Team Foundation Server (TFS), and Borland's Silk Central are good options. These tools provide not only strong QA and test management functionality but also complete application lifecycle management and integrations with many tool stacks including automation tools and include the collaborations features that distributed teams require.
Quality Center is the centerpiece of HPE's ALM software and provides 360-degree traceability from requirements through test cases and defects. In the most recent versions, HPE refers to Quality Center as ALM; however, this tool provides all the test management functions and integrates with other tools in the HPE suite to form a true ALM product. It provides integration with HPE's Performance Center, Load Runner and Storm Runner for performance and load testing and HPE's United Functional Test (UFT) as well as other third-party tools. UFT is comprised of Service Test, which is used for testing Web services, and Quick Test Professional, which is a premier tool for automated functional testing. Although licensing is pricey and complicated, it is easy to use and most testing organizations throughout the world have at least some familiarity with it.
Microsoft's TFS is both an ALM and DevOps product, providing version control, work item tracking, reporting, and automated builds. Through its integration with Microsoft's Visual Studio Developer tools, TFS offers a collaborative environment in which to manage the entire requirements gathering, development, test and release management processes. TFS's team rooms facilitate communication among decentralized team members. Although TFS is geared to more at QA testers with more technical skills than Quality Center, it is a particularly good choice for organizations that are already using Visual Studio.
Borland's Silk Central is a complete test management product that provides application lifecycle management by offering out-of-box integrations with many requirements management, development, versioning and defect management tracking tools. Through integration with Silk Performer, Silk Test and Silk Mobile, it provides complete performance and automation capabilities. Silk Central's Test Book feature tracks test activities and allow for collaboration among test teams.
QA and test management software for agile or CI methodologies
Organizations that are currently using an agile or CI methodology or are engaged in an agile transformation will need QA and test management software that features collaboration, support for exploratory testing, and integration with development, automation and CI tools. Product choices for this scenario include Microsoft's TFS as well as Telerik's Test Studio, QA Symphony's qTest and JIRA's Clover.
Microsoft's TFS provides integrations and collaboration features as described above. TFS is a particularly good choice for agile and CI shops because it features integration with DevOps as well as ALM tools. With its requirements management features, product owners become part of the team with developers and testers.
Telerik's Test Studio is a great choice for agile organizations because it features strong exploratory testing and provides a Visual Studio plug-in for collaboration between testers and developers through automation. Test Studio supports CI by providing functionality that allows for integrating any build server with recorded tests.
QA Symphony's qTest is a QA and test management tool developed specifically to support agile methodology. It provides integration with agile ALM tools including Rally and VersionOne as well as with JIRA and other defect management tools. The seamless test management that qTest creates for all types of testing including manual, exploratory and automated tests, by integrating with QA Symphony's tool suite, qTest eXplorer and qTest Scenario. Exploratory testing capabilities fall under the purview of qTest eXplorer, and qTest Scenario provides support for shift left test approaches including test driven development. The tools are intuitive and easy to use, and qTest has strong collaboration features that provide the ability to share test cases.
For agile and continuous integration shops that are focused on code coverage, JIRA's Clover offers complete code coverage that easily exposes incomplete test coverage. Clover also includes a test optimization feature that automates test in order to run them faster and it also runs integrated development environments and continuous integration systems. Although Clover is not a complete QA and test management tool, it offers test management value, especially in the CI arena.
For organizations with specialized needs
Organizations with specialized testing needs, including mobile applications or COTS software, and organizations within regulated industries should evaluate vendors that offer superior capabilities in those areas.
Telerik's Test Studio, Borland's Silk Central and SmartBear's QAComplete offer strong capabilities in mobile testing. Test Studio supports many platforms and browsers by using object identification and allows for automating tests directly on the device or through emulation. Silk Central provides support for mobile testing for both Web and native applications.
SmartBear's QAComplete and its companion products strongly support both mobile and Web testing. It has integrations with JIRA and Bugzilla for defect management and integrates with SoapUI and Selenium as well as SmartBear's companion automation tools, TestComplete, MobileComplete and LoadComplete.
Although both HPE's Quality Center and Borland's Silk Central provide support for COTS testing, if your organization needs to test specific COTS packages, it is worthwhile to look at vendors who provide support specifically for COTS packages. Worksoft provides a suite of tools for analyzing and testing business processes. Worksoft Analyze is a tool within that suite that documents business processes, and Worksoft Certify including Certify Mobile and Certify Performance are Worksoft tools geared specifically to automate business process testing.
Organizations in safety critical or regulated industries should look at vendors whose traceability, documentation and reporting capabilities are strong. Although all of the market leaders including HPE's Quality Center, Microsoft's TFS and Borland's Silk Central have efficient dashboards, Seapine Software's TestTrack provides all of the traceability and the reporting capabilities that regulated organizations need. Requirements, test plans, source code and a help desk are linked in one platform, providing visibility and traceability. TestTrack's reporting capabilities include analyzing bugs across releases, tracing project work from design and development through testing as well as test coverage and effectiveness.
If your organization needs QA and test management software, but the budget is limited, Gurock's Test Rails or SmartBear's QAComplete would be perfect choices.
SmartBear's QAComplete is a comprehensive test management package that provides traceability between requirements, test cases and defects as well as slick reporting features. SmartBear provides a complete package much like the market leaders at a much lower price. Gurock's Test Rail also provides complete test management including integrations to defect management tools. Test Rail's metrics and reporting features are efficient and easy to use and it provides nice collaboration features for distributed teams. Although they provide fewer "bells and whistles" than the market leaders, Test Rail and QAComplete offer core test management functionality at a reasonable cost.
Buying QA and test management software is not difficult when it is approached in three steps. First, perform a needs analysis to document the organizations requirements. Then, build the business case by reviewing the pain points or specific issues that the purchase of QA and test management software will mitigate. Finally, use these analyses to narrow down the choice of vendors who best meet the needs of the organization.
This is part four of a four-part series covering quality assurance software. Part one examines the role QA software plays in enterprises. Part two presents use cases for QA software. Part three highlights features to consider when procuring QA software.
The choice between emulators or real devices doesn't have to be a choice in testing anymore.
Get on the Lean bandwagon.
QA testers should act like journalists.