A guide to purchasing the right quality assurance software
A collection of articles that takes you from defining technology needs to purchasing options
There are many scenarios that indicate an organization has a clear need for quality assurance and test management software. However, once the need is established, an organization needs to somehow evaluate the multitude of products on the market today in order to determine what QA software to purchase. To do this, an organization must look not only at its pain points, but also at what it is hoping to achieve through the purchase.
Performing a needs analysis is the first and most critical step in the purchasing process. The needs analysis should focus on a review of the current QA and testing process including test and QA professionals, stakeholders and test methodology; the types of testing performed by the organization; and the business climate and market conditions including industry- and organization-specific regulatory requirements.
Starting the QA and testing needs analysis
A needs analysis begins with a review of the current QA and testing process, or the approach the organization takes to testing. Look to the current process to determine the features that are required. If there isn't a current process, try to envision what QA process the organization will use. Processes determine the features of QA software, features that usually include the ability to trace requirements to test cases, possibly even to defects; creation and execution of test cases; defect management; status reporting; and collaboration. It may also include some form of automated testing.
The quality assurance and testing process analysis includes not only a review of the test methodology, but also an understanding of the state of the testing and of the quality assurance professionals and others who are or may become involved in the process, because they are the primary users of QA software. Specifically, it is important to know their skill sets, levels of experience and where they are located. Knowing if the QA staff is comprised of dedicated testers or if testing is handled by the development or business team is an important factor to consider, because different workers coming from different departments have different skill sets. The number of testers employed by the organization and their location within the organization and geographically play into the purchasing process.
Knowing your testers will provide information needed for the number of licenses, licensing model (concurrent vs. named) and cost, service level agreements for support, training options, and whether to choose an on-premises or software as a service (SaaS) product.
In addition, it is important to consider the needs of project managers, developers, business systems analysts, product owners, scrum masters, steering committees, senior management and even user representatives -- all of whom could be stakeholders in the test process. Understanding their roles will also generate requirements for QA and test management software. For example, QA software offers various features for creating status reports and generating metrics.
Defining the QA and testing process
The next step in the quality assurance and testing process analysis is a review of the company's methodology for the entire software development lifecycle (SDLC). This review includes not only the methodology, but also how it may change in response to competition, regulatory requirements or any other organization-specific criteria.
A methodology analysis is important as it will flush out requirements for integrations and possibly the need for application lifecycle management (ALM) software rather than -- or in conjunction with -- quality and testing software. Even if the organization does not currently need ALM software, there may be a need in the future, so it is critical to include integration with ALM as a requirement.
During the methodology analysis, review the SDLC methodology that is currently in use in the organization as well as any changes that may be under consideration. In today's business climate, most organizations that are not currently using an Agile methodology are considering implementing it. For example, if the current process is Agile, there may be plans to implement continuous integration (CI). It is important to look at market forces that may impact the choice of methodology as these may drive an Agile transformation or a move to CI.
Organizations using Agile or continuous development will need a test management tool that integrates with requirements management and release management tools in order to support the rapid speed to market. For CI, it is also useful to have a tool that supports BVT (Build Verification Testing). There are tools with add-ons for Agile and CI, which might be worth considering if a methodology change is not currently under consideration.
The testing drill down
Since the main role of QA and test management software is to facilitate testing, it is important to take an in-depth look at not only the test process, but also at what is tested. Although some organizations develop custom applications in house, many companies purchase commercial off-the-shelf (COTS) or SaaS products. Testing these products requires a different focus, which can generate special requirements for your test management and quality assurance software. For COTS and SaaS testing, consider features that focus on business process tools that have integrations with COTS packages. If the organization is heavily invested in one vendor, such as Oracle or SAP, it might be worthwhile to consider a vendor-specific tool.
Test automation strategy is another important consideration. Most organizations have a need for some automation, especially for data testing, Web services, performance and mobile testing. For Agile and CI shops, automated unit and regression testing is a must. If automation is already a component of the organization's testing strategy, then it is important to include integration with automation tools you are currently using as a requirement for QA and testing software. If the organization is considering purchasing automation tools, it is worthwhile to consider products that have a test suite that includes both test management and automation tools.
Taking the temperature
The final part of the needs analysis includes a review of the competitive business climate and industry- and organization-specific needs. In most industries, organizations have a need for quicker time to market. Most customers demand a flawless customer experience across all digital channels, and, with the proliferation of social media, customers will share their experiences freely. This means that a poor experience can negatively impact a company's reputation.
These market forces may show capabilities to look for in test management and quality assurance software. Examples of these capabilities might include a need to integrate with application performance management software for monitoring performance in production or integration with specialized mobile testing emulation or device clouds. As another example, traceability features -- which link defects back to test cases, requirements, easy-to-prepare status reports and metrics -- are usually important to organizations in regulated industries.
The needs analysis provides a framework for evaluating QA and testing software by clarifying the organization's requirements and expectations. The importance of the needs analysis and product-related considerations cannot be underestimated; an in-depth understanding of needs and expectations forms the basis for a well-informed buying decision.
Where do QA engineers fall in the software design process?
Make your QA reports acceptable.
Taking QA on the road.