This content is part of the Buyer's Guide: A guide to purchasing the right quality assurance software
Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Needs analysis determines QA and testing software direction

Performing a needs analysis and identifying pain points, methodologies and vision for the future can help you determine which QA and testing software fits your enterprise.

There are many scenarios that indicate an organization has a clear need for quality assurance and test management software. However, once the need is established, an organization needs to somehow evaluate the multitude of products on the market today in order to determine what QA software to purchase. To do this, an organization must look not only at its pain points, but also at what it is hoping to achieve through the purchase.

Performing a needs analysis is the first and most critical step in the purchasing process. The needs analysis should focus on a review of the current QA and testing process including test and QA professionals, stakeholders and test methodology; the types of testing performed by the organization; and the business climate and market conditions including industry- and organization-specific regulatory requirements.

Starting the QA and testing needs analysis

A needs analysis begins with a review of the current QA and testing process, or the approach the organization takes to testing. Look to the current process to determine the features that are required. If there isn't a current process, try to envision what QA process the organization will use. Processes determine the features of QA software, features that usually include the ability to trace requirements to test cases, possibly even to defects; creation and execution of test cases; defect management; status reporting; and collaboration. It may also include some form of automated testing.  

The quality assurance and testing process analysis includes not only a review of the test methodology, but also an understanding of the state of the testing and of the quality assurance professionals and others who are or may become involved in the process, because they are the primary users of QA software. Specifically, it is important to know their skill sets, levels of experience and where they are located. Knowing if the QA staff is comprised of dedicated testers or if testing is handled by the development or business team is an important factor to consider, because different workers coming from different departments have different skill sets. The number of testers employed by the organization and their location within the organization and geographically play into the purchasing process.

The importance of the needs analysis and product-related considerations cannot be underestimated.

Knowing your testers will provide information needed for the number of licenses, licensing model (concurrent vs. named) and cost, service level agreements for support, training options, and whether to choose an on-premises or software as a service (SaaS) product.

In addition, it is important to consider the needs of project managers, developers, business systems analysts, product owners, scrum masters, steering committees, senior management and even user representatives -- all of whom could be stakeholders in the test process. Understanding their roles will also generate requirements for QA and test management software. For example, QA software offers various features for creating status reports and generating metrics.  

Defining the QA and testing process

The next step in the quality assurance and testing process analysis is a review of the company's methodology for the entire software development lifecycle (SDLC). This review includes not only the methodology, but also how it may change in response to competition, regulatory requirements or any other organization-specific criteria.

A methodology analysis is important as it will flush out requirements for integrations and possibly the need for application lifecycle management (ALM) software rather than -- or in conjunction with -- quality and testing software. Even if the organization does not currently need ALM software, there may be a need in the future, so it is critical to include integration with ALM as a requirement.

During the methodology analysis, review the SDLC methodology that is currently in use in the organization as well as any changes that may be under consideration. In today's business climate, most organizations that are not currently using an Agile methodology are considering implementing it. For example, if the current process is Agile, there may be plans to implement continuous integration (CI). It is important to look at market forces that may impact the choice of methodology as these may drive an Agile transformation or a move to CI.

Organizations using Agile or continuous development will need a test management tool that integrates with requirements management and release management tools in order to support the rapid speed to market. For CI, it is also useful to have a tool that supports BVT (Build Verification Testing). There are tools with add-ons for Agile and CI, which might be worth considering if a methodology change is not currently under consideration.

The testing drill down

Since the main role of QA and test management software is to facilitate testing, it is important to take an in-depth look at not only the test process, but also at what is tested. Although some organizations develop custom applications in house, many companies purchase commercial off-the-shelf (COTS) or SaaS products. Testing these products requires a different focus, which can generate special requirements for your test management and quality assurance software. For COTS and SaaS testing, consider features that focus on business process tools that have integrations with COTS packages. If the organization is heavily invested in one vendor, such as Oracle or SAP, it might be worthwhile to consider a vendor-specific tool.

Test automation strategy is another important consideration. Most organizations have a need for some automation, especially for data testing, Web services, performance and mobile testing. For Agile and CI shops, automated unit and regression testing is a must. If automation is already a component of the organization's testing strategy, then it is important to include integration with automation tools you are currently using as a requirement for QA and testing software. If the organization is considering purchasing automation tools, it is worthwhile to consider products that have a test suite that includes both test management and automation tools.

Taking the temperature

The final part of the needs analysis includes a review of the competitive business climate and industry- and organization-specific needs. In most industries, organizations have a need for quicker time to market. Most customers demand a flawless customer experience across all digital channels, and, with the proliferation of social media, customers will share their experiences freely. This means that a poor experience can negatively impact a company's reputation.

These market forces may show capabilities to look for in test management and quality assurance software. Examples of these capabilities might include a need to integrate with application performance management software for monitoring performance in production or integration with specialized mobile testing emulation or device clouds. As another example, traceability features -- which link defects back to test cases, requirements, easy-to-prepare status reports and metrics -- are usually important to organizations in regulated industries.

The needs analysis provides a framework for evaluating QA and testing software by clarifying the organization's requirements and expectations. The importance of the needs analysis and product-related considerations cannot be underestimated; an in-depth understanding of needs and expectations forms the basis for a well-informed buying decision.

Next Steps

Where do QA engineers fall in the software design process?

Make your QA reports acceptable.

Taking QA on the road.

Dig Deeper on Topics Archive

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

What would your needs analysis reveal about your organization's need for QA and testing products?
One thing it would show is the change we’ve made from cumbersome, process-heavy tools to more agile, lightweight tools that address more than just the needs of the QA team.
I got an impression that the article describes a very "heavy weight" and formal approach. One critical thing I couldn't find is an emphasis on learning through experimentation. Proof of concept aka "pilot approach" is an industry common nowadays.
I need flexibility.  If my team decides that prescriptive test cases are a waste of time, then I need a way to provide documentation of what we have tested and capture defects as we find them rather than a tool that feels cold to the touch and cannot change as our team changes.
There are so many products out there that, even with a needs assessment, its hard to narrow down the selection. I would also add other members of the team into the needs assessment, such as developers, BA/SA, UX, etc, because they will have needs that the software should address as well.
The common fallacy I've seen too frequently during my 2 decades in IT is buying tools first then thinking how to utilize them. The process should come first. Skills should come first. Then we can think if we need any tools. Highly skilled professionals tend to be more efficient with lightweight tools and flexible processes.
There is some good advice here, but be careful not to look at the tool and try to make it fit into your model.  And don't buy it because 'ooo it creates flashy metrics'.  There is much misuse of metrics in the software industry and testing software may also have the same problem.

It is far more important to have a tool that is flexible and that you use it for your needs, not just because someone else suggests they use it in a way that is different than yours.  The grass is not always greener on the other side, the same is true with practices when using testing software.
We don't utilize any kind of test management tool, and for the most part, we're doing fine. We do not have any defect management. We get by because defects get fixed, and no one is asking for reporting/tracking of defects, so we go without it.

Also, our status reporting is manual - each day at our standup we verbally communicate a status.

This process seems to work well for us.