With applications that have lines of code numbering in the millions, how do testers know what's been looked at and tested? Does each line do something, or are there dead ends, and a graveyard of methods and syntax lying around in your application? Outside of the code, how much of a product's features have been tested? Is there a way to tell? As an important aspect of application testing, test management software provides coverage, telling testers what percentage of software has been exercised by tests, and how many requirements -- and subsequent tests -- have been run. This article covers the two types of coverage software: code coverage and test case management tools.
What is measured by a code coverage tool?
Each one of a programmer's automatic microtests -- sometimes called unit tests -- covers a small sliver of functionality. Combined, they might, or might not, cover a large part of the application. Code coverage tools run all the unit tests, and analyze what functionality was called and what is missing. Results like this can help programmers add tests or find new, unconsidered paths through the software. It is also possible to track if all the code is exercised during customer-facing testing, see what lines of code have not run and change the test strategy to address them. Froglogic Squish, JetBrains, and Atlassian Clover are all software products designed to provide code coverage for the enterprise.
How does test management software differ from code coverage?
Test management software asks how many features in the system were tested, and highlights what has not been tested. It is possible to track this manually, with a traceability matrix that lists every requirement and the tests that exercise that requirement. Those tests could be stored in wiki or Microsoft Word documents. The pain happens when an auditor asks if the tests have been run, by whom, when and against what version of the code. Testers track what they do in those tools, and, at the end of each release, the organizations can determine if the coverage is high enough, if the test cases they have compiled have been run and so on.
HPE Quality Center
Hewlett Packard Enterprise Quality Center is more than just coverage software. HPE considers Quality Center to be a unified platform for requirements gathering, test planning and scheduling, analyzing of results, and reporting and tracking of issues. HPE Quality Center is based on tools originally created by Mercury Interactive, which was purchased by HPE in 2006. Software testers with experience using Mercury's tools will feel right at home with HPE Quality Center.
Quality Center is designed to interact with a suite of other HPE applications, including its Unified Functional Testing tool and Sprinter, which is an exploratory test coverage tool. If you are already using these tools, Quality Center fits right in with them.
HPE Quality Center runs on Windows Server, and, therefore, would be appropriate for environments based on Windows and .NET. On-site installations require registration and licensing. A free community version is available -- limited to five named users. Quality Center is also available in a software-as-a-service model. For more details about HPE Quality Center pricing and trial downloads, visit its site.
IBM Rational Quality Manager
Rational Quality Manager from IBM allows software testing teams to track multiple areas, including goals, schedules, milestones and exit criteria, along with test cases, requirements and development initiatives. It is designed to be run and managed through a web interface, so users can access it by web browser.
Rational Quality Manager can share requirements information with Rational Requirements Composer, and share work items and defects with Rational Team Concert. It utilizes Jazz.net, which is a unifying framework for Rational tools, and utilizes an API to interact and integrate with third-party applications. This is a platform that is primarily designed to take advantage of .NET programming environments. IBM offers a free 90-day trial.
SpiraTeam from Inflectra is an all-in-one test management software, with an emphasis on combining requirement, test and defect management. By mapping tests to requirements, SpiraTeam can help inform test coverage and time to completion. Reporting and dashboards can be customized to show the details most important to the team. Additionally, SpiraTeam has the ability through plug-ins to integrate with automated unit, functional and load testing tools from different vendors.
Each project has a dashboard showing all the information regarding the overall status and health of the project in an easy-to-view and easy-to-read format. SpiraTeam is designed with Agile development in mind, and aims to be the tool of choice for those who wish to use Scrum, Agile unified process, XP and dynamic systems development method, among others.
SpiraTeam also provides mobile web capabilities, so that smartphones, tablets and other devices can view dashboards, as well as execute tests from mobile devices. SpiraTeam supports iOS, Android and Windows Phone. Additionally, SpiraTeam has a broad array of tools with which it can be integrated and for which it can provide additional functionality. This allows organizations to leverage their existing infrastructure and artifacts while they transition from a traditional development approach to a more Agile approach.
QASymphony qTest platform
The QASymphony qTest platform integrates a number of elements, including requirements management, test design and execution, defect tracking and report creation. Additionally, qTest also offers eXplorer, which is an application designed to aid with exploratory testing charters. EXplorer lets users record interactions to create a record of what they have seen during their software testing sessions. QMap allows managers to identify which features have been tested, bugs discovered and amount of time testing.
QTest is cloud-based software, which is good for organizations that do not want to deal with administration issues. Additionally, since it is in the cloud, all that is needed to interact with it is a web browser. This makes it available to organizations with multiple sites and globally distributed team members. With an emphasis on providing rich functionality out of the box, less time is needed for setup and modification.
TestTrack from Seapine is another hybrid service that aims to cover requirement management, test case management, and bug reporting and tracking. Rather than try to make your team test in a particular way, TestTrack uses task boards to allow you to design your tests, requirements and tracking to match the way you work. If you are Agile, Waterfall or a hybrid of both, TestTrack allows the team to communicate using the processes that work for them. TestTrack shines as an integrated suite, yet allows teams to license only what they need, with the option of adding different components later if desired.
The test management software component can be purchased and run as a stand-alone service. TestTrack runs in the cloud and is offered via a subscription model. The test case management component offers unlimited projects, along with free support and upgrades. To request a trial, see Seapine's TestTrack product page.
Monitoring your apps
Application performance monitoring (APM) tools running on a development or test environment can give a combined -- application level and code level -- view of coverage. APM tools monitor and report on exactly what part of the code is being called when a user is working in the software. If a user logs in, navigates to the bookstore and adds two books to her cart and then checks out, each API call or method used will be noted. This creates a technical view of coverage that relates back to how customers will use the program.
TestCaddy is a tool focused on manual tests, and the development and management of those tests. TestCaddy uses the idea of "test sets" that can be applied to multiple application testing cycles.
TestCaddy is a native Win32 desktop application, which means installation and setup can be handled quickly on individual machines where it is needed. TestCaddy also integrates with Atlassian JIRA, and can also be used alongside Bugzilla.
TestCaddy utilizes dashboards that allow all information to be seen on one screen and minimizes the need to run multiple reports. Additionally, through the use of a central database, users can access any project's data without having to reconfigure or move machines. For organizations that want the flexibility of a central database with the quick response of a native application, TestCaddy aims to provide both. TestCaddy is not available at this time for Linux or Mac.
Zephyr emphasizes "real-time test management" by providing its own set of tools and integrating with other tools. Zephyr integrates the ability to create tests, plan application testing efforts, execute tests and track those efforts. By utilizing add-ons, Zephyr allows teams to extend their reach into other tools -- such as JIRA and Confluence -- and by using ZBots, allows remote automation systems to be kicked off and report back results for display in Zephyr. For teams with existing assets in other tools, Zephyr aims to integrate with a broad range of tools, rather than force adoption of their way of working. A trial version of the enterprise edition is available, and a community version is available for free for teams of 10 or fewer users.
We want to know: How does your company test mobile applications?
As testing costs are set to rise, the right testing tools are even more important
These testing best practices can help with efficiency and release time
Does Rational Quality Manager meet your app testing needs?