Choosing the best software testing tools for your business
A collection of articles that takes you from defining technology needs to purchasing options
Application testing's wide variety of technologies and styles make creating a single, universal test tool impossible -- companies need a toolbox. In this article, we focus on automated functional testing tools, a type of software testing tool that replaces human activities with a computer.
Automated functional testing tools typically drive the UI, mimicking the actions of a human, and then check for expected results. Record-playback tools let a user record interactions. This creates a literal script that can be played back again and again. The drawback to record-playback tools is how they work; they can often fail when a code change alters the appearance of a screen, making it different than the expected results. Because of this, many of these scripts are stripped down and made less literal.
Checking the text on a button, instead of the entire window or webpage, makes the tests less brittle. In that process, we replicate the other approach often used with these tools: keyword-driven tests. Keyword testing is often more robust and less error-prone because it is able to look at the specifics of interactions, rather than every aspect for each line in the script.
In this area, there's an important question to ask: What type of application am I testing? Are we looking at a compiled desktop application, a website, a contained web application, a mobile website or a mobile application? Each requires a different approach and different tools. Native mobile applications can be extremely challenging to automate or test in a desktop; mobile web applications can run in a desktop, but may look or behave differently. Unless it's an all-in-one tool, it is unlikely that a web automation test tool supports mobile applications.
All-in-one automated functional testing products
Less-technical testers who work on Windows machines and don't want to write code should consider SmartBear's TestComplete. Based on record-playback, TestComplete has two views: visual and code. The visual view of TestComplete shows the commands, such as click, wait, type and so on followed by parameters -- where to click, how long to wait and what to type. That means a business user can look at the test and read it, agreeing that it's what the software should do.
Programmers can dive into code in TestComplete, but wrap that code in a function. For example, the login function can be a single line in the visual view, taking a username and password as inputs.
TestComplete is ideal for testers who don't want to program and work on Windows devices, and who want the flexibility to test Windows, web and mobile -- iOS and Android -- applications all in one package. Teams that want to version control tests right along with code will want to use Git, Subversion, Mercurial or Microsoft Team Foundation Server; the application uses its own format for data, and you'll want to check code in from the TestComplete development environment.
TestComplete provides additional paid support, including a training package of videos you can purchase; a monthly, two-day remote video training course; in-person training; and hourly support. The biggest issues we have experienced with TestComplete are supporting just-released versions of browsers, especially Internet Explorer, along with Java Applets running in the browser. SmartBear does work actively to support new browsers, but automatic updates can sometimes trigger before the SmartBear team has a new version in place.
Pricing for the TestComplete Platform is based on two models. You can purchase a Node-Locked License, meaning the software is tied to a single machine, or a Floating User License, which can be installed on multiple machines, but is tied to a single user account. All users purchase the TestComplete Platform license first, and then choose to purchase licenses for web, desktop, mobile or all of the above.
TestComplete has been expanding into the open source world, adding recent support for Jenkins and Selenium by way of Java's JUnit. If your team is using those tools already and you want to run every application test in one place, TestComplete might be the right choice.
Hewlett Packard Enterprise UFT
If your concerns go beyond automating the software tests to providing evidence the tests ran, when they ran and on what build and server, you might consider Hewlett Packard Enterprise's UFT, which stands for Unified Functional Testing.
Hewlett Packard Enterprise's product, formerly known as QuickTest Pro, is designed to integrate with Quality Center, an application test coverage tool that measures what was tested, along with Sprinter, an exploratory test coverage tool. Together, the three apps provide documentation and evidence that an insurance, medical or financial services company might need, especially to stand up to auditors. The applications work together to create a sort of quality suite.
Better known as part of a suite than a stand-alone application, UFT becomes a clear recommendation for organizations that already work in a Hewlett Packard Enterprise environment. Companies that already rely on Hewlett Packard Enterprise for service and support may find UFT the clear winner. Like TestComplete, UFT uses a visual model. But its model is more like a flowchart than a grid, with the possibility of loops and conditionals -- if statements -- visualized.
Telerik Test Studio
Teams using Microsoft Visual Studio to create web or desktop applications might be interested in Telerik Test Studio. This is especially true for the more technical testers who are not afraid of code, or teams that want to have the programmers contribute to the automated testing.
That's because Telerik's code is the same native programming language as Visual Studio. "Drop in" to the code in Telerik to extend it, and you can find yourself in the Visual Studio development environment, writing code in the exact same language as the programmers on the team. That means the programmers and testers can even share code libraries, which can be very popular. TestComplete's C# and VBScript modules are not true Visual Studio code and have minor syntax differences.
Teams that create in Linux or on Apple, for example, creating open source applications, will have problems with all the tools above, because they will need to have application testers using Windows. Because of the tight Visual Studio-Telerik integration, we do not recommend Telerik for teams on Linux and Mac OS X.
Teams writing code on Linux, Mac or in a mixed environment will want to look at eggPlant. EggPlant records the user interface, like the record-playback tools above, then creates a simple script. The script looks very much like BASIC code; testers can add simple if statements, variables and for loops. In our experience, most testers are able to pick up this kind of programming very easily -- although it is possible for testers to get out of their depth very easily, creating complex test code that is hard to maintain and debug.
TestStudio, UFT, eggPlant and TestComplete all have free trials. If the winner is not clear -- UFT for a Hewlett Packard Enterprise organization and eggPlant for Linux -- then explore, learn, pick and then build a proof of concept with the winner.
Stand-alone web and mobile automated testing tools
Teams that are comfortable with code and only need to test web applications through the UI are bound to look to Selenium. Selenium is a free, open source project for testing websites and web applications. Its most popular component is WebDriver, a code library to use in many programming languages that automates the web browser with commands like open, click, type and so on. Selenium provides a number of components that need to be installed to work effectively.
The on-ramp to Selenium WebDriver is the Selenium integrated development environment (IDE), which is an in-browser record-playback tool that can not only produce, but record-playback scripts and also convert them to WebDriver code in Java. As a record-playback tool, the IDE only runs in FireFox; WebDriver runs in Internet Explorer, FireFox, Chrome and Safari.
Commercial software support exists around Selenium. Sauce Labs provides virtual test environments to simulate any modern browser and the grid to run many application tests at once. Learn more about Sauce Labs by visiting their site.
Companies that develop mobile applications and mobile websites will want to look beyond Selenium to Appium, an open source mobile testing tool. Appium uses the same API as Selenium WebDriver, which means the application testing frameworks available for WebDriver also are available for Appium. Sauce Labs provides Appium support, as well.
Although it started out offering a virtual lab of real devices for users to test on, Perfecto Mobile has gone on to extend its features to support automated testing against that network of real devices, along with boot camps and training options. The company has a proprietary interface to design software tests easily -- claiming little to no coding required -- which also supports Selenium and other open source frameworks.
If your product will be used by a wide variety of mobile devices -- Android, iOS, tablet and phone -- and you need to support both older devices with smaller screen resolutions or newer devices with larger screen resolutions, you might want to look into what Perfecto Mobile has to offer.
Managing a library of test devices is one of the first problems makers of mobile software run into. If you are working with Apple only, there are four iPads, three iPhones and the watch. Those are just the devices running the latest version of iOS. Add older versions and then the entire Android ecosystem and you have a problem. Sauce Labs simplifies this problem by offering a large set of emulated or real mobile devices that are available as a service when you need them. This service works for hands-on and automated testing.
While enterprises are recognizing the need for mobile, few are recognizing the need for mobile application testing.
Getting out of your own way by implementing automated testing is a big step in releasing apps faster.
Here are the top three testing objectives to help direct your tests.
TestComplete Platform allows automated app testing