Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Should you purchase an automated testing tool or build your own solution?

Many organizations are unsure whether to purchase automated testing tools, use free tools available online or build their own test suite. Expert John Overbaugh weighs in on this common software testing decision.

When it comes to automation, is it better to buy a tool or build your own?
The build-vs-buy argument isn't just for applications. Test organizations struggle with this question all the time, especially when it comes to test automation. There are a number of automation solutions available commercially, of which promise to solve your automation problems. For a price. But when it comes to automating anything but the easiest, static web application, I'm a firm believer in building.

I prefer building internally when I'm dealing with a web application which is highly complex (requires a lot of very technical data coming in) or not very static (the UI is constantly changing, new features are being added, etc.). I work in the health care industry, and the applications we build handle vast amounts of variable data. No commercially available products offer an off-the shelf solution for submitting this data, so regardless of the solution, I'd have to build an interface to submit data to the application during automated testing. Due to the complex nature of what our applications can do, commercial products also don't provide a robust way to automate testing. Finally, in an agile organization producing software in an incredibly competitive and now fast-changing industry, our application UI is changing frequently. Commercial automation solutions, especially those offering click-and-record capabilities, do not allow me to write thousands of automated tests, yet tweak them in one place. More on that in a minute.

When I design an automation solution, I build it in layers. The first layer of my solution is the test harness – this layer queues up tests, executes them, and records results. I prefer to use open-source unit testing frameworks at this layer—they're very flexible, they're cheap, and they're purpose-built for testing. They're not just for unit testing! NUnit and JUnit are my favorite open-source solutions; if I'm in a .NET environment and running Visual Studio, I really like Microsoft's MSTest tool.

The next layer is what I call the framework. This layer is an abstraction layer – it allows me to abstract my tests from the application with which the tests interact. This is hugely important, and the key differentiator between purpose-built solutions and most off the shelf commercial solutions. Components in this layer include tools like Watij or WebAII, for driving browsers, and custom tools for stubbing out executables. I'll also include classes in this layer which abstract interaction with features – in the case of a web application, my classes will abstract and expose each page in the application. I will provide references to page elements such as controls, and I'll also include a number of 'utilities' to interact with the page. For example, a login page class will expose the user name textbox, password textbox, and login button on the page. It'll also include a quick "DoLogin" function for times when the login page is simply a step in a test, and the test just needs a one-line function to log in.

Commercial applications do have their place. I find, for instance, that Visual Studio's new click-and-record tools give me a quick approach to automated BVTs on new feature sets. While my automation engineers are busy writing the long-term automation, my QA analysts can click-and-record their way to an inexpensive automated build verification test.

Dig Deeper on Topics Archive

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

Hi, do you have any tutorials on creating an actual tool? I am looking to create one for myself and any future organizations.
It depends on what you need. Licensing, capabilities, intended purpose, availability of knowledge regarding tools, etc., are all considerations. We were comfortable, for many years, using HP’s QTP/UFT and it met all of our needs. More recently, however, we’ve started moving more towards a fourth-generation automation framework that makes use of model-based testing, which required us to create our own framework.
My job was using QTP, then we moved into an in house application that a developer there created. It was basically the same as QTP but alot faster. Now I am at a new Organization and would like for us to create our own in the same way he did before. I would like help getting that much started, if needed I can explain exactly what it is I need it to do?