I prefer building internally when I'm dealing with a web application which is highly complex (requires a lot of very technical data coming in) or not very static (the UI is constantly changing, new features are being added, etc.). I work in the health care industry, and the applications we build handle vast amounts of variable data. No commercially available products offer an off-the shelf solution for submitting this data, so regardless of the solution, I'd have to build an interface to submit data to the application during automated testing. Due to the complex nature of what our applications can do, commercial products also don't provide a robust way to automate testing. Finally, in an agile organization producing software in an incredibly competitive and now fast-changing industry, our application UI is changing frequently. Commercial automation solutions, especially those offering click-and-record capabilities, do not allow me to write thousands of automated tests, yet tweak them in one place. More on that in a minute.
When I design an automation solution, I build it in layers. The first layer of my solution is the test harness – this layer queues up tests, executes them, and records results. I prefer to use open-source unit testing frameworks at this layer—they're very flexible, they're cheap, and they're purpose-built for testing. They're not just for unit testing! NUnit and JUnit are my favorite open-source solutions; if I'm in a .NET environment and running Visual Studio, I really like Microsoft's MSTest tool.
The next layer is what I call the framework. This layer is an abstraction layer – it allows me to abstract my tests from the application with which the tests interact. This is hugely important, and the key differentiator between purpose-built solutions and most off the shelf commercial solutions. Components in this layer include tools like Watij or WebAII, for driving browsers, and custom tools for stubbing out executables. I'll also include classes in this layer which abstract interaction with features – in the case of a web application, my classes will abstract and expose each page in the application. I will provide references to page elements such as controls, and I'll also include a number of 'utilities' to interact with the page. For example, a login page class will expose the user name textbox, password textbox, and login button on the page. It'll also include a quick "DoLogin" function for times when the login page is simply a step in a test, and the test just needs a one-line function to log in.
Commercial applications do have their place. I find, for instance, that Visual Studio's new click-and-record tools give me a quick approach to automated BVTs on new feature sets. While my automation engineers are busy writing the long-term automation, my QA analysts can click-and-record their way to an inexpensive automated build verification test.
Dig Deeper on Topics Archive
Related Q&A from John Overbaugh
Learn what's behind AWS outages and how to fix failures before they happen. Continue Reading
Learn strategies for best security test strategies for SaaS cloud. Continue Reading
Expert John Overbaugh identifies the three top concerns of the test manager and offers advice on how to stay ahead of the curve when it comes to ... Continue Reading