Manage Learn to apply best practices and optimize your operations.

Running user interface, unit and integration tests suites

An expert tester explains differences between User Interface and integration tests. Read these test demonstrations to learn how to utilize each test type of testing in this tip.

The problem of long running tests

User Interface, (UI) tests and certain kinds of integration tests like SOAP or REST API, are different than unit tests in many ways. For one thing, they take a long time to run. For another thing, they test not only the code, but also the whole environment in which the tests run. Also, they are not coupled to the code itself. We can be working in the test suite without regard for what is happening in the code base; and we can work in the code base without regard for the state of the test suite; and we can run such tests any time, as long as the underlying application has passed all its unit tests and is known to be in good shape.

The best approach I know to running a test suite that takes a long time to run is to do so inside an infinite loop: check out the latest tests; run them; get the test results and publish them; and then do the cycle over again, 24 hours a day, 7 days a week.

The tricky part is that last step: how can we publish the test results so that everyone can see the current state of the test suite?

Some might be tempted to have the test-runner script send an email or create a document with the test results, but there are better solutions.

Using information radiators

A good practice for development teams is to have information radiators, communication channels that tell everyone the current state of the work, like a red light that shows a broken build or index cards on the wall to show what the team is working on. A lot of teams using information radiators move away from low-bandwidth communication channels like email and documents and replace them with higher-bandwidth channels like blogging and wikis.

I like to publish high-level test results on a wiki, because every reasonable wiki has a 'What's New' feature, which makes it easy to see the current state of the test suite.

An example is always handy

This example is going to use Ruby and Watir to drive SVN and FitNesse, and to report test results to a Confluence wiki.

First we need a command line SVN client. I'm working on Windows, so I chose SlikSVN. Alternatively, you could install the SVN bindings for your particular language, but SlikSVN seems easier. This script assumes that you already have an SVN repo in place, which just needs SVN UPDATE for each run of the test suite.

Then we'll use Watir to bring up FitNesse, launch the suite of tests, and collect the test status once the run is finished.

After that we need to use Ruby's text-handling ability to munge our results message. The Confluence API won't accept the newline character (\n), so we have to replace each newline with two backslashes "\\". We also use Ruby's Time class to get the time that the run ended, and then we paste the whole message together with a little string interpolation.

system() is a very handy feature of Ruby and any number of other languages. Using system() we can launch an outside process in its own shell, let it run, and capture any output from the process. We use system() here to launch the SVN client for the SVN UPDATE, and then to launch the Confluence Command Line Interface. Note that we do not have access to any PATH information when using system(), so we have to specify the full path to the application to be run.

Finally, we wrap the whole thing in an infinite loop (while 1), so it runs over and over and over.

  require 'watir'
  while 1
  svn = system("C:\\Program Files\\SlikSvn\\bin\\svn.exe update C:\\Path\\To\\Repo""") 
  b = Watir::Browser.start("
  "), "Suite").click
  status = b.div(:id, "test-summary")
  msg = status.text
  msg = msg.gsub(/\n/, '\\')
  endtime =
  msg = "Run ended at #{endtime} with this status: #{msg}"
  x = system("C:\\Users\\ChrisMc\\Desktop\\confluence-cli-1.5.0-distribution\\confluence-cli-1.5.0\\confluence.bat
 --server --user Username --password Password --action storePage --title TestReport --space Test --content \"#{msg}\"")
  end #while


When this script runs, it yields on a wiki page entitled "TestReport" a message like "Run ended at Wed Oct 14 15:10:05 -0400 2009 with this status: Test Pages 45 right, 15 wrong, 0 ignored, 0 exceptions. Assertions: 1443 right 9 wrong 345 ignored 0 exceptions." And it replaces the message in the same page with a new message every time around the loop, so we always have the latest information. Furthermore, page history is available in the wiki, so we could even find trends over time if we wanted.

Now anytime we need to know the current status of our long-running suite of tests, we can just look in the wiki for that page to find out when it ran last and how many tests passed and failed.

What do you call that?

This sort of work goes by a number of different names. In one sense it is a "mashup", in that it uses public APIs to take data from one application and insert it in another application. But "mashup" may not be the best fit; usually a mashup applies to consumer software, for example, if you use Google Maps' API to make a map showing where all your relatives live.

A more appropriate term is enterprise application integration, or EAI. EAI is when various applications run by the business and connect in ways that accomplish some sort of business function, in this case the ability to tell at a glance the status of a test suite. Other EAI operations might be to send messages when certain records appear in a database, or to start or stop applications under certain circumstances, or any number of other operations.

There is an argument that EAI, particularly what is called "point to point" EAI, is unmanageable in the long run because of the number of unique kinds of connections between applications, all of which require maintenance. When EAI gets to that scale, many people call it SOA, for service-oriented architecture. SOA systems require that all business applications that interact with each other use a centralized messaging system and obey a uniform procedure for producing and receiving messages. While SOA systems can be valuable when there are a very large number of disparate applications to be integrated, for most day-to-day integration projects, SOA is too expensive to be worthwhile.

Because all we really want for ourselves most of the time is a little information radiator. And here in 12 lines of Ruby we can make it so we have up-to-the-minute status on a long-running test suite available 24/7/365. That's a good investment.

Dig Deeper on Topics Archive

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.