Wouldn’t it be great if programming languages were as easy to understand as English? Well, we have almost gotten to that point. Using a common language that both provides test code and documentation, we have “executable specifications.” In this interview, SSQ contributor Chris McMahon talks to Dawn Cannan about her upcoming half-day presentation at STAREAST on May 3, “Creating Executable Specifications and Tests with FitNesse and S...
SSQ: What are "executable specifications," and why are they important? Can you give some examples?
Dawn Cannan: For me, the concept behind "executable specifications" involves having a single place for descriptions of what a product should do, and those descriptions should be self-verifying. That is, when using executable specifications of any type, all of the team members are reading the same exact sentences to understand the desired behavior, and those sentences are also tests that can be run and verify that the behavior exists as expected. With a single place for this type of documentation, the whole team develops a common language, and translation errors and misunderstandings are minimized.
In a system using executable specifications, a specification might look something like this:
As an administrative user,
I want to be able to edit the first and last name of a standard user
so that I can handle the fact that people's names change
Log in as an administrative user
Edit the last name of a standard user
Verify that the changes are saved
Ensure that the first and last name fields follow the existing field validation rules
Please note that the above example is just an example -- it includes a user story, a verification test, and some scoping information. In this example, the user story itself would likely not be executable, but provides the context for the rest of the sentences which also run as tests.
SSQ: Tell us how executable specifications in FitNesse are hooked up with Selenium, the browser driver tool.
Cannan: FitNesse provides a simple, easy to learn, plain text wiki for creating tests and associated descriptive information. The Slim engine allows for writing test steps in the wiki in any way you desire; scenario tables will take your natural language sentence and translate it into more technical steps. This is a fantastic way of including business people in the definition of a behavior and scoping out what it should and should not do. However, FitNesse is not and does not do browser driven automated UI testing. The Selenesse project that I'll be demonstrating at STAREast gives FitNesse access to the Selenium library, so that Selenium tests can be written in FitNesse, and then FitNesse calls out to Selenium and reports back the results. FitNesse, in this case, is simply calling out to Selenium the same way it would call out to any other library.
In this way, both below-the-UI and end-to-end UI automated tests can be created, executed and reported on in a single place. Managing these layers of testing in a single place removes some complexity in systems for teams.
SSQ: FitNesse and Selenium both offer users a lot of flexibility when creating tests. Do you have any test design tips to make the most of that flexibility?
Cannan: This combination is incredibly flexible, for sure. Perhaps for the light of heart, too flexible. With the ability to write sentences in absolutely any way that works for you, your team's tests can represent whatever way your team naturally organizes its thinking. For some teams, they write a user story and maybe some textual description to provide context, followed by some acceptance criteria, and then a set of specific functional tests.
The best tip I can offer is to create tests in FitNesse in a way that mirrors your current process. However you normally organize tests, organize them the same way in the FitNesse framework. However you normally write requirements, use those same language patterns (which may evolve over time).
In general, I would suggest keeping all of the information and tests around a specific behavior together. That is, I would not suggest having hierarchies in FitNesse that start with categories like "Selenium tests" and "Business logic tests," but rather for example, "User management tests" and "Feature X tests" and "Feature Y tests."
SSQ: It is possible to create really large numbers of tests for an application of reasonable size. How do you use these tools to keep your test suites in good order and organized?
Cannan: Oh, boy. Large numbers of tests are generally difficult no matter what your framework or stack is. In general, I try to push tests down as far as is reasonable with the developers on my team. Anything that can be pushed down into unit tests or integration tests, should be. This will help to minimize the number of tests that will end up in the FitNesse or Selenium layer.
For the tests that are still managed through FitNesse, utilizing the hierarchical page structure will help to organize tests, though too many levels of hierarchy will eventually be just as confusing. FitNesse also allows you to enter tags for your test pages. Using tags, you can create categories that can then be grouped in any combination you need for testing. For example, you can run just the tests that are tagged as "administration" or "smoke," or a combination of tags. Using tags will help to make the hierarchical organization less important.
Dawn Cannan is the author of the blog “Confessions of a Passionate Tester.” She speaks and writes on software testing and development, test automation and Agile software development. SSQ interviewed Dawn ahead of her appearance at the STAREast conference in which she will be presenting on May 3rd.