A question I often receive is, "Which tools are best for agile testing?" This is almost like trying to answer the question, "Which tool is best for working on a car?" There are many things to consider, such as:
- Who is performing the tests?
How much funding is available for the tools?
- Which kinds of testing are to be performed?
Let's explore what can be learned by answering these questions, then I'll present some tools for your consideration.
Who is performing the tests?
This is important to consider because of the learning curve that may be involved, and the type of tool that may be most applicable. For example, agile testing is often characterized by early testing with close interaction between software developers and testers. At this level of testing, testers may be using tools such as the X-unit family of tools to be testing at the code level.
However, if the testers are envisioned to be at a functional level where business rules are being tested, then tools such as Fitnesse are commonly used to define functional tests.
If the testers have technical skills, the X-unit tools may be fairly easily learned and applied. Otherwise, the more functional tools like Fitnesse may be the better choice to learn and use.
How much funding is available for the tools?
Software test tools can be a major investment, which adds an element of risk. Some people simply do not have thousands of dollars to invest in tools. Plus, test tools are not a single-tool solution. You need tools for test execution, test management, defect tracking, and specialty areas such as performance testing, security testing and test environment control. There are also maintenance costs to consider as time goes on. So, tool costs can add up quickly.
Interestingly, agile practitioners have been able to build or adapt some free and low-cost solutions for software test automation as well as other testing tasks such as defect tracking.
On the other hand, when commercial tools such as HP's Quick Test Pro are available, they can also be applied in agile projects for functional testing. Just keep in mind that it takes time and practice to use test automation tools effectively. Many times, people give up due to frustrations in actually applying the tools. This may be due to issues with the application being tested, the time available to automate tests and learn the tools.
Which kinds of testing are to be performed?
For unit testing at the code level, tools that allow you trace execution of the code and build stubs are helpful. For unit testing at the functional level, the same tools can be applied as would be used in system or user acceptance testing.
For structural testing, the main goal is to test the stability, efficiency and integrity of the code. In other words, you are trying to determine if the code is behaving correctly as written. It is very important to understand that structural testing is essential, but only takes you so far in evaluating software quality. It is functional testing that validates the correctness of the business rules implemented in the code.
As an example, consider this IF statement:
IF X > current_date
Move "5" to available_status_code
Move "0" to available_status_code;
A good structural test would be to test X greater than, equal to, and less than the value held in current_date. Using a tool that would allow viewing the value held in current_date, we could determine if "5" and "0" are stored in the right conditions. However, this test does nothing to determine if "5" and "0" are the right values to store. Likewise, perhaps the business rule is, "If the order date (X) is greater than or equal to the current date then move..." Once again, all the structural test tells us is whether or not the IF statement is working as coded. It is functional testing based on requirements, business rules or some other source that determines if the implementation of the rule is correct.
In agile testing, Test-driven Development (TDD) is a very important practice for developers. In this practice, instead of writing the code first and then testing it, the tests are designed first and the code is considered correct once those tests pass. If TDD were used in the previous example, the developer would work with the business analyst or user to understand the correct business rules and design tests. Then, the developer would write the code and run the tests until the code works as defined by the tests.
The X-unit family of tools is open source. Other tools such as C++ Test, Jtest and AgitatorOne are commercial. I have been impressed by how both the free and commercial tools have found a home with many developers. The advantage of the more robust commercial tools is that they can create the structural tests, scan code for good practice violations and provide code coverage metrics. A general rule with test tools is that it takes time to learn and implement them. However, structural unit test tools such as these commercial products are fairly quickly learned and applied.
What is the envisioned roles for tools?
By answering this question, you are addressing the purpose of tool usage and the effort needed to implement the tools. The purpose can include things such as structural and functional test automation, defect tracking, structural test coverage measurement, test management and test design. The specific tool approach needed will depend on the envisioned purposes. For example, test automation is much more complex to get the tool in place and create the automated tests than other tools such as defect tracking.
Another example is the use of test design tools. One of the techniques used by some agile testers is pairwise testing, in which tests are optimized to test all pairs of conditions. There are free and commercial tools to handle this task, a few of which are shown below in the tools listing.
In learning and applying test design tools for pairwise testing, the tool adds power but it takes time to learn the technique and how best to apply it. Not every situation lends itself to pairwise testing, so you must understand which conditions to input in the tool. Also, you must understand the application you are testing and what constitutes correct behavior. All of this needs to occur before you ever start using the tool. When you focus on using the tool before learning the concepts about the functions to be performed, the result is often performing an inadequate test faster.
Tools often used in agile testing
The following tools are those I see used often in agile projects, but they can be used in other test and development approaches as well. There are many tools to consider, but these are some that I see used most often in agile projects.
Fitnesse is a popular wiki-based tool which is free at www.fitnesse.org. This is a functional test tool which is often used for agile acceptance testing. In Fitnesse, you can create tables of test inputs and expected outputs. One of the nice features of Fitnesse is that you can copy from a spreadsheet and paste it in Fitnesse to create a table.
Selenium IDE is a free Firefox add-in which allows the recording of Web-based tests. The nice thing about being an integrated IDE is that you can record, edit and debug tests. While you can use the capture/playback automation approach with Selenium IDE, you can also write tests from scratch if that is your desired approach. Selenium IDE is part of a family of test tools, such as Selenium Remote Control which can be downloaded at www.seleniumhq.org/projects.
Watir is an open source tool browser-based test tool that allows functional web testing. Watir is implemented in Ruby but you can test applications based in other technologies. Watir also has the ability to read from databases and files. This allows data-driven testing.
X-Unit is a very popular family of unit test tools which has been created for a variety of languages, including Java, php, Python, and Visual Basic. These tools have become the unit testing tool of choice for many agile developers and testers.
Bromine is a robust test management and execution tool which allows you to create test projects, enter requirements, document test cases and drive tests created in Selenium IDE. You can execute the tests on a variety of platforms and browser/OS configurations. The results can then be viewed by management.
You must have Adobe Flash Player 7 or above to view this content.See https://www.adobe.com/products/flashplayer to download now.
Download for later:
• Internet Explorer: Right Click > Save Target As
• Firefox: Right Click > Save Link As
Testopia integrates with Mozilla's Bugzilla defect management and tracking tool. If you are using Bugzilla for defect reporting and tracking, it is helpful to have a test management tool such as Testopia that is designed to compatible and a central point of test case management.
Bugzilla is a free solution that has been around for some time and continues to be a popular option for tracking defects. Although Bugzilla is free, it has many features found in commercial defect tracking tools.
Bugaware is a hosted bug tracking product that is Web-based and free for up to five named users. It is also free for not-for-profit and academic organizations. It has a clean, easy-to-use interface, is customizable, and if you do not have the ability to host your own defect tracking system, this is an option. The paid version for enterprise use is affordable at $2,160 (hosted annually) or $2,375 for locally installed with 12-month maintenance.
CTE-XL is a free tool to identify test conditions and define test cases using the classification tree method. Since agile projects proceed very quickly, testers need to get the greatest test coverage with the most efficient number of tests. CTE-XL not only generates all possible combinations of tests, it also generates pairwise and triple-wise tests. Unlike many of the other free pairwise tools, CTE-XL has a clean user interface and can also apply user-defined dependency rules.
Apache Jmeter continues to be a popular option on agile projects for performance testing. Jmeter is implemented in Java and is used to simulate heavy load on a Web server, network, webpage, or application to determine if it can handle required load levels when deployed.
Agile testers must be able to keep up with the pace of constant changes – not only to the code being written, but to the user requirements. Along with this challenge comes the need for tools to provide speed and accuracy in testing. Regardless of the amount of money available for tools, there are options for agile testers to build their own toolkit. Hopefully, some of the tools presented in this article will be a good start in obtaining and using tools on your agile projects.
About the author: Randy Rice is a leading author, speaker and consultant in the field of software testing and software quality. He has over 30 years experience building and testing mission-critical projects in a variety of environments and has authored over 60 training courses in software testing and software engineering. He is a popular speaker at international conferences on software testing and is also publisher of The Software Quality Advisor newsletter. He is co-author of the book, Surviving the Top Ten Challenges of Software Testing, published by Dorset House Publishing Co.
You can contact Randy from his website at www.riceconsulting.com.