In this tip, we'll take a look at running your first load test with JMeter. JMeter is a popular open source Java application used for load testing. It supports a variety of technologies like HTTP/HTTPS, SOAP, JDBC, LDAP, JMS. These are all commonly used by teams looking to introduce load on a server, a network or even specific objects. While it's not as full featured as some commercial tools, it's a great tool with a lot of firepower for its price -- free.
We won't be looking at getting JMeter setup and running in this article. That's not because it's difficult - typically all I have to do to get it setup is download and double-click to fire it up. It's because operating systems and versions change, so the best place to get setup advice is on the "Getting Started" section of the JMeter user manual on the JMeter website.
This article was written using JMeter 2.3.4 running on Java 1.5.0_20. The application used in this article is Yahoo Search since it's a publically available application. The example uses a very low load to simply illustrate how JMeter works. If you would like to test with larger loads, do not use Yahoo. Instead, please use your own applications.
First some terminology
JMeter tests are built around the idea of a test plan. Within a test plan you can have Thread Groups, controllers, listeners, timers, assertions, and other elements. Each test plan is a performance test scenario - it's the series of steps JMeter will execute when you run the plan. The following table gives a brief overview of the basic elements you can include in a test plan.
|Thread Group||As with any load test, execution is multi-threaded. The thread group element is what controls concurrent connection to your application. In end user terms, it's your group of users.|
|Samplers||A sampler is the basic type of controller. Very simply, samplers tell JMeter to send requests (HTTP, SOAP, etc…) to a server. There are also Logic Controllers, but we won't be using one in this tip.|
|Listeners||Listeners are what you use to access to the information JMeter gathers while running. They give you the pretty charts|
|Timers||Timers are how you set delays in JMeter. Timers fire before each request that a thread makes.|
|Assertions||Assertions, like in any test tool, allow you to check for specific behavior as your test executes. Assertions provide the standard pass/fail results.|
For full details on these and other elements, please read the Elements of a Test Plan section of the JMeter user manual. However, this short list is enough for us to get a simple test running. If you have troubles with any particular element when trying to get it to work with your own application, the Component Reference section of the JMeter user manual is awesome for understanding each element and it's settings.
Building your first test
When you first open JMeter, you should be greeted with an empty Test Plan.
To add elements to a Test Plan, you can either right-click on it, or with it highlighted select Edit -> Add from the menu. The option of either right-clicking to add elements or using the menu to add elements will be true for all the items we talk about in this article. To reduce confusion, I'll be referencing object by asking you to right-click on them.
The first thing we need to get a test running is a Thread Group. To add a Thread Group, right-click on the Test Plan and select Add -> Thread Group.
For our example, we don't want to run with a lot of load. Yahoo won't like that. So we're only going to test with 10 threads - or users. In addition, we'll set the ramp-up period to 2 seconds, and we'll have each user search ten times. The only reason I'm having them loop ten times is so the charts you see at the end will actually contain some data in them.
Once we have our Thread Group configured, we're ready to add a timer. We'll be using a Constant Timer between transactions to space them apart. To add a timer, right-click on the Thread Group and select Add -> Timer -> Constant Timer.
Set the thread delay to three seconds - or 3000 milliseconds. At this point we're ready for our HTTP request. To add a request, right-click on the Thread Group again and this time select Add -> Sampler -> HTTP Request.
For the example, I want to have you run a simple search on Yahoo. To do that, set the Server Name to 'www.search.yahoo.com'. That tells JMeter what server you're hitting. Then for the path, I want you to enter '/search?p=testing'. That's the same as if you went to Yahoo's homepage and did a search on the word 'testing'. You can leave everything else on the page set to the defaults.
When I'm testing, I always want to know that what I did had the desired effect. So we're going to add an assertion to our test. To do that, right-click on the HTTP Request element and select Add -> Assertion -> Response Assertion.
There are a number of options here that might be confusing. We're not going to look at them in detail, but what we want to do is test to make sure we're getting search results back in our response. To do that, we're going to test the 'Main sample only' and we're going to be looking at the 'Text Response.' Looking at the text response means we're looking at the HTML coming back.
Because we want to make sure it has results, we're doing a 'Contains' pattern match on the response. At the time I wrote this article, the top response to the search was the Wikipedia entry for Software Testing. I simply searched for the starting text to that definition which was 'Software Testing is an empirical investigation'.
You'll notice that there's a bold HTML tag in the check. That's because that's what you'll see if you do a view source on the page. Always check for what you see in the source code - not what you see on the screen.
Next, we're going to add a couple of listeners so we can see our results. I like the Graph Results and View Results as Table listeners. They both offer intuitive interfaces. To add a Graph Results listener, right-click on the Thread Group and select Add -> Listener -> Graph Results.
For this example, you might want to uncheck the Deviation and Throughput checkboxes. Next, add the View Results in Table listener. Right-click on the Test Group and select Add -> Listener -> View Results in Table.
You now have a really simple test ready to go. Go ahead and save your Test Plan.
Running your test
Once you've saved your test, you're ready to run. From the menu, select Run -> Start. It's very possible you won't notice anything happen. Many first time users of JMeter don't. Here are some tips.
First, when you don't have a test running, you won't see any active users. In the upper right-hand corner there's a user activity indicator. If you're not running a test, it looks like this:
When the test is running, this indicator changes to show you how many of your users are active, like in the following figure:
Second, you can select your listeners to see what's happening during your tests. This information is updated real-time. This is also how you view your results after your test is complete. In the figure below, I've selected the Graph Results view after my test has executed:
You can see that in the Graph Results, it's charted each of the individual data points during the test (that's all those black dots scattered around) along with two lines showing the median and average response times. This is a hand chart to notice trends and odd behaviors.
Alternatively, the View Results in Table listener shown below is a more structured view of the data:
With this listener you can see stats for each data point, along with a graphic representation of the status of the response. If you wanted to export data, this is likely one of the listeners you'd use to save off your results to import into another tool like Excel.
In future tips, we'll look a bit more at some other ways you can use JMeter in your load testing. These tips will include some suggestions for debugging your tests, recording tests, and how to make sense of all the results. In the meantime, for more on JMeter, you can always checkout the project website at jakarta.apache.org.
If you're new to performance testing, I recommend the following resources:
- Beyond Performance Testing by Scott Barber: This 14-part series focuses on performance tuning and is structured in an easy-to-follow format with great references.
- User Experience, not Metrics by Scott Barber: This 13-part series focuses on understanding performance testing from an end-user perspective, and what that means for your approach to testing.
- Testing for Performance by Mike Kelly: This 3-part series focuses on the process of delivering a performance testing project - from problem assessment, to building out assets, to delivering results.