Ten quick attacks for Web-based software

An expert tester offers his insight on common Web-based application flaws and intuitive ways to handle their repair. In this tip learn how to identify weaknesses before going live.

As a "Boutique Tester," my job is to jump in and add value in a chaotic environment. One tool for the Boutique Tester is the quick attack. What follows are techniques that anyone can use to attack Web-based software, with or without up-front time and planning.

1. Overwhelm the software with invalid data
If the system expects letters, type in numbers. If it expects numbers, try fractions, or negative numbers, or letters, or words like "one", "two" or "three." In either case you can try special characters like [email protected]#$%^&*()_+={}[]<>,.?/~`. On a web application, you can try to enter HTML elements to see if they show up as elements for example HR/, BR/, or  , and other HTML elements. For phone numbers, try words, or phone numbers that are extremely long.

The point of all of these is to feed the system input it doesn't expect, and see how it handles the unexpected. If the system does well, it means the developers are doing a good job of considering odd combinations, which bodes well for general, happy-path use. If it doesn't, well, we've got some bugs to file, which will give us some time to dig into the software. Our second quick attack is similar to the first

2. Overwhelm the software by pushing beyond boundaries
This attack is similar to invalid data, but instead of being just plain wrong, the data we'll send in will be technically correct, just bigger than the developers expected. We can then look for errors two ways: Processing errors, where the program itself chokes on the input, or rendering errors, where the output is re-displayed, but too large for the space available. For example, if asked for geometric degrees for a calculation, we'll type in 360, or 480, or 620, or 62000 -- or just keep typing nines until we run out of space. We can do the same thing with words. One particularly crafty trick is to type in a paragraph with no spaces and click save; it's very common for rendering errors to occur if the software can't generate any line breaks. I like to use the free program perlclip to generate these long text strings. Perlclip has a function called counterstring; counterstring embeds the lengths of the string in the string itself. Using perlclip, it's relatively easy to figure out exactly what the real limits of the application are.

3. Perform server-process interrupts
When I click the submit button on a form, the browser sends a signal to the server and I wait for the form to process. This can take a long time; a really long time. If the application gives me no indication that it's processing, I may click again. This can result in two books in my shopping cart, or, maybe, two books shipped to my house and double credit-card bills.

Since clicking twice "to make sure it took" is so common, one thing we can do to test is to force keyboard interrupts - to get the server to think two different things at the same time. Beyond clicking twice, if the app is not supposed to allow re-submits, we could click-back and resubmit, or pick a different option. We can also do things the software doesn't expect, like use the browser back->next buttons instead of the expected set of links in a page.

Also look for the hidden use cases. For example, it's pretty common to need to type a password twice when creating an account. What happens if I type a different password the second time and then click save? Do I re-type all my information? (I did on MacHeist.com while writing this essay!) If there is javascript or flash that generates pop-ups, try switching between tabs quickly to see if the images repaint. Here's that registration screen from MacHeist again, under FireFox 3 for the Macintosh after switching tabs.

We can even move beyond this behavior by doing interrupts in space in time, our next item.

4. Try unexpected combination of users and race conditions
Imagine sally creates a page that has a picture of her drinking at a party. Now Joe logs in, sees the picture, and leaves the comment "U R busted if the dorm RA sees this." Now, between the time that Joe saw the picture and clicks submit, Sally deletes the picture. What will Joe see on his screen?

Unless the programmers are aware of this condition, Joe might see a bad error message, poorly rendered HTML, or just maybe some sort of security hole.

Testing for race conditions is as trivial as logging in on two different browsers as two different users and simulating the use; sometimes you can just do the tests in different tabs in the same browser.

5. Try the software from outside the firewall - from a public terminal, in secure mode
If your application is designed to be encrypted, it should support https. Likewise, if it has any eye candy, it might be slow, or it might serve the graphics up without https, which generally pops up a browser warning. Try testing the app on old hardware, on slow connections, from a public environment, and see what you can find. I've worked on "public" applications that looked just fine behind the firewall, but when we got out from the outside we found they were linking to internal "blocked" sites for inner frames and graphics. Whoopsie.

6. Have the team bang on a slow-running operation simultaneously. Use JMeter or LoadStorm if you must.
The line between functional testing and performance testing can get pretty blurry; I once had to tell a project manager "you can't call it 'just; a performance issue if it doesn't work at all for one user." Yet performance testing can be painful and expensive. A quick lunchtime bug bash session can do more than find bugs; it can demonstrate race conditions and performance problems beyond what one person can find alone.

Invite the team to bash on the software, and spring for the pizza yourself. It'll be worth the twenty bucks.

7. Try different browsers - Safari, Opera, FireFox, Internet Explorer 6
One of my favorite tricks to find HTML and javscript issues is to find out what browsers the developers are using, then find out what browsers the customer will use, and exploit the difference.

8. Look at the business rules the software operates on.
By this point, we've found the cheap and easy bugs any "Boutique-er" can find without context. Now we'll go deeper. One way to do this is to take the requirements and, if you can, express them as a table. Now look for holes in the table, behavior that is undefined. For every rule, look at the suggested boundaries, and test just before, at, and after each (perceived) boundary. Also look for data boundaries. Ask how many bits the integers are, and enter values just over or under. (An 8-bit integer can handle up to 255, a 16-bit, thirty-two thousand, seven hundred and sixty seven.) Look for differences between the data types stored in memory and the way the data is saved to disk or database. Exploit those differences.

Another tactic is to look for mutually exclusive constraints in the requirements, and see if the user interface allows you a way to configure both of them at the same time. If it does, try it, and see what happens.

9. Internationalization
It's likely that if you offer your site to anyone on the internet, you'll get people from outside the United States, who might have non-traditional Arabic names. Take a look at "Markus Gärtner" for example -- see the dots over the a? That's call an umlaut, and it can cause traditional "ASCII" strings to choke. We call this choking, Mojibake. On windows computers, you can just start->run charmap and see the charmap program, which gives you a whole host of non-standard characters to select, copy, and paste into your application. (Here's a German set just for fun: ö,ä,ü,Ö,Ä,Ü and ß)

Beyond the extended ASCII chart there are also true international characters. Ideally, your application would have a list of keywords and translations into other languages. We could cover this in a future tip, but the key things to ask are (A) Has internationalization been done at all? (B) What languages are supported? (C) Are there any rendering errors that appear because some words are longer than others when translated? and (D) Do we see mojibake when we input, save and display these characters?

Now it is possible you are working on an internal, time tracking application to be used only in New Jersey, so this advice might not hold. A big part of the challenge of the boutique tester is not only finding bugs, but finding out which avenues of attack will be most valuable for the customer.

10. Common defects
My personal favorite: Just pull out whatever bugs the developers have commonly encountered before and look for them. Of course, in team retrospectives, you can pull these problems out and try to prevent them in the future. Do this well enough, and you'll encounter the old pesticide paradox: The old bug-finding tricks stop working as the developers learn to prevent common errors.

That's good; it means the team is getting stronger. It also means it's time to crank up our bug hunting skills a notch or two.

Where to go for more

James Whittaker's book "How To Break Web Software" provides a much more in-depth look at quick attacks; Elizabeth Hendrickson's Test Heuristic Cheat Sheet provides a much more compressed list, and Cem Kaner's Black Box Software Testing Video on c provides perhaps the best tutorial format. Note that the quick attacks video is only part of a greater, comprehensive open-source course on software testing.

One more place to look is into bug taxonomies; ordered hierarcal lists of what could go wrong. A good public example of this is by Also Giri Vijayaraghavan and Cem Kaner "Bug taxonomies: Use them to generate better tests. Vijaraghavan and Kaner presented the material at the STAREast conference in Orlando, Florida in 2003.

Now get out there and find some bugs!

About the author: Currently a technical staff member at Socialtext, Matt Heusser has been developing, testing or managing software projects for over a decade. He teaches information systems courses at Calvin College and is the original lead organizer of the Great Lakes Software Excellence Conference, now in it's fourth year. You can read more of Matt's writings on his blog,Edge of Chaos.

Dig Deeper on Topics Archive