What are the benefits of automation in software testing? What are the drawbacks?
This is a question that, if you ask two knowledgeable people, you are likely to get three answers. Part of this is because “automation” covers a great deal of ground, all of which is dependent on the context they work in.
From my experience, the question revolves around what people hope to achieve from automation. Most are looking to reduce costs or speed the testing process or make testing more efficient or “better” in some way.
This is fine as far as it goes. The issue is that most people are looking to automation to fix all their testing ills. They are hoping that automation tools will address problems that manual testing practices have not been able to address. They are looking for a silver bullet to address their problems. The simple fact is, if you have problems with manual testing, automation will not fix them – it is likely to compound them.
In reality, automation can help you by executing mundane tasks, allowing good testers to examine the results carefully for anomalies that may inform their testing effort. Depending on your situation, it can also help you create scenarios that are not capable of being tested manually.
For example, you may send an invalid transaction to a system you are testing to make sure it is handled correctly. You may send three or four other invalid transactions. To send 300 or 400 transactions, invalid in a combination of ways, may be more than a manual test effort can absorb.
One area where I believe automation is an absolute, hands-down success is in a continuous integration environment. Allowing a series of basic processes to run, exercising sample functions of the application, can quickly find problems that would delay or prevent other, more directed tests. This can free up testers to focus on less mundane tests that can reveal interesting behavior about the system.
In this way, some basic automation exercising happy-path functions can allow more detail-oriented work to be done by the testers to investigate aspects of the application they may not be able to otherwise. I believe this is the greatest benefit of test automation – not reducing testing, but allowing more productive, result-oriented work.
This was first published in March 2012