Q
Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Know when to choose automated vs. manual testing

Vendors have inched toward automated application testing for a long time, yet there is still room for growth. Software tester Amy Reichert offers her insights.

Automated application testing has been at the forefront of the testing discussion for decades. However, manual testing is enjoying a renaissance of sorts as it becomes more credible and valued, due to the expertise and nuance QA engineers bring to the table -- especially when it comes to exploratory tests and edge cases.

Several questions remain for all software development businesses: When should they use automated vs. manual testing? Which type of testing provides the most business value? Does automated testing provide more valid, reliable testing results? Does manual testing irreplaceably represent how human customers work with software applications?

The choice of automated vs. manual testing comes down to their distinctive strengths: verification vs. validation. Let's explore those criteria to determine which is a better fit for your application testing requirements.

Manual testing: Better for validation

With manual testing, QA engineers both verify and validate software code with a variety of techniques. Validation is a unique strength of manual testing, a human judgement call to assess whether the application performs functions promised to the customer. Validation is less about logic and calculation, and more about if the application meets customer expectations during use.

To validate applications, you can perform internal manual QA testing, or augment it with temporary testing services. Internal QA testing provides an advantage in that testers understand the application's functions and use. However, testing services provide insight on how well an application delivers on various users' needs. For example, crowd-sourced testing services provide a group of global users with different types of devices. Consider a mix of both internal and testing services; or select one that best represents your application's customer base. You can always decide to add either, if a single approach doesn't provide enough testing coverage to keep your application users happy and productive.

Automated testing: Better for verification

Automated test suites provide exceptional verification testing for application code. Verification-type test execution is more consistent and reliable, because automated test suites repeat the exact same actions with each execution and verify the exact same workflow. Each logical calculation, when scripted with appropriate verification points, is tested in exactly the same way every time.

For verification-type testing, automated testing provides more reliable, consistent test coverage. For automated tests to provide verification test coverage, however, they must be coded with enough depth and complexity to truly test how the application functions under use.

Historically, organizations' poor planning and unrealistic expectations led to frequent failures with automated testing. It takes time to ensure that an automated test tool handles simple and complex functions in application code. Take time to research automated testing tools to determine which will be most effective for your application. Take advantage of free trial periods to help make a purchasing decision.

After you select a tool, plan the testing approach you'll use. Which types of tests will you automate first? Start with critical functions, and automate critical smoke tests first. This adds instant and significant value, because the smoke tests will run automatically while you refine the tool and remaining scripts. As you expand the depth of coverage on your critical functional test suite, add the complexity that your application offers and coverage for end-to-end workflows. If you can build out these types of tests first, you'll improve quality for the most valuable and critical functions of the application.

When to use automated vs. manual testing

Both manual and automated testing have advantages. Automated testing provides better verification test coverage when your test development strategy has specific, realistic functional goals. Manual tests are better at covering validation in all its typical, non-logical, end-user glory. If you invest the time in proper test automation development, you get more reliable, consistent test coverage time after time with no human-induced variation.

Ideally, a combination of manual testing and automated testing gives you the best value. If you must choose one or the other, determine whether validation or verification is more crucial to an application's success. Decide if you need human testing to gauge how an application performs, or if pure verification of logic, calculations and coding is enough to ensure an application delivers the intended functionality for your customer base.

Next Steps

Automated dinosaurs? Check. Automated testing? It depends.

Finding the best automated testing tools for your business

How autonomous software testing could change QA

Dig Deeper on Automated and autonomous testing

Join the conversation

12 comments

Send me notifications when other members comment.

Please create a username to comment.

Should automated testing eliminate manual testing? Why?
Cancel
That's an interesting question. I can offer a few other.
  • Can automated management eliminate manual management?
  • Can automated recruitment eliminate manual recruitment?
  • Can automated parenting eliminate manual parenting?
  • Can automated kissing eliminate...

Well, you got it. It's not about "manual" way. It's about human way.

For further amusement I suggest the following article: http://www.satisfice.com/articles/cdt-automation.pdf


Cancel
This question feels like a misunderstanding of definitions.

What is manual testing?

What is automated?

Programmers used to use machines to do program for them automatically on chips, later the got disk drives and RAM, and then had compilers to convert higher level languages to lower language which machines understand.

The answer is no, because all testing is automated to a degree when it involves a computer.   And unless you are testing dead tree manuals (which I doubt) then you probably aren't doing manual testing at all.
Cancel
Very valid question in the current DevOps scenario.
Automation is a buzz word used in testing to attract developers into testing team. Assume if no manual testing and the code quality is bad, the tool cannot be blamed. This has to be addressed in manual testing phase to decide whether to automate or not. Also, proper feasibility analysis will help cost saving on tools and effort. 
Cancel
It’s not really a question of manual vs. automated. It’s more a question of to what extent is your testing is automated?
Cancel
Yes, manual testing could not be replaced by the automation testing. The automation testing shall rerun the scripted test n-number of time. But in manual testing we tend to explore the application based on the current change and identify more scenario and unveils defects. More over automation could be done only on fairly stable application other wise effort would be spent on making the code work.
Cancel
You need both. Automated can test the bulk of the conditions and parameters set much quicker. Manual is going to catch more of the human error situations that the automated process may not have been looking for. When we assume certain things when designing the automated process it the obviously wrong exceptions that will make it fail.
Cancel
I agree, automated testing can't replace manual testing, but it can complement it. And it can be better in some ways. For instance, performance testing and load testing is definitely best automated, if possible. In a lot of cases, it's too difficult for a tester to manually create the volume of data/connections/transactions (whatever the case may be) to accurately & efficiently test those aspects.
Cancel
@Abby - but it's impossible for automation to design workload models and to analyze execution results. From this perspective, it's not automated testing but automation-assisted or tool-assisted testing.
And we use tools all the time. In general, all testing - and all programming - is tool assisted.
Cancel
I do not think it's better, just faster. There is still to much human error that automation does not take into effect to solely rely on just one method.
Cancel
Automated to what degree? Let's face it we don't live in an age of Ghost in the Shell and we don't have wide spread use of androids to catch many visual problems. But are that many people really testing manuals? (I kid)

There is tremendous low hanging fruit in Unit and API testing. Its when he UI gets into play that thins get tricky and a little bit of wisdom is needed.

Cancel
Certainly. You can have a very unproductive commodity test effort that misses every single bug in the system, and you can have a well-planned and implemented automation suite. You can also have a totally useless, haphazard automation suite and a well-planned and implemented manual test effort. It’s not whether the effort is manual or automated, but rather how well thought-out and executed it is.
Cancel

SearchCloudComputing

SearchAppArchitecture

SearchITOperations

TheServerSide.com

SearchAWS

Close