This content is part of the Essential Guide: Next generation Agile: Guide to continuous development
Manage Learn to apply best practices and optimize your operations.

Continuous development brings changes for software testers

Continuous development calls for lots of automated testing and a little manual testing. Form a continuous testing plan based on the right balance between the two.

Testing in a traditional software development shop has become a 10-headed monster for most corporations and especially for Agile organizations with a continuous release cycle. Like software developers, testers need to update their techniques to add greater efficiency and value as the organization adopts continuous development.

Continuous development is practiced as part of an Agile approach to software development. DevOps is essential and the involvement of a full team in a rapid release cycle means that testing techniques need to change.

In continuous development, programmers integrate code frequently, or even daily, rather than at the end of a long development cycle. Integrating code builds frequently exposes defects because of the merging of different code by developers on software code simultaneously. It's practically impossible not to find defects when code is merged. It's imperative in a shortened, rapid development effort that testers are not left holding the bag of defects. Programmers, testers and operations need to work together at each stage of the process to add the right tests in the most effective manner.

Test planning -- automated versus manual

The first step in any testing effort is a plan. It doesn't have to be a formal test plan; testers can use anything that works for them including a written outline, a spreadsheet or even a full-blown test matrix. The objective is to have a detailed plan to:

  • gather input from development;
  • gather input from operations;
  • review with the product manager.
The combination [of automated and manual testing] is essential to demonstrate that user stories are implemented and functional.

Gathering input on a test plan ensures the tester understands the design and acceptance criteria and defines the testing types used.

The tester needs to determine which testing areas are to be covered by automated code tests (unit tests), back-end automated code tests, coded UI tests and manual tests. The test plan determines what type of testing is written and how it's executed.

In a rapid, continuous development system, manual testing is too cumbersome to be the only method used. Automated tests need to be developed in multiple areas to assist in finding defects prior to a production release. The combination is essential to demonstrate that user stories are implemented and functional. Manual testing is the most effective method for verifying end user workflows. However, automation is required so that as new builds are continuously deployed, smoke or regression type tests can automatically generate a level of confidence that the code is working as expected. Manually retesting each feature continuously is impossible and coding automated tests that run quickly and frequently is essential to preventing defects in a production release.

Consider using manual testing for new stories, stories with substantial changes and user workflows. Use automated testing to test code merges and code builds, and execute unit tests to test components within the software and to confirm the code base is stable.

Test execution -- valid, useful automated tests

The trick is developing useful, valid and verifiable automated tests in different "zones" of the application and making them flexible enough to find defects and not require extensive maintenance. Unit tests provide an effective method of maintaining a working code base. Developers update them at the same time they update the code and so the action of maintenance is built in. Unit tests can be automated to execute each time a line of code is altered and when a build is created.

Another benefit is adding integration unit tests and automating them to execute every time a new line of code is checked or a release build is created. Integration unit tests go beyond the component level and include external databases, websites, or other external connections and transfer points. Automated unit and unit integration tests add significant value and their maintenance becomes part of the coding effort.

Consider creating coded UI tests for manual tests if they need to persist as regression tests. The generated code executes the same as a manual test. Make sure the test developer adds accurate validation points so the test is verified against a result. A typical hazard of coded UI automated tests is the maintenance involved in keeping them updated for an application that changes constantly. Keep maintenance effort in mind and have a backup testing suite in place so automated tests needing maintenance don't negatively impact a release schedule by generating false failures.

Testing reality -- step through the workflows

Should continuous development teams automate all testing and abandon any manual testing? It's a popular notion but a guaranteed method of releasing bugs into production that pass automated code tests but not end user expectations. Especially in Web and mobile application development where user experience becomes increasingly critical it's essential to use both to complement each other.

Manual testers continuously test new and changed features in order to find workflow, visual and other end-user defects. Manual exploratory testing simulates user acceptance testing and serves to ensure the user-facing workflows truly function as expected. I've used many tools that tout their 100% test automation and they release significant user-facing defects and workflow flaws apparently not detected by automation. Mobile applications are even more user-focused than Web applications and therefore the need for manual testing on top of automated tests is more apparent. The combination of manual and automated tests is the most effective method of ensuring a high-quality release in a continuous fashion.

Next Steps

Check out the lessons learned as they moved into continuous development

Read up on solid strategies for leading multiple teams in Agile

Dig Deeper on Topics Archive

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

Automation is essential for rapid deployment environments, but it's a mistake to think that all testing, even in these environments, will be automated. There's still a lot of manual and interactive testing that needs to be done. Who does it, and at what level of the process, is another discussion entirely (does a dedicated tester need to do it?).
1. I wish the word continuous would stop being butchered.  Continuous implies an unbroken line.  There are clearly times when people stop and do things that aren't development.  Just like they stop from testing at times to do other activities in the development workflow.

2. As teams move to more CI/CD (as it is called) style of development there's a lot that needs to change with testing.  For one thing the old way of waiting till the end to test is too late.  Testers will have to be involved up front, asking what and how they will be able to test certain things, and what and how  the pieces that need tested are being delivered.  

There's a lot of value to be had there, and automation can help in some ways, but with these 'fast' development timelines, in CI/CD, it seems all the more critical to identify early where human eyes are needed, over the machine's pre-planned evaluations in doing automated checks.