Testing in a traditional software development shop has become a 10-headed monster for most corporations and especially for Agile organizations with a continuous release cycle. Like software developers, testers need to update their techniques to add greater efficiency and value as the organization adopts continuous development.
Continuous development is practiced as part of an Agile approach to software development. DevOps is essential and the involvement of a full team in a rapid release cycle means that testing techniques need to change.
In continuous development, programmers integrate code frequently, or even daily, rather than at the end of a long development cycle. Integrating code builds frequently exposes defects because of the merging of different code by developers on software code simultaneously. It's practically impossible not to find defects when code is merged. It's imperative in a shortened, rapid development effort that testers are not left holding the bag of defects. Programmers, testers and operations need to work together at each stage of the process to add the right tests in the most effective manner.
Test planning -- automated versus manual
The first step in any testing effort is a plan. It doesn't have to be a formal test plan; testers can use anything that works for them including a written outline, a spreadsheet or even a full-blown test matrix. The objective is to have a detailed plan to:
- gather input from development;
- gather input from operations;
- review with the product manager.
Gathering input on a test plan ensures the tester understands the design and acceptance criteria and defines the testing types used.
The tester needs to determine which testing areas are to be covered by automated code tests (unit tests), back-end automated code tests, coded UI tests and manual tests. The test plan determines what type of testing is written and how it's executed.
In a rapid, continuous development system, manual testing is too cumbersome to be the only method used. Automated tests need to be developed in multiple areas to assist in finding defects prior to a production release. The combination is essential to demonstrate that user stories are implemented and functional. Manual testing is the most effective method for verifying end user workflows. However, automation is required so that as new builds are continuously deployed, smoke or regression type tests can automatically generate a level of confidence that the code is working as expected. Manually retesting each feature continuously is impossible and coding automated tests that run quickly and frequently is essential to preventing defects in a production release.
Consider using manual testing for new stories, stories with substantial changes and user workflows. Use automated testing to test code merges and code builds, and execute unit tests to test components within the software and to confirm the code base is stable.
Test execution -- valid, useful automated tests
The trick is developing useful, valid and verifiable automated tests in different "zones" of the application and making them flexible enough to find defects and not require extensive maintenance. Unit tests provide an effective method of maintaining a working code base. Developers update them at the same time they update the code and so the action of maintenance is built in. Unit tests can be automated to execute each time a line of code is altered and when a build is created.
Another benefit is adding integration unit tests and automating them to execute every time a new line of code is checked or a release build is created. Integration unit tests go beyond the component level and include external databases, websites, or other external connections and transfer points. Automated unit and unit integration tests add significant value and their maintenance becomes part of the coding effort.
Consider creating coded UI tests for manual tests if they need to persist as regression tests. The generated code executes the same as a manual test. Make sure the test developer adds accurate validation points so the test is verified against a result. A typical hazard of coded UI automated tests is the maintenance involved in keeping them updated for an application that changes constantly. Keep maintenance effort in mind and have a backup testing suite in place so automated tests needing maintenance don't negatively impact a release schedule by generating false failures.
Testing reality -- step through the workflows
Should continuous development teams automate all testing and abandon any manual testing? It's a popular notion but a guaranteed method of releasing bugs into production that pass automated code tests but not end user expectations. Especially in Web and mobile application development where user experience becomes increasingly critical it's essential to use both to complement each other.
Manual testers continuously test new and changed features in order to find workflow, visual and other end-user defects. Manual exploratory testing simulates user acceptance testing and serves to ensure the user-facing workflows truly function as expected. I've used many tools that tout their 100% test automation and they release significant user-facing defects and workflow flaws apparently not detected by automation. Mobile applications are even more user-focused than Web applications and therefore the need for manual testing on top of automated tests is more apparent. The combination of manual and automated tests is the most effective method of ensuring a high-quality release in a continuous fashion.
Check out the lessons Ancestry.com learned as they moved into continuous development
Read up on solid strategies for leading multiple teams in Agile