Maintaining software quality can be challenging, certainly when you need to deliver software to customers fast....
Continuous integration is one of the three traits of continuous software development and it can help reveal problems quickly. It is defined as:
Continuous integration (CI) is a software engineering practice in which isolated changes are immediately tested and reported on when they are added to a larger code base. The goal of CI is to provide rapid feedback so that if a defect is introduced into the code base, it can be identified and corrected as soon as possible.
When using a continuous integration process, developers are expected to integrate frequently -- committing code changes at least daily or even more often. When they commit their code, it will be merged with the code in the main branch and tested automatically. As multiple developers are committing code, conflicts may arise between parallel code changes, or the automated regression testing might fail. The advantage of a continuous integration process is that it reveals these problems quickly. It is a practice that provides fast feedback to developers on code changes that they have made.
If the build breaks or automated testing reports errors, immediate action is needed. You need to "stop the line," investigate what happens and solve the problem. A stop-the-line approach helps you to catch problems when they are still small and avoids getting into big problems, as Kevlin Henney describes in a recent article.
In most organizations, the expectation is that the developer who caused the build to fail will have to solve the problem. At first sight, this sounds reasonable, because it is that developer's code that is conflicting with the code in the main branch, causing the failure. Although the developer is the first person to take action and explore the problem, it might be that other people should get involved to solve it. Also, the way to solve the failure depends on the underlying faults.
It's important to find and correct the fault that is causing the failure. Faults might be in the code that the developer committed, but it is also possible that there are faults in another part of the code, which are triggered by the code changes. In some cases, the failure might be traced back to the design or architecture, maybe even to the requirements to uncover the origins or root causes of the failure.
You need to understand the fault to be able to solve the problem effectively. For some failures, it's easy to find the faults, while others might be more complex to explore. Root cause analysis can be used to investigate major or repeating problems, and come up with a structural solution.
Based on the fault, other people may need to be involved to solve it, rather than just the developer who caused the failure. You can discuss architecture or design problems with architects or tech leads. For complex problems, my suggestion is to use pairing or agile swarming to solve them. Problems revealed by a continuous integration process have high priority, so everybody will benefit when people are working together to solve them quickly.
You can strive for a culture where breaking the build is considered wrong, but there are some pitfalls to be aware of. It's better to have small problems than big problems, so you want to make sure developers don't become afraid to break the build and hence do not commit their code frequently. Also, you have to balance testing by developers with regression testing, and make sure to invest enough time and money in automated testing to have a test set that guards the quality of your products.
In Lean Software Development, defects are waste. You may want to look for ways to prevent failure due to defects instead of catching them with continuous integration. Double loop learning and root cause analysis are some of the techniques that can be used to learn from defects, as well as define actions to prevent them from happening. Using such techniques, continuous integration and development becomes a foundation for continuous improvement.
Agile best practices
How to measure software quality