How to address 5 common CI/CD implementation challenges
CI/CD removes the manual steps involved when you create and install application code. Address these CI/CD challenges and ease the flow of your software pipeline.
A common mantra in today's business world is, "How can I move my product faster?" Companies of all sizes want to accelerate their product cycles -- whether they want to release something entirely new, add features to an existing product, or push out bug fixes to an application.
Historically, an organization might have developed code in house, added the code to a server and then managed the code in perpetuity. Today, companies embrace faster and more efficient methodologies, in particular, those that use continuous integration/continuous delivery (CI/CD).
CI/CD has many pros and cons -- better code quality, shorter testing cycles and easier rollback; changes that unexpectedly propagate across services, and continuous monitoring, reporting and resource management. Let's explore several challenges that you may encounter when implementing CI/CD, as well as some ways to mitigate those challenges.
What is CI/CD?
First, here is a primer on the methodologies involved in CI/CD and how it all works together:
- Continuous integration (CI) takes code from a source control repository and puts it in a centralized location (the build). This build takes the shape of an artifact or a consolidation of code to deploy.
- Continuous delivery (CD) takes that built code and runs various tests for performance, security and usability. If the build passes, it can be deployed with a trigger or button that requires human intervention.
- Continuous deployment automatically deploys the code to customers, with zero human intervention -- if the finished build passes tests, it deploys straight to production. Consider this only if you're confident that the code has zero chance to create an issue (bugs, incorrect resource allocations, platform stability, etc.)
The first step in the CI/CD process is to build the code and consolidate the code in one location. If you have code in say, GitHub, you would run the CI process against a GitHub repository and then store the code in that pipeline.
From here, the CI/CD process consumes the build, configures monitoring for the code, runs various tests and deploys the code to its destination, whether to staging for final manual signoff or automatically straight to a production environment.
What could go wrong? You may confront one of these five scenarios.
Challenge 1: Prioritizing which processes to automate with code
The first big hurdle is to determine what level of code to build: application code, automation or scripting code, or even infrastructure as code? The answer is simple: build it all. No matter which it is, you'll use it later -- say, to automatically deploy services to AWS using Terraform.
The goal is to never manually run any code, regardless of its function. Where you start depends on how much improvement you'll see by automating a task vs. a manual process. What is the frequency of the project or task, what time and resources are spent to code and test and deploy, and what are the risk of errors? Add them all together and what are the cost differences?
Challenge 2: Store code for CI/CD in the right place
Think about where you want to store code. Git is a popular open-source version control system for many developers and organizations, but there are other options, such as Apache Subversion (SVN).
IT shops that already use cloud-based CI/CD might prefer to use those platforms' deployment options instead. Azure Repos, for example, enables teams to store source code inside the Azure DevOps platform. However, this sets up a problem if an organization chooses to move away from Azure DevOps and use another CI/CD platform.
The best approach to storing code is to keep it more universal and use something Git-based such as GitHub, GitLab or Bitbucket, and not lock into a specific platform. If you decide to move to a different Git-based repo, you won't worry about nuances that arise with moving an entire codebase to a different type of repository.
Challenge 3: Which deployment path is best
As the build exits the source and build phases and enters the testing and delivery/deployment end of the process, all prerequisites are installed for the application to run (modules, libraries, dependencies, etc.). The build is then subjected to various tests against performance, usability and security criteria. If the build passes, it is ready for deployment.
An executed build can be an application that runs on a VM, container or as a serverless function, or as a script that runs against a server. It can also be infrastructure as code to run locally or in the cloud, such as Terraform, to create or trigger a service in AWS or Azure.
Regardless, the same rules apply here as in the CI process -- deploy any code that you would have to otherwise run manually on a terminal.
Whether you choose continuous delivery or continuous deployment depends on your comfort level to let the process run on its own. Many shops choose to perform one more round of shakedown testing and checks before shipping to production. IT shops already comfortable with automation and confident in their build and deployment processes may prefer to skip straight to automated deployment. Organizations getting started with CI/CD might be better off to choose delivery for most builds.
Challenge 4: Avoid single, broad pipelines
All CI pipelines should be unique. You shouldn't have one CI pipeline for multiple source control repositories. When you have more than one CI pipeline for each application or codebase, this opens up more flexibility in terms of testing and ensuring code quality.
For example, if your run multiple CI pipelines with different languages, such as Python and JavaScript, you can create and automatically run unit tests for each language, rather than set up multiple unit tests for different languages into one CI pipeline.
Separate pipelines also make sense for projects even if they share a deployment process and destination. One pipeline to deploy an entire environment for AWS quickly becomes cumbersome and difficult to monitor, or even to define individual pipeline tasks.
Challenge 5: Pick the right combination of CI/CD tools
Choosing the right CI/CD platform is another hurdle. There are dozens of CI/CD tool options, some of which cover specific aspects (version control, CI engine or delivery only) and some which offer more complete CI/CD functionality. How do you choose which tool or tools best fit your needs?
Under the hood, many of these tools appear to be similar. Some CI/CD platforms are more feature-complete than others, such as broader CI/CD coverage vs. just CI or delivery functions. Some offer stronger integration with certain cloud services and other platforms. For example, Azure DevOps has a plethora of extensions and tasks for most clouds and applications, while the newer GitHub Actions is slowly catching up with available extensions.
Another consideration is the CI/CD platform's feature richness. Octopus Deploy, for example, is a delivery/deployment platform only, with no build process, so you'll need to implement a CI tool or tools. This may be a deal breaker if you need a platform with functions that more broadly span CI/CD steps.
Additionally, many CI/CD tools are open source and free of charge, or mostly free, while others cost money. You'll want to research this aspect while testing out the CI/CD platform. Keep in mind that "open source" and "free" are anything but. You'll have to manage it yourself, and support may rely on the user community.
CI/CD has quickly become the most popular way to ensure that code properly and efficiently runs on any system, whether it is a virtual machine, serverless or a container. Staying alert to common challenges in CI/CD will help ensure your software development efforts meet your organization's business and IT goals.
