Deploy is a critical step in the SAFe for DevOps Health Radar. After we have built a deployable package and tested it in a staging environment, we now want to continuously deploy our changes into the production environment. The goal is to deploy with high frequency and low risk. In this video, I explain how we achieve this and what practices enable continuous deployment.
The Journey So Far#
Before we reach the Deploy step, we have already completed several steps in the SAFe DevOps Health Radar. It started with a hypothesis where we captured bright ideas in epic form. We then collaborated and researched to understand customer and market needs. In the architect step, we designed the minimal architecture to prove our hypothesis. We synthesized by breaking down the epic into features for the program backlog. In the develop step, we coded the feature by breaking it into user stories. We committed the code and built a deployable package. After that, we deployed the package into a staging environment for end-to-end testing.
Now, in the deploy step, we bring our changes into the production environment.
Why Continuous Deployment?#
The reason we want to continuously deploy our changes into production is simple: small changes carry lower risk than large changes. When we deploy small changes frequently, each deployment has a lower chance of failure. In contrast, if changes accumulate over three months, the deployment becomes much larger and the risk increases significantly.
Continuous deployment means deploying with high frequency and low risk into the production system.
Separating Deployment from Release#
To continuously deploy into production, we need to separate deployment from release. A deployment is bringing the compiled code into production with the feature switch turned off. A release is switching the feature switch on in the production environment.
This distinction is fundamental. We already discussed this concept in the architect step, because it is important to architect systems so that deployment and release can be separated from the start.
Feature Toggles#
Feature toggles (also called feature switches) enable us to switch features on and off. At its core, a feature toggle is nothing more than an if-statement. If the flag is set, the new behavior is active. If the flag is not set, the old behavior continues.
There are important rules to follow when working with feature toggles:
- Each toggle adds a test case. Multiple feature toggles create a combinatorial explosion of scenarios that need testing.
- Remove toggles after release. A feature toggle is temporary. Once the feature is released and stable, remove the toggle from the code.
- Do not confuse toggles with application configuration. Application configuration is long-lived and allows you to configure application behavior. A feature toggle is short-lived and only exists to enable continuous deployment.
Dark Launches#
Separating deployment from release and using feature toggles enables dark launches. In a dark launch, we deploy new code to production with the feature toggle turned off. This lets us test monitoring and alerting in the production environment. We can observe how the newly deployed code behaves with real production data, all without any impact on end users.
Version Control for Everything#
When we want to continuously deploy into production, we need to have everything in version control. Not just the code, but also the infrastructure configuration, test data, requirements, and architecture documentation. Having everything in version control enables fast rollbacks and rapid investigation of production problems, because we know exactly what was deployed and what changed compared to the previous version.
Infrastructure as Code#
Infrastructure as code enables automated environment setup. We define how our infrastructure should look in configuration files, and these configuration files are stored in version control. Using tools like Chef, Puppet, and others, we can automatically provision and configure our infrastructure. This approach saves costs, increases speed, and lowers risk because the infrastructure setup is consistent and repeatable every time.
Deployment Automation#
To continuously deploy to production, we need to automate all the steps required to bring compiled code into production. This is deployment automation. It is important that developers always receive feedback about whether a deployment succeeded or failed.
If full automation is not permitted (for example, in regulated environments with segregation of duty requirements), we can use self-service deployment. This is also known as push-the-button deployment. The deployment process is still fully automated, but a person triggers it by pressing a button.
Selective Deployments#
Selective deployments give us flexibility when releasing. With selective deployments, we can deploy to a single geographic region, a single data center, one server cluster, or one network segment. This lowers risk and enables more flexible releases.
Blue-Green Deployments#
Blue-green deployments are another powerful practice for deployment. We maintain two identical production environments: one is live (green) and one is idle (blue). All users work on the live environment. While they do, we deploy the next version to the idle environment.
Once the deployment to the idle environment is complete, we can test the new version there without affecting any users. When everything looks good, we switch the load balancer so the blue environment becomes live and the green environment becomes idle. If something goes wrong, we can quickly switch back to the previous version.
Blue-green deployments come with trade-offs. You need two full production environments, which increases cost. The approach is straightforward for stateless applications, but becomes more complex when databases or other stateful components are involved, requiring careful architecture and design.
The Maturity Levels#
The SAFe DevOps Health Radar provides a maturity assessment for the Deploy step:
- Sit: Features are deployed to production every three or more months. Deployments are manual and painful. Deployed implies released.
- Crawl: Features are deployed to production at the PI boundary (three-month cycle). Deployments are mostly manual. Deployed implies released.
- Walk: Features are deployed to production every iteration. Deployments are mostly automated. Some features can be deployed without being released.
- Run: Features are deployed to production every iteration, fully automated through the pipeline. Dark releases are common.
- Fly: Features are deployed continuously throughout each iteration. Dev teams initiate deployments directly via pipeline tools. Release is completely decoupled from deployment. Dark releases are the norm.
Key Takeaways#
- Deploy with high frequency and low risk. Small, frequent deployments are safer than large, infrequent ones.
- Separate deployment from release. Deployment brings code into production with the feature toggle off. Release switches the feature toggle on.
- Use feature toggles wisely. They enable continuous deployment, but remember to remove them after the feature is released.
- Leverage dark launches. Deploy to production without releasing to users so you can monitor and validate the new code.
- Automate everything. From infrastructure to deployment, automation is the foundation of continuous deployment.
- Consider blue-green deployments. They enable zero-downtime deployments and fast rollback, but require careful planning for stateful components.