In this video, I take a deep dive into the science behind DevOps. Specifically, I look at the DORA metrics, where they come from, and the book Accelerate that provides the scientific foundation for everything we know about high-performing software delivery organizations.
The Book Accelerate#
The book Accelerate: The Science of Lean Software and DevOps was published in 2018 and has 288 pages. I highly recommend it to everyone working on DevOps or on a DevOps transformation. It is the essential book to read.
The book was written by three remarkable authors. Dr. Nicole Forsgren did the research and created the startup DORA (DevOps Research and Assessment). Jez Humble is well known for his books Continuous Delivery and The DevOps Handbook. Gene Kim is known for The Phoenix Project, The Unicorn Project, and is co-author of The DevOps Handbook. He also created the DevOps Enterprise Summit. These three books, Accelerate, The DevOps Handbook, and The Phoenix Project, are the essential reading list for anyone in the DevOps space.
The book has three parts. Part one describes the findings: the 24 key capabilities they discovered through their research. Part two covers the science, including all the statistical analysis techniques they used. Part three is a case study from ING about their DevOps transformation, serving as a blueprint. If you have limited time, at least read part one.
The 24 Key Capabilities#
The most amazing thing about Accelerate is that they scientifically analyzed companies doing DevOps and found 24 key capabilities that drive improvements in software delivery performance. These are organized into five categories:
Continuous Delivery Capabilities: Version control, deployment automation, continuous integration, trunk-based development, test automation, test data management, shift-left security, and continuous delivery.
Architecture Capabilities: Loosely coupled architecture and empowering teams to make their own decisions.
Product and Process Capabilities: Customer feedback, organizing across the value stream, working in small batches, and allowing teams to experiment.
Lean Management and Monitoring Capabilities: Change approval boards are ineffective (science has confirmed this). You should have monitoring systems, proactive notifications, limit work in process, and visualize work across the value stream.
Cultural Capabilities: Westrum organizational culture (generative over bureaucratic or pathological), supported learning, collaboration across teams, job satisfaction, and transformational leadership.
“Science has found out that change approval boards are for nothing. Many of the processes are redundant because you have already automated them with your pipeline.”
What makes this research extraordinary is not just the capabilities themselves, but the relationships between them. The book provides a picture showing how these capabilities influence each other. On the right side is what you want to achieve. On the left side is what you need to do to achieve that. For me, this is the reference picture for any DevOps transformation.
The DORA Metrics#
How do you measure software delivery performance? It is not team velocity. It is not lines of code. It is not the number of commits or test coverage. The scientifically proven metrics are the DORA metrics:
Lead Time for Changes: How long from code commit until production. Note that the time before code commit (ideation, requirements, implementation) is not included because it is statistically not significant. Elite performers do this in less than an hour. High is one day to one week. Medium is one month to six months. Low is more than six months.
Deployment Frequency: How often you deploy to production. Remember: deployment is bringing code into production with the feature toggle off. Release is switching the toggle on. Elite organizations deploy on demand, multiple times per day. High is once per day to once per week.
Time to Restore Service: How long to recover from an incident. Elite teams restore within an hour. This metric is closely related to lead time and deployment frequency.
Change Failure Rate: The percentage of deployments causing a failure. Elite is 0 to 15%. High and medium are 16 to 30%. Low is 31 to 45%.
Important: always consider at least two metrics, one from speed (lead time, deployment frequency) and one from stability (time to restore, change failure rate). Optimizing only for speed or only for stability gives a distorted picture.
The Numbers Speak for Themselves#
The State of DevOps reports compare elite performers with low performers. The 2019 report shows that elite companies have:
- 208x more frequent deployments
- 106x faster lead time
- 2604x faster time to restore service
- 7x lower change failure rate
By 2021, these numbers had grown even larger. The market is accelerating. Elite organizations are getting faster, and the gap between elite and low performers continues to widen.
The benefits are clear: faster time to market, more value for money, higher quality, higher customer satisfaction, and top qualified employees. This is exactly what CIOs want.
The Shift Over Time#
Looking at the state of DevOps data over time, in 2018 there were only a few elite companies, many high performers, some medium, and a large portion of low performers. Over the years, more companies moved to elite and high levels. The low performers either improved or went out of business.
Key Takeaways#
- Read Accelerate. It is the essential book for anyone working in DevOps. At minimum, read part one.
- Focus on the 24 key capabilities. They are scientifically proven to drive software delivery performance.
- Use the DORA metrics. Lead time for changes, deployment frequency, time to restore service, and change failure rate are the only scientifically valid KPIs.
- Distinguish deployment from release. Deployment brings code to production with the feature toggle off. Release switches it on.
- Balance speed and stability. Always measure both dimensions to get an accurate picture.
- The evidence is overwhelming. Elite DevOps organizations outperform low performers by orders of magnitude across all metrics.