Measure is the step of the SAFe for DevOps Health Radar where everything comes together. After deploying to production and stabilizing, we now collect qualitative and quantitative information about our epics and features. The goal is to validate our hypotheses and make informed strategic decisions. In this video, I walk through what the Measure step involves and why it is essential for building the right thing.
The Full Journey to Measure#
Before we reach the Measure step, we have gone through the entire continuous delivery pipeline. It starts with bright ideas from the customer or the business. We take these ideas and put them into epics, identifying the real hypothesis behind each epic using the epic hypothesis statement.
From there, we collaborate and research to find the real market need or customer need. Then we architect the minimal amount of architecture needed to prove the hypothesis. In the synthesize step, we break down the epic into features, prioritize them on a backlog, build up a roadmap, and elaborate on our vision.
With these features, we move into development. We break features into user stories, develop them, commit source code to version control, and build deployable artifacts. These artifacts are tested end-to-end and deployed to a staging environment for final verification. Then we deploy to production with the feature toggle off.
We verify with a subset of tests that everything works in production and continuously monitor the system. If anything happens, we get alerted and respond as one team. When the business says the time is right, we switch on the feature toggle, stabilize the production environment, and check that everything works according to SLAs.
Now we enter the Measure step.
What Happens in the Measure Step#
In the Measure step, we collect both qualitative and quantitative information about our epics and features in production. The core purpose is to identify the business value behind our epics and features, and to prove or disprove the hypotheses we stated at the very beginning.
This data enables us to make informed strategic decisions. For example, we might decide to stop investing in certain epics or features, or we might confirm that a hypothesis is true and decide to invest more.
Connecting Back to the Hypothesis#
Back in the Hypothesize step, we defined epics and their underlying hypotheses using the epic hypothesis statement. Every significant initiative has a hypothesis behind it, and with the lean startup cycle, we try to identify the minimal viable product that we can build to prove if that hypothesis is true or false.
The key to this validation is the use of leading indicators. With leading indicators, we can identify if we are building the right thing. To gather these indicators, we use the application telemetry that we have built into our solution.
Of course, we need to correlate the business results with the telemetry to determine if the hypothesis is true or false. It is important to be aware of vanity metrics. The number of downloads, for example, might not be the right metric or leading indicator in your case. What counts as a vanity metric can also depend on the specific context you are working in.
Measuring Feature Benefit Hypotheses#
We do not only measure the hypothesis of our epics. We also measure the benefit hypothesis of our features and enabler features. In the Synthesize step, we broke the epic down into features. Each feature has a title, a set of acceptance criteria, and important non-functional requirements. But the most important thing about a feature is its benefit hypothesis, and this is exactly what we validate in the Measure step.
The Maturity Levels#
The SAFe DevOps Health Radar provides a maturity assessment for the Measure step:
- Sit: We don’t define or measure the value of features.
- Crawl: We’ve defined what “value” is but don’t know how to measure it.
- Walk: We capture qualitative feedback from the business about the value of our features.
- Run: We capture qualitative and quantitative feedback from the business and our monitoring systems about the value of our features.
- Fly: We aggregate the quantitative and qualitative feedback to objectively validate the original hypothesis and inform pivot-or-persevere decisions.
Where does your team fall on this scale? The goal is to move toward a state where data objectively validates your hypotheses and directly informs your investment decisions.
Tools for the Measure Step#
Looking at the periodic table of DevOps tools, the tools relevant for the Measure step fall into the AIOps and analytics category. Three examples:
- Matomo: A strong choice when you are in an on-premises environment.
- Google Analytics: A widely used option when you are in the cloud.
- Dynatrace: A comprehensive observability platform for monitoring and analytics.
What Measure Produces#
The output of the Measure step is clear:
Validated hypotheses: For epics, we can determine whether the hypothesis is true or false. For features, we verify whether the benefit hypothesis has been fulfilled.
Informed decisions: With qualitative and quantitative data, we can make strategic decisions about where to invest more, where to invest less, and where to stop entirely.
Business value identification: We gain a clear picture of whether our epics and features deliver the expected business value.
Key Takeaways#
- Measure is where everything comes together. All previous steps in the SAFe DevOps Health Radar lead to this point, where we validate whether we built the right thing.
- Collect both qualitative and quantitative data. Both types of feedback are needed to get a complete picture of the value delivered by features and epics.
- Validate hypotheses with leading indicators. Use application telemetry and correlate it with business results to prove or disprove your hypotheses.
- Watch out for vanity metrics. Not every metric that looks impressive is actually meaningful. Choose leading indicators that truly reflect the value you are trying to deliver.
- Enable informed strategic decisions. The data from the Measure step empowers teams and leadership to make objective, fact-based decisions about investment and direction.
- This step feeds into Learn. The insights gathered here flow directly into the Learn step, where pivot-or-persevere decisions are made.