Verify is a critical step in the SAFe for DevOps Health Radar. After deploying our package into the production environment, we need to confirm that the new functionality works correctly and does not negatively affect the integrity or robustness of the existing system. Only after this verification is complete can we confidently release the new features to end-users. In this video, I walk through what the Verify step involves and how it fits into the overall continuous delivery pipeline.
Where Verify Fits in the Pipeline#
Before reaching the Verify step, we have gone through continuous exploration and continuous integration. We transformed bright ideas from the customer and the business into hypothesis statements, collaborated on research, architected the solution, and broke down epics into features and user stories. The code was built, tested, and packaged into a deployable artifact. This artifact was deployed first to a staging environment and then to the production environment.
Now that the code is deployed to production with the feature toggle off, it is time to verify that everything works as expected before we switch the feature toggle on and release it to users.
Separating Deployment from Release#
One of the cornerstones of DevOps is separating deployment from release. Deployment is the act of bringing the compiled code into production with the feature toggle turned off. Release is the act of switching the feature toggle on so that the new functionality can be used by the end-users.
This separation is what makes the Verify step possible. Because the code is in production but not yet visible to users, we have the opportunity to test it thoroughly in the real production environment without any risk to end-users.
Production Testing#
The core activity in the Verify step is production testing. We execute tests that we have already run in lower stages again in the production environment. We do this by simulating end-user behavior with synthetic transactions. These synthetic transactions allow us to verify two things:
- The existing functionality still behaves as expected.
- The new functionality does what it should do and is ready to be released.
This approach gives us confidence that the deployment has not introduced any regressions and that the new features are working correctly in the real production environment.
Build, Test, and Deployment Automation#
To verify the new package in production, we need solid build, test, and deployment automation. With this automation in place, we can trigger a build on every commit, execute all unit tests, run code analysis and security checks, and automatically deploy the package into staging and production environments.
What is important here is the fast feedback loop. Developers need to know quickly whether their changes pass all verification checks. The faster this feedback, the faster issues can be resolved.
Test Data Management#
Proper test data management is crucial for the Verify step. We need a well-maintained set of synthetic test data so that we can execute our tests reliably. This test data must be stored in version control.
What we do not want to do is use production data or an anonymized production database for testing. Production data changes constantly, and those changes lead to flaky tests. A flaky test has zero value because it erodes confidence in the test suite and makes it impossible to distinguish real failures from noise.
Non-Functional Requirements#
When verifying functionality in production, we also need to verify non-functional requirements. These are system attributes like security, reliability, performance, maintainability, scalability, and usability. Non-functional requirements are a constraint on the entire backlog, which means that every user story, every feature, and every epic must comply with them.
We verify non-functional requirements first in the staging environment, but we also need to confirm that they are met in the production environment. The production environment may have different characteristics than staging, so this verification is essential.
The Maturity Levels#
The SAFe DevOps Health Radar provides a maturity assessment for the Verify step:
- Sit: Deployments are not verified in production before being released to end-users.
- Crawl: Deployments are verified with manual smoke tests and user acceptance testing (UAT). Deployment issues are addressed within a stated triage warranty period. Issues are often corrected directly in production.
- Walk: Deployments are verified with manual tests prior to releasing to end-users. Rolling back is painful or impossible. No changes are made directly in production.
- Run: Deployments are verified with automated production tests. Rolling back is painful or impossible. No changes are made directly in production.
- Fly: Automated production tests run on an ongoing basis and feed monitoring systems. Failed deployments can be rolled back instantly or fixed forward through the entire pipeline.
Key Takeaways#
- Verify before releasing. After deploying to production, test the new functionality and existing functionality before switching the feature toggle on for end-users.
- Use synthetic transactions. Simulate end-user behavior in production to confirm that everything works as expected without affecting real users.
- Maintain good test data. Use synthetic test data stored in version control. Never rely on production data for testing because it leads to flaky tests.
- Verify non-functional requirements. Security, performance, reliability, and other system attributes must be checked in the production environment, not just in staging.
- Automate the verification. Build, test, and deployment automation is essential for fast feedback and reliable production verification.
- Separate deployment from release. This separation is what makes safe production verification possible in the first place.