In this article, I explain what Test End to End means within the SAFe® DevOps Health Radar and why it is essential for delivering high-quality software. Please note that everything discussed here is under the license of Scaled Agile, and that the Scaled Agile Framework is a framework to be used as a toolbox. Take out what fits your needs or what solves your problem.
Where Test End to End Fits in the Pipeline#
In the SAFe DevOps pipeline, big ideas come from customers and the business side and get transferred into epics with a clear hypothesis statement. After that, we collaborate and research with the customer, do interviews, and conduct market research. Then we architect the minimal architecture to prove the hypothesis. In the synthesize step, we break down the epic into features and create a roadmap and a vision. In the develop step, we break features into user stories, develop them, and commit the source code into the version control system. The build server integrates the source code, compiles it, and runs unit tests. The output of the build step is a deployable artifact, which we now test end to end.
What Happens in the Test End to End Step#
In the Test End to End step, we validate the solution against the acceptance criteria in a production-like environment. During the build step, we already executed unit and component tests. Now in the Test End to End step, we perform tests on a feature level, which are system-level or integration tests.
To run these integration tests reliably, we need an environment that is as close as possible to the production system. This ensures we can trust the test results. Everything needs to be in version control, including the configurations of the test environments.
Test Data#
To execute tests in a test environment, we need test data. This topic starts in the architect step, where we design the system to be testable, and continues into the develop step, where we create the test data.
It is essential to use synthetic, production-like test data. In many projects I see people using production data or anonymized production data for testing. This is possible but not recommended. When you use production data, you get flaky tests because the data can always change or be removed. A flaky test has zero value. All test data needs to be stored in version control.
Service Virtualization#
Usually, your system depends on other systems through interfaces. To move fast, you need to be decoupled from these other systems and test independently. We achieve this through service virtualization: creating simulators for external systems so we can test independently of their release cycles.
This has the great benefit that we can use our synthetic test data for the other systems as well. We enrich the simulators with the correct synthetic test data, and these simulators also need to be maintained under version control.
Non-Functional Testing#
We not only need to do functional testing but also non-functional testing. Non-functional requirements are system attributes such as security, reliability, performance, maintainability, scalability, and usability. These are the “-ilities” of our system.
An important distinction: a non-functional requirement is not a backlog item, not a user story, and not a feature. It is a constraint on our backlog that applies to all backlog items.
The Agile Test Matrix#
To get a clearer picture of test automation, I use the agile test matrix, which consists of four quadrants:
Q1: Unit and Component Tests#
These are the tests we execute during the build step. They can easily be automated, and we use test-driven development in this area.
Q2: Functional Tests (User Acceptance Tests)#
Here we execute acceptance tests defined in features and user stories. We use behaviour-driven development to create a good set of acceptance criteria. The tests in this area are preferably automated.
Q3: System-Level Acceptance Tests#
In this quadrant, we validate the behaviour of the whole system, including interactions with other systems. We use exploratory testing or workflow testing. Automation is hard in this area, and maintaining these tests is expensive, so there are usually many manual tests.
Q4: Non-Functional Tests#
Here we execute the non-functional requirements tests against our system. There are specialized tools for security testing, performance testing, and more. Because of the strong tool support, it is easy to automate these tests, resulting in a high degree of automation.
The Output#
The output of the Test End to End step is a verified solution. We have executed functional tests, integration tests, regression tests, performance tests, and exploratory testing. We also have test evidence, which is usually needed in regulated environments.
SAFe DevOps Health Radar Assessment#
The SAFe DevOps Health Radar provides a maturity model for Test End to End:
- Sit: Testing is performed manually in environments that do not mimic production. Testing occurs in large batches during a scheduled testing phase.
- Crawl: Testing is mostly manual in non-production-like environments. Stories are implemented and tested independently within a single PI (three-month cycle).
- Walk: Half the testing is automated and performed in production-like or production-simulated environments every PI.
- Run: The majority of tests are automated and run in production-like environments. Stories are implemented and fully tested every iteration (two-week sprint).
- Fly: Successful builds trigger automatic deployments to production-like test environments. All tests are automated, tests run in parallel, and changes are fully validated after every commit.
Test End to End means validating our system end to end in a production-like environment, covering both functional and non-functional testing. It is a critical step in the Continuous Integration pipeline that ensures we deliver verified, high-quality solutions.