Reasons for testing Software in the Cloud
The current state
Test executions are computationally intensive and time-consuming.
The complete testing of a software application can take several days to weeks. Most often, the tests are still conducted on individual workstations.
Scaling the infrastructure to accelerate test execution times quickly encounters organizational, technological, operational, and/or financial limits. We wanted to understand exactly how setting up tests in the cloud works.
To test software, it takes
System under Test
For example, software models (Simulink, Targetlink), code (C/C++).
Test Cases
At best, they represent a meaningful stimulation of the System under Test.
Test Environment
The test environment stimulates the System under Test and measures and evaluates results.
A test environment requires many IT resources (especially memory and CPU) because it performs multiple tasks simultaneously:
Stimulation of the System under Test
Execution of the System under Test
Measurement of the System under Test
Assessment of the System under Test´s behavior
In Model/Software-in-the-Loop tests, stimulation is one of the computationally intensive components, as it involves runtime calculations of the environment simulations for the stimulation. Therefore, tests require almost all resources of a computing unit.
To accelerate and streamline testing, the use of cloud solutions is recommended. In the cloud, computing units can be virtually cloned, and the parallel instantiation of multiple cloned computing units allows for arbitrary, horizontal scaling. The tests are divided among the instances, executed in parallel, and independent of other instances. Test execution times can be almost infinitely reduced by cleverly dividing the tests.
What we have accomplished
We have extensively and repeatedly examined how test execution times can be reduced for automotive software development using state-of-the-art cloud technologies.
The focus of the analysis was primarily on the usability for setup and operation, as well as the costs for operation, with the overarching goal of accelerating development through shorter feedback cycles to reduce the time-to-market of software-based vehicle functions.
For development teams, IT, and management, the cloud-based approach offers interesting use cases, which could be familiar to many as real-world examples:
Typical scenario: Just before the release, a last-minute change comes in that urgently needs to be included in the current version. A CCB or management decision has been made. Implementation only takes a day. Unfortunately, it’s in a central software module, so three-quarters of all tests need to be re-run.
The necessary test run on the high-performance machine takes at least five days, two days later than the promised delivery date. And the documentation still needs to be created. The deadline must be met at all costs.
In critical situations, such as impending shifts in the start-of-production, short-term, significantly higher costs may be justified. With test solutions in the cloud, it’s possible to create as many computers as needed at short notice. And with the principle that a machine is set up as desired and then cloned multiple times, more virtual machines can test much more.
If one were to say, for example, that one test case is executed per machine, then with 7,200 tests, there would be 7,200 machines. If one test case takes one minute, then a complete test run could be completed in a matter of minutes due to the large number of virtual machines, instead of five days on a single physical machine.
Feedback is Breakfast for Champions. And Testing is the Feedback in Development.
In our experience, this holds true for all software development: The faster the development receives feedback on the success of the changes made, the more effective and efficient it becomes.
Especially in large and complex software products with numerous dependencies, a small change can lead to significant problems. According to the Rule of Ten, the cost of fixing errors increases exponentially with the time it takes to discover the error. This means that the later an error is found, the more expensive it is. That’s why early testing and fast results are so important.
For quick starts, Continuous Integration (CI) environments have become established in professional software development. With every change made to the codebase (Push/Commit), tests are automatically initiated. The CI orchestrates the entire integration process by incrementally loading, building, and testing software components from unit to end product. Tests can be executed on one or more different computing units. Additionally, cloud integration with CI environments is possible to further accelerate testing.
Companies often have to invest a considerable amount in hardware to be able to run a large number of tests simultaneously. The setup and operation tie up valuable internal resources, as there is usually not enough personnel in the IT departments for the growing future demand. Scalability is therefore fundamentally inadequate.
During start-of-production launches and just before new releases, development teams often have temporarily increased and unpredictable infrastructure needs for their tests. With virtual machines in the cloud, temporarily occurring demand peaks can be absorbed, which also means relief for the IT department.
Hybrid approaches, such as combining on-premise and cloud, in conjunction with sophisticated IT cloud operation strategies, can lead to optimal cost-benefit ratios, as well as cloud-only approaches, with the option for immediate, rapid scaling.
And then there’s the release from five years ago. Now, there’s a need to implement a small change for only a small number of vehicles. The infrastructure has since evolved, been enhanced, and software has been updated. Now, all old tests need to be migrated to the new infrastructure. A huge effort for a small change.
Here too, the cloud offers an interesting solution: Once set up, virtual machines can be archived and reactivated at a later time. One would need to re-familiarize themselves with the processes, methods, and tools used at that time. In our experience, however, this is easier and faster than a complete migration from old to new.