Page 45 - SMILESENG

Intl. Summer School on Search- and Machine Learning-based Software Engineering
Regression Testing for Self-driving Cars as Cyber-physical Systems in Virtual Environments
Christian Birchler
Zurich University of Applied Sciences
Winterthur, Switzerland [email protected]
Abstract—Detecting bugs early in the software testing process is crucial to lowering the costs. Testing cyber-physical systems (CPS) is more expensive than traditional software systems, so regression testing for CPS is a promising approach to deal with the higher testing costs. Regression testing consists of test selection, prioritization, and minimization. Our work focuses on using self-driving cars (SDC) as a CPS use case.
First, we tackled the problem of test selection for CPS in virtual environments. For this, we developed a tool to select test scenarios from the test suite with machine-learning models for SDCs. SDC-Scissor is a cost-effective test selector for SDC software that can predict failing tests with an F1-score up to 96% and speed up the test execution in simulation by 170%. This selection approach reduces the time spent for running tests that likely pass but increases those that likely fail.
Secondly, to deal with the test prioritizing problem, we devel- oped a multi-objective genetic algorithm called SDC-Prioritizer. It finds the optimal test execution so that the diversity of tests maximizes and the execution time minimizes. The evaluation of SDC-Prioritizer shows that our approach outperforms a random and a greedy baseline prioritizer statistically significantly. I.e., SDC-Prioritizer detects more defects as the baseline with the same execution time. With this tool, the testing process is getting more efficient by revealing more defects in a shorter time.
Both tools use labeled datasets of road scenarios that consist of lists of consecutive road points for evaluation. The execution of the scenarios labels the tests as safe or unsafe, i.e., if the SDC is driving off the lane or not. Future work will minimize the test scenarios and complete the regression testing framework.
I. INTRODUCTION
The future of mobility will be driven by Artificial Intel- ligence (AI). There are several examples of the replacement of humans with AIs [20], [24], [26]. In 2021, Tesla began to deliver their SDCs that only rely on camera input [25]. The AI that drives the car relies only on the camera data and the according vision-based algorithms. These examples show the direction toward autonomous mobility that heavily rely on AI and algorithms.
However, some CPS, like SDCs are life-critical and require proper testing. Several incidents with humans happened in the past [18], [19], [21], [22] which clearly shows that some of these incidents could have been prevented with proper testing. Proper testing of CPS is also challenging [1], and several studies have addressed already some aspects [9], [12], [27].
Nevertheless, testing CPS is still costly, especially in-field tests and even simulation-based tests. In-field tests require testing real, physical, and most likely also expensive hardware in a real-world environment that might damage the test subject.
To mitigate this issue, testing in simulation environments can prevent damaging the real test subject, but the computation of the simulation environment is more expensive than testing traditional software systems since complex computation re- garding the virtual environment must be performed [13], [15].
We adapt the concept of regression testing to the CPS domain for system-level testing in virtual environments. Our contribution is a cost-effective test selector SDC-Scissor [7] that covers the selection process in regression testing, whereas SDC-Prioritizer [8] orders the selected test cases based on a multi-objective genetic algorithm.
We aim to minimize the test scenarios to execute only the critical part of a scenario. With this approach, we try to minimize further the execution costs of tests in virtual environ- ments. The work on test case minimization will complement our central research vision:
Development of a regression testing framework for cyber-physical systems.
This paper will give a brief overview of the related work, methodology, preliminary results, and conclusion in Sec- tions II, III, IV and, V respectively.
II. RELATED WORK
Several studies about testing CPS efficiently based on model checking or simulations [10], [16], [23]. A more concrete ap- proach proposes Humeniuk et al. [14] with a search-based test generation. They applied a multi-objective search algorithm to have diversity and track the deviation of the observed system’s behavior from its expected behavior. A diverse test suite can be generated for testing the lane-keeping ability of SDCs in simulation.
First studies regarding regression testing for CPS have also been performed [2]–[6], [17]. They tackled the problem of test case selection and prioritization for different levels, e.g., MiL, SiL, HiL, etc. The use cases mainly were CPS with different configurations in product lines by applying different search algorithms.
In our vision, we want to have a complete regression testing framework for SDCs that includes test case selection, prior- itization, and minimization. The techniques use ML models and evolutionary genetic algorithms to minimize the costs in
33