Simulation-Based Metamorphic Testing of Cyber-Physical Systems

Abstract

Cyber-Physical Systems (CPSs) are heterogeneous systems in which software components and physical processes interact with each other. These systems have strict requirements that are difficult to verify, resulting in very costly verification processes which often lack automation. As CPSs and their requirements become more complex, development practices that still rely on manual intervention struggle to scale up. As such, there is a need to identify the fundamental problems that currently prevent the automation of those processes, and find solutions to them. One such problem is the inability to accurately define the expected behaviour of the systems for their verification, namely, the test oracle problem. Due to factors such as ambiguous requirements or non-determinism, testing activities for CPSs often involve human oracles, since automated oracles cannot be implemented without defining specific properties to verify. Metamorphic Testing (MT) is an alternative verification approach which can be used to define test oracles. Instead of focusing on a single system execution, MT defines properties that compare the behaviours from two or more system executions with known relationships between their inputs and outputs, the so-called Metamorphic Relations (MRs). This thesis aims to advance the current practice on testing CPSs by proposing methods that alleviate the test oracle problem by using MT. First, we present a general CPS DevOps architecture which will enable the automation of all the development tasks, focusing on the verification components where the MT techniques will be implemented. Second, we propose methods for identifying MRs for CPSs. On the one hand, we propose a pattern for performance-based MRs, which can be used to instantiate MRs for specific CPSs more easily. On the other hand, we implement a fully-automated approach for generating MRs for a given system based on samples of correct and incorrect system behaviours. We implement two prototypes of this technique. The first prototype enables automatic performance MR generation for CPSs, given user-provided test execution pairs. The second prototype demonstrates the full automation of a generic whole MR generation process. Finally, we propose an optimization approach for MT, metamorphic test selection. The goal of this approach is to maximize the cost-effectiveness of MT by reducing the number of tests that need to be executed, which is important due to the high cost of executing CPSs.