Autonomous systems are intended to operate without human intervention over prolonged time periods, perceive their operating environment, and adapt to changes – while pursuing defined goals or generating new ones. The perception functions process the inputs of various sensors and generate an internal model of the operating environment. By relying on this model, the decision functions plan and execute the actions required to achieve the goals of the mission.

To achieve safety for an autonomous system, the engineers should ensure that the perception functions can sufficiently accurately build the model of the environment, i.e., perception and establishing a context for prediction are reliable. They also seek to ensure that the planned actions are safe, i.e., decisions do not result in actions that endanger humans or other agents in the operating environment.

Both sensing and decision usually rely on Artificial Intelligence (AI), in particular Machine Learning (ML). The problem of safe AI has received a significant amount of research and industrial attention over the last few years, but there has been a divergence in the approaches taken by the safety and the ML communities. Moreover, it has become clear that the safety assurance problems cannot be resolved by improving the ML algorithms alone. Hence, the research communities should collaborate in creating methods and tools enabling a holistic approach to safety of autonomous systems. It is increasingly acknowledged that there needs to be work on ML methods, e.g. explainability to make algorithms transparent, predictable updates (learning without forgetting), and other areas. This should be complemented by a systems approach enabling safe autonomy through an integration of dedicated architectural, modelling, verification and validation as well as assurance methods.

Clearly, the engineering and assurance of safe autonomous systems require more fundamental research work that goes well beyond efforts of near-term industry deployment. In particular, we should address such open research problems as building a robust world model; creating resilient architectures enabling graceful degradation and fail-operational behavior; making safety assurances for high-consequence long-tail events; and establishing ways to measure and regulate safety for learning-enabled systems. To develop a holistic view on the safety of autonomous systems, we are planning to discuss, systematize and integrate these problems during our seminar.

This Dagstuhl Seminar aims at bringing together researchers and practitioners from safety engineering, systems and software engineering, modelling, verification and validation, machine learning, robotics, and autonomous systems to identify the state-of-the-art and key research and industrial challenges in engineering safe autonomous systems and defining the research roadmap for safe autonomy.

Creative Commons BY 4.0

Ignacio J. Alvarez, Philip Koopman, John McDermid, Mario Trapp, and Elena Troubitsyna