Dagstuhl Seminar 22202
Anticipatory Human-Machine Interaction
( May 15 – May 20, 2022 )
(Click in the middle of the image to enlarge)
Permalink
Please use the following short url to reference this page:
https://www.dagstuhl.de/22202
Organizers
- Jelmer Borst (University of Groningen, NL)
- Andreas Bulling (Universität Stuttgart, DE)
- Cleotilde Gonzalez (Carnegie Mellon University - Pittsburgh, US)
- Nele Rußwinkel (TU Berlin, DE)
Contact
- Marsha Kleinbauer (for scientific matters)
- Simone Schilke (for administrative matters)
There is growing interest in interdisciplinary understanding of the anticipatory processes regarding other people's actions and also one's own actions. This surge in interest is especially important given that anticipatory interaction is needed in the growing area of intelligent systems that work closely together with human partners. Decades of research on human-machine interaction (HMI) have resulted in significant advances in theories, tools, and methods to facilitate, support, and enhance interactions between humans and computing systems. Despite the fundamental importance of HMI for our information society and numerous advances towards making interactions with machines more human-like, current systems still fall short in one core human ability - Theory of Mind (ToM). ToM allows us to attribute mental states to others and anticipate their actions; and is thus essential for us to interact naturally, effortlessly, and seamlessly.
ToM shapes how we interact with each other and is most easily observable in physical tasks, such as moving a table together. In this scenario, we rely on ToM abilities to attribute intentions to others and in turn, continuously adapt our own behaviour to accommodate the intentions of others, resulting in seamless collaboration. ToM begins to develop in early childhood. Even small children who are not able to develop the full cognitive ability of ToM can anticipate others' intentions using simple mental models of tasks and following familiar sequences of actions (e.g. Fiebich, 2018).
Deficits in ToM are closely linked to developmental disorders, such as autism, and current machines are similarly mind-blind. That is, they fail to sense users' attention and predict their intentions, and therefore lack the ability to anticipate and pro-actively adapt to users' actions. This limits machines to operating after the fact, i.e. to merely react to user input. This fundamentally limits the naturalness, efficiency, and user experience of current interactions. Imagine how difficult it would be to move a table with a robot: as the robot cannot anticipate easy and hard actions for the user (e.g., the table is heavy and needs to be rested for a moment, a corner has to be navigated for which the user has to switch her grip, etc.), it would not be a very helpful teammate. We believe anticipatory HMI has significant potential to bring systems that possess ToM to a new, exciting level of development. However, the building blocks required for anticipatory HMI are still at a very early research stage. This is in stark contrast to the large body of work on (Computational) Theory of Mind in the cognitive and neurosciences and the variety of potential applications in which artificial anticipatory behaviour could have a revolutionary and paradigm-changing influence.
With this seminar we have made a first step in bridging this gap and have discussed theoretical foundations, key research challenges and opportunities, new computational methods, as well as applications and use cases of anticipatory HMI. Research on anticipatory HMI is inherently interdisciplinary and draws from a number of fields, most notably human-computer interaction, machine learning, computer vision, computational neuroscience, computational cognitive science, privacy and security, as well as context-aware computing. Consequently, in this seminar, we have brought together junior and established researchers within these communities for the first time to explore the scientific foundations of anticipatory HMI.
Three perspectives on human-machine anticipation
In the seminar we have discussed three perspectives on human-machine anticipation.
Anticipating the intentions, beliefs, mental states and next actions of the user. This perspective focuses on the challenges, as well as possible solutions, for machine anticipation of user intentions, goals, motivations, and behaviors. This includes the underlying question of which mental states need to be modeled and anticipated in the first place. This might be different depending on the purpose of the collaboration or on the specific circumstances.
Anticipating the outcome of the machine's own actions. The ability of a machine to anticipate its own actions and outcomes is important, especially for interaction in teams and for learning purposes. Artificial agents need to have a concept of the outcome of their own actions, as that will naturally affect the collaboration with the human or other agents. In addition, in case the expected outcome of an action is not reached, they have to learn how they can achieve the expected outcome in an alternative manner.
Anticipating the outcome of collaborative work and human-machine teaming. The third perspective builds on the previous two. In human-machine teaming, a machine needs to anticipate the human partners' actions, identify problems, and develop ideas on how the partners in a team can be supported. We will discuss how artificial agents can engage in effective teamwork with humans and which specific abilities are required. We will also discuss whether we can construct artificial agents that behave indistinguishable from human agents – and whether this is even desirable.
Underlying concepts and methods
In addition to the three different perspectives on anticipatory behaviour, we have discussed the underlying concepts and computational methods.
On the conceptual side, we have discussed definitions and developed a common language. This included, for example, concepts such as cognitive state vs. mental states (which are not always concisely defined across disciplines). Furthermore, to support our cross-disciplined, multi-context approach, we have collected different forms and definitions of "anticipation''. A related but separate concept that we have discussed is what different disciplines understand by ToM and what related terms that should be differentiated?. We found that there are considerable differences of opinion on this that we formulated and exchanged.
On the methodological side, we have discussed the ways in which different domains and disciplines have their own methods, approaches, and challenges. Subsequently, groups have exchanged and discussed these collections to gain a deeper understanding of available methodological approaches. We have also talked about current approaches in modelling and cognitive science, and exchange ideas with those of us from other disciplines. Which approaches support the development of better applications? More general is the question how user behaviour can be measured. What data can or should be collected (CV, natural language, neurobiological measures, behavior,..), how can we process it most effectively, and how can we incorporate it into cognitive or computational models of the other users? Finally, we have discussed what the best way is to integrate and fuse different sources of information about the user, and how this can be integrated with the current context.
Creative Commons BY 4.0
Jelmer Borst, Andreas Bulling, Cleotilde Gonzalez, and Nele Rußwinkel
Even after three decades of research on human-machine interaction (HMI), current systems still lack the ability to predict mental states of their users, i.e., they fail to understand users' intentions, goals, and needs and therefore cannot anticipate their actions. This lack of anticipation drastically restricts their capabilities to interact and collaborate effectively with humans. The goal of this Dagstuhl Seminar is to discuss the scientific foundations of a new generation of human-machine systems that anticipate, and proactively adapt to, human actions by monitoring their attention, behavior, and predicting their mental states. Anticipation might be realized by using mental models of tasks, specific situations and systems to build up expectations about intentions, goals, and mental states that gathered evidence can be tested against.
The seminar will provide an inter-disciplinary forum to discuss this emerging topic by bringing together researchers from a range of fields, including human-computer interaction, cognitive-inspired AI, machine learning, computational cognitive science, and social and decision sciences. We will discuss theoretical foundations, key research challenges and opportunities, new computational methods, and future applications of anticipatory human-machine interaction from three perspectives:
1) Anticipating the intentions, beliefs, mental states and next actions of the user
This perspective focuses on the challenges, as well as possible solutions, for machine anticipation of user intentions, goals, motivations, and behaviors. This includes the underlying question of which mental states need to be modeled and anticipated in the first place. This might be different depending on the purpose of the collaboration or on the specific circumstances.
2) Anticipating the outcome of the machine’s own actions
The ability of a machine to anticipate its own actions and outcomes is important, especially for interaction in teams and for learning purposes. Artificial agents need to have a concept of the outcome of their own actions, as that will naturally affect the collaboration with the human or other agents. In addition, in case the expected outcome of an action is not reached, they have to learn how they can achieve the expected outcome in an alternative manner.
3) Anticipating the outcome of collaborative work and human-machine teaming
The third perspective builds on the previous two. In human-machine teaming, a machine needs to anticipate the human partners’ actions, identify problems, and develop ideas on how the partners in a team can be supported. We will discuss how artificial agents can engage in effective teamwork with humans and which specific abilities are required. We will also discuss whether we can construct artificial agents that behave indistinguishable from human agents - and whether this is even desirable.
As one key outcome of the seminar we plan to prepare a report that we aim to become a key reference for researchers interested in a multi-disciplinary approach to anticipatory HMI and, as such, to provide both an overview of the state of the art as well as a starting point into this emerging area of research.
Jelmer Borst, Andreas Bulling, Cleotilde Gonzalez, and Nele Rußwinkel
- Henny Admoni (Carnegie Mellon University - Pittsburgh, US) [dblp]
- Elisabeth André (Universität Augsburg, DE) [dblp]
- Mihai Bâce (Universität Stuttgart, DE)
- Leslie Blaha (AFRL - Wright Patterson, US) [dblp]
- Jelmer Borst (University of Groningen, NL)
- Matteo Bortoletto (Universität Stuttgart, DE)
- André Brechmann (Leibniz-Institut für Neurobiologie - Magdeburg, DE)
- Andreas Bulling (Universität Stuttgart, DE) [dblp]
- Cristina Conati (University of British Columbia - Vancouver, CA) [dblp]
- Emmanuelle Dietz (Airbus - Hamburg, DE)
- Steffen Frey (University of Groningen, NL) [dblp]
- Stella (Chenxu) Hao (Universität Erlangen-Nürnberg, DE)
- Susanne Hindennach (Universität Stuttgart, DE)
- Christian P. Janssen (Utrecht University, NL) [dblp]
- Samuel Kaski (Aalto University, FI & University of Manchester, GB) [dblp]
- Frank Keller (University of Edinburgh, GB) [dblp]
- Oliver Klaproth (Airbus - Hamburg, DE)
- Alvitta Ottley (Washington University - St. Louis, US) [dblp]
- Nele Rußwinkel (TU Berlin, DE) [dblp]
- Andreea Sburlea (University of Groningen, NL)
- Florian Strohm (Universität Stuttgart, DE)
- Niels A. Taatgen (University of Groningen, NL) [dblp]
Classification
- Artificial Intelligence
- Human-Computer Interaction
- Machine Learning
Keywords
- Computational Cognitive Modelling
- Computational Theory of Mind
- Human-Machine Teaming
- Cognition-aware Computing
- Goal and Plan Recognition
