February 5 – 10 , 2017, Dagstuhl Seminar 17062

Beyond VR and AR: Reimagining Experience Sharing and Skill Transfer Towards an Internet of Abilities


Anind K. Dey (Carnegie Mellon University – Pittsburgh, US)
Jonna Häkkilä (University of Lapland – Rovaniemi, FI)
Niels Henze (Universität Stuttgart, DE)
Kai Kunze (Keio University – Yokohama, JP)
Jun Rekimoto (University of Tokyo, JP)

For support, please contact

Dagstuhl Service Team


Dagstuhl Report, Volume 7, Issue 2 Dagstuhl Report
Aims & Scope
List of Participants
Dagstuhl's Impact: Documents available


Sharing experiences and knowledge have always been essential for human development. They enable skill transfers and empathy. Over history, mankind developed from oral traditions to cultures of writing. With the ongoing digital revolution, the hurdles to share knowledge and experiences vanish. Already today it is, for example, technically feasible to take and store 24/7 video recordings of one's life. While this example creates massive collections of data, it makes it even more challenging to share experiences and knowledge with others in meaningful ways. Facilitating the third wave of VR and AR technologies we are currently witnessing, researchers started to broadly look at VR and AR again. Topics ranging from using AR to mitigate skills gaps [1] and understanding user interaction with commercial AR games [4], to using focus depth as an input modality for VR [2], and understanding the effect of gender in VR [3]. The goal of the seminar was to take a step back from the technical research to look at the fundamental aspects of interactive media.

A recurring theme in science fiction literature is the act of downloading another human's abilities to one's mind. Although current cognitive science and neuroscience strongly suggest that this is impossible, as our minds are embodied; we believe that skill transfer and effective learning will accelerate tremendously given recent technological trends; just to name a few of the enabling technologies, human augmentation using virtual/augmented reality, new sensing modalities (e.g. affective computing) and actuation (e.g. haptics), advances in immersive storytelling (increasing empathy, immersion, communication) etc.

Ultimately, we believe this will lead to "downloadable" experiences and abilities. The effects will definitely not be instant and it will most likely be very different from the Sci-Fi theme. Yet, these differences are exactly what we want to explore in this seminar. Computer scientists in wearable computing, ubiquitous computing, human computer interaction, affective computing, virtual reality and augmented reality have been working on related topics and enabling technologies for years. However, these developments are disjointed from each other. With this seminar we want to bring them together working in the virtual/augmented/mixed reality, ubiquitous computing, sensing and HCI fields discussing also with experts in cognitive science, psychology and education.

While sharing experiences and knowledge through communication and socializing are a long time focus of various research efforts, we believe it is necessary to rethink and redefine experience sharing and skill transfer in light of the following current technological advances like the following:

  1. Affordable Virtual Reality and Augmented Reality systems will become available to consumers in the near future (or already are available).
  2. Advances in new sense sharing technologies (e.g. eye gaze, haptics, odors).
  3. Advances in real-life tracking of physical and cognitive activities and emotional states.
  4. Educators, cognitive scientists and psychologists have now a better understanding of individual and group behaviors, empathy and fundamentals of learning.

The seminar was structured around lightning talks by the participants, two hands-on workshops and three thematic sessions. In the lightning talks, the participants introduced themselves and shared their vision with the group. The first hands-on workshop by Shunichi Kasahara introduced the term Superception and showcased prototypes in this domain. The second workshop organized by Pedro Lopes enabled participants to experiment with electrical muscle stimulation by connecting off-the-shelf devices to embedded systems. Three days of the seminar started with thematic sessions run by one of the organizers. The sessions explored the future of human-computer symbiosis, human augmentation, and enabling technologies.


  1. Yamashita, S., Matsuda, A., Hamanishi, N., Suwa, S., & Rekimoto, J. (2017, March). Demulti Display: A Multiplayer Gaming Environment for Mitigating the Skills Gap. In Proceedings of the Tenth International Conference on Tangible, Embedded, and Embodied Interaction (pp. 457–463). ACM.
  2. Pai, Y. S., Outram, B., Vontin, N., & Kunze, K. (2016, October). Transparent Reality: Using Eye Gaze Focus Depth as Interaction Modality. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (pp. 171–172). ACM.
  3. Schwind, Valentin; Knierim, Pascal; Tasci, Cagri; Franczak, Patrick; Haas, Nico; Henze, Niels. “These are not my hands!”: Effect of Gender on the Perception of Avatar Hands in Virtual Reality Inproceedings CHI ’17 Proceedings of the 2017 Annual Symposium on Computer-Human Interaction, ACM Press, New York, NY, USA, 2017, ISBN: 9781450346559,
  4. Colley, Ashley, Jacob Thebault-Spieker, Allen Yilun Lin, Donald Degraen, Benjamin Fischman, Jonna Häkkilä, Kate Kuehl, Valentina Nisi, Nuno Jardim Nunes, Nina Wenig, Dirk Wenig, Brent Hecht and Johannes Schöning. 2017. The Geography of Pokémon GO: Beneficial and Problematic Effects on Places and Movement. In Proc. CHI’17.
Summary text license
  Creative Commons BY 3.0 Unported license
  Anind K. Dey, Jonna Häkkilä, Niels Henze, and Kai Kunze


  • Mobile Computing
  • Society / Human-computer Interaction


  • Augmented Human
  • HCI
  • Virtual Reality
  • Augmented Reality
  • Educational Psychology


In the series Dagstuhl Reports each Dagstuhl Seminar and Dagstuhl Perspectives Workshop is documented. The seminar organizers, in cooperation with the collector, prepare a report that includes contributions from the participants' talks together with a summary of the seminar.


Download overview leaflet (PDF).

Dagstuhl's Impact

Please inform us when a publication was published as a result from your seminar. These publications are listed in the category Dagstuhl's Impact and are presented on a special shelf on the ground floor of the library.


Furthermore, a comprehensive peer-reviewed collection of research papers can be published in the series Dagstuhl Follow-Ups.