https://www.dagstuhl.de/23031

15. – 20. Januar 2023, Dagstuhl-Seminar 23031

Frontiers of Information Access Experimentation for Research and Education

Organisatoren

Christine Bauer (Utrecht University, NL)
Ben Carterette (University of Delaware – Newark, US)
Nicola Ferro (University of Padova, IT)
Norbert Fuhr (Universität Duisburg-Essen, DE)

Auskunft zu diesem Dagstuhl-Seminar erteilen

Susanne Bach-Bernhard zu administrativen Fragen

Michael Gerke zu wissenschaftlichen Fragen

Motivation

This Dagstuhl Seminar will address technology-enhanced information access (information retrieval, recommender systems, etc.) and specifically focus on developing more responsible experimental practices leading to more valid results, both for research as well as scientific education.

Information access has a long tradition of relying heavily on experimental evaluation dating back to the mid 1950s, this tradition has driven the research and evolution of the field. However, nowadays, research and development of information access systems are confronted with new challenges: information access systems are called to support a much wider set of user tasks (informational, educational, entertainment, just to name a few) which are increasingly challenging, and as a result, research settings and available opportunities have evolved substantially (e.g., better platforms, richer data, but also developments within the scientific culture) and shape the way in which we do research and experimentation. Consequently, it is critical that the next generation of scientists is equipped with a portfolio of evaluation methods that reflect the field’s challenges and opportunities, and help ensure internal validity (e.g., how many measures, statistical analyses, effect sizes, etc., to support establishing a trustworthy cause-effect relationship between treatments and outcomes), construct validity (e.g., measuring the right thing rather than a partial proxy), and external validity (e.g., critically assessing to which extent results hold in other situations, domains, and user groups). A robust portfolio of such methods will contribute to developing more responsible experimental practices.

Therefore, we face two problems: Can we re-innovate how we do research and experimentation in the field by addressing emerging challenges in experimental processes to develop the next generation of information access systems? How can a new paradigm of experimentation be leveraged to improve education to give an adequate basis to the new generation of researchers and developers?

To cope with these problems, in this Dagstuhl Seminar we aim at addressing specifically the following questions – among others:

  • Which experimentation methodologies are most promising to further develop and create a culture around?
  • In which ways can we consider the concerns related to FAccT (Fairness, Accountability, and Transparency) in the experimentation practices? How can we establish FaccT-E, i.e. FaccT in Experimentation?
  • How can industry and academia better work together on experimentation?
  • How can critical experimentation methods and skills be taught and developed in academic teaching?
  • How can we foster collaboration and run shared infrastructures enabling collaborative and joint experimentation? How to organize shared evaluation activities taking the opportunity of new hybrid forms of participation?

Motivation text license
  Creative Commons BY 4.0
  Christine Bauer, Ben Carterette, Nicola Ferro, and Norbert Fuhr

Classification

  • Artificial Intelligence
  • Information Retrieval

Keywords

  • Information Access Systems
  • Experimentation
  • Evaluation
  • User Interaction
  • Simulation

Dokumentation

In der Reihe Dagstuhl Reports werden alle Dagstuhl-Seminare und Dagstuhl-Perspektiven-Workshops dokumentiert. Die Organisatoren stellen zusammen mit dem Collector des Seminars einen Bericht zusammen, der die Beiträge der Autoren zusammenfasst und um eine Zusammenfassung ergänzt.

 

Download Übersichtsflyer (PDF).

Dagstuhl's Impact

Bitte informieren Sie uns, wenn eine Veröffentlichung ausgehend von Ihrem Seminar entsteht. Derartige Veröffentlichungen werden von uns in der Rubrik Dagstuhl's Impact separat aufgelistet  und im Erdgeschoss der Bibliothek präsentiert.

Publikationen

Es besteht weiterhin die Möglichkeit, eine umfassende Kollektion begutachteter Arbeiten in der Reihe Dagstuhl Follow-Ups zu publizieren.