21. – 26. Februar 2021, Dagstuhl-Seminar 21082

CANCELLED Frontiers of Information Access Experimentation for Research and Education

Due to the Covid-19 pandemic, this seminar was cancelled. A related Dagstuhl-Seminar was scheduled to 15. – 20. Januar 2023 – Seminar 23031.


Ben Carterette (University of Delaware – Newark, US)
Nicola Ferro (University of Padova, IT)
Norbert Fuhr (Universität Duisburg-Essen, DE)
Nava Tintarev (TU Delft, NL)

Auskunft zu diesem Dagstuhl-Seminar erteilt

Dagstuhl Service Team


Information access is concerned with searching, ranking, recommending, and accessing information resources with respect to user information needs, delivering a wide range of key applications for industry and society, such as web search engines, biomedical search, expert finding, intellectual property and patent search, multilingual information access, digital libraries, e-commerce, news recommendation, job and candidate search, just to name a few.

Information access relies on a strong experimental evaluation tradition, dating back to the mid 1950s, which has successfully driven the research and evolution of the field. However, nowadays, on the one side, information access systems are called to support increasingly challenging and diversified user tasks (e.g. voice interactions, item discovery, explanations) and, on the other side, the way in which we do research and experimentation has continued to evolve (e.g., better platforms, richer data, the fourth paradigm about data-intensive science, but also developments within the scientific culture).

After many decades of successful research and experimentation in the information access field, it is probably time to stop for a moment and think about what to do next and how to improve our methodologies and practices. Indeed, our current methods are showing their limits when it comes to reproducibility, validity, and generalizability of our experimental findings. Evaluation initiatives have followed more or less the same format for decades, but experimentation needs have evolved and move at a much faster pace today. Consider Kaggle-like competitions or the unprecedented use of leaderboards and the effect these may have on scientific inquiry.

It is critical that the next generation of scientists are equipped with a portfolio of evaluation methods that help ensure internal validity (the extent to which measures, statistical analyses, effect sizes, etc. support establishing a trustworthy causal relationship between treatments and outcomes), construct validity (measuring the right thing rather than a partial proxy), and external validity (critically assessing whether extent results hold in other situations, domains, and user groups).

The Dagstuhl Seminar will deal with how to develop responsible experimental practices for the information access fields (information retrieval, recommender systems, etc), for research and scientific education. We face two problems: how can the next generation of responsible information access systems transform and renovate the way in which research and experimentation are carried out in order to keep up with today's challenges? How can this new paradigm be leveraged to improve education and grow the new generation of researchers and developers? Therefore, in this seminar, we aim at addressing the following questions among others:

  • How do we focus more on scientific insights than on raw performance numbers?
  • Which experimentation methodologies should we further develop or create a culture around? Validity? Reproducibility? Generalizability?
  • What are the FAT (Fairness, Accountability, and Transparency) concerns in experimentation?
  • How can industry and academia work together to enable better experimentation?
  • How do we share experimental data and research artifacts?
  • How can we better teach experimental methods and develop critical experimentation skills?
  • What is the role of evaluation initiatives such as TREC, CLEF, NTCIR, and FIRE for education?

Motivation text license
  Creative Commons BY 3.0 DE
  Ben Carterette, Nicola Ferro, Norbert Fuhr, and Nava Tintarev

Related Dagstuhl-Seminar


  • Information Retrieval
  • Other Computer Science


  • Information Access Systems
  • Experimentation
  • Methodologies
  • Education


In der Reihe Dagstuhl Reports werden alle Dagstuhl-Seminare und Dagstuhl-Perspektiven-Workshops dokumentiert. Die Organisatoren stellen zusammen mit dem Collector des Seminars einen Bericht zusammen, der die Beiträge der Autoren zusammenfasst und um eine Zusammenfassung ergänzt.


Download Übersichtsflyer (PDF).

Dagstuhl's Impact

Bitte informieren Sie uns, wenn eine Veröffentlichung ausgehend von Ihrem Seminar entsteht. Derartige Veröffentlichungen werden von uns in der Rubrik Dagstuhl's Impact separat aufgelistet  und im Erdgeschoss der Bibliothek präsentiert.


Es besteht weiterhin die Möglichkeit, eine umfassende Kollektion begutachteter Arbeiten in der Reihe Dagstuhl Follow-Ups zu publizieren.