February 21 – 26 , 2021, Dagstuhl Seminar 21082

CANCELLED Frontiers of Information Access Experimentation for Research and Education

Due to the Covid-19 pandemic, this seminar was cancelled. A related Dagstuhl Seminar was scheduled to January 15 – 20 , 2023 – Seminar 23031.


Ben Carterette (University of Delaware – Newark, US)
Nicola Ferro (University of Padova, IT)
Norbert Fuhr (Universität Duisburg-Essen, DE)
Nava Tintarev (TU Delft, NL)

For support, please contact

Dagstuhl Service Team


Information access is concerned with searching, ranking, recommending, and accessing information resources with respect to user information needs, delivering a wide range of key applications for industry and society, such as web search engines, biomedical search, expert finding, intellectual property and patent search, multilingual information access, digital libraries, e-commerce, news recommendation, job and candidate search, just to name a few.

Information access relies on a strong experimental evaluation tradition, dating back to the mid 1950s, which has successfully driven the research and evolution of the field. However, nowadays, on the one side, information access systems are called to support increasingly challenging and diversified user tasks (e.g. voice interactions, item discovery, explanations) and, on the other side, the way in which we do research and experimentation has continued to evolve (e.g., better platforms, richer data, the fourth paradigm about data-intensive science, but also developments within the scientific culture).

After many decades of successful research and experimentation in the information access field, it is probably time to stop for a moment and think about what to do next and how to improve our methodologies and practices. Indeed, our current methods are showing their limits when it comes to reproducibility, validity, and generalizability of our experimental findings. Evaluation initiatives have followed more or less the same format for decades, but experimentation needs have evolved and move at a much faster pace today. Consider Kaggle-like competitions or the unprecedented use of leaderboards and the effect these may have on scientific inquiry.

It is critical that the next generation of scientists are equipped with a portfolio of evaluation methods that help ensure internal validity (the extent to which measures, statistical analyses, effect sizes, etc. support establishing a trustworthy causal relationship between treatments and outcomes), construct validity (measuring the right thing rather than a partial proxy), and external validity (critically assessing whether extent results hold in other situations, domains, and user groups).

The Dagstuhl Seminar will deal with how to develop responsible experimental practices for the information access fields (information retrieval, recommender systems, etc), for research and scientific education. We face two problems: how can the next generation of responsible information access systems transform and renovate the way in which research and experimentation are carried out in order to keep up with today's challenges? How can this new paradigm be leveraged to improve education and grow the new generation of researchers and developers? Therefore, in this seminar, we aim at addressing the following questions among others:

  • How do we focus more on scientific insights than on raw performance numbers?
  • Which experimentation methodologies should we further develop or create a culture around? Validity? Reproducibility? Generalizability?
  • What are the FAT (Fairness, Accountability, and Transparency) concerns in experimentation?
  • How can industry and academia work together to enable better experimentation?
  • How do we share experimental data and research artifacts?
  • How can we better teach experimental methods and develop critical experimentation skills?
  • What is the role of evaluation initiatives such as TREC, CLEF, NTCIR, and FIRE for education?

Motivation text license
  Creative Commons BY 3.0 DE
  Ben Carterette, Nicola Ferro, Norbert Fuhr, and Nava Tintarev

Related Dagstuhl Seminar


  • Information Retrieval
  • Other Computer Science


  • Information Access Systems
  • Experimentation
  • Methodologies
  • Education


In the series Dagstuhl Reports each Dagstuhl Seminar and Dagstuhl Perspectives Workshop is documented. The seminar organizers, in cooperation with the collector, prepare a report that includes contributions from the participants' talks together with a summary of the seminar.


Download overview leaflet (PDF).

Dagstuhl's Impact

Please inform us when a publication was published as a result from your seminar. These publications are listed in the category Dagstuhl's Impact and are presented on a special shelf on the ground floor of the library.


Furthermore, a comprehensive peer-reviewed collection of research papers can be published in the series Dagstuhl Follow-Ups.