https://www.dagstuhl.de/21082

21. – 26. Februar 2021, Dagstuhl-Seminar 21082

Frontiers of Information Access Experimentation for Research and Education

Organisatoren

Ben Carterette (University of Delaware – Newark, US)
Nicola Ferro (University of Padova, IT)
Norbert Fuhr (Universität Duisburg-Essen, DE)
Nava Tintarev (TU Delft, NL)

Auskunft zu diesem Dagstuhl-Seminar erteilen

Annette Beyer zu administrativen Fragen

Andreas Dolzmann zu wissenschaftlichen Fragen

Dokumente

Programm des Dagstuhl-Seminars (Hochladen)

(Zum Einloggen bitte persönliche DOOR-Zugangsdaten verwenden)

Motivation

Information access is concerned with searching, ranking, recommending, and accessing information resources with respect to user information needs, delivering a wide range of key applications for industry and society, such as web search engines, biomedical search, expert finding, intellectual property and patent search, multilingual information access, digital libraries, e-commerce, news recommendation, job and candidate search, just to name a few.

Information access relies on a strong experimental evaluation tradition, dating back to the mid 1950s, which has successfully driven the research and evolution of the field. However, nowadays, on the one side, information access systems are called to support increasingly challenging and diversified user tasks (e.g. voice interactions, item discovery, explanations) and, on the other side, the way in which we do research and experimentation has continued to evolve (e.g., better platforms, richer data, the fourth paradigm about data-intensive science, but also developments within the scientific culture).

After many decades of successful research and experimentation in the information access field, it is probably time to stop for a moment and think about what to do next and how to improve our methodologies and practices. Indeed, our current methods are showing their limits when it comes to reproducibility, validity, and generalizability of our experimental findings. Evaluation initiatives have followed more or less the same format for decades, but experimentation needs have evolved and move at a much faster pace today. Consider Kaggle-like competitions or the unprecedented use of leaderboards and the effect these may have on scientific inquiry.

It is critical that the next generation of scientists are equipped with a portfolio of evaluation methods that help ensure internal validity (the extent to which measures, statistical analyses, effect sizes, etc. support establishing a trustworthy causal relationship between treatments and outcomes), construct validity (measuring the right thing rather than a partial proxy), and external validity (critically assessing whether extent results hold in other situations, domains, and user groups).

The Dagstuhl Seminar will deal with how to develop responsible experimental practices for the information access fields (information retrieval, recommender systems, etc), for research and scientific education. We face two problems: how can the next generation of responsible information access systems transform and renovate the way in which research and experimentation are carried out in order to keep up with today's challenges? How can this new paradigm be leveraged to improve education and grow the new generation of researchers and developers? Therefore, in this seminar, we aim at addressing the following questions among others:

  • How do we focus more on scientific insights than on raw performance numbers?
  • Which experimentation methodologies should we further develop or create a culture around? Validity? Reproducibility? Generalizability?
  • What are the FAT (Fairness, Accountability, and Transparency) concerns in experimentation?
  • How can industry and academia work together to enable better experimentation?
  • How do we share experimental data and research artifacts?
  • How can we better teach experimental methods and develop critical experimentation skills?
  • What is the role of evaluation initiatives such as TREC, CLEF, NTCIR, and FIRE for education?

Motivation text license
  Creative Commons BY 3.0 DE
  Ben Carterette, Nicola Ferro, Norbert Fuhr, and Nava Tintarev

Classification

  • Information Retrieval
  • Other Computer Science

Keywords

  • Information Access Systems
  • Experimentation
  • Methodologies
  • Education

Dokumentation

In der Reihe Dagstuhl Reports werden alle Dagstuhl-Seminare und Dagstuhl-Perspektiven-Workshops dokumentiert. Die Organisatoren stellen zusammen mit dem Collector des Seminars einen Bericht zusammen, der die Beiträge der Autoren zusammenfasst und um eine Zusammenfassung ergänzt.

 

Download Übersichtsflyer (PDF).

Publikationen

Es besteht weiterhin die Möglichkeit, eine umfassende Kollektion begutachteter Arbeiten in der Reihe Dagstuhl Follow-Ups zu publizieren.

Dagstuhl's Impact

Bitte informieren Sie uns, wenn eine Veröffentlichung ausgehend von
Ihrem Seminar entsteht. Derartige Veröffentlichungen werden von uns in der Rubrik Dagstuhl's Impact separat aufgelistet  und im Erdgeschoss der Bibliothek präsentiert.