TOP
Search the Dagstuhl Website
Looking for information on the websites of the individual seminars? - Then please:
Not found what you are looking for? - Some of our services have separate websites, each with its own search option. Please check the following list:
Schloss Dagstuhl - LZI - Logo
Schloss Dagstuhl Services
Seminars
Within this website:
External resources:
  • DOOR (for registering your stay at Dagstuhl)
  • DOSA (for proposing future Dagstuhl Seminars or Dagstuhl Perspectives Workshops)
Publishing
Within this website:
External resources:
dblp
Within this website:
External resources:
  • the dblp Computer Science Bibliography


Dagstuhl Seminar 21082

Frontiers of Information Access Experimentation for Research and Education Cancelled

( Feb 21 – Feb 26, 2021 )

Permalink
Please use the following short url to reference this page: https://www.dagstuhl.de/21082

Replacement
Dagstuhl Seminar 23031: Frontiers of Information Access Experimentation for Research and Education (2023-01-15 - 2023-01-20) (Details)

Organizers

Contact

Motivation

Information access is concerned with searching, ranking, recommending, and accessing information resources with respect to user information needs, delivering a wide range of key applications for industry and society, such as web search engines, biomedical search, expert finding, intellectual property and patent search, multilingual information access, digital libraries, e-commerce, news recommendation, job and candidate search, just to name a few.

Information access relies on a strong experimental evaluation tradition, dating back to the mid 1950s, which has successfully driven the research and evolution of the field. However, nowadays, on the one side, information access systems are called to support increasingly challenging and diversified user tasks (e.g. voice interactions, item discovery, explanations) and, on the other side, the way in which we do research and experimentation has continued to evolve (e.g., better platforms, richer data, the fourth paradigm about data-intensive science, but also developments within the scientific culture).

After many decades of successful research and experimentation in the information access field, it is probably time to stop for a moment and think about what to do next and how to improve our methodologies and practices. Indeed, our current methods are showing their limits when it comes to reproducibility, validity, and generalizability of our experimental findings. Evaluation initiatives have followed more or less the same format for decades, but experimentation needs have evolved and move at a much faster pace today. Consider Kaggle-like competitions or the unprecedented use of leaderboards and the effect these may have on scientific inquiry.

It is critical that the next generation of scientists are equipped with a portfolio of evaluation methods that help ensure internal validity (the extent to which measures, statistical analyses, effect sizes, etc. support establishing a trustworthy causal relationship between treatments and outcomes), construct validity (measuring the right thing rather than a partial proxy), and external validity (critically assessing whether extent results hold in other situations, domains, and user groups).

The Dagstuhl Seminar will deal with how to develop responsible experimental practices for the information access fields (information retrieval, recommender systems, etc), for research and scientific education. We face two problems: how can the next generation of responsible information access systems transform and renovate the way in which research and experimentation are carried out in order to keep up with today's challenges? How can this new paradigm be leveraged to improve education and grow the new generation of researchers and developers? Therefore, in this seminar, we aim at addressing the following questions among others:

  • How do we focus more on scientific insights than on raw performance numbers?
  • Which experimentation methodologies should we further develop or create a culture around? Validity? Reproducibility? Generalizability?
  • What are the FAT (Fairness, Accountability, and Transparency) concerns in experimentation?
  • How can industry and academia work together to enable better experimentation?
  • How do we share experimental data and research artifacts?
  • How can we better teach experimental methods and develop critical experimentation skills?
  • What is the role of evaluation initiatives such as TREC, CLEF, NTCIR, and FIRE for education?
Copyright Ben Carterette, Nicola Ferro, Norbert Fuhr, and Nava Tintarev

Participants
  • Ben Carterette (University of Delaware - Newark, US) [dblp]
  • Nicola Ferro (University of Padova, IT) [dblp]
  • Norbert Fuhr (Universität Duisburg-Essen, DE) [dblp]
  • Nava Tintarev (TU Delft, NL) [dblp]

Classification
  • Information Retrieval
  • Other Computer Science

Keywords
  • Information Access Systems
  • Experimentation
  • Methodologies
  • Education