Information access is concerned with searching, ranking, recommending, and accessing information resources with respect to user information needs, delivering a wide range of key applications for industry and society, such as web search engines, biomedical search, expert finding, intellectual property and patent search, multilingual information access, digital libraries, e-commerce, news recommendation, job and candidate search, just to name a few.

Information access relies on a strong experimental evaluation tradition, dating back to the mid 1950s, which has successfully driven the research and evolution of the field. However, nowadays, on the one side, information access systems are called to support increasingly challenging and diversified user tasks (e.g. voice interactions, item discovery, explanations) and, on the other side, the way in which we do research and experimentation has continued to evolve (e.g., better platforms, richer data, the fourth paradigm about data-intensive science, but also developments within the scientific culture).

After many decades of successful research and experimentation in the information access field, it is probably time to stop for a moment and think about what to do next and how to improve our methodologies and practices. Indeed, our current methods are showing their limits when it comes to reproducibility, validity, and generalizability of our experimental findings. Evaluation initiatives have followed more or less the same format for decades, but experimentation needs have evolved and move at a much faster pace today. Consider Kaggle-like competitions or the unprecedented use of leaderboards and the effect these may have on scientific inquiry.

It is critical that the next generation of scientists are equipped with a portfolio of evaluation methods that help ensure internal validity (the extent to which measures, statistical analyses, effect sizes, etc. support establishing a trustworthy causal relationship between treatments and outcomes), construct validity (measuring the right thing rather than a partial proxy), and external validity (critically assessing whether extent results hold in other situations, domains, and user groups).

The Dagstuhl Seminar will deal with how to develop responsible experimental practices for the information access fields (information retrieval, recommender systems, etc), for research and scientific education. We face two problems: how can the next generation of responsible information access systems transform and renovate the way in which research and experimentation are carried out in order to keep up with today's challenges? How can this new paradigm be leveraged to improve education and grow the new generation of researchers and developers? Therefore, in this seminar, we aim at addressing the following questions among others:

• How do we focus more on scientific insights than on raw performance numbers?
• Which experimentation methodologies should we further develop or create a culture around? Validity? Reproducibility? Generalizability?
• What are the FAT (Fairness, Accountability, and Transparency) concerns in experimentation?
• How can industry and academia work together to enable better experimentation?
• How do we share experimental data and research artifacts?
• How can we better teach experimental methods and develop critical experimentation skills?
• What is the role of evaluation initiatives such as TREC, CLEF, NTCIR, and FIRE for education?

Creative Commons BY 3.0 DE

Ben Carterette, Nicola Ferro, Norbert Fuhr, and Nava Tintarev