Technology-Assisted Review (TAR) systems and automated systematic review (SR) pipelines are designed to achieve very high recall in domains where missing relevant material is unacceptable. These systems combine i) functional primitives such as ranking, filtering, and active learning, and ii) human-in-the-loop interaction, where experts guide and validate system decisions. With the rise of large-scale language models and autonomous research agents, TAR workflows are increasingly deployed in strategic fields like law, medicine, policy, and science.

TAR challenges “classical” evaluation methodology. In fact, traditional benchmarks and leaderboard-driven tasks capture only isolated components, while real TAR workflows are adaptive, iterative, and collaborative. This raises open questions around reproducibility, fairness, transparency, and the embedding of domain expertise. Importantly, these challenges are not domain-specific: experiences from law, medicine, and beyond illustrate the same higher-level methodological issues, without requiring deep expertise in any single field.

This Dagstuhl Seminar will focus on the expectations, promises, and limits of evaluation for TAR and SR systems. Relevant questions include:

• How can evaluation protocols capture dynamic, human-in-the-loop workflows?
• How do we make TAR systems transparent, accountable, and reproducible?
• What experimental designs best reflect real-world conditions across domains?
• How can we align evaluation initiatives toward shared goals?
• How should we teach responsible experimentation to the next generation of researchers?

To work on these and related questions, the seminar will bring together experts in information retrieval, recommender systems, machine learning, legal tech, and medical informatics, with academic, industrial, and non-profit backgrounds.

Creative Commons BY 4.0

Giorgio Maria Di Nunzio, Evangelos Kanoulas, Udo Kruschwitz, and Jeremy Pickens

This seminar qualifies for Dagstuhl's LZI Junior Researchers program. Schloss Dagstuhl wishes to enable the participation of junior scientists with a specialisation fitting for this Dagstuhl Seminar, even if they are not on the radar of the organizers. Applications by outstanding junior scientists are possible until January 16, 2026.