Dagstuhl Seminar 26382
Frontiers of Technology Assisted Review Systems
( Sep 13 – Sep 16, 2026 )
Permalink
Please use the following short url to reference this page:
https://www.dagstuhl.de/26382
Organizers
- Giorgio Maria Di Nunzio (University of Padova, IT)
- Evangelos Kanoulas (University of Amsterdam, NL)
- Udo Kruschwitz (Universität Regensburg, DE)
- Jeremy Pickens (Elevate - New York, US)
Contact
- Michael Gerke (for scientific matters)
- Susanne Bach-Bernhard (for administrative matters)
Dagstuhl Seminar Wiki
- Dagstuhl Seminar Wiki (Use personal credentials as created in DOOR to log in)
Shared Documents
- Dagstuhl Materials Page (Use personal credentials as created in DOOR to log in)
Schedule
- Upload (Use personal credentials as created in DOOR to log in)
Technology-Assisted Review (TAR) systems and automated systematic review (SR) pipelines are designed to achieve very high recall in domains where missing relevant material is unacceptable. These systems combine i) functional primitives such as ranking, filtering, and active learning, and ii) human-in-the-loop interaction, where experts guide and validate system decisions. With the rise of large-scale language models and autonomous research agents, TAR workflows are increasingly deployed in strategic fields like law, medicine, policy, and science.
TAR challenges “classical” evaluation methodology. In fact, traditional benchmarks and leaderboard-driven tasks capture only isolated components, while real TAR workflows are adaptive, iterative, and collaborative. This raises open questions around reproducibility, fairness, transparency, and the embedding of domain expertise. Importantly, these challenges are not domain-specific: experiences from law, medicine, and beyond illustrate the same higher-level methodological issues, without requiring deep expertise in any single field.
This Dagstuhl Seminar will focus on the expectations, promises, and limits of evaluation for TAR and SR systems. Relevant questions include:
- How can evaluation protocols capture dynamic, human-in-the-loop workflows?
- How do we make TAR systems transparent, accountable, and reproducible?
- What experimental designs best reflect real-world conditions across domains?
- How can we align evaluation initiatives toward shared goals?
- How should we teach responsible experimentation to the next generation of researchers?
To work on these and related questions, the seminar will bring together experts in information retrieval, recommender systems, machine learning, legal tech, and medical informatics, with academic, industrial, and non-profit backgrounds.
Creative Commons BY 4.0
Giorgio Maria Di Nunzio, Evangelos Kanoulas, Udo Kruschwitz, and Jeremy Pickens
This seminar qualifies for Dagstuhl's LZI Junior Researchers program. Schloss Dagstuhl wishes to enable the participation of junior scientists with a specialisation fitting for this Dagstuhl Seminar, even if they are not on the radar of the organizers. Applications by outstanding junior scientists are possible until July 24, 2026.
Please log in to DOOR to see more details.
- Gordon Cormack
- Hanna Denecke
- Giorgio Maria Di Nunzio
- Laura Dietz
- Lenora Gray
- Maura Grossman
- Allan Hanbury
- Yiran Hu
- Evangelos Kanoulas
- Udo Kruschwitz
- Wojciech Kusa
- David Lewis
- Malcolm MacLeod
- Jeremy Pickens
- Adam Roegiest
- Tony Russell-Rose
- Harrisen Scells
- Veronika Solopova
- Rene Spijker
- Benno Stein
- Mark Stevenson
- Tanja Svarre
- James Thomas
- Henning Wachsmuth
- Eugene Yang
Classification
- Databases
- Information Retrieval
- Machine Learning
Keywords
- Technology Assisted Review Systems
- Systematic Reviews
- eDiscovery
- Content Moderation
- Information Access Systems
