Dagstuhl Seminar 25391
Retrieval-Augmented Generation – The Future of Search?
( Sep 21 – Sep 26, 2025 )
(Click in the middle of the image to enlarge)
Permalink
Please use the following short url to reference this page:
https://www.dagstuhl.de/25391
Organizers
- Matthias Hagen (Friedrich-Schiller-Universität Jena, DE)
- Josiane Mothe (Toulouse University, FR)
- Smaranda Muresan (Barnard College, Columbia University - New York, US)
- Martin Potthast (Universität Kassel, DE)
- Min Zhang (Tsinghua University - Beijing, CN)
Contact
- Andreas Dolzmann (for scientific matters)
- Jutka Gasiorowski (for administrative matters)
Shared Documents
- Dagstuhl Materials Page (Use personal credentials as created in DOOR to log in)
Schedule
Background and Motivation
Retrieval-augmented generation (RAG) has proven effective in conditioning the output of large language models (LLMs) on relevant documents and for grounding LLM-generated statements, this way combatting the so-called hallucination or confabulation problem. Basically, RAG combines (1) a retrieval phase where a search system identifies relevant documents for a user prompt and (2) a generation phase, where an LLM synthesizes a tailored answer, probably linking to the retrieved sources.
RAG challenges “classical” retrieval technology and has the potential to revolutionize information-seeking behavior overall by reducing the searcher’s effort to extract the desired information from individual search results. The revolution becomes evident, among others, in a change in the design of a search engine results page (SERP): Instead of presenting the proverbial list of “ten blue links”, the list SERP, a generated text with references is shown, the text SERP. The first public prototypes of this kind were You.com’s You Chat and the now discontinued Neeva AI, followed by Microsoft’s Bing Chat, Google’s Bard, Perplexity.ai, and Baidu’s Ernie. However, many unsolved problems and relevant research questions still lurk under the hood (i.e., the user interface).
The proposed Dagstuhl Seminar focused on the expectations, the promises, the potential, and the limits of integrating RAG in information retrieval (IR). Relevant questions include:
- Will we ever search again?
- Will RAG bias retrieval results?
- Is RAG more than fact checking for conversational IR?
- How can we measure the effectiveness of RAG-based systems?
- How can we keep RAG-based systems transparent and accountable?
To work on these and related questions, the Dagstuhl Seminar brought together experts from the fields of information retrieval, natural language processing, and generative AI who have academic, non-profit (e.g., Open Search Foundation), or industrial backgrounds (e.g., Cohere).
Seminar Program
The 5-day program of the seminar consisted of five introductory and background sessions, one perspectives talk session, one industry talk session, and nine breakout discussion and reporting sessions. The program also had three social events and is available online.
Pre-Seminar Activities
Prior to the seminar, participants were asked to provide inputs to the following questions and request:
- Will RAG replace ranked search for end users?
- Please list, from the perspective of your research interests, important open questions or challenges in RAG.
- What are the three papers a PhD student in RAG should read and why?
The first question has been answered by 30 out of the thirty-nine participants. The answers were almost evenly distributed between “No” with 16 votes and “Yes” with 13 votes. In addition to asking this question, we asked participants to give a reason for their vote. These answers turned out to be nuanced, including various forms of hedging one way of the other. A list of the arguments made is provided in Section 5 of the full report.
From the survey, the following topics initially emerged as interests of participants. Many of these topics were discussed at length in the seminar.
- Can the methodologies that underpin RAG solve the enterprise search problems that sparse retrieval struggled to deal with?
- How can biases in the responses of RAG systems be detected?
- How to effectively inject external knowledge into LLMs?
- How to conduct RAG efficiently and dynamically?
- What is the analog of the Cranfield-style evaluation of ranked retrieval for RAG?
- What should the role and prominence of citations be in RAG outputs?
- How does RAG influence information behavior and how does it affect relevance feedback?
- How should search engines be designed that are used by RAG agents?
- What role does reasoning play in RAG?
- How can RAG systems be trained and how can they aid in training generative AIs?
- How can RAG systems notice and express uncertainty in their response?
Another outcome of the above pre-seminar questions was the compilation of a list of recommended reading to gain a solid understanding of topics and technologies related to retrieval-augmented generation. The reading list is provided in Section 6 of the full report.
Invited Talks
One of the main goals and challenges of this seminar was to bring a broad range of researchers together to discuss retrieval-augmented generation, which required to establish common terminologies among participants. Therefore, we had a series of 26 invited talks throughout the seminar program to facilitate the understanding and discussion of retrieval-augmented generation and its potential enabling technologies. Section 3 of the full report contains the abstracts of all talks.
Working Groups
In the afternoon of Day 2, initial working groups were formed based on the inputs to the pre-seminar questionnaires, introductory, and background talks, and discussions among participants. Eventually, the following five groups were formed:
- Potential & Limitations with Respect to Cognitive Tasks
- Information Behavior and Result Presentation in RAG
- Retrieval-Augmented Generation: The System's Perspective
- Societal and Ethical Motivations for Inverting RAG to GAR
- An Unexamined RAG Is Not Worth Interrogating
We have summarized the working groups' outcomes in the following. Please refer to the main part of the full report for the full description of the findings.
Potential & Limitations with Respect to Cognitive Tasks
The aim of this group was to provide an overview of the cognitive tasks in which a user can be supported by retrieval-augmented generation (RAG) systems. These cognitive tasks were organized in a taxonomy and linked to Bloom’s taxonomy. With Bloom’s taxonomy in mind, it became obvious that RAG is the superior technology in comparison to ranked search for information needs for which the solution requires synthesis (i.e., activates all layers in Bloom’s taxonomy). However, if a user outsources these tasks to RAG systems frequently, there are risks of cognitive decline for the user since cognitively demanding synthesis tasks are rarely performed manually. These risks were outlined by this group and scaffolding strategies to mitigate them were proposed.
Information Behavior and Result Presentation in RAG
and tailor the result presentation to the requirements of different users and tasks. They discussed how RAG systems allow for more natural interactions in multiple modalities and the formulation of complex queries that are contextualized by the rich history of previous interactions.
The group emphasized the need for adaptive, transparent, and inclusive RAG systems that accommodate diverse users and tasks – from factual retrieval to creative generation. They analyzed interaction dimensions such as user versus system initiative, information complexity, and human – machine collaboration. Their discussions resulted in open research questions around provenance tracking, adaptive presentation, and fostering user engagement while counteracting cognitive offloading.
Retrieval-Augmented Generation: The System’s Perspective
This group took a systems-level view of RAG, examining its architecture and contrasting today’s naive setup with an ideal RAG system. First, they highlighted that retrieval models were originally built for human users, informing evaluation methods and result formats that may not be suited for LLMs as users. Second, they emphasized the many sources of uncertainty in RAG, such as retrieval, model reasoning, and data quality, and the need for better ways to detect and communicate this uncertainty. Third, they explored efficiency – effectiveness trade-offs, arguing that future systems should dynamically allocate computation based on query complexity. Finally, the group discussed the advantages of federated RAG systems, such as unified access to open and proprietary sources, as well as challenges that this kind of architecture would face.
Societal and Ethical Motivations for Inverting RAG to GAR
This group advocated for a shift from retrieval-augmented generation (RAG) to Generation-Augmented Retrieval (GAR) to develop next-generation information access tools. This framing prioritizes information retrieval and the design of transparent, ethical, and sustainable information access systems that encourage active user engagement with information sources and diverse knowledge ecosystems. First, the group discussed the intersection of knowledge, ethics, and human rights, and its implications for RAG vs. GAR. Next, they offered perspectives on several (though not exhaustive) sociotechnical issues for GAR: scholarly communication, user cognition, emotional and mental well-being, democracy and political discourse, and language and culture. Each section references existing challenges related to the rapid and widespread use of generative AI to encourage critical and informed thinking about GAR development for individuals, organizations, and society as a whole. Throughout, the group proposed considerations for information retrieval researchers to create information access systems, learn from user studies, and foster interdisciplinary partnerships.
An Unexamined RAG Is Not Worth Interrogating
This group focused on how one can determine in which scenarios RAG systems work, where they fail, and how one can identify which RAG system is suitable for which scenario. The motivation of the group was that paradigm shifts in information access technology might also require paradigm shifts for corresponding evaluations. It therefore started by hypothesizing which evaluation properties might change between mature information retrieval evaluation methods and the methods applied in RAG systems. The group recognized that RAG evaluation is currently in an exploratory stage where many different evaluation ideas are being explored. The group identified research gaps and proposed directions for future research. Many parts of the discussion evolved around the idea of enabling in-depth analysis of RAG responses by multiple experts. This activity motivated the group to brainstorm how concepts from the Talmud can be transferred to RAG evaluations and labeling.
Conclusions
Leading researchers from diverse domains in academia and industry investigated the essence, attributes, architecture, applications, challenges, and opportunities of retrieval-augmented generation in the seminar. One clear signal from the seminar is that research opportunities to advance retrieval-augmented generation are available to many areas and collaboration in an interdisciplinary community is essential to achieve this goal. The full report should serve as one of the main sources to facilitate such diverse research programs on retrieval-augmented generation.
Creative Commons BY 4.0
Matthias Hagen, Josiane Mothe, Smaranda Muresan, Martin Potthast, Min Zhang, and Benno Stein
Retrieval-augmented generation (RAG) has proven effective in conditioning the output of large language models (LLMs) on relevant documents and for grounding LLM-generated statements, this way combatting the so-called confabulation or hallucination problem. Basically, RAG combines (1) a retrieval phase, where a search system identifies relevant documents for a user prompt, and (2) a generation phase, where an LLM synthesizes a tailored answer, probably linking to the retrieved sources.
RAG challenges “classical” retrieval technology and has the potential to revolutionize information-seeking behavior overall by reducing a searcher's effort to extract the desired information from individual search results. The revolution becomes evident, among others, in a change in the design of search engine results pages (SERPs): Instead of presenting the proverbial list of “ten blue links”, the classic list SERP, a generated text with references is shown, a text SERP. The first public prototype of this kind was You.com’s You Chat, followed by Microsoft’s Copilot, Google’s Gemini, Baidu’s Ernie, and many others. Still, plenty of unsolved problems and interesting research questions lurk under the hood of this new user interface.
This Dagstuhl Seminar will focus on the expectations, the promises, the potential, and the limits of integrating RAG in search systems. Relevant questions include
- Will we ever search again?
- Will RAG bias retrieval results?
- Is RAG more than fact checking for conversational search?
- How can we measure the effectiveness of RAG-based systems?
- How can we keep RAG-based systems transparent and accountable?
To work on these and related questions, the Dagstuhl Seminar will bring together experts from the fields of information retrieval, natural language processing, and generative AI who have academic, industrial, or non-profit backgrounds.
Matthias Hagen, Josiane Mothe, Smaranda Muresan, Martin Potthast, Benno Stein and Min Zhang
Please log in to DOOR to see more details.
- Qingyao Ai (Tsinghua University - Beijing, CN) [dblp]
- Mohammad Alian Nejadi (University of Amsterdam, NL) [dblp]
- Liesbeth Allein (KU Leuven, BE)
- Sophia Althammer (Cohere - München, DE) [dblp]
- Avishek Anand (TU Delft, NL) [dblp]
- Nolwenn Bernard (TH Köln, DE)
- Arjen P. de Vries (Radboud University Nijmegen, NL) [dblp]
- Niklas Deckers (Universität Kassel, DE) [dblp]
- Gianluca Demartini (University of Queensland - Brisbane, AU) [dblp]
- Laura Dietz (University of New Hampshire - Durham, US) [dblp]
- Carsten Eickhoff (Universität Tübingen, DE) [dblp]
- Nicola Ferro (University of Padova, IT) [dblp]
- Maik Fröbe (Friedrich-Schiller-Universität Jena, DE) [dblp]
- Norbert Fuhr (Universität Duisburg-Essen, DE) [dblp]
- Marcel Gohsen (Bauhaus-Universität Weimar, DE) [dblp]
- Michael Granitzer (Universität Passau, DE) [dblp]
- Faegheh Hasibi (Radboud University Nijmegen, NL) [dblp]
- Sebastian Heineking (Universität Leipzig, DE)
- Djoerd Hiemstra (Radboud University Nijmegen, NL) [dblp]
- Adam Jatowt (Universität Innsbruck, AT) [dblp]
- Abhinav Joshi (Indian Institute of Technology Kanpur, IN) [dblp]
- Johannes Kiesel (GESIS - Köln, DE) [dblp]
- Wojciech Kusa (NASK - Warsaw, PL)
- Sean MacAvaney (University of Glasgow, GB) [dblp]
- Bhaskar Mitra (Montréal, CA) [dblp]
- Josiane Mothe (Toulouse University, FR) [dblp]
- Smaranda Muresan (Barnard College, Columbia University - New York, US) [dblp]
- Jian-Yun Nie (University of Montreal, CA) [dblp]
- Heather O'Brien (University of British Columbia - Vancouver, CA) [dblp]
- Birte Platow (TU Dresden, DE)
- Martin Potthast (Universität Kassel, DE) [dblp]
- Mark Sanderson (RMIT University - Melbourne, AU) [dblp]
- Harrisen Scells (Universität Tübingen, DE) [dblp]
- Alan Smeaton (Dublin City University, IE) [dblp]
- Damiano Spina (RMIT University - Melbourne, AU) [dblp]
- Benno Stein (Bauhaus-Universität Weimar, DE) [dblp]
- Johanne Trippas (RMIT University - Melbourne, AU) [dblp]
- Stefan Voigt (Open Search Foundation - Starnberg, DE)
- Guido Zuccon (University of Queensland - Brisbane, AU) [dblp]
Classification
- Computation and Language
- Computers and Society
- Information Retrieval
Keywords
- Information Retrieval
- Retrieval-Augmented Generation
- Large Language Models
- Information Seeking Behavior
- Conversational Search
