Large Language Models (LLMs) are reshaping Natural Language Processing (NLP) and Human-Computer Interaction (HCI). Yet, the communities that build such systems and study them to improve their impact on people and society operate largely in isolation. NLP evaluation focuses on benchmark performance and automated metrics, often treating humans as data sources rather than active users. HCI centers on user experience and real-world use but treats the underlying NLP system as a black box. Researchers and end users alike hold incomplete mental models of how LLMs work, which shape how they prompt, interpret, and trust these systems. This gap between communities leads to systems that perform well on benchmarks but fail in real life, and to user studies that struggle with differentiating problems stemming from the model, the interface, or the interaction.

This Dagstuhl Seminar aims to bring together members from the NLP, HCI, and related communities to work and discuss three interdisciplinary challenges & research questions:

• How do NLP and HCI define and measure success in research, and where do these metrics fall short?
• How do both communities evaluate language technology, and what ethical issues arise?
• What collaboration strategies and evaluation approaches are needed to ensure responsible research in both fields?

Discussion & Collaboration

The seminar wants to build a shared vocabulary, identify evaluation methods that neither field has adopted on its own, and produce a joint agenda that connects computational metrics to human-centered outcomes. We hope to bring together experts in NLP and HCI to develop approaches for evaluating LLM systems that reflect their real-world impact. Participants are expected to take part in active group discussions and structured activities that allow them to exchange ideas freely.

Expected Outcomes

While there have been individual seminars on NLP and HCI at Dagstuhl before, to the best of our knowledge this seminar will be the first to bring experts from these two fields together physically. By bringing in open-minded scholars from both fields and building interdisciplinary bridges, our seminar will foster a better understanding across fields and new research collaborations (e.g., joint publications, scientific research grants). To increase the impact of our seminar beyond the discussions at Dagstuhl, we aim for tangible outcomes. These include (1) a synthesized framework of shared and conflicting terminology, research goals, and project success criteria, (2) a synthesized, visual mapping of current evaluation methodology of LLM systems (including multi-agent systems) and key gaps that can be the basis for potential position papers on evaluation challenges, ethical guidelines, and interdisciplinary best practices, learning resources, strategies, and (3) interdisciplinary ideas for projects and publications covering the fields of NLP and HCI.

Creative Commons BY 4.0

A. Seza Dogruöz, Michael A. Hedderich, Thomas Kosch, and Qian Yang

This seminar qualifies for Dagstuhl's LZI Junior Researchers program. Schloss Dagstuhl wishes to enable the participation of junior scientists with a specialization fitting for this Dagstuhl Seminar, even if they are not on the radar of the organizers. Applications by outstanding junior scientists are possible until November 6, 2026.