The language barrier remains a significant impediment to seamless global communication and interaction, a challenge particularly pronounced within multilingual communities and institutions. Recent advancements in AI have led to the increasing availability and use of automatic speech translation tools. However, current tools still seem to fall short of fully supporting people to overcome language barriers in everyday life, showing a gap between improvements as measured by computer scientists and as perceived by end users. The rise of powerful multimodal and multilingual pre-trained models gives us an opportunity to focus research efforts on bridging this gap. For instance, what are the unique aspects of translation quality that matter when translating speech? How can we evaluate them manually and automatically? What are the communicative goals of users of speech translation systems? Can we design tools that can go beyond translation to answer questions, summarize information, or selectively translate content as required? How should speech and multimodal AI tools be integrated in communication scenarios? What are the interaction strategies that could empower users to use the full potential of speech and multimodal translation technology for communicating across language barriers?

This 3-day Dagstuhl Seminar wants to bring together an interdisciplinary group of researchers from speech translation, interpretation, and human-computer interactions to foster a new interdisciplinary community and redefine together how AI-mediated multilingual communication is studied, designed, and practiced. This will be informed by investigating the limitations of spoken language translation in real-life scenarios. We will also consider how insights from Interpretation Studies might help AI-based solutions to better support multilingual communication. We will draw from Human-Computer interaction expertise to identify new directions for how end users adopt, evaluate, and act upon translation outputs in alignment with their ecological goals.

During the workshop, each day will start with a plenary talk to set the stage, followed by focused breakout sessions for in-depth discussion, and plenary presentation and discussion of findings.

The plan is that Day 1 will focus on identifying a broad range of use cases that can serve as a foundation for rethinking how new technologies should support multilingual communication. Our aim is to move beyond siloed perspectives and simple attempts to mimic human interpreters. Instead, we will explore how advances in automatic translation can be meaningfully integrated with insights from interpreting practices and with the critical user experience requirements that shape multilingual settings. Day 2 is dedicated to the investigation of the potential positive and negative impacts of multilingual support on stakeholders, including opportunities and pain points, and identify critical technological gaps hindering effective multilingual support. Finally, Day 3 will be used to identify open research directions inspired by the use cases discussed and promising tasks, benchmarks, and evaluation frameworks to guide progress in these areas.

Creative Commons BY 4.0

Marine Carpuat, Claudio Fantinuoli, Ge Gao, and Jan Niehues

This seminar qualifies for Dagstuhl's LZI Junior Researchers program. Schloss Dagstuhl wishes to enable the participation of junior scientists with a specialisation fitting for this Dagstuhl Seminar, even if they are not on the radar of the organizers. Applications by outstanding junior scientists are possible until December 5, 2025.