Benchmarking is a cornerstone of empirical research in ML-driven computer science and its subfields, such as machine learning, computer vision, and optimization. Community-curated benchmarks are the gold standard for rigorous evaluations in AI subdomains. Competitions complement this effort by exposing the community to emerging, novel, and real-world challenges across modalities and domains. Benchmarks enable fair comparison of algorithms, measure scientific progress, pose ambitious challenges, and help align research communities around shared objectives. In short, they promote progress. Yet despite their central role, the foundations of benchmarking remain underexplored, fragmented across subfields, and often overshadowed by the push for novel algorithms and new datasets.

Large language models (LLMs), foundation models, and agentic systems emerge as new horizontal paradigms and are becoming critical components of state-of-the-art ML systems. Because these systems are pre-trained on vast amounts of data, including data that might overlap with benchmarks, their performance and leaderboard rankings must be interpreted with particular care. Furthermore, their broad generality, open-ended task space, and continual evolution contrast with traditional static benchmarking efforts, underscoring the need for scientifically grounded approaches to benchmarking.

Meta-science, also referred to as the science of benchmarking, treats benchmarking as a first-class scientific object rather than an auxiliary engineering task. It investigates how benchmark tasks should be designed, validated, and maintained to promote progress. While benchmarking has been discussed as a side aspect in many previous Dagstuhl Seminars and Dagstuhl Perspectives Workshops, this seminar will be the first Dagstuhl Seminar to focus entirely on the scientific foundations of benchmarking.

We aim to bring together researchers from core ML communities with experts from other areas of computer science that increasingly rely on ML solutions and discuss the spectrum from long-standing optimization testbeds to evolving benchmarks for foundation models and LLMs. A special focus will be put on how the different communities handle the evaluation of foundation models and LLMs.

The seminar will cover the following topics:

• Benchmark Design, Scope, and Validity: e.g., objectives beyond accuracy; dynamic benchmarks vs static benchmarks vs competitions; data leakage and pre-training contamination
• Empirical Evaluation: e.g., multiple comparisons and p-hacking; benchmark reliability and sensitivity; uncertainty and significance of leaderboard rankings; LLMs as evaluators
• Meta-Scientific and Philosophical Foundations: e.g., meta-science of benchmarking and evaluation; construct validity; philosophy of science; external validity to downstream utility

and will be structured around the following guiding questions:

1. Benchmarking as a cross-discipline practice. What are common foundations, goals, and challenges, and what are domain-specific differences?
2. Benchmarking with and for foundation models. Many models train on benchmarks, and many saturate quickly. How can we resolve this? Do we need to retire and adapt benchmarks, and do we need generative/dynamic benchmarks, or are there alternatives?
3. Leaderboard obsession. Many communities are obsessed with leaderboards and competitions. What would be alternative ways to measure and foster progress that still engage communities?

Creative Commons BY 4.0

Matthias Feurer, Katharina Eggensperger, Sanmi Koyejo, Joaquin Vanschoren, and Moritz Hardt

This seminar qualifies for Dagstuhl's a LZI Junior Researchers program. Schloss Dagstuhl wishes to enable the participation of junior scientists with a specialization fitting for this Dagstuhl Seminar, even if they are not on the radar of the organizers. Applications by outstanding junior scientists are possible until December 18, 2026.