Suche auf der Schloss Dagstuhl Webseite
Sie suchen nach Informationen auf den Webseiten der einzelnen Seminare? - Dann:
Nicht fündig geworden? - Einige unserer Dienste laufen auf separaten Webseiten mit jeweils eigener Suche. Bitte beachten Sie folgende Liste:
Schloss Dagstuhl - LZI - Logo
Schloss Dagstuhl Services
Innerhalb dieser Seite:
Externe Seiten:
  • DOOR (zum Registrieren eines Dagstuhl Aufenthaltes)
  • DOSA (zum Beantragen künftiger Dagstuhl Seminare oder Dagstuhl Perspektiven Workshops)
Innerhalb dieser Seite:
Externe Seiten:
Innerhalb dieser Seite:
Externe Seiten:
  • die Informatik-Bibliographiedatenbank dblp

Dagstuhl-Seminar 24052

Reviewer No. 2: Old and New Problems in Peer Review

( 28. Jan – 02. Feb, 2024 )

(zum Vergrößern in der Bildmitte klicken)

Bitte benutzen Sie folgende Kurz-Url zum Verlinken dieser Seite:



Dagstuhl Reports

As part of the mandatory documentation, participants are asked to submit their talk abstracts, working group results, etc. for publication in our series Dagstuhl Reports via the Dagstuhl Reports Submission System.

  • Upload (Use personal credentials as created in DOOR to log in)

Gemeinsame Dokumente



Peer review is the best mechanism for assessing scientific validity of new research that we have so far. But this mechanism has many well-known issues, such as the different incentives of the authors and reviewers, difficulties with preserving reviewer and author anonymity, confirmation and other cognitive biases that even researchers fall prey to. These intrinsic problems are exacerbated in interdisciplinary fields like Natural Language Processing (NLP) and Machine Learning (ML), where groups of researchers may vary so much in their methodology, terminology, and research agendas, that sometimes they have trouble even recognizing each other's contributions as "research".

This Dagstuhl Seminar will cover a range of topics related to organization of peer review in NLP and ML, including the following:

  • Improving the paper-reviewer matching by processes/algorithms that take into account both topic matches and reviewer interest in a given research question
  • Peer review vs methodological and demographic diversity in the interdisciplinary fields
  • Better practices for designing peer-review policies
  • Improving the structural incentives for reviewers
  • Use of NLP and ML for suitable automation of (parts of) the paper reviewing process
  • Peer-reviewing and research integrity

The seminar will serve as a point of reflection on decades of personal experience of the participants in organizing different kinds of peer-reviewed venues, enabling an in-depth discussion of what has been tried, what seems to work and what doesn't. It will also incorporate the fast-improving capabilities of NLP/ML systems. The outcomes of the seminar may include joint research publications on the methodological challenges of peer review, NLP and ML for intelligent support of peer-reviewing and actionable proposals, informed by the experience of participants as researchers as well as in various roles including chairs, editors, conference organizers, reviewers, and authors.

Copyright Iryna Gurevych, Anna Rogers, and Nihar B. Shah


  • Artificial Intelligence
  • Computation and Language
  • Machine Learning

  • peer review
  • diversity
  • natural language processing
  • incentives