Peer review is the best mechanism for assessing scientific validity of new research that we have so far. But this mechanism has many well-known issues, such as the different incentives of the authors and reviewers, difficulties with preserving reviewer and author anonymity, confirmation and other cognitive biases that even researchers fall prey to. These intrinsic problems are exacerbated in interdisciplinary fields like Natural Language Processing (NLP) and Machine Learning (ML), where groups of researchers may vary so much in their methodology, terminology, and research agendas, that sometimes they have trouble even recognizing each other's contributions as "research".

This Dagstuhl Seminar will cover a range of topics related to organization of peer review in NLP and ML, including the following:

• Improving the paper-reviewer matching by processes/algorithms that take into account both topic matches and reviewer interest in a given research question
• Peer review vs methodological and demographic diversity in the interdisciplinary fields
• Better practices for designing peer-review policies
• Improving the structural incentives for reviewers
• Use of NLP and ML for suitable automation of (parts of) the paper reviewing process
• Peer-reviewing and research integrity

The seminar will serve as a point of reflection on decades of personal experience of the participants in organizing different kinds of peer-reviewed venues, enabling an in-depth discussion of what has been tried, what seems to work and what doesn't. It will also incorporate the fast-improving capabilities of NLP/ML systems. The outcomes of the seminar may include joint research publications on the methodological challenges of peer review, NLP and ML for intelligent support of peer-reviewing and actionable proposals, informed by the experience of participants as researchers as well as in various roles including chairs, editors, conference organizers, reviewers, and authors.

Creative Commons BY 4.0

Iryna Gurevych, Anna Rogers, and Nihar B. Shah