Dagstuhl Seminar 24052
Reviewer No. 2: Old and New Problems in Peer Review
( Jan 28 – Feb 02, 2024 )
(Click in the middle of the image to enlarge)
Permalink
Please use the following short url to reference this page:
https://www.dagstuhl.de/24052
Organizers
- Iryna Gurevych (TU Darmstadt, DE)
- Anna Rogers (IT University of Copenhagen, DK)
- Nihar Shah (Carnegie Mellon University - Pittsburgh, US)
Contact
- Christopher Michels (for scientific matters)
- Christina Schwarz (for administrative matters)
Dagstuhl Reports
As part of the mandatory documentation, participants are asked to submit their talk abstracts, working group results, etc. for publication in our series Dagstuhl Reports via the Dagstuhl Reports Submission System.
- Upload (Use personal credentials as created in DOOR to log in)
Shared Documents
- Dagstuhl Materials Page (Use personal credentials as created in DOOR to log in)
Peer review is the best mechanism for assessing scientific validity of new research that we have so far. But this mechanism has many well-known issues, such as the different incentives of the authors and reviewers, difficulties with preserving reviewer and author anonymity, confirmation and other cognitive biases that even researchers fall prey to. These intrinsic problems are exacerbated in interdisciplinary fields like Natural Language Processing (NLP) and Machine Learning (ML), where groups of researchers may vary so much in their methodology, terminology, and research agendas, that sometimes they have trouble even recognizing each other's contributions as "research".
This Dagstuhl Seminar will cover a range of topics related to organization of peer review in NLP and ML, including the following:
- Improving the paper-reviewer matching by processes/algorithms that take into account both topic matches and reviewer interest in a given research question
- Peer review vs methodological and demographic diversity in the interdisciplinary fields
- Better practices for designing peer-review policies
- Improving the structural incentives for reviewers
- Use of NLP and ML for suitable automation of (parts of) the paper reviewing process
- Peer-reviewing and research integrity
The seminar will serve as a point of reflection on decades of personal experience of the participants in organizing different kinds of peer-reviewed venues, enabling an in-depth discussion of what has been tried, what seems to work and what doesn't. It will also incorporate the fast-improving capabilities of NLP/ML systems. The outcomes of the seminar may include joint research publications on the methodological challenges of peer review, NLP and ML for intelligent support of peer-reviewing and actionable proposals, informed by the experience of participants as researchers as well as in various roles including chairs, editors, conference organizers, reviewers, and authors.
Creative Commons BY 4.0
Iryna Gurevych, Anna Rogers, and Nihar B. Shah
- Osama Mohammed Afzal (MBZUAI - Abu Dhabi, AE)
- Koen Dercksen (Radboud University Nijmegen, NL)
- Nils Dycke (TU Darmstadt, DE)
- Alexander Goldberg (Carnegie Mellon University - Pittsburgh, US)
- Iryna Gurevych (TU Darmstadt, DE) [dblp]
- Jason Hartline (Northwestern University - Evanston, US) [dblp]
- Tom Hope (The Hebrew University of Jerusalem, IL)
- Dirk Hovy (Bocconi University - Milan, IT) [dblp]
- Eddie Kohler (Harvard University - Allston, US)
- Jonathan Kummerfeld (The University of Sydney, AU)
- Ilia Kuznetsov (TU Darmstadt, DE) [dblp]
- Anne Lauscher (Universität Hamburg, DE)
- Kevin Leyton-Brown (University of British Columbia - Vancouver, CA) [dblp]
- Sheng Lu (TU Darmstadt, DE)
- Dorsa Majdi (Sharif University of Technology - Tehran, IR)
- Mausam (Indian Institute of Technology - New Delhi, IN)
- Bahar Mehmani (Elsevier BV - Amsterdam, NL)
- Margot Mieskes (Hochschule Darmstadt, DE) [dblp]
- Aurélie Névéol (CNRS - Orsay, FR)
- Danish Pruthi (Indian Institute of Science - Bangalore, IN)
- Lizhen Qu (Monash University - Clayton, AU)
- Anna Rogers (IT University of Copenhagen, DK)
- Roy Schwartz (The Hebrew University of Jerusalem, IL)
- Nihar Shah (Carnegie Mellon University - Pittsburgh, US)
- Noah A. Smith (University of Washington - Seattle, US)
- Thamar Solorio (MBZUAI - Abu Dhabi, AE)
- Jingyan Wang (Georgia Institute of Technology, US)
- Xiaodan Zhu (Queen's University - Kingston, CA)
Classification
- Artificial Intelligence
- Computation and Language
- Machine Learning
Keywords
- peer review
- diversity
- natural language processing
- incentives
