https://www.dagstuhl.de/22232

June 6 – 10 , 2022, Dagstuhl Seminar 22232

Efficient and Equitable Natural Language Processing in the Age of Deep Learning

Organizers

Jesse Dodge (AI2 – Seattle, US)
Iryna Gurevych (TU Darmstadt, DE)
Roy Schwartz (The Hebrew University of Jerusalem, IL)
Emma Strubell (Carnegie Mellon University – Pittsburgh, US)

For support, please contact

Christina Schwarz for administrative matters

Marsha Kleinbauer for scientific matters

Dagstuhl Reports

As part of the mandatory documentation, participants are asked to submit their talk abstracts, working group results, etc. for publication in our series Dagstuhl Reports via the Dagstuhl Reports Submission System.

Documents

List of Participants
Shared Documents
Dagstuhl Seminar Wiki
Dagstuhl Seminar Schedule [pdf]

(Use personal credentials as created in DOOR to log in)

Motivation

Since 2012, the field of artificial intelligence (AI) has reported remarkable progress on a broad range of capabilities including object recognition, game playing, speech recognition, and machine translation. Much of this progress has been achieved by increasingly large and computationally intensive deep learning models: training costs for state-of-the-art deep learning models have increased 300,000 times between 2012 and 2018 [1]. Perhaps the epitome of this trend is the subfield of natural language processing (NLP) that over the past three years has experienced even sharper growth in model size and corresponding computational requirements in the word embedding approaches (e.g. ELMo, BERT, openGPT-2, Megatron-LM, T5, and GPT-3, one of the largest models ever trained with 175B dense parameters) that are now the basic building blocks of nearly all NLP models. Recent studies indicate that this trend is both environmentally unfriendly and prohibitively expensive, raising barriers to participation in NLP research [2, 3]. The goal of this seminar is to mitigate these concerns and promote equity of access in NLP. We plan to bring together a diverse group of researchers and practitioners in NLP and adjacent fields, and develop actionable policies, incentives and a joint strategy towards these goals.

This Dagstuhl Seminar will cover a range of topics relating to efficiency in NLP. A key method for mitigating the concerns raised above is reducing costs by making models more efficient. We will survey the different methods that exist for making NLP technology more efficient. We will discuss their tradeoffs, prioritize them, and aim to identify new opportunities to promote efficiency in NLP.

We will also address systemic issues in the field relating to the reporting of computational budgets in NLP research, and how we can use incentive structures such as the NLP Reproducibility Checklist to motivate researchers throughout the field to improve reporting. We have privileged access to the survey responses for the reproducibility checklist used at four major NLP conferences, and we will release a report of this data, in addition to planning the future direction of the checklist.

A third topic of discussion will be equity of access to computational resources and state-of-the-art NLP technologies. Prior to the seminar we will conduct a survey of different stakeholders across the NLP community. During the seminar we will analyze and discuss the results of this survey to better understand who is most affected and how, and develop informed strategies and policies to mitigate this inequity moving forward.

All of the above endeavors require establishing the right metrics and standards to measure our current status and progress towards efficiency and equity goals. We will devise metrics and evaluation frameworks that capture the bigger picture of how different approaches compare in terms of energy efficiency not just in the research environment but in practice and over the entire ML model lifecycle (development, training and deployment), and that work under a wide range of computational budgets.

The result of this seminar may include joint research publications, e.g., towards the efficiency of NLP models; conceptualization of an efficiency benchmark and the corresponding evaluation; and actionable policies informed by analysis of survey data.

[1] D. Amodei and D. Hernandez. 2018. AI and Compute. https://openai.com/blog/ai-and-compute/ [2] R. Schwartz, D. Dodge, N. A. Smith, and O. Etzioni. 2020. Green AI. Communications of the ACM (CACM). [3] E. Strubell, A. Ganesh, and A. McCallum. 2019. Energy and Policy Considerations for Deep Learning in NLP. In Proc. of ACL.

Motivation text license
  Creative Commons BY 4.0
  Jesse Dodge, Iryna Gurevych, Roy Schwartz, and Emma Strubell

Classification

  • Artificial Intelligence
  • Computation And Language
  • Machine Learning

Keywords

  • Natural language processing (NLP)
  • Efficiency
  • Equity
  • Deep learning

Documentation

In the series Dagstuhl Reports each Dagstuhl Seminar and Dagstuhl Perspectives Workshop is documented. The seminar organizers, in cooperation with the collector, prepare a report that includes contributions from the participants' talks together with a summary of the seminar.

 

Download overview leaflet (PDF).

Dagstuhl's Impact

Please inform us when a publication was published as a result from your seminar. These publications are listed in the category Dagstuhl's Impact and are presented on a special shelf on the ground floor of the library.

Publications

Furthermore, a comprehensive peer-reviewed collection of research papers can be published in the series Dagstuhl Follow-Ups.