https://www.dagstuhl.de/23122

19. – 24. März 2023, Dagstuhl-Seminar 23122

Deep Continual Learning

Organisatoren

Bing Liu (University of Illinois – Chicago, US)
Vincenzo Lomonaco (University of Pisa, IT)
Tinne Tuytelaars (KU Leuven, BE)
Gido van de Ven (Baylor College of Medicine – Houston, US)

Auskunft zu diesem Dagstuhl-Seminar erteilen

Jutka Gasiorowski zu administrativen Fragen

Marsha Kleinbauer zu wissenschaftlichen Fragen

Motivation

Continual learning, also referred to as incremental learning or lifelong learning, is a sub-field of machine learning focusing on the challenging setting where data distributions and/or task specifications vary over time. This includes learning a sequence of tasks as well as learning from data streams. This calls for learning algorithms that can acquire new knowledge over time, with minimal forgetting of what they have learned previously, transfer knowledge across tasks, and smoothly adapt to new circumstances as needed. This contrasts with the traditional setting of machine learning, which largely builds on the premise that all data, both for training and testing, are sampled i.i.d. from a single, stationary data distribution.

Deep learning models in particular are in need of continual learning capabilities. A first reason for this is the strong data-dependence of these models. When trained on a stream of data whose underlying distribution changes over time, deep learning models tend to fully adapt to the most recently seen data, thereby "catastrophically" forgetting the skills they had learned earlier in their training process. Another reason that continual learning capabilities could be especially beneficial for deep learning models, is that they can help deal with the very long training times of these models. The current practice applied in industry, where models are completely re-trained on a regular basis to avoid being outdated, is time inefficient, unsustainable and sub-optimal. Freezing the feature extraction layers is not an option, as the power of deep learning in many challenging applications, be it in computer vision, NLP or audio processing, hinges on the learned representations.

Open research questions we would like to address in this Dagstuhl Seminar include:

  • How do we tackle continual learning at scale on real-world problems, where domain shifts may be unpredictable and data can be long-tailed?
  • To what extent can recent advances in representation learning and insights in model generalisability help continual learning?
  • Rather than relying on tools developed and optimized for machine learning under i.i.d. conditions, should we consider completely different learning strategies?
  • In case old data can be revisited, are there more efficient strategies than simply retraining on all available data over and over again?
  • What are the open challenges in open world learning and automated continual learning, where the agent discovers new tasks by itself, collects its own training data and incrementally learns the new tasks?
  • What can we learn from related fields such as online learning, meta-learning, Bayesian deep learning, robotics and neuroscience?

By aiming to bring together world-class researchers in the field of deep continual learning, as well as in the related fields of online learning, meta-learning, Bayesian deep learning, robotics and neuroscience to discuss and to brainstorm, we plan to set the research agenda for years to come.

Motivation text license
  Creative Commons BY 4.0
  Gido van de Ven, Bing Liu, Vincenzo Lomonaco, Tinne Tuytelaars

Classification

  • Machine Learning

Keywords

  • Continual learning
  • Incremental learning

Dokumentation

In der Reihe Dagstuhl Reports werden alle Dagstuhl-Seminare und Dagstuhl-Perspektiven-Workshops dokumentiert. Die Organisatoren stellen zusammen mit dem Collector des Seminars einen Bericht zusammen, der die Beiträge der Autoren zusammenfasst und um eine Zusammenfassung ergänzt.

 

Download Übersichtsflyer (PDF).

Dagstuhl's Impact

Bitte informieren Sie uns, wenn eine Veröffentlichung ausgehend von Ihrem Seminar entsteht. Derartige Veröffentlichungen werden von uns in der Rubrik Dagstuhl's Impact separat aufgelistet  und im Erdgeschoss der Bibliothek präsentiert.

Publikationen

Es besteht weiterhin die Möglichkeit, eine umfassende Kollektion begutachteter Arbeiten in der Reihe Dagstuhl Follow-Ups zu publizieren.