TOP
Search the Dagstuhl Website
Looking for information on the websites of the individual seminars? - Then please:
Not found what you are looking for? - Some of our services have separate websites, each with its own search option. Please check the following list:
Schloss Dagstuhl - LZI - Logo
Schloss Dagstuhl Services
Seminars
Within this website:
External resources:
  • DOOR (for registering your stay at Dagstuhl)
  • DOSA (for proposing future Dagstuhl Seminars or Dagstuhl Perspectives Workshops)
Publishing
Within this website:
External resources:
dblp
Within this website:
External resources:
  • the dblp Computer Science Bibliography


Dagstuhl Seminar 22052

The Human Factors Impact of Programming Error Messages

( Jan 30 – Feb 02, 2022 )

(Click in the middle of the image to enlarge)

Permalink
Please use the following short url to reference this page: https://www.dagstuhl.de/22052

Organizers

Contact



Schedule

Summary

Programming error messages (commonly called compiler error messages) pose challenges to programmers - from novices to professionals - with evidence dating from the 1960s to present day. In this seminar, we explored the nature of these challenges and particularly why they remain largely unaddressed. We further investigated the specific challenges that different users including children, non-native English speakers, and those of varying ability experience when faced with programming error messages. Finally, we sought to identify the most promising avenues to assess the effectiveness of error messages, how to improve them with large, demonstrated effect, and how to produce appropriate messages for different users with different needs. To this end, we assembled experts from many sub-disciplines of Computer Science, including Programming Languages, HCI, Computer Science Education, and Software Engineering as well as Learning Sciences. Due to travel restrictions imposed by the COVID-19 pandemic, we ran the seminar in a hybrid format, with 11 on-site participants collaborating with 17 remote participants. We formulated a schedule for the seminar that sought to maximise outcomes for all participants.

By combining the expertise of these different disciplines, we could identify gaps in knowledge and high-priority areas to build a basis for future work. This is the starting point for a long-lasting contribution to the field. By uniting communities that have to date been working largely in isolation, we sought to gather appropriate and useful data, broaden perspectives, build consensus among diverse stakeholders, and begin cross-community efforts working in unison going forward.

During the seminar, we had sessions focusing on existing literature and practice, including experience reports from language maintainers, expert users, researchers, and educators, to develop a shared understanding of the evidence that exists with regard to effective programming error messages. This included the group working together to synthesise existing data sets, which we view as an important exercise given that large corpora of errors and error message data are generally not openly available. This can make it difficult for researchers to answer seemingly simple questions such as: "What are the most frequent error messages encountered by [students, professionals, blind, non-native English speakers] in language x?" or, "What are evidence-based examples of effective and/or ineffective error messages?". As a result, we identified several fruitful avenues in the form of cross-discipline collaboration, data sharing opportunities, and improved focus on what steps are needed to improve efforts to answer these questions.

One of the key objectives of the seminar was to identify areas for immediate research. Several problems have been identified, including but not limited to:

  1. linking error messages with the context of the problems and programs being worked on when error messages are generated;
  2. understanding the metacognitive aspects involved in the process of interpreting and reacting to error messages;
  3. identifying what error messages arise from inconsistent conceptions of how a program runs or how it is structured;
  4. the effects of error message presentation on error message interpretation (e.g. visual queues, structured error messages, etc.);
  5. error message classification schemes;
  6. metrics to measure and assess error messages;
  7. identifying when error messages are "wrong", what occurs when they steer the programmer in the wrong direction, and how this can be avoided;
  8. determining the design factors of programming error messages that differ across demographic groups (e.g., expertise level, disabilities, native/natural language, etc.).

Another key objective was to establish new cross-community efforts to improve programming error messages in practice, leveraging the strengths of each community. Participants discussed open research problems and identified those of high priority and interest including those listed above. Several interdisciplinary research objectives have been established, and seeds were sewn to form teams to collaboratively address these research questions.

Copyright Brett A. Becker, Paul Denny, Janet Siegmund, and Andreas Stefik

Motivation

Many of the Human Factors that impact how people program are poorly understood, presenting significant challenges to efforts aimed at improving programmer productivity and education. Programming error messages (commonly called compiler error messages) are one aspect that is known to be problematic, with evidence dating back to the 1960s. The information that is presented to users via these messages impacts different people in different ways, from children to students to seasoned professionals. Despite clear evidence of their human impact, no consensus exists on how to assess the effectiveness of error messages, how to improve them with large, demonstrated effect, nor how to produce appropriate messages for different users with different needs.

Work on improving programming error messages presents a complex challenge as it spans many sub-disciplines of Computer Science including Programming Languages, HCI, Computer Science Education, and Software Engineering as well as Learning Sciences more broadly. The past several years has seen a marked increase in interest within some communities. However, in order to best proceed it would be highly beneficial for experts across these areas to identify gaps in knowledge, high-priority areas, and a basis upon which future work can be built. This Dagstuhl seminar will seek to establish a long-lasting contribution to the field by uniting communities that have to date been working largely in isolation. We seek to gather appropriate and useful data, broaden perspectives, build consensus among diverse stakeholders, and begin cross-community efforts working in unison going forward.

During this Dagstuhl Seminar, participants will engage in a variety of activities to meet three broad objectives. First, we will have sessions focusing on existing literature and practice, including experience reports from language maintainers, expert users, researchers, and educators, to develop a shared understanding of the evidence that exists with regard to effective programming error messages. This will include the group working together to synthesize existing data sets, initially contributed by attendees with access to such. This is an important exercise, as large corpora of errors and error message data are generally not openly available, making it difficult for researchers to answer seemingly simple questions (without conducting large scale studies) such as: "What are the most frequent error messages encountered by [students, professionals, blind, non-native English speakers] in language x?" or, "What are evidence-based examples of effective and/or ineffective error messages?" Organizing such data will help meet the second objective of the seminar: to identify areas for immediate research including determining the design factors of programming error messages that differ across demographic groups (e.g., expertise level, disabilities, native/natural language, etc.). The third objective of the seminar is to establish new cross-community efforts to improve programming error messages in practice, leveraging the strengths of each community. Participants will be able to discuss open research problems and identify those of high priority and interest, and join collaborative research teams involving multi-site data collection.

Throughout the seminar breakout groups will investigate diverse topics such as how the needs of professionals and students differ, how metrics for measuring error message performance can be defined, and how to characterize the unique requirements of programmers with disabilities. Attendees are encouraged to collect/organize examples of effective and ineffective error messages, as well as lists/collections of error messages from various languages, including frequency data for given users, and share these prior to the seminar. These will be used during the seminar as a basis for discussion and moving forward. We will include opportunities within the seminar to explore in greater detail any topics identified in early sessions that are of particular interest. This flexibility will enable us to capitalize on the synergies we are confident will develop quickly at this seminar. The wide-ranging issue of problematic error messages will not be solved easily or in short order. This seminar seeks to start this process with a future-focused outlook based on community consensus of the objectives that must be met immediately and a roadmap of concrete steps to take once these are achieved.

Copyright Brett A. Becker, Paul Denny, Janet Siegmund, and Andreas Stefik

Participants
On-site
  • Annabelle Bergum (Universität des Saarlandes - Saarbrücken, DE)
  • Joe Dillane (University College Dublin, IE)
  • Kerstin Haring (University of Denver, US)
  • Elisa Madeleine Hartmann (TU Chemnitz, DE)
  • Felienne Hermans (Leiden University, NL) [dblp]
  • Tobias Kohn (University of Cambridge, GB)
  • Jens Mönig (SAP SE - Walldorf, DE) [dblp]
  • Raymond Pettit (University of Virginia - Charlottesville, US)
  • Maria Mercedes T. Rodrigo (Ateneo de Manila University - Quezon City, PH)
  • Eddie Antonio Santos (University College Dublin, IE)
  • Janet Siegmund (TU Chemnitz, DE) [dblp]
Remote:
  • Brett A. Becker (University College Dublin, IE) [dblp]
  • Dennis Bouvier (Southern Illinois Univ. Edwardsville, US)
  • Neil Brown (King's College London, GB) [dblp]
  • Paul Denny (University of Auckland, NZ) [dblp]
  • Kathi Fisler (Brown University - Providence, US) [dblp]
  • Ioannis Karvelas (University College Dublin, IE)
  • Amy Ko (University of Washington - Seattle, US) [dblp]
  • Michael Kölling (King's College London, GB) [dblp]
  • Shriram Krishnamurthi (Brown University - Providence, US) [dblp]
  • Linda McIver (ADSEI - Glen Waverley, AU)
  • Jan Pearce (Berea College, US)
  • Prajish Prasad (IIT Bombay, IN)
  • James Prather (Abilene Christian University, US)
  • Seán Russell (University College Dublin, IE)
  • Bonita Sharif (University of Nebraska - Lincoln, US) [dblp]
  • Andreas Stefik (University of Nevada - Las Vegas, US) [dblp]
  • Toni Taipalus (University of Jyväskylä, FI)
  • Jan Vahrenhold (Universität Münster, DE) [dblp]

Classification
  • Human-Computer Interaction
  • Programming Languages
  • Software Engineering

Keywords
  • Compiler Error Messages
  • Computer Science Education
  • Empirical Studies
  • Human Factors
  • Programming Language Design