https://www.dagstuhl.de/22052

January 30 – February 2 , 2022, Dagstuhl Seminar 22052

The Human Factors Impact of Programming Error Messages

Organizers

Brett A. Becker (University College Dublin, IE)
Paul Denny (University of Auckland, NZ)
Janet Siegmund (TU Chemnitz, DE)
Andreas Stefik (University of Nevada – Las Vegas, US)

For support, please contact

Dagstuhl Service Team

Documents

Dagstuhl Report, Volume 12, Issue 1 Dagstuhl Report
Aims & Scope
List of Participants
Shared Documents
Dagstuhl Seminar Schedule [pdf]

Summary

Programming error messages (commonly called compiler error messages) pose challenges to programmers - from novices to professionals - with evidence dating from the 1960s to present day. In this seminar, we explored the nature of these challenges and particularly why they remain largely unaddressed. We further investigated the specific challenges that different users including children, non-native English speakers, and those of varying ability experience when faced with programming error messages. Finally, we sought to identify the most promising avenues to assess the effectiveness of error messages, how to improve them with large, demonstrated effect, and how to produce appropriate messages for different users with different needs. To this end, we assembled experts from many sub-disciplines of Computer Science, including Programming Languages, HCI, Computer Science Education, and Software Engineering as well as Learning Sciences. Due to travel restrictions imposed by the COVID-19 pandemic, we ran the seminar in a hybrid format, with 11 on-site participants collaborating with 17 remote participants. We formulated a schedule for the seminar that sought to maximise outcomes for all participants.

By combining the expertise of these different disciplines, we could identify gaps in knowledge and high-priority areas to build a basis for future work. This is the starting point for a long-lasting contribution to the field. By uniting communities that have to date been working largely in isolation, we sought to gather appropriate and useful data, broaden perspectives, build consensus among diverse stakeholders, and begin cross-community efforts working in unison going forward.

During the seminar, we had sessions focusing on existing literature and practice, including experience reports from language maintainers, expert users, researchers, and educators, to develop a shared understanding of the evidence that exists with regard to effective programming error messages. This included the group working together to synthesise existing data sets, which we view as an important exercise given that large corpora of errors and error message data are generally not openly available. This can make it difficult for researchers to answer seemingly simple questions such as: "What are the most frequent error messages encountered by [students, professionals, blind, non-native English speakers] in language x?" or, "What are evidence-based examples of effective and/or ineffective error messages?". As a result, we identified several fruitful avenues in the form of cross-discipline collaboration, data sharing opportunities, and improved focus on what steps are needed to improve efforts to answer these questions.

One of the key objectives of the seminar was to identify areas for immediate research. Several problems have been identified, including but not limited to:

  1. linking error messages with the context of the problems and programs being worked on when error messages are generated;
  2. understanding the metacognitive aspects involved in the process of interpreting and reacting to error messages;
  3. identifying what error messages arise from inconsistent conceptions of how a program runs or how it is structured;
  4. the effects of error message presentation on error message interpretation (e.g. visual queues, structured error messages, etc.);
  5. error message classification schemes;
  6. metrics to measure and assess error messages;
  7. identifying when error messages are "wrong", what occurs when they steer the programmer in the wrong direction, and how this can be avoided;
  8. determining the design factors of programming error messages that differ across demographic groups (e.g., expertise level, disabilities, native/natural language, etc.).

Another key objective was to establish new cross-community efforts to improve programming error messages in practice, leveraging the strengths of each community. Participants discussed open research problems and identified those of high priority and interest including those listed above. Several interdisciplinary research objectives have been established, and seeds were sewn to form teams to collaboratively address these research questions.

Summary text license
  Creative Commons BY 4.0
  Brett A. Becker, Paul Denny, Janet Siegmund, and Andreas Stefik

Classification

  • Human-Computer Interaction
  • Programming Languages
  • Software Engineering

Keywords

  • Compiler Error Messages
  • Computer Science Education
  • Empirical Studies
  • Human Factors
  • Programming Language Design

Documentation

In the series Dagstuhl Reports each Dagstuhl Seminar and Dagstuhl Perspectives Workshop is documented. The seminar organizers, in cooperation with the collector, prepare a report that includes contributions from the participants' talks together with a summary of the seminar.

 

Download overview leaflet (PDF).

Dagstuhl's Impact

Please inform us when a publication was published as a result from your seminar. These publications are listed in the category Dagstuhl's Impact and are presented on a special shelf on the ground floor of the library.

Publications

Furthermore, a comprehensive peer-reviewed collection of research papers can be published in the series Dagstuhl Follow-Ups.