February 4 – 9 , 2018, Dagstuhl Seminar 18061

Evidence About Programmers for Programming Language Design


Stefan Hanenberg (Universität Duisburg-Essen, DE)
Brad A. Myers (Carnegie Mellon University – Pittsburgh, US)
Bonita Sharif (Youngstown State University, US)
Andreas Stefik (Univ. of Nevada – Las Vegas, US)

For support, please contact

Dagstuhl Service Team


List of Participants
Shared Documents
Dagstuhl Seminar Schedule [pdf] (Update here)

(Use seminar number and access code to log in)

Press Room


In computer science as it stands today, there exists thousands of competing programming language products, with new ones (including both domain-specific and general purpose languages) being created regularly. These technologies are important, with applications ranging from educational or business tools to critical infrastructure in society or the military. While programming is by its nature mathematical, meaning that these products have a rigorous foundation to check that they work on a technical level, they are programmed by people that may have challenges and are prone to mistakes. While one might assume that existing communities, like the International Conference on Software Engineering (ICSE), Empirical Software Engineering (ESE), or the programming language conferences themselves (e.g., ICFP, OOPSLA, POPL) might be suitable venues for discussions about human-factors evidence to guide language design, this has not historically been the case. Notably, recent studies by Kaijanaho and others has shown that these venues contain almost no scientific data on the issue, with only 22 studies from the 1950s to 2012. Further, one might reasonably assume that smaller workshops, like the Psychology of Programmers Interest Group (PPIG), older workshops like Empirical Studies on Programmers (ESP), or newer workshops like PLATEAU cover this space, but rigorous investigations into their evidence practices show that these venues rarely include reports with rigorous scientific evidence.

In this Dagstuhl Seminar, we have four primary objectives. First, we are 1) striving to form a diverse community of scholars around programming language design based on evidence of programmer impacts. Given that top-tier venues in the area have not carefully investigated this long-standing and important issue in the field, we want to form a new community that will. Second, given that the design space for programming languages is very large, it makes sense to 2) create an organized set of priorities for new studies in the area along with 3) the evidence standard to be used. Notably, if a new community that contains evidence were to appear, it makes sense to create rigorous standards so that scholars can "test" each other's claims in replication, as opposed to the "trust" model that exists today. Finally, research can often make progress if it adheres to guiding principles, which brings us to our fourth objective, 4) to discuss or create theories that can guide the community in understanding and framing the evidence gathered over time. Historically, scholars have relied on somewhat subjective guiding theories. In contrast, we strive for theories and data that emphasize precision and objectivity.

During the seminar, participants will engage in a variety of activities. First, we will have sessions on the existing literature to provide attendees with historical context, a shared sense of what evidence exists, and methods that have been used to gather that evidence. Second, as there are considerable holes in the scholarly literature, we will generate lists of possible studies and try to prioritize them as a group. Third, to better understand what evidence standard could be used for these future studies, we will discuss evidence standards as they exist in other scholarly disciplines and their respective pros and cons. In summary, seminar participants will investigate the impact of programming language design on the productivity of programmers using systematic science with established standards.

  Creative Commons BY 3.0 DE
  Stefan Hanenberg, Brad A. Myers, Bonita Sharif, and Andreas Stefik


  • Programming Languages / Compiler
  • Society / Human-computer Interaction
  • Software Engineering


  • Human Factors
  • Programming Language Design
  • Community Evidence Standards
  • Domain Specific Languages

Book exhibition

Books from the participants of the current Seminar 

Book exhibition in the library, ground floor, during the seminar week.


In the series Dagstuhl Reports each Dagstuhl Seminar and Dagstuhl Perspectives Workshop is documented. The seminar organizers, in cooperation with the collector, prepare a report that includes contributions from the participants' talks together with a summary of the seminar.


Download overview leaflet (PDF).


Furthermore, a comprehensive peer-reviewed collection of research papers can be published in the series Dagstuhl Follow-Ups.

Dagstuhl's Impact

Please inform us when a publication was published as a result from your seminar. These publications are listed in the category Dagstuhl's Impact and are presented on a special shelf on the ground floor of the library.

NSF young researcher support