Evidence About Programmers for Programming Language Design

Press Room

• Designing and evaluating programming languages: Dagstuhl trip report
  Blog entry by Andy J. Ko in Bits and Behaviour on Medium, February 9, 2018
• Dagstuhl reflections
  Blog entry by Amelia McNamara in Letters from a young statistician, February 11, 2018
• Dagstuhl Seminar 18061: Evidence about programmers for programming language design
  Blog entry by Brett Becker in CS0, February 11, 2018
• Tag Archive: dagstuhl
  Live blog entries by Felienne in Programming and Education and Programming Education, February 5 - 8, 2018

Motivation

In computer science as it stands today, there exists thousands of competing programming language products, with new ones (including both domain-specific and general purpose languages) being created regularly. These technologies are important, with applications ranging from educational or business tools to critical infrastructure in society or the military. While programming is by its nature mathematical, meaning that these products have a rigorous foundation to check that they work on a technical level, they are programmed by people that may have challenges and are prone to mistakes. While one might assume that existing communities, like the International Conference on Software Engineering (ICSE), Empirical Software Engineering (ESE), or the programming language conferences themselves (e.g., ICFP, OOPSLA, POPL) might be suitable venues for discussions about human-factors evidence to guide language design, this has not historically been the case. Notably, recent studies by Kaijanaho and others has shown that these venues contain almost no scientific data on the issue, with only 22 studies from the 1950s to 2012. Further, one might reasonably assume that smaller workshops, like the Psychology of Programmers Interest Group (PPIG), older workshops like Empirical Studies on Programmers (ESP), or newer workshops like PLATEAU cover this space, but rigorous investigations into their evidence practices show that these venues rarely include reports with rigorous scientific evidence.

In this Dagstuhl Seminar, we have four primary objectives. First, we are 1) striving to form a diverse community of scholars around programming language design based on evidence of programmer impacts. Given that top-tier venues in the area have not carefully investigated this long-standing and important issue in the field, we want to form a new community that will. Second, given that the design space for programming languages is very large, it makes sense to 2) create an organized set of priorities for new studies in the area along with 3) the evidence standard to be used. Notably, if a new community that contains evidence were to appear, it makes sense to create rigorous standards so that scholars can "test" each other's claims in replication, as opposed to the "trust" model that exists today. Finally, research can often make progress if it adheres to guiding principles, which brings us to our fourth objective, 4) to discuss or create theories that can guide the community in understanding and framing the evidence gathered over time. Historically, scholars have relied on somewhat subjective guiding theories. In contrast, we strive for theories and data that emphasize precision and objectivity.

During the seminar, participants will engage in a variety of activities. First, we will have sessions on the existing literature to provide attendees with historical context, a shared sense of what evidence exists, and methods that have been used to gather that evidence. Second, as there are considerable holes in the scholarly literature, we will generate lists of possible studies and try to prioritize them as a group. Third, to better understand what evidence standard could be used for these future studies, we will discuss evidence standards as they exist in other scholarly disciplines and their respective pros and cons. In summary, seminar participants will investigate the impact of programming language design on the productivity of programmers using systematic science with established standards.