June 2 – 7 , 2019, Dagstuhl Seminar 19231

Empirical Evaluation of Secure Development Processes


Adam Shostack (Seattle, US)
Matthew Smith (Universität Bonn and Fraunhofer FKIE, DE)
Sam Weber (Carnegie Mellon University – Pittsburgh, US)
Mary Ellen Zurko (MIT Lincoln Laboratory – Lexington, US)

For support, please contact

Dagstuhl Service Team


List of Participants
Shared Documents
Dagstuhl Seminar Wiki
Dagstuhl Seminar Schedule [pdf]

(Use seminar number and access code to log in)


The problem of how to design and build secure systems has been long-standing – although much progress has been made in software engineering, cybersecurity and industrial practices, many of the fundamental scientific foundations have not been laid and there is little empirical data to quantify the effects that our existing principles, architectures and methodologies have on the resulting systems.

This situation leaves developers and industry in a rather undesirable situation. The lack of data makes it difficult for organizations to choose practices that will cost-effectively reduce security vulnerabilities in a given system and help development teams achieve their security objectives. Without answers as to why proposed secure development practices are beneficial, and by how much, it is extremely difficult for organizations to rationally improve these processes, or to evaluate the cost-effectiveness of any specific technique.

The ultimate goal of this seminar is to create a community for empirical science in software engineering for secure systems. Naturally, such community-building is a long-term activity, which can be initiated during this seminar but will require continuous involvement. Our more immediate goals are to develop a manifesto for the community elucidating the need for research in this area, and to provide actionable and concrete guidance on how to overcome the obstacles that have hindered progress. The emphasis on being actionable and concrete is critical: the difficulties involved in empirically investigating security development processes, especially those in the early part of the development lifecycle, are already well-known, and instead we wish to focus on making forward progress.

Such forward progress requires not only the skills and knowledge of cybersecurity experts, but members of the empirical software engineering, usable security researchers and industrial communities as well. This seminar will bring together people from all four spheres. The majority of the seminar will be devoted to breakout groups, with each group focused on tackling a challenging problem that would have a large potential impact on secure development. Potential breakout topics include evaluating the effectiveness of different threat modeling methodologies, the security impact of different API design choices, and the merits of capabilities versus access-control-lists in real systems. Participants will be highly encouraged to develop and explore other similar challenges – the intent is that by focusing on more specific issues we are more likely to be able to develop actionable results.

This seminar aims to produce a manifesto to the community elucidating the need for empirical research of secure development methodologies and a report detailing both general guidance and advice on specific high-impact subtopics. However, the main outcome will be an active and growing research community tackling this new research field.

Motivation text license
  Creative Commons BY 3.0 DE
  Adam Shostack, Matthew Smith, Sam Weber, and Mary Ellen Zurko


  • Security / Cryptology
  • Society / Human-computer Interaction
  • Software Engineering


  • Empirical software engineering
  • Usable security for developers


In the series Dagstuhl Reports each Dagstuhl Seminar and Dagstuhl Perspectives Workshop is documented. The seminar organizers, in cooperation with the collector, prepare a report that includes contributions from the participants' talks together with a summary of the seminar.


Download overview leaflet (PDF).


Furthermore, a comprehensive peer-reviewed collection of research papers can be published in the series Dagstuhl Follow-Ups.

Dagstuhl's Impact

Please inform us when a publication was published as a result from your seminar. These publications are listed in the category Dagstuhl's Impact and are presented on a special shelf on the ground floor of the library.