02. – 07. Juni 2019, Dagstuhl-Seminar 19231

Empirical Evaluation of Secure Development Processes


Adam Shostack (Seattle, US)
Matthew Smith (Universität Bonn and Fraunhofer FKIE, DE)
Sam Weber (Carnegie Mellon University – Pittsburgh, US)
Mary Ellen Zurko (MIT Lincoln Laboratory – Lexington, US)

Auskunft zu diesem Dagstuhl-Seminar erteilt

Dagstuhl Service Team


Gemeinsame Dokumente
Dagstuhl-Seminar Wiki
Programm des Dagstuhl-Seminars [pdf]

(Zum Einloggen bitte Seminarnummer und Zugangscode verwenden)


The problem of how to design and build secure systems has been long-standing – although much progress has been made in software engineering, cybersecurity and industrial practices, many of the fundamental scientific foundations have not been laid and there is little empirical data to quantify the effects that our existing principles, architectures and methodologies have on the resulting systems.

This situation leaves developers and industry in a rather undesirable situation. The lack of data makes it difficult for organizations to choose practices that will cost-effectively reduce security vulnerabilities in a given system and help development teams achieve their security objectives. Without answers as to why proposed secure development practices are beneficial, and by how much, it is extremely difficult for organizations to rationally improve these processes, or to evaluate the cost-effectiveness of any specific technique.

The ultimate goal of this seminar is to create a community for empirical science in software engineering for secure systems. Naturally, such community-building is a long-term activity, which can be initiated during this seminar but will require continuous involvement. Our more immediate goals are to develop a manifesto for the community elucidating the need for research in this area, and to provide actionable and concrete guidance on how to overcome the obstacles that have hindered progress. The emphasis on being actionable and concrete is critical: the difficulties involved in empirically investigating security development processes, especially those in the early part of the development lifecycle, are already well-known, and instead we wish to focus on making forward progress.

Such forward progress requires not only the skills and knowledge of cybersecurity experts, but members of the empirical software engineering, usable security researchers and industrial communities as well. This seminar will bring together people from all four spheres. The majority of the seminar will be devoted to breakout groups, with each group focused on tackling a challenging problem that would have a large potential impact on secure development. Potential breakout topics include evaluating the effectiveness of different threat modeling methodologies, the security impact of different API design choices, and the merits of capabilities versus access-control-lists in real systems. Participants will be highly encouraged to develop and explore other similar challenges – the intent is that by focusing on more specific issues we are more likely to be able to develop actionable results.

This seminar aims to produce a manifesto to the community elucidating the need for empirical research of secure development methodologies and a report detailing both general guidance and advice on specific high-impact subtopics. However, the main outcome will be an active and growing research community tackling this new research field.

Motivation text license
  Creative Commons BY 3.0 DE
  Adam Shostack, Matthew Smith, Sam Weber, and Mary Ellen Zurko


  • Security / Cryptology
  • Society / Human-computer Interaction
  • Software Engineering


  • Empirical software engineering
  • Usable security for developers


In der Reihe Dagstuhl Reports werden alle Dagstuhl-Seminare und Dagstuhl-Perspektiven-Workshops dokumentiert. Die Organisatoren stellen zusammen mit dem Collector des Seminars einen Bericht zusammen, der die Beiträge der Autoren zusammenfasst und um eine Zusammenfassung ergänzt.


Download Übersichtsflyer (PDF).


Es besteht weiterhin die Möglichkeit, eine umfassende Kollektion begutachteter Arbeiten in der Reihe Dagstuhl Follow-Ups zu publizieren.

Dagstuhl's Impact

Bitte informieren Sie uns, wenn eine Veröffentlichung ausgehend von
Ihrem Seminar entsteht. Derartige Veröffentlichungen werden von uns in der Rubrik Dagstuhl's Impact separat aufgelistet  und im Erdgeschoss der Bibliothek präsentiert.