Dagstuhl Seminar 19231
Empirical Evaluation of Secure Development Processes
( Jun 02 – Jun 07, 2019 )
(Click in the middle of the image to enlarge)
Permalink
Please use the following short url to reference this page:
https://www.dagstuhl.de/19231
Organizers
- Adam Shostack (Seattle, US)
- Matthew Smith (Universität Bonn, DE & Fraunhofer FKIE - Bonn, DE)
- Sam Weber (Carnegie Mellon University - Pittsburgh, US)
- Mary Ellen Zurko (MIT Lincoln Laboratory - Lexington, US)
Contact
- Shida Kunz (for scientific matters)
Dagstuhl Seminar Wiki
- Dagstuhl Seminar Wiki (Use personal credentials as created in DOOR to log in)
Shared Documents
- Dagstuhl Materials Page (Use personal credentials as created in DOOR to log in)
Impacts
- "Do this! Do that!, And Nothing will happen" : Do specifications lead to securely stored passwords? : article in 43rd International Conference on Software Engineering ICSE 2021 - Hallett, Joseph; Patnaik, Nikhil; Shreeve, Ben; Rashid, Awais - Los Alamitos : IEEE, 2021. - pp. 486-499.
- Contextualisation of Data Flow Diagrams for security analysis : article - Faily, Shamal; Scandariato, Riccardo; Shostack, Adam; Sion, Laurens; Ki-Aries, Duncan - Cornell University : arXiv.org, 2020. - 12 pp..
- Contextualisation of Data Flow Diagrams for Security Analysis : article in International Workshop on Graphical Models for Security GraMSec 2020 : Graphical Models for Security, LNCS 12419 - Faily, Shamal; Scandariato, Riccardo; Shostack, Adam; Sion, Laurens; Ki-Aries, Duncan - Berlin : Springer, 2020. - pp. 186-197.
- Exploring the Shift in Security Responsibility : article - Weir, Charles; Migues, Samuel; Williams, Laurie - Los Alamitos : IEEE, 2022. - pp. 2-11 - (IEEE Security & Privacy ; 2022).
- Infiltrating security into development : exploring the world’s largest software security study : article in ESEC/FSE 2021: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering - Weir, Charles; Migues, Samuel; Ware, Mike; Williams, Laurie - New York : ACM, 2021. - Pages 1326 - 1336.
- Secure Development Tools and Techniques Need More Research That Will Increase Their Impact and Effectiveness in Practice : article - Shostack, Adam; Zurko, Mary Ellen - New York : ACM, 2020. - pp.39-41 - (Communications of the ACM ; 63. 2020, 5 : article).
- “Do this! Do that!, and Nothing will Happen” Do Specifications Lead to Securely Stored Passwords? : article in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) - Hallett, Joseph; Patnaik, Nikhil; Shreeve, Ben; Rashid, Awais - Los Alamitos : IEEE, 2021. - pp. 486-498.
Schedule
The problem of how to design and build secure systems has been long-standing – although much progress has been made in software engineering, cybersecurity and industrial practices, many of the fundamental scientific foundations have not been laid and there is little empirical data to quantify the effects that our existing principles, architectures and methodologies have on the resulting systems.
This situation leaves developers and industry in a rather undesirable situation. The lack of data makes it difficult for organizations to choose practices that will cost-effectively reduce security vulnerabilities in a given system and help development teams achieve their security objectives. Without answers as to why proposed secure development practices are beneficial, and by how much, it is extremely difficult for organizations to rationally improve these processes, or to evaluate the cost-effectiveness of any specific technique.
The ultimate goal of this seminar is to create a community for empirical science in software engineering for secure systems. Naturally, such community-building is a long-term activity, which can be initiated during this seminar but will require continuous involvement. Our more immediate goals are to develop a manifesto for the community elucidating the need for research in this area, and to provide actionable and concrete guidance on how to overcome the obstacles that have hindered progress. The emphasis on being actionable and concrete is critical: the difficulties involved in empirically investigating security development processes, especially those in the early part of the development lifecycle, are already well-known, and instead we wish to focus on making forward progress.
Such forward progress requires not only the skills and knowledge of cybersecurity experts, but members of the empirical software engineering, usable security researchers and industrial communities as well. This seminar will bring together people from all four spheres. The majority of the seminar will be devoted to breakout groups, with each group focused on tackling a challenging problem that would have a large potential impact on secure development. Potential breakout topics include evaluating the effectiveness of different threat modeling methodologies, the security impact of different API design choices, and the merits of capabilities versus access-control-lists in real systems. Participants will be highly encouraged to develop and explore other similar challenges – the intent is that by focusing on more specific issues we are more likely to be able to develop actionable results.
This seminar aims to produce a manifesto to the community elucidating the need for empirical research of secure development methodologies and a report detailing both general guidance and advice on specific high-impact subtopics. However, the main outcome will be an active and growing research community tackling this new research field.
Creative Commons BY 3.0 DE
Adam Shostack, Matthew Smith, Sam Weber, and Mary Ellen Zurko
The problem of how to design and build secure systems has been long-standing. For example, as early as 1978 Bisbey and Hollingworth[6] complained that there was no method of determining what an appropriate level of security for a system actually was. In the early years various design principles, architectures and methodologies were proposed: in 1972 Anderson[5] described the “reference monitor” concept, in 1974 Saltzer[7] described the “Principle of least privilege”, and in 1985 the US Department of Defense issued the Trusted Computer System Evaluation Criteria[8].
Since then, although much progress has been made in software engineering, cybersecurity and industrial practices, much of the fundamental scientific foundations have not been addressed – there is little empirical data to quantify the effects that these principles, architectures and methodologies have on the resulting systems.
This situation leaves developers and industry in a rather undesirable situation. The lack of this data makes it difficult for organizations to effectively choose practices that will cost-effectively reduce security vulnerabilities in a given system and help development teams achieve their security objectives. There has been much work creating security development lifecycles, such as the Building Security In Maturity Model[1], Microsoft Security Development LifeCycle[3] OWASP[4] and ISECOM[2] and these incorporate a long series of recommended practices on requirements analysis, architectural threat analysis, and hostile code review. It is agreed that these efforts are, in fact, beneficial. However, without answers as to why they are beneficial, and how much, it is extremely difficult for organizations to rationally improve these processes, or to evaluate the cost-effectiveness of any specific technique.
The ultimate goal of this seminar was to create a community for empirical science in software engineering for secure systems. This is particularly important in this nascent of research in this domain stage since there is no venue in which researchers meet and exchange. Currently single pieces of work are published at a wide variety of venues such as IEEE S&P, IEEE EuroS&P, ACM CCS, USENIX Security, SOUPS, SIGCHI, ICSE, USEC, EuroUSEC, and many more. The idea was that bringing together all researchers working separately and creating an active exchange will greatly benefit the community.
Naturally, community-building is a long-term activity – we can initiate it at a Dagstuhl seminar, but it will require continuous activity. Our more immediate goals were to develop a manifesto for the community elucidating the need for research in this area, and to provide actionable and concrete guidance on how to overcome the obstacles that have hindered progress.
One aspect of this was information gathering on how to conduct academic research which is able to be transitioned and consumed by developers. We felt that all too frequently developer needs aren’t fully understood by academics, and that developers underestimate the relevance of academic results. Our information gathering will help foster mutual understanding between these two groups and we specifically looked for ways to build bridges between them.
A second obstacle which we aimed to address is how to produce sufficiently convincing empirical research at a foundational level as well as in the specific application areas. Currently there is no consensus on what are ecologically valid studies and there are sporadic debates on the merits of the different approaches. This seminar included a direct and focused exchange of experience and facilitated the creation of much needed guidelines for researchers. In accordance with our bridge building, we also looked at what developers find convincing, and how that aligns with research requirements.
Seminar Format
Our seminar brought together thirty-three participants from industry, government and both the security and software engineering academic communities. Before the seminar started we provided participants with the opportunity to share background readings amongst themselves.
We began our seminar with level-setting and foundational talks from industrial, software engineering and security participants aimed to foster a common level of understanding of the differing perspectives of the various communities.
Following this the seminar was very dynamic: during each session we broke into break-out groups whose topics were dynamically generated by the participants. The general mandate for each group was to tackle an aspect of the general problem and be actionable and concrete: we wished to avoid vague discussions of the difficulties involved with studying secure development but instead focus on how to improve our understanding and knowledge. After each session we met again as a group and summarized each group’s progress.
At the conclusion of the seminar we brought together all the participants in a general discussion about further activities. In all, a total of eighteen further activities, ranging from papers to research guideline documents, were proposed and organized by the participants.
References
- Building security in maturity model. http://www.bsimm.com/.
- Isecom. http://www.isecom.org.
- Microsoft security development lifecycle. https://www.microsoft.com/en-us/securityengineering/sdl/.
- Owasp. https://www.owasp.org.
- Anderson, J. P. Computer Security Technology Planning Study, Volume II. Tech. Rep. ESD-TR-73-51, 1972.
- Bisbey, R., and Hollingworth, D. Protection analysis: Final report. Information Sciences Institute, University of Southern California: Marina Del Rey, CA, USA, Technical Report ISI/SR-78-13 (1978).
- Saltzer, J. Protection and the control of information sharing in Multics. Communications of the ACM 17, 7 (1974), 388–402.
- United States Department of Defense. Trusted computer system evaluation criteria ( orange book ).
Adam Shostack, Matthew Smith, Sam Weber, and Mary Ellen Zurko
- Florian Alt (Universität der Bundeswehr - München, DE) [dblp]
- Adam J. Aviv (U.S. Naval Academy - Annapolis, US) [dblp]
- Eric Bodden (Universität Paderborn, DE) [dblp]
- Michael Coblenz (Carnegie Mellon University - Pittsburgh, US) [dblp]
- Tamara Denning (University of Utah - Salt Lake City, US) [dblp]
- Serge Egelman (ICSI - Berkeley, US) [dblp]
- Sascha Fahl (Leibniz Universität Hannover, DE) [dblp]
- Shamal Faily (Bournemouth University, GB) [dblp]
- Tobias Fiebig (TU Delft, NL) [dblp]
- Joseph Hallett (University of Bristol, GB) [dblp]
- Trent Jaeger (Pennsylvania State University - University Park, US) [dblp]
- Mike Lake (CISCO Systems - Research Triangle Park, US) [dblp]
- Carl E. Landwehr (George Washington University - Washington, US) [dblp]
- Steven B. Lipner (SAFECode - Seattle, US) [dblp]
- Luigi Lo Iacono (FH Köln, DE) [dblp]
- Fabio Massacci (University of Trento, IT) [dblp]
- Michelle Mazurek (University of Maryland - College Park, US) [dblp]
- Brendan Murphy (Microsoft Research - Cambridge, GB) [dblp]
- Brad Myers (Carnegie Mellon University - Pittsburgh, US) [dblp]
- Xinming (Simon) Ou (University of South Florida - Tampa, US) [dblp]
- Olgierd Pieczul (IBM Research - Dublin, IE) [dblp]
- Heather Richter Lipford (University of North Carolina - Charlotte, US) [dblp]
- Riccardo Scandariato (Chalmers and University of Gothenburg, SE) [dblp]
- Reinhard Schwarz (Fraunhofer IESE - Kaiserslautern, DE) [dblp]
- Adam Shostack (Seattle, US) [dblp]
- Laurens Sion (KU Leuven, BE) [dblp]
- Matthew Smith (Universität Bonn, DE & Fraunhofer FKIE - Bonn, DE) [dblp]
- Walter F. Tichy (KIT - Karlsruher Institut für Technologie, DE) [dblp]
- Daniel Votipka (University of Maryland - College Park, US) [dblp]
- Sam Weber (Carnegie Mellon University - Pittsburgh, US) [dblp]
- Charles Weir (Lancaster University, GB) [dblp]
- Laurie Williams (North Carolina State University - Raleigh, US) [dblp]
- Mary Ellen Zurko (MIT Lincoln Laboratory - Lexington, US) [dblp]
Related Seminars
- Dagstuhl Seminar 23181: Empirical Evaluation of Secure Development Processes (2023-05-01 - 2023-05-05) (Details)
Classification
- security / cryptology
- society / human-computer interaction
- software engineering
Keywords
- empirical software engineering
- usable security for developers
