Dagstuhl Seminar 18122
Automatic Quality Assurance and Release
( Mar 18 – Mar 21, 2018 )
(Click in the middle of the image to enlarge)
Permalink
Please use the following short url to reference this page:
https://www.dagstuhl.de/18122
Organizers
- Bram Adams (Polytechnique Montreal, CA)
- Benoit Baudry (KTH Royal Institute of Technology - Stockholm, SE)
- Sigrid Eldh (Ericsson AB - Stockholm, SE)
- Andy Zaidman (TU Delft, NL)
Contact
- Shida Kunz (for scientific matters)
- Jutka Gasiorowski (for administrative matters)
Schedule
This seminar brings together researchers and practitioners who share an interest in the quality assurance perspective of release engineering and DevOps. The overarching goal is twofold: (1) sharing views on the state of the art and state of the practice regarding techniques to ensure quality within time-constrained release cycles; (2) identifying challenges and best practices that are relevant both for the software industry and as scientific investigations.
The bridge between quality assurance and release engineering is part of a wider movement to "shift right" the testing activities. For example, companies are increasingly applying A/B tests on subsets of their actual users, i.e., in production, to decide about design questions, or are using canary releases to test (and possibly roll back) a new version in the field. As such, “shift right” test strategies are becoming a key approach to reconcile the fast delivery of increasingly complex software systems with a level of quality that is higher than ever before.
Seminar topics to be included are:
Reconciling industry and academic perspectives
Both academia and industry (either open or closed source) participate in building techniques to ensure high quality within release engineering and DevOps. Yet, their understanding of these techniques varies in terms of implementation and, more importantly, with respect to the issues that need to be resolved to improve the applicability and efficiency of DevOps. The seminar aims to align academic perspectives with the industry’s needs, while sharing recent academic breakthroughs and techniques with industry.
Shift left versus shift right
Early testing (shift left) has been advocated as a good practice for decades. Meanwhile, DevOps and associated methodologies push for late (shift right) approaches to testing. The seminar wants to discuss both approaches and how one should balance both views to achieve effective quality assurance for release engineering processes.
Best practices
Can we define a set of best practices for implementing effective release engineering and DevOps pipelines? Aspects of these go beyond mere automation of the "build to release" pipeline, and also consider managing multiple system variants in parallel, limiting the impact of a feature change, having fast and automatic support for fault localization and debugging, or dealing with automatic installation and rollback in complex environments and configurations. Other practices advocate the creation of abstraction layers and complex architectures to limit fault propagation, yet this appears to stand in conflict with system testing to release fast.
Technical and organizational DevOps issues
What are both the technical and organizational issues in building effective QA into a release engineering and/or DevOps process? Here, we aim at identifying software quality risks that arise as a consequence of DevOps adoption. These risks can be related to the lack of tools or techniques, to the gap between existing techniques and the requirements for DevOps, but they can also arise because of organizational decisions taken to implement a DevOps process. The exchange between academic and industrial participants will be essential to identify these risks and crystallize challenges for future research.
Creative Commons BY 3.0 DE
Bram Adams, Benoit Baudry, Sigrid Eldh, and Andy Zaidman
The seminar explored the relationship between DevOps and quality assurance from a software engineering perspective. DevOps has been gaining traction since around 2012, with initiatives formed both in industry and academia. While the importance of DevOps as an enabler in higher quality software is intuitively clear to both industry and academia, we have discussed commonalities in views, but also the challenges that lie ahead for this discipline.
In essence, human factors are very important, because DevOps is not only a technology, it is a way of working and organizing teams. In this light, we have also discussed the resistance that some team members or even entire organisations seem to have towards automating quality assurance through DevOps. Section 4.2 summarizes a group discussion that eventually triggered a set of reflections on this topic of human aspects of DevOps. Yet, we have also discussed how DevOps can be an enabler for onboarding new team members through the availability of a standardized DevOps infrastructure (Section 4.4). The whole group observed the general lack of empirical evidence on the importance and benefits of DevOps in modern software engineering. This final point is tightly connected to another important theme in our discussion: educating software engineers in the ways and associated technologies of DevOps.
The main goal of this seminar was to bridge the knowledge divide on how researchers and industry professionals reason about and implement DevOps for automatic quality assurance. Through the seminar, we have built up a common understanding of DevOps tools and practices, but we have also identified major challenges for this field of research as well as for the teaching of DevOps principles and practices.
This Dagstuhl was a 2.5 day seminar, which we structured around 4 invited talks that served as keynotes to introduce key topics for discussions. These talks, summarized in Sections 3.1 through 3.3, were given at the beginning of each morning and afternoon to inspire topics for further discussions on a given topic. The group split into smaller sub-groups after each keynote, in order to focus discussions and reflections on a specific topic. All these discussions have been summarized in the form of a blog post, while in Dagstuhl, and are provided in this report.
In addition to keynotes and subgroup discussions, we had a plenary session to start the seminar, where each participant had 2 slides for a short introduction; we had a "speed-dating" session on Tuesday evening; and we organized a panel discussion about the future of the field on the last morning (Section 6.3).
Bram Adams, Benoit Baudry, Sigrid Eldh, and Andy Zaidman
- Bram Adams (Polytechnique Montreal, CA) [dblp]
- Benoit Baudry (KTH Royal Institute of Technology - Stockholm, SE) [dblp]
- Moritz Beller (TU Delft, NL) [dblp]
- Benjamin Danglot (INRIA Lille, FR) [dblp]
- Tamara Dumic (Ericsson Research - Stockholm, SE)
- Sigrid Eldh (Ericsson AB - Stockholm, SE) [dblp]
- Daniele Gagliardi (Engineering Ingegneria Informatica S.p.A - Roma, IT) [dblp]
- Georgios Gousios (TU Delft, NL) [dblp]
- Zhen Ming (Jack) Jiang (York University - Toronto, CA) [dblp]
- Foutse Khomh (Polytechnique Montreal, CA) [dblp]
- Philipp Leitner (Chalmers University of Technology - Göteborg, SE) [dblp]
- Lucy Ellen Lwakatare (University of Oulu, FI) [dblp]
- Vincent Massol (XWIKI - Paris, FR) [dblp]
- Shane McIntosh (McGill University - Montreal, CA) [dblp]
- Martin Monperrus (KTH Royal Institute of Technology - Stockholm, SE) [dblp]
- Sarah Nadi (University of Alberta - Edmonton, CA) [dblp]
- Andrew Neitsch (Cisco Systems Canada Co. - Toronto, CA) [dblp]
- Christopher J. Parnin (North Carolina State University - Raleigh, US) [dblp]
- Gerald Schermann (Universität Zürich, CH) [dblp]
- Weiyi (Ian) Shang (Concordia University - Montreal, CA) [dblp]
- Hui Song (SINTEF ICT - Oslo, NO) [dblp]
- Oscar Luis Vera Perez (INRIA - Rennes, FR) [dblp]
- Hyrum K. Wright (Duolingo - Pittsburgh, US) [dblp]
- Andy Zaidman (TU Delft, NL) [dblp]
- Fiorella Zampetti (University of Sannio - Benevento, IT) [dblp]
Classification
- software engineering
Keywords
- Software testing
- software release
- software deployment
- Quality assurance
- continuous integration
