March 18 – 21 , 2018, Dagstuhl Seminar 18122

Automatic Quality Assurance and Release


Bram Adams (Polytechnique Montreal, CA)
Benoit Baudry (KTH Royal Institute of Technology – Stockholm, SE)
Sigrid Eldh (Ericsson AB – Stockholm, SE)
Andy Zaidman (TU Delft, NL)

For support, please contact

Jutka Gasiorowski for administrative matters

Shida Kunz for scientific matters


List of Participants
Shared Documents
Dagstuhl Seminar Schedule [pdf] (Update here)

(Use seminar number and access code to log in)


This seminar brings together researchers and practitioners who share an interest in the quality assurance perspective of release engineering and DevOps. The overarching goal is twofold: (1) sharing views on the state of the art and state of the practice regarding techniques to ensure quality within time-constrained release cycles; (2) identifying challenges and best practices that are relevant both for the software industry and as scientific investigations.

The bridge between quality assurance and release engineering is part of a wider movement to "shift right" the testing activities. For example, companies are increasingly applying A/B tests on subsets of their actual users, i.e., in production, to decide about design questions, or are using canary releases to test (and possibly roll back) a new version in the field. As such, “shift right” test strategies are becoming a key approach to reconcile the fast delivery of increasingly complex software systems with a level of quality that is higher than ever before.

Seminar topics to be included are:

Reconciling industry and academic perspectives

Both academia and industry (either open or closed source) participate in building techniques to ensure high quality within release engineering and DevOps. Yet, their understanding of these techniques varies in terms of implementation and, more importantly, with respect to the issues that need to be resolved to improve the applicability and efficiency of DevOps. The seminar aims to align academic perspectives with the industry’s needs, while sharing recent academic breakthroughs and techniques with industry.

Shift left versus shift right

Early testing (shift left) has been advocated as a good practice for decades. Meanwhile, DevOps and associated methodologies push for late (shift right) approaches to testing. The seminar wants to discuss both approaches and how one should balance both views to achieve effective quality assurance for release engineering processes.

Best practices

Can we define a set of best practices for implementing effective release engineering and DevOps pipelines? Aspects of these go beyond mere automation of the "build to release" pipeline, and also consider managing multiple system variants in parallel, limiting the impact of a feature change, having fast and automatic support for fault localization and debugging, or dealing with automatic installation and rollback in complex environments and configurations. Other practices advocate the creation of abstraction layers and complex architectures to limit fault propagation, yet this appears to stand in conflict with system testing to release fast.

Technical and organizational DevOps issues

What are both the technical and organizational issues in building effective QA into a release engineering and/or DevOps process? Here, we aim at identifying software quality risks that arise as a consequence of DevOps adoption. These risks can be related to the lack of tools or techniques, to the gap between existing techniques and the requirements for DevOps, but they can also arise because of organizational decisions taken to implement a DevOps process. The exchange between academic and industrial participants will be essential to identify these risks and crystallize challenges for future research.

  Creative Commons BY 3.0 DE
  Bram Adams, Benoit Baudry, Sigrid Eldh, and Andy Zaidman


  • Software Engineering


  • Software testing
  • Software release
  • Software deployment
  • Quality assurance
  • Continuous integration

Book exhibition

Books from the participants of the current Seminar 

Book exhibition in the library, ground floor, during the seminar week.


In the series Dagstuhl Reports each Dagstuhl Seminar and Dagstuhl Perspectives Workshop is documented. The seminar organizers, in cooperation with the collector, prepare a report that includes contributions from the participants' talks together with a summary of the seminar.


Download overview leaflet (PDF).


Furthermore, a comprehensive peer-reviewed collection of research papers can be published in the series Dagstuhl Follow-Ups.

Dagstuhl's Impact

Please inform us when a publication was published as a result from your seminar. These publications are listed in the category Dagstuhl's Impact and are presented on a special shelf on the ground floor of the library.

NSF young researcher support