Automatic Quality Assurance and Release

Motivation

This seminar brings together researchers and practitioners who share an interest in the quality assurance perspective of release engineering and DevOps. The overarching goal is twofold: (1) sharing views on the state of the art and state of the practice regarding techniques to ensure quality within time-constrained release cycles; (2) identifying challenges and best practices that are relevant both for the software industry and as scientific investigations.

The bridge between quality assurance and release engineering is part of a wider movement to "shift right" the testing activities. For example, companies are increasingly applying A/B tests on subsets of their actual users, i.e., in production, to decide about design questions, or are using canary releases to test (and possibly roll back) a new version in the field. As such, “shift right” test strategies are becoming a key approach to reconcile the fast delivery of increasingly complex software systems with a level of quality that is higher than ever before.

Seminar topics to be included are:

Reconciling industry and academic perspectives

Both academia and industry (either open or closed source) participate in building techniques to ensure high quality within release engineering and DevOps. Yet, their understanding of these techniques varies in terms of implementation and, more importantly, with respect to the issues that need to be resolved to improve the applicability and efficiency of DevOps. The seminar aims to align academic perspectives with the industry’s needs, while sharing recent academic breakthroughs and techniques with industry.

Shift left versus shift right

Early testing (shift left) has been advocated as a good practice for decades. Meanwhile, DevOps and associated methodologies push for late (shift right) approaches to testing. The seminar wants to discuss both approaches and how one should balance both views to achieve effective quality assurance for release engineering processes.

Best practices

Can we define a set of best practices for implementing effective release engineering and DevOps pipelines? Aspects of these go beyond mere automation of the "build to release" pipeline, and also consider managing multiple system variants in parallel, limiting the impact of a feature change, having fast and automatic support for fault localization and debugging, or dealing with automatic installation and rollback in complex environments and configurations. Other practices advocate the creation of abstraction layers and complex architectures to limit fault propagation, yet this appears to stand in conflict with system testing to release fast.

Technical and organizational DevOps issues

What are both the technical and organizational issues in building effective QA into a release engineering and/or DevOps process? Here, we aim at identifying software quality risks that arise as a consequence of DevOps adoption. These risks can be related to the lack of tools or techniques, to the gap between existing techniques and the requirements for DevOps, but they can also arise because of organizational decisions taken to implement a DevOps process. The exchange between academic and industrial participants will be essential to identify these risks and crystallize challenges for future research.