The seminar was conducted as an open space. On the first day the participants collected the topics they wanted to discuss and present. Based on this an agenda for the whole week was developed with plenary sessions and working sessions in parallel tracks. In the course of the seminar the agenda was restructured based on the needs of the participants. Every day the participants assigned themselves to the parallel sessions. In the plenary sessions overview talks were given, summaries of the parallel tracks presented and discussed. This scheme ensured that the groups in each track were small enough for intensive discussions, but on the other hand every participant was informed about the overall results. In the final session the results were put together to a general picture of the pros and cons of the integration of requirements engineering and quality assurance.
The outcome of this seminar is summarized in the following. The main contribution of the seminar was a better understanding
- between the different communities: requirements engineering, formal methods, inspection and testing
- of practical problems of customer/supplier relationships (C/SR) and
- of the practical and scientific problems of integrating requirements engineering (RE) and quality assurance (QA).
In the closing session many participants emphasized that they will base future work on this understanding and work in particular on the synergies of these techniques. All participants agreed that an integrated RE and QA process would be of utmost practical value. In the following we first highlight the findings wrt. C/SR. Then we sketch an integrated RE and QA process which evolved during the course of the seminar. Finally, we list the most important research questions identified in the discussions.
Customer and Supplier
Many presentations and discussions of the seminar dealt with the integration of RE and QA without regard of the issue of C/SR. Thus, during the seminar the question arose whether there is anything specific to integration of RE and QA for C/SR. In particular, the participants from industry felt that C/SR issues are very important. Thus, the workshop by Frank Houdek and Maud Schlich aimed at identifying these specific issues (see also abstract 3.7).
First of all, it was agreed that C/SR are quite complex, but also quite diverse. Customers are interested to buy or procure software. Suppliers produce software from scratch or by adapting a product. The intensity of the relationship varies between joint development work and very formal settings where only management or lawyers of both sides interact.
Second, it was agreed that both, customers and suppliers, need to be involved in RE and QA activities. Thus, both of them have responsibility for the success of these activities. Also, it is equally important for them that this process is successful.
So, the following issues for RE and QS in C/SR-situation were identified:
- The integrated RE and QA process must accommodate both, customer and suppliers, that means people with quite diverse backgrounds and interests. In addition, C/SR-situations typically induce issues of geographical distribution, cultural and organization differences on the process.
- For C/SR projects an integrated RE & QA process is especially important, because these are exactly the activities where customer and supplier interact.
The following picture shows the outline of an integrated RE and QA process. Boxes represent artifacts, ovals represent activities. Arrows represent dependencies.
Roles involved in this process are not shown. It depends on the situation which role is involved in which activity. In the upper half of the picture the RE and Design artifacts are shown. One important distinction here – which is too often not recognized when the RE and the formal method community interact – is that there are two kinds of requirements:
- Goal, domain and product-level requirements describe the effects of the software system in its environment as well as the features of the software system that induce these effects.
- Design-level requirements detail the product-level requirements such that developers exactly know what to build.
The first kind of requirements is the basis for the discussions with the users and customers, the second kind is the basis for the developers. Customers must approve both, since only they can assure that design-level requirements detail the other kinds.
The arrows between these artifacts indicate activities to develop code from the requirements. To ease change and reuse, it is important that the resulting dependencies between the different artifacts are traced. In the lower half of the picture the QA artifacts and activities are shown. The outcome of the activities is requirements and code defects. The arrows indicate the inputs and outputs of these activities as well as transformations between the artifacts (where the activities are not explicitly shown). A solid arrow indicates that there are powerful automation tools to support the activity delivering the result.
One important point here is that all QA activities can be used to identify requirements defects and code defects. (Note that for the purpose of this discussion it is not necessary to distinguish between failures and defects.)
So inspections can be used to find requirements defects and code defects (and of course also design defects, not shown here).
Similarly, metrics can be applied to requirements artifacts and code artifacts. However, typically they will only be applied to design-level requirements, because at this level there is an emphasis on formal properties.
The development of test cases (out of the product-level requirements or the design-level requirements) helps to identify requirements defects. . If the user manual is developed early, it is also possible to identify test cases from the user manual.
Powerful automatic analysis of the requirements documents is only possible, if requirements are transformed into an executable specification. There is evidence from several industrial applications that this transformation is worth the effort when starting from design-level requirements. It is an open question whether an executable specification can be directly developed from product-level requirements (therefore the arrow is highlighted). The problem is that product-level requirements are often not detailed enough and not stable enough. If there is a formal specification of the requirements as well as of the realization (code), then the analysis tools can also reveal code defects. Furthermore, it is possible to automatically derive test cases from executable requirements specifications. In this case the executable specification is used as an oracle for the outcome of the test cases
Defects detected during testing are either code defects (when the code does not conform to the specified requirements and the specified requirements are correct) or requirements defects (when the code does not conform to the specified requirements, but to the customer's needs). One can use the evidence of requirements defects identified early to tailor the set of test cases to specific problems expected in the code. However, so far this step has not been exploredFinally, the analysis of the defects helps to make experiences explicit. These can be captured in standards, templates and patterns and serve to improve following projects.
Altogether, this picture reveals many possibilities for QA of RE through inspection, metrics, formal methods and testing. There was general agreement that all possibilities are worthwhile in some contexts. The major open questions concern the selection of the right technique depending on the context and effort savings through synergies of these techniques. These questions are detailed in the following section.
The seminar lead to a common understanding of the possibilities of integrating RE and QA techniques. At the same time it revealed many open issues wrt. the problems and benefits of such an integration - in particular, given the cost and time restrictions of industrial projects
- Documentation effort:
- In practice, it is not possible to specify requirements completely. However, QA can only work on the basis of requirements documents. Thus, one needs guidance on where to focus the requirements engineering effort. Here experiences from previous QA-activities (e.g. defects classes) can help.
- Technique selection:
- It is not possible in general to execute all QA activities on all artifacts. Thus, one needs guidance when to select which technique. So for example, it would be nice to guide the selection depending on the application domain (e.g. formal specification seems to be more suitable for embedded systems than for information systems) or on the project context (e.g. size of the project, people involved) or on the classes of defects investigated. Clearly, the technique has to be understandable by the participants. One important prerequisite for creating such guidance is data about the effort and defect detection rates of the different techniques.
- Synergies between QA-techniques:
- There are three ways to achieve synergies
- Through substitution: this means that one QA-activity is carried out instead of another one. So, for example one could substitute a certain set of tests through inspection. (e.g. specific defects only through inspection).
- Through preparation: this means that one QA-activity is used to alleviate another technique. So, for example, automatic derivation of test cases from formal specification eases the testing process.
- Through leveraging of QA results: This means that analysis of the defects found through one technique is used to guide the application of another technique in the same project. For example, defect rates from requirements inspections can be used to focus testing.
- Synergies between RE and QA:
- There are four ways to achieve synergies
- Through substitution: this means that a RE artifact or activity is substituted by a QA artifact or activity. For example, instead of requirements documents, test cases are written. This is one measure now advocated in Extreme Programming. In this case the test cases serve as the basis for the development.
- Through preparation: this means that during RE activities are performed and artifacts created that are especially suited to perform QA activities. For example, use cases ease the derivation of test cases.
- Through leveraging of QA results: This means that analysis of the defects found through QA is used to guide the RE activities of the next project. For example, defect rates from requirements inspections can be used to improve the elicitation or specification activities.
- Through mixed teams: This means that QA people are involved early in RE activities, e.g. as inspectors. Another example is to let requirements engineers write the test cases so that they get to know the level of details needed by the testers.
- Aybüke Aurum (UNSW - Sydney, AU) [dblp]
- Daniel M. Berry (University of Waterloo, CA) [dblp]
- Ole Fischer (ERICSSON Eurolab - Herzogenrath, DE)
- Mario Friske (Fraunhofer Institut - Berlin, DE)
- Martins Gills (Riga Information Technology Institute, LV)
- Martin Glinz (Universität Zürich, CH) [dblp]
- Wolfgang Grieskamp (Microsoft Research - Redmond, US) [dblp]
- Hans-Gerhard Gross (FhG IESE - Kaiserslautern, DE)
- Constance L. Heitmeyer (Naval Research - Washington, US) [dblp]
- Frank Houdek (Daimler R&D - Ulm, DE)
- Filippo Lanubile (University of Bari, IT)
- Soren Lauesen (University of Copenhagen, DK)
- Dirk B. Meyerhoff (SQS AG - Köln, DE)
- Barbara Paech (Universität Heidelberg, DE) [dblp]
- Stacy Prowell (University of Tennessee, US) [dblp]
- Isabel Ramos (Escola Superior de Tecnologia e Gestco - Viana, PT)
- Gianna Reggio (University of Genova, IT)
- H. Dieter Rombach (Fraunhofer ITWM - Kaiserslautern, DE)
- Bernhard Schätz (TU München, DE) [dblp]
- Maud Schlich (FhG IESE - Kaiserslautern, DE)
- Dirk Seifert (TU Berlin, DE)
- Natasha Sharygina (Carnegie Mellon University - Pittsburgh, US) [dblp]
- Andreas Spillner (Hochschule Bremen, DE)
- Carsten Sühl (Fraunhofer Institut - Berlin, DE)
- Jan Tretmans (Radboud University Nijmegen, NL) [dblp]
- Rudolf van Megen (SQS AG - Köln, DE)
- Antje von Knethen (FhG IESE - Kaiserslautern, DE)
- Roel J. Wieringa (University of Twente, NL) [dblp]
- Burkhart Wolff (ETH Zürich, CH) [dblp]
- Thomas Zink (Daimler R&D - Ulm, DE)