Evolutionary Test Generation

Summary

The ``Evolutionary Test Generation'' Dagstuhl seminar that was held from September 24th to September 29th 2008. The organisation of the seminar was initiated by the EvoTest project, a project funded by the European Commission under the contract number IST-33472. The goal of our seminar was to bring together researchers from the software testing and evolutionary algorithms communities for the discussion of problems and challenges in evolutionary test generation. This goal has been satisfactorily met and has led to a comprehensive list of open problems and challenges identified and discussed during the seminar. The seminar has been attended by 33 people: 30 were researchers from all over the world working on evolutionary testing, test generation and/or evolutionary computing; 3 were industrial participants with experience and feedback from real-life challenges were present: Microsoft, IBM and Berner & Mattner. The abstract collection indicates the talks that were given by the participants.

Systematic testing is the most widely used method to ensure that a program meets its specification. The effectiveness of testing for quality assurance largely depends on the chosen test suite. Currently, test suites are constructed either manually or semi-automatically from the program code or program specification. For large systems, however, manual test case construction is tedious and error-prone, whereas semi-automatic procedures often achieve only insufficient coverage. Therefore, new methods for the automated generation of "good" test suites are necessary.

Evolutionary adaptive search techniques offer a promising perspective for this problem. Genetic algorithms have been investigated for complex search problems in various fields. Their basic principles are selection, mutation, and recombination. These principles can be beneficially applied to the automated generation and optimisation of test suites, both from code (white-box testing) and specification (black-box testing). However, to make this approach successful in practice, a lot of problems remain to be solved: the question of adequate testing objectives, coverage and reliability measures, representation issues for test cases and test suites, seeding, recombination and mutation strategies, and others.

Results of the discussions: Open problems and future challenges

The future challenges identified at the Dagstuhl seminar have been categorized as follows:

• Theoretical foundations
• Search Technique improvements
• New testing objectives
• Tool environment/testing infrastructure
• New application areas