16. – 20. März 1998, Dagstuhl-Seminar 98111

Evaluation and Validation of Computer Vision Algorithms


R. Haralick (Seattle), R. Klette (Auckland), S. Stiehl (Hamburg), M. Viergever (Delft)

Auskunft zu diesem Dagstuhl-Seminar erteilt

Dagstuhl Service Team


Externe Homepage
Dagstuhl's Impact: Dokumente verfügbar
Dagstuhl-Seminar-Report 205


This Seminar addressed a subject which has been under active discussion in computer vision for several years. The evaluation and validation of algorithms is of basic importance for the configuration of computer vision applications. In the ideal case certain "data sheets" should allow to qualify algorithmic solutions in a specific context, e.g. defined by image data, goal of image analysis, or software environment ("edge detection is not equal to edge detection"). There is a lack of methodological fundamentals in the field of performance analysis.

The Seminar follows typical topics in computer vision. There are some fields in computer vision where an extensive literature about evaluation and validation of algorithms is available, e.g. motion analysis, digital geometry, shape reconstruction, or image registration. But still we are quite far away from "final answers" in these fields. "Compression" could play the role of the "good example" because the evaluation of compression methods is well developed based on comparisons of compression rates, behaviour on specific test sequences, and evaluation of image quality after decompression.

The range of fundamental problems encompasses, e.g.,

  • the value of synthetic images in experimental computer vision,
  • the selection of a representative set of real images related to specific domains and tasks,
  • the definition of ground truth given different tasks and applications,
  • the definition of experimental test-beds,
  • the analysis of algorithms with respect to general features such as computation time, convergence, stability, or domains of admissible input data,
  • the definition and analysis of performance measures for classes of algorithms,
  • the role of statistics-based performance measures,
  • the generation of data sheets with performance measures of algorithms supporting the system engineer in his configuration problem, etc.

The workshop thus attempted to bring together experienced colleagues from the international computer vision community both to discuss the state-of-the-art and to formulate recommendations for future activities.

36 talks, grouped in several topical sessions, were given by 32 speakers from 14 countries. Out of a total of 41 participants, 11 were from Germany; 5 from The Netherlands and the U.S.A. each; 3 from Denmark and United Kingdom each; 2 from Canada, Czech Republic, Japan, New Zealand, and Slovenia each; and 1 participant from Hungary, Israel, Slovakia, and Taiwan each. In addition to the oral presentations, four working groups - partly working during wood-walking tours – were established to discuss issues of common interest in more detail (see appendix for summaries). The participation of a number of younger scientists from Denmark, Germany, The Netherlands, and United Kingdom was rendered possible through financial support from the TMR (Training and Mobility of Researchers) programme of the European Union which is gratefully acknowledged. Moreover we are pleased to state that the presentations at our meeting were of such a high quality that a refereed proceedings book is planned to be published soon by Kluwer Academic Publishers in the computational Imaging and Vision series.

We are also grateful to the administration of the Dagstuhl enterprise for creating such an inspiring and free-of-duty environment as well as for providing excellent facilities which significantly contributed to the success of our meeting.

Eventually the workshop has stimulated different future activities, ranging from the establishment of an algorithmic web site for the international computer vision community, to the recommendation of organising a similar meeting at Schloß Dagstuhl in Y2K on the subject of theory, methodology, and practice of performance measures.

Dagstuhl-Seminar Series


In der Reihe Dagstuhl Reports werden alle Dagstuhl-Seminare und Dagstuhl-Perspektiven-Workshops dokumentiert. Die Organisatoren stellen zusammen mit dem Collector des Seminars einen Bericht zusammen, der die Beiträge der Autoren zusammenfasst und um eine Zusammenfassung ergänzt.


Download Übersichtsflyer (PDF).

Dagstuhl's Impact

Bitte informieren Sie uns, wenn eine Veröffentlichung ausgehend von Ihrem Seminar entsteht. Derartige Veröffentlichungen werden von uns in der Rubrik Dagstuhl's Impact separat aufgelistet  und im Erdgeschoss der Bibliothek präsentiert.


Es besteht weiterhin die Möglichkeit, eine umfassende Kollektion begutachteter Arbeiten in der Reihe Dagstuhl Follow-Ups zu publizieren.