15. – 19. September 1997, Dagstuhl-Seminar 9738

Performance Evaluation - Origins and Directions


G. Haring (Wien), Ch. Lindemann (GMD-FIRST Berlin), M. Reiser (Zürich)

Auskunft zu diesem Dagstuhl-Seminar erteilt

Dagstuhl Service Team


Externe Homepage
Dagstuhl's Impact: Dokumente verfügbar
Dagstuhl-Seminar-Report 189

Goals of this Dagstuhl Seminar

Performance Evaluation is a discipline of Computer Science for some thirty years. It seems time to take stock of what we were doing. That is, provide answers to the following questions:

  • What are its scientific contributions?
  • What is its relevance in industry and business?
  • What is its standing in academia?
  • Where is the field headed?
  • What are its success stories and failures?
  • What are its current burning questions?

At this Dagstuhl seminar, we were taking the term workshop literally and have broken up in five working groups. These groups worked in parallel on different areas of Performance Evaluation. Each group prepared a written report during the stay at Dagstuhl. These reports constitute the remainder of this text. The ultimate goal of this seminar is to publish a White Book on Performance Evaluation that will become a catalyst for the advancements of this field as well as a book of reference. Furthermore, each group discussed Success Stories, Failures, Impact of Methods, and Burning Questions of Performance Evaluation. A summary of this discussion is given in the following:

Success Stories

  • Computer architecture, in particular RISC, cache coherence, and multiprogramming.
  • IEEE 802 suite of protocols, in particular medium access control level.
  • Data networks, in particular TCP/IP.
  • Storage systems, in particular RAID and virtual memory.
  • Operating systems, in particular timesharing, MVS, and scheduling.
  • Realization and implementation of methodological results in easy-to-use software packages.


  • Late consideration of performance issues in system design process.
  • Bad quality of measured input data.
  • Wrong level of abstraction.
  • Educational failures
    • Lack of making known methods to customers.
    • Lack of training of modeling skills
Impact of Methods
  • BCMP theorem made impact to e.g., capacity planning, multiprogramming, and communication networks.
  • MVA algorithm substantially increased the applicability of BCMP queueing networks with several customer classes for supporting the design of computer and communication systems.
  • Particular stochastic models with closed-form solutions made impact to design of IEEE 802 medium access control protocols.
  • Trace-driven simulation made impact to cache and processor architecture design
Burning Questions
  • Complexity of today’s systems
    • Scalability of performance evaluation techniques.
    • Validation
    • Parameter estimation, large amount of measured data.
    • Develop methods for handling very large models with sufficient level of detail
  • Bridging the gap between theoretical and applied work:
    • Seamless integration of performance models into design process.
    • Application-specific parametrizable performance models.
  • Academic education and university research:
    • Curriculum for computer scientists and engineers has not enough exposure to Performance Evaluation methods.
    • Faculty positions and research funding is inadequate for Performance Evaluation methodology. I.e., is pure Performance Evaluation research an endangered species?
  • Acceptance in industry:
    • Performance Evaluation experts must be routinely be involved in system design process.
    • Short development cycle, short live cycles of technologies.
    • Education of university graduates.


In der Reihe Dagstuhl Reports werden alle Dagstuhl-Seminare und Dagstuhl-Perspektiven-Workshops dokumentiert. Die Organisatoren stellen zusammen mit dem Collector des Seminars einen Bericht zusammen, der die Beiträge der Autoren zusammenfasst und um eine Zusammenfassung ergänzt.


Download Übersichtsflyer (PDF).

Dagstuhl's Impact

Bitte informieren Sie uns, wenn eine Veröffentlichung ausgehend von Ihrem Seminar entsteht. Derartige Veröffentlichungen werden von uns in der Rubrik Dagstuhl's Impact separat aufgelistet  und im Erdgeschoss der Bibliothek präsentiert.


Es besteht weiterhin die Möglichkeit, eine umfassende Kollektion begutachteter Arbeiten in der Reihe Dagstuhl Follow-Ups zu publizieren.