https://www.dagstuhl.de/9738

September 15 – 19 , 1997, Dagstuhl Seminar 9738

Performance Evaluation - Origins and Directions

Organizer

G. Haring (Wien), Ch. Lindemann (GMD-FIRST Berlin), M. Reiser (Zürich)

For support, please contact

Dagstuhl Service Team

Documents

External Homepage
Dagstuhl's Impact: Documents available
Dagstuhl-Seminar-Report 189

Goals of this Dagstuhl Seminar

Performance Evaluation is a discipline of Computer Science for some thirty years. It seems time to take stock of what we were doing. That is, provide answers to the following questions:

  • What are its scientific contributions?
  • What is its relevance in industry and business?
  • What is its standing in academia?
  • Where is the field headed?
  • What are its success stories and failures?
  • What are its current burning questions?

At this Dagstuhl seminar, we were taking the term workshop literally and have broken up in five working groups. These groups worked in parallel on different areas of Performance Evaluation. Each group prepared a written report during the stay at Dagstuhl. These reports constitute the remainder of this text. The ultimate goal of this seminar is to publish a White Book on Performance Evaluation that will become a catalyst for the advancements of this field as well as a book of reference. Furthermore, each group discussed Success Stories, Failures, Impact of Methods, and Burning Questions of Performance Evaluation. A summary of this discussion is given in the following:

Success Stories

  • Computer architecture, in particular RISC, cache coherence, and multiprogramming.
  • IEEE 802 suite of protocols, in particular medium access control level.
  • Data networks, in particular TCP/IP.
  • Storage systems, in particular RAID and virtual memory.
  • Operating systems, in particular timesharing, MVS, and scheduling.
  • Realization and implementation of methodological results in easy-to-use software packages.

Failures

  • Late consideration of performance issues in system design process.
  • Bad quality of measured input data.
  • Wrong level of abstraction.
  • Educational failures
    • Lack of making known methods to customers.
    • Lack of training of modeling skills
Impact of Methods
  • BCMP theorem made impact to e.g., capacity planning, multiprogramming, and communication networks.
  • MVA algorithm substantially increased the applicability of BCMP queueing networks with several customer classes for supporting the design of computer and communication systems.
  • Particular stochastic models with closed-form solutions made impact to design of IEEE 802 medium access control protocols.
  • Trace-driven simulation made impact to cache and processor architecture design
Burning Questions
  • Complexity of today’s systems
    • Scalability of performance evaluation techniques.
    • Validation
    • Parameter estimation, large amount of measured data.
    • Develop methods for handling very large models with sufficient level of detail
  • Bridging the gap between theoretical and applied work:
    • Seamless integration of performance models into design process.
    • Application-specific parametrizable performance models.
  • Academic education and university research:
    • Curriculum for computer scientists and engineers has not enough exposure to Performance Evaluation methods.
    • Faculty positions and research funding is inadequate for Performance Evaluation methodology. I.e., is pure Performance Evaluation research an endangered species?
  • Acceptance in industry:
    • Performance Evaluation experts must be routinely be involved in system design process.
    • Short development cycle, short live cycles of technologies.
    • Education of university graduates.

Documentation

In the series Dagstuhl Reports each Dagstuhl Seminar and Dagstuhl Perspectives Workshop is documented. The seminar organizers, in cooperation with the collector, prepare a report that includes contributions from the participants' talks together with a summary of the seminar.

 

Download overview leaflet (PDF).

Dagstuhl's Impact

Please inform us when a publication was published as a result from your seminar. These publications are listed in the category Dagstuhl's Impact and are presented on a special shelf on the ground floor of the library.

Publications

Furthermore, a comprehensive peer-reviewed collection of research papers can be published in the series Dagstuhl Follow-Ups.