Benchmarking or system evaluation has a strong tradition as a research method for information retrieval. Details of aggregation of performance over data sets and test buckets and of quantification of results with respect to the target notion of relevance are discussed regularly, and sometimes parametrized to cover several usage scenarios. 

Success of future information systems is only in part due to system qualities of the type typically tested in benchmarking excercises. Collection coverage, interface design, uptime, branding and marketing concerns all have as strong or stronger effect on system take-up and eventual success than the qualities of the retrieval algorithm. The carefully designed evaluation schemes cannot predict usefulness of the enveloping system: they are only intended to test some central components.

Moving from topic- and task-oriented retrieval to other usages such as diversion and entertainment and from text to other media it is even clearer that the bare retrieval component of the system is only a small factor in predicting usefulness of a system. Aggregated quantified measures of whatever type do not model the various usage scenarios that a system may be intended to address. 

Validating the various tests in terms of predicted usage is the crucial further step for establishing system effectiveness and usefulness. This requires system developers to engage in user studies, preferably situated in realistic or near-realistic usage contexts, but including partners competent in the craft of designing and performing user studies is not usually a part of multimedia system development efforts. 

Bridging the two activities of paramterized system benchmarking and realistic or near-realistic usage evaluation is the formulation of use cases. Use cases could be formulated to lock down parameters for informed system evaluation and to establish appropriate contextual constraints for validating usage scenarios for the informed user studies. This would enable informatino flow between system developers and studies of usage and would allow systems to be compared even across various system architectures.

The various commission-funded projects on multimedia information access have been asked by CHORUS, a commission-funded coordination action, to describe the use cases they intend to address. The reports from the participating projects has been composed into a use case typology, intended to capture the most significant dimensions of variation with respect to system evaluation. This work is still under progress, but will be presented at Dagstuhl.