Evaluation of interactive information retrieval (IIR) systems has typically focused on outputs, neither on outcomes nor on search process. Outputs are products delivered by systems, outcomes are the benefits systems produce for the users.  In evaluations outputs have been the number of relevant items retrieved. The aim of this presentation is to sketch an evaluation framework for IIR, which takes into account search process, and outcomes also to some extent.

The goal of information searching is to support actor’s task performance. Information searching consists of search goals and actions for accomplishing these goals. We suppose that the goals of IR systems are to support actor’s to proceed in the task performance for finding useful information for the task. These consist of the following sub-goals: support in structuring the search topic, supporting search process, and supporting evaluation of the search results.

Evaluation of IIR should focus on assessing to which extent systems support searchers in attaining these three goals, not only on the output of the systems. In the evaluation model search process is divided into five stages, structuring information needs, query formulation, querying, evaluation of results, and the use of information in results. For the first four stages exemplary search tools are introduced, and evaluation measures for these stages are proposed. 

It is proposed that the evaluation concerns both to which extent the system supports attaining search goals and attaining goals of the task. In order to be able to relate search process with its output and outcome, it is necessary to analyze how process variables influence output and outcome variables. It is discussed also to which extent it is possible to evaluate the outcomes of the system.