While the laboratory was once the primary location where researchers studied IIR, researchers are increasingly using different delivery modes to gather data from human subjects.  Researchers have used mechanical Turk to gather relevance assessments (https://www.mturk.com/mturk/welcome), online games such as ESP to generate training data for machine classifiers (see http://www.gwap.com/gwap/ for more examples) and remote usability testing to interact with geographically dispersed clients. One of the primary benefits of conducting a study in a laboratory is that the researcher has increased control over the study situation.  Many sources of variance can be identified and controlled and alternative explanations can be ruled-out.  Furthermore, the researcher can ensure that a standard protocol is followed and that subjects are not engaged in extracurricular activities.  When studies are conducted online the researcher is no longer able to fully monitor the subject and cannot even be certain that there is a real human participant.  However, an online delivery mode allows researchers to study a wider range of more diverse users and researchers do not always have to be present when subjects are participating (in other words, you can collect data while you eat, sleep, run and holiday!).  It could further be argued that studies delivered via the Internet are particularly well-suited to IIR since a large amount of information seeking and retrieval takes places in online environments.

In this talk, I will review different study delivery modes that have been used in IIR, as well as in other related research areas. I will present preliminary results of an IIR experiment that was completed by two cohorts of subjects:  the first cohort completed the study online and the second completed the study in the laboratory.  Finally, I will discuss the benefits and limitations of these various delivery modes with respect to different types of IIR studies (e.g., experiment, evaluation, usability test).  It is hoped that this talk will generate further consideration and discussion of which techniques are appropriate for IIR research.