Evaluating Embodied Conversational Agents

Summary

In the past years we have seen an increasing number of ECAs (Embodied Conversational Agents), as results (and sometimes also as medium) of academic research as well as in commercial applications in fields like information providing (news reader, info kiosk), user interfaces for deaf (lip-reading, signing avatars), education, commerce and entertainment. Due to the novelty of the field and the complexity of the problems, different paradigms and tools have been used to produce the individual ECAs, often tailored to the special needs of an application or of a research issue.

The comparison and re-use of ECAs has not been dealt with to a sufficient degree, in spite of the high costs of developing yet new ECAs. There have been studies that indicate added value of particular features of ECAs, but it remains to be determined in what situations fully functional ECAs are advantageous, and for what purpose. The sporadic evaluation experiments concentrated on special issues and under restricted circumstances. All the same, several of these partial evaluations have already produced surprising and for the future development and application of ECAs essential results, e.g. on the different preferences of different users.

For the evaluation and design guidelines of ECAs one should rely on results and methodologies of related fields like HCI, psychology, cultural anthropology and even arts. The different language and methodologies of these disciplines and lack of common forums have hindered the cross-fertilization. Feedback from the industry on commercial applications would also very useful.

Motivated by the above observations, the main objectives of the seminar were:

• to review the results, tools and resources;
• identify key problems and future research directions;
• initiate further activities to trigger and dissimilate work on ECA design and evaluation.