September 1 – 4 , 2013, Dagstuhl Seminar 13361

Crowdsourcing: From Theory to Practice and Long-Term Perspectives


Claudio Bartolini (HP Labs – Palo Alto, US)
Tobias Hoßfeld (Universität Würzburg, DE)
Phuoc Tran-Gia (Universität Würzburg, DE)
Maja Vukovic (IBM TJ Watson Research Center – Yorktown Heights, US)

For support, please contact

Dagstuhl Service Team


Dagstuhl Report, Volume 3, Issue 9 Dagstuhl Report
Aims & Scope
List of Participants
Dagstuhl's Impact: Documents available
Dagstuhl Seminar Schedule [pdf]

Press Room


Over the past several years crowdsourcing has emerged as a new research theme, but also as a new service platform and Internet for harnessing the skills of the large, network-connected crowd on-line. Whilst the research community has not just yet recognized crowdsourcing as an entirely new discipline, many research challenges remain open and need to be addressed to ensure its successful applications in academia, industry and public sectors. Crowdsourcing research intersects many existing domains and brings to the surface new challenges, such as crowdsourcing as a novel methodology for user-centered research; development of new services and applications based on human sensing, computation and problem solving; engineering of improved crowdsourcing platforms including quality control mechanisms; incentive design and gamification of work; usage of crowdsourcing for professional business; theoretical frameworks for evaluation. Crowdsourcing, as a new means of engaging human capital online is increasingly having an impact on the Internet and its technical infrastructure, on society, and the future of work.

With crowdsourcing gaining momentum and becoming mainstream, the objective of this Dagstuhl seminar was to lead coordination of research efforts in the different communities, especially in US currently leading the crowdsourcing market and in Europe. The seminar engaged experts from the different research fields (e.g. sociology to image processing) as well as experts from industry with a practical background on the deployment, operation or usage of crowdsourcing platforms. From industry, real-world problem statements, requirements and challenges, position statements, innovative use cases, and practical experiences are tackled and discussed. The collection and analysis of practical experiences of the different crowdsourcing stakeholders were key outcomes of the Dagstuhl Seminar. The seminar was structured so that the participants use existing use cases, as a driver in the discussion to envisions future perspectives of this domain. To move forward, we identified the need for a common terminology, classification and taxonomy of crowdsourcing systems, as well as evaluation frameworks; and have already proposed a blueprint of the same. The impact of crowdsourcing from different perspectives has been discussed, by participants' viewpoints stemming from societal, business, economic, legal and infrastructure perspectives.

From platform provider side, Nhatvi Nguyen (Sec. 3.11) showed the actual challenges in operating a crowdsourcing platform. As industry use case, the example of enterprise crowdsourcing was presented by Maja Vukovic (Sec. 3.14), where the rapid generation of a snapshot of the state of IT systems and operation is conducted by means of crowdsourcing. This allows for massive cost savings within the company by uncovering knowledge critical to IT services delivery. Crowdsensing is another industry use case presented in the seminar by Florian Zeiger (Sec. 3.15). Environmental sensing in the area of safety and security was discussed from industry point of view along with the challenges and open questions, e.g. user privacy, data quality and integrity, efficient and reliable data collection, as well as architectural decisions and flexible support of various business models. A concrete application for crowdsensing is radiation sensing as shown by Shinichi Konomi (Sec. 3.7).

Beyond this, there were also discussions on multimedia related use cases. Crowdsourcing can be efficiently used for describing and interpreting multimedia on the Internet and allows to better address other aspects of multimedia with meaning for human beings. Martha Larson (Sec. 3.10) provided examples of these aspects like the emotional impact of multimedia content, and judgments concerning which multimedia is best suited for a given purpose. Klaus Diepold (Sec. 3.6) applied crowdsourcing to move subjective video quality tests from the lab into the crowd. The resulting ratings are used to train mathematical model for predicting subjective quality of video sequences. Multivariate data analysis tools are recommended to incorporate contextual information to further validate the mathematical model. Vassilis Kostakos (Sec. 3.8) showed that the data quality of appropriate subjective tests may be increased by using public displays and touch screens in cities compared to online surveys. While gamification pops up as buzzword aiming among others at increased data quality, Markus Krause (Sec. 3.9) mentioned that the player should be put first i.e. the desires of player are paramount. In particular, task and game ideas need to be able to be linked, while fun has to be the main motivator for the game.

General approaches to improve crowdsourcing and the resulting data quality were a topic of interest by several participants. Gianluca Demartini (Sec. 3.5) proposes to model workers in the crowd as basis for quality assurance mechanisms. Alessandro Bozzon (Sec. 3.2) demanded for better conceptual abstractions for crowd tasks and processes design and (automatic) generation; better understanding of crowds properties such as (soft and hard) skills, reliability, availability, capacity, precision; and better tools for measuring and driving worker engagement. Cristina Cabanillas (Sec. 3.3) considered the human resource management aspects starting from workflows to crowdsourcing. Abraham Bernstein (Sec. 3.1) discussed human computers as part of computational processes, however, with their own strengths and issues. The three traits on human computation, that are motivational diversity, cognitive diversity, and error diversity, are embraced as strengths instead of weaknesses. While the main focus of the seminar was on technical challenges, the potential impact and long-term perspectives were discussed from an interdisciplinary point of view too, given the social and human aspects of crowdsourcing. Those issues were also raised by Phuoc Tran-Gia (Sec. 3.13) and Joseph G. Davis (Sec. 3.14). Overall there were 22 participants from 9 countries and 16 institutions. The seminar was held over 2.5 days, and included presentations by researcher and specific hands-on discussion sessions to identify challenges, evaluate viewpoints and develop a research agenda for crowdsourcing. While the abstracts of the talks can be found in Section 3, a summary of the discussions arising from those impulse talks is given in Section 7. Additional abstracts and research statements without any presentation in the plenary are also included in the report in Section 4. The different aspects of crowdsourcing were discussed in more detail in four different working groups formed during the seminar: (W1) long-term perspectives & impact on economics in five years, (W2) theory: taxonomy and dimensions of crowdsourcing, (W3) industry use cases, (W4) crowdsourcing mechanisms and design. The summary of those working groups can be found in Section 5.

Please note that a related seminar on "Cloud-based Software Crowdsouring" (Dagstuhl Seminar 13362), organized by Michael N. Huhns, Wei Li, Martin Schader and Wei-Tek Tsai, took place in parallel to this seminar. We held a joint social event and a session on discussing research challenges and planned publications. In this late night session, on one hand ethical issues in the area of crowdsourcing were raised in a stimulus talk by Martha Larson (TU Delft). On the other hand, Munindar P. Singh (North Carolina State University) intended to provoke with his talk on the critique of current research in the area of social computing and crowdsourcing. A summary can also be found in Section 7.

A comprehensive list of open problems and challenges in the area of crowdsourcing as observed and stated by the participants is another key outcome of the seminar which is provided in Section 6.

Summary text license
  Creative Commons BY 3.0 Unported license
  Tobias Hoßfeld and Phuoc Tran-Gia and Maja Vukovic


  • Mobile Computing
  • Networks
  • Society / Human-computer Interaction


  • Crowdsourcing
  • Human computation
  • Mobile crowdsourcing
  • Enterprise crowdsourcing


In the series Dagstuhl Reports each Dagstuhl Seminar and Dagstuhl Perspectives Workshop is documented. The seminar organizers, in cooperation with the collector, prepare a report that includes contributions from the participants' talks together with a summary of the seminar.


Download overview leaflet (PDF).

Dagstuhl's Impact

Please inform us when a publication was published as a result from your seminar. These publications are listed in the category Dagstuhl's Impact and are presented on a special shelf on the ground floor of the library.


Furthermore, a comprehensive peer-reviewed collection of research papers can be published in the series Dagstuhl Follow-Ups.