September 1 – 4 , 2013, Dagstuhl Seminar 13362

Cloud-based Software Crowdsourcing


Michael N. Huhns (University of South Carolina – Columbia, US)
Wei Li (Beihang University – Beijing, CN)
Wei-Tek Tsai (Arizona State University – Tempe, US)


Wenjun Wu (Beihang University – Beijing, CN)

For support, please contact

Dagstuhl Service Team


Dagstuhl Report, Volume 3, Issue 9 Dagstuhl Report
Aims & Scope
List of Participants
Dagstuhl's Impact: Documents available
Dagstuhl Seminar Schedule [pdf]


Crowdsourcing software development or software crowdsourcing is an emerging software engineering approach. Software development has been outsourced for a long time, but the use of Internet with a cloud to outsource software development to the crowd is new. Most if not all software development tasks can be crowdsourced including requirements, design, coding, testing, evolution, and documentation. Software crowdsourcing practices blur the distinction between end users and developers, and allow the co-creation principle, i.e., a regular end-user becomes a co-designer, co-developer, and co-maintainer. This is a paradigm shift from conventional industrial software development to a crowdsourcing-based peer-production software development. This seminar focused on the notion of cloud-based software crowdsourcing, with the following goals:

  1. to establish a theoretical framework for applying software crowdsourcing, and identify the important design patterns and highly interactive and iterative processes in a cloud-based infrastructure.
  2. to propose and design a reference architecture for software crowdsourcing
  3. to develop and finalize the research roadmap for software crowdsourcing for the next five years

The grand research challenge in cloud-based software crowdsourcing is how to embrace elements from the two aspects: cloud infrastructure and software crowsourcing. Metaphorically, it can be regarded as synergy between two clouds -- machine cloud and human cloud, towards the ultimate goal of developing high-quality and low cost software products. This seminar intended to bring together scientists from both fields to tackle the major research problems in this emerging research area.

More than twenty researchers, who work on different domains such as crowsourcing, human-computer interaction, cloud computing, service oriented computing, software engineering and business management attended the seminar. In addition to regular 5-minute talks from every participant in the seminar, the organizer arranged a keynote speech delivered by Prof Schahram Dustdar, which summarizes large-scale collective problems solving research enabling software crowsourcing. The topics covered by their presentations can be roughly categorized into three groups: software crowdsourcing process and models, crowdsourcing cloud infrastructure and human crowd management. To promote in-depth discussion among these topics, we also divided people into five discussion groups including:

Crowd Source Software Engineering Design-Group: This group identified the three main areas in the design of software crowdsourcing: processes, models, and techniques. It highlighted the importance of standardized generic models of software crowdsourcing study, and explored multiple crowdsurcing techniques, especially virtual team formation and quality assessment.

Worker-centric design for software crowdsourcing: This group focused on the crowd management in software crowdsourcing and aimed to answering the question about how to make a sustainable software crowdsourcing industry. Discussion in the group covered the major issues such as careers and reputation development of workers, trust among workers and ``employers'' (task solicitors) on crowdsourcing markets, virtual team selection and team building.

Cloud-based Software Crowdsourcig Architecture: This group discussed the possible common architectures of crowd-sourcing applications and explored two complementary architectural approaches.

Experimentation Design for Software Crowdsourcing: The central topic of this group is about how to design a valid and reproducible experiment for software crowdsourcing research. The group had extensive discussion on software crowdsourcing experiment approaches and the major crowdsourcing infrastructures.

Infrastructure and Platform: This group reviewed the motivations to construct the crowdsourcing platform, analyzed architecture design issues, and proposed a educational platform for software crowdsourcing.

During the session of our seminar, Dagstuhl also set up a parallel seminar named "Crowdsourcing: From Theory to Practice and Long-Term Perspectives", which mostly focused on general crowdsourcing research and service platforms. Software crowdsourcing can be regarded as one of the most complex crowdsourcing activities that often need intense dedication from workers with high-level skills of software engineering. Thus, there are some interesting overlapping areas such as worker incentive and quality assurance, between our seminar and the parallel seminar. To foster collaboration among the two groups, we hold a joint discussion session for introducing and sharing findings from each group, followed by an evening session with two presentations from the general crowdsourcing group.

We believe this seminar is a good start for software crowdsourcing research. Finding and consensus generated from the seminar have been formalized in the wiki page of software crowdsourcing ( to give a clear definition and initial reference architecture of cloud-based crowdsourcing software development. More efforts will be put into the growth of the research community and production of joint research publications.

Summary text license
  Creative Commons BY 3.0 Unported license
  Michael N. Huhns, Wei Li, and Wei-Tek Tsai


  • Modelling / Simulation
  • Society / Human-computer Interaction
  • Software Engineering


  • Software Engineering
  • Crowdsourcing
  • Socio-technical ecosystem
  • Cloud Computing


In the series Dagstuhl Reports each Dagstuhl Seminar and Dagstuhl Perspectives Workshop is documented. The seminar organizers, in cooperation with the collector, prepare a report that includes contributions from the participants' talks together with a summary of the seminar.


Download overview leaflet (PDF).

Dagstuhl's Impact

Please inform us when a publication was published as a result from your seminar. These publications are listed in the category Dagstuhl's Impact and are presented on a special shelf on the ground floor of the library.


Furthermore, a comprehensive peer-reviewed collection of research papers can be published in the series Dagstuhl Follow-Ups.