http://www.dagstuhl.de/15452

November 1 – 4 , 2015, Dagstuhl Perspectives Workshop 15452

Artifact Evaluation for Publications

Organizers

Bruce R. Childers (University of Pittsburgh, US)
Grigori Fursin (cTuning – Cachan, FR)
Shriram Krishnamurthi (Brown University – Providence, US)
Andreas Zeller (Universität des Saarlandes, DE)


1 / 3 >

For support, please contact

Dagstuhl Service Team

Documents

Dagstuhl Report, Volume 5, Issue 11 Dagstuhl Report
Aims & Scope
List of Participants
Shared Documents

Summary

Computer systems researchers have developed numerous artifacts that encompass a broad collection of software tools, benchmarks, and data sets. These artifacts are used to prototype innovations, evaluate trade-offs and analyze implications. Unfortunately, methods used in the evaluation of computing system innovation are often at odds with sound science and engineering practice. The ever-increasing pressure to publish more and more results poses an impediment to accountability, which is a key component of the scientific and engineering process. Experimental results are not usually disseminated with sufficient metadata (i.e., software extensions, data sets, benchmarks, test cases, scripts, parameters, etc.) to achieve repeatability and/or reproducibility. Without this information, issues surrounding trust, fairness and building on and comparing with previous ideas becomes problematic. Efforts in various computer systems research sub-communities, including programming languages/compilers, computer architecture, and high-performance computing, are underway to address the challenge.

This Dagstuhl Perspectives Workshop (PW) brought together stakeholders of associated CSR sub-communities to determine synergies and to identify the most promising directions and mechanisms to push the broader community toward accountability. The PW assessed current efforts, shared what does and doesn't work, identified additional processes, and determined possible incentives and mechanisms. The outcomes from the workshop, including recommendations to catalyze the community, are separately documented in an associated Dagstuhl Manifesto.

License
  Creative Commons BY 3.0 Unported license
  Bruce R. Childers and Grigori Fursin and Shriram Krishnamurthi and Andreas Zeller

Classification

  • Hardware
  • Optimization / Scheduling
  • Software Engineering

Keywords

  • Empirical Evaluation of Software Tools
  • Documentation of Research Processes
  • Artifact Evaluation
  • Experimental Reproducibility

Book exhibition

Books from the participants of the current Seminar 

Book exhibition in the library, ground floor, during the seminar week.

Documentation

In the series Dagstuhl Reports each Dagstuhl Seminar and Dagstuhl Perspectives Workshop is documented. The seminar organizers, in cooperation with the collector, prepare a report that includes contributions from the participants' talks together with a summary of the seminar.

 

Download overview leaflet (PDF).

Publications

Furthermore, a comprehensive peer-reviewed collection of research papers can be published in the series Dagstuhl Follow-Ups.

Dagstuhl's Impact

Please inform us when a publication was published as a result from your seminar. These publications are listed in the category Dagstuhl's Impact and are presented on a special shelf on the ground floor of the library.