From Big Data Theory to Big Data Practice

Motivation

Some recent advances in the theory of algorithms for big data – sublinear/local algorithms, streaming algorithms and external memory algorithms – have translated into impressive improvements in practice, whereas others have remained stubbornly resistant to useful implementations. This Dagstuhl Seminar aims to glean lessons for those aspects of these algorithms that have led to practical implementation to see if the lessons learned can both improve the implementations of other theoretical ideas and to help guide the next generation of theoretical advances.

As data has grown faster than RAM, the theory of algorithms has expanded to provide approaches for tackling such problems. These fall into three broad categories:

• Streaming and semi-streaming algorithms
• Sublinear or local algorithms
• External memory algorithms

Each of these areas has a vibrant literature, and many of the results from the theory literature have made their way into practice. Other results are not suitable for implementation and deployment. The seminar aims to address several questions by bringing together algorithmicists from these subcommunities, as well as algorithms engineers. Specifically, we aim to address the following questions:

• What themes emerge from considering practical algorithms from the theory literature?
• Can we use these insights to create new models or to capture interesting new optimization criteria?

By bringing together researchers in these disparate areas and by including researchers in algorithms engineering, we hope to bring to light these deep connections. The goals are to:

• Extract shared lessons to help guide theoretical research towards practical solutions;
• Create a feedback loop where commonalities of practical solutions can help guide future theoretical research;
• Help cross-pollinate these research areas.