Theoretical Foundations of Storage Systems

Summary

Storage systems, including databases and file systems, are at the heart of all large data applications. Recently, some storage systems have incorporated theoretical advances in data organization techniques, with substantial improvements in performance. However, there is little contact between the systems designers who build storage systems and theoreticians who design new ways to organize data. This Seminar worked to bridge this gap, to the benefit of both communities and to improve the design of all storage systems.

External-memory algorithms are those where the data is too large to fit in memory, and hence needs to be stored on disk and accessed using I/O. Algorithmic analysis of such algorithms therefore focuses on the number of I/Os needed to complete a computation, rather than the number of instructions. This is because an I/O can be many orders of magnitude slower than a machine instruction and therefore I/Os can be the bottleneck in such computations.

The theoretical analysis of such external-memory algorithms has produced many exciting results in the last two decades. Many of these are directly relevant to practical applications, but only a few have made the leap to deployment. This Seminar brought together theoreticians, who have an extensive understanding of the state of the art in external-memory data structures, and storage systems researchers and practitioners, who understand the details of the problems that need to be solved.

Specific questions that were addressed in the workshop include the following:

• How can we use the huge improvements in string data structures to improve storage systems that manipulate strings? Many data structures, such as LSMs and B^varepsilon-trees, rely heavily on the assumption that keys are indivisible and small.
• How can we use new multi-dimensional data indexes in working systems?
• Many indexes suffer from fragmentation. Are there data structure improvements that would allow efficient storage on disks that are nearly full? Currently, disks are kept only a fraction full because the performance of existing data structures decays dramatically as the disk fills. This suggests another problem:
• How can theoretical models be improved to capture such issues as:
  • full disk: The external memory model, the disk is of unbounded size.
  • sequential access: Both hard disks and SSDs require very large blocks of sequential I/O to capture a large fraction of bandwidth. The external-memory model assumes that disks are random access.
• Concurrency: Can data structures be designed to exploit memory locality on disk while maintaining concurrency in RAM?