Tensor Computations: Applications and Optimization

Summary

This seminar was planned for 40 participants, but due to travel restrictions resulting from Covid-19, only 15 were able to attend - though several key talks were delivered via teleconferencing. As a result, the Seminar was very focused, and very productive. Some aspects were lost, perhaps in particular representation in-person of the full breadth of applications communities.

It was very evident from the presentations and lively discussions at the Seminar that the field of "Tensor Computations" is vibrant, multi-faceted, interdisciplinary, and fundamental to progress in a diverse range of important areas which are driving researchers in different fields to search for common foundations and common tools.

One of the communities with an interest in tensor computations can be described as "classical" computational science, focusing, for example, on partial differential equations in fluid dynamics, and electronic structure computations in chemistry and materials science. Tensor contractions have been identified as a powerful way of representing the computational structure in the architecture of compilers for domain-specific languages serving these communities. Exploiting this algebraic intermediate representation in the compiler has enabled important performance optimizations far beyond the scope of conventional compilers based on loop nests and polyhedral techniques.

Another major community is primarily concerned with tensor decomposition - finding low-rank approximations of tensors. This is fundamental to data analytics and machine learning applications. Tensor factorization also provides a powerful framework for deep learning and representation learning, and provides a promising strategy for weight compression in convolutional neural networks.

Tensor contractions, in the form of tensor networks, have enormous importance as a tool for understanding and computation in particle and quantum physics. Indeed mapping the connections between these topics, as exposed through the structure of the tensor network representation, offers an exciting frontier with the potential to underpin these different disciplines with common language and shared software.

The Seminar developed a focus, to some extent as a result of the participants able to attend, on tensor contractions, recognising that this provides a foundation for implementation of numerical methods for tensor decompositions. Revisiting this is a key topic to be addressed in following up this Seminar in the future.

A major focus for progress was identified in characterization of safety and correctness properties - ensuring that tensor contraction expressions are well-formed and meaningful. A related topic that was identified as critical concerns how structure is captured, represented and used. This is not only conceptually valuable, but provides a pathway to exploiting block, band and symmetry structure in generating efficient code.

An open question remains in how to capture, track and exploit the properties of tensors with unstructured (i.e. data-dependent) sparsity.

A key outcome from the Seminar was to recognize the massive replication of efforts in terms of software development. Many tools and libraries are being re-developed within different communities, while failing to share techniques and experience in high-performance implementation. The aim of this Seminar was to address this lack of cohesive, coherent community effort to develop computational building blocks. The results of this effort are being realized in the form of a "white paper", offering a manifesto for how to bridge the discipline divides and realize the potential for tensor computations in the future.