Prof. Matei Zaharia: Composable High-Performance Data Processing with Weld
Seminar | August 31 | 12:30-1:30 p.m. | 405 Soda Hall
Matei Zaharia, Stanford University
The main way users are productive writing software is by composing libraries written by other developers. Unfortunately, the traditional interface for composing software -- function calls that exchange data through memory -- is increasingly inefficient on modern hardware. Even when developers write applications using highly optimized libraries, such as BLAS, their performance can be orders of magnitude below hardware limits due to extensive data movement between these functions. To address this problem, we propose Weld, a runtime for data-intensive applications that can automatically optimize across disjoint libraries. Weld represents computations from different libraries in a simple functional intermediate language (IL) and optimizes across them to generate highly efficient code for parallel hardware. Weld can be integrated into existing libraries such as Spark, TensorFlow, Pandas and NumPy without changing their user-facing APIs. We show that Weld can accelerate applications using these frameworks by as much as 30x.