Naiad

Naiad: A Timely Dataflow System. Naiad is a distributed system for executing data parallel, cyclic dataflow programs. It offers the high throughput of batch processors, the low latency of stream proces- sors, and the ability to perform iterative and incremental computations. Although existing systems offer some of these features, applications that require all three have re- lied on multiple platforms, at the expense of efficiency, maintainability, and simplicity. Naiad resolves the com- plexities of combining these features in one framework. A new computational model, timely dataflow, under- lies Naiad and captures opportunities for parallelism across a wide class of algorithms. This model enriches dataflow computation with timestamps that represent logical points in the computation and provide the basis for an efficient, lightweight coordination mechanism. We show that many powerful high-level programming models can be built on Naiad’s low-level primitives, en- abling such diverse tasks as streaming data analysis, it- erative machine learning, and interactive graph mining. Naiad outperforms specialized systems in their target ap- plication domains, and its unique features enable the de- velopment of new high-performance applications.