Dryad: distributed data-parallel programs from sequential building blocks. Dryad is a general-purpose distributed execution engine for coarse-grain data-parallel applications. A Dryad application combines computational ”vertices” with communication ”channels” to form a dataflow graph. Dryad runs the application by executing the vertices of this graph on a set of available computers, communicating as appropriate through flies, TCP pipes, and shared-memory FIFOs. The vertices provided by the application developer are quite simple and are usually written as sequential programs with no thread creation or locking. Concurrency arises from Dryad scheduling vertices to run simultaneously on multiple computers, or on multiple CPU cores within a computer. The application can discover the size and placement of data at run time, and modify the graph as the computation progresses to make efficient use of the available resources. Dryad is designed to scale from powerful multi-core single computers, through small clusters of computers, to data centers with thousands of computers. The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.

This software is also peer reviewed by journal TOMS.

References in zbMATH (referenced in 21 articles )

Showing results 1 to 20 of 21.
Sorted by year (citations)

1 2 next

  1. Nanongkai, Danupon; Scquizzato, Michele: Equivalence classes and conditional hardness in massively parallel computations (2022)
  2. Becker, Florent; Montealegre, Pedro; Rapaport, Ivan; Todinca, Ioan: The role of randomness in the broadcast congested clique model (2021)
  3. Czumaj, Artur; Davies, Peter; Parter, Merav: Simple, deterministic, constant-round coloring in congested clique and MPC (2021)
  4. Hao, Rong-Xia; Tian, Zengxian: The vertex-pancyclicity of data center networks (2021)
  5. Czumaj, Artur; Łącki, Jakub; Mądry, Aleksander; Mitrović, Slobodan; Onak, Krzysztof; Sankowski, Piotr: Round compression for parallel matching algorithms (2020)
  6. Fotakis, Dimitris; Milis, Ioannis; Papadigenopoulos, Orestis; Vassalos, Vasilis; Zois, Georgios: Scheduling MapReduce jobs on identical and unrelated processors (2020)
  7. Montealegre, P.; Perez-Salazar, S.; Rapaport, I.; Todinca, I.: Graph reconstruction in the congested clique (2020)
  8. Li, Xiaoyan; Fan, Jianxi; Lin, Cheng-Kuan; Cheng, Baolei; Jia, Xiaohua: The extra connectivity, extra conditional diagnosability and (t/k)-diagnosability of the data center network DCell (2019)
  9. Rompf, Tiark; Amin, Nada: A SQL to C compiler in 500 lines of code (2019)
  10. Convolbo, Moïse W.; Chou, Jerry; Hsu, Ching-Hsien; Chung, Yeh Ching: GEODIS: towards the optimization of data locality-aware job scheduling in geo-distributed data centers (2018)
  11. Haller, Philipp; Miller, Heather; Müller, Normen: A programming model and foundation for lineage-based distributed computation (2018)
  12. Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, Ion Stoica: Ray: A Distributed Framework for Emerging AI Applications (2017) arXiv
  13. Lee, Anthony; Whiteley, Nick: Forest resampling for distributed sequential Monte Carlo (2016)
  14. Wang, Xi; Fan, Jianxi; Jia, Xiaohua; Lin, Cheng-Kuan: An efficient algorithm to construct disjoint path covers of DCell networks (2016)
  15. Philip Chen, C. L.; Zhang, Chun-Yang: Data-intensive applications, challenges, techniques and technologies: a survey on big data (2014) ioport
  16. Ahmad, Faraz; Lee, Seyong; Thottethodi, Mithuna; Vijaykumar, T. N.: MapReduce with communication overlap (MaRCO) (2013) ioport
  17. Han, Liangxiu; Liew, Chee Sun; van Hemert, Jano; Atkinson, Malcolm: A generic parallel processing model for facilitating data mining and integration (2011) ioport
  18. Nicolae, Bogdan; Antoniu, Gabriel; Bougé, Luc; Moise, Diana; Carpen-Amarie, Alexandra: BlobSeer: Next-generation data management for large scale infrastructures (2011) ioport
  19. Wilde, Michael; Hategan, Mihael; Wozniak, Justin M.; Clifford, Ben; Katz, Daniel S.; Foster, Ian: Swift: A language for distributed parallel scripting (2011) ioport
  20. Raicu, Ioan; Foster, Ian; Wilde, Mike; Zhang, Zhao; Iskra, Kamil; Beckman, Peter; Zhao, Yong; Szalay, Alex; Choudhary, Alok; Little, Philip: Middleware support for many-task computing (2010) ioport

1 2 next