On Optimization of Scientific Workflows to Support Streaming Applications in Distributed Network Environments
Document Type
Conference Proceeding
Publication Date
12-1-2010
Abstract
Large-scale data-intensive streaming applications in various science fields feature complex DAG-structured workflows comprised of distributed computing modules with intricate inter-module dependencies. Supporting such workflows in high-performance network environments and optimizing their throughput are crucial to collaborative scientific exploration and discovery. We formulate workflow mapping as a frame rate optimization problem and propose an efficient heuristic solution, which is integrated into the Condor-based Scientific Workflow Automation and Management Platform (SWAMP) in place of Condor's default mapping scheme. The SWAMP system is also augmented with several new components to improve the workflow management process. The performance superiority of the proposed solution is verified using both simulations and a real-life scientific workflow for climate modeling deployed in a distributed heterogeneous network environment.
DOI
10.1109/WORKS.2010.5671851
Montclair State University Digital Commons Citation
Wu, Qishi; Gu, Yi; Lu, Xukang; Zhu, Michelle; Brown, Patrick; Lin, Wuyin; and Liu, Yangang, "On Optimization of Scientific Workflows to Support Streaming Applications in Distributed Network Environments" (2010). Department of Computer Science Faculty Scholarship and Creative Works. 442.
https://digitalcommons.montclair.edu/compusci-facpubs/442