Admission Control in YARN Clusters Based on Dynamic Resource Reservation
Document Type
Conference Proceeding
Publication Date
1-1-2015
Abstract
Hadoop YARN is an open project developed by the Apache Software Foundation to provide a resource management framework for large scale parallel data processing. However, there exists a resource waiting deadlock under the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by ApplicationMasters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. Therefore, we develop a new admission control mechanism which dynamically reserves resources for processing tasks in order to avoid resource waiting deadlocks and meanwhile obtain good performance. We implement and evaluate our new mechanism in Hadoop YARN v2.2.0. The experimental results show the effectiveness of this mechanism under MapReduce benchmarks.
DOI
10.1109/INM.2015.7140389
Montclair State University Digital Commons Citation
Yao, Yi; Lin, Jason; Wang, Jiayin; Mi, Ningfang; and Sheng, Bo, "Admission Control in YARN Clusters Based on Dynamic Resource Reservation" (2015). Department of Computer Science Faculty Scholarship and Creative Works. 90.
https://digitalcommons.montclair.edu/compusci-facpubs/90