Residential College | false |
Status | 已發表Published |
MEER: Online estimation of optimal memory reservations for long lived containers in in-memory cluster computing | |
Guoyao Xu1,2,3; Chengzhong Xu4 | |
2019-07 | |
Conference Name | 39th IEEE International Conference on Distributed Computing Systems (ICDCS) |
Source Publication | Proceedings - International Conference on Distributed Computing Systems |
Volume | 2019-July |
Pages | 23-34 |
Conference Date | 07-10 July 2019 |
Conference Place | Dallas, TX, USA |
Country | USA |
Publisher | IEEE |
Abstract | Modern in-memory data-intensive computing systems like Spark create long-lived containers to execute diverse types of applications. They rely on a cluster manager like YARN or Mesos to perform resource allocation to the containers. The cluster manager or scheduler requires users of the containers to reserve resources beforehand. It is a challenge to estimate just right amounts of memory to run the applications before execution, so as to avoid over-or under-provisioning of memory space. We discover a general property of memory reservation elasticity, which allows applications to run with a reservation limit smaller than they would ideally need while only paying a moderate performance penalty. Based on the property, we designed a system, namely MEER, which performs online estimation of minimum necessary amount of memory limit that achieves nearly optimal performance. We referred to it as optimal reservation, which divides memory over-provisioning from under-provisioning. It is non-trivial to efficiently estimate optimal reservations on line through one step without runtime history. MEER uses a two-step approach to dealing with the challenge: 1) Do robust profiling and probability density analysis of applications' memory footprints in two pilot runs. By using confidence levels for the prediction, we reduce the negative effects of container footprints' randomness and achieve a highly accurate online initial estimation (over 80% accuracy) of optimal reservation. 2) By exploiting a self-decay property of the analytical results, MEER adaptively performs iterative search based on a feed-back control mechanism over subsequent recurring executions. We implemented MEER atop of YARN and evaluated the prototype by running 15 benchmark workloads on a 16-node local cluster. Evaluation results show that it achieves an average accuracy of more than 95%. By deploying MEER on schedulers and allocating memory according to the optimal reservations, one could improve cluster memory utilization by about 40%. It reduces individual application execution time by 2 to 6 times on average compared to the state-of-the-art approaches. A 90 times peak speedup for PageRank in comparison with the default Spark/Yarn is observed. |
Keyword | Memory Resource Demands Optimal Reservation Memory Elasticity Long-lived Containers Long-running Applications (Lra) Yarn Spark Intelligent Cluster SchedulIng In Cloud Datacenter |
DOI | 10.1109/ICDCS.2019.00012 |
URL | View the original |
Indexed By | CPCI-S |
Language | 英語English |
WOS Research Area | Computer Science |
WOS Subject | Computer Science, Hardware & Architecture ; Computer Science, Information Systems ; Computer Science, Software Engineering ; Computer Science, Theory & Methods |
WOS ID | WOS:000565234200003 |
The Source to Article | https://ieeexplore.ieee.org/document/8885207 |
Scopus ID | 2-s2.0-85074823870 |
Fulltext Access | |
Citation statistics | |
Document Type | Conference paper |
Collection | DEPARTMENT OF COMPUTER AND INFORMATION SCIENCE Faculty of Science and Technology THE STATE KEY LABORATORY OF INTERNET OF THINGS FOR SMART CITY (UNIVERSITY OF MACAU) |
Affiliation | 1.Shenzhen Institutes of Advanced Technology 2.Wayne State University 3.Alibaba Group 4.State Key Lab on IoTSC Dept of Computer and Information Sciences University of Macau, Macao SAR, China |
Recommended Citation GB/T 7714 | Guoyao Xu,Chengzhong Xu. MEER: Online estimation of optimal memory reservations for long lived containers in in-memory cluster computing[C]:IEEE, 2019, 23-34. |
APA | Guoyao Xu., & Chengzhong Xu (2019). MEER: Online estimation of optimal memory reservations for long lived containers in in-memory cluster computing. Proceedings - International Conference on Distributed Computing Systems, 2019-July, 23-34. |
Files in This Item: | There are no files associated with this item. |
Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.
Edit Comment