Residential Collegefalse
Status已發表Published
MEER: Online estimation of optimal memory reservations for long lived containers in in-memory cluster computing
Guoyao Xu1,2,3; Chengzhong Xu4
2019-07
Conference Name39th IEEE International Conference on Distributed Computing Systems (ICDCS)
Source PublicationProceedings - International Conference on Distributed Computing Systems
Volume2019-July
Pages23-34
Conference Date07-10 July 2019
Conference PlaceDallas, TX, USA
CountryUSA
PublisherIEEE
Abstract

Modern in-memory data-intensive computing systems like Spark create long-lived containers to execute diverse types of applications. They rely on a cluster manager like YARN or Mesos to perform resource allocation to the containers. The cluster manager or scheduler requires users of the containers to reserve resources beforehand. It is a challenge to estimate just right amounts of memory to run the applications before execution, so as to avoid over-or under-provisioning of memory space. We discover a general property of memory reservation elasticity, which allows applications to run with a reservation limit smaller than they would ideally need while only paying a moderate performance penalty. Based on the property, we designed a system, namely MEER, which performs online estimation of minimum necessary amount of memory limit that achieves nearly optimal performance. We referred to it as optimal reservation, which divides memory over-provisioning from under-provisioning. It is non-trivial to efficiently estimate optimal reservations on line through one step without runtime history. MEER uses a two-step approach to dealing with the challenge: 1) Do robust profiling and probability density analysis of applications' memory footprints in two pilot runs. By using confidence levels for the prediction, we reduce the negative effects of container footprints' randomness and achieve a highly accurate online initial estimation (over 80% accuracy) of optimal reservation. 2) By exploiting a self-decay property of the analytical results, MEER adaptively performs iterative search based on a feed-back control mechanism over subsequent recurring executions. We implemented MEER atop of YARN and evaluated the prototype by running 15 benchmark workloads on a 16-node local cluster. Evaluation results show that it achieves an average accuracy of more than 95%. By deploying MEER on schedulers and allocating memory according to the optimal reservations, one could improve cluster memory utilization by about 40%. It reduces individual application execution time by 2 to 6 times on average compared to the state-of-the-art approaches. A 90 times peak speedup for PageRank in comparison with the default Spark/Yarn is observed.

KeywordMemory Resource Demands Optimal Reservation Memory Elasticity Long-lived Containers Long-running Applications (Lra) Yarn Spark Intelligent Cluster SchedulIng In Cloud Datacenter
DOI10.1109/ICDCS.2019.00012
URLView the original
Indexed ByCPCI-S
Language英語English
WOS Research AreaComputer Science
WOS SubjectComputer Science, Hardware & Architecture ; Computer Science, Information Systems ; Computer Science, Software Engineering ; Computer Science, Theory & Methods
WOS IDWOS:000565234200003
The Source to Articlehttps://ieeexplore.ieee.org/document/8885207
Scopus ID2-s2.0-85074823870
Fulltext Access
Citation statistics
Document TypeConference paper
CollectionDEPARTMENT OF COMPUTER AND INFORMATION SCIENCE
Faculty of Science and Technology
THE STATE KEY LABORATORY OF INTERNET OF THINGS FOR SMART CITY (UNIVERSITY OF MACAU)
Affiliation1.Shenzhen Institutes of Advanced Technology
2.Wayne State University
3.Alibaba Group
4.State Key Lab on IoTSC Dept of Computer and Information Sciences University of Macau, Macao SAR, China
Recommended Citation
GB/T 7714
Guoyao Xu,Chengzhong Xu. MEER: Online estimation of optimal memory reservations for long lived containers in in-memory cluster computing[C]:IEEE, 2019, 23-34.
APA Guoyao Xu., & Chengzhong Xu (2019). MEER: Online estimation of optimal memory reservations for long lived containers in in-memory cluster computing. Proceedings - International Conference on Distributed Computing Systems, 2019-July, 23-34.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Guoyao Xu]'s Articles
[Chengzhong Xu]'s Articles
Baidu academic
Similar articles in Baidu academic
[Guoyao Xu]'s Articles
[Chengzhong Xu]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Guoyao Xu]'s Articles
[Chengzhong Xu]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.