Lifetime-based memory management for distributed data processing systems

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

Standard

Lifetime-based memory management for distributed data processing systems. / Lu, Lu; Shi, Xuanhua; Zhou, Yongluan; Zhang, Xiong; Jin, Hai; Pei, Cheng; He, Ligang; Geng, Yuanzhen.

Proceedings of the 42nd International Conference on Very Large Data Bases, New Delhi, India. ed. / Surajit Chaudhuri; Jayant Haritsa. VLDB Endowment, 2016. p. 936-947 (Proceedings of the VLDB Endowment; No. 12, Vol. 9).

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

Harvard

Lu, L, Shi, X, Zhou, Y, Zhang, X, Jin, H, Pei, C, He, L & Geng, Y 2016, Lifetime-based memory management for distributed data processing systems. in S Chaudhuri & J Haritsa (eds), Proceedings of the 42nd International Conference on Very Large Data Bases, New Delhi, India. VLDB Endowment, Proceedings of the VLDB Endowment, no. 12, vol. 9, pp. 936-947, 42nd International Conference On Very Large Data Bases, New Delhi, India, 05/09/2016. https://doi.org/10.14778/2994509.2994513

APA

Lu, L., Shi, X., Zhou, Y., Zhang, X., Jin, H., Pei, C., ... Geng, Y. (2016). Lifetime-based memory management for distributed data processing systems. In S. Chaudhuri, & J. Haritsa (Eds.), Proceedings of the 42nd International Conference on Very Large Data Bases, New Delhi, India (pp. 936-947). VLDB Endowment. Proceedings of the VLDB Endowment, No. 12, Vol.. 9 https://doi.org/10.14778/2994509.2994513

Vancouver

Lu L, Shi X, Zhou Y, Zhang X, Jin H, Pei C et al. Lifetime-based memory management for distributed data processing systems. In Chaudhuri S, Haritsa J, editors, Proceedings of the 42nd International Conference on Very Large Data Bases, New Delhi, India. VLDB Endowment. 2016. p. 936-947. (Proceedings of the VLDB Endowment; No. 12, Vol. 9). https://doi.org/10.14778/2994509.2994513

Author

Lu, Lu ; Shi, Xuanhua ; Zhou, Yongluan ; Zhang, Xiong ; Jin, Hai ; Pei, Cheng ; He, Ligang ; Geng, Yuanzhen. / Lifetime-based memory management for distributed data processing systems. Proceedings of the 42nd International Conference on Very Large Data Bases, New Delhi, India. editor / Surajit Chaudhuri ; Jayant Haritsa. VLDB Endowment, 2016. pp. 936-947 (Proceedings of the VLDB Endowment; No. 12, Vol. 9).

Bibtex

@inproceedings{a6e47ddf92084c32b233c07a5a1f1db2,
title = "Lifetime-based memory management for distributed data processing systems",
abstract = "In-memory caching of intermediate data and eager combining of data in shuffle buffers have been shown to be very effective in minimizing the re-computation and I/O cost in distributed data processing systems like Spark and Flink. However, it has also been widely reported that these techniques would create a large amount of long-living data objects in the heap, which may quickly saturate the garbage collector, especially when handling a large dataset, and hence would limit the scalability of the system. To eliminate this problem, we propose a lifetime-based memory management framework, which, by automatically analyzing the user-defined functions and data types, obtains the expected lifetime of the data objects, and then allocates and releases memory space accordingly to minimize the garbage collection overhead. In particular, we present Deca, a concrete implementation of our proposal on top of Spark, which transparently decomposes and groups objects with similar lifetimes into byte arrays and releases their space altogether when their lifetimes come to an end. An extensive experimental study using both synthetic and real datasets shows that, in comparing to Spark, Deca is able to 1) reduce the garbage collection time by up to 99.9{\%}, 2) to achieve up to 22.7x speed up in terms of execution time in cases without data spilling and 41.6x speedup in cases with data spilling, and 3) to consume up to 46.6{\%} less memory.",
author = "Lu Lu and Xuanhua Shi and Yongluan Zhou and Xiong Zhang and Hai Jin and Cheng Pei and Ligang He and Yuanzhen Geng",
year = "2016",
month = "8",
day = "1",
doi = "10.14778/2994509.2994513",
language = "English",
series = "Proceedings of the VLDB Endowment",
publisher = "VLDB Endowment",
number = "12",
pages = "936--947",
editor = "Surajit Chaudhuri and Jayant Haritsa",
booktitle = "Proceedings of the 42nd International Conference on Very Large Data Bases, New Delhi, India",
note = "42nd International Conference On Very Large Data Bases, VLDB ; Conference date: 05-09-2016 Through 09-09-2016",

}

RIS

TY - GEN

T1 - Lifetime-based memory management for distributed data processing systems

AU - Lu, Lu

AU - Shi, Xuanhua

AU - Zhou, Yongluan

AU - Zhang, Xiong

AU - Jin, Hai

AU - Pei, Cheng

AU - He, Ligang

AU - Geng, Yuanzhen

N1 - Conference code: 42

PY - 2016/8/1

Y1 - 2016/8/1

N2 - In-memory caching of intermediate data and eager combining of data in shuffle buffers have been shown to be very effective in minimizing the re-computation and I/O cost in distributed data processing systems like Spark and Flink. However, it has also been widely reported that these techniques would create a large amount of long-living data objects in the heap, which may quickly saturate the garbage collector, especially when handling a large dataset, and hence would limit the scalability of the system. To eliminate this problem, we propose a lifetime-based memory management framework, which, by automatically analyzing the user-defined functions and data types, obtains the expected lifetime of the data objects, and then allocates and releases memory space accordingly to minimize the garbage collection overhead. In particular, we present Deca, a concrete implementation of our proposal on top of Spark, which transparently decomposes and groups objects with similar lifetimes into byte arrays and releases their space altogether when their lifetimes come to an end. An extensive experimental study using both synthetic and real datasets shows that, in comparing to Spark, Deca is able to 1) reduce the garbage collection time by up to 99.9%, 2) to achieve up to 22.7x speed up in terms of execution time in cases without data spilling and 41.6x speedup in cases with data spilling, and 3) to consume up to 46.6% less memory.

AB - In-memory caching of intermediate data and eager combining of data in shuffle buffers have been shown to be very effective in minimizing the re-computation and I/O cost in distributed data processing systems like Spark and Flink. However, it has also been widely reported that these techniques would create a large amount of long-living data objects in the heap, which may quickly saturate the garbage collector, especially when handling a large dataset, and hence would limit the scalability of the system. To eliminate this problem, we propose a lifetime-based memory management framework, which, by automatically analyzing the user-defined functions and data types, obtains the expected lifetime of the data objects, and then allocates and releases memory space accordingly to minimize the garbage collection overhead. In particular, we present Deca, a concrete implementation of our proposal on top of Spark, which transparently decomposes and groups objects with similar lifetimes into byte arrays and releases their space altogether when their lifetimes come to an end. An extensive experimental study using both synthetic and real datasets shows that, in comparing to Spark, Deca is able to 1) reduce the garbage collection time by up to 99.9%, 2) to achieve up to 22.7x speed up in terms of execution time in cases without data spilling and 41.6x speedup in cases with data spilling, and 3) to consume up to 46.6% less memory.

U2 - 10.14778/2994509.2994513

DO - 10.14778/2994509.2994513

M3 - Article in proceedings

T3 - Proceedings of the VLDB Endowment

SP - 936

EP - 947

BT - Proceedings of the 42nd International Conference on Very Large Data Bases, New Delhi, India

A2 - Chaudhuri, Surajit

A2 - Haritsa, Jayant

PB - VLDB Endowment

T2 - 42nd International Conference On Very Large Data Bases

Y2 - 5 September 2016 through 9 September 2016

ER -

ID: 179407884