Lifetime-based memory management for distributed data processing systems

Publikation: Bidrag til bog/antologi/rapportKonferencebidrag i proceedingsForskningfagfællebedømt

Dokumenter

  • pdf

    Indsendt manuskript, 1 MB, PDF-dokument

  • Lu Lu
  • Xuanhua Shi
  • Zhou, Yongluan
  • Xiong Zhang
  • Hai Jin
  • Cheng Pei
  • Ligang He
  • Yuanzhen Geng
In-memory caching of intermediate data and eager combining of data in shuffle buffers have been shown to be very effective in minimizing the re-computation and I/O cost in distributed data processing systems like Spark and Flink. However, it has also been widely reported that these techniques would create a large amount of long-living data objects in the heap, which may quickly saturate the garbage collector, especially when handling a large dataset, and hence would limit the scalability of the system. To eliminate this problem, we propose a lifetime-based memory management framework, which, by automatically analyzing the user-defined functions and data types, obtains the expected lifetime of the data objects, and then allocates and releases memory space accordingly to minimize the garbage collection overhead. In particular, we present Deca, a concrete implementation of our proposal on top of Spark, which transparently decomposes and groups objects with similar lifetimes into byte arrays and releases their space altogether when their lifetimes come to an end. An extensive experimental study using both synthetic and real datasets shows that, in comparing to Spark, Deca is able to 1) reduce the garbage collection time by up to 99.9%, 2) to achieve up to 22.7x speed up in terms of execution time in cases without data spilling and 41.6x speedup in cases with data spilling, and 3) to consume up to 46.6% less memory.
OriginalsprogEngelsk
TitelProceedings of the 42nd International Conference on Very Large Data Bases, New Delhi, India
RedaktørerSurajit Chaudhuri, Jayant Haritsa
Antal sider12
ForlagVLDB Endowment
Publikationsdato1 aug. 2016
Sider936-947
DOI
StatusUdgivet - 1 aug. 2016
Eksternt udgivetJa
Begivenhed42nd International Conference On Very Large Data Bases - New Delhi, Indien
Varighed: 5 sep. 20169 sep. 2016
Konferencens nummer: 42

Konference

Konference42nd International Conference On Very Large Data Bases
Nummer42
LandIndien
ByNew Delhi
Periode05/09/201609/09/2016
NavnProceedings of the VLDB Endowment
Nummer12
Vol/bind9
ISSN2150-8097

Antal downloads er baseret på statistik fra Google Scholar og www.ku.dk


Ingen data tilgængelig

ID: 179407884