Big data analysis with skeletons on SOFA

Publikation: Bidrag til bog/antologi/rapportKonferencebidrag i proceedingsForskningfagfællebedømt

  • Kenneth Skovhede
  • Brian Vinter

This paper explores how a skeleton based approach can be used to perform big data analysis. We introduce a restricted storage system based on blocks with a fixed maximum size. The storage design removes the residual data problem commonly found in storage systems, and enables processing on individual blocks. We then introduce a stream-oriented query system that can be used on top of the distributed storage system. The query system is built on a limited number of core operations. Each of the perform a specified function, such as filtering elements, but are skeleton operations where the programmer needs to fill in how to perform the operation. The operations are designed to allow splitting across the blocks in the storage system, giving concurrent execution while maintaining a completely sequential program description. To assist in understanding the data flow, we also introduce a graphical representation for each of the methods, enabling a visual expression of an algorithm. To evaluate the query system we implement a number of classic Big-Data queries and show how to implement them with code, and how the queries can be visualized with the graphical representation.

OriginalsprogEngelsk
TitelCommunicating Process Architectures 2017 and 2018, WoTUG-39 and WoTUG-40 - Proceedings of CPA 2017 (WoTUG-39) and Proceedings of CPA 2018 (WoTUG-40)
RedaktørerJan Baekgaard Pedersen, Kevin Chalmers, Jan F. Broenink, Brian Vinter, Kevin Vella, Peter H. Welch, Marc L. Smith, Kenneth Skovhede
Antal sider13
ForlagIMIA and IOS Press
Publikationsdato2019
Sider5-17
ISBN (Elektronisk)9781614999485
DOI
StatusUdgivet - 2019
Begivenhed39th WoTUG Conference on Communicating Process Architectures, CPA 2017 and 40th WoTUG Conference on Communicating Process Architectures, CPA 2018 - Dresden, Tyskland
Varighed: 19 aug. 201822 aug. 2018

Konference

Konference39th WoTUG Conference on Communicating Process Architectures, CPA 2017 and 40th WoTUG Conference on Communicating Process Architectures, CPA 2018
LandTyskland
ByDresden
Periode19/08/201822/08/2018
NavnConcurrent Systems Engineering Series
Vol/bind70
ISSN1383-7575

ID: 241091096