A mediation system for continuous spatial queries on a unified schema using Apache Spark - INRAE - Institut national de recherche pour l’agriculture, l’alimentation et l’environnement Access content directly
Journal Articles Big Earth Data Year : 2024

A mediation system for continuous spatial queries on a unified schema using Apache Spark

Abstract

Recent advances in big and streaming data systems have enabled real-time analysis of data generated by Internet of Things (IoT) systems and sensors in various domains. In this context, many applications require integrating data from several heterogeneous sources, either stream or static sources. Frameworks such as Apache Spark are able to integrate and process large datasets from different sources. However, these frameworks are hard to use when the data sources are heterogeneous and numerous. To address this issue, we propose a system based on mediation techniques for integrating stream and static data sources. The integration process of our system consists of three main steps: configuration, query expression and query execution. In the configuration step, an administrator designs a mediated schema and defines mapping between the mediated schema and local data sources. In the query expression step, users express queries using customized SQL grammar on the mediated schema. Finally, our system rewrites the query into an optimized Spark application and submits the application to a Spark cluster. The results are continuously returned to users. Our experiments show that our optimizations can improve query execution time by up to one order of magnitude, making complex streaming and spatial data analysis more accessible.
Fichier principal
Vignette du fichier
A mediation system for continuous spatial queries on a unified schema using Apache Spark.pdf (2.84 Mo) Télécharger le fichier
Origin : Publisher files allowed on an open archive
Licence : CC BY - Attribution

Dates and versions

hal-04283788 , version 1 (28-02-2024)

Licence

Attribution

Identifiers

Cite

Thi Thu Trang Ngo, François Pinet, David Sarramia, Myoung-Ah Kang. A mediation system for continuous spatial queries on a unified schema using Apache Spark. Big Earth Data, 2024, 8 (1), pp.1-27. ⟨10.1080/20964471.2023.2275854⟩. ⟨hal-04283788⟩
41 View
10 Download

Altmetric

Share

Gmail Facebook X LinkedIn More