A mediation system for continuous spatial queries on a unified schema using Apache Spark - INRAE - Institut national de recherche pour l’agriculture, l’alimentation et l’environnement
Article Dans Une Revue Big Earth Data Année : 2024

A mediation system for continuous spatial queries on a unified schema using Apache Spark

Résumé

Recent advances in big and streaming data systems have enabled real-time analysis of data generated by Internet of Things (IoT) systems and sensors in various domains. In this context, many applications require integrating data from several heterogeneous sources, either stream or static sources. Frameworks such as Apache Spark are able to integrate and process large datasets from different sources. However, these frameworks are hard to use when the data sources are heterogeneous and numerous. To address this issue, we propose a system based on mediation techniques for integrating stream and static data sources. The integration process of our system consists of three main steps: configuration, query expression and query execution. In the configuration step, an administrator designs a mediated schema and defines mapping between the mediated schema and local data sources. In the query expression step, users express queries using customized SQL grammar on the mediated schema. Finally, our system rewrites the query into an optimized Spark application and submits the application to a Spark cluster. The results are continuously returned to users. Our experiments show that our optimizations can improve query execution time by up to one order of magnitude, making complex streaming and spatial data analysis more accessible.
Fichier principal
Vignette du fichier
A mediation system for continuous spatial queries on a unified schema using Apache Spark.pdf (2.84 Mo) Télécharger le fichier
Origine Fichiers éditeurs autorisés sur une archive ouverte
Licence

Dates et versions

hal-04283788 , version 1 (28-02-2024)

Licence

Identifiants

Citer

Thi Thu Trang Ngo, François Pinet, David Sarramia, Myoung-Ah Kang. A mediation system for continuous spatial queries on a unified schema using Apache Spark. Big Earth Data, 2024, 8 (1), pp.115-141. ⟨10.1080/20964471.2023.2275854⟩. ⟨hal-04283788⟩
96 Consultations
29 Téléchargements

Altmetric

Partager

More