The traditional OLAP (On-Line Analytical Processing) systems store data in relational databases. Unfortunately, it is difficult to manage big data volumes with such systems. As an alternative, NoSQL systems (Not-only SQL) provide scalability and flexibility for an OLAP system. We define a set of rules to map star schemas and its optimization structure, a precomputed aggregate lattice, into two logical NoSQL models: column-oriented and document-oriented. Using these rules we analyse and implement two decision support systems, one for each model (using MongoDB and HBase).We compare both systems during the phases of data (generated using the TPC-DS benchmark) loading, lattice generation and querying.
1 March 2018 in Information Processing & Management
This paper presents a concise exploration of the evolving landscape of sustainable agriculture through the lens of data engineering. The aim is to highlight the most important challenges in the field and sketch ideas for how to address them. This position paper delves into key research challenges such as multi-modal data integration, data quality assurance, network optimisation and edge versus fog data processing strategies. Additionally, it emphasises the significance of performance enhancement in driving innovation within sustainable agriculture. By addressing these challenges and following the proposed visionary approaches for future research endeavours, we claim that data engineering will serve as a catalyst for advancing sustainable agriculture practices.
This paper highlights the advantages of an affordable multi-wavelength ground-based camera, called WaltRCam, for monitoring Aerosol Optical Depth (AOD) in a clear case over a peri-urban area. To simulate the performance of this low-cost camera, for which data are not yet available, we use data from an expensive hyperspectral camera (HSI) to mimic its characteristics. Our methodology is based on the construction of look-up tables using the DART (Discrete Anisotropic Radiative Transfer) 3D radiative transfer model. DART simulates the different spectra observed by the WaltRCam camera, which then provides the AODs for all image pixels in near-real-time. Moreover, DART is coupled to a 3D scale-model of the city of Toulouse (dating from 2014) to model complex urban geometries and to associate specific optical properties to the various objects that make up the environment. Moreover, we use a neural-network-based method to recognize the various objects in the image in order to take into account only pixels common between the observations. In this way, we take account of changes to the peri-urban area, such as vegetation growth, construction, demolition of buildings, etc. The results of this study show that the WaltRCam camera, by capturing eight wavelengths, can deliver convincing results compared with ground and satellite reference data, with a correlation coefficient of 0.9 and an average RMSE of less than 0.02.