With the increasing prevalence of IoT environments, the demand for processing massive distributed data streams has become a critical challenge. Data Stream Processing on the Edge (DSPoE) systems have emerged as a solution to address this challenge, but they often struggle to cope with the heterogeneity of hardware and platforms. To address this issue, we propose a new hybrid DSPoE architecture named GALOIS, which is based on WebAssembly (Wasm) and is hardware-, platform-, and language-agnostic. GALOIS employs a multi-layered approach that combines P2P and master-worker concepts for communication between components. We present experimental results showing that operators executed in Wasm outperform those in Docker in terms of energy and CPU consumption, making it a promising option for streaming operators in DSPoE. We therefore expect Wasm-based solutions to significantly improve the performance and resilience of DSPoE systems.
With the rapid growth of data from heterogeneous, distributed sources, data streams need to be increasingly processed in the cloud-edge continuum. Processing is distributed between diverse edge environments and homogeneous, but powerful data centers, to optimally utilize available resources, alleviate infrastructure bottlenecks and follow the principle of data locality. Compared to cloud infrastructure, compute and data resources on the edge are distributed across geographical regions, infrastructures and organizational units with independent data processing systems. However, existing data stream processing frameworks provide integrated systems, which require matching software components to be used across the whole, distributed infrastructure or even assume centralized control over all resources. We argue, that this cloud-like, centralized approach does not fit to the decentralized nature of the edge environment. Focusing on fully integrated systems, which are either limited to single organizational units or require a certain degree of homogeneity limits data sovereignty and the overall potential of distributed data stream processing on the edge. Instead, we propose to develop data stream processing as part of data ecosystems, and connect locally independent and sovereign systems through a lightweight set of common standards, protocols, and semantic descriptions.
With the increasing prevalence of IoT devices, efficiently using these resources to process data has become a topic of great relevance and the interest in systems for distributed data stream processing on heterogeneous edge hardware is rising. A core challenge in developing and evaluating those systems lies in the dynamic and heterogeneous environments that such systems need to operate in. With GALOISim, we present a simulation software to develop and test algorithms for operator placement and clustering in hierarchical, distributed environments. The application enables users to submit algorithms and simulate their behavior against user-defined topologies and queries and interactively explore the results.