Wireless sensor networks (WSNs) have taken a giant leap in scale, expanding their applicability to a large variety of technological domains and applications, ranging from the Internet of things (IoT) for smart cities and smart homes to wearable technology healthcare applications, underwater, agricultural and environmental monitoring and many more. This expansion is rapidly growing every passing day in terms of the variety, heterogeneity and the number of devices which such applications support. Data collection is commonly the core application in WSN and IoT networks, which are typically composed of a large variety of devices, some constrained by their resources (e.g., processing, storage, energy) and some by highly diverse demands. Many challenges span all the conceptual communication layers, from the Physical to the Applicational. Many novel solutions devised in the past do not scale well with the exponential growth in the population of the devices and need to be adapted, revised, or new innovative solutions are required to comply with this massive growth. Furthermore, recent technological advances present new opportunities which can be leveraged in this context. This paper provides a cross-layer perspective and review of data gathering in WSN and IoT networks. We provide some background and essential milestones that have laid the foundation of many subsequent solutions suggested over the years. We mainly concentrate on recent state-of-the-art research, which facilitates the scalable, energy-efficient, cost-effective, and human-friendly functionality of WSNs and the novel applications in the years to come.

This study introduces a novel approach to data gathering in energy-harvesting wireless sensor networks (EH-WSNs) utilizing cooperative multi-agent reinforcement learning (MARL). In addressing the challenges of efficient data collection in resource-constrained WSNs, we propose and examine a decentralized, autonomous communication framework where sensors function as individual agents. These agents employ an extended version of the Q-learning algorithm, tailored for a multi-agent setting, enabling independent learning and adaptation of their data transmission strategies. We introduce therein a specialized $ϵ$-p-greedy exploration method which is well suited for MAS settings. The key objective of our approach is the maximization of report flow, aligning with specific applicative goals for these networks. Our model operates under varying energy constraints and dynamic environments, with each sensor making decisions based on interactions within the network, devoid of explicit inter-sensor communication. The focus is on optimizing the frequency and efficiency of data report delivery to a central collection point, taking into account the unique attributes of each sensor. Notably, our findings present a surprising result: despite the known challenges of Q-learning in MARL, such as non-stationarity and the lack of guaranteed convergence to optimality due to multi-agent related pathologies, the cooperative nature of the MARL protocol in our study obtains high network performance. We present simulations and analyze key aspects contributing to coordination in various scenarios. A noteworthy feature of our system is its perpetual learning capability, which fosters network adaptiveness in response to changes such as sensor malfunctions or new sensor integrations. This dynamic adaptability ensures sustained and effective resource utilization, even as network conditions evolve. Our research lays grounds for learning-based WSNs and offers vital insights into the application of MARL in real-world EH-WSN scenarios, underscoring its effectiveness in navigating the intricate challenges of large-scale, resource-limited sensor networks.

We address a cost optimization problem of running instances of applications in a remote cloud configuration of multiple virtual machines (VMs). Managing VMs involves a sophisticated trade-off between demands, which are expressed by the internal provisional costs of leased cloud resources, and exogenous costs expressed by service revenues that are typically bound to SLAs. The internal costs may include VM deployment/termination cost, and VM lease cost. The exogenous costs include rewards accumulated due to accomplished tasks. In the case where the SLA restricts the performance level, tasks incoming at VMs that reached that level are rejected, causing fines deducted against the rewards. The performance level is also quantified, namely, by means of a delay cost. We model this problem by cost-optimal load balancing to a queuing system with a flexible number of queues. We analyze the system by Markov decision process (MDP) and numerically solve it to find the optimal policy, which captures the aforementioned costs and performance constraints. Within this constrained framework, we also investigate the impact of average VM deployment time. We show that the optimal policy possesses decision thresholds which depend on several parameters. We validate policies found by MDP through a set-up implemented on AWS.

Managing network-related resources involves sophisticated trade-off between flexibility, high performance standards, latency demands which adhere to service level agreements (SLAs) and cost constraints. Network functioning virtualization (NFV) opens new challenges to the remote network management which combines activation and control of virtual machines (VMs) according to variable demand. Currently, this functionality is being handled by harnessing the traditional orchestration algorithms using suboptimal heuristics. We model the problem of virtual network function (VNF) allocation by queuing system with flexible number of queues, which captures variety of constraints including queue deployment and displacement, delay cost, holding cost, scheduling reward and fine. Next, we model the system by Markov decision process (MDP) and numerically solve it to find the optimal policy. We show analytically and by simulations that the optimal policy possesses decision thresholds which depend on several parameters¹.

Deep Neural Networks (DNNs) have demonstrated exceptional performance in diverse tasks. However, deploying DNNs on resource-constrained devices presents challenges due to energy consumption and delay overheads. To mitigate these issues, early-exit DNNs (EE-DNNs) incorporate exit branches within intermediate layers to enable early inferences. These branches estimate prediction confidence and employ a fixed threshold to determine early termination. Nonetheless, fixed thresholds yield suboptimal performance in dynamic contexts, where context refers to distortions caused by environmental conditions, in image classification, or variations in input distribution due to concept drift, in NLP. In this article, we introduce Upper Confidence Bound in EE-DNNs (), an online algorithm that dynamically adjusts early exit thresholds based on context. leverages confidence levels at intermediate layers and learns without the need for true labels. Through extensive experiments in image classification and NLP, we demonstrate that achieves logarithmic regret, converging after just a few thousand observations across multiple contexts. We evaluate for image classification and text mining. In the latter, we show that can reduce cumulative regret and lower latency by approximately 10%–20% without compromising accuracy when compared to fixed threshold alternatives. Our findings highlight as an effective method for enhancing EE-DNN efficiency.