6 March 2020 in IEEE Internet of Things Journal
Vehicular edge computing (VEC) is a new computing paradigm that has great potential to enhance the capability of vehicle terminals (VT) to support resource-hungry in-vehicle applications with low latency and high energy efficiency. In this paper, we investigate an important computation offloading scheduling problem in a typical VEC scenario, where a VT traveling along an expressway intends to schedule its tasks waiting in the queue to minimize the long-term cost in terms of a trade-off between task latency and energy consumption. Due to diverse task characteristics, dynamic wireless environment, and frequent handover events caused by vehicle movements, an optimal solution should take into account both where to schedule (i.e., local computation or offloading) and when to schedule (i.e., the order and time for execution) each task. To solve such a complicated stochastic optimization problem, we model it by a carefully designed Markov decision process (MDP) and resort to deep reinforcement learning (DRL) to deal with the enormous state space. Our DRL implementation is designed based on the state-of-the-art proximal policy optimization (PPO) algorithm. A parameter-shared network architecture combined with a convolutional neural network (CNN) is utilized to approximate both policy and value function, which can effectively extract representative features. A series of adjustments to the state and reward representations are taken to further improve the training efficiency. Extensive simulation experiments and comprehensive comparisons with six known baseline algorithms and their heuristic combinations clearly demonstrate the advantages of the proposed DRL-based offloading scheduling method.
26 November 2021 in IEEE Transactions on Computers
Edge computing is an emerging promising computing paradigm that brings computation and storage resources to the network edge, hence significantly reducing the service latency and network traffic. In edge computing, many applications are composed of dependent tasks where the outputs of some are the inputs of others. How to offload these tasks to the network edge is a vital and challenging problem which aims to determine the placement of each running task in order to maximize the Quality-of-Service (QoS). Most of the existing studies either design heuristic algorithms that lack strong adaptivity or learning-based methods but without considering the intrinsic task dependency. Different from the existing work, we propose an intelligent task offloading scheme leveraging off-policy reinforcement learning empowered by a Sequence-to-Sequence (S2S) neural network, where the dependent tasks are represented by a Directed Acyclic Graph (DAG). To improve the training efficiency, we combine a specific off-policy policy gradient algorithm with a clipped surrogate objective. We then conduct extensive simulation experiments using heterogeneous applications modelled by synthetic DAGs. The results demonstrate that: 1) our method converges fast and steadily in training; 2) it outperforms the existing methods and approximates the optimal solution in latency and energy consumption under various scenarios.
14 January 2020 in IEEE Transactions on Vehicular Technology
18 March 2024 in IEEE Transactions on Mobile Computing
Internet-of-Things (IoT) has played a critical role in developing sustainable smart cities and emerging numerous latency-sensitive IoT applications. Mobile edge computing (MEC) federation has the capability to incorporate a transparent resource management approach, which enables the sharing and utilization of MEC services from edge infrastructure providers (EIPs) and provides agile access services to mobile devices (MDs). In this paper, we investigate the joint optimization problem of computation offloading, task migration, and resource allocation in the MEC federation. The objective is to minimize the weighted sum of latency and energy consumption while maintaining load balancing under the constraint of the long-term migration cost budget of EIPs. To address the problem, we decompose it into two sub-problems: 1) the MDs clustering sub-problem and 2) the sub-problem of joint computation offloading, task migration, and resource allocation. First, an MDs clustering matching (MDCM) algorithm is proposed to cluster the MDs in edge servers (ESs) according to the differences in channel gains. Afterward, the second sub-problem is simplified by the Lyapunov optimization technique, and then we propose a Transformer-based mobility prediction model and a decentralized deep deterministic policy gradient (DDPG)-based framework to solve it. Extensive simulation results demonstrate the cost-efficiency of the proposed algorithm.
2 January 2025 in IEEE Internet of Things Journal
Workflow scheduling plays a critical role in optimizing completion time and throughput in distributed cloud environments, leveraging the parallelism of heterogeneous computing resources. However, existing workflow scheduling algorithms often fall short due to heuristic limitations and the challenges in adaptability within heterogeneous settings, leading to suboptimal scheduling solutions. In this article, we present a novel deep reinforcement learning (DRL) framework tailored for continuous workflow scheduling in heterogeneous environments. First, we propose an intelligent scheduler that updates the policy network through interactions with a multitenant environment, triggered by scheduling events. Next, the framework incorporates a graph attention network (GAT) and a self-attention multilayer perceptron (MLP) to preprocess the workflow topology and embed dynamic features of ready tasks and available processors into the state input at each scheduling step. Additionally, a k-dimensional tree-based k-nearest neighbors (kNN) algorithm is employed to map the output action vector to a pair of executed ready task and processor, facilitating the transition from continuous to discrete action spaces and addressing challenges associated with dynamic action spaces. Experimental results demonstrate that our method converges effectively in continuous workflow scheduling scenarios and significantly outperforms the best-known methods in terms of average makespan and load balancing efficiency.
25 October 2022 in IEEE Transactions on Circuits and Systems for Video Technology
Transformers are transforming the landscape of computer vision, especially for image-level recognition and instance-level detection tasks. Human-object interaction detection transformer (HOI-TR) is the first transformer-based end-to-end learning system for human-object interaction (HOI) detection; vision transformers build a simple multi-stage structure for multi-scale representation with single-scale patch and are the first patch-based transformer architecture for image-level recognition and instance-level detection. In this paper, we build a transformer-based multi-scale human-object interaction detector (MHOI), a novel method to integrate Vision and HOI detection Transformer, instead of directly incorporating two types of transformers, since the vision transformer lacks hierarchical architecture to handle the large variations in the scale of visual entities due to the single-scale patch partitioning. Specifically, MHOI embeds features of the same size (i.e., sequence length) with patches of variable scales simultaneously by utilizing overlapping convolutional patch embedding, then introduces an efficient transformer decoder that designs the query based on anchor points and essential auxiliary techniques to boost the HOI detection performance. Numerically, extensive experiments on several benchmarks demonstrate that our proposed framework outperforms prior existing methods coherently and achieves the impressive performance of 29.67 mAP on HICO-DET and 58.7 mAP on V-COCO, respectively.