The accelerating convergence of artificial intelligence and edge computing has sparked a recent wave of interest in edge intelligence. While pilot efforts focused on edge DNN inference serving for a single user or DNN application, scaling edge DNN inference serving to multiple users and applications is however nontrivial. In this paper, we propose an online optimization framework EdgeAdaptor for multi-user and multi-application edge DNN inference serving at scale, which aims to navigate the three-way trade-off between inference accuracy, latency, and resource cost via jointly optimizing the application configuration adaption, DNN model selection and edge resource provisioning on-the-fly. The underlying long-term optimization problem is difficult since it is NP-hard and involves future uncertain information. To address these dual challenges, we fuse the power of online optimization and approximate optimization into a joint optimization framework, via i) decomposing the long-term problem into a series of single-shot fractional problems with a regularization technique, and ii) rounding the fractional solution to a near-optimal integral solution with a randomized dependent scheme. Rigorous theoretical analysis derives a parameterized competition ratio of our online algorithms, and extensive trace-driven simulations verify that its empirical value is no larger than 1.4 in typical scenarios.
29 December 2023 in IEEE Transactions on Mobile Computing
Serverless computing is envisioned as the de-facto standard for next-generation cloud computing. However, the cold start dilemma has impeded its adoption by delay-sensitive and burst applications. In this paper, we propose to tame serverless cold start in a cloud inference system with edge computing. Specifically, the proposed solution smooths the serverless cloud workload with user-owned edge computing, reducing the number of cold starts. Leveraging the configurability of requests and serverless functions, the proposed solution further reduces the transmission latency and serverless cost by adapting request configuration (e.g., image resolution) and function configuration (e.g., memory). To alleviate the potential inference accuracy degradation incurred by configuration adaption, we aim to strike a nice balance between inference latency, cost, and accuracy. However, achieving this goal is non-trivial since the underlying optimization is non-convex and involves future uncertain information. To simultaneously address dual challenges, the presented cold-start-aware online algorithms apply the regularization technique to decompose the problem into separate convex subproblems. Then, it applies lazy switching to smooth the number of provisioned functions and thus reduces the cold start. Through rigorous theoretical analysis, realistic prototype evaluations on AWS Lambda, and trace-driven simulations, we comprehensively validate the theoretical and empirical performance of our proposed solution.
24 September 2024 in IEEE Transactions on Mobile Computing
Benefiting from hardware upgrades and deep learning techniques, more and more end devices can independently support a variety of intelligent applications. Further powered by edge computing technologies, the end-edge collaboration paradigm becomes one mainstream approach for achieving advanced edge intelligence (EI). To fully exploit the system resources, it is desirable to coordinate diverse EI services efficiently. Thus, we present a novel framework to jointly optimize the cost-performance trade-off for two distinct but typical EI services, where end devices simultaneously perform federated learning (FL) model training and conduct model inference with the assistance of edge offloading. However, balancing the long-term cost-performance trade-off is highly non-trivial, especially in the absence of knowledge of future system dynamics. Moreover, the capacity heterogeneity further increases the difficulty of service coordination among resource-limited end devices. To overcome these challenges, we first analyze the optimality of inference offloading decisions with and without FL model training and quantify their mutual effects due to local resource contention. By incorporating the loss estimation of FL training model, we then propose a novel proactive policy with theoretical guarantees, which proactively controls the stopping of FL training procedure to balance well the trade-offs between FL model performance and resource costs while fulfilling the inference performance requirements. Extensive results show the efficiency and robustness of our proposed algorithm for EI service coordination in dynamic end-edge collaboration scenarios.
11 February 2025 in IEEE Transactions on Networking
Due to the penetration of edge computing, a wide variety of workloads are sunk down to the network edge to alleviate huge pressure of the cloud. With the presence of high input workload dynamics and intensive edge resource contention, it is highly non-trivial for an edge proxy to optimize the scheduling of heterogeneous services with diverse QoS requirements. In general, online services should be quickly completed in a quite stable running environment to meet their tight latency constraint, while offline services can be processed loosely for their elastic soft deadlines. To well coordinate such services at the resource-limited edge cluster, in this paper, we study an edge-centric resource provisioning optimization for dynamic online and offline services co-location, where the proxy seeks to maximize timely online service performances while maintaining satisfactory long-term offline service performances. However, intricate hybrid couplings for provisioning decisions arise due to heterogeneous constraints of the co-located services and their different time-scale performances. We hence first propose a reactive provisioning approach without requiring a prior knowledge of future system dynamics, which leverages a Lagrange relaxation for devising constraint-aware stochastic subgradient algorithm to deal with the challenge of hybrid couplings. To further boost the performance by integrating powerful machine learning techniques, we then advocate a predictive provisioning approach, where future request arrivals can be estimated accurately. To align with practical deployments, we incorporate a tunable prediction window mechanism, which well balances the potential improvement and degradation of online performance in imperfect prediction scenarios. With rigorous theoretical analysis and extensive trace-driven evaluations, we show the superior performance of our proposed algorithms for online and offline services co-location at the edge.
Driven by the accelerated convergence of artificial intelligence (AI) and the Internet of Things (IoT), the recent years have witnessed the booming of Artificial Intelligence of Things (AIoT). Edge clouds place computing and service capabilities at the network edges to reduce network transmission overhead, which has been widely recognized as the critical infrastructure for AIoT applications. Meanwhile, to accelerate computationintensive edge cloud AI operations, specialized AI accelerators such as GPU, NPU, and TPU have been increasingly integrated into edge clouds. For such emerging XPU edge clouds, utilizing costly XPUs more efficiently has become a significant challenge. In this paper, we present an online optimization framework for joint resource provisioning and batch scheduling for more costefficient AIoT inference serving in an XPU edge cloud. The essential optimization process for the online framework is to first adaptively batch inference tasks to increase the system throughput without compromising the service level agreement (SLA). Next, heterogeneous XPU resources are provisioned for the batches. Finally, the resource instance is consolidated to a minimum of physical servers. Via extensive trace-driven simulations, we verify the performance of the presented online optimization framework.