1 June 2023 in IEEE Access
With the development of AI technology, the parameters and calculation overhead of advanced models have increased exponentially, resulting in the existing low-end GPUs(Graphic Processing Unit) being unable to meet the computing power required for model operation. In order to speed up the inference speed in edge scenarios, various manufacturers have launched NPU(Neural Processor Unit), a special chip for neural networks, which can improve the overall inference efficiency and reduce energy consumption through a certain loss of precision. However, in the current common edge-side solutions, the problem of CPU+GPU+NPU co-processing is not well considered. At the same time, edge-side devices are more easily affected by the ambient temperature. In this paper, CPU+GPU+NPU is used to jointly process edge-side inference tasks, and we first establishes a heterogeneous device temperature perception model based on the ambient temperature of the edge device, then proposed a TAS(temperature-aware schedule) algorithm to control the running speed of the heterogeneous device, and then proposed a task scheduling algorithm for the heterogeneous device, namely TASTS(TAS-based task schedule). At the same time, we also use a hungarian matching algorithm to optimize the final result. This paper finally verified several models in real edge environment, found that it can improve the performance by 20-50% compared with conventional methods under temperature constraints.
30 May 2024 in IEEE Transactions on Emerging Topics in Computing
Driven by the accelerated convergence of artificial intelligence (AI) and the Internet of Things (IoT), the recent years have witnessed the booming of Artificial Intelligence of Things (AIoT). Edge clouds place computing and service capabilities at the network edges to reduce network transmission overhead, which has been widely recognized as the critical infrastructure for AIoT applications. Meanwhile, to accelerate computationintensive edge cloud AI operations, specialized AI accelerators such as GPU, NPU, and TPU have been increasingly integrated into edge clouds. For such emerging XPU edge clouds, utilizing costly XPUs more efficiently has become a significant challenge. In this paper, we present an online optimization framework for joint resource provisioning and batch scheduling for more costefficient AIoT inference serving in an XPU edge cloud. The essential optimization process for the online framework is to first adaptively batch inference tasks to increase the system throughput without compromising the service level agreement (SLA). Next, heterogeneous XPU resources are provisioned for the batches. Finally, the resource instance is consolidated to a minimum of physical servers. Via extensive trace-driven simulations, we verify the performance of the presented online optimization framework.