Recently, Deep Learning (DL) models have demonstrated great success for its attractive ability of high accuracy used in artificial intelligence Internet of Things applications. A common deployment solution is to run such DL inference tasks on edge servers. In a DL inference, each operator takes tensors as input and run in a tensor virtual machine, which isolates resource usage among operators. Nevertheless, existing edge-based DL inference approaches can not efficiently use heterogeneous resources (e.g., CPU and low-end GPU) on edge servers and result in sub-optimal DL inference performance, since they can only partition operators in a DL inference with equal or fixed ratios. It is still a big challenge to support partition optimizations over edge servers for a wide range of DL models, such as Convolution Neural Network (CNN), Recurrent Neural Network (RNN) and Transformers. In this paper, we present EOP, an Efficient Operator Partition approach to optimize DL inferences over edge servers, to address this challenge. Firstly, we carry out a large-scale performance evaluation on operators running on heterogeneous resources, and reveal that many operators do not follow similar performance variation when input tensors change. Secondly, we employ three categorized patterns to estimate the performance of operators, and then efficiently partition key operators and tune partition ratios. Finally, we implement EOP on TVM, and experiments over a typical edge server show that EOP improves the inference performance by up to 1.25−1.97× for various DL models compared to state-of-the-art approaches.
JointCloud gives developers the freedom to create applications that are portable across multiple clouds (e.g., Amazon EC2, Aliyun ECS). However, any modification in a vendor-lock API will make JointCloud unusable due to the problem of API synchronization. We assume that adapter flows should be decoupled from the manners in which they are executed. To verify this idea, we built Wukong, a framework that can dynamically map front-end adapter flows' descriptions to a broad range of back-end clouds by the generic APIs. Generic APIs for VM services have been proposed to evaluate our Wukong framework. The experimental results have shown three contributions: (1) the overhead of reflection is acceptable when WuKong generates the generic APIs, (2) the feasibility of Wukong has been verified by the major cloud vendor-lock APIs, (3) the usability of Wukong heuristic rules can accelerate the generation of APIs.