19 August 2021 in IEEE Transactions on Mobile Computing
Many mobile applications have been developed to apply deep learning for video analytics. Although these advanced deep learning models can provide us with better results, they also suffer from the high computational overhead which means longer delay and more energy consumption when running on mobile devices. To address this issue, we propose a framework called FastVA, which supports deep learning video analytics through edge processing and Neural Processing Unit (NPU) in mobile. The major challenge is to determine when to offload the computation and when to use NPU. Based on the processing time and accuracy requirement of the mobile application, we study three problems: \textit{Max-Accuracy} where the goal is to maximize the accuracy under some time constraints, \textit{Max-Utility} where the goal is to maximize the utility which is a weighted function of processing time and accuracy, and \textit{Min-Energy} where the goal is to minimize the energy under some time and accuracy constraints. We formulate them as integer programming problems and propose heuristics based solutions. We have implemented FastVA on smartphones and demonstrated its effectiveness through extensive evaluations.
25 August 2020 in IEEE Transactions on Mobile Computing
High quality video streaming for mobile devices implies high energy consumption due to the transmitted data and the variation of wireless signals. As an example, transmissions in mobile scenarios (e.g., inside a moving bus) consumes more energy for devices than when accessing from a static environment (e.g., at home). The QoE for the user does not substantially increase when watching high bitrate videos in a vibrating environment (i.e., a moving vehicle), as the context, in this case vehicle’s vibration, affects the perceived QoE. To address this problem, we propose to save energy by considering the context (environment) of video streaming. To model the impact of context, we exploit the embedded accelerometer in smartphones to record the vibration level during video streaming. Based on quality assessment experiments, we collect traces and model the impact of video bitrate and vibration level on QoE, and model the impact of video bitrate and signal strength on power consumption. Based on the QoE model and the power model, we formulate the context-aware and energy-aware video streaming problem as an optimization problem. We present an optimal algorithm which can maximize QoE and minimize energy. Since the optimal algorithm requires perfect knowledge of future tasks, we propose an online bitrate selection algorithm. To further improve the performance of the online algorithm, we propose a crowdsourcing based bitrate selection algorithm. Through real measurements and trace-driven simulations, we demonstrate that the proposed algorithms can significantly outperform existing approaches when considering both energy and QoE.
25 March 2024 in IEEE Transactions on Mobile Computing
As Deep Neural Networks (DNNs) have been successfully applied to various fields, there is a tremendous demand for running DNNs on mobile devices. Although mobile GPU can be leveraged to improve performance, it consumes a large amount of energy. After a short period of time, the mobile device may become overheated and the processors are forced to reduce the clock speed, significantly reducing the processing speed. A different approach to support DNNs on mobile device is to leverage the Neural Processing Units (NPUs). Compared to GPU, NPU is much faster and more energy efficient, but with lower accuracy due to the use of low precision floating-point numbers. We propose to combine these two approaches to improve the performance of running DNNs on mobile devices by studying the thermal-aware scheduling problem, where the goal is to achieve a better tradeoff between processing time and accuracy while ensuring that the mobile device is not overheated. To solve the problem, we propose a heuristic-based scheduling algorithm to determine when to run DNNs on GPU and when to run DNNs on NPU based on the current states of the mobile device. The heuristic-based algorithm makes scheduling decisions greedily and ignores their future impacts. Thus, we propose a deep reinforcement learning based scheduling algorithm to further improve performance. Extensive evaluation results show that the proposed algorithms can significantly improve the performance of running DNNs on mobile devices while avoiding overheating.
30 December 2022 in IEEE Transactions on Mobile Computing
Tile-based streaming techniques have been widely used to save bandwidth in $360^{\circ }$ video streaming. However, it is a challenge to determine the right tile size which directly affects the bandwidth usage. Moreover, downloading and processing many small tiles consume a large amount of energy on mobile devices. To solve this problem, we propose to encode the video by taking into account the viewing popularity, where the popularly viewed areas are encoded as macrotiles. We propose techniques for identifying and building macrotiles, and adjusting their sizes to take into account practical issues such as head movement randomness. In some cases, the user's viewing area may not be covered by the constructed macrotiles, and then the conventional tiling scheme is used. To support macrotile based $360^{\circ }$ video streaming, the client selects the right tiles (a macrotile or a set of conventional tiles) with the right quality level to maximize the QoE under bandwidth constraint. We formulate this problem as an optimization problem which is NP-hard, and then propose a heuristic algorithm to solve it. Through extensive evaluations based on real head movement traces, we demonstrate that the proposed algorithm can significantly improve QoE, save bandwidth usage, and reduce energy consumption.
3 August 2023 in Computer
Although deep neural networks (DNNs) can require less processing time when running on neural processing units (NPUs) instead of CPUs, their accuracy can degrade. We point out three research directions to improve DNN performance on mobile devices with NPUs.