9 June 2021 in IEEE Embedded Systems Letters
As deep learning (DL) inference applications are increasing, an embedded device tends to equip neural processing units (NPUs) in addition to a CPU and a GPU. For fast and efficient development of DL applications, TensorRT is provided as the software development kit for the NVIDIA hardware platform, including optimizer and runtime that delivers low latency and high throughput for DL inference. Like most DL frameworks, TensorRT assumes that the inference is executed on a single processing element, GPU, or NPU, not both. In this letter, we propose a parallelization methodology to maximize the throughput of a single DL application using both GPU and NPU by exploiting various types of parallelism on TensorRT. With six real-life benchmarks, we could achieve 81%–391% throughput improvement over the baseline inference using GPU only.
2 November 2022 in IEEE Transactions on Computers
To cope with the increasing demand for deep learning applications in embedded systems, emerging embedded devices tend to equip multiple heterogeneous processors, including GPU and deep learning hardware accelerator, called neural processing unit (NPU). It becomes popular to run multiple deep learning (DL) applications simultaneously to provide several functionalities. In this work, we assume that applications have real-time constraints that may vary at run time. While extensive studies have been conducted recently to find an efficient mapping of multiple DL applications on various hardware platforms, they do not consider the constraints imposed by the NPU and the associated software development kit (SDK) in a real embedded platform. In this paper, we propose a novel energy-aware mapping methodology of multiple DL applications onto a real embedded system that has multiple heterogeneous processors. The objective is to minimize energy consumption while satisfying the real-time constraints of all applications. In the proposed scheme, we first select Pareto-optimal mapping solutions for each application. Then mapping combination is explored, considering the scenario that indicates the dynamism of applications while satisfying the constraints. Also, we reduce energy consumption by tuning the frequency of processors. We could satisfy up to 40% higher deadline constraints and reduce the energy consumption by 22% ∼ 31% compared to the static mapping methods with real-life applications and different scenarios on a real platform.
1 January 2021 in IEEE Access
Parallel scheduling of multiple real-time applications onto heterogeneous processors is needed in the emerging embedded systems such as self-driving cars, smart cameras, and smartphones. Assuming that an embedded application is specified as a synchronous dataflow (SDF) graph or its extension, we propose a novel parallel scheduling methodology based on an evolutionary algorithm where the mapping of tasks onto processors is evolved to optimize a given objective function in an iterative fashion. In each iteration, we use an existing worst-case response time (WCRT) analysis tool to check if all applications satisfy their real-time requirements by translating each SDF graph into a directed acyclic graph (DAG) that is assumed in the WCRT analysis tool. Since the WCRT analysis must be performed in each iteration of evolution, we propose a clustering technique to reduce drastically the analysis time that depends on the number of nodes and their dependency. We formally prove that the proposed clustering technique does not change the estimated WCRT of each application. The effectiveness of the proposed scheduling methodology with the clustering technique is verified with extensive experiments using real-life benchmarks, randomly generated graphs, and the comparison with the existing technique.
7 July 2021 in Design Automation for Embedded Systems
1 June 2023 in IEEE Access
As deep learning applications are getting popular in embedded systems, how to support deep learning applications in the model-based embedded software design methodology becomes a challenging problem. A previous solution is to represent each deep learning application with a model. However, it requires significant efforts to translate specifications and obtain good performance by applying optimization techniques to deep learning applications. In this work, we propose a novel methodology that leverages the benefits of using deep learning software development kit (SDK) for performance optimization. In the proposed methodology, we first obtain the Pareto-optimal mapping solutions of deep learning applications using the SDK associated with the hardware platform. Afterward, we perform mapping of dataflow tasks and selection of mapping solutions of deep learning (DL) applications together through a genetic algorithm. We could confirm 4.5 times higher throughput performance than the previous approach that translate a DL application to a dataflow model. Experiments with a real-life example and randomly generated graphs show that we could reduce at least 5% of the maximum utilization compared to our previous work that maps DL applications and dataflow applications sequentially.
31 October 2021 in KIISE Transactions on Computing Practices
임베디드 시스템에서 딥 러닝 응용에 대한 필요가 증가함에 따라, 응용을 가속하는 데에 있어서 CPU가 아닌 처리 요소(processing element)를 임베디드 기기에 포함되고 있다. NVIDIA Jetson AGX Xavier는 대표적인 예제로 8-core CPU 뿐만 아니라 GPU와 2개의 딥러닝 가속기를 함께 갖고 있어서 자원이 제한된 환경에서 딥 러닝 응용의 성능을 끌어올리는 데에 활용된다. 임베디드 기기가 이기종처리 요소를 제공한다고 하여도, 이런 다양한 요소들을 함께 활용하여 성능을 올리는 것은 상당한 노력을 필요로 한다. 본 논문에서는 기존의 존재하는 여러 기법들과 우리가 제안하는 네트워크 파이프라이닝 기법을 함께 조합하여 이기종 처리요소를 가진 Xavier에서 딥 러닝 응용의 처리량을 최대화 하는 기법을 제안한다. 여러 개의 이미지 분류 예제와 사물 인식 예제를 통해 하나의 GPU를 사용하는 기존의 방법 대비 최대 355%까지 성능이 향상되는 것을 확인하였다.