6 February 2020 in IEEE Internet of Things Journal
Recent advancements in deep neural networks (DNNs) have enabled us to solve traditionally challenging problems. To deploy a service based on DNNs, since DNNs are compute intensive, consumers need to rely on compute resources in the cloud. This approach, in addition to creating a dependency on the high-quality network infrastructure and data centers, raises new privacy concerns because of the sharing of private data. These concerns and challenges limit the widespread use of DNN-based applications, so many researchers and companies are trying to optimize DNNs for fast in-the-edge execution. Executing DNNs is further pushed to the edge with the widespread use of embedded processors and ubiquitous wireless networks in Internet-of-Things (IoT) devices. However, inadequate power and computing resources of edge devices, along with the small number of local requests, limit the use of prevalent optimization techniques such as batch processing. In this article, we enable the utilization of the aggregated computing power of several IoT devices by creating a local collaborative network for a subset of DNNs, visual-based applications. In this approach, IoT devices cooperate to conduct single-batch inferencing in real time while exploiting several new model-parallelism methods, which will be introduced in this article. Our approach enhances the collaborative system by creating a balanced and distributed processing pipeline while adjusting the tasks in real time. For experiments, we deploy a system with up to 10 Raspberry Pis and execute state-of-the-art visual models, such as AlexNet, VGG16, Xception, and C3D.
1 October 2018 in IEEE Robotics and Automation Letters
Recognition ability and, more broadly, machine learning techniques enable robots to perform complex tasks and allow them to function in diverse situations. In fact, robots can easily access an abundance of sensor data that is recorded in real-time such as speech, image, and video. Since such data is time sensitive, processing it in real-time is a necessity. Moreover, machine learning techniques are known to be computationally intensive and resource hungry. As a result, an individual resource-constrained robot, in terms of computation power and energy supply, is often unable to handle such heavy real-time computations alone. To overcome this obstacle, we propose a framework to harvest the aggregated computational power of several low-power robots for enabling efficient, dynamic, and real-time recognition. Our method adapts to the availability of computing devices at runtime and adjusts to the inherit dynamics of the network. Our framework can be applied to any distributed robot system. To demonstrate, with several Raspberry-Pi3-based robots (up to 12) each equipped with a camera, we implement a state-of-the-art action recognition model for videos and two recognition models for images. Our approach allows a group of multiple low-power robots to obtain a similar performance (in terms of the number of images or video frames processed per second) compared to an high-end embedded platform, Nvidia Tegra TX2.
25 July 2019 in IEEE Micro
Systolic arrays with promising attributes such as high degree of concurrent computation and high data-reuse rate are attractive solutions for dense linear algebra. Recently, systolic arrays have also been used for accelerating the inference of deep neural networks (DNNs). However, since DNN inference is a sparse problem, it cannot fully benefit from the fundamental advantages offered by systolic arrays. To solve this challenge, we propose Eridanus, an approach to structured pruning that produces DNNs compatible with the synchronous and rhythmic flow of data from memory to the compute units of systolic arrays.
5 September 2012 in ACM SIGARCH Computer Architecture News
Exclusive last-level caches (LLCs) reduce memory accesses by effectively utilizing cache capacity. However, they require excessive on-chip bandwidth to support frequent insertions of cache lines on eviction from upper-level caches. Non-inclusive caches, on the other hand, have the advantage of using the on-chip bandwidth more effectively but suffer from a higher miss rate. Traditionally, the decision to use the cache as exclusive or non-inclusive is made at design time. However, the best option for a cache organization depends on application characteristics, such as working set size and the amount of traffic consumed by LLC insertions. This paper proposes FLEXclusion, a design that dynamically selects between exclusion and non-inclusion depending on workload behavior. With FLEXclusion, the cache behaves like an exclusive cache when the application benefits from extra cache capacity, and it acts as a non-inclusive cache when additional cache capacity is not useful, so that it can reduce on-chip bandwidth. FLEXclusion leverages the observation that both non-inclusion and exclusion rely on similar hardware support, so our proposal can be implemented with negligible hardware changes. Our evaluations show that a FLEXclusive cache reduces the on-chip LLC insertion traffic by 72.6% compared to an exclusive design and improves performance by 5.9% compared to a non-inclusive design.
16 September 2022 in ACM Transactions on Architecture and Code Optimization
As CUDA becomes the de facto programming language among data parallel applications such as high-performance computing or machine learning applications, running CUDA on other platforms becomes a compelling option. Although several efforts have attempted to support CUDA on devices other than NVIDIA GPUs, due to extra steps in the translation, the support is always a few years behind CUDA’s latest features. In particular, the new CUDA programming model exposes the warp concept in the programming language, which greatly changes the way the CUDA code should be mapped to CPU programs. In this article, hierarchical collapsing that correctly supports CUDA warp-level functions on CPUs is proposed. To verify hierarchical collapsing , we build a framework, COX , that supports executing CUDA source code on the CPU backend. With hierarchical collapsing , 90% of kernels in CUDA SDK samples can be executed on CPUs, much higher than previous works (68%). We also evaluate the performance with benchmarks for real applications and show that hierarchical collapsing can generate CPU programs with comparable or even higher performance than previous projects in general.