31 July 2019 in IEEE Micro
Systems specialized for Artificial Intelligence (AI) workloads are increasingly deployed across the spectrum of computing devices from edge to the cloud. However, their ubiquitous adoption requires bridging two seemingly conflicting challenges - the need to deliver extreme processing efficiencies while employing intuitive (and familiar) programming and user interfaces, making AI systems compelling even for common (non-expert) users. In this paper, we take a significant first step towards this goal and present an end-to-end software stack for the RAPID AI accelerator developed by IBM Research. We achieve this by developing a set of software extensions, called DEEPTOOLS, that leverage and work within popular deep learning frameworks (e.g., TensorFlow), thus requiring no additional user input while still enabling aggressive, accelerator-specific performance optimization that might otherwise be expected from a full, custom framework. DEEPTOOLS has two key components: (i) a compiler runtime called DEEPRT, which given a DNN graph automatically identifies how best to execute it on RAPID and constructs the requisite program binaries, (ii) an execution runtime called RAPIDLIB, which triggers and manages the execution of compute and data-transfer operations on RAPID. We demonstrate the utility of DEEPTOOLS by plugging it into TensorFlow and mapping multiple large and practical DNNs (AlexNet, VGG, ResNet, and LSTM) to RAPID. Further, our unique optimizations yielded substantial improvement in performance over hand-tuned homogeneous mappings.
18 October 2022 in ACM Transactions on Embedded Computing Systems
Hardware acceleration of Artificial Intelligence (AI) workloads has gained widespread popularity with its potential to deliver unprecedented performance and efficiency. An important challenge remains in how AI accelerators are programmed to sustain high utilization without impacting end-user productivity. Prior software optimizations start with an input graph and focus on node-level optimizations, viz. dataflows and hierarchical tiling, and graph-level optimizations such as operation fusion. However, little effort has been devoted to inter-node on-chip scratchpad memory (SPM) management in Deep Learning (DL) accelerators, whose significance is bolstered by the recent trends in complex network topologies and the emergence of eager execution in DL frameworks. We characterize and show that there exists up to a 5.2× performance gap in DL inference to be bridged using SPM management and propose OnSRAM, a novel SPM management framework integrated with the compiler runtime of a DL accelerator. We develop two variants, viz. OnSRAM-Static, which works on static graphs to identify data structures that can be lucratively held on-chip based on their size, liveness and significance, and OnSRAM-Eager, which targets an eager execution model (no graph) and uses a history-based speculative scheme to hold/discard data structures. We integrate OnSRAM with TensorFlow and analyze it on multiple accelerator configurations. Across a suite of 12 images, objects, and language networks, on a 3 TFLOP system with a 2 MB SPM and 32 GBps external memory bandwidth, OnSRAM-Static and OnSRAM-Eager achieve 1.02–4.8× and 1.02–3.1× reduction in inference latency (batch size of 1), over a baseline with no SPM management. In terms of energy savings, we observe average reductions of 1.51× (up to 4.1×) and 1.23× (up to 2.9×) for the static and eager execution scenarios, respectively.
16 October 2015 in International Journal of Parallel Programming
12 July 2024 in IEEE Micro
Artificial Intelligence Unit (AIU) is a specialized accelerator card from IBM offering state-of-the-art compute capabilities (100s of TOPS) through dataflow-driven compute arrays attached to a multi-level hierarchy of distributed memory elements. In mapping entire AI models, functional correctness hinges on maintaining dataflow compatibility between producer-consumer operations i.e., the element organization with which a tensor is produced in memory must match the organization expected by the consumer(s). This paper presents a key component in AIU’s compiler stack, DNNDASHER 1 , a systematic framework to analyze such dataflow incompatibilities and invoke an intermediate operation to shuffle tensor elements within and/or across memory elements to resolve the discrepancy. It targets opportunities to eliminate shuffles, increase granularity of memory accesses, and schedule transactions to maximize interconnect utilization. Compared to well-optimized baseline implementations of 4 CNN and Transformer benchmarks, DNNDaSher achieves of 1.27×-4.12× (average 2.3×) end-to-end latency improvement based on measured execution cycles on the AIU.