16 July 2021 in IEEE Micro
Near-memory processing (NMP) is a prospective paradigm enabling memory-centric computing. By moving the compute capability next to the main memory (DRAM modules), it can fundamentally address the CPU-memory bandwidth bottleneck and thus effectively improve the performance of memory-constrained workloads. Using the personalized recommendation system as a driving example, we developed a scalable, practical DIMM-based NMP solution tailor-designed for accelerating the inference serving. Our solution is demonstrated on a versatile FPGA-enabled NMP platform called AxDIMM that allows rapid prototyping and evaluation of NMP's performance potential on real hardware under a realistic system setting using industry-representative recommendation framework. We experimentally validated the performance of a two-ranked AxDIMM prototype which achieves up to 1.89x speedup in latency and 31.6% memory energy saving for embedding operations. For end-to-end recommendation inference serving, AxDIMM improves the throughput up to 1.5x and latency-bounded throughput up to 1.77x, respectively.
3 August 2020 in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
Recent work has demonstrated great potentials of neural network-inspired analog-to-digital converters (NNADCs) in many emerging applications. These NNADCs often rely on resistive random-access memory (RRAM) devices to realize basic NN operations, and usually need high-precision RRAM (6–12 b) to achieve moderate quantization resolutions (4–8 b). Such an optimistic assumption of RRAM precision, however, is not well supported by practical RRAM arrays in the large-scale production process. In this article, we evaluate two new designs of NNADC with low-precision RRAM devices. They take advantage of traditional two-stage/pipelined hardware architecture and a custom deep-learning-based building block design methodology. Results obtained from SPICE simulations demonstrate a robust design of an 8-b subranging NNADC using 4-b RRAM devices, as well as a 14-b pipelined NNADC using 3-b RRAM devices. The evaluations on the two NNADCs suggest that pipelined architecture is better to achieve higher-resolution using lower precision RRAM. We also perform design space exploration on the building blocks of NNADCs to achieve a balanced performance tradeoff. Comprehensive comparisons reveal improved power, speed performance, and competitive figure of merits (FoMs) of the pipelined NNADC, compared with state-of-the-art NNADCs and traditional ADCs. In addition, the proposed pipelined NNADC can support reconfigurable high-resolution nonlinear quantization with high conversion speed and low conversion energy, enabling intelligent analog-to-information interfaces for near-sensor processing.
25 September 2019 in IEEE Design & Test
To reduce the excess energy footprint of deep neural networks (DNNs), diverse solutions have been proposed across the system stacks. For instance, algorithm-level network pruning techniques remove insignificant connections to simplify the NN models, whereas architecture-level techniques leverage approximate units to perform computations in a cost-efficient manner. However, the collaborations across different optimization layers are rarely explored. We observe that with careful characterization, cross-stack techniques can be leveraged to further reduce the energy consumption of DNNs with minimal accuracy impact compared with employing a single technique. Specifically, in this paper, we bridge the gap between network pruning and approximate multiplication by proposing a collaborative solution to optimize computational power. We first characterize the error resilience and energy consumption of DNNs in each layer and apply layer-wise network pruning. Based on the pruned model, we propose an incremental approximation and retraining scheme to apply approximate multiplication. The experimental results from three datasets demonstrate that the proposed cross-stack approach achieves 26.23% and 29.97% energy reduction compared with the approximation-only and pruning-only method with only 1.39% accuracy degradation compared to the optimal accuracy of the original networks on average.
24 January 2025 in IEEE Transactions on Dependable and Secure Computing
Trusted Executing Environments (TEEs) in hardware designs protect program execution from other untrusted software programs in the processor as well as untrusted off-chip hardware components. Meanwhile, Near-Memory Processing (NMP) has shown performance and energy benefits on memory-intensive workloads. Recently, novel memory encryption schemes have been proposed to allow TEEs to leverage the benefit of NMP without trusting the NMP components. In this paper, we present a system design of confidential computing with NMP that can be directly used in Intel SGX, a TEE platform available in commercial processors today. We develop the full software stack and evaluate the results on commercial processors with the emulated AxDIMM, an FPGA-based NMP platform. In our case study on personalized Deep Learning Recommendation Model (DLRM) inference, the proposed confidential computing in NMP achieves up to 1.51× latency reduction and up to 2.57× throughout improvement.
12 December 2022 in arXiv
Online personalized recommendation services are generally hosted in the cloud where users query the cloud-based model to receive recommended input such as merchandise of interest or news feed. State-of-the-art recommendation models rely on sparse and dense features to represent users' profile information and the items they interact with. Although sparse features account for 99% of the total model size, there was not enough attention paid to the potential information leakage through sparse features. These sparse features are employed to track users' behavior, e.g., their click history, object interactions, etc., potentially carrying each user's private information. Sparse features are represented as learned embedding vectors that are stored in large tables, and personalized recommendation is performed by using a specific user's sparse feature to index through the tables. Even with recently-proposed methods that hides the computation happening in the cloud, an attacker in the cloud may be able to still track the access patterns to the embedding tables. This paper explores the private information that may be learned by tracking a recommendation model's sparse feature access patterns. We first characterize the types of attacks that can be carried out on sparse features in recommendation models in an untrusted cloud, followed by a demonstration of how each of these attacks leads to extracting users' private information or tracking users by their behavior over time.