This paper presents a communication-aware processing-in-memory deep neural network accelerator, which implements an in-memory entry-counting scheme for low bitwidth quantized multiplication-and-accumulations (MACs). To maintain good accuracy on ImageNet, the proposed design adopts a full-stack co-design methodology, from algorithms, circuits to architectures. In the algorithm level, an entry-counting based MAC is proposed to fit the learned step-sized quantization scheme, and exploit the sparsity of both activations and weights intrinsically. In the circuit level, content addressable memory cells and multiplexed arrays are developed in the processing-inmemory macro. In the architecture level, the proposed design is compatible with different stationary dataflow mappings, further reducing the memory access. An in-memory entry-counting silicon prototype and its entire peripheral circuits are fabricated in 65nm LP CMOS technology with an active area of 0.76×0.66 mm2. The 7.36-Kb processing-in-memory macro with 128 search entries can reduce the multiplication number by 12.8×. The peak throughput is 3.58 GOPS, achieved at a clock rate of 143MHz and a power supply of 1.23V. The peak energy efficiency of the processing-in-memory macro is 11.6 TOPS/W, achieved at a clock rate of 40MHz and a power supply of 1.01V. Note that the physical design of the entry-counting memory is completed in a standard digital placement and routing flow by augmenting the library with two dedicated memory cells. A 3-bit quantized ResNet-18 on the ImageNet dataset is performed, where the top-1 accuracy is 64.4%.
Blockchain technology is increasingly being used in Internet of things (IoT) devices for information security and data integrity. However, it is challenging to implement complex hash algorithms with limited resources in IoT devices owing to large energy consumption and a long processing time. This paper proposes a RISC-V processor with memristor-based in-memory computing (IMC) for blockchain technology in IoT applications. The IMC-adapted instructions were designed for the Keccak hash algorithm by virtue of the extendibility of the RISC-V instruction set architecture (ISA). Then, a RISC-V processor with area-efficient memristor-based IMC was developed based on an open-source core for IoT applications, Hummingbird E200. The general compiling policy with the data allocation method is also disclosed for the IMC implementation of the Keccak hash algorithm. An evaluation shows that >70% improvements in both performance and energy saving were achieved with limited area overhead after introducing IMC in the RISC-V processor.
22 May 2023 in IEEE Transactions on Image Processing
Although deep learning-based (DL-based) image processing algorithms have achieved superior performance, they are still difficult to apply on mobile devices (e.g., smartphones and cameras) due to the following reasons: 1) the high memory demand and 2) large model size. To adapt DL-based methods to mobile devices, motivated by the characteristics of image signal processors (ISPs), we propose a novel algorithm named LineDL. In LineDL, the default mode of the whole-image processing is reformulated as a line-by-line mode, eliminating the need to store large amounts of intermediate data for the whole image. An information transmission module (ITM) is designed to extract and convey the interline correlation and integrate the interline features. Furthermore, we develop a model compression method to reduce the model size while maintaining competitive performance; that is, knowledge is redefined, and compression is performed in two directions. We evaluate LineDL on general image processing tasks, including denoising and superresolution. The extensive experimental results demonstrate that LineDL achieves image quality comparable to that of state-of-the-art (SOTA) DL-based algorithms with a much smaller memory demand and competitive model size.
The tongue is the only human organ that can stick out of the body. Using the human tongue is considered to be a novel biometrics method because of its rich individual characteristics. How to represent the dynamic shape changes of the tongue is a challenge for identity verification. A new framework for human tongue modeling and recognition based on image sequences is proposed in this paper. In this framework, we exploit appearance manifold learning to obtain a low-dimensional embedding of the sequence of tongue images, and we propose nearest manifold measurement for measuring the similarities in multiple manifolds. Based on the database of tongue image sequences, the results of our experiments showed that the proposed framework not only can effectively perform tongue biometric recognition but can also provide robustness, which is very important for the Internet of medical things platform.
1 May 2025 in IEEE Transactions on Circuits and Systems I: Regular Papers
This paper proposed an algorithm-hardware co-design of an event-driven spiking neural network (SNN) accelerator for classification tasks of event-based data from dynamic vision sensors (DVS), which can implement a feed-forward SNN with a maximum network size of 1 million synapses. Configurable structured sparsity is introduced between the first layer and the second layer to improve energy efficiency and balance the workload between different processing elements (PEs). The number of available neurons in the accelerator is sparsity-dependent and ranges from 1024 to 4096. The modified leaky-integrate-fire (LIF) neuron model and an event-driven neuron update scheme are employed in both algorithm and hardware to fully utilize the natural sparsity of DVS event stream. Early-stop inference strategy on the hardware enables a trade-off between inference accuracy and efficiency. A three-layer fully connected SNN is trained through backpropagation through time (BPTT) and is implemented and evaluated on Xilinx ZCU104 FPGA. Our design can achieve 96.0% accuracy on the N-MNIST dataset and 79.0% accuracy on the DVS128-Gesture dataset both at 50% sparsity. The top performance of the accelerator on ZCU104 is 3.22 GSOP/S and 3.99 GSOP/W at 250 MHz.