Neuromorphic hardware systems have been gaining ever-increasing focus in many embedded applications as they use a brain-inspired, energy-efficient spiking neural network (SNN) model that closely mimics the human cortex mechanism by communicating and processing sensory information via spatiotemporally sparse spikes. In this paper, we fully leverage the characteristics of spiking convolution neural network (SCNN), and propose a scalable, cost-efficient, and high-speed VLSI architecture to accelerate deep SCNN inference for real-time low-cost embedded scenarios. We leverage the snapshot of binary spike maps at each time-step, to decompose the SCNN operations into a series of regular and simple time-step CNN-like processing to reduce hardware resource consumption. Moreover, our hardware architecture achieves high throughput by employing a pixel stream processing mechanism and fine-grained data pipelines. Our Zynq-7045 FPGA prototype reached a high processing speed of 1250 frames/s and high recognition accuracies on the MNIST and Fashion-MNIST image datasets, demonstrating the plausibility of our SCNN hardware architecture for many embedded applications.
28 December 2023 in IEEE Journal of Solid-State Circuits
Edge vision devices face real 3-D scenes with lighting fluctuations. They require a vision chip capable of acquiring 2-D/3-D vision with a high dynamic range and performing real-time intelligent in situ processing, while maintaining low end-to-end latency. Aimed at this versatile edge vision, this article presents a bio-inspired full-spiking vision chip that consists of a single-photon-avalanche-diode (SPAD) image sensor and a spiking vision processor monolithically. To decrease latency while realizing versatile functions, we build a bio-inspired full spiking vision on chip by directly coupling spike-like SPAD imaging with a neuroinspired spike-computing paradigm. By modeling this spiking visual flow, we theoretically demonstrate direct coupling and derive a spike preprocessing method for dim-vision enhancement. The SPAD image sensor equipped with a configurable exposure gating scheme can realize three modes: 2-D, 3-D depth, and dim-vision imaging. The spiking vision processor contains a parallel processing element (PE) array and can be reconfigured as a preprocessor or a spiking neural network (SNN) processor. A prototype chip integrated with a $128\ttimes 128$ SPAD image sensor with a reconfigurable spiking vision processor (256 PEs) was implemented in a 0.18- $\mu$ m CMOS technology. The chip experimentally demonstrates three imaging modes, 99.30% classification accuracy on the MNIST dataset with 2-D vision, only 3.9% accuracy loss under 0.02 lux with dim-vision, 1.68 cm obstacle detection error with 3-D depth vision, and a 3.58 $\mu$ s latency for light adaptation. The chip realizes a 100 dB dynamic range, 100 000 spiking maps per second (SMps), and 81.92 GSOPS at an 80-MHz clock.
Low-light imaging capabilities are in urgent demand in many fields, such as security surveillance, night-time autonomous driving, wilderness rescue, and environmental monitoring. The excellent performance of SPAD devices gives them significant potential for applications in low-light imaging. This article presents a 64 (rows) × 128 (columns) SPAD image sensor designed for low-light imaging. The chip utilizes a three-dimensional stacking architecture and microlens technology, combined with compact gated pixel circuits designed with thick-gate MOS transistors, which further enhance the SPAD’s photosensitivity. The configurable digital control circuit allows for the adjustment of exposure time, enabling the sensor to adapt to different lighting conditions. The chip exhibits very low dark noise levels, with an average DCR of 41.5 cps at 2.4 V excess bias voltage. Additionally, it employs a denoising algorithm specifically developed for the SPAD image sensor, achieving two-dimensional grayscale imaging under 6 × 10−4 lux illumination conditions, demonstrating excellent low-light imaging capabilities. The chip designed in this paper fully leverages the performance advantages of SPAD image sensors and holds promise for applications in various fields requiring low-light imaging capabilities.
8 July 2025 in IEEE Journal of Solid-State Circuits
Edge machine vision requires a vision chip to perform imaging and visual recognition with high speed, high accuracy, and compact size. Current vision chips adopt a two-stage workflow, which uses computer vision processing and a deep neural network (DNN) to perform signal enhancement and recognition, respectively. However, the two stages involve different operations and are separately optimized, which causes degraded recognition accuracy, low inference rate, and huge hardware overhead. To address it, we propose a vision chip that integrates a single-photon avalanche diode (SPAD) image sensor and a reconfigurable spiking neural network (SNN) processor. Inspired by the human visual system that employs a full SNN for signal enhancement and recognition, the vision chip acquires spike-based images and performs an end-to-end SNN embedding image signal enhancement (ISE) for object recognition. The ISE and recognition are trained jointly in a unified deep learning framework to increase recognition accuracy and inference speed. The vision chip only integrates an SNN processor to perform ISE and recognition by being reconfigured into the low-precision multiply-accumulation (MAC) mode or accumulation (ACC) mode, respectively, thereby avoiding a dedicated image signal processor (ISP). Based on a 55-nm CMOS technology, the fabricated vision chip integrates $128{\times }128$ pixels and can implement 4096 ACC/MAC components. The chip was measured to exhibit: 1) a 99.2% accuracy with 10000 inferences/s at 10 lx and 95.3% at 1.43 mlx on the MNIST dataset and 2) a 94.32% accuracy with color mosaic images on the ETH-80 dataset. Our chip maintains a relatively high recognition performance even under an illumination one order of magnitude lower than in prior vision chips.
5 September 2025 in IEEE Signal Processing Letters
High dynamic range (HDR) imaging is essential for edge devices. Hybrid event-frame cameras can provide relative intensity events under HDR conditions and absolute intensity frames. Their image reconstruction pipeline needs to fuse different imaging data in real-time. Due to a lack of large-scale real-world datasets, deep learning algorithms can only rely on synthetic data for training, which face challenges with generalization ability. Model-based computational vision methods do not require training data. However, their event-driven update strategy results in slow processing speeds, making it difficult to achieve real-time performance on edge devices. In this paper, we present a fast HDR image reconstruction pipeline comprising Deblur, Calibration and Fusion, which respectively correspond to frame enhancement, decompression error calibration and nonlinear regression fusion. Our experiments are evaluated on public datasets with challenging lighting conditions. Our algorithm demonstrates improved performance metrics, achieving a 54% faster speed compared to state-of-the-art computational vision methods.
15 December 2023 in IEEE Transactions on Circuits and Systems I: Regular Papers
Edge computing of reliable multimodal (2D RGB/3D RGB-Depth) data has a wide range of applications. However, many of currently reported visual processors cannot flexibly handle multimodal data, e.g., the visual streams of RGB-Depth data. The key challenge exists that these prior visual processors do not come with efficient and unified instruction set architecture (ISA) for both conventional and intelligent cognition on the 2D/3D multimodal sensory data. To fill such a gap, this paper proposes a programmable intelligent visual vector processor compatible with multimodal 2D/3D visual data processing ( $1920\times 1080$ -pixel resolution). The processor consists of a reconfigurable processing element (PE) array, a memory access network flexibly configurable to be fine-or coarse-grained, and a high throughput I/O interface. The vectorial PE array with neighbor PE access increases the data reuse rate and parallel computation efficiency, and can implement both convolutional neural networks (CNNs) and conventional image processing algorithms. The proposed ISA is customized and optimally tailored targeting 2D/3D image processing from RGB/Time-of-Flight(ToF) raw data to intelligent inference results. The chip is fabricated in a 55-nm CMOS process. The experimental results showed that the area efficiency, peak performance, and peak throughput of our chip attained as high as 14.41 GOPS/mm $^2$ , 409.6 GOPS, and 9.6 Gbps at 200 MHz, respectively. The measured processing speeds of this chip on ToF depth reconstruction is 87 fps ( $480\times 270$ ) or 31 fps( $1920\times 1080$ ),on 3D object classification is 219 fps ( $256\times 256$ ), and on CNN-based 2D object tracking is 36 fps ( $256\times 256$ ).