4 November 2021 in IEEE Transactions on Circuits and Systems II: Express Briefs
With the development of artificial intelligence research, the convolutional neural networks (CNN) triumphantly continue to conquer diverse visual tasks, with lots of technical and practical problems emerging. When the detection networks need to meet the high-speed processing requirement and the low power consumption constraint in edge computing, dedicated circuits and algorithm optimization are necessary. In this brief, we propose a non-maximum suppression (NMS) coprocessor that cooperated with a Vision Processor (VP) to speed up the detection. It contains a parallel process unit and dynamic memory management to accelerate NMS processing. Meanwhile, the software workflow provides a comprehensive method to convert the one-stage detection network to our processor in a post-training manner. The experimental results show that our co-design solution reaches 166 FPS to run a MobileNet-SSD, promising to deploy in edge-computing scenarios. Moreover, the detection accuracy on the Airbus dataset is 88.6%, favorable in practical applications.
8 June 2022 in IEEE Transactions on Circuits and Systems II: Express Briefs
Vision chips perform image capture and real-time intelligent image processing by integrating an imager and a vision processor on a single chip, having broad application prospects. This brief proposes a programmable and flexible vision processor with a dual-issue micro-architecture. The processor consists of a reconfigurable vector unit, a flexible memory access network, and a non-maximum suppression (NMS) block. It can efficiently implement both deep neural network (DNN) and traditional computer vision (CV) algorithms. The vector unit performs single-instruction multiple-vector (SIMV) parallel operations with reconfigurable vector width. The flexible memory access network adaptively supports multiple vector operations under different vector widths. A four-MAC processing element (PE) in the vector unit is designed to increase computational power and data reuse rate. The NMS block can speed up the object location processing of the detection networks. The chip is fabricated in a 28nm process. The experimental results show that the maximum clock frequency, peak performance, and peak energy efficiency are 600MHz, 1.2TOPS, and 2.03TOPS/W, respectively. The Mobilenet-V1 processing achieves a throughput of 404 fps under a $256\times 224$ image size and an 87.15%(top-5) accuracy on the ImageNet dataset.
15 December 2023 in IEEE Transactions on Circuits and Systems I: Regular Papers
Edge computing of reliable multimodal (2D RGB/3D RGB-Depth) data has a wide range of applications. However, many of currently reported visual processors cannot flexibly handle multimodal data, e.g., the visual streams of RGB-Depth data. The key challenge exists that these prior visual processors do not come with efficient and unified instruction set architecture (ISA) for both conventional and intelligent cognition on the 2D/3D multimodal sensory data. To fill such a gap, this paper proposes a programmable intelligent visual vector processor compatible with multimodal 2D/3D visual data processing ( $1920\times 1080$ -pixel resolution). The processor consists of a reconfigurable processing element (PE) array, a memory access network flexibly configurable to be fine-or coarse-grained, and a high throughput I/O interface. The vectorial PE array with neighbor PE access increases the data reuse rate and parallel computation efficiency, and can implement both convolutional neural networks (CNNs) and conventional image processing algorithms. The proposed ISA is customized and optimally tailored targeting 2D/3D image processing from RGB/Time-of-Flight(ToF) raw data to intelligent inference results. The chip is fabricated in a 55-nm CMOS process. The experimental results showed that the area efficiency, peak performance, and peak throughput of our chip attained as high as 14.41 GOPS/mm $^2$ , 409.6 GOPS, and 9.6 Gbps at 200 MHz, respectively. The measured processing speeds of this chip on ToF depth reconstruction is 87 fps ( $480\times 270$ ) or 31 fps( $1920\times 1080$ ),on 3D object classification is 219 fps ( $256\times 256$ ), and on CNN-based 2D object tracking is 36 fps ( $256\times 256$ ).