3 August 2021 in IEEE Transactions on Circuits and Systems I: Regular Papers
The semi-global matching (SGM) algorithm in stereo vision is a well-known depth-estimation method since it can generate dense and robust disparity maps. However, the real-time processing and low power dissipation, the specifications of the Internet-of-Thing (IoT) applications, are challenging for their computational complexity. In this paper, we propose a hardware-oriented SGM algorithm with pixel-level pipeline and region-optimized cost aggregation for high-speed processing and low hardware-resource usage. Firstly, the matching costs in a region are integrated with an optimization strategy to significantly reduce memory usage and improve the processing speed of the cost aggregation. Then, a two-layer parallel two-stage pipeline (TPTP) architecture, which enables pixel-level processing, is designed to calculate two directions (0° and 135°) aggregation to further solve the crucial computational bottleneck of the SGM algorithm. Finally, the architecture is demonstrated on a low-cost XILINX Spartan-7 device and an advanced Stratix-V FPGA device for VGA ( $640\times 480$ ) depth estimation. The experimental results show that the proposed architecture with compact hardware architecture also ensures accuracy. The pixel-level pipeline architecture enables a processing speed of 355 frames per second (fps) at 109MHz on the Spartan-7 FPGA device and 508 fps at 156MHz on the Stratix-V FPGA. Besides, the coprocessor respectively achieves an energy efficiency of 4.74 nJ/pixel with a power dissipation of 517mW and 4.29nJ/pixel with a power dissipation of 669mW on these two FPGAs.
14 March 2022 in IEEE Transactions on Circuits and Systems II: Express Briefs
Bilateral filter (BF) is a widely applied method for image denoising due to the characteristic of edge-preserving. This brief presents a super-compact hardware architecture for the BF by a piecewise approximate computing algorithm. The contributions are summarized as follows: 1) The architecture significantly reduces the storage and the arithmetic logic for the piecewise approximated filter weights and can attain comparable smoothy and edge-preserving performance to the standard BF. 2) The BF is accelerated by the parallel pixel-level pipeline architecture and the LUT-based divider for normalization. 3) The synthesized result shows that this architecture, even on a low-cost XINILX Zynq-7000 FPGA, can reach real-time denoising at more than 30 frames/second for 8M-pixel ( $3268 \times 2448$ ) videos at the maximum working frequency of 278 MHz with the power dissipation of only 168 mW.
18 July 2022 in IEEE Transactions on Circuits and Systems II: Express Briefs
Stereo matching is an essential topic in computer vision. However, the high complexity and massive computation restrict its applicability in real-time. This brief presents a pixel-level pipeline coprocessor for configurable binocular stereo rectification, region-optimized semi-global matching (SGM), left-right consistency checking (LRC), and disparity refinement. The contributions are summarized as follows: 1) Instead of storing the entire image of a large number of lines, the buffer size can be significantly reduced by the adjustment range computed according to the pre-calibrated coefficients. 2) An approximate divider with Newton iterations can reach the precision of 10−7 for high accuracy rectification with low latency. 3) A four-layer parallel two-stage pipeline is leveraged to address the computational bottleneck of the SGM, i.e., cost aggregation. 4) The pseudo-dual-port SRAMs and stacks mechanism solve the problem of inverse pixel flow in hole filling with low resource consumption. Finally, the proposed architecture is implemented on an Intel Stratix-IV FPGA and can process 144 frames ( $1024\mathbf {\times }768$ -pixel) per second at a maximum working frequency of 113.13 MHz.
27 February 2024 in IEEE Transactions on Circuits and Systems I: Regular Papers
Binocular stereo vision is a depth estimation technique by imitating human eyes. It is widely used in various fields, such as self-driving cars, SLAM, and 3D reconstruction. However, designing a hardware architecture that can balance resource utilization, processing speed, and estimation accuracy remains a significant challenge. This paper proposes a compact and efficient hardware-based design that incorporates linear fitting-based cost fusion, disparity optimization with subpixel interpolation, and multi-directional occlusion filling techniques. Firstly, a gradient-enhanced pipelined matching costs architecture with a resource-saving scheme and re-computation paradigm is proposed to improve the accuracy of edge information. Then, we approximate the nonlinear exponential function by linear fitting to save the hardware resource. Moreover, we design the subpixel interpolation with an SRT radix-4 divider to refine the disparity, which significantly enhances the accuracy of the disparity map in real situations. Finally, we proposed a resource-reused architecture for synchronous hole filling and median filter in post-processing. The disparity map quality of the proposed architecture is evaluated on KITTI2015 datasets, which delivers leading accuracy compared to other state-of-the-art works. The architecture has been successfully implemented and demonstrated on the Stratix-V FPGA platform and achieved 54 frames per second operating at 112 MHz under a resolution of $1920\times 1080$ .
15 September 2023 in IEEE Transactions on Circuits and Systems I: Regular Papers
With the increasing demand for high-resolution video and real-time processing, the limited efficiency of video-denoising algorithms has become a critical factor. This paper proposes a spatio-temporal video denoising co-processor to suppress an image sequence’s spatial and temporal noise. Temporal denoising is achieved by merging the current and previous frames at the pixel level in which the current frame is processed by a spatial filter. After exploiting noise estimation and motion detection, the Wiener filter calculates the merge ratio. Rather than buffering the entire previous frame, the JPEG-like codec can dynamically adjust the compression ratio through a predefined quantization table to satisfy the designed on-chip storage. The experimental results demonstrate that the spatio-temporal denoising co-processor can effectively eliminate the fluctuation of the grayscale value of the noise in videos. Simultaneously, the adaptive codec can reduce the storage space consumption for the frame buffer by at least 80% of the original size. To the best of our knowledge, this is the first fully integrated spatio-temporal denoising co-processor without any external memory. Additionally, the grayscale, RGB, and RAW versions of the co-processor are also implemented on the Stratix V FPGA platform and synthesized in 28nm CMOS technology.