27 February 2018 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Light-field stereo matching problems are commonly modeled by Markov Random Fields (MRFs) for statistical inference of depth maps. Nevertheless, most previous approaches did not adapt to image statistics but instead adopted fixed model parameters. They explored explicit vision cues, such as depth consistency and occlusion, to provide local adaptability and enhance depth quality. However, such additional assumptions could end up confining their applicability, e.g. algorithms designed for dense view sampling are not suitable for sparse one. In this paper, we get back to MRF fundamentals and develop an empirical Bayesian framework-Robust Pseudo Random Field-to explore intrinsic statistical cues for broad applicability. Based on pseudo-likelihoods with hidden soft-decision priors, we apply soft expectation-maximization (EM) for good model fitting and perform hard EM for robust depth estimation. We introduce novel pixel difference models to enable such adaptability and robustness simultaneously. Accordingly, we devise a stereo matching algorithm to employ this framework on dense, sparse, and even denoised light fields. It can be applied to both true-color and grey-scale pixels. Experimental results show that it estimates scene-dependent parameters robustly and converges quickly. In terms of depth accuracy and computation speed, it also outperforms state-of-the-art algorithms constantly.
3 August 2021 in IEEE Transactions on Circuits and Systems I: Regular Papers
This paper presents a block-based embedded convolutional neural network (CNN) for gesture classification on field-programmable gate array (FPGA) in real time. Gesture recognition is an important tool to spontaneous interact with human machine interface. Many CNN architectures using RGB images have been proposed for gesture classification. RGB based gesture classification may cause incorrect results under insufficient light or similar gestures. In addition, most of the CNN architectures cannot run in real time on edge devices due to their large number of parameters and DRAM data access. In this paper, a block-based CNN using RGB-D data is proposed for gesture classification. Adding depth images to RGB images boots the classification accuracy. A CNN architecture with block-based feature maps is built for embedded FPGA implementations. The total number of parameters of the proposed RGB-D embedded CNN (eCNN) model is only 0.17M and it achieves 99.96% and 99.88% accuracy with 32-bit floating point and 8-bit fixed point implementation for America Sign Language (ASL) data set. The RTL simulation of the proposed eCNN model has the average inference speed of 0.171 milliseconds at frequency of 250MHz for a single pair RGB-D image. Implemented on a FPGA integrated with Microsoft Kinect v2 achieve an inference time in 19.42 ms which achieves high accuracy and real-time performance.
22 July 2022 in IEEE Transactions on Circuits and Systems I: Regular Papers
Face classification is important in many applications such as surveillance, border control, and security systems. However, wide variations in environments such as insufficient light, large distances or pose angles make the task challenging. Depth sensors are added with RGB cameras for improving classification accuracy but commercial RGB-D sensors are most targeted for indoors applications. In this paper, we present and design a Chaos LiDAR depth sesnor that provides high-precision depth images through intelligent correlation processing for both indoors and outdoors applications. Our Chaos LiDAR depth sensor detects range from 2 to 40 meters with precision around 8mm at 20-meter. With the Chaos LiDAR depth as input, we design a RGB-D based face classification embedded CNN (eCNN) model for wide range applications such as dim illumination, various distances and large poses. Our Chaos LiDAR increases around 14.27% classification accuracy compared to RealSense D435i for distance from 3 to 5 meter. The eCNN face classification subsystem is implemented in Xilinx ZCU 102 and achieves 11.11 ms inference time. The eCNN engine achieves a peak throughput at 614.4 GOPS. The overall system including Chaos LiDAR, correlation and eCNN FPGA achieves face classification inference rate of 10fps.
22 November 2016 in IEEE Transactions on Image Processing
Digital refocusing has a tradeoff between complexity and quality when using sparsely sampled light fields for lowstorage applications. In this paper, we propose a fast physically correct refocusing algorithm to address this issue in a twofold way. First, view interpolation is adopted to provide photorealistic quality at infocus-defocus hybrid boundaries. Regarding its conventional high complexity, we devised a fast line-scan method specifically for refocusing, and its 1D kernel can be 30× faster than the benchmark View Synthesis Reference Software (VSRS)1D-Fast. Second, we propose a block-based multi-rate processing flow for accelerating purely infocused or defocused regions, and a further 3-34× speedup can be achieved for high-resolution images. All candidate blocks of variable sizes can interpolate different numbers of rendered views and perform refocusing in different subsampled layers. To avoid visible aliasing and block artifacts, we determine these parameters and the simulated aperture filter through a localized filter response analysis using defocus blur statistics. The final quadtree block partitions are then optimized in terms of computation time. Extensive experimental results are provided to show superior refocusing quality and fast computation speed. In particular, the run time is comparable with the conventional single-image blurring, which causes serious boundary artifacts.
12 March 2025 in IEEE Journal of Solid-State Circuits
Video convolutional neural networks (V-CNNs) take multiple frames as input and leverage temporal information to enhance quality and temporal consistency, making them promising solutions for high-resolution imaging tasks, such as video super-resolution (VSR) and video frame interpolation (VFI). Previous works have proposed CNN accelerators for single-image high-resolution imaging tasks, using layer-fusion (LF) workflows to reduce the need for external memory access (EMA) of intermediate feature maps (FMs). However, V-CNNs demand more EMA and computational complexity, posing implementation challenges for edge devices. Additionally, using deformable convolution (DC) to break through the fixed shape of the kernel receptive field can improve image quality and temporal consistency but requires additional storage and computational logic. In this article, we present a memory-efficient V-CNN processor, VISTA. We introduce a cuboid-based LF (CBLF) workflow for V-CNNs to reuse temporal information from overlapped FMs at different time points, reducing EMA and computational complexity. Moreover, the VISTA adopts a heterogeneous reuse-recomputing approach to handle overlaps between region-of-influence (ROI) pyramids and uses reference-frame-first scheduling (RFFS) to reduce the need for extensive memory usage during cross-frame alignment computations. Furthermore, we apply a hardware-model co-design to devise tile-based offset-confined DC (TODC), which reduces computational logic and saves line buffer usage for the search window with 0.06–0.18 dB of peak signal-to-noise ratio (PSNR) drop in image quality. The 12.6-mm2 VISTA is fabricated using 40-nm CMOS technology and achieves peak throughput of 4K-UHD 60 and 50 frames/s for supporting VSR and VFI applications, respectively. It reduces 33%–53% of input EMA, 19% of activation static random-access memory (SRAM), and 19%–42% of computational complexity.