3 May 2021 in IEEE Transactions on Multimedia
Most existing studies performed video denoising by using multiple adjacent noisy frames to recover one clean frame; however, despite achieving relatively good quality for each individual frame, these approaches may result in visual flickering when the denoised frames are considered in sequence. In this paper, instead of separately restoring each clean frame, we propose a multiframe-to-multiframe (MM) denoising scheme that simultaneously recovers multiple clean frames from consecutive noisy frames. The proposed MM denoising scheme uses a training strategy that optimizes the denoised video from both the spatial and temporal dimensions, enabling better temporal consistency in the denoised video. Furthermore, we present an MM network (MMNet), which adopts a spatiotemporal convolutional architecture that considers both the interframe similarity and single-frame characteristics. Benefiting from the underlying parallel mechanism of the MM denoising scheme, MMNet achieves a highly competitive denoising efficiency. Extensive analyses and experiments demonstrate that MMNet outperforms the state-of-the-art video denoising methods, yielding temporal consistency improvements of at least 13.3 $\%$ and running more than 2 times faster than the other methods.
8 June 2022 in IEEE Transactions on Instrumentation and Measurement
Images captured in low-light environments often suffer from issues related to dark illumination and damaged details, which results in poor visibility. To address these problems, existing methods have attempted to enhance the visibility of low-light images using convolutional neural networks (CNNs). However, due to the insufficient consideration of crucial features such as illumination and edge details, most of them yield unnatural illumination and blurry details. In this work, to fully exploit these features, we present a detailed analysis of the illumination and edge features of low-light images, observing that the frequency components of these two features are considerably different. Therefore, we explore the frequency distributions of the feature maps extracted from different layers of a CNN model and try to seek the best representation for the illumination and edge information. Based on this, we present a hierarchical feature mining network (HFMNet) that extracts illumination and edge features in different network layers. Specifically, we build a feature mining attention (FMA) module combined with a hierarchical supervised loss to mine crucial features in appropriate network layer. Since deep hierarchical supervision tends to cause overfitting, we introduce an unpaired adversarial loss for improving the generality of the enhancement model. Through extensive experiments and analysis, we demonstrate the advantages of the proposed network, which achieves the state-of-the-art performance in terms of image quality.
6 April 2022 in IEEE Transactions on Instrumentation and Measurement
The 3-D shape reconstruction is a hot topic in computational imaging and many related techniques have been developed. However, most of these techniques have a limited field-of-view (FOV), which results in difficulties on general application. In this article, an FOV-enlarged single-camera 3-D shape reconstruction system is proposed. By placing a saccade mirror in the light path, the proposed system generates a series of virtual cameras and reconstructs the object with multiview images. As the virtual cameras have the same resolution with the real camera, the system gets a larger FOV for reconstruction without sacrificing image resolution. Besides, compared with conventional stereovision systems, the proposed system gives a prior of postures of virtual cameras and simplifies the calibration procedure of extrinsic parameters, which makes it a compact and economical system for application. Reconstruction experiments demonstrate that the proposed system provides an FOV-enlarged and accurate 3-D shape reconstruction. Furthermore, the robustness of the system to environmental conditions is also verified by real-life experiments.
2 March 2023 in IEEE Transactions on Instrumentation and Measurement
3-D reconstruction of a moving object is a hotpot in structured-light research. However, large-scale motion causes sparse point cloud density and low reconstruction accuracy in a conventional structured-light system (SLS) because the field-of-view (FOV) of the camera is enlarged to prevent object loss. In this article, we propose a gaze tracking 3-D reconstruction system (GT3DRS), which utilizes a saccade mirror to construct a dynamic relationship between the camera and projector for an object with large-scale motion. In the GT3DRS, the real-time position of the moving object is obtained using a tracking algorithm, and a saccade mirror is adopted to keep the moving object constantly located at the center of the FOV according to the position, which enables the camera’s FOV to compactly cover the object. The GT3DRS increases the resolution of the object, effectively enhancing the point cloud density of the reconstructed 3-D profile and improving the reconstruction accuracy. In addition, a new calibration framework is established for the proposed GT3DRS, which reduces the system errors by introducing an assumption of nonideal installation and improves calibration efficiency by building an external parameter transformation model (EPTM) to describe the dynamic relationship. Experimental results verify that the point cloud density is increased by 5.2 times, and the reconstruction accuracy is improved by an average of 10.6 times compared with a conventional SLS. Moreover, the tracking reconstruction video sequence reaches 15 fps.