8 May 2024 in IEEE Access
Single-frame optical flow estimation is a more challenging task than predicting the optical flow between adjacent frames in a video. This paper presents a two-stream network that combines motion information from correlation images with appearance information from intensity images to estimate optical flow in a single-frame manner. The correlation image generated by a three-phase correlation image sensor (3PCIS) records changes in incident intensity during the exposure time and conveys motion information about moving objects. This is crucial to assist motion-blurred images in estimating the motion state of moving objects. Due to the lack of directly available datasets to train our network, we generate a synthetic dataset. Importantly, we introduce a definition that describes optical flow on a single motion-blurred image, which is essential for creating reasonable ground truth. Our experimental results demonstrate that 1) network trained on our synthetic dataset achieves an average End Point Error (EPE) of 0.357 and generalizes well to real-world scenes; 2) our proposed single-frame method outperforms conventional two-frame-based methods in motion-blurred scenes.
19 February 2025 in IEEE Access
Motion deblurring in scenes involving small moving objects or low-illumination conditions is challenging. This paper presents an effective deep-learning solution that utilizes correlation images as key auxiliaries to address the problem. The correlation image, produced by a three-phase correlation image sensor (3PCIS), is a temporal correlation between incident light and reference signals within a frame time, which encodes intensity changes of incident light over the exposure time. Since correlation images explicitly record motion information lost during the blurring process during exposure, they can be used for accurately identifying the location and degree of blur, especially in low-illumination conditions and scenarios with small moving objects. Therefore, we combine correlation images and motion-blurred images as inputs and build a two-stream network for motion deblurring. Two key designs in our model are 1) Shared-gated Block (SGB), which enables information exchange between the two encoders and selectively allows useful information to pass through the network to obtain high-quality output; 2) a Motion-guided Block (MGB), decoding process that can draw more attention to the blurred areas in the image, thereby achieving clearer textures and details restoration in the blurred areas. The experimental results show that our model not only can successfully eliminate the motion blur in the above challenging scenarios, but also achieves a state-of-the-art 36.02dB in Peak Signal-to-Noise Ratio (PSNR) on the GoPro dataset with simulated correlation images.
6 August 2025 in Neurocomputing
19 February 2025 in SICE Journal of Control, Measurement, and System Integration