High-resolution high-frame-rate videos can record motion scenes detailedly and smoothly, but usually only professional cameras have enough transmission bandwidth to meet the video capture requirement. The conventional solutions use video processing methods such as video super-resolution (VSR) and video frame interpolation (VFI), but their results suffer from unreal spatial-temporal details in complex dynamic cases. To address this problem, we reconstruct a more real high-resolution high-frame-rate video using a hybrid video input, including a low-resolution high-frame-rate video (main video) and a high-resolution low-frame-rate video (auxiliary video). We propose a deep learning model named HIS-VSR, which consists of three parts: super-resolution of the main video, detail feature extraction of the auxiliary video and hybrid video information aggregation. Among them, the first part processes the main video to generate preliminary high-resolution frames; the second part warps the auxiliary frames for alignment and extracts their high-resolution detail features; the last part uses a weighted aggregation method to fuse the results of the first and second part. We train our model on synthetic datasets and demonstrate its excellent performance of reconstructing dynamic scenes by comparing it with Deep-SloMo on synthetic and real videos.
Moving object detection under a moving camera is a challenging question, especially in a complex background. This paper proposes a background orientation field reconstruction method based on Poisson fusion for detecting moving objects under a moving camera. As enlightening by the optical flow orientation of a background is not dependent on the scene depth, this paper reconstructs the background orientation through Poisson fusion based on the modified gradient. Then, the motion saliency map is calculated by the difference between the original and the reconstructed orientation field. Based on the similarity in appearance and motion, the paper also proposes a weighted accumulation enhancement method. It can highlight the motion saliency of the moving objects and improve the consistency within the object and background region simultaneously. Furthermore, the proposed method incorporates the motion continuity to reject the false positives. The experimental results obtained by employing publicly available datasets indicate that the proposed method can achieve excellent performance compared with current state-of-the-art methods.
1 May 2023 in IEEE Transactions on Geoscience and Remote Sensing
Due to cluttered backgrounds, long-range imaging, rapid relative motion, etc., the detection of small aerial targets remains a challenge. Existing works have illustrated the importance of temporal cues for robust small aerial target detection. However, the strong assumptions about the temporal cues give such methods high computational complexity and weak adaptability. To address this problem, we treat trajectory verification as a classification problem using convolutional neural networks. A novel small aerial target detection method that uses trajectory hypothesis and verification is proposed in this study. First, the IMU assisted inter-frame difference is used to extract target candidates from each single image. Then, based on the continuous and smoothness characteristics of the target trajectory, hypotheses of the required length are generated by linking target candidates in adjacent frames. Finally, the trajectory hypotheses are sent to a trained classification convolution neural network for verification. The detected targets are traced back from the verified trajectory. Furthermore, trajectory merge is conducted to link adjacent trajectory segments. The proposed method can achieve real-time processing. Experimental results on public datasets show that the proposed method performs better than existing methods.
5 August 2025 in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
Infrared small target detection holds great potential for various applications but also faces numerous challenges. Among these, detection algorithms for moving small targets are increasingly gaining attention. Most algorithms focus solely on extracting features from the spatial domain. However, this approach often leads to suboptimal detection performance. While some methods attempt to incorporate temporal information during detection, non-learning-based approaches are highly sensitive to noises and require manual parameter adjustments based on the input data. To tackle this problem, this paper proposes a novel multi-frame network called as A Spatio-Temporal Attention Network with Multi-Frame Information. This proposed method thinks of the infrared detection of moving small targets as the trajectory classification. First, the candidate targets are extracted by analyzing the spatial information of each image frame. The trajectories formed by candidate targets contain temporal information. The subsequent analysis of the temporal motion and appearance information serves to distinguish the real target from the background interference among the candidate targets. Concurrently, the coordinate attention block is integrated into the model to direct the algorithm to emphasize the local detail features of the target, thereby enhancing the processing capability of the temporal information. The experimental results on two public datasets demonstrate that the proposed method can substantially reduce the number of false alarm targets without significantly losing real targets.
1 July 2020 in International Journal of Advanced Robotic Systems
Due to the clutter background motion, accurate moving object segmentation in unconstrained videos remains a significant open problem, especially for the slow-moving object. This article proposes an accurate moving object segmentation method based on robust seed selection. The seed pixels of the object and background are selected robustly by using the optical flow cues. Firstly, this article detects the moving object’s rough contour according to the local difference in the weighted orientation cues of the optical flow. Then, the detected rough contour is used to guide the object and the background seed pixel selection. The object seed pixels in the previous frame are propagated to the current frame according to the optical flow to improve the robustness of the seed selection. Finally, we adopt the random walker algorithm to segment the moving object accurately according to the selected seed pixels. Experiments on publicly available data sets indicate that the proposed method shows excellent performance in segmenting moving objects accurately in unconstraint videos.
In the field of infrared small target detection (ISTD), single-frame ISTD (SISTD), using only spatial features, cannot deal well with dim targets in cluttered backgrounds. In contrast, multi-frame ISTD (MISTD), utilizing spatio-temporal information from videos, can significantly enhance moving target features and effectively suppress background interference. However, current MISTD algorithms are limited by fixed-size time windows, resulting in an inability to adaptively adjust the input amount of spatio-temporal information for different detection scenarios. Moreover, utilizing spatio-temporal features remains a significant challenge in MISTD, particularly in scenarios involving slow-moving targets and fast-moving backgrounds. To address the above problems, we propose a memory-based temporal Transformer U-Net (MTTU-Net), which integrates a memory-based temporal Transformer module (MTTM) into U-Net. Specifically, MTTM utilizes the proposed D-ConvLSTM to sequentially transmit the temporal information in the form of memory, breaking through the limitation of the time window paradigm. And we propose a Transformer-based interactive fusion approach, which is dominated by spatial features of the to-be-detected frame and supplemented by temporal features in the memory, thereby effectively dealing with targets and backgrounds with various motion states. In addition, MTTM is divided into a temporal channel-cross Transformer module (TCTM) and a temporal space-cross Transformer module (TSTM), which achieve target feature enhancement and global background perception through feature interactive fusion in the channel and space dimensions, respectively. Extensive experiments on IRDST and IDSMT datasets demonstrate that our MTTU-Net outperforms existing MISTD algorithms, and they verify the effectiveness of the proposed modules.