Video captioning refers to automatic generate natural language sentences which summarize the video contents. Inspired by the visual attention mechanism of human beings, temporal attention mechanism has been widely used in video description to selectively focus on important frames. However, most existing methods based on temporal attention mechanism suffer from the problems of recognition error and detail missing, because temporal attention mechanism cannot further catch significant regions in frames. In order to address above problems, we propose the use of a novel spatial-temporal attention mechanism (STAT) within an encoder-decoder neural network for video captioning. The proposed STAT successfully takes into account both the spatial and temporal structures in a video, so it makes the decoder to automatically select the significant regions in the most relevant temporal segments for word prediction. We evaluate our STAT on two well-known benchmarks: MSVD and MSR-VTT-10K. Experimental results show that our proposed STAT achieves the state-of-the-art performance with several popular evaluation metrics: BLEU-4, METEOR and CIDEr.

Existing CNN-based light field de-occlusion (LF-DeOcc) methods suffer from occlusion removal performance degradation in the presence of large-size occlusions. In this paper, we infer that it is possibly caused by the limited receptive field of CNN and experimentally demonstrate that the de-occlusion performance is high-related to the receptive field. Therefore, a novel LF-DeOcc network based on Swin Transformer and CNN, which aims to exploit both global and local receptive fields, is firstly proposed for light field de-occlusion task. CNNs are employed at shallow layers to compensate for the deficiency of Transformers in extracting local features, while Transformers are employed at deep layers to capture the global patterns of large size occlusions. Hence, by integrating the global and local features, one could restore occlusion-free images effectively. For performance evaluation, a large dataset with mild-to-severe occlusions is developed and tested, with average occlusion rates of 19.82%, 32.13% and 40.56%, respectively. Experimental results show that the proposed network is superior to state-of-the-art methods, achieving 27.87 dB and 29.46 dB on the public dataset and our developed dataset, respectively. Finally, a new evaluation method has been presented in our work, i.e., by utilizing the real target detection task to evaluate the performance of LF de-occlusion algorithms. The practicability of our algorithm is validated using the new evaluation method.

Lane detection plays an important role in improving autopilot’s safety. In this paper, a novel lane-division-lines detection method is proposed, which exhibits good performances in abnormal illumination and lane occlusion. It includes three major components: First, the captured image is converted to aerial view to make full use of parallel lanes’ characteristics. Second, a ridge detector is proposed to extract each lane’s feature points and remove noise points with an adaptable neural network (ANN). Last, the lane-division-lines are accurately fitted by an improved random sample consensus (RANSAC), termed the (regional) gaussian distribution random sample consensus (G-RANSAC). To test the performances of this novel lane detection method, we proposed a new index named the lane departure index (LDI) describing the departure degree between true lane and predicted lane. Experimental results verified the superior performances of the proposed method over others in different testing scenarios, respectively achieving 99.02%, 96.92%, 96.65% and 91.61% true-positive rates (TPR); and 66.16, 54.85, 55.98 and 52.61 LDIs in four different types of testing scenarios.

Current low-light light-field (LF) image enhancement algorithms tend to produce blurry results, for (1) loss of spatial details during enhancement and (2) inefficient exploitation of angular correlations, which helps to recover spatial details. Therefore, in this paper, we propose a parallel multi-scale network (PMSNet), which attempts to (1) process features of different scales in parallel to aggregate the different contributions of multi-scale features at each layer, thus fully preserve spatial details, and (2) integrate multi-resolution 3D convolution streams to efficiently utilize angular correlations. Specifically, PMSNet consists of three stages: Stage-I employs multi-scale modules (MSMs) to generate local understanding with the aid of adjacent views. Notably, MSM retains high-resolution feature extraction to minimize loss of spatial details. Stage-II processes all views to encode global information. Based on the above extracted local and global information, Stage-III utilizes 3D multi-scale modules (3D-MSMs) to efficiently exploit angular correlations. To validate our idea, we comprehensively evaluate the performance of PMSNet on three publicly available datasets. Experimental results show that our method is superior to the current state-of-the-art methods, achieving an average PSNR of 24.76 dB.