23 September 2021 in IEEE Transactions on Instrumentation and Measurement
Semantic segmentation is of great importance and a challenge in computer vision. One of its main problems is how to efficiently obtain rich information (geometric structure) and identify useful features from higher dimensions. A light field camera, due to its special microlens array structure, can completely record the angular-spatial information of scenes, which is attractive and has great potential to improve the performance of semantic segmentation tasks. Inspired by this, we propose an end-to-end semantic segmentation network that can process light field macropixel images robustly and extract their features. In addition, this network can flexibly and efficiently load the different popular deep learning backbones. Furthermore, we propose an efficient angular model, which, to learn the angular features between the different viewpoints of the macropixel image, improves the nonlinearity of angular-spatial features and enhances multichannel semantic correlations. To evaluate the network, we construct a new real scene light field dataset comprising 800 high-quality samples. The quantitative and qualitative results show that the highest mean intersection over union (mIoU) based on our algorithm is greater than 57%. Our algorithm achieves a 10.30% increase compared with state-of-the-art semantic segmentation algorithms. In combination with different backbones or multiscale light field macropixel images, the network can also achieve comparable results. This preliminary work demonstrates that the combination of light field imaging and deep learning technology has potential applications in the future study of semantic segmentation.
16 March 2022 in IEEE Transactions on Industrial Informatics
Identifying moving target from nearby similar distractors is a crucial challenge in visual object tracking due to the lack of angular information. In this work, we propose a visual object tracking method based on 4D light field imaging which captures abundant spatial and angular information in a single snapshot. Our method first presents a novel image representation of epipolar plane images (EPIs) attained from light field multi-view images, i.e., the macro-epipolar plane image (macro-EPI), to highlight both spatial and angular information of the objects. On this basis, we design a feature extractor for macro-EPI to learn the visual angle information. Thereafter, we design an object tracking framework based on a two-pattern convolution filters method which merges the confidence scores of macro-EPI and center view image. Experimental results demonstrate that the proposed method successfully discriminates the target from the similar distractors and achieves better performance than state-of-the-art visual object tracking methods.
10 April 2025 in IEEE Transactions on Circuits and Systems for Video Technology
Event data can asynchronously capture variations in light intensity, thereby implicitly providing valuable complementary cues for RGB-Event tracking. Existing methods typically employ a direct interaction mechanism to fuse RGB and event data. However, due to differences in imaging mechanisms, the representational disparity between these two data types is not fixed, which can lead to tracking failures in certain challenging scenarios. To address this issue, we propose a novel prior knowledge-driven hybrid prompter learning framework for RGB-Event tracking. Specifically, we develop a frame-event hybrid prompter that leverages prior tracking knowledge from the foundation model as intermediate modal support to mitigate the heterogeneity between RGB and event data. By leveraging its rich prior tracking knowledge, the intermediate modal reduces the gap between the dense RGB and sparse event data interactions, effectively guiding complementary learning between modalities. Meanwhile, to mitigate the internal learning disparities between the lightweight hybrid prompter and the deep transformer model, we introduce a pseudo-prompt learning strategy that lies between full fine-tuning and partial fine-tuning. This strategy adopts a divide-and-conquer approach to assign different learning rates to modules with distinct functions, effectively reducing the dominant influence of RGB information in complex scenarios. Extensive experiments conducted on two public RGB-Event tracking datasets show that the proposed HPL outperforms state-of-the-art tracking methods, achieving exceptional performance.
16 August 2022 in IEEE Transactions on Intelligent Transportation Systems
Multi-object tracking is of great importance in autonomous driving. However, with the outbreak of COVID-19, multi-object tracking faces new challenges in areas gripped by epidemics because of complex motion blur, frequent occlusions, and appearance deformations. To reliably improve object trajectory association in epidemic-plagued areas, we propose a temporal-spatial aggregation embedding network (TSAEN) for multi-object tracking. Our embedding network contains a temporal-aware correlation module (TACM) and spatial-aggregate embedding module (SAEM) that can fully obtain and aggregate appearance clues related to moving objects in previous frames. The TACM learns the temporal homogeneity features of the current and previous frames to perceive features with correlated appearance cues. Then, the SAEM adjusts the spatial deformation for each perceived temporal homogeneity feature and aggregates them for re-ID embedding learning. The experimental results demonstrate that our proposed method is able to achieve excellent overall performance.
21 November 2025 in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
Accurate ship detection is essential for maritime safety. However, existing convolutional neural network-based and transformer-based methods struggle to balance computational efficiency and detection accuracy, particularly in capturing global contextual relationships and fine-grained features of small ships. To address these challenges, we propose GLMFNet, a novel lightweight global-local multiscale fusion network for small ship detection. GLMFNet propose a lightweight backbone, global-local hybrid visual state-space (GL-HVSS), which employs the neighbor 2D-selective-scan (NS2D) to dynamically capture spatial-semantic correlations. This mechanism preserves ship continuity in complex maritime scenes while efficiently modeling global information. Furthermore, to amplify the saliency of tiny ships, we design the micro-similarity aware activation module block, which precisely accentuates the edge and texture details of vessels without incurring any additional parameters or computational overhead. Finally, the multiscale auxiliary hierarchical fusion block embeds intrinsic multiscale feature fusion, markedly enhancing localization accuracy while simultaneously optimizing parameter efficiency. Experimental evaluations on the Kaggle and FAIR1M marine ship detection datasets demonstrate that GLMFNet outperforms state-of-the-art models in detection accuracy while maintaining a lightweight structure.
7 March 2026 in Digital Signal Processing