The multi-camera array has drawn attention of researchers in recent years, and has been configured and deployed on intelligent vehicle to capture the panoramic views. Understanding surroundings is crucial for the ego-vehicle. This paper presents a Multi-perspective Tracking (MPT) framework for intelligent vehicle. An iterative search procedure is proposed to associate detections and tracklets in different perspectives. This procedure iteratively assigns determined states and estimates non-determined states for the detections and tracklets. An inherent determined and non-determined graph is utilized to reinforce this procedure. For more reliable associations between perspectives, a Siamese convolutional neural network is employed to learn feature representation. The supervised classification and verification signals are added to train the network. The features in different conventional stages are integrated together as the discriminative appearance model. The experiments are conducted on a MPT data set with five perspectives. The proposed framework is tested in each pair of adjacent perspectives for the ability to associate target objects between perspectives.
In this paper, a deep learning method for video‐based action recognition is proposed. On the one hand, boundary compensation on the basis of a deep neural network is performed to achieve action proposal. Boundary compensation considering non‐maximum suppression according to sliding window priority is applied to remove redundant windows. To accurately detect boundaries, a boundary compensation network is established with multiple networks to process different numbers of segments. On the other hand, action recognition based on the resultant action proposals is performed. To further utilise boundary compensation, three methods are introduced for key frame selection. Optical flow and RGB features are combined via a channel fusion to realise feature representation. A two‐stream network with a spatiotemporal structure is adopted for action recognition. The proposed method is evaluated on three public datasets. The experimental results demonstrate that the proposed method achieves a superior performance to that of state‐of‐the‐art methods.
23 January 2025 in IEEE Transactions on Circuits and Systems for Video Technology
Monocular visual odometry (VO) is crucial for the application of various autonomous systems. However, the inherent scale ambiguity issue in monocular methods greatly limits their performance in pose estimation. In this paper, we propose a hybrid monocular VO system named KPDepth-VO, which solves camera pose from monocular video based on sparse keypoints. To estimate the scale-consistent relative pose, we present a novel photometric-sensitive depth uncertainty model that accounts for the depth uncertainty introduced by limitations in the photometric error constraint. We also introduce an uncertainty-aware scale recovery strategy that incorporates depth uncertainty for reliable scale alignment. Additionally, we propose a novel difference attention mechanism to construct a point filter that effectively filters out less distinctive points, ensuring high-quality matches for more accurate and efficient pose estimation in the proposed system. Experimental results on the KITTI dataset and Oxford Robotcar dataset demonstrate that our system can predict scale-consistent trajectories from monocular videos and achieve state-of-the-art performance among similar methods. Meanwhile, the depth network within our system achieves competitive depth estimation performance on KITTI depth benchmark.
Lane detection plays a critical role in the field of autonomous driving. Most previous lane detection methods focus exclusively on analyzing individual images and overlook the inter-frame dynamics, while car-mounted cameras capture continuous streams amenable to leveraging intra-video context. In challenging scenes with blur, occlusion, or illumination variations, considering visible lanes from previous frames can aid current frame interpretation. To tackle above challenges, we introduce a parallel heterogeneous framework, called PHNet, for video lane detection. First, a novel router automatically analyzes multi-level features to determine the inference route of candidates with different visual cues. Then, a cross-frame attention mechanism leverages relationships between current candidates and positive embeddings from past frame to aggregate contextual cues. Unlike existing offline video lane detection methods, which face limitations in handling long video sequences and real-time video streams due to computational constraints, our proposed method operates as an online framework capable of processing clips of any length. Our method demonstrates robust lane detection through temporally association modeling and efficient online inference. The extensive experiments on public benchmark show that PHNet has superior performance versus state-of-the-art video lane detection methods.