Railway track detection, which is crucial for train operational safety, faces numerous challenges such as the curved track, obstacle occlusion, and vibrations during the train’s operation. Most existing methods for railway track detection use a camera or LiDAR. However, the vision-based approach lacks essential 3D environmental information about the train, while the LiDAR-based approach tends to detect tracks of insufficient length due to the inherent limitations of LiDAR. In this study, we propose a real-time method for railway track detection and 3D fitting based on camera and LiDAR fusion sensing. Semantic segmentation of the railway track in the image is performed, followed by inverse projection to obtain 3D information of the distant railway track. Then, 3D fitting is applied to the inverse projection of the railway track for track vectorization and LiDAR railway track point segmentation. The extrinsic parameters necessary for inverse projection are continuously optimized to ensure robustness against variations in extrinsic parameters during the train’s operation. Experimental results show that the proposed method achieves desirable accuracy for railway track detection and 3D fitting with acceptable computational efficiency, and outperforms existing approaches based on LiDAR, camera, and camera–LiDAR fusion. To the best of our knowledge, our approach represents the first successful attempt to fuse camera and LiDAR data for real-time railway track detection and 3D fitting.

Most existing 3D pose representations cannot completely decouple the overlapping two or more human joints of the same type. In this paper, the authors propose a novel 2.5 D representation of the human pose by projecting human joints in 3D space onto the three orthogonal planes. The authors apply for the first time the permutation module to a multi‐person 3D human pose estimation task and use Geometric Constraints Loss (GCL) to guide the learning of the model. The authors overcome the negative effects of the inductive bias of convolutional neural networks (CNNs) by aligning the intermediate feature space with the output feature space. The effectiveness of the authors’ approach is validated on the carnegie mellon university (CMU) panoptic dataset and MuPoTS‐3D dataset. The authors’ proposed representations can effectively decouple the human joints in their selected data from overlapping human joints.

Semantic scene completion is a crucial end‐to‐end 3D perception task, and the 3D information perception subjects is vital for autonomous driving. This paper presents CASSC, a novel adaptive context‐aware method based on Transformer networks, aimed at realizing camera‐based semantic scene completion algorithms. The key idea is to leverage rich context information from images to obtain pixel‐level label proposals, followed by designing a multiscale fusion mechanism to merge this information and match it with voxel space. A weakly supervised training strategy is proposed to obtain semantic label distribution features from images and introduce an adaptive multiscale fusion module to fuse and adaptively match these features with voxel space. Here, CASSC achieves state‐of‐the‐art performance on the SemanticKITTI dataset and demonstrates excellent performance on the SSC‐Bench dataset. Ablation experiments validate the rationality and effectiveness of our design, and the model and code of CASSC will be open‐sourced on https://github.com/dogooooo/CASSC .

Large language models have brought revolutionary changes to autonomous driving algorithms, ushering them into the era of multimodality. However, existing vehicle trajectory planning methods primarily focus on obstacle avoidance in autonomous driving scenarios, overlooking interactions with entities within the scene, such as humans. In this letter, we propose a new research direction: vehicle trajectory planning that takes into account human actions. We establish ATHENA, the first autonomous driving dataset that integrates multimodal human actions, comprising 33,855 scenarios. Each scenario contains status information about ego vehicle, as well as pedestrian actions that actively or passively interact with the vehicle, such as signaling the vehicle to proceed by waving and the unexpected falls by pedestrians that force the vehicle to stop. Based on each type of interaction, ATHENA also provides the corresponding driving suggestions and the reasons behind them. Moreover, we present an LLM-based baseline that consists of two agents: the Action Understanding Agent and the Vehicle Control Agent. Our baseline implements the generation of driving recommendations and vehicle control functions, which are guided by pedestrian actions. Experiments demonstrate the effectiveness and strong performance of our method. Our dataset and code will be publicly available at https://github.com/dogooooo/ATHENA.

The large-scale generation of real-world scenario datasets is a pivotal task in the field of autonomous driving. Existing methods emphasize solely on single-frame rendering, which need complex inputs for continuous scenario rendering. In this letter, ARES: a text-driven automatic realistic simulator is proposed, which can generate extensive realistic datasets with just a single text input. Its core idea is to generate vehicle trajectories based on the textual description, and then render the scenario by vehicle attributes associated with these trajectories. For learning trajectories generating, supervisory signal temporal logic is proposed to assist conditional diffusion model, which incorporates prior physical information. We annotate textual descriptions for KITTI-MOT dataset and establish an objective quantitative evaluation system. The superiority of our method is demonstrated by its high performance, which is reflected in a matching score of 3.54 and an FID of 8.93 in the trajectory reconstruction task, along with a speed accuracy of 0.99 and a direction accuracy of 0.93 in the trajectory editing task. The scenarios rendered by the proposed method exhibit high quality and realism, which indicates its great potential in testing of autonomous driving algorithms with vehicle-in-the-loop simulations. Our work will be open-sourced at https://github.com/dogooooo/ARES .