This paper presents a lightweight stereo visionbased driving lane detection and classification system to achieve the ego-car’s lateral positioning and forward collision warning to aid Advanced Driver Assistance Systems (ADAS). For lane detection, we design a self-adaptive traffic lanes model in Hough Space with a maximum likelihood angle and dynamic pole detection ROIs, which is robust to road bumpiness, lane structure changing while the ego-car’s driving and interferential markings on the ground. What’s more, this model can be improved with geographic information system (GIS) or electronic map to achieve more accurate results. Besides, the 3D information acquired by stereo matching is used to generate an obstacle mask to reduce irrelevant objects’ interfere and detect forward collision distance. For lane classification, a Convolutional Neural Network (CNN) is trained by using manually labelled Region Of Interest (ROI) from KITTI dataset to classify the left/right-side line of host lane so that we can provide significant information for lane changing strategy making in ADAS. Quantitative experimental evaluation shows good true positive rate on lane detection and classification with a real-time (15Hz) working speed. Experimental results also demonstrate a certain level of system robustness on variation of the environment.

This paper presents real-time obstacles detection and their status classification method for collision warning in the vehicle active safety system. Specifically, stereo cameras and millimeter wave (mmw)-radar are fused to help the driving ego-vehicle to find “Danger” or “Potential Danger” in a timely way through combining with the vehicle kinematic model. The proposed method makes full use of the unique advantages of stereo cameras and mmw-radar to sense the environment through several modules. Cameras are mainly used to detect the near or lateral dynamic objects and to obtain the obstacles region of interest (ROI) considering its rich information and high sensitivity to the lateral displacement, while far or longitudinal relative dynamic objects are detected by mmw-radar according to its observational ability to make up for the disadvantage of cameras. In detail, a cameras detector utilizes ”error vectors” rather than the optical flow to obtain dynamic classes through two times clustering. Mmw-radar mainly detects relative dynamic objects, whose absolute speed can be computed according to the ego-vehicle's state. Then, the detected objects of these two detectors are integrated in an obstacles ROI map, which is obtained through an UV-disparity obstacles detection algorithm to get the final dynamic and relative dynamic objects. Finally, they are classified by comparing them with a dangerous area that is acquired according to the vehicle kinematic model in a special vehicle coordinate system, which is fixed to the ground temporarily. This method is tested on our mobile platforms and the results prove that it can work effectively even though the ego-vehicle drives quickly.

Deep reinforcement learning (DRL) has been successfully applied to end-to-end autonomous driving, especially in simulation environments. However, common DRL approaches used in complex autonomous driving scenarios sometimes are unstable or difficult to converge. This paper proposes two approaches to improve the stability of the training policy model with as few manual data as possible. For the first approach, reinforcement learning is combined with imitation learning to train a feature network with a small amount of manual data for parameters initialization. For the second approach, an auxiliary network is added to the reinforcement learning framework, which can leverage the real-time measurement information to deepen the understanding of environment, without any guide of demonstrators. To verify the effectiveness of these two approaches, simulations in image information-based and lidar information-based end-to-end autonomous driving systems are conducted, respectively. These approaches are not only tested in the virtual game world, but also applied in Gazebo, in which we build a 3D world based on the real vehicle model of Ranger XP900 platform, the real 3D obstacle model, and the real motion constraints with inertial characteristics, so as to ensure that the trained end-to-end autonomous driving model is more suitable for the real world. Experimental results show that the performance is increased by over 45% in the virtual game world, and can converge quickly and stably in Gazebo in which previous methods can hardly converge.

Environment representations endowed with sophisticated semantics are pivotal for facilitating seamless interaction between robots and humans, enabling them to effectively carry out various tasks. Open-vocabulary representation, powered by Visual-Language models (VLMs), possesses inherent advantages, including zero-shot learning and open-set cognition. However, existing open-vocabulary maps are primarily designed for small-scale environments, such as desktops or rooms, and are typically geared towards limited-area tasks involving robotic indoor navigation or in-place manipulation. They face challenges in direct generalization to outdoor environments characterized by numerous objects and complex tasks, owing to limitations in both understanding level and map structure. In this work, we propose OpenGraph, a novel open-vocabulary hierarchical graph representation designed for large-scale outdoor environments. OpenGraph initially extracts instances and their captions from visual images, enhancing textual reasoning by encoding captions. Subsequently, it achieves 3D incremental object-centric mapping with feature embedding by projecting images onto LiDAR point clouds. Finally, the environment is segmented based on lane graph connectivity to construct a hierarchical representation. Validation results from SemanticKITTI and real-world scene demonstrate that OpenGraph achieves high segmentation and query accuracy.

Autonomous driving, including intelligent decision-making and path planning, in dynamic environments (like highway) is significantly more difficult than the navigation in static scenarios because of the additional time dimension. Therefore, correlating the time dimension and the space dimension through prediction to create a spatio-temporal navigation map can make decision-making and path planning in such kinds of environment much easier. In this article, NGSIM data is analysed and processed from the perspective of the ego-vehicle (using the data as an ego-vehicle’s perception results). Based on the data, we develop an LSTM (Long-Short Term Memory)-based framework to predict possible trajectories of multiple surrounding vehicles within a certain range of the ego-vehicle. Then, the multiple predicted trajectories in a series of continuous dynamic highway scenes are projected into a spatio-temporal domain to create an octree map. Thus, dynamic targets and static obstacles can be unified into the same domain or map so that the dynamic disturbance problem for autonomous driving in highway environments can be resolved. Experimental results show that the proposed model is capable of predicting all the future trajectories around the ego-vehicle efficiently and the corresponding spatio-temporal map can be generated accurately in different dynamic scenarios.