Multispectral images are increasingly used for pedestrian detection. Preliminary fusion strategies would fail to exploit informative features from cross-spectral images, or worse, may introduce additional interference. In this paper, we propose an attention based multi-layer fusion network in the triple-stream deep convolutional neural network architecture for multispectral pedestrian detection. The effectiveness of multi-layer fusion is examined and verified in this work. Furthermore, a channel-wise attention module (CAM) and a spatial-wise attention module (SAM) are developed and incorporated into the network aimng at more subtle adjustment to weights of multispectral features along both the channel and spatial dimensions respectively. Channel-wise attention is trained with self-supervision while spatialwise attention is trained with external supervision as we remodel its learning process as saliency detection. Both attention-based weighting mechanisms are evaluated separately and then sequentially. Experimental results on the KAIST dataset show that the proposed multi-layer cross-spectral fusion R-CNN (CS-RCNN), with spatial-wise weighting applied alone, achieves state-of-the-art performance on all-day detection while outperforming compared methods at nighttime.

This paper presents a two-layer controller for accurate and robust lateral path tracking control of highly automated vehicles. The upper-layer controller, which produces the front wheel steering angle, is implemented with a Linear Time-Varying MPC (LTV-MPC) whose prediction and control horizon are both optimized offline with particle swarm optimization (PSO) under varying working conditions. A constraint on the slip angle is imposed to prevent lateral forces from saturation to guarantee vehicle stability. The lower layer is a radial basis function neural network proportion-integral-derivative (RBFNN-PID) controller that generates electric current control signals executable by the steering motor to rapidly track the target steering angle. The nonlinear characteristics of the steering system are modeled and are identified on-line with the RBFNN so that the PID controller’s control parameters can be adjusted adaptively. The results of CarSim-Matlab/Simulink joint simulations show that the proposed hierarchical controller achieves a good level of path tracking accuracy while maintaining vehicle stability throughout the path tracking process, and is robust to dynamic changes in vehicle velocities and road adhesion coefficients.

Effective and friendly human-machine cooperative driving is considered as a feasible solution to filling the gap between assisted driving and highly automated driving. This paper proposes an indirect shared control strategy, in both longitudinal and lateral directions, for safe cooperative driving on curvy roads, which often pose hazards in real-world driving. At the tactical decision level of the proposed strategy, driving authorities are dynamically allocated between human and machine based on the risk assessed with a data-driven gaussian processes regression (GPR) model, which is trained to generate dynamic safety envelopes of longitudinal and lateral control commands, namely vehicle speed and front wheel steering angle, from observed road conditions and vehicle states. At the maneuver execution level, a multi-objective hierarchical MPC controller is built to combine human-machine control commands smoothly for collision-free driving while simultaneously maintaining vehicle stability and mitigating human-machine conflicts. Both matching the driver's commands and tracking the reference trajectories are formulated into objectives of the optimization-based controller. The weight factor of each objective is adjusted according to the authorities allocated at the decision level. The results of experiments conducted on a 3 degree of freedom motion-base driving simulator show that the proposed strategy accomplishes effective and friendly cooperative driving on curvy roads. It's also verified that the proposed strategy is adaptable to unseen scenarios, mainly owing to the superiority of the GPR-based risk assessment model, driven by class-balanced training data, in modeling the nonlinearity and uncertainty of driving risks.

A core issue of safe human-machine cooperative driving lies in dynamic assessment of the rapidly evolving driving risks. Given that, this paper proposes a shared controller for safe and human-friendly cooperative driving based on predictive risk assessment enabled by Digital Twin technologies. In the digital world, we create a fine-grained digital replica of the driving scene comprising historical motions of vehicles, as well as roadway geometries and topologies. On the basis of that, spatial-temporal interactive features are obtained with a deep learning-based model and subsequently decoded to predict future trajectories of each target vehicle in the neighborhood of the ego-vehicle. In the physical world, the predicted trajectories of neighboring vehicles are integrated into the risk distribution to construct predictive risk fields. A novel shared controller in the framework of multi-objective MPC is designed to minimize the driving risk while matching driver's commands, so that safe cooperative driving is achieved in a smooth and minimal-intervention manner. The results of driver-in-the-loop simulation experiments demonstrate the enabling role of the Digital Twin in improving the assessment of risk in highly dynamic scenes through taking the motion trends of dynamic agents into account. The results also show the superiority of the Digital Twin-based shared controller in terms of implementing cooperation in time while honoring the driver's commands whenever possible.

Accurate trajectory prediction for neighboring agents is crucial for autonomous vehicles navigating complex scenes. Recent deep learning (DL) methods excel in encoding complex interactions but often generate invalid predictions due to difficulties in modeling transient and contingency interactions. This paper proposes a hierarchical hybrid framework that combines DL and reinforcement learning (RL) for multi-agent trajectory prediction, capturing multi-scale interactions that shape future motion. In the DL stage, Transformer-style graph neural network (GNN) is employed to encode heterogeneous interactions at intermediate and global scales, predicting multi-modal intentions as key future positions for agents. In the RL stage, we divide the scene into local scenes based on DL predictions. A Transformer-based Proximal Policy Optimization (PPO) model, incorporated with vehicle kinematics, generates future trajectories in the form of motion planning shaped by microscopic interactions and guided by a multi-objective reward for balanced agent-centric accuracy and scene-wise compatibility. Experimental results on the Argoverse benchmark and driver-in-loop simulations demonstrate that our framework enhances trajectory prediction feasibility and plausibility in interactive scenes.