Human drivers instinctively reason with commonsense knowledge to predict hazards in unfamiliar scenarios and to understand the intentions of other road users. However, this essential capability is entirely missing from traditional decision-making systems in autonomous driving. In response, this paper presents DriveLLM, a decision-making framework that integrates large language models (LLMs) with existing autonomous driving stacks. This integration allows for commonsense reasoning in decision-making. DriveLLM also features a unique cyber-physical feedback system, allowing it to learn and improve from its mistakes. In real-world case studies, the proposed framework outperforms traditional decision-making methods in complex scenarios, including difficult edge cases. Furthermore, we propose a novel approach that allows the decision-making system to interact with human inputs while guarding against adversarial attacks. Empirical evaluations demonstrate that this framework responds correctly to complex human instructions. The code and test data is available: https://github.com/DriveLLM/DriveLLM

The human-vehicle collaboration in automated vehicles is an effective transitional means to overcome the difficulty of rapidly transitioning to a highly automated level of intelligence. Furthermore, it can fully leverage the strengths of both drivers and autonomous driving systems, embodying a design philosophy of human-centered. Therefore, this paper provides a review and perspectives of the human-vehicle collaboration for automated vehicles. First, the concept, forms and methods of human-vehicle collaboration are reviewed. Then, a human-vehicle mutual trust collaboration framework based on complementary advantages of humans and vehicles and brain-like intelligence is proposed. Specifically, the framework focuses on driver behavior understanding and brain-like cognitive decision planning. After that, the methods of driver behavior understanding and brain-like cognitive decision planning are summarized. Finally, challenges and future works are analyzed to contribute the develop of understandable, trustable, and acceptable human-vehicle collaboration systems.

Human drivers possess spatial and causal intelligence, enabling them to perceive driving scenarios, anticipate hazards, and react to dynamic environments. In contrast, autonomous vehicles lack these abilities, making it challenging to manage perception-related Safety of the Intended Functionality (SOTIF) risks, especially under complex or unpredictable driving conditions. To address this gap, we propose fine-tuning multimodal large language models (MLLMs) on a customized dataset specifically designed to capture perception-related SOTIF scenarios. Benchmarking results show that fine-tuned MLLMs achieve an 11.8% improvement in close-ended VQA accuracy and a 12.0% increase in open-ended VQA scores compared to baseline models, while maintaining real-time performance with a 0.59-second average inference time per image. We validate our approach through real-world case studies in Canada and China, where fine-tuned models correctly identify safety risks that challenge even experienced human drivers. This work represents the first application of domain-specific MLLM fine-tuning for the SOTIF domain in autonomous driving. The dataset and related resources are available at github.com/s95huang/DriveSOTIF.git