Highly integrated information sharing among people, vehicles, roads, and cloud systems, along with the rapid development of autonomous driving technologies, has spurred the evolution of automobiles from simple “transportation tools” to interconnected “intelligent systems”. The intelligent cockpit is a comprehensive application space for various new technologies in intelligent vehicles, encompassing the domains of driving control, riding comfort, and infotainment. It provides drivers and passengers with safety, comfort, and pleasant driving experiences, serving as the gateway for traditional automobile manufacturing to upgrade towards an intelligent automotive industry ecosystem. This is the optimal convergence point for the intelligence, connectivity, electrification, and sharing of automobiles. Currently, the form, functions, and interaction methods of the intelligent cockpit are gradually changing, transitioning from the traditional “human adapts to the vehicle” viewpoint to the “vehicle adapts to human”, and evolving towards a future of natural interactive services where “humans and vehicles mutually adapt”. This article reviews the definitions, intelligence levels, functional domains, and technical frameworks of intelligent automotive cockpits. Additionally, combining the core mechanisms of human–machine interactions in intelligent cockpits, this article proposes an intelligent-cockpit human–machine interaction process and summarizes the current state of key technologies in intelligent-cockpit human–machine interactions. Lastly, this article analyzes the current challenges faced in the field of intelligent cockpits and forecasts future trends in intelligent cockpit technologies.

End-to-end autonomous driving, which directly maps raw sensor inputs to low-level vehicle controls, is an crucial part of Embodied AI. Despite successes in applying Multimodal Large Language Models (MLLMs) for high-level traffic scene semantic understanding, it remains challenging to effectively translate these conceptual semantics understandings into low-level motion control commands and achieve cross-scene driving generalization and consensus. We propose Sce2DriveX, a human-like chain-of-thought (CoT) driving reasoning MLLM framework, designed to achieve progressive learning from multi-view scene understanding to behavior analysis, motion planning, and vehicle control driving process. Sce2DriveX utilizes multimodal joint learning of local scene videos and global Bird’s Eye View (BEV) maps to deeply understand long-range spatiotemporal relationships and road topology, enhancing its 3D dynamic/static scene perception and reasoning capabilities and achieving cross-scene generalization. Meanwhile, it reconstructs the implicit cognitive chain inherent in human driving, further enhancing the consensus between autonomous driving and human thought. To improve model performance, we construct the first comprehensive Visual Question Answering (VQA) driving instruction dataset, which tailored for 3D spatial understanding and long-axis task reasoning, and introduce a task-oriented three-stage training pipeline to support supervised fine-tuning. Extensive experiments demonstrate that Sce2DriveX achieves state-of-the-art performance across tasks from scene understanding to end-to-end driving, as well as robust generalization in handling diverse driving scenes on the CARLA Bench2Drive benchmark.

Human-level driving is the ultimate goal of autonomous driving. As the top-level decision-making aspect of autonomous driving, behavior decision establishes short-term driving behavior strategies by evaluating road structures, adhering to traffic rules, and analyzing the intentions of other traffic participants. Existing behavior decisions are primarily implemented based on rule-based methods, exhibiting insufficient generalization capabilities when faced with new and unseen driving scenarios. In this paper, we propose a novel behavior decision method that leverages the inherent generalization and commonsense reasoning abilities of visual language models (VLMs) to learn and simulate the behavior decision process in human driving. We constructed a novel instruction-following dataset containing a large number of image–text instructions paired with corresponding driving behavior labels, to support the learning of the Drive Large Language and Vision Assistant (DriveLLaVA) and enhance the transparency and interpretability of the entire decision process. DriveLLaVA is fine-tuned on this dataset using the Low-Rank Adaptation (LoRA) approach, which efficiently optimizes the model parameter count and significantly reduces training costs. We conducted extensive experiments on a large-scale instruction-following dataset, and compared with state-of-the-art methods, DriveLLaVA demonstrated excellent behavior decision performance. DriveLLaVA is capable of handling various complex driving scenarios, showing strong robustness and generalization abilities.

The cockpit is evolving from passive, reactive interaction toward proactive, cognitive interaction, making precise predictions of driver intent a key factor in enhancing proactive interaction experiences. This paper introduces Cockpit-Llama, a novel language model specifically designed for predicting driver behavior intent. Cockpit-Llama predicts driver intent based on the relationship between current driver actions, historical interactions, and the states of the driver and cockpit environment, thereby supporting further proactive interaction decisions. To improve the accuracy and rationality of Cockpit-Llama’s predictions, we construct a new multi-attribute cockpit dataset that includes extensive historical interactions and multi-attribute states, such as driver emotional states, driving activity scenarios, vehicle motion states, body states and external environment, to support the fine-tuning of Cockpit-Llama. During fine-tuning, we adopt the Low-Rank Adaptation (LoRA) method to efficiently optimize the parameters of the Llama3-8b-Instruct model, significantly reducing training costs. Extensive experiments on the multi-attribute cockpit dataset demonstrate that Cockpit-Llama’s prediction performance surpasses other advanced methods, achieving BLEU-4, ROUGE-1, ROUGE-2, and ROUGE-L scores of 71.32, 80.01, 76.89, and 81.42, respectively, with relative improvements of 92.34%, 183.61%, 95.54%, and 201.27% compared to ChatGPT-4. This significantly enhances the reasoning and interpretative capabilities of intelligent cockpits.

Learning and simulating the decision processes of real-world human drivers is a key research direction in autonomous driving (AD). As the core of AD, existing decision systems typically face challenges in cross-scene generalization and decision interpretability: it requires understanding diverse dynamic driving scenarios and formulate transparent strategies that earn broad user trust. We propose VLM-Driver, a Vision-Language Model (VLM) framework with human-like chained driving decision thought, designed to progressively achieve global scene observation, high-level behavior planning, and low-level motion planning based on full-view driving videos, multi-round question queries and optional environmental perception information. Specifically, the full-view videos are resized to unified base resolution using the AnyRes strategy, and video features are extracted by a vision encoder. These video features are then mapped to the text embedding space via a vision-language projector and jointly fed into large language model (LLM) backbone with tokenized text embeddings. During this process, we introduce a Bilinear Interpolation method to efficiently compress the number of video features, ensuring an optimal balance between model performance and computational cost. Meanwhile, a dedicated motion head is designed to output refined future waypoints, improving the model's motion planning efficiency. Additionally, we construct a novel multimodal driving instruction dataset to support VLM-Driver training and introduce a decision-oriented training strategy to further enhance its chained reasoning capability. Extensive experiments show that VLM-Driver excels in scene observation and behavior planning, demonstrating high human-like consistency, and significantly outperforms existing baseline methods in motion planning, achieving SOTA performance. VLM-Driver also enable to handle unseen complex driving scenarios, exhibiting robust cross-scene generalization.