26 December 2022 in IEEE Transactions on Systems, Man, and Cybernetics: Systems
Foundation models are used to train a broad system of general data to build adaptations to new bottlenecks. Typically, they contain hundreds of billions of hyperparameters that have been trained with hundreds of gigabytes of data. However, this type of black-box vulnerability places foundation models at risk of data poisoning attacks that are designed to pass on misinformation or purposely introduce machine bias. Moreover, ordinary researchers have not been able to completely participate due to the rise in deployment standards. This study introduces the theoretical framework of scenarios engineering (SE) for building accessible and reliable foundation models in metaverse, namely, “SE-enabled foundation models in metaverse.” Particularly, the research framework comprises a six-layer architecture (infrastructure layer, operation layer, knowledge layer, intelligence layer, management layer, and interaction layer), which can provide controllability, trustworthiness, and interactivity for the foundation models in metaverse. This creates closed-loop, virtual–real, and human–machine environments that provides the best indices and goals for the foundation models, which allows us to fully validate and calibrate the corresponding models. Then, examples of use cases from the automotive industry are listed to provide transparency on the possible use and benefits of our approach. Finally, the open research topics of related frameworks are discussed.
23 August 2023 in IEEE Transactions on Intelligent Vehicles
Diverse transport demands have resulted in the wide existence of heterogeneous vehicle automation systems. While these systems have demonstrated effectiveness, they also pose challenges in terms of the share of technological advancements among different organizations and lead to poor generalization ability of individual systems. This paper proposes a Transformer-based unified framework, VistaGPT, to address these challenges. VistaGPT, composed of Modular Federations of Vehicular Transformers (M-FoV) and Automated Composing of Autonomous Driving Systems (AutoAuto), aims to overcome the information barriers due to system-level and module-level heterogeneity. M-FoV collects and organizes Transformer-based models in a modular fashion to facilitate system integration by providing diversity and versatility. AutoAuto utilizes large language models (LLMs) to automatically compose end-to-end autonomous driving systems with a “Dividing and Recombining” strategy. Besides, we deploy Scenario Engineering systems to evaluate the composed systems and provide systematic feedback for the optimization of AutoAuto, and Federated intelligence to contribute to diverse training samples and applications. With its capacity, scalability, and diversity, VistaGPT provides a new paradigm of LLM-aided system development for transport automation, which promotes virtual-real interactive parallel driving and advances progress toward “6S” objectives.
18 March 2024 in IEEE/CAA Journal of Automatica Sinica
During our discussion at workshops for writing “What Does ChatGPT Say: The DAO from Algorithmic Intelligence to Linguistic Intelligence” [1], we had expected the next milestone for Artificial Intelligence (AI) would be in the direction of Imaginative Intelligence (II), i.e., something similar to automatic words-to-videos generation or intelligent digital movies/theater technology that could be used for conducting new “Artificiofactual Experiments” [2] to replace conventional “Counterfactual Experiments” in scientific research and technical development for both natural and social studies [2]–[6]. Now we have OpenAI's Sora, so soon, but this is not the final, actually far away, and it is just the beginning.
24 April 2023 in IEEE Transactions on Intelligent Vehicles
As driving scenarios and autonomous vehicles (AVs) become increasingly intricating, there is an increasing need for innovative frameworks that can enhance and test AV capabilities across diverse scenarios. At present, the design and validation for AVs predominantly rely on simulation software or real-road testing methodologies. However, these approaches possess inherent limitations, leading to inaccuracy, reduced efficiency, and potential hazards during the development process. This letter reports our first DHW (decentralized and hybrid workshop) on Scenarios Engineering (SE), that aims to calibrate and validate AV modules through advanced scenarios. Specifically, the DHW discusses combining SE with existing simulation software and real-road testing strategies, enabling developers to maximize the advantages of each approach while ensuring the safety and efficacy in the development of autonomous vehicles. Our findings demonstrate the benefits of enhancing realism of simulation scenarios in the aspects of both content and appearance. Furthermore, the use of scenarios engineering is explored in order to enhance the diversity and safety of real-road testing by means of integrating virtual and physical components.
7 February 2024 in IEEE Transactions on Intelligent Vehicles
The latest developments in parallel driving foreshadow the possibility of delivering intelligence across organizations using foundation models. As is well-known, there are limitations in scenario acquisition in the field of intelligent vehicles (IV), such as efficiency, diversity, and complexity, which hinder in-depth research of vehicle intelligence. To address this issue, this manuscript draws inspiration from scenarios engineering, parallel driving and introduces a pioneering framework for scenario generation, leveraging the ChatGPT, denoted as SeGPT. Within this framework, we define a trajectory scenario and design prompts engineering to generate complex and challenging scenarios. Furthermore, SeGPT, in combination with “Three Modes”, foundation models, vehicle operating system, and other advanced infrastructure, holds the potential to achieve higher levels of autonomous driving. Experimental outcomes substantiate SeGPT's adeptness in producing a spectrum of varied scenarios, underscoring its potential to augment the development of trajectory prediction algorithms. These findings mark significant progress in the domain of scenario generation, also pointing towards new directions in the research of vehicle intelligence and scenarios engineering.
20 August 2019 in IEEE Transactions on Vehicular Technology
Virtual simulation testing is becoming indispensable for the intelligence testing of intelligent vehicles. However, even the most advanced simulation software provides rather limited test conditions. In the long run, intelligent vehicles are expected to work at SAE (Society of Automotive Engineers) level 4 or level 5. Researchers should make full use of virtual simulation scenarios to test the visual intelligence algorithms of intelligent vehicles under various imaging conditions. In this paper, we create realistic artificial scenes to simulate the self-driving scenarios, and collect a dataset of synthetic images from the virtual driving scenes, named “ParallelEye-CS”. In the artificial scenes, we can flexibly change environmental conditions and automatically acquire accurate and diverse ground-truth labels. As a result, ParallelEye-CS has six ground-truth labels and includes twenty types of tests, which are divided into normal, environmental, and difficult tasks. Furthermore, we utilize ParallelEye-CS in combination with other publicly available datasets to conduct experiments for visual object detection. The experimental results indicate that: 1) object detection algorithms of intelligent vehicles can be tested under various scenario challenges; 2) mixed dataset can improve the accuracy of object detection algorithms, but domain shift is a serious issue worthy of attention.