17 October 2023 in IEEE Transactions on Intelligent Vehicles
One of the most challenging problems in human-machine co-work is the gap between human intention and the machine's understanding and execution. Large Language Models (LLMs) have been showing superior abilities in solving such issue. In this paper, we design a universal framework that embeds LLMs as a vehicle “Co-Pilot” of driving, which can accomplish specific driving tasks with human intention satisfied based on the information provided. Meanwhile, a utilization workflow is defined to handle the interaction between humans and vehicles, and memory mechanism is introduced to organize the information involved in the tasks. Expert Oriented Black-Box tuning is proposed to improve the performance of the Co-Pilot without finetuning or training the LLMs. In the experiment, the Co-Pilot is applied to two different tasks, i.e., path tracking control and trajectory planning. The Co-Pilot adjusts vehicle operating conditions by selecting a proper controller or planning a certain trajectory to fit human intentions. Simulation tests are conducted to evaluate the performance and generality of the proposed module. The results show that the Co-Pilot can accomplish most of the tasks based on only natural language processing, although it is not flawless. Finally, a discussion about human-machine hybrid intelligence and further applications of LLMs in autonomous driving is made. We believe that such a framework has promising potential in further applications in the field of automous vehicles.
9 April 2024 in IEEE Journal on Emerging and Selected Topics in Circuits and Systems
Realizing high-fidelity three-dimensional (3D) scene representation through holography presents a formidable challenge, primarily due to the unknown mechanism of the optimal hologram and huge computational load as well as memory usage. Herein, we propose a Physically Guided Generative Adversarial Network (PGGAN), which is the first generative model to transform the multi-view light field directly to holographic 3D content. PGGAN harmoniously fuses the fidelity of data-driven learning with the rigor of physical optics principles, ensuring a stable reconstruction quality across wide field of view, which is unreachable by current central-view-centric approaches. The proposed framework presents an innovative encoder-generator-discriminator, which is informed by a physical optics model. It benefits from the speed and adaptability of data-driven methods to facilitate rapid learning and effectively transfer to novel scenes, while its physics-based guidance ensures that the generated holograms adhere to holographic standards. A unique, differentiable physical model facilitates end-to-end training, which aligns the generative process with the “holographic space”, thereby improving the quality of the reconstructed light fields. Employing an adaptive loss strategy, PGGAN dynamically adjusts the influence of physical guidance in the initial training stages, later optimizing for reconstruction accuracy. Empirical evaluations reveal PGGAN’s exceptional ability to swiftly generate a detailed hologram in as little as 0.002 seconds, significantly eclipsing current state-of-the-art techniques in speed while maintaining superior angular reconstruction fidelity. These results demonstrate PGGAN’s effectiveness in producing high-quality holograms rapidly from multi-view datasets, advancing real-time holographic rendering significantly.