Maritime transport plays a critical role in global logistics. Compared to road transport, the pace of research and development is much slower for maritime transport. It faces many major challenges, such as busy ports, long journeys, significant accidents, and greenhouse gas emissions. The problems have been exacerbated by recent regional conflicts and increasing international shipping demands. Maritime Autonomous Surface Ships (MASSs) are widely regarded as a promising solution to addressing maritime transport problems with improved safety and efficiency. With advanced sensing and path-planning technologies, MASSs can autonomously understand environments and navigate without human intervention. However, the complex traffic and water conditions and the corner cases are large barriers in the way of MASSs being practically deployed. In this paper, to address the above issues, we investigated the application of Large Language Models (LLMs), which have demonstrated strong generalization abilities. Given the substantial computational demands of LLMs, we propose a framework for LLM-assisted navigation in connected MASSs. In this framework, LLMs are deployed onshore or in remote clouds, to facilitate navigation and provide guidance services for MASSs. Additionally, certain large oceangoing vessels can deploy LLMs locally, to obtain real-time navigation recommendations. To the best of our knowledge, this is the first attempt to apply LLMs to assist with ship navigation. Specifically, MASSs transmit assistance requests to LLMs, which then process these requests and return assistance guidance. A crucial aspect, which has not been investigated in the literature, of this safety-critical LLM-assisted guidance system is the knowledge and safety performance of the LLMs, in regard to ship handling, navigation rules, and skills. To assess LLMs’ knowledge of navigation rules and their qualifications for navigation assistance systems, we designed and conducted navigation theory tests for LLMs, which consisted of more than 1500 multiple-choice questions. These questions were similar to the official theory exams that are used to award the Officer Of the Watch (OOW) certificate based on the Standards of Training, Certification, and Watchkeeping (STCW) for Seafarers. A wide range of LLMs were tested, which included commercial ones from OpenAI and Baidu and an open-source one called ChatGLM, from Tsinghua. Our experimental results indicated that among all the tested LLMs, only GPT-4o passed the tests, with an accuracy of 86%. This suggests that, while the current LLMs possess significant potential in regard to navigation and guidance systems for connected MASSs, further improvements are needed.

Security check-in life-support areas, e.g., bridge, engine room, is crucial for cruise ships due to numerous and diverse passenger identities. Instead of conventional security check approaches, such as facial recognition and fingerprint identification, device-free approaches enabled by WiFi-based gait recognition have attracted considerable attention owing to their low cost, non-intrusiveness, and privacy protection. Despite the excellent performance of existing indoor methods, they cannot be trivially extended to cruise ships because of the unique characteristics of hull deformation caused by vibrating engines and waves. This stems from the flexible structure of cruise ships, which introduces additional noise to the WiFi signals. To address this challenge, we propose WiCrew, a device-free gait recognition system that detects crew identity anomalies in cruise ships. WiCrew consists of two components: 1) a spatial separation algorithm that separates the signal components from ship vibration and human activity; 2) a speed-independent adversarial learning framework that identifies the ship’s crew using human gaits at an arbitrary walking speed. Extensive experiments on a cruise ship demonstrate the effectiveness of WiCrew. While the crew members walk at speed of 0.7 to 1.8 m/s, the average recognition accuracy reaches 82%, which is similar to vision-based approaches.

To ensure the safety of marine traffic, ship watch-keeping officers must maintain a high level of vigilance during their watchkeeping period. Although advanced driving assistance systems are available that can effectively measure driver alertness and provide early warnings before any hazardous maneuvers, such systems are primarily designed for road vehicles and are considerably different from those used when operating a marine vehicle. In this paper, we propose to use fine-grained channel state information (CSI) obtained using commercial off-the-shelf wireless fidelity (WiFi) to track an officer-on-watch’s vigilant activities and determine, in real time, whether the ship officer is adhering to safety guidelines.We developed a CSI path model and a 2D multiple signal classification (MUSIC) algorithm to estimate signal propagation path parameters, such as angle of arrival and Doppler shift, in a low signal-to-noise ratio (SNR) ship environment. We then developed CSI velocity and activity models based on finer-grained signal parameters, using wavelet transforms and deep learning techniques to determine bridge-officer activities. Finally, we developed a watchkeeping vigilance evaluation module to determine whether a bridge officer was complying with safe driving guidelines during their watchkeeping. Our proposed system was implemented on a commercial WiFi platform, and we extensively evaluated the system on an actual passenger ship. Our proposed system achieved accuracies of 95.4% and 93.8% for tracking walking movements and recognizing careless activities, respectively.

Large Language Models (LLMs), with their strong generalization and inference capabilities, have been increasingly leveraged to address the challenges of handling corner cases in autonomous driving (AD). However, a critical unresolved issue remains: the lack of a comprehensive understanding and formal assessment of LLMs’ driving theory knowledge and practical skills. To address this issue, we propose the first dedicated driving theory test framework and benchmark for LLMs. That is a crucial yet unexplored area in the literature, particularly for safety-critical applications in autonomous driving and driver assistance. Our framework systematically evaluates LLMs’ competence in driving theory and hazard perception, akin to the official UK driving theory test, ensuring their qualification for critical driving-related tasks. To facilitate rigorous benchmarking, we construct a comprehensive dataset comprising over 700 multiple-choice questions (MCQs) and 54 hazard perception video tests sourced from the official UK driving theory examination. Additionally, we incorporate two standardized MCQ sets from the UK’s Driver and Vehicle Standards Agency (DVSA). For these two types of theoretical test items, we design tailored assessment methodologies and evaluation metrics, including accuracy, recall, precision, F1-score, real-time performance, and computational efficiency. The experimental results reveal that among all LLMs tested, only GPT-4o achieved an accuracy of 88. 21% in the MCQs test, successfully passing this component. However, in hazard perception testing, none of the evaluated models met the passing criteria under the given settings, highlighting the substantial improvements required before these models can be practically deployed for real-world driving applications. Our key insight is that the specific test questions LLMs fail to answer correctly directly reflect their deficiencies in understanding and flexibly applying traffic regulations, as well as in analyzing and responding to complex driving scenarios. This provides clear directions for future improvements.

Handling long tail corner cases is a major challenge faced by autonomous vehicles (AVs). While large language models (LLMs) hold great potentials to handle the corner cases with excellent generalization and explanation capabilities and received increasing research interest on application to autonomous driving, there are still technical barriers to be tackled, such as strict model performance and huge computing resource requirements of LLMs, which are difficult to be met locally at AVs. In this paper, we investigate a new approach of applying remote or edge LLMs to support autonomous driving. With this approach connected autonomous vehicles (CAVs) send driving assistance requests to the LLMs. LLMs deployed at the edge of the networks or remote clouds process the requests and generate driving assistance instructions for the CAVs. A key issue for such LLM assisted driving system is the assessment of LLMs on their understanding of driving theory and skills, ensuring they are qualified to undertake safety critical driving assistance tasks for CAVs. As there is no published work on assessing LLM of driving theory and skills, we design and run driving theory tests for several proprietary LLM models (OpenAI GPT models, Baidu Ernie and Ali QWen) and open-source LLM models (Tsinghua MiniCPM-2B and MiniCPM-Llama3-V2.5) with more than 500 multiple-choices theory test questions. These questions are close to the official UK driving theory test ones. Model accuracy, cost and processing latency are measured from the experiments. Experiment results show that while model GPT-4 passes the test with improved domain knowledge and Ernie has an accuracy of 85% (just below the 86% passing threshold), other LLM models including GPT-3.5 fail the test. For the test questions with images, the multimodal model GPT4-o has an excellent accuracy result of 96%, and the MiniCPM-Llama3-V2.5 achieves an accuracy of 76%. While GPT-4 holds stronger potential for CAV driving assistance applications, the cost of using model GPT4 is much higher, almost 50 times of that of using GPT3.5. The results can help make decision on the use of the existing LLMs for CAV applications and balancing on the model performance and cost.

The safeguarding of critical zones aboard a marine vehicle, such as the engine room, wheelhouse, and pump room, assumes crucial significance while navigating through the open sea. Despite the existing pre-boarding security measures, Concealed Threat Detection (CTD) systems have emerged as a pressing need to prevent the ship from post-boarding damage with concealed dangers. Due to concerns regarding deployment cost and privacy, mmWave-based CTD systems have received significant attention. However, current solutions are not easily adapted to work in ships because of the large number of ghost targets resulting from multipath reflections in full metal cabins. To address these challenges, this paper proposes a new CTD system, called mmCTD, which utilizes two mmWave commercial radars. The proposed system addresses the multipath challenge by unifying multi-view perceptions with two distinct designs. First, we propose a ghost-point elimination algorithm that extracts the point clouds from real objects. Then, we design a multi-view domain adversarial framework to predict concealed threats in the human body using the extracted RF features. mmCTD is validated by both simulations and real ship experiments, and results demonstrate that the recognition accuracy in three scenarios reaches 89% with a low false alarm rate.