In recent years, unmanned surface vehicles (USVs) have played an increasingly important role in various applications. Due to the expansion of USV application scenes from common marine areas to inland waters with complex environments, environmental perception has become an essential requirement for autonomous navigation systems of USVs. Traditional perception methods utilize either LiDAR or radar to construct volumetric maps for environmental perception. To improve the accuracy of perception systems and reduce deployment costs, this paper proposes a novel radar and camera fusion volumetric map network named FVMNet for real-time volumetric perception. FVMNet is based on a novel radar and image fusion architecture and comprises four modules. 1) The radar and image encoders can extract different features; 2) Only using in training stage without extra valid time costs, auxiliary segmentation head advances the image encoder; 3) To eliminate the representation difference between image features and radar features, the BEV spatial transformer module transfers image feature representations from the perspective view to BEV space; 4) The fusion segmentation head predicts the volumetric perception results. Compared to other baseline methods that use a single modality, FVMNet achieves state-of-the-art accuracy in public USVInland dataset and our collected wharf dataset. We conducted comprehensive ablation experiments to validate the efficacy of the designed modules. Moreover, the proposed method demonstrates generalization in zero-shot real-world scenarios and robustness under extreme weather conditions.

In recent years, the significance of unmanned surface vehicles (USVs) has grown substantially across a wide range of applications. Monocular cameras, as the most common perception sensors deployed in USVs, offer the inherent advantages of rich semantic information and low-cost deployment. However, visual perception methods for USVs face challenges from water surface application and lose efficiency in harsh weather cases. To achieve robust perception capabilities through monocular cameras for USVs, we propose a panoptic water surface visual perception framework that can accomplish various perception tasks, including drivable area segmentation, object detection and raindrop segmentation in water surface scenes. In addition, our framework provides a SAM-driven training method to improve the robustness of the proposed model through low-cost model iteration. Moreover, the proposed perception method demonstrates excellent accuracy, robustness and high inference speed compared to other light-weight baseline methods and can be deployed to low-power embedded platforms in USVs.

Autonomous surface vessels (ASVs) are growing rapidly due to their ability to execute hazardous and time-consuming missions over water surfaces. Recently, camera-based surround-view bird's eye view (BEV) segmentation has attracted increasing attention because of its popular application in autonomous driving. However, most existing surround-view BEV segmentation studies have focused on road scenes for autonomous cars, and the development of water surface surround-view BEV segmentation has been relatively slow due to the absence of water surface BEV datasets. Different from road scenes with complex dynamic scenes, water surface BEV segmentation task also face new interference like vessel swaying and reflection interference. To address these problems and stimulate relevant research, we first introduce the WSBEV dataset, a surround-view visual segmentation dataset on water surface scenes. Notably, our compiled WSBEV utilizes a hybrid camera configuration and contains pinhole cameras and fisheye cameras with different focal lengths, which also brings new challenges for the surround-view BEV segmentation method. To address the issues above, we propose a novel water surface BEV segmentation network named Hybrid-BEV that supports hybrid camera inputs. The geometry-based view transformer module is designed to directly transfer the extracted feature from the perspective view to BEV space. The pose query adjustment module introduces vessel pose data to suppress vessel swaying interference. Compared to other baseline methods, our Hybrid-BEV achieves excellent accuracy in our WSBEV dataset. Extensive experiments validate the effectiveness of our approach and the exceptional generalizability of the WSBEV dataset.

Recently, unmanned surface vessels (USVs) have played an increasingly important role in autonomous exploration, and boat detection is an important task for USVs. While most existing boat detection methods focus on 2-dimensional (2D) detection, 3-dimensional (3D) detection which provides valuable spatial direction for moving target estimation has not been studied in the boat detection field. However, 3D boat detection on water surfaces faces challenging problems, such as small sizes of detected targets and diverse moving directions. Considering that traditional LiDAR-based 3D boat detection methods require high hardware costs, we fuse millimeter wave (MMW) radar and high semantic camera to achieve low-cost and high-quality 3D boat detection. We propose a novel radar-camera fusion boat 3D detection model named RCBDet. The proposed RCBDet employs a new dual radar encoder and first introduces Doppler speed information from MMW radar into neural network to overcome sparse radar points. A new radar-camera attention module is designed to effectively combine camera features, radar spatial features and radar velocity features, encapsulating not only shape and semantic attributes but also spatial orientation information. In our collected boat 3D detection dataset, RCBDet achieves state-of-the-art accuracy compared to other single-modality baselines and radar-camera fusion baselines. Moreover, we conducted comprehensive ablation experiments to validate the efficacy of the designed modules. The experimental results demonstrated that the proposed radar-camera fusion model effectively fuses MMW radar features and camera features.

Infrared image segmentation in marine environments is crucial for enhancing nighttime observations and ensuring maritime safety. While recent advancements in deep learning have significantly improved segmentation accuracy, challenges remain due to nighttime marine scenes including low contrast and noise backgrounds. This paper introduces a cross-granularity infrared image segmentation network CGSegNet designed to address these challenges specifically for infrared images. The proposed method designs a hybrid feature framework with cross-granularity to enhance segmentation performance in complex water surface scenarios. To suppress feature semantic disparity against different feature granularity, we propose an adaptive multi-scale fusion module (AMF) that combines local granularity extraction with global context granularity. Additionally, incorporating a handcrafted histogram of oriented gradients (HOG) features, we designed a novel HOG feature fusion module to improve edge detection accuracy under low-contrast conditions. Comprehensive experiments conducted on the public infrared segmentation dataset demonstrate that our method outperforms state-of-the-art techniques, achieving superior segmentation results compared to professional infrared image segmentation methods. The results highlight the potential of our approach in facilitating accurate infrared image segmentation for nighttime marine observation, with implications for maritime safety and environmental monitoring.

Marine object detection is an essential component for maritime autonomous surface ship (MASS) systems, enabling them to perceive obstacle objects and avoid them. Existing detection algorithms primarily rely on visual sensors and face serious limitations in complex maritime environments. In this paper, we propose a novel joint object detection method based on the fusion of marine radar and camera data for robust maritime object detection. By employing a multi-stage hybrid-level fusion architecture, the proposed method efficiently fuses marine radar data with camera images, thereby enhancing the accuracy and robustness of object detection in complex maritime contexts. In addition, inspired by vision transformer detectors, a new transformer-based fusion module is proposed to mitigate semantic disparity between the two modalities and enhance radar information guidance through extracted object queries. Extensive experiments on public maritime perception datasets demonstrate the significant advantages of the proposed method in various maritime environments. The framework also exhibits strong generalization capability, making it applicable to diverse maritime settings, including challenging conditions like low visibility and high background noise. In public maritime perception datasets, the proposed method achieves state-of-the-art accuracy compared to other single-modality baselines and radar-camera fusion baselines, outperforming the excellent visual object detection model YOLOv8-x in terms of mAP0.35 increased by 13.72% in common scenarios and 24.85% in complex scenarios.