The digital twin can reconstruct the scenes of real traffic operation of inland waterways in virtual space, giving birth to new ideas for the digital transformation and upgrading of water safety management. Aiming at the problems of video fragmentation, sub-system data separation and untimely emergency response in the current inland waterway safety monitoring, this paper proposes a new method for reconstructing and applying a digital twin of inland waterways based on 3D video fusion which consists of four parts: (1) Combining drone tilt photography and BIM modeling technology to construct a three-dimensional scene model of inland waterways; (2) Proposing a multi-channel video three-dimensional registration and rendering method to integrate in real time discrete surveillance videos of different angles of view with three-dimensional scene model to establish the spatial correlation between different video images in the scene; (3) Integrating multi-source IoT sensing data to form a scene of digital twin of inland waterways; (4) Exploring innovative application methods for water safety management based on digital twins to meet the water safety management requirements and business logic of inland waterways. A certain segment of the Changhu Shen Line was selected for demonstration and application. The application results show that the construction and application of digital twins for inland waterways combined with 3D video reduces the pressure on managers to understand real-time video footage and improves the efficiency of daily monitoring, evidence collection, and emergency response.

Automated detection of road damage (ADRD) is a challenging topic in road maintenance. It focuses on automatically detecting road damage and assessing severity by deep learning. Because of the sparse distribution of characteristic pixels, it is more challenging than object detection. Although some public datasets provide a database for the development of ADRD, their amounts of data and the standard of classification cannot meet network training and feature learning. With the aim of solving this problem, this work publishes a new road damage dataset named CNRDD, which is labeled according to the latest evaluation standard for highway technical conditions in China (JTG5210-2018). The dataset is collected by professional onboard cameras and is manually labeled in eight categories with three different degrees (mild, moderate and severe), which can effectively help promote research of automated detection of road damage. At the same time, a novel baseline with attention fusion and normalization is proposed to evaluate and analyze the published dataset. It explicitly leverages edge detection cues to guide attention for salient regions and suppresses the weights of non-salient features by attention normalization, which can alleviate the interference of sparse pixel distribution on damage detection. Experimental results demonstrate that the proposed baseline significantly outperforms most existing methods on the existing RDD2020 dataset and the newly released CNRDD dataset. Further, the CNRDD dataset is proved more robust, as its high damage density and professional classification are more conducive to promote the development of ADRD.

Road damage detection (RDD) is indispensable for safe autonomous driving. Existing RDD models focus on designing feature representations following expert knowledge. However, collecting and labeling all types of samples is time-consuming and leads to insufficient training data. To alleviate the adverse effect of few training samples, a novel few-shot road damage detector (FSRDD) is proposed in this paper to detect rare road damages. The proposed FSRDD includes three stages. First, fully annotated abundant base classes are leveraged to train a base detector, where ghost attention (GA) and proposal feature metric (PFM) modules are developed to eliminate the redundant information and measure the proposal features, respectively. Second, the recognition branch of the detector is fine-tuned using a few samples of all classes. Finally, the test set is inferred with the help of an offline scale-aware prototypical calibration block (SPCB). Extensive experiments show that our FSRDD achieves 10-shot rare road damage detection with 33.4% and 12.9% mAP50 on RDD and CNRDD datasets, respectively, significantly outperforming state-of-the-art methods.

The more sensory channels are equipped in a virtual assembly system, the more real users feel in the whole system, however, most of the existing virtual assembly systems are based on the natural interaction method of one or two sensory channels. Thus, this paper proposes a novel virtual assembly system integrating multi-sensory channels, including gesture interaction, Chinese speech interaction, tactile interaction, 3D display and real-time display of real environment pictures in the virtual environment. To improve the operability of the virtual environment, we analyze the parallel virtual assembly sequence on the basis of two-hand interaction, and the assembly priority is prompted based on UI interface. For ease of operation, we present a method of viewpoint control based on gesture interaction and the coordinate threshold of spatial position. A hierarchical bounding box collision detection algorithm based on volume difference is proposed to improve the efficiency of collision feedback and collision avoidance. In addition, the power equipment models are exhibited in the virtual scene, as the exhibits of the virtual roaming process. Finally, to evaluate the training effect of this system, a comparative experiment is designed to compare the participants’ experience and the effect of assembly training. The experimental results show that the virtual assembly system of natural interaction with multi-sensory channels is flexible and immersive.