Despite the emergence of various human-robot collaboration frameworks, most are not sufficiently flexible to adapt to users with different habits. In this article, a Multimodal Reinforcement Learning Human-Robot Collaboration (MRLC) framework is proposed. It integrates reinforcement learning into human-robot collaboration and continuously adapts to the user's habits in the process of collaboration with the user to achieve the effect of human-robot cointegration. With the user's multimodal features as states, the MRLC framework collects the user's speech through natural language processing and employs it to determine the reward of the actions made by the robot. Our experiments demonstrate that the MRLC framework can adapt to the user's habits after repeated learning and better understand the user's intention compared to traditional solutions.

Detecting objects in remote sensing images is essential for intelligent interpretation. Although deep neural networks have made significant progress in recent years, they often struggle with complex backgrounds in remote sensing images, which can lead to inaccurate detection. To tackle this problem, a self-supervised adaptive fusion network (SSAFN) has been developed. The SSAFN includes an adaptive fusion module (AFM) and a self-supervised task module (SSTM). The AFM mainly fuses the deep semantic information to the shallow features with appropriate weights to enhance the semantic information of the shallow features. The SSTM is mainly to constrain the AFM through self-supervised tasks to fulfill the function similar to the attention mechanism: to make the AFM enhance the target feature representation and suppress the background information. The SSAFN reduces the impact of complex backgrounds on object representation, resulting in better detection results for various types of objects such as buildings, ships and more. The proposed method has been tested on various datasets and has not only improved the detection accuracy for different types of objects but also enhanced the performance of popular object detection algorithms.

Change detection (CD) from remote sensing images has been widely used in land management and urban planning. Benefiting from deep learning, numerous methods have achieved significant results in the CD of clearly changed targets. However, there are still significant challenges in the CD of weak targets, such as targets with small size, targets with blurred boundaries, and targets with low distinguishability from the background. Feature extraction from these targets can result in the loss of critical spatial features, potentially leading to decreased CD performance. Inspired by the improvement of multi-scale features for CD of weak target, a hierarchical feature interaction network with multi-scale fusion (HFIFNet) was proposed. Firstly, a hierarchical feature interactive fusion module (HFIFM) is proposed, which achieves optimized multi-channel feature interaction and enhances the distinguishability between weak targets and background. Moreover, the module also achieves cross scale feature fusion, which compensates for the loss of spatial feature of changed targets at a single scale during feature extraction. Secondly, VMamba Block (VMB) is utilized to obtain global features, and a spatial feature localization module (SFLM) was proposed to enhance the saliency of spatial features such as edges and textures. The distinguishability between weak targets and irrelevant spatial features is further enhanced. Our method has been experimentally evaluated on three public datasets, and outperformed state-of-the-art approaches by 1.06%, 1.41% and 2.63% in F1 score on the LEVIR-CD, S2Looking, and NALand datasets, respectively. These results affirm the effectiveness of our method for weak targets in CD tasks.

The current virtual system for secondary school experiments poses several issues, such as limited methods of operation for students and an inability of the system to comprehend the users’ operational intentions, resulting in a greater operational burden for students and hindering the goal of the experimental practice. However, many traditional multimodal fusion algorithms rely solely on individual modalities for the analysis of users’ experimental intentions, failing to fully utilize the intention information for each modality. To rectify these issues, we present a new multimodal fusion algorithm, MFIRA, which intersects and blends intention probabilities between channels by executing parallel processing of multimodal information at the intention layer. Additionally, we developed an augmented reality (AR) virtual experiment platform based on the Hololens 2, which enables students to conduct experiments using speech, gestures, and vision. Employing the MFIRA algorithm, the system captures users’ experimental intent and navigates or rectifies errors to guide students through their experiments. The experimental results indicate that the MFIRA algorithm boasts a 97.3% accuracy rate in terms of interpreting users’ experimental intent. Compared to existing experimental platforms, this system is considerably more interactive and immersive for students and is highly applicable in secondary school experimental chemistry classrooms.