In order to implement more accurate and fine-grained monitoring of robot-environment interaction (REI) in real time, this article extends the concept of digital twin (DT) to REI description and proposes a novel semantic-enhanced DT system with exteroceptive and proprioceptive modalities of robots as mechanisms for states monitoring and semantics inference associated with interactions. We create a 3-D generic graphic model embedded with contact dynamics based on domain descriptions and sensor data to mirror a real interactive scene and simultaneously extract the appearance and physical properties of real entities. In addition, we present a multifeature fused visual relations detector and introduce a lightweight spatial relation data set to measure the visual relationships between entity pairs. To keep the DT consistent with the response of a real system, the model is updated with interactive subprocess-triggered control. In our system, the whole framework is developed with virtual reality techniques, which provides a high degree of observability. The experiments on the objects sorting task in a kitchen scene are conducted, and the abilities for system monitoring in real time and semantic reasoning are evaluated; the results demonstrate the effectiveness and feasibility of our proposal.

Robots are now working more and more closely with humans outside of traditional fences in industrial scenes. Their real-time tactile interaction perception is crucial to the safety of human–robot collaboration (HRC). In this work, we present a customized, wearable, and modular robot skin (TacSuit), which is scalable for large-area surface coverage of robot with easily accessible multimodal sensors, including pressure, proximity, acceleration, and temperature sensors. The TacSuit is co-designed for mechanical structure and data fusion algorithm, consisting of three levels of design: sensor, cell (of multimodal sensors), and block (of multiple cells). These sensors are stored with custom-designed and 3-D printed capsules to achieve the conformity, scalability, and easy installation to the arbitrary robot surface. A multilevel event-driven data fusion algorithm enables efficient information processing for large number of tactile sensors. Furthermore, a virtual interaction force fusion method takes both the proximity and force perception information into consideration in order to achieve safety of whole interaction process before and after direct physical contacts. A humanoid robotic platform successfully realizes the TacSuit wear of 159 tactile cells. Validation experiments of obstacle detection demonstrate the effective collision avoidance capability of the TacSuit for safe HRC.

Wireless sensor networks (WSNs) are widely used in the field of monitoring. For data collection of sensor nodes in large-scale monitoring scenarios, unmanned aerial vehicle (UAV)-assisted WSNs have emerged. For the security and validity of data collection, a blockchain-enhanced data collection framework for UAV-assisted WSNs is presented in this article. To reduce data redundancy in WSNs, a sparsity-optimized and compressed sensing-based spatiotemporal data aggregation model is built. A UAV identity authentication mechanism based on a Merkle tree is also designed to ensure the security of data transmission. By combining blockchain building and data aggregation, a disaster semantic blockchain (DSB) based on a data reconstruction-directed consensus mechanism is presented. Disaster semantics are extracted by analyzing the semantic association relationship of disaster, background, event, and sensor data. The experimental results show that the blockchain-enhanced spatiotemporal data aggregation effectively increases the network life cycle and data reconstruction accuracy. The disaster situation can be described accurately through the DSB.

Learning to achieve a user-specified objective from a random position in unseen environments is challenging for image-guided navigation agents. The abilities of long-horizon reasoning and semantic understanding are still lacking. Inspired by the human memory mechanism, we introduce a neural multi-store memory network to the reinforcement learning framework for target-driven visual navigation. The proposed memory network utilizes three temporal stages of memory to build time dependency for better scene understanding. Sensory memory encodes observations and embeds transient information into working memory, which is short-term and realized by a gated recurrent neural network (RNN). Then, the long-term memory stores the latent state from each step of the RNN into a single slot. Finally, a self-attention reading mechanism is designed to retrieve goal-related information from long-term memory. In addition, to improve the scene generalization capability of the agent, we facilitate training of the visual representation with a self-supervised auxiliary task and image augmentation. This method can navigate agents in unknown visual-realistic environments using only egocentric observations, without the need for any position sensors or pretrained models. The evaluation results on the Matterport3D dataset through the Habitat simulator demonstrate that our method outperforms the state-of-the-art approaches.

SUMMARY: Cable is the most important bearing structure of the cable-stayed bridges. Its safety has been of crucial public concern. Traditional manual cable inspection method has many defects such as low inspection efficiency, poor reliability and hazardous working environment. In this paper, a new wirelessly controlled cable-climbing robot enabling safe and convenient inspection of stay cables is proposed. The designed robot is composed of two modules, joined by four turnbuckles to form a closed structure that clasps the cable. The robot is controlled wirelessly by a ground-based station, and a DC power is supplied via an onboard lithium battery. The climbing principle and mechanical structure of this robot are introduced. The static model of the robot during obstacle negotiation is established. The relationships of the driving force and resistance with obstacle height to determine the obstacle-negotiation capability of the robot are obtained. The effects of cable diameter, cable inclination and preload force on obstacle climbing ability of the robot are also analyzed. The experiments verify that the robot could climb random inclined cables and overcome an obstacle of 2.42 mm in height with a mass of 5 kg payload.

Humanoid robots are designed and expected to resemble humans in structure and behavior, showing increasing application potentials in various fields. Like their biological counterparts, their environmental perception ability is fundamental. In particular, the visual and tactile perception are the two main sensory modes that humanoids use to understand and interact with the environment. Vision-Tactile Fusion Perception (VTFP) has shown multiple possibilities for better sensing understanding in challenging conditions, causing new research interests and questions. The overlap between visual and tactile perception in humanoids is continually growing. This work has reviewed the current state of the art of VTFP. It starts with the physiological basis of biological vision and tactile systems as well as the VTFP mechanisms as inspirations for humanoid perception. Then, the bioinspired visual-tactile fusion systems for humanoids are reviewed as the emphasis. After the survey on the vision and tactile sensors of robots, seven currently publicly available VTFP datasets are introduced. They are the data sources for several studies on neural network-inspired fusion algorithms. Furthermore, the applications of VTFP on humanoids are summarized. Finally, the challenges and future work are discussed. This review aims to provide several references for further exploitation of VTFP and its applications on humanoids.