In order to implement more accurate and fine-grained monitoring of robot-environment interaction (REI) in real time, this article extends the concept of digital twin (DT) to REI description and proposes a novel semantic-enhanced DT system with exteroceptive and proprioceptive modalities of robots as mechanisms for states monitoring and semantics inference associated with interactions. We create a 3-D generic graphic model embedded with contact dynamics based on domain descriptions and sensor data to mirror a real interactive scene and simultaneously extract the appearance and physical properties of real entities. In addition, we present a multifeature fused visual relations detector and introduce a lightweight spatial relation data set to measure the visual relationships between entity pairs. To keep the DT consistent with the response of a real system, the model is updated with interactive subprocess-triggered control. In our system, the whole framework is developed with virtual reality techniques, which provides a high degree of observability. The experiments on the objects sorting task in a kitchen scene are conducted, and the abilities for system monitoring in real time and semantic reasoning are evaluated; the results demonstrate the effectiveness and feasibility of our proposal.

Robots are now working more and more closely with humans outside of traditional fences in industrial scenes. Their real-time tactile interaction perception is crucial to the safety of human–robot collaboration (HRC). In this work, we present a customized, wearable, and modular robot skin (TacSuit), which is scalable for large-area surface coverage of robot with easily accessible multimodal sensors, including pressure, proximity, acceleration, and temperature sensors. The TacSuit is co-designed for mechanical structure and data fusion algorithm, consisting of three levels of design: sensor, cell (of multimodal sensors), and block (of multiple cells). These sensors are stored with custom-designed and 3-D printed capsules to achieve the conformity, scalability, and easy installation to the arbitrary robot surface. A multilevel event-driven data fusion algorithm enables efficient information processing for large number of tactile sensors. Furthermore, a virtual interaction force fusion method takes both the proximity and force perception information into consideration in order to achieve safety of whole interaction process before and after direct physical contacts. A humanoid robotic platform successfully realizes the TacSuit wear of 159 tactile cells. Validation experiments of obstacle detection demonstrate the effective collision avoidance capability of the TacSuit for safe HRC.

Learning to achieve a user-specified objective from a random position in unseen environments is challenging for image-guided navigation agents. The abilities of long-horizon reasoning and semantic understanding are still lacking. Inspired by the human memory mechanism, we introduce a neural multi-store memory network to the reinforcement learning framework for target-driven visual navigation. The proposed memory network utilizes three temporal stages of memory to build time dependency for better scene understanding. Sensory memory encodes observations and embeds transient information into working memory, which is short-term and realized by a gated recurrent neural network (RNN). Then, the long-term memory stores the latent state from each step of the RNN into a single slot. Finally, a self-attention reading mechanism is designed to retrieve goal-related information from long-term memory. In addition, to improve the scene generalization capability of the agent, we facilitate training of the visual representation with a self-supervised auxiliary task and image augmentation. This method can navigate agents in unknown visual-realistic environments using only egocentric observations, without the need for any position sensors or pretrained models. The evaluation results on the Matterport3D dataset through the Habitat simulator demonstrate that our method outperforms the state-of-the-art approaches.

SUMMARY: Cable is the most important bearing structure of the cable-stayed bridges. Its safety has been of crucial public concern. Traditional manual cable inspection method has many defects such as low inspection efficiency, poor reliability and hazardous working environment. In this paper, a new wirelessly controlled cable-climbing robot enabling safe and convenient inspection of stay cables is proposed. The designed robot is composed of two modules, joined by four turnbuckles to form a closed structure that clasps the cable. The robot is controlled wirelessly by a ground-based station, and a DC power is supplied via an onboard lithium battery. The climbing principle and mechanical structure of this robot are introduced. The static model of the robot during obstacle negotiation is established. The relationships of the driving force and resistance with obstacle height to determine the obstacle-negotiation capability of the robot are obtained. The effects of cable diameter, cable inclination and preload force on obstacle climbing ability of the robot are also analyzed. The experiments verify that the robot could climb random inclined cables and overcome an obstacle of 2.42 mm in height with a mass of 5 kg payload.

Humanoid robots are designed and expected to resemble humans in structure and behavior, showing increasing application potentials in various fields. Like their biological counterparts, their environmental perception ability is fundamental. In particular, the visual and tactile perception are the two main sensory modes that humanoids use to understand and interact with the environment. Vision-Tactile Fusion Perception (VTFP) has shown multiple possibilities for better sensing understanding in challenging conditions, causing new research interests and questions. The overlap between visual and tactile perception in humanoids is continually growing. This work has reviewed the current state of the art of VTFP. It starts with the physiological basis of biological vision and tactile systems as well as the VTFP mechanisms as inspirations for humanoid perception. Then, the bioinspired visual-tactile fusion systems for humanoids are reviewed as the emphasis. After the survey on the vision and tactile sensors of robots, seven currently publicly available VTFP datasets are introduced. They are the data sources for several studies on neural network-inspired fusion algorithms. Furthermore, the applications of VTFP on humanoids are summarized. Finally, the challenges and future work are discussed. This review aims to provide several references for further exploitation of VTFP and its applications on humanoids.

Optoelectronic synapses can be crucial for advancing artificial intelligence and visual systems. Optoelectronic synapses based on organic field-effect transistors have been widely studied but still face significant challenges including obvious programming nonlinearity, restricted response wavelength, high operation voltage, and limited storage memory. Organic electrochemical transistors can be another candidate but lack intensive studies. Additionally, wafer-scale photolithographic fabrication on optoelectronic synapses responding to near-infrared (NIR) light is highly desirable but rarely reported. Here, we propose the NIR organic photoelectrochemical transistor (OPECT) array capable of low voltage multi-level memories fabricated by photolithography. Based on NIR photo-induced electrochemical doping mechanism, the OPECTs enable linear weight programming with ultra-low nonlinearity (−0.015) over a wide range (47.3). We further demonstrate OPECTs arrays for image sensing, memorization, and visualization. Eventually, a convolutional computing system is constructed, executing accurate recognition of noisy handwritten digits. This work offers a promising insight into neuromorphic sensory computing applications.