Robots are now working more and more closely with humans outside of traditional fences in industrial scenes. Their real-time tactile interaction perception is crucial to the safety of human–robot collaboration (HRC). In this work, we present a customized, wearable, and modular robot skin (TacSuit), which is scalable for large-area surface coverage of robot with easily accessible multimodal sensors, including pressure, proximity, acceleration, and temperature sensors. The TacSuit is co-designed for mechanical structure and data fusion algorithm, consisting of three levels of design: sensor, cell (of multimodal sensors), and block (of multiple cells). These sensors are stored with custom-designed and 3-D printed capsules to achieve the conformity, scalability, and easy installation to the arbitrary robot surface. A multilevel event-driven data fusion algorithm enables efficient information processing for large number of tactile sensors. Furthermore, a virtual interaction force fusion method takes both the proximity and force perception information into consideration in order to achieve safety of whole interaction process before and after direct physical contacts. A humanoid robotic platform successfully realizes the TacSuit wear of 159 tactile cells. Validation experiments of obstacle detection demonstrate the effective collision avoidance capability of the TacSuit for safe HRC.

With the development and popularization of the Internet of Things, big data, cloud computing, and other technologies, Digital twin technology (DTT) is increasingly applied to building operation and maintenance (O&M). However, most of the research focuses on building energy consumption, safety, and other management, and relatively little research on the monitoring of building terminal energy using information. The information is closely related to occupants’ behavior, such as air conditioning, lighting, shading, window status information, as well as personnel number and location, and it has a great impact on building energy consumption. Due to different occupants’ behaviors, the level of building energy consumption varies several times or even more. Take an office building as an example. Based on digital twin technology, the framework of building energy using intelligent monitoring is constructed. It mainly includes four parts, namely building physical space, virtual twin space, predictive control simulation engine, and twin big data. For each part, functions are realized through building Information Modelling (BIM), smart sensors, and Internet of Things (IoT) technologies. Based on the standard framework and every function realization method, the DTT can used for building O&M effectually. The application of building an intelligent control system based on the occupants’ characteristics is simulated and analyzed in Designbuilder software 6.1.2. The results show that the digital technology application in building intelligent control systems can realize maximum energy saving for 30%. However, the DTT in building O&M is not widely used now. There is a lot of research to be completed in the future.

Purpose: Vision, audition, olfactory, tactile and taste are five important senses that human uses to interact with the real world. As facing more and more complex environments, a sensing system is essential for intelligent robots with various types of sensors. To mimic human-like abilities, sensors similar to human perception capabilities are indispensable. However, most research only concentrated on analyzing literature on single-modal sensors and their robotics application. Design/methodology/approach: This study presents a systematic review of five bioinspired senses, especially considering a brief introduction of multimodal sensing applications and predicting current trends and future directions of this field, which may have continuous enlightenments. Findings: This review shows that bioinspired sensors can enable robots to better understand the environment, and multiple sensor combinations can support the robot’s ability to behave intelligently. Originality/value: The review starts with a brief survey of the biological sensing mechanisms of the five senses, which are followed by their bioinspired electronic counterparts. Their applications in the robots are then reviewed as another emphasis, covering the main application scopes of localization and navigation, objection identification, dexterous manipulation, compliant interaction and so on. Finally, the trends, difficulties and challenges of this research were discussed to help guide future research on intelligent robot sensors.

Human tactile perception enables rapid assessment of deformable objects and the application of appropriate force to prevent slip or excessive deformation. However, this task remains challenging for robots. To address this issue, we propose the T-TD3 algorithm, which utilizes a multi-scale fusion neural network (MSF-Net) for the fused perception of multi-scale features, including the tactile prior information obtained through preprocessing. Our approach decomposes the robot task of grasping deformable objects into three subtasks: slip detection, stable grasping evaluation, and minimum grasping force tracking. We develop a simulation environment called CR5GraspStable-Env using PyBullet and TACTO for the network training. Our work reports a success rate of 94.81% in the robot task of grasping deformable objects in real, demonstrating an excellent sim-to-real capability. Moreover, the proposed approach has the potential to be extended to other stable grasping tasks that utilize tactile perception. Note to Practitioners —Traditional grasping strategies in stable grasping tasks typically apply significant grasping force to prevent slip. However, excessive grasping force for deformable objects may lead to excessive deformation and damage. While humans can flexibly control grasping force through tactile perception, it poses a significant challenge for robots. To overcome this challenge, we propose a novel method for the robot to learn an improved stable grasping strategy by incorporating tactile priors. Our method establishes a unified tactile prior representation for the visual-based tactile sensors mounted on the robot grippers, which enables them to sense the contact state. Additionally, it utilizes sensor distortion correction based on spatial symmetry to ensure the applicability of the tactile prior representation to any kind of visual-based tactile sensor. Furthermore, this method integrates multi-scale tactile priors and robot states, and utilizes reinforcement learning to autonomously make real-time decisions, aiming to minimize the grasping force while maintaining stable grasps on deformable objects. The primary research objective of this article is to address the challenge of achieving stable grasps on variable objects, while also being applicable to grasping rigid objects. We validated the practicality of this method by successfully achieving a 94.81% success rate in stably grasping deformable objects using the robotic arm CR5 in both simulated and real-world environments. Furthermore, this method can be readily applied to other robot systems. In the future, we plan to extend the application of our proposed method to more dexterous manipulators and perform more complex manipulation tasks. Additionally, we aim to introduce new fusion algorithms and decision-making strategies to further enhance the applicability of this method.

Deep learning (DL) models such as multilayer perceptrons (MLPs) and convolutional neural networks (CNNs) have strong feature representation and nonlinear mapping capabilities, and their effectiveness has been demonstrated in fault diagnosis. However, fault features usually occur at different scales and are always disturbed by noise, making it difficult for DL-based models to learn local and global information in mechanical vibration signals. To address this issue, a multigrained hybrid neural network named MgHNN is proposed to extract robust features that seamlessly integrate CNN into vision MLP. First, the short-time Fourier transform is performed on original vibration signals to obtain time-frequency images, and each image is then divided into multiple nonoverlapping patches. Second, a novel multigrained feature representation (MFR) block is proposed by constructing hierarchical residual-like connections within one single wave block, which is more suitable for learning hierarchical local and global feature representations among different image patches. Third, we propose a depthwise wave (DWwave) block by integrating depthwise convolution and feature concatenation operations, which can make MFR block better focus on the local information and effectively overcome vanishing gradient problem. Finally, experimental results on two fault diagnosis datasets demonstrate that the MgHNN has improved diagnostic accuracy and reduced model complexity compared to state-of-the-art models. The results of noise interference experiments indicate that the MgHNN exhibits superior robustness against noise.

The emerging embodied AI paradigm enables intelligent robots to learn like humans from interaction, and is thus considered an effective way approach to general artificial intelligence. Unfortunately, even the best performing agents still overfit and generalize poorly to unseen scenes, due to the limited scenes provided by embodied AI simulators. To alleviate this issue, we propose a scene augmentation strategy to scale up the scene diversity for interactive tasks and make the interactions more like the real world. Compared to existing methods focusing on improving diversity in observation space, our approach aims to automatically derive a new distribution of scene layout or object states to provide sufficient conditional transfer models for the agent to learn environmental invariant and irrelevant features through interaction. Specifically, we provide four representative and systematical scene augmentation methods that can derive scene variants for entities from different levels of a scene graph. We demonstrate the efficiency of our methods in the popular embodied AI simulator iGibson. To verify the effectiveness for interactive agents, we also conduct two representative interactive tasks with a proposed continuous action parameterized method. The evaluation results show that our scene augmentation strategy can boost the performance of interactive agents and generalize well to unseen scenes. Finally, we present a systematic generalization analysis using the proposed methods to explicitly estimate the ability of agents to generalize to new layouts, new objects, and new object states. We claim that the proposed methods are not limited to iGibson and can be extended to other interactive simulators. The code and additional information are available at: https://github.com/sanghongrui/SceneAug.