Recent advances in machine learning have paved the way for the development of musical and entertainment robots. However, human-robot cooperative instrument playing remains a challenge, particularly due to the intricate motor coordination and temporal synchronization. In this paper, we propose a theoretical framework for human-robot cooperative piano playing based on non-verbal cues. First, we present a music improvisation model that employs a recurrent neural network (RNN) to predict appropriate chord progressions based on the human's melodic input. Second, we propose a behavior-adaptive controller to facilitate seamless temporal synchronization, allowing the cobot to generate harmonious acoustics. The collaboration takes into account the bidirectional information flow between the human and robot. We have developed an entropy-based system to assess the quality of cooperation by analyzing the impact of different communication modalities during human-robot collaboration. Experiments demonstrate that our RNN-based improvisation can achieve a 93% accuracy rate. Meanwhile, with the MPC adaptive controller, the robot could respond to the human teammate in homophony performances with real-time accompaniment. Our designed framework has been validated to be effective in allowing humans and robots to work collaboratively in the artistic piano-playing task.
13 November 2024 in IEEE Robotics and Automation Letters
The human hand is capable of executing a wide range of complex movements due to its biomechanical structure and skin sensing system. Designing an anthropomorphic hand that mimics the biomechanical structure of the human hand and incorporates skin sensing, presents a long-term challenge in the field of robotics. In this paper, we proposed a concept for structure design, which is to combine rigid, flexible and soft components, and designed a rigid-flexible-soft coupled dexterous hand based on this concept. For enhancing dexterous hand's adaptivity to environment, we also developed a soft piezoresistive tactile module inspired by human skin, and mounted it on fingertips to detect sliding states. Meanwhile, we have also designed an integrated system for dexterous manipulation including sensing, actuation and control, based on the concept of embodied intelligence, aiming to achieve a closed-loop control to dexterous hand. This work provides a reliable structure and control strategy to enrich the perceptual abilities of the dexterous hand and enable their applications in unstructured environments.
While vacuum grippers offer an economical solution, the environmental impact of energy waste through their suction cups (SCs) cannot be overlooked. This waste stems from three key factors: i) air losses from idle SCs arranged in arrays, ii) inadequate sealing on target surfaces leading to air leaks, and iii) damage from sharp objects resulting in leaking perforations. To overcome these challenges, in this article, a comprehensive approach is presented that involves the development of a i) self‐closing, ii) multi‐material, and iii) self‐healing system based on reversible elastomers cross‐linked via the Diels–Alder (DA) reaction. The system incorporates a fully autonomous self‐closing mechanism to prevent energy waste in SC arrays during periods of non‐contact. Fluid–structure interaction simulations are utilized to analyze the design. Versatility and stability are achieved by incorporating hyper‐flexible and stiff elastomers in a multi‐material design, supported by covalent DA cross‐links that ensure robustness through high‐strength multi‐material interfaces. These DA cross‐links also enable self‐healing capabilities, allowing the SCs to recover from macroscopic damages within 1 day at ambient conditions or in a single hour with mild heating (80–90 °C), restoring full performance. Additionally, in the article, a recycling method is introduced for multi‐material SCs based on the mechanical separation of reversible polymers.
Soft robots are characterized by their mechanical compliance, making them well-suited for various bio-inspired applications. However, the challenge of preserving their flexibility during deployment has necessitated using soft sensors which can enhance their mobility, energy efficiency, and spatial adaptability. Through emulating the structure, strategies, and working principles of human senses, soft robots can detect stimuli without direct contact with soft touchless sensors and tactile stimuli. This has resulted in noteworthy progress within the field of soft robotics. Nevertheless, soft, touchless sensors offer the advantage of non-invasive sensing and gripping without the drawbacks linked to physical contact. Consequently, the popularity of soft touchless sensors has grown in recent years, as they facilitate intuitive and safe interactions with humans, other robots, and the surrounding environment. This review explores the emerging confluence of touchless sensing and soft robotics, outlining a roadmap for deployable soft robots to achieve human-level dexterity.
Understanding the coordination of multiple biomechanical degrees of freedom in biological organisms is crucial for unraveling the neurophysiological control of sophisticated motor tasks. This study focuses on the cooperative behavior of upper-limb motor movements in the context of octave playing on the piano. While the vertebrate locomotor system has been extensively investigated, the coherence and precision timing of rhythmic movements in the upper-limb system remain incompletely understood. Inspired by the spinal cord neuronal circuits (central pattern generator, CPG), a computational neuro-musculoskeletal model is proposed to explore the coordination of upper-limb motor movements during octave playing across varying tempos and volumes. The proposed model incorporates a CPG-based nervous system, a physiologically-informed mechanical body, and a piano environment to mimic human joint coordination and expressiveness. The model integrates neural rhythm generation, spinal reflex circuits, and biomechanical muscle dynamics while considering piano playing quality and energy expenditure. Based on real-world human subject experiments, the model has been refined to study tempo transitions and volume control during piano playing. This computational approach offers insights into the neurophysiological basis of upper-limb motor coordination in piano playing and its relation to expressive features.
Flexible, soft materials are increasingly used for the fabrication of soft robots, as the inherent compliance and shock‐absorbance protect the robot from mechanical impact. Soft universal grippers take full advantage of this adaptability, facilitating effective and safe grasping of various objects. However, due to their predominantly soft material composition, these grippers have limited lifetimes, especially when operating in unstructured and unfamiliar environments. The self‐healing universal gripper (SHUG) is proposed, which can grasp various objects and recover from substantial realistic damages autonomously. It integrates damage detection, heat‐assisted healing, and healing evaluation. Notably, unlike other universal grippers, the entire SHUG can be fully reprocessed and recycled. The gripper's functionality relies on the particle jamming of steel balls enclosed within a self‐healing membrane. Thanks to the thermoreversible covalent Diels–Alder bonds in self‐healing polymer membrane, the gripper is able to recover from macroscopic damages including scratches and punctures. Temperature‐assisted healing is regulated in a closed‐loop manner using an embedded thermocouple and Joule heater. Experimental validation demonstrates the adaptability, resilience, and recyclability of the SHUG.