Electronic skin (e-skin) is designed to mimic the comprehensive nature of human skin. Various advances in e-skin continue to drive the development of the multimodal tactile sensor technology on flexible and stretchable platforms. e-skin incorporates pressure, temperature, texture, photographic imaging, and other sensors as well as data acquisition and signal processing units formed on a soft substrate for humanoid robots, wearable devices, and health monitoring electronics that are the most critical applications of soft electronics. This artificial skin has developed very rapidly toward becoming real technology. However, the complex nature of e-skin technology presents significant challenges in terms of materials, devices, sophisticated integration methods, and interference-free data acquisition. These challenges range from functional materials, device architecture, pixel design, array structure, and data acquisition method to multimodal sensing performance with negligible interference. In this article, we present recent research trends and approaches in the field of flexible and stretchable multimodal sensors for e-skin focusing on the following aspects: 1) flexible and stretchable platforms; 2) operating principles and materials suitable for pressure, temperature, strain, photograph, and hairy sensor devices; 3) device and integration architectures, including multimodal single cells, three-axis tactile sensors, vertical-stacked sensor arrays, active matrix sensor arrays, and integration electronics; 4) reliable acquisition methods for various texture sensing and machine-learning algorithms for processing tactile sensing data; and 5) future outlook.

Executing contact-rich manipulation tasks necessitates the fusion of tactile and visual feedback. However, the distinct nature of these modalities poses significant challenges. In this paper, we introduce a system that leverages visual and tactile sensory inputs to enable dexterous in-hand manipulation. Specifically, we propose Robot Synesthesia, a novel point cloudbased tactile representation inspired by human tactile-visual synesthesia. This approach allows for the simultaneous and seamless integration of both sensory inputs, offering richer spatial information and facilitating better reasoning about robot actions. Comprehensive ablations are performed on how the integration of vision and touch can improve reinforcement learning and Sim2Real performance. Our project page is available at https://yingyuan0414.github.io/visuotactile/.

In this paper, we present an interaction force estimation method that uses visual information rather than that of a force sensor. Specifically, we propose a novel deep learning-based method utilizing only sequential images for estimating the interaction force against a target object, where the shape of the object is changed by an external force. The force applied to the target can be estimated by means of the visual shape changes. However, the shape differences in the images are not very clear. To address this problem, we formulate a recurrent neural network-based deep model with fully-connected layers, which models complex temporal dynamics from the visual representations. Extensive evaluations show that the proposed learning models successfully estimate the interaction forces using only the corresponding sequential images, in particular in the case of three objects made of different materials, a sponge, a PET bottle, a human arm, and a tube. The forces predicted by the proposed method are very similar to those measured by force sensors.

In this paper, we propose a compact force sensor system for a hip-mounted exoskeleton for seniors with difficulties in walking due to muscle weakness. It senses and monitors the delivered force and power of the exoskeleton for motion control and taking urgent safety action. Two FSR (force-sensitive resistors) sensors are used to measure the assistance force when the user is walking. The sensor system directly measures the interaction force between the exoskeleton and the lower limb of the user instead of a previously reported force-sensing method, which estimated the hip assistance force from the current of the motor and lookup tables. Furthermore, the sensor system has the advantage of generating torque in the walking-assistant actuator based on directly measuring the hip-assistance force. Thus, the gait-assistance exoskeleton system can control the delivered power and torque to the user. The force sensing structure is designed to decouple the force caused by hip motion from other directional forces to the sensor so as to only measure that force. We confirmed that the hip-assistance force could be measured with the proposed prototype compact force sensor attached to a thigh frame through an experiment with a real system.

Compared to laparoscopy, robotics-assisted minimally invasive surgery has the problem of an absence of force feedback, which is important to prevent a breakage of the suture. To overcome this problem, surgeons infer the suture force from their proprioception and 2D image by comparing them to the training experience. Based on this idea, a deep-learning-based method using a single image and robot position to estimate the tensile force of the sutures without a force sensor is proposed. A neural network structure with a modified Inception Resnet-V2 and Long Short Term Memory (LSTM) networks is used to estimate the suture pulling force. The feasibility of proposed network is verified using the generated DB, recording the interaction under the condition of two different artificial skins and two different situations (in vivo and in vitro) at 13 viewing angles of the images by changing the tool positions collected from the master-slave robotic system. From the evaluation conducted to show the feasibility of the interaction force estimation, the proposed learning models successfully estimated the tensile force at 10 unseen viewing angles during training.