Sliding detection can effectively enhance the stability of robot grasping operations. Methods relying solely on 2D vision or tactile information for sliding detection often exhibit limited applicability across diverse scenarios. Furthermore, 2D visual’s sensitivity to the changes of light intensity leads to inadequate performance in overexposure scenarios. In this paper, we propose a novel deep neural network called PTFD-Net, which serves the purpose of detecting the sliding state when robots grasp objects. Specifically, we utilize an improved PointRNN algorithm and 3D convolution to extract features from the point cloud sequences and tactile sequences to construct a multi-modal sequence information prediction model. In addition, we construct a dataset that integrates point cloud sequences with tactile sequences. The impact of different point cloud sequences lengths and tactile image sizes on the performance of PTFD-Net is discussed to determine the optimal parameters. The model can attain a sliding state detection accuracy of 95.49% on the dataset. Moreover, we conduct performance comparisons with baseline algorithms under overexposure scenarios, demonstrating PTFD-Net’s superiority over baseline algorithms and achieving a sliding detection accuracy of 92.71%. Finally, we conduct robustness experiments on unknown objects under different grasping poses and occlusion scenarios, resulting in sliding detection accuracies of 86.02% and 89.43% respectively, both surpassing single-modal models. These results demonstrate the effectiveness of PTFD-Net in extracting multi-modal features, enabling it to perform better robot grasping tasks in complex environments.

Due to the frailty of elderly individuals’ physical condition, falling can lead to severe bodily injuries. Effective fall detection can significantly reduce the occurrence of such incidents. However, current fall detection methods heavily rely on visual and multi-sensor devices, which incur higher costs and complex wearable designs, limiting their wide-ranging applicability. In this paper, we propose a fall detection method based on nursing aids integrated with multi-array flexible tactile sensors. We design a kind of multi-array capacitive tactile sensor and arrange the distribution of tactile sensors on the foot based on plantar force analysis and measure tactile sequences from the sole of the foot to develop a dataset. Then we construct a fall detection model based on a graph convolution neural network and long-short term memory network (GCN-LSTM), where the GCN module and LSTM module separately extract spatial and temporal features from the tactile sequences, achieving detection on tactile data of foot and walking states for specific time series in the future. Experiments are carried out with the fall detection model, the Mean Squared Error (MSE) of the predicted tactile data of the foot at the next time step is 0.0716, with the fall detection accuracy of 96.36%. What is more, the model can achieve fall detection on 5-time steps with 0.2-s intervals in the future with high confidence results. It exhibits outstanding performance, surpassing other baseline algorithms. Besides, we conduct experiments on different ground types and ground morphologies for fall detection, and the model showcases robust generalization capabilities.

Three-fingered hands can offer more contact points and flexible fingertip configurations, enabling complex grasping modes and finer manipulations for objects of various shapes and sizes. However, existing research on visual-tactile integrated robotic grasping primarily focuses on grippers, and lacks a general dataset merging visual and tactile information for the entire grasping process. In this letter, we introduce the VTG dataset, which can be used for various aspects of three-fingered robotic grasping control. The VTG dataset includes three-view point clouds of objects, grasping modes, and finger angles of three-fingered hands, as well as tactile data at multiple spatial contact locations during the grasping process. By integrating visual and tactile information, we develop a robotic grasping controller that leverages a grasping stability prediction module and a grasping adjustment module. By representing tactile data as a static graphical structure based on the spatial distribution of tactile sensors, the grasping stability prediction module is constructed based on a multi-scale graph neural network, MS-GCN. It combines multi-scale graph topological features with various grasping modes, and achieving an accuracy of 98.4% in robotic grasping stability prediction. Additionally, this controller successfully adapts to unknown objects of varying hardness and shapes, providing stable grasping within approximately 0.4s after contact.

Reliable tasks execution of wheeled platform requires high perceptive ability in terrains. Currently, vision perception is susceptible to external factors such as lighting conditions and air particles, and vibration perception reflects no surface features of terrains. In this paper, we propose a novel system geared towards terrain classification based on tactile perception, well addressing those shortcomings. We develop a type of capacitive flexible tactile sensors array for three-dimensional forces with wide measuring range, high sensitivity, considerable adaptability, and strongly durability. To fully exploit the terrain features of the collected data, we propose a characterization method that encode tactile information as image flow encompassing spatiotemporal information, and establish a novel tactile-based terrain classification dataset. We construct the image flow as special tokens and feed them to a multi-head spatiotemporal attention network, with spatial and temporal heads evenly constructed, to ultimately realize terrain classification. Our network achieves an accuracy of 91.9%, demonstrating the superiority over existing algorithms. Accuracies achieve are 81.3% and 76.3%, respectively with 8 kilograms burden and at triple speed. Moreover, the performance degradation caused by increasing speed can be alleviated by decreasing time steps.

Accurately grasping objects with varying levels of hardness and softness is a challenging task. Current robotic grasping methods rely solely on visual to determine the grasping posture, without considering the grasping force of different objects, which might increase the risk of slippage or damage. For the purpose of realize the improvement and supplementation of visual information from the dimension of force, several robots include tactile sensors installed at fingertips. In this paper, we propose a method that integrates visual and tactile information to accurately determine the 6D posture and force for robot grasping. The main challenges of this research include (1) grasping representation of visual and tactile data from heterogeneous sources, (2) Construction of the implicit link between grasping posture and clutching force , (3) a visual–tactile training dataset. To address these challenges, we first use a grasping quality assessment module to filter the grasping posture and a transformer-depth separation convolution module to predict the grasping force; second, we propose a diffusion model with multi-stage convolution module to generate the optimal grasping. In addition, we created a visual–tactile grasping data representation form that included the grasping quality of grasping postures, and matching tactile data for objects of 8 distinct hardness. Experiments show that our method achieves a 98% success rate and decreases grasping force redundancy by 78% compared to purely visual methods,which significantly improves the accuracy and stability of the robot grasping.