Robots execute diverse load operations, including carrying, lifting, tilting, and moving objects, involving load changes or transfers. This dynamic process can result in the shift of interactive operations from stability to instability. In this paper, we respond to these dynamic changes by utilizing tactile images captured from tactile sensors during interactions, conducting a study on the dynamic stability and instability in operations, and propose a real-time dynamic state sensing network by integrating convolutional neural networks (CNNs) for spatial feature extraction and long short-term memory (LSTM) networks to capture temporal information. We collect a dataset capturing the entire transition from stable to unstable states during interaction. Employing a sliding window, we sample consecutive frames from the collected dataset and feed them into the network for the state change predictions of robots. The network achieves both real-time temporal sequence prediction at 31.84 ms per inference step and an average classification accuracy of 98.90%. Our experiments demonstrate the network’s robustness, maintaining high accuracy even with previously unseen objects.

With the advancement of tactile sensors, researchers increasingly integrate tactile perception into robotics, but only for tasks such as object reconstruction, classification, recognition, and grasp state assessment. In this paper, we rethink the relationship between visual and tactile perception and propose a novel robotic grasping detection method based on visual–tactile perception. Initially, we construct a visual–tactile dataset containing the grasp stability for each potential grasping position. Next, we introduce a novel Grasp Stability Prediction Module (GSPM) to generate a grasp stability probability map, providing prior knowledge regarding grasp stability to the grasp detection network for each possible grasp position. Finally, the map is multiplied element-wise with the corresponding colored image and inputted into the grasp detection network. Experimental results demonstrate that our novel visual–tactile fusion method significantly enhances robotic grasping detection accuracy.

Robotic grasping is a core capability for embodied AI agents, but traditional methods focus solely on stable object holding and fail to account for object functional semantics, which limits their effectiveness in unstructured, human-centric scenarios. This gap highlights the need to integrate “affordance" (object properties indicating usable interactions) into grasping systems. To address this, we proposes an MLLM-driven affordance-guided grasping framework. Unlike existing approaches that rely on large annotated datasets or additional memory modules, the framework leverages the inherent world knowledge and commonsense reasoning of multimodal large language models (MLLMs) to infer object affordances. It comprises three key modules: 1) an MLLM-based Affordance Reasoning Module that analyzes visual-language inputs to identify graspable and avoidable regions, generating structured outputs with confidence/risk scores and normalized coordinates; 2) a Segmentation with Affordance Guidance Module that converts MLLM-inferred affordance cues into binary masks, optimized via metrics for coverage, avoidance penalty, and core-point consistency; 3) a Grasp Strategy Generation Module that refines initial grasp candidates using affordance masks to output optimal grasp poses. Experiments are conducted on the real-world robotic platform. Results show the framework effectively identifies functional regions, validating its ability to enable task-driven, safe grasping. This work advances robotic adaptability in real-world contexts and provides a foundation for extending affordance-based manipulation to more complex scenarios.