Peg-in-hole assembly, a crucial component of robotic automation in manufacturing, continues to pose challenges due to its strict tolerance requirements. To date, most conventional peg-in-hole assembly algorithms have been validated only within simulated environments or under limited observational scenarios. In this paper, an environment-agnostic coarse-to-fine visual servoing (EA-CTFVS) assembly algorithm is proposed. Firstly, to solve the frequent issue of visual blindness during visual servoing, a bottleneck pose is proposed to be used as the desired pose for the visual servoing. Secondly, to achieve accurate assembly, a coarse-to-fine framework is constructed, in which a rough pose is given by the coarse controller to remove large initial alignment errors. For the fine controller, a twin network-based fine controller is provided to improve assembly accuracy. Furthermore, EA-CTFVS utilizes the Oriented Bounding Box (OBB) of objects as the input for visual servoing, which guarantees the system’s ability to operate effectively in diverse and complex scenes. The proposed EA-CTFVS achieves a successful assembly rate of 0.92/0.89 for initial alignment errors of 15/30 cm and 0.6 mm tolerance in real-world D-sub plug assembly tasks under complex scenarios.

Grasping specific objects in complex and irregularly stacked scenes is still challenging for robotics. Because the robot is not only required to identify the object's grasping posture but also needs to reason the manipulation relationship between the objects. In this paper, we propose a manipulation relationship reasoning network with a multi-scale feature aggregation (MSFA) mechanism for robot grasping tasks. MSFA aggregates high-level semantic information and low-level spatial information in a cross-scale connection way to improve the generalization ability of the model. Furthermore, to improve the accuracy, we propose to use intersection features with rich location priors for manipulation relationship reasoning. Experiments are validated in VMRD datasets and real environments, respectively. The experimental results demonstrate that our proposed method can accurately predict the manipulation relationship between objects in the scene of multi-object stacking. Compared with previous methods, it significantly improves reasoning speed and accuracy.

In stacked object scenes, effectively perceiving the positional relationships between objects is crucial for enabling robots to grasp safely. However, most existing methods rely on single-scale features with limited representational capability, ignoring the benefits of multi-scale features. In this paper, we propose a multi-task grasp detection model for stacked object scenes based on multi-scale feature fusion. We design a Multi-Scale Feature Aggregation (MSFA) module to fuse features, enhancing the feature maps' ability to represent objects of varying sizes. Additionally, we leverage features from multiple views to predict object positional relationships. In particular, the local Intersection Region Feature (IRF) significantly enhances the model's capability to identify positional relationships between objects while improving computational efficiency. Experimental results on the Visual Manipulation Relationship Dataset (VMRD) demonstrate that our model surpasses current state-of-the-art methods. Furthermore, we conducted grasping experiments in real-world stacking scenarios, which verified the model's effectiveness and generalization capability.