This letter addresses the problem of contact-based manipulation of deformable linear objects (DLOs) towards desired shapes with a dual-arm robotic system. To alleviate the burden of high-dimensional continuous state-action spaces, we model DLOs as kinematic multibody systems via our proposed keypoint encoding network. This novel encoding is trained on a synthetic labeled image dataset without requiring any manual annotations and can be directly transferred to real manipulation scenarios.Our goal-conditioned policy efficiently rearranges the configuration of the DLO based on the keypoints. The proposed hierarchical action framework tackles the manipulation problem in a coarse-to-fine manner (with high-level task planning and low-level motion control) by leveraging two action primitives. The identification of deformation properties is bypassed since the algorithm replans its motion after each bimanual execution. The conducted experimental results reveal that our method achieves high performance in state representation and shaping manipulation of the DLO under environmental constraints.

Soft object manipulation has recently gained popularity within the robotics community due to its potential applications in many economically important areas. Although great progress has been recently achieved in these types of tasks, most state-of-the-art methods are case-specific; They can only be used to perform a single deformation task (e.g., bending), as their shape representation algorithms typically rely on “hard-coded” features. In this letter, we present LaSeSOM, a new feedback latent representation framework for semantic soft object manipulation. Our new method introduces internal latent representation layers between low-level geometric feature extraction and high-level semantic shape analysis; This allows the identification of each compressed semantic function and the formation of a valid shape classifier from different feature extraction levels. The proposed latent framework makes soft object representation more generic (independent from the object's geometry and its mechanical properties) and scalable (it can work with 1D/2D/3D tasks). Its high-level semantic layer enables to perform (quasi) shape planning tasks with soft objects, a valuable and underexplored capability in many soft manipulation tasks. To validate this new methodology, we report a detailed experimental study with robotic manipulators.

Garment folding is a ubiquitous domestic task that is difficult to automate due to the highly deformable nature of fabrics. In this article, we propose a novel method of learning from demonstrations that enables robots to autonomously manipulate an assistive tool to fold garments. In contrast to traditional methods (that rely on low-level pixel features), our proposed solution uses a dense visual descriptor to encode the demonstration into a high-level hand-object graph (HoG) that allows to efficiently represent the interactions between the manipulated tool and robots. With that, we leverage graph neural network to autonomously learn the forward dynamics model from HoGs, then, given only a single demonstration, the imitation policy is optimized with a model predictive controller to accomplish the folding task. To validate the proposed approach, we conducted a detailed experimental study on a robotic platform instrumented with vision sensors and a custom-made end-effector that interacts with the folding board.

The automatic inspection of surface defects is an essential task for quality control in the computers, communications, and consumer electronics (3C) industry. Traditional inspection mechanisms (viz. line-scan sensors) have a limited field of view, thus, prompting the necessity for a multifaceted robotic inspection system capable of comprehensive scanning. Optimally selecting the robot’s viewpoints and planning a path is regarded as coverage path planning (CPP), a problem that enables inspecting the object’s complete surface while reducing the scanning time and avoiding misdetection of defects. In this paper, we present a new approach for robotic line scanners to detect surface defects of 3C free-form objects automatically. A two-stage region segmentation method defines the local scanning based on the random sample consensus (RANSAC) and K-means clustering to improve the inspection coverage. The proposed method also consists of an adaptive region-of-interest (ROI) algorithm to define the local scanning paths. Besides, a Particle Swarm Optimization (PSO)-based method is used for global inspection path generation to minimize the inspection time. The developed method is validated by simulation-based and experimental studies on various free-form workpieces, and its performance is compared with that of two state-of-the-art solutions. The reported results demonstrate the feasibility and effectiveness of our proposed method.

In this paper, we present a novel sensor-based system to perform defect inspection tasks automatically over free-form specular surfaces. The inspection procedure is performed by a robotic manipulator equipped with a line scanner system. Taking the geometric and optical parameters into consideration, our algorithm computes a flexible scanning path. Based on a mesh model, the system segments the convex surface into areas with similar curvatures, then adaptively adjusts the line sensor’s scanning range to ensure the complete coverage of the surface. We consider the scanning efficiency through constrained optimization. To align the computed scanning path with the specular object, a real-time workpiece localization algorithm in a coarse-to-fine manner is developed. Our control framework synchronizes the motion of the manipulator and the acquisition of the line scanner for robotic inspection. We also provide an image processing pipeline for automatic defect detection. To validate the proposed formal methodology, we report a detailed experimental study with the developed robotic inspection prototype.

In this article, we propose a new action planning approach to automatically pack long linear elastic objects into common-size boxes with a bimanual robotic system. For that, we developed a hybrid geometric model to handle large-scale occlusions combining an online vision-based method and an offline reference template. Then, a reference point generator is introduced to automatically plan the reference poses for the predesigned action primitives. Finally, an action planner integrates these components enabling the execution of high-level behaviors and the accomplishment of packing manipulation tasks. To validate the proposed approach, we conducted a detailed experimental study with multiple types and lengths of objects and packing boxes.