We propose a CNN based visual servoing scheme for precise positioning of an eye-to-hand manipulator in which the control input of a robot is calculated directly from images by a neural network. In this paper, we propose Difference of Encoded Features driven Interaction matrix Network (DEFINet), a new convolutional neural network (CNN), for eye-to-hand visual servoing. DEFINet estimates a relative pose between desired and current end-effector from desired and current images captured by an eye-to-hand camera. DEFINet includes two branches of the same CNN that share weights and encode target and current images, which is inspired by the architecture of Siamese network. Regression of the relative pose from the difference of the encoded target and current image features leads to a high positioning accuracy of visual servoing using DEFINet. The training dataset is generated from sample data collected by operating a manipulator randomly in task space. The performance of the proposed visual servoing is evaluated through numerical simulation and experiments using a six-DOF industrial manipulator in a real environment. Both simulation and experimental results show the effectiveness of the proposed method.

Pose estimation that locates objects in a bin is necessary for a robotic bin picking system. Although many algorithms have shown high performance in pose estimation, most algorithms estimate the poses of objects regardless of their occlusion. This can reduce the success rate in picking up the object. To resolve this issue, we propose a novel pipeline that estimates a pose only for occlusion-free objects based on point pair feature-based pose estimation with multiple edge appearance model (PPF-MEAM). The proposed method detects occlusion-free objects in the 2D image captured by a camera with a convolutional neural network framework. Next, corresponding point clouds of occlusion-free objects need to be extracted by using their locations in the 2D image. we propose a robust extraction method that finds the 3D points corresponding to image pixels in the 2D image to reduce the effect of the calibration errors between the camera and 3D sensor. The point cloud of the occlusion-free objects is finally input into a pipeline of PPF-MEAM to estimate the pose of the object. The experiment results prove that the proposed method is about 50% faster 30% higher in terms of pose estimation success rate compared with the original PPF. Moreover, it increases the success rate of picking tasks compared with the original PPF-MEAM.

Automating the sewing process presents significant challenges due to the inherent softness of fabrics and the limited control capabilities of sewing systems. To realize sewing automation, we propose a time-scaling modeling and control architecture of the robotic sewing system. By using the time-scaling modeling, the nonholonomic kinematics of the sewing process of the industrial sewing machine is linearized precisely. Based on this model, a two-layer real-time control architecture is proposed. The upper layer controls the sewn seam line trajectory using the model-based feedback control implemented in the time-scaling domain, while the lower layer controls the manipulator and the sewing machine using geometric-based trajectory generation and coordinated motion control of the robot and the sewing system in the time domain. The experimental results demonstrate that the sewing trajectories exponentially converge to the desired trajectories without overshooting under different initial conditions and sewing speeds. Besides, the same sewing trajectories under different sewing speeds are obtained for a given stitch size. The sewing results show the good performance and application potential of the proposed robotic sewing system.

This paper proposes a fixture-free 2D sewing system using a dual-arm manipulator, i.e., the seam lines of the top and bottom fabric parts are the same. The proposed 2D sewing system sews two stacked fabric parts together along a desired seam line printed on the top fabric part without the use of a fixture. In the proposed system, the set of aligned and stacked fabric parts is held by the end-effectors of the dual-arm manipulator in coordination. The dual-arm manipulator controls the motion of the fabric parts on the flat sewing table stitch by stitch in coordination, while keeping the manipulated fabric parts flat using the internal force applied to the set of fabric parts. A novel vision-based seam line tracking control is proposed to control the motion of the set of fabric parts along the printed seam line on the top fabric part. The convergence of the tracking error is analyzed for sewing along both straight and curved seam lines and is shown to be specified by the control parameters. Sewing experiments show that the tracking error converges to zero as analyzed. The sewing experiments also show that the newly proposed trajectory generation method, which synchronizes the coordinated motion of the manipulators and the motion of the sewing needle, is essential for achieving accurate sewing. Note to Practitioners —Most semi-automatic sewing machines and pattern sewers on the market use fixtures to handle the stacked fabric parts. They require the user to customize the fixture depending on the shape, size, and material of the fabric parts to be sewn together. Users are required to redesign/reconfigure the fixture to sew different fabric parts. Our robotic sewing system is based on the concept of fixture-free sewing, i.e., the pose of the set of stacked fabric parts is controlled by the end-effectors without using the fixture. The internal force applied to the fabric parts by the end-effectors is used to keep the fabric parts flat, and the position of the fabric parts is controlled by the motions of the end-effectors in coordination with the proposed vision-based seam line tracking control. The proposed robotic sewing system provides practitioners with a new approach to fixture-free automatic sewing of fabric parts.

In this article, we propose a Passive Actuator-Less Gripper (PALGRIP) for picking a piece of fabric from a stack of fabric parts and placing the picked fabric part. The picking of a piece of fabric from a stack is a simple but difficult process to automate. The proposed gripper can pick a piece of fabric from the stack by simply pressing the fingertips of the gripper against the stack. The fingers are closed and opened by the relative motion between the fingers and the housing of the gripper. The grasping motion of the gripper is generated by two mechanisms: a passive pinching mechanism and a self-locking mechanism. These mechanisms allow the fingers to perform opening and closing movements and to maintain the fingers in either open or closed state. The kinematics of the mechanisms are analyzed to design the gripper. The relation between the movement of the fingers and the force required to operate the gripper is also investigated through static force analysis and the experiment. Finally, experiments using PALGRIP are conducted, and the experimental results illustrate how the pick-and-place operations are carried out using the prototype of PALGRIP. The proposed gripper allows the robot to automate fabric pick-and-place operations easily by attaching it to the robot's endpoint.

Seams are information-rich components of garments. The presence of different types of seams and their combinations helps to select grasping points for garment handling. In this paper, we propose a new Seam-Informed Strategy (SIS) for finding actions for handling a garment, such as grasping and unfolding a T-shirt. Candidates for a pair of grasping points for a dual-arm manipulator system are extracted using the proposed Seam Feature Extraction Method (SFEM). A pair of grasping points for the robot system is selected by the proposed Decision Matrix Iteration Method (DMIM). The decision matrix is first computed by multiple human demonstrations and updated by the robot execution results to improve the grasping and unfolding performance of the robot. Note that the proposed scheme is trained on real data without relying on simulation. Experimental results demonstrate the effectiveness and generalization ability of the proposed strategy. The project home page is available at https://github.com/lancexz/sis.