Object tracking is one of the most important problems in computer vision applications such as robotics, autonomous driving, and pedestrian movement. There has been a significant development in camera hardware where researchers are experimenting with the fusion of different sensors and developing image processing algorithms to track objects. Image processing and deep learning methods have significantly progressed in the last few decades. Different data association methods accompanied by image processing and deep learning are becoming crucial in object tracking tasks. The data requirement for deep learning methods has led to different public datasets that allow researchers to benchmark their methods. While there has been an improvement in object tracking methods, technology, and the availability of annotated object tracking datasets, there is still scope for improvement. This review contributes by systemically identifying different sensor equipment, datasets, methods, and applications, providing a taxonomy about the literature and the strengths and limitations of different approaches, thereby providing guidelines for selecting equipment, methods, and applications. Research questions and future scope to address the unresolved issues in the object tracking field are also presented with research direction guidelines.

Robot hand-to-eye calibration is a necessary process for a robot arm to perceive and interact with its environment. Past approaches required collecting multiple images using a calibration board placed at different locations relative to the robot. When the robot or camera is displaced from its calibrated position, hand–eye calibration must be redone using the same tedious process. In this research, we developed a novel method that uses a semi-automatic process to perform hand-to-eye calibration with a stereo camera, generating a transformation matrix from the world to the camera coordinate frame from a single image. We use a robot-pointer tool attached to the robot’s end-effector to manually establish a relationship between the world and the robot coordinate frame. Then, we establish the relationship between the camera and the robot using a transformation matrix that maps points observed in the stereo image frame from two-dimensional space to the robot’s three-dimensional coordinate frame. Our analysis of the stereo calibration showed a reprojection error of 0.26 pixels. An evaluation metric was developed to test the camera-to-robot transformation matrix, and the experimental results showed median root mean square errors of less than 1 mm in the x and y directions and less than 2 mm in the z directions in the robot coordinate frame. The results show that, with this work, we contribute a hand-to-eye calibration method that uses three non-collinear points in a single stereo image to map camera-to-robot coordinate-frame transformations.

Robot bin picking is one of the most important steps in industrial automation. The robot bin-picking problem involves detecting objects in a bin using a perception system such as a camera, identifying the object pose, and planning a path to pick the objects efficiently. In this research, we provide a novel approach to detecting objects in a bin using a segmentation method based on deep learning. Firstly, we build and calibrate a stereo camera rig using two USB3 Vision industrial cameras and the GenICam (Generic Interface for Cameras) programming interface. Then, we detect the objects and use a segmentation method, along with post-processing steps, followed by pose estimation using the principal components of the object silhouette. Finally, we perform depth detection on the detected objects. Experimental results showed that our developed method can correctly detect objects, estimate object pose, and accurately measure object depth, with a mean error of 5 mm.

Robotic welding has been widely used to improve manufacturing efficiency. However, for small batches or one-off jobs, the time and effort required for robotic welding may be economically prohibitive. An intuitive ‘mimicking’-like robotic welding approach can help non-robotics experts instruct the robot to perform the welding operations by letting them demonstrate the welding path via hand gestures without requiring any programming. In this paper, we present a human–robot interaction (HRI)-based weld seam detection for such a robotic welding application. Our approach consists of a vision-based hand detection and tracking method for the user to demonstrate the welding path to the robot’s vision system. We then isolated the welding seam lines and developed a search algorithm to identify the welding paths for the robot. The continuous weld seam path is observed within the bounds of the seam edge in the image and is projected to the robot coordinate space. The seam detection is thoroughly tested with a real-world application on the UR10e robot. The evaluation has revealed an accuracy of 1 pixel in the image plane, equivalent to 1 mm in physical space in our setup.