Mobile robots have an important role in material handling in manufacturing and can be used for a variety of automated tasks. The accuracy of the robot’s moving trajectory has become a key issue affecting its work efficiency. This paper presents a method for optimizing the trajectory of the mobile robot based on the digital twin of the robot. The digital twin of the mobile robot is created by Unity, and the trajectory of the mobile robot is trained in the virtual environment and applied to the physical space. The simulation training in the virtual environment provides schemes for the actual movement of the robot. Based on the actual movement data returned by the physical robot, the preset trajectory of the virtual robot is dynamically adjusted, which in turn enables the correction of the movement trajectory of the physical robot. The contribution of this work is the use of genetic algorithms for path planning of robots, which enables trajectory optimization of mobile robots by reducing the error in the movement trajectory of physical robots through the interaction of virtual and real data. It provides a method to map learning in the virtual domain to the physical robot.

This paper presents a fiber Bragg grating (FBG)-based six-dimensional (6-D) force/torque (F/T) sensor that can be mounted on robot joints for the detection of comprehensive force/torque information. The sensor is designed based on a new concept of multilayer measurement in which the elastic structure is divided into four layers. A single optical fiber inscribed with 12 FBGs is symmetrically attached to the elastic structure to sense force/torque-induced strains by means of differential measurements. Because of the mechanical decoupling function of the structure and the reasonable arrangement of the FBG sensing elements, the designed sensor possesses a great advantage of low cross coupling compared with existing 6-D force/torque sensors, which simplifies the calibration and maintenance of the sensor. Calibration experiments demonstrate high resolutions of 7.8 N mm, 7.2 N mm, 15.8 N mm, 0.0067 N, 0.0066 N and 0.0825 N within the range of ±1.56 Nm, ±1.56 Nm, ±1.56 Nm, ±10 N, ±10 N and ±25 N for the detection of Mx, My, Mz, Fx, Fy and Fz, respectively. Dynamic performances of the designed sensor have been evaluated in comparison with a commercial force/torque sensor.

In order to improve the recognition rate of hand gestures a new interactive image segmentation method for hand gesture recognition is presented, and popular methods, e.g., Graph cut, Random walker, Interactive image segmentation using geodesic star convexity, are studied in this article. The Gaussian Mixture Model was employed for image modelling and the iteration of Expectation Maximum algorithm learns the parameters of Gaussian Mixture Model. We apply a Gibbs random field to the image segmentation and minimize the Gibbs Energy using Min-cut theorem to find the optimal segmentation. The segmentation result of our method is tested on an image dataset and compared with other methods by estimating the region accuracy and boundary accuracy. Finally five kinds of hand gestures in different backgrounds are tested on our experimental platform, and the sparse representation algorithm is used, proving that the segmentation of hand gesture images helps to improve the recognition accuracy.

As a key technology for the non-invasive human-machine interface that has received much attention in the industry and academia, surface EMG (sEMG) signals display great potential and advantages in the field of human-machine collaboration. Currently, gesture recognition based on sEMG signals suffers from inadequate feature extraction, difficulty in distinguishing similar gestures, and low accuracy of multi-gesture recognition. To solve these problems a new sEMG gesture recognition network called Multi-stream Convolutional Block Attention Module-Gate Recurrent Unit (MCBAM-GRU) is proposed, which is based on sEMG signals. The network is a multi-stream attention network formed by embedding a GRU module based on CBAM. Fusing sEMG and ACC signals further improves the accuracy of gesture action recognition. The experimental results show that the proposed method obtains excellent performance on dataset collected in this paper with the recognition accuracies of 94.1%, achieving advanced performance with accuracy of 89.7% on the Ninapro DB1 dataset. The system has high accuracy in classifying 52 kinds of different gestures, and the delay is less than 300 ms, showing excellent performance in terms of real-time human-computer interaction and flexibility of manipulator control.

Camera calibration is a crucial problem in many applications, such as 3D reconstruction, structure from motion, object tracking and face alignment. Numerous methods have been proposed to solve the above problem with good performance in the last few decades. However, few methods are targeted at joint calibration of multi-sensors (more than four devices), which normally is a practical issue in the real-time systems. In this paper, we propose a novel method and a corresponding workflow framework to simultaneously calibrate relative poses of a Kinect and three external cameras. By optimizing the final cost function and adding corresponding weights to the external cameras in different locations, an effective joint calibration of multiple devices is constructed. Furthermore, the method is tested in a practical platform, and experiment results show that the proposed joint calibration method can achieve a satisfactory performance in a project real-time system and its accuracy is higher than the manufacturer’s calibration.

There are a variety of objects, random postures and multiple objects stacked in a disorganized manner in unstructured home applications, which leads to the object grasping posture estimation and grasping planning based on machine vision become very complicated. This paper proposes a method of cluttering pose detection based on convolutional neural network with multiple self-powered sensors information. Firstly, a search strategy for candidate grasping poses based on the 3D point cloud is proposed, and the single-channel grasping image dataset representing grasping posture in this paper is established by using Bigbird dataset. Secondly, the ResNet is constructed to rank and filter the single channel captured images representing the captured bit pose. It is also compared with three mainstream classification networks, Inception V2, VGG-A and LetNet, and the perception analysis function and the execution planning function are developed under ROS. The effective grasping of the manipulator in the scene of scattered piles is realized based on the detection results of grasping position and combined with the information of multiple self-powered sensors, and the ResNet network is compared with other three classification networks. In a scattered and stacked environment of objects, the results of experiment show that the method based on ResNet network is superior to the other three networks, and the average success rate of grasping pose detection based on ResNet, InceptionV2, VGGA and LetNet networks is 90.67%, 82.67%, 86.67% and 87.33% respectively, which verifies the effectiveness and superiority of the deep learn-based grasping pose detection model proposed in this paper.