Monitoring the assembly process is a challenge in the manual assembly of mass customization production, in which the operator needs to change the assembly process according to different products. If an assembly error is not immediately detected during the assembly process of a product, it may lead to errors and loss of time and money in the subsequent assembly process, and will affect product quality. To monitor assembly process, this paper explored two methods: recognizing assembly action and recognizing parts from complicated assembled products. In assembly action recognition, an improved three-dimensional convolutional neural network (3D CNN) model with batch normalization is proposed to detect a missing assembly action. In parts recognition, a fully convolutional network (FCN) is employed to segment, recognize different parts from complicated assembled products to check the assembly sequence for missing or misaligned parts. An assembly actions data set and an assembly segmentation data set are created. The experimental results of assembly action recognition show that the 3D CNN model with batch normalization reduces computational complexity, improves training speed and speeds up the convergence of the model, while maintaining accuracy. Experimental results of FCN show that FCN-2S provides a higher pixel recognition accuracy than other FCNs.

Locating and identifying the components mounted on a printed circuit board (PCB) based on machine vision is an important and challenging problem for automated PCB inspection and automated PCB recycling. In this paper, we propose a PCB semantic segmentation method based on depth images that segments and recognizes components in the PCB through pixel classification. The image training set for the PCB was automatically synthesized with graphic rendering. Based on a series of concentric circles centered at the given depth pixel, we extracted the depth difference features from the depth images in the training set to train a random forest pixel classifier. By using the constructed random forest pixel classifier, we performed semantic segmentation for the PCB to segment and recognize components in the PCB through pixel classification. Experiments on both synthetic and real test sets were conducted to verify the effectiveness of the proposed method. The experimental results demonstrate that our method can segment and recognize most of the components from a real depth image of the PCB. Our method is immune to illumination changes and can be implemented in parallel on a GPU.

In the process of product assembly, parts need to be assembled in a given assembly sequence. Failure to detect the correctness of the newly assembled parts in time can affect the quality and assembly efficiency of products. For effective detection of newly assembled parts in the assembly process from different viewing angles, this study applies scene change detection to mechanical assembly monitoring for the first time and proposes a mechanical assembly monitoring method based on depth image multiview change detection. This method includes a semantic fusion network and an attention-based feature extraction network (AFE Net) for multiview change detection. To make this method suitable for the change detection of the mechanical assembly, this study involves the following innovations. 1) Considering the assembly parts with a single color, symmetry, and no texture, this study employs depth images as the input of the semantic fusion network. Subsequently, the semantic segmentation network is used to segment the parts on the depth images for the generation of color semantic segmentation images. Then, the semantic segmentation images and depth images are merged and input into the multiview change detection network. 2) In the multiview change detection network, an attention-based feature extraction module is designed to rapidly focus on the key information of the current task from a large amount of input information and improve the processing efficiency and accuracy of the task. Furthermore, through up-sampling, the size of the feature map is unified, and high-dimensional semantic information and low-dimensional spatial information are merged, which effectively increases the amount of feature information. 3) To verify the effectiveness of the multiview change detection of the assembly, this study establishes a multiview assembly process dataset and evaluates the proposed method using this dataset. The results show that, compared with other change detection networks, the comprehensive index F1 of the method based on the abovementioned dataset reaches an optimal value of 96.9% while consuming less time and with clearer boundary processing. Overall, the proposed network structure is suitable for the change detection of the mechanical assembly and can also be applied to multiview monitoring in product assembly.

The precise application of tightening torque is one of the important measures to ensure accurate bolt connection and improvement in product assembly quality. Currently, due to the limited assembly space and efficiency, a wrench without the function of torque measurement is still an extensively used assembly tool. Therefore, wrench torque monitoring is one of the urgent problems that needs to be solved. This study proposes a multi-segmentation parallel convolution neural network (MSP-CNN) model for estimating assembly torque using surface electromyography (sEMG) signals, which is a method of torque monitoring through classification methods. The MSP-CNN model contains two independent CNN models with different or offset torque granularities, and their outputs are fused to obtain a finer classification granularity, thus improving the accuracy of torque estimation. First, a bolt tightening test bench is established to collect sEMG signals and tightening torque signals generated when the operator tightens various bolts using a wrench. Second, the sEMG and torque signals are preprocessed to generate the sEMG signal graphs. The range of the torque transducer is divided into several equal subdivision ranges according to different or offset granularities, and each subdivision range is used as a torque label for each torque signal. Then, the training set, verification set, and test set are established for torque monitoring to train the MSP-CNN model. The effects of different signal preprocessing methods, torque subdivision granularities, and pooling methods on the recognition accuracy and torque monitoring accuracy of a single CNN network are compared experimentally. The results show that compared to maximum pooling, average pooling can improve the accuracy of CNN torque classification and recognition. Moreover, the MSP-CNN model can improve the accuracy of torque monitoring as well as solve the problems of non-convergence and slow convergence of independent CNN network models.

Human hands engage in interactive activities in many practical working scenarios, among which the interactions between human hands and objects are the most common. Tracking the movement of the human hand during hand-object interactions is an important research task that is also challenging due to the high-dimensionality and occlusions. In this paper, we track hand-object interactions from depth observations with a model-based method. To overcome the difficulties of optimum searching in the hand-object high-dimensional space, we propose a new algorithm — collaborative differential evolution filtering (CoDEF) — for tracking hand-object interactions. The proposed CoDEF algorithm integrates the differential evolution (DE) algorithm into a particle filtering (PF) framework to accelerate the convergence of particles. Particles are driven to the regions with a high probability by optimizing the matching error under the current observation with DE. To decompose the state space and decrease the complexity of optimum searching, CoDEF tracks the movement of the hand and object by using two collaborative trackers. Based on the proposed CoDEF algorithm, we develop a model-based tracking system with 3D graphic techniques. According to the experimental results, the proposed CoDEF algorithm can achieve robust tracking of hand-object interactions using fewer particles.