The widespread installation of inertial sensors in smartphones and other wearable devices provides a valuable opportunity to identify people by analyzing their gait patterns, for either cooperative or non-cooperative circumstances. However, it is still a challenging task to reliably extract discriminative features for gait recognition with noisy and complex data sequences collected from casually worn wearable devices like smartphones. To cope with this problem, we propose a novel image-based gait recognition approach using the Convolutional Neural Network (CNN) without the need to manually extract discriminative features. The CNN’s input image, which is encoded straightforwardly from the inertial sensor data sequences, is called Angle Embedded Gait Dynamic Image (AE-GDI). AE-GDI is a new two-dimensional representation of gait dynamics, which is invariant to rotation and translation. The performance of the proposed approach in gait authentication and gait labeling is evaluated using two datasets: (1) the McGill University dataset, which is collected under realistic conditions; and (2) the Osaka University dataset with the largest number of subjects. Experimental results show that the proposed approach achieves competitive recognition accuracy over existing approaches and provides an effective parametric solution for identification among a large number of subjects by gait patterns.

Nowadays, robots have been widely used in various manufacturing assemblies. However, in the mobile phone flexible printed circuit (FPC) assembly task, robots are still far from trustworthy because the connector and receiver of FPC are very small in size, and detection for them is influenced by the partial occlusion of the robot end manipulator in operation. This makes it difficult to measure the spatial relationship between the connector and receiver in robot picking and buckling. To this end, this work proposes a Digit Twin (DT)-driven measurement method for partial occlusion FPC assembly. The contributions include: (1) we introduce a homography-based strategy to calibrate the FPC placement panel and its virtual placement panel in digit twin. It makes the partial occlusion views in physical space visible in digital twin within tolerable errors. This helps to improve the measurement by eliminating the observation occlusion which originally exits in physical space; (2) we propose a DT-driven reward function measured by the distance and rotation difference between the FPC connector hold by the manipulator and its ideal position in the digital twin. This makes the deep architecture can learn the actions guided by the measurements of the FPC positions in physical space and digital twin simultaneously. The experiments are carried out on a practical assembly platform for the mobile phone Redmi Note 11. With the number of failed trails limited, the proposed network obviously outperforms the pure vision-based measurement and the network without DT support. The proposed method provides a primitive but novel measurement strategy for mobile phone FPC assembly. It can be applied to the real mobile phone assembly lines to reduce the labor burden in the near future.

The utilization of robots in computer, communication, and consumer electronics (3C) assembly has the potential to significantly reduce labor costs and enhance assembly efficiency. However, many typical scenarios in 3C assembly, such as the assembly of flexible printed circuits (FPCs), involve complex manipulations with long-horizon steps and high-precision requirements that cannot be effectively accomplished through manual programming or conventional skill-learning methods. To address this challenge, this article proposes a learning-based framework for the acquisition of complex 3C assembly skills assisted by a multimodal digital-twin environment. First, we construct a fully equivalent digital-twin environment based on the real-world counterpart, equipped with visual, tactile force, and proprioception information, and then collect multimodal demonstration data using virtual reality (VR) devices. Next, we construct a skill knowledge base through multimodal skill parsing of demonstration data, resulting in primitive policy sequences for achieving 3C assembly tasks. Finally, we train primitive policies via a combination of curriculum learning, residual reinforcement learning, and domain randomization methods and transfer the learned skill from the digital-twin environment to the real-world environment. The experiments are conducted to verify the effectiveness of our proposed method.

In the industrial assembly of flexible printed circuits (FPCs), the FPC connectors and receivers are required to connect successfully with the least possible buckling times. However, precise and autonomous positioning of the FPC connectors is still a great challenge, as they are very small and visually blocked during assembly. This article proposes a strategy for the industrial assembly of FPC by: 1) unlike the traditional vision based methods, the task of FPC position identification in this work is considered as a haptic signals alignment problem. We mount a haptic sensor at the top end of the robotic arm manipulator and record various mismatched haptic signals around the ideal FPC connector position; 2) converting both the correct and incorrect haptic signals into images; and 3) we introduce a novel deep Siamese-network (DSN) structure to align the incorrect haptic images to the ground truth. The experiments are carried out on a practical FPC assembly platform for the Redmi Note 11 mobile phone. It is found that the haptic images combined with DSN can significantly improve the FPC location accuracy. When the buckle times are limited to 10, 5, and 1, the proposed method outperforms the state-of-the-art methods in a realistic FPC assembly platform.

When the camera moves quickly and the image is blurred or the texture in the scene is missing, the Simultaneous Localization and Mapping (SLAM) algorithm based on point feature experiences difficulty tracking enough effective feature points, and the positioning accuracy and robustness are poor, and even may not work properly. For this problem, we propose a monocular visual odometry algorithm based on the point and line features and combining IMU measurement data. Based on this, an environmental-feature map with geometric information is constructed, and the IMU measurement data is incorporated to provide prior and scale information for the visual localization algorithm. Then, the initial pose estimation is obtained based on the motion estimation of the sparse image alignment, and the feature alignment is further performed to obtain the sub-pixel level feature correlation. Finally, more accurate poses and 3D landmarks are obtained by minimizing the re-projection errors of local map points and lines. The experimental results on EuRoC public datasets show that the proposed algorithm outperforms the Open Keyframe-based Visual-Inertial SLAM (OKVIS-mono) algorithm and Oriented FAST and Rotated BRIEF-SLAM (ORB-SLAM) algorithm, which demonstrates the accuracy and speed of the algorithm.