The mismatched-pins inspection of complex aviation connector is a critical process to ensure the correct wiring harness assembly, of which the existing manual operation is error-prone and time-consuming. Aiming to fill this gap, this work proposes an augmented reality (AR)-assisted deep learning-based approach to tackle three major challenges in aviation connector inspection, including small pins detection, multi-pins sequencing, and mismatched pins visualization. Firstly, the proposed spatial-attention pyramid network approach extracts image features in multilayers and searches for their spatial relationships among images. Secondly, based on the cluster-generation sequencing algorithm, these detected pins are clustered into annuluses of expected layers and numbered according to their polar angles. Lastly, AR glass as the inspection visualization platform, highlights mismatched pins in the augmented interface to warn operators automatically. Compared with other existing methodologies, the experimental result shows that the proposed approach can achieve better performance accuracy, and support operators inspection process efficiently.

In mobile augmented/virtual reality (AR/VR), real-time 6-Degree of Freedom (DoF) motion tracking is essential for the registration between virtual scenes and the real world. However, due to the limited computational capacity of mobile terminals today, the latency between consecutive arriving poses would damage the user experience in mobile AR/VR. Thus, a visual-inertial based real-time motion tracking for mobile AR/VR is proposed in this paper. By means of high frequency and passive outputs from the inertial sensor, the real-time performance of arriving poses for mobile AR/VR is achieved. In addition, to alleviate the jitter phenomenon during the visual-inertial fusion, an adaptive filter framework is established to cope with different motion situations automatically, enabling the real-time 6-DoF motion tracking by balancing the jitter and latency. Besides, the robustness of the traditional visual-only based motion tracking is enhanced, giving rise to a better mobile AR/VR performance when motion blur is encountered. Finally, experiments are carried out to demonstrate the proposed method, and the results show that this work is capable of providing a smooth and robust 6-DoF motion tracking for mobile AR/VR in real-time.

In recent years, deep learning (DL)-based fault diagnosis methods have demonstrated significant success in various domains due to their high accuracy. Similar to other data-driven techniques, DL models are application-specific and strongly depend on the data which they are developed upon. Different DL models need to be designed for different tasks. In addition, manual-tuning of the DL structures and associated parameters as part of the model design is a trial-and-error optimization process, which is time-consuming and sensitive to changes. To address these limitations, this article presents a reinforcement learning (RL) and neural architecture search (NAS)-based automatic modeling framework (RL-NAS) for fault diagnosis of machinery. The RL-NAS method can adaptively and automatically design high-accuracy network models according to the intended diagnosis tasks. A weight-sharing mechanism has been developed to alleviate the bottleneck of NAS where the computation time increases exponentially with the expansion of the search space and the deepening of the model structure. To evaluate its effectiveness, efficiency, and reproducibility, the proposed framework is validated on five datasets from different institutes, spanning the applications of rolling element bearings, gears, and ball screws. The results show that for all five applications, the RL-NAS method successfully searched out diagnostics models with over 97% accuracy within a short time, from 40 s to 30 min depending on the complexity of the diagnosis task, showing good performance of the proposed method.

Monitoring surface quality during machining has considerable practical significance for the performance of high-value products, particularly for their assembly interfaces. Surface roughness is the most important metric of surface quality. Currently, the research on online surface roughness prediction has several limitations. The effect of tool wear variation on surface roughness is seldom considered in machining. In addition, the deterioration trend of surface roughness and tool wear differs under variable cutting parameters. The prediction models trained under one set of cutting parameters fail when cutting parameters change. Accordingly, to timely monitor the surface quality of assembly interfaces of high-value products, this paper proposes a surface roughness prediction method that considers the tool wear variation under variable cutting parameters. In this method, a stacked autoencoder and long short-term memory network (SAE–LSTM) is designed as the fundamental surface roughness prediction model using tool wear conditions and sensor signals as inputs. The transfer learning strategy is applied to the SAE–LSTM such that the surface roughness online prediction under variable cutting parameters can be realized. Machining experiments for the assembly interface (using Ti6Al4V as material) of an aircraft’s vertical tail are conducted, and monitoring data are used to validate the proposed method. Ablation studies are implemented to evaluate the key modules of the proposed model. The experimental results show that the proposed method outperforms other models and is capable of tracking the true surface roughness with time. Specifically, the minimum values of the root mean square error and mean absolute percentage error of the prediction results after transfer learning are 0.027 μm and 1.56%, respectively.