In this paper we present an approach where ultrasonic testing data (UT) is linked with its spatial coordinates and direction vector to the examined specimen. Doing so, the processed nondestructive testing (NDT) results can be visualized directly on the sample in real-time using augmented or virtual reality. To enable the link between NDT data and physical object, a 3D-tracking system is used. Spatial coordinates and NDT sensor data are stored together. For visualization, texture mapping was applied on a 3D model. The testing process consists of data recording, processing and visualization. All three steps are performed in real-time. The data is recorded by an UT-USB interface, processed on a PC workstation and displayed using a Mixed-Reality-system (MR). Our system allows real-time 3D visualization of ultrasonic NDT data, which is directly drawn into the virtual representation. Therefore, the possibility arises to assist the operator during the manual testing process. This new approach results in a much more intuitive testing process and a data set optimally prepared to be saved in a digital twin environment. The size of the samples is not limited to a laboratory scale, but also works for larger objects, e.g. a helicopter fuselage. Our approach is inspired by concepts of NDE 4.0 to create a new kind of smart inspection systems.

In this work, we present a system architecture that is especially designed for remote ultrasound testing inspections (UT). The system itself is realized using a real-time session service and a web service based on a REST API (REST: Representational State Transfer; API: application programming interface). This web service is used to store data persistently, e.g., sample geometries, raw nondestructive testing (NDT) data and derived inspection results in a shared dataspace. In the current development state, the results consist of textures mapped onto the sample geometry. This approach allows us to display the UT results directly on the real sample using mixed reality technologies. We also implemented a feature to assist the inspector remotely by making use of the availability of this digital representation. Hence, it is necessary to share additional data like the current UT-signal, temporary position marks, user and device positions, etc. between the different participants. A real-time distribution of this highly dynamic data is required to create an effective assistance environment. Therefore, a separate session service is used. The inspection data generated in this temporary session can also be transferred to the afore mentioned dataspace to be saved persistently. The system features mixed reality visualization for the inspector and optionally a virtual reality or a 3D Desktop environment for one or more remote assistants.

This abstract presents the outcomes of the AIRPASS project, a collaborative effort between Royal Netherlands Aerospace Centre (NLR) (an applied research organisation in the aerospace domain) and Bundeswehr Research Institute for Materials, Fuels and Lubricants (WIWeB) (an independent Bundeswehr research center which is responsible for the safety, technology and chemistry of materials and petroleum, oils and lubricants), which aimed to develop a novel inspection concept for aircraft structures by leveraging automation and digitalization. The project's core idea revolves around the central role of the digital twin, fed by non-destructive testing (NDT) data. Within AIRPASS, we successfully demonstrated an automated approach to link inspection data to the digital twin. Notably, NLR and WIWeB employed distinct methods to achieve this goal: NLR utilized visual properties of the inspected object to register the position of its contactless NDT sensor with respect to the digital twin, whereas WIWeB employed positioning devices that recognized reference points in 3D space, suitable for probe-like sensors. The method employed by NLR combines the simplicity of a Structure from Motion (SfM) system for camera positioning and orientation, with the geometric accuracy of high fidelity 3D scanning or CAD geometry for the projection of NDT measurements. The WIWeB’s system makes use of an optical tracking system, which enables precise locating of the sample, probe and Mixed Reality glasses. It features real-time feedback for the user by drawing the NDT results as a color-coded map into the texture of the 3D model and displaying it in the users field of view. A comparative analysis of the results from both approaches was also conducted, providing valuable insights into the effectiveness of each method. The AIRPASS project's findings pave the way for enhanced automation and digitalization in aircraft inspection processes.