Metaverses embedded in our lives create virtual experiences inside of the physical world. Moving towards metaverses in aircraft maintenance, mixed reality (MR) creates enormous opportunities for the interaction with virtual airplanes (digital twin) that deliver a near-real experience, keeping physical distancing during pandemics. 3D twins of modern machines exported to MR can be easily manipulated, shared, and updated, which creates colossal benefits for aviation colleges who still exploit retired models for practicing. Therefore, we propose mixed reality education and training of aircraft maintenance for Boeing 737 in smart glasses, enhanced with a deep learning speech interaction module for trainee engineers to control virtual assets and workflow using speech commands, enabling them to operate with both hands. With the use of the convolutional neural network (CNN) architecture for audio features and learning and classification parts for commands and language identification, the speech module handles intermixed requests in English and Korean languages, giving corresponding feedback. Evaluation with test data showed high accuracy of prediction, having on average 95.7% and 99.6% on the F1-Score metric for command and language prediction, respectively. The proposed speech interaction module in the aircraft maintenance metaverse further improved education and training, giving intuitive and efficient control over the operation, enhancing interaction with virtual objects in mixed reality.

Digital twins have revolutionized manufacturing and maintenance, allowing us to interact with virtual yet realistic representations of the physical world in simulations to identify potential problems or opportunities for improvement. However, traditional digital twins do not have the ability to communicate with humans using natural language, which limits their potential usefulness. Although conventional natural language processing methods have proven to be effective in solving certain tasks, neuro-symbolic AI offers a new approach that leads to more robust and versatile solutions. In this paper, we propose neuro-symbolic reasoning (NSR)—a fundamental method for interacting with 3D digital twins using natural language. The method understands user requests and contexts to manipulate 3D components of digital twins and is able to read maintenance manuals and implement installations and removal procedures autonomously. A practical neuro-symbolic dataset of machine-understandable manuals, 3D models, and user queries is collected to train the neuro-symbolic reasoning interaction mechanism. The evaluation demonstrates that NSR can execute user commands accurately, achieving 96.2% accuracy on test data. The proposed method has industrial importance since it provides the technology to perform maintenance procedures, request information from manuals, and serve as a tool to interact with complex virtual machinery using natural language.

A number of studies have focused on how to constructan academic social network. The relational ties among researchers are either co-authorship or shared keywords that are captured from scientific journals. The problem with such a network is that researchers are limited within their professional social network. In this paper, we propose a novel method for building a social network explicitly based on researchers’ knowledge interests. The researcher’s profile is automatically generated from metadata of scientific publications and homepage. By measuring the similarity between topics of interest, we are able to construct a researcher social network with relational linkages among authors that are produced by matching the their corresponding profiles. A direct loop graph-based social network is proposed. The graph naturally represents such a social network. Interestingly, our results showed that a social network based on profile matching is more representative than network based on publication co-authorship or shared keywords. Researcher mining in the academic social network has been explored via two problems Researcher Ranking and Expert Finding.

This paper addresses the problem of unsupervised video summarization. Video summarization helps people browse large-scale videos easily with a summary from the selected frames of the video. In this paper, we propose an unsupervised video summarization method with piecewise linear interpolation (Interp-SUM). Our method aims to improve summarization performance and generate a natural sequence of keyframes with predicting importance scores of each frame utilizing the interpolation method. To train the video summarization network, we exploit a reinforcement learning-based framework with an explicit reward function. We employ the objective function of the exploring under-appreciated reward method for training efficiently. In addition, we present a modified reconstruction loss to promote the representativeness of the summary. We evaluate the proposed method on two datasets, SumMe and TVSum. The experimental result showed that Interp-SUM generates the most natural sequence of summary frames than any other the state-of-the-art methods. In addition, Interp-SUM still showed comparable performance with the state-of-art research on unsupervised video summarization methods, which is shown and analyzed in the experiments of this paper.