In the field of the acquisition of sports skills, a common way to improve sports skills, such as golf swings, is to imitate professional players’ motions. However, it is difficult for beginners to specify the keyframes on which they should focus and which part of the body they should correct because of inconsistent timing and lack of knowledge. In this study, a golf swing analysis tool using neural networks is proposed to address this gap. The proposed system compares two motion sequences and specifies keyframes in which significant differences can be observed between the two motions. In addition, the system helps users intuitively understand the differences between themselves and professional players by using interpretable clues. The main challenge of this study is to target the fine-grained differences between users and professionals that can be used for self-training. Moreover, the significance of the proposed approach is the use of an unsupervised learning method without prior knowledge and labeled data, which will benefit future applications and research in other sports and skill training processes. In our approach, neural networks are first used to create a motion synchronizer to align motions with different phases and timing. Next, a motion discrepancy detector is implemented to find fine-grained differences between motions in latent spaces that are learned by the networks. Furthermore, we consider that learning intermediate motions may be feasible for beginners because, in this way, they can gradually change their pose to match the ideal form. Therefore, based on the synchronization and discrepancy detection results, we utilize a decoder to restore the intermediate human poses between two motions from the latent space. Finally, we suggest possible applications for analyzing and visualizing the discrepancy between the two input motions and interacting with the users. With the proposed application, users can easily understand the differences between their motions and those of various experts during self-training and learn how to improve their motions.

Spatial and temporal clues found in a professional’s motion are essential for designing a training system for learning a motor skill. We investigate the potential of using neural networks to learn spatial and temporal features of advanced players in sports and to detect the fine-grained differences between motions. As a training system, we implement an AI Coach prototype application that finds the differences between two input motions and visualizes a recommendation motion for the users to correct their forms. In the user study, we investigate the effects of the proposed AI Coach and discuss the findings based on quantitative questionnaires and qualitative interviews. In the study, the proposed system can help the user better understand the difference between them and the coach. The study also reveals the necessity of coaching beginners in the early learning phases.

In this work, we present an analysis tool to help golf beginners compare their swing motion to the swing motion of experts. The proposed application synchronizes videos with different swing phase timings using the latent features extracted by a neural network-based encoder and detects key frames where discrepant motions occur. We visualize synchronized image frames and 3D poses that help users recognize the differences and key factors that can be important for their swing skill improvement.

Microscopic suturing in neurosurgery is a challenging medical technique that requires time to master. Since skilled surgeons are too busy to spend much time with novice surgeons, novice surgeons need to train alone in monotonous tasks to acquire skills. To address this problem, this study proposes a system that incorporates gamification elements, such as scoring and displaying real-time feedback, to improve motivation for training. This system detects the technical factors necessary for microscopic suturing from video capture, and calculates a score using these factors. According to neurosurgeons, suturing has three important factors: speed, accuracy, and carefulness. These technical factors are detected by tracking instruments and gauze using machine learning and image processing. The experiment was conducted with ten novices using this system. The results of this experiment showed that the system is easy to use and contributed to increased motivation, according to the User Experience Questionnaire (UEQ) and System Usability Scale (SUS).

MonoMR is a system that synthesizes pseudo-2.5D content from monocular videos for mixed reality (MR) head-mounted displays (HMDs). Unlike conventional systems that require multiple cameras, the MonoMR system can be used by casual end-users to generate MR content from a single camera only. In order to synthesize the content, the system detects people in the video sequence via a deep neural network, and then the detected person’s pseudo-3D position is estimated by our proposed novel algorithm through a homography matrix. Finally, the person’s texture is extracted using a background subtraction algorithm and is placed on an estimated 3D position. The synthesized content can be played in MR HMD, and users can freely change their viewpoint and the content’s position. In order to evaluate the efficiency and interactive potential of MonoMR, we conducted performance evaluations and a user study with 12 participants. Moreover, we demonstrated the feasibility and usability of the MonoMR system to generate pseudo-2.5D content using three example application scenarios.

This work proposes a motion style transfer network that transfers motion style between different motion categories using variational autoencoders. The proposed network effectively transfers style among various motion categories and can create stylized motion unseen in the dataset. The network contains a content-conditioned module to preserve the characteristic of the content motion, which is important for real applications. We implement the network with variational autoencoders, which enable us to control the intensity of the style and mix different styles to enrich the motion diversity.