The study developed and assessed a novel soft exoskeleton glove generating dexterous finger joint movements with little constraints on volitional motions. Four pneumatic artificial muscles were attached to each finger, which formed two antagonistic pairs of muscles similar to the human anatomy and thereby enabled various postural control of the individual joints. This unique structure provided 20 DOFs with the exoskeleton. A unique adjustable mechanism also allowed for fixing the soft exoskeleton suitable for user's hand shape and size semi-automatically. Our glove generated static force of approximately 8 N at the fingertip, and moved a single finger at approximately 10 Hz and all five digits sequentially at 25 Hz. Through co-contracting the antagonistic muscles, the fingertip force generated by the tapping was increased by 1.5 times. Compared with hard exoskeleton gloves, our soft exoskeleton glove had lower constraints on the volitional finger motions performed by pianists, and also enabled to passively move the fingers quicker. Finally, after the soft exoskeleton passively moved the fingers so as to play a musical excerpt with the piano over a half hour, force variability of the keypresses was reduced when playing even without wearing the glove. Passive movement guidance by the soft exoskeleton may facilitate fine control of sequential force production.

In the field of the acquisition of sports skills, a common way to improve sports skills, such as golf swings, is to imitate professional players’ motions. However, it is difficult for beginners to specify the keyframes on which they should focus and which part of the body they should correct because of inconsistent timing and lack of knowledge. In this study, a golf swing analysis tool using neural networks is proposed to address this gap. The proposed system compares two motion sequences and specifies keyframes in which significant differences can be observed between the two motions. In addition, the system helps users intuitively understand the differences between themselves and professional players by using interpretable clues. The main challenge of this study is to target the fine-grained differences between users and professionals that can be used for self-training. Moreover, the significance of the proposed approach is the use of an unsupervised learning method without prior knowledge and labeled data, which will benefit future applications and research in other sports and skill training processes. In our approach, neural networks are first used to create a motion synchronizer to align motions with different phases and timing. Next, a motion discrepancy detector is implemented to find fine-grained differences between motions in latent spaces that are learned by the networks. Furthermore, we consider that learning intermediate motions may be feasible for beginners because, in this way, they can gradually change their pose to match the ideal form. Therefore, based on the synchronization and discrepancy detection results, we utilize a decoder to restore the intermediate human poses between two motions from the latent space. Finally, we suggest possible applications for analyzing and visualizing the discrepancy between the two input motions and interacting with the users. With the proposed application, users can easily understand the differences between their motions and those of various experts during self-training and learn how to improve their motions.

Previous studies have explored generating accurately lip-synced talking faces for arbitrary targets given audio conditions. However, most of them deform or generate the whole facial area, leading to non-realistic results. In this work, we delve into the formulation of altering only the mouth shapes of the target person. This requires masking a large percentage of the original image and seamlessly inpainting it with the aid of audio and reference frames. To this end, we propose the Audio-Visual Context-Aware Transformer (AV-CAT) framework, which produces accurate lip-sync with photo-realistic quality by predicting the masked mouth shapes. Our key insight is to exploit desired contextual information provided in audio and visual modalities thoroughly with delicately designed Transformers. Specifically, we propose a convolution-Transformer hybrid backbone and design an attention-based fusion strategy for filling the masked parts. It uniformly attends to the textural information on the unmasked regions and the reference frame. Then the semantic audio information is involved in enhancing the self-attention computation. Additionally, a refinement network with audio injection improves both image and lip-sync quality. Extensive experiments validate that our model can generate high-fidelity lip-synced results for arbitrary subjects.

Spatial and temporal clues found in a professional’s motion are essential for designing a training system for learning a motor skill. We investigate the potential of using neural networks to learn spatial and temporal features of advanced players in sports and to detect the fine-grained differences between motions. As a training system, we implement an AI Coach prototype application that finds the differences between two input motions and visualizes a recommendation motion for the users to correct their forms. In the user study, we investigate the effects of the proposed AI Coach and discuss the findings based on quantitative questionnaires and qualitative interviews. In the study, the proposed system can help the user better understand the difference between them and the coach. The study also reveals the necessity of coaching beginners in the early learning phases.

While considerable progress has been made in achieving accurate lip synchronization for 3D speech-driven talking face generation, the task of incorporating expressive facial detail synthesis aligned with the speaker’s speaking status remains challenging. Existing efforts either focus on learning a dynamic talking head pose synchronized with speech rhythm or aim for stylized facial movements guided by external reference such as emotional labels or reference video clips. The former works often yield coarse alignment, neglecting the emotional nuances present in the audio content while the latter studies lead to unnatural applications, requiring manual style source selection by users. Our goal is to directly leverage the inherent style information conveyed by human speech for generating an expressive talking face that aligns with the speaking status. In this paper, we propose AVI-Talking, an Audio-Visual Instruction system for expressive Talking face generation. This system harnesses the robust contextual reasoning and hallucination capability offered by Large Language Models (LLMs) to instruct the realistic synthesis of 3D talking faces. Instead of directly learning facial movements from human speech, our two-stage strategy involves the LLMs first comprehending audio information and generating instructions implying expressive facial details seamlessly corresponding to the speech. Subsequently, a diffusion-based generative network executes these instructions. This two-stage process, coupled with the incorporation of LLMs, enhances model interpretability and provides users with flexibility to comprehend instructions and specify desired operations or modifications. Specifically, given a speech clip, we first employ a Q-Former for contrastive alignment the speech features with visual instructions, which is then projected to input text embedding of LLMs. It functions as a prompting strategy, prompting LLMs to generate plausible visual instructions that encompass diverse facial details. In order to use these predicted instructions, a language-guided talking face generation system with disentangled latent space is delicately derived, where the speech content related lip movements and emotion correlated facial expressions are separately represented in speech content space and content irrelevant space. Additionally, we introduce a contrastive instruction-style alignment and diffusion technique within the content-irrelevant space to fully exploit the talking prior network for diverse instruction-following synthesis. Extensive experiments showcase the effectiveness of our approach in producing vivid talking faces with expressive facial movements and consistent emotional status.

This article examines the attentional mechanism of in-person collaboration by means of System Dynamics-based simulations using an eye tracking experiment. Three experimental conditions were tested: in-person collaboration, remote collaboration, and single user. We hypothesized that collaboration focuses users’ attention on key information facilitating decision-making. Collaborating participants dwelt longer on key elements of the simulation than single users. Moreover, in-person collaboration and single users yielded a strategy of decision-making similar to an optimal strategy. Finally, in-person collaboration was less cognitively demanding and of higher quality. The contribution of this article is a deeper understanding of how in-person collaboration on a large display can help users focus their visual attention on the most important areas. With this novel understanding, we believe collaborative systems designers will be better equipped to design more effective attention-guiding mechanisms in remote collaboration systems. The present work has the potential to advance the study of collaborative, interactive technologies.