The Sign Language Production (SLP) project aims to automatically translate spoken languages into sign sequences. Our approach focuses on the transformation of sign gloss sequences into their corresponding sign pose sequences (G2P). In this paper, we present a novel solution for this task by converting the continuous pose space generation problem into a discrete sequence generation problem. We introduce the Pose-VQVAE framework, which combines Variational Autoencoders (VAEs) with vector quantization to produce a discrete latent representation for continuous pose sequences. Additionally, we propose the G2P-DDM model, a discrete denoising diffusion architecture for length-varied discrete sequence data, to model the latent prior. To further enhance the quality of pose sequence generation in the discrete space, we present the CodeUnet model to leverage spatial-temporal information. Lastly, we develop a heuristic sequential clustering method to predict variable lengths of pose sequences for corresponding gloss sequences. Our results show that our model outperforms state-of-the-art G2P models on the public SLP evaluation benchmark. For more generated results, please visit our project page: https://slpdiffusier.github.io/g2p-ddm.

Dynamic curved interfaces are fundamental and ubiquitous structures in biological systems. However, replicating the structure and function associated with these interfaces for mechanobiology and drug screening is challenging. Here, we develop a dynamic curvature-enabled microfluidic organ chip of two fluid–solid dynamic curved interfaces. One interface effectively integrates adjustable biomechanics, and the other controls drug release with open microfluidics. The fluid–solid interface sensed by the cells can modulate the residual stress, stiffness, strain of the solid phase, and the flow shear stress of the fluid phase. Using the chip, we investigate the mechanotransductive responses of endothelial and epithelial cells, including Piezo1, Ca2+, and YAP, and reveal that the response of the endothelium to combined dynamic cyclic strain and flow shear stress is different from separate stimulation and also disparate from the epithelium. Furthermore, direct and high-efficiency drug release to cells is realized by constructing the other fluid–solid interface on the back side of cells, where drugs are encapsulated within cross-linked alginate hydrogel in the open microfluidic channel. Then, we replicate object-specific and location-specific biomechanical environments within carotid bifurcation and prove the effectiveness of drug delivery. Our design exemplifies dynamic curved biological interfaces with controlled mechanical environments and holds potential for patient-specific medicine.

The advancement of deep learning techniques has significantly propelled the development of the continuous sign language recognition (cSLR) task. However, the spatial feature extraction of sign language videos in the RGB space tends to focus on the overall image information while neglecting the perception of traits at different granularities, such as eye gaze and lip shape, which are more detailed, or posture and gestures, which are more macroscopic. Exploring the efficient fusion of visual information of different granularities is crucial for accurate sign language recognition. In addition, applying a vanilla Transformer to sequence modeling in cSLR exhibits weak performance because specific video frames could interfere with the attention mechanism. These limitations constrain the capability to understand potential semantic characteristics. We introduce a feature fusion method for integrating visual features of disparate granularities and refine the metric of attention to enhance the Transformer’s comprehension of video content. Specifically, we extract CNN feature maps with varying receptive fields and employ a self-attention mechanism to fuse feature maps of different granularities, thereby obtaining multi-scale spatial features of the sign language framework. As for video modeling, we first analyze why the vanilla Transformer failed in cSLR and observe that the magnitude of the feature vectors of video frames could interfere with the distribution of attention weights. Therefore, we utilize the Euclidean distance among vectors to measure the attention weights instead of scaled-dot to enhance dynamic temporal modeling capabilities. Finally, we integrate the two components to construct the model MSF-ET (Multi-Scaled feature Fusion–Euclidean Transformer) for cSLR and train the model end-to-end. We perform experiments on large-scale cSLR benchmarks—PHOENIX-2014 and Chinese Sign Language (CSL)—to validate the effectiveness.

Sign language translation aims to transform continuous sign language videos into text, thereby mapping sign language actions onto natural language. Current research primarily focuses on the Transformer’s encoder-decoder architecture, employing an autoregressive (AR) approach for SLT. This one-by-one token generation method has a slow speed to translate. We design a Non-Autoregressive Sign Language Translation (NA-SLT) model based on Transformer Encoder. The proposed model consists of three components: the text-encoder, sign-encoder, and sign-text-encoder. Both the text-encoder and sign-encoder utilize the Encoder to model single-modality data. The sign-text-encoder effectively engages with multi-modal data, producing high-quality corresponding text after minimal refinement iterations. Moreover, during model training, we devise a double-mask strategy. In addition to traditional random masking, the schedule allocates mask probabilities based on each word’s frequency, ensuring long-tail tokens have an increased chance of being masked and receiving extensive training. We evaluate the model on PHOENIX-2014T, and the results indicate that NA-SLT approaches the performance of AR models on BLEU (21.06) and ROUGE(49.59) metrics while significantly improving translation speed (5.4x-7.9x).

Sign Language Production (SLP) is the tough task of turning sign language into sign videos. The main goal of SLP is to create these videos using a sign gloss. In this research, we've developed a new method to make high-quality sign videos without using human poses as a middle step. Our model works in two main parts: first, it learns from a generator and the video's hidden features, and next, it uses another model to understand the order of these hidden features. To make this method even better for sign videos, we make several significant improvements. (i) In the first stage, we take an improved 3D VQ-GAN to learn downsampled latent representations. (ii) In the second stage, we introduce sequence-to-sequence attention to better leverage conditional information. (iii) The separated two-stage training discards the realistic visual semantic of the latent codes in the second stage. To endow the latent sequences semantic information, we extend the token-level autoregressive latent codes learning with perceptual loss and reconstruction loss for the prior model with visual perception. Compared with previous state-of-the-art approaches, our model performs consistently better on two word-level sign language datasets, i.e., WLASL and NMFs-CSL.