5 July 2019 in IEEE Transactions on Circuits and Systems for Video Technology
This paper aims at learning to score the figure skating sports videos. To address this task, we propose a deep architecture that includes two complementary components, i.e., Self-Attentive LSTM and Multi-scale Convolutional Skip LSTM. These two components can efficiently learn the local and global sequential information in each video. Furthermore, we present a large-scale figure skating sports video dataset FisV dataset. This dataset includes 500 figure skating videos with the average length of 2 minutes and 50 seconds. Each video is annotated by two scores of nine different referees, i.e., Total Element Score(TES) and Total Program Component Score (PCS). Our proposed model is validated on FisV and MIT-skate datasets. The experimental results show the effectiveness of our models in learning to score the figure skating videos. The codes and datasets would be downloaded from https://github.com/loadder/MS_LSTM.git.
21 November 2022 in IEEE Transactions on Pattern Analysis and Machine Intelligence
The task of Few-shot learning (FSL) aims to transfer the knowledge learned from base categories with sufficient labelled data to novel categories with scarce known information. It is currently an important research question and has great practical values in the real-world applications. Despite extensive previous efforts are made on few-shot learning tasks, we emphasize that most existing methods did not take into account the distributional shift caused by sample selection bias in the FSL scenario. Such a selection bias can induce spurious correlation between the semantic causal features, that are causally and semantically related to the class label, and the other non-causal features. Critically, the former ones should be invariant across changes in distributions, highly related to the classes of interest, and thus well generalizable to novel classes, while the latter ones are not stable to changes in the distribution. To resolve this problem, we propose a novel data augmentation strategy dubbed as PatchMix that can break this spurious dependency by replacing the patch-level information and supervision of the query images with random gallery images from different classes from the query ones. We theoretically show that such an augmentation mechanism, different from existing ones, is able to identify the causal features. To further make these features to be discriminative enough for classification, we propose Correlation-guided Reconstruction (CGR) and Hardness-Aware module for instance discrimination and easier discrimination between similar classes. Moreover, such a framework can be adapted to the unsupervised FSL scenario. The utility of our method is demonstrated on the state-of-the-art results consistently achieved on several benchmarks including mini ImageNet, tiered ImageNet, CIFAR-FS, CUB, Cars, Places and Plantae, in all settings of single-domain, cross-domain and unsupervised FSL. By studying the intra-variance property of learned features and visualizing the learned features, we further quantitatively and qualitatively show that such a promising result is due to the effectiveness in learning causal features.
18 September 2020 in IEEE Transactions on Image Processing
Generating realistic images with the guidance of reference images and human poses is challenging. Despite the success of previous works on synthesizing person images in the iconic views, no efforts are made towards the task of poseguided image synthesis in the non-iconic views. Particularly, we find that previous models cannot handle such a complex task, where the person images are captured in the non-iconic views by commercially-available digital cameras. To this end, we propose a new framework – Multi-branch Refinement Network (MR-Net), which utilizes several visual cues, including target person poses, foreground person body and scene images parsed. Furthermore, a novel Region of Interest (RoI) perceptual loss is proposed to optimize the MR-Net. Extensive experiments on two non-iconic datasets, Penn Action and BBC-Pose, as well as an iconic dataset – Market-1501, show the efficacy of the proposed model that can tackle the problem of pose-guided person image generation from the non-iconic views. The data, models, and codes are downloadable from https://github.com/loadder/MR-Net.
17 July 2023 in Systems Biology in Reproductive Medicine
To clarify the effect of next-generation sequencing (NGS)-based preimplantation genetic testing for aneuploidy (PGT-A) combined with trophectoderm (TE) biopsy on the pregnancy outcomes of idiopathic recurrent pregnancy loss (iRPL) and idiopathic recurrent implantation failure (iRIF), we conducted a retrospective cohort study of 212 iRPL couples and 66 iRIF couples who underwent PGT-A or conventional in vitro fertilization/intracytoplasmic sperm injection (IVF/ICSI) treatment. The implantation rate (IR) per transfer (64.2%), clinical pregnancy rate (CPR) per transfer (57.5%), and live birth rate (LBR) per transfer (45%) of iRPL couples of the PGT-A treatment group were significantly higher (p < 0.05) than those of the conventional IVF/ICSI group (IR per transfer,38.2%; CPR per transfer,33.3%; LBR per transfer, 28.4%), whereas the pregnancy loss rate (PLR) per transfer was similar between the two groups. These effects were also significant (p < 0.05) in iRPL couples with advanced maternal age (AMA, ≥35 years), whereas no significant differences were found in clinical outcomes between the PGT-A and conventional IVF/ICSI groups in younger iRPL couples (<35 years). The cumulative clinical outcomes of iRPL couples were comparable between the PGT-A and conventional IVF/ICSI groups. No significant differences were found in any clinical outcomes between the PGT-A and conventional IVF/ICSI groups for young or AMA couples with iRIF. In conclusion, NGS-based PGT-A involving TE biopsy may be useful for iRPL women to shorten the time to pregnancy and reduce their physical and psychological burden, especially for iRPL women with AMA; however, couples with iRIF may not benefit from PGT-A treatment. Considering the small sample size of the iRIF group, further investigations with a larger sample size are needed to verify our findings.