Xiaojie Jin

Xiaojie Jin

Auto-generated profile, based on publicly available information

Xiaojie Jin

Bytedance Inc

Research Focus & Metrics

Academic Career Length

11Years (2016-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 April 2016 in Expert Systems with Applications
Verified
Journal Article
Closed access
WCWenchong Chen
Wenchong Chen
JLJing Li
Jing Li
XJXiaojie Jin
Xiaojie Jin

23 October 2022
Verified
Book Chapter
Closed access
PDF file
YSYaojie Shen
Yaojie Shen
LZLibo Zhang
Libo Zhang
KXKai Xu
Kai Xu
XJXiaojie Jin
Xiaojie Jin

24 March 2024 in Proceedings of the AAAI Conference on Artificial Intelligence
Verified
Journal Article
Open access
PDF file
FMFan Ma
Fan Ma
XJXiaojie Jin
Xiaojie Jin
HWHeng Wang
Heng Wang
JHJingjia Huang
Jingjia Huang
LZLinchao Zhu
Linchao Zhu
YYYi Yang
Yi Yang
Video-language pre-training models have recently achieved remarkable results on various multi-modal downstream tasks. However, most of these models rely on contrastive learning or masking modeling to align global features across modalities, neglecting the local associations between video frames and text tokens. This limits the model’s ability to perform fine-grained matching and generalization, especially for tasks that selecting segments in long videos based on query texts. To address this issue, we propose a novel stitching and matching pre-text task for video-language pre-training that encourages fine-grained interactions between modalities. Our task involves stitching video frames or sentences into longer sequences and predicting the positions of cross-model queries in the stitched sequences. The individual frame and sentence representations are thus aligned via the stitching and matching strategy, encouraging the fine-grained interactions between videos and texts. in the stitched sequences for the cross-modal query. We conduct extensive experiments on various benchmarks covering text-to-video retrieval, video question answering, video captioning, and moment retrieval. Our results demonstrate that the proposed method significantly improves the generalization capacity of the video-text pre-training models.

1 August 2024
Verified
Conference Paper
Open access
PDF file
BWBo Wang
Bo Wang
ZZZhao Zhao Zhang
Zhao Zhao Zhang
MZMingbo Zhao
Mingbo Zhao
XJXiaojie Jin
Xiaojie Jin
MXMingliang Xu
Mingliang Xu
MWMeng Wang
Meng Wang
Mainstream image captioning models are usually two-stage captioners, i.e., encoding the region features by a pre-trained detector and then feeding them into a language model to generate the captions. However, such a two-stage procedure will lead to a task-based information gap that decreases the performance, because the region features in the detection task are suboptimal representations and cannot provide all the necessary information for subsequent captions generation. Besides, the region features are usually represented from the last layer of the detectors that lose the local details of images. In this paper, we propose a novel One-Stage Image Captioner (OSIC) with dynamic multi-sight learning, which directly transforms the images into descriptive sentences in one stage for eliminating the information gap. Specifically, to obtain rich features, multi-level features are captured by Swin Transformer, and then fed into a novel dynamic multi-sight embedding module to exploit both the global structure and local texture of input images. To enhance the global modeling capacity of the visual encoder, we propose a new dual-dimensional refining to non-locally model the features interaction. As a result, OSIC can directly obtain rich semantic information to improve the captioner. Extensive comparisons on the benchmark MS-COCO, Flickr8K and Flickr30K datasets verified the superior performance of our method.

24 May 2023 in arXiv
Verified
Preprint
Open access
PDF file
CLCheng-Ze Lu
Cheng-Ze Lu
XJXiaojie Jin
Xiaojie Jin
QHQibin Hou
Qibin Hou
JLJun Hao Liew
Jun Hao Liew
MCMing-Ming Cheng
Ming-Ming Cheng
JFJiashi Feng
Jiashi Feng
Understanding whether self-supervised learning methods can scale with unlimited data is crucial for training large-scale models. In this work, we conduct an empirical study on the scaling capability of masked image modeling (MIM) methods (e.g., MAE) for visual recognition. Unlike most previous works that depend on the widely-used ImageNet dataset, which is manually curated and object-centric, we take a step further and propose to investigate this problem in a more practical setting. Specifically, we utilize the web-collected Coyo-700M dataset. We randomly sample varying numbers of training images from the Coyo dataset and construct a series of sub-datasets, containing 0.5M, 1M, 5M, 10M, and 100M images, for pre-training. Our goal is to investigate how the performance changes on downstream tasks when scaling with different sizes of data and models. The study reveals that: 1) MIM can be viewed as an effective method to improve the model capacity when the scale of the training data is relatively small; 2) Strong reconstruction targets can endow the models with increased capacities on downstream tasks; 3) MIM pre-training is data-agnostic under most scenarios, which means that the strategy of sampling pre-training data is non-critical. We hope these observations could provide valuable insights for future research on MIM.

22 May 2023 in arXiv
Verified
Preprint
Open access
PDF file
XHXingjian He
Xingjian He
SCSihan Chen
Sihan Chen
FMFan Ma
Fan Ma
ZHZhicheng Huang
Zhicheng Huang
XJXiaojie Jin
Xiaojie Jin
JFJiashi Feng
Jiashi Feng
Large-scale image-text contrastive pre-training models, such as CLIP, have been demonstrated to effectively learn high-quality multimodal representations. However, there is limited research on learning video-text representations for general video multimodal tasks based on these powerful features. Towards this goal, we propose a novel video-text pre-training method dubbed VLAB: Video Language pre-training by feature Adapting and Blending, which transfers CLIP representations to video pre-training tasks and develops unified video multimodal models for a wide range of video-text tasks. Specifically, VLAB is founded on two key strategies: feature adapting and feature blending. In the former, we introduce a new video adapter module to address CLIP's deficiency in modeling temporal information and extend the model's capability to encompass both contrastive and generative tasks. In the latter, we propose an end-to-end training method that further enhances the model's performance by exploiting the complementarity of image and video features. We validate the effectiveness and versatility of VLAB through extensive experiments on highly competitive video multimodal tasks, including video text retrieval, video captioning, and video question answering. Remarkably, VLAB outperforms competing methods significantly and sets new records in video question answering on MSRVTT, MSVD, and TGIF datasets. It achieves an accuracy of 49.6, 61.0, and 79.0, respectively. Codes and models will be released.