Li Tao

Li Tao

Auto-generated profile, based on publicly available information

Li Tao

Department of Information and Communication Engineering, the University of Tokyo, Tokyo, Japan

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
7 January 2022 in IEEE Transactions on Circuits and Systems for Video Technology
Verified
Journal Article
Open access
LTLi Tao
Li Tao
XWXueting Wang
Xueting Wang
TYToshihiko Yamasaki
Toshihiko Yamasaki
In this paper, we propose a self-supervised contrastive learning method to learn video feature representations. In traditional self-supervised contrastive learning methods, constraints from anchor, positive, and negative data pairs are used to train the model. In such a case, different samplings of the same video are treated as positives, and video clips from different videos are treated as negatives. Because the spatio-temporal information is important for video representation, we set the temporal constraints more strictly by introducing intra-negative samples. In addition to samples from different videos, negative samples are extended by breaking temporal relations in video clips from the same anchor video. With the proposed Inter-Intra Contrastive (IIC) framework, we can train spatio-temporal convolutional networks to learn feature representations from videos. Strong data augmentations, residual clips, as well as head projector are utilized to construct an improved version. Three kinds of intra-negative generation functions are proposed and extensive experiments using different network backbones are conducted on benchmark datasets. Without using pre-computed optical flow data, our improved version can outperform previous IIC by a large margin, such as 19.4% (from 36.8% to 56.2%) and 5.2% (from 15.5% to 20.7%) points improvements in top-1 accuracy on UCF101 and HMDB51 datasets for video retrieval, respectively. For video recognition, over 3% points improvements can also be obtained on these two benchmark datasets. Discussions and visualizations validate that our IICv2 can capture better temporal clues and indicate the potential mechanism.

4 November 2021 in IEEE Transactions on Image Processing
Verified
Journal Article
Open access
LTLi Tao
Li Tao
XWXueting Wang
Xueting Wang
TYToshihiko Yamasaki
Toshihiko Yamasaki
Recently, 3D convolutional networks yield good performance in action recognition. However, an optical flow stream is still needed for motion representation to ensure better performance, whose cost is very high. In this paper, we propose a cheap but effective way to extract motion features from videos utilizing residual frames as the input data in 3D ConvNets. By replacing traditional stacked RGB frames with residual ones, 35.6% and 26.6% points improvements over top-1 accuracy can be achieved on the UCF101 and HMDB51 datasets when trained from scratch using ResNet-18-3D. We deeply analyze the effectiveness of this modality compared to normal RGB video clips, and find that better motion features can be extracted using residual frames with 3D ConvNets. Considering that residual frames contain little information of object appearance, we further use a 2D convolutional network to extract appearance features and combine them together to form a two-path solution. In this way, we can achieve better performance than some methods which even used an additional optical flow stream. Moreover, the proposed residual-input path can outperform RGB counterpart on unseen datasets when we apply trained models to video retrieval tasks. Huge improvements can also be obtained when the residual inputs are applied to video-based self-supervised learning methods, revealing better motion representation and generalization ability of our proposal.

1 August 2023 in Multimedia Tools and Applications
Verified
Journal Article
Open access
PDF file
LTLi Tao
Li Tao
SNShunsuke Nakamura
Shunsuke Nakamura
XWXueting Wang
Xueting Wang
TKTatsuya Kawahara
Tatsuya Kawahara
GTGen Tamura
Gen Tamura
TYToshihiko Yamasaki
Toshihiko Yamasaki
Creating impressive video content such as movies and advertisements is a very important yet challenging task in business that requires both a sense of creativity and a lot of experience. Even professionals cannot necessarily invoke the impressions and emotions that they have aimed at. Many video advertisements are created and then disappear without giving a large impact on viewers. This paper presents a large-scale dataset of television (TV) advertisements that consists of 14,490 videos. The impressions of each video such as the recognition rate and interestingness rate are from the results of questionnaires answered by 620 participants. We also present a baseline for predicting the impression effects of TV advertisements by using visual and audio information, metadata such as broadcasting pattern, business category, the popularity of the casts, and text information including texts appearing on videos and narrations in audios. We predict four impressions of the viewers: 1) how much participants remember the video afterward, 2) how much they feel like buying the product/service, 3) how much they become interested in the product/service, and 4) how much they like the content of the advertisement itself. By combining images, audio, metadata, cast data, and text data, our baseline method is able to predict such impressions with a correlation of 0.69-0.82, much better than using a single-modal feature such as visual data or audio data only. This paper also gives some possible applications such as estimating the importance scores of each key frame, which gives us informative insights about how to make the advertisement content more impressive.

1 October 2020
Verified
Conference Paper
Closed access
LTLi Tao
Li Tao
XWXueting Wang
Xueting Wang
TYToshihiko Yamasaki
Toshihiko Yamasaki
Recently, 3D convolutional networks (3D ConvNets) yield good performance in action recognition. However, optical flow stream is still needed to ensure better performance, the cost of which is very high. In this paper, we propose a fast but effective way to extract motion features from videos utilizing residual frames as the input data in 3D ConvNets. By replacing traditional stacked RGB frames with residual ones, 35.6% and 26.6% points improvements over top-l accuracy can be obtained on the UCF101 and HMDB51 datasets when ResNet-18 models are trained from scratch. And we achieved the state-of-the-art results in this training mode. Analysis shows that better motion features can be extracted using residual frames compared to RGB counterpart. By combining with a simple appearance path, our proposal can be even better than some methods using optical flow streams.