It is a challenging task to learn discriminative representation from images and videos, due to large local redundancy and complex global dependency in these visual data. Convolution neural networks (CNNs) and vision transformers (ViTs) have been two dominant frameworks in the past few years. Though CNNs can efficiently decrease local redundancy by convolution within a small neighborhood, the limited receptive field makes it hard to capture global dependency. Alternatively, ViTs can effectively capture long-range dependency via self-attention, while blind similarity comparisons among all the tokens lead to high redundancy. To resolve these problems, we propose a novel Unified transFormer (UniFormer), which can seamlessly integrate the merits of convolution and self-attention in a concise transformer format. Different from the typical transformer blocks, the relation aggregators in our UniFormer block are equipped with local and global token affinity respectively in shallow and deep layers, allowing tackling both redundancy and dependency for efficient and effective representation learning. Finally, we flexibly stack our blocks into a new powerful backbone, and adopt it for various vision tasks from image to video domain, from classification to dense prediction. Without any extra training data, our UniFormer achieves 86.3 top-1 accuracy on ImageNet-1 K classification task. With only ImageNet-1 K pre-training, it can simply achieve state-of-the-art performance in a broad range of downstream tasks. It obtains 82.9/84.8 top-1 accuracy on Kinetics-400/600, 60.9/71.2 top-1 accuracy on Something-Something V1/V2 video classification tasks, 53.8 box AP and 46.4 mask AP on COCO object detection task, 50.8 mIoU on ADE20 K semantic segmentation task, and 77.4 AP on COCO pose estimation task. Moreover, we build an efficient UniFormer with a concise hourglass design of token shrinking and recovering, which achieves 2-4 $\bm {\times }$ higher throughput than the recent lightweight models. Code is available at https://github.com/Sense-X/UniFormer .

Temporal action localization (TAL) is an important and challenging problem in video understanding. However, most existing TAL benchmarks are built upon the coarse granularity of action classes, which exhibits two major limitations in this task. First, coarse-level actions can make the localization models overfit in high-level context information, and ignore the atomic action details in the video. Second, the coarse action classes often lead to the ambiguous annotations of temporal boundaries, which are inappropriate for temporal action localization. To tackle these problems, we develop a novel large-scale and fine-grained video dataset, coined as FineAction , for temporal action localization. In total, FineAction contains 103K temporal instances of 106 action categories, annotated in 17K untrimmed videos. Compared to the existing TAL datasets, our FineAction takes distinct characteristics of fine action classes with rich diversity, dense annotations of multiple instances, and co-occurring actions of different classes, which introduces new opportunities and challenges for temporal action localization. To benchmark FineAction, we systematically investigate the performance of several popular temporal localization methods on it, and deeply analyze the influence of fine-grained instances in temporal action localization. As a minor contribution, we present a simple baseline approach for handling the fine-grained action detection, which achieves an mAP of 13.17% on our FineAction. We believe that FineAction can advance research of temporal action localization and beyond. The dataset is available at https://deeperaction.github.io/datasets/fineaction.

Graph Convolution Network (GCN) has been successfully used for 3D human pose estimation in videos. However, it is often built on the fixed human-joint affinity, according to human skeleton. This may reduce adaptation capacity of GCN to tackle complex spatio-temporal pose variations in videos. To alleviate this problem, we propose a novel Dynamical Graph Network (DG-Net), which can dynamically identify human-joint affinity, and estimate 3D pose by adaptively learning spatial/temporal joint relations from videos. Different from traditional graph convolution, we introduce Dynamical Spatial/Temporal Graph convolution (DSG/DTG) to discover spatial/temporal human-joint affinity for each video exemplar, depending on spatial distance/temporal movement similarity between human joints in this video. Hence, they can effectively understand which joints are spatially closer and/or have consistent motion, for reducing depth ambiguity and/or motion uncertainty when lifting 2D pose to 3D pose. We conduct extensive experiments on three popular benchmarks, e.g., Human3.6M, HumanEva-I, and MPI-INF-3DHP, where DG-Net outperforms a number of recent SOTA approaches with fewer input frames and model size.

In this paper, we develop a large-scale vision-language fish benchmark, namely WildFish++, for comprehensive studies in multimedia research. Concretely, WildFish++ consists of 2,348 fish categories with 103,034 images in the wild, and 3,817 fish descriptions with 213,858 words. Based on these distinct characteristics, we mainly introduce four challenging research tasks on WildFish++. (1) Fine-Grained Recognition with Comparison Texts . WildFish++ naturally contains subtle difference among fish categories, which leads to fine-grained classification. Most approaches resort to tackle this problem by capturing discriminative regions in the view of each image. However, this paradigm may be still far way from extracting the most distinct features when the context on visual difference is not available. In this case, we propose to introduce comparison fish descriptions, a unique corpus that can directly point out subtle difference between highly-confused species and naturally serve as a kind of valuable context information. With such texts, we creatively elaborate a multi-modal fish network, aiming at incorporating those comparison textual information as prior knowledge and consequently leveraging it to guide CNNs to find subtle yet distinct regions in the context of comparison texts. (2) Open-Set Classification . We often confront with unknown categories in practice, e.g., there may still exist unknown fishes in our planet. Hence, we creatively adapt WildFish++ for a novel open-set classification task, which aims at correctly assigning each test image into the unknown class or one of known classes. More importantly, we investigate a number of practical designs to boost accuracy of deep learning models in open-set scenarios. (3) Cross-Modal Retrieval . WildFish++ not only contains diversified fish images in the wild but also has rich fish descriptions about morphology diagnosis, biology information, etc. Hence, we design a challenging cross-modal retrieval task, which leverages three subtasks such as text-to-text, text-to-image, image-to-text retrieval in a unified end-to-end framework. (4) Automatic Fish Classification . Automatic fish classification is a long-term research in marine biology, while current studies are unsatisfactory due to the lack of large-scale data. In this case, we train a number of CNNs with WildFish++, and use its pre-trained models to boost fish classification on most existing benchmarks of wild fishes. We will release WildFish++ with codes/protocols ( https://github.com/PeiqinZhuang/WildFish++ ). We believe it can promote relevant studies in multimedia and beyond.