We address the challenging problem of weakly supervised temporal action localization from unconstrained web videos, where only the video-level action labels are available during training. Inspired by the Adversarial Erasing strategy in weakly supervised semantic segmentation, we propose a novel iterative-winners-out network. Specifically, we make two technical contributions: 1) we propose an iterative training strategy, namely winners-out, to select the most discriminative action instances in each training iteration and remove them in the next training iteration. This iterative process alleviates the “winner-takes-all” phenomenon that existing approaches tend to choose the video segments that strongly correspond to the video label, but neglect other less discriminative video segments. With this strategy, our network is able to localize not only the most discriminative instances but also the less discriminative ones. 2) to better select the target action instances in winners-out, we devise a class-discriminative localization technique. By employing the attention mechanism and the information learned from data, our technique is able to identify the most discriminative action instances effectively. The two key components are integrated into an end-to-end network to localize actions without using the frame-level annotations. Extensive experimental results demonstrate that our method outperforms the state-of-the-art weakly supervised approaches on ActivityNet1.3 and improves mAP from 16.9% to 20.5% on THUMOS14. Notably, even with weak video-level supervision, our method attains comparable accuracy to those employing frame-level supervisions.

Temporal action localization, which aims at recognizing the location and the category of action instances in videos, has long been researched. Existing methods divide each video into multiple action units (i.e., proposals in two-stage methods and segments in one-stage methods) and then perform recognition/regression on each of them individually without explicitly exploiting their relations, which, however, play an important role in action localization. In this paper, we propose a general graph convolutional module (GCM) that can be easily plugged into existing action localization methods, including two-stage and one-stage paradigms. Specifically, we first construct a graph, where each action unit is represented as a node and their relations as edges. We use two types of relations, one for capturing the temporal connections, and the other one for characterizing the semantic relationship. Then, we apply graph convolutional networks (GCNs) on the graph to model the relations and learn more informative representations for action localization. Experimental results show that GCM consistently improves the performance of both two-stage action localization methods (e.g., CBR and R-C3D) and one-stage methods (e.g., D-SSAD), verifying the generality and effectiveness of GCM. Moreover, with the aid of GCM, our approach significantly outperforms the state-of-the-art on THUMOS14 and ActivityNet.

We study unsupervised video representation learning that seeks to learn both motion and appearance features from unlabeled video only, which can be reused for downstream tasks such as action recognition. This task, however, is extremely challenging due to 1) the highly complex spatial-temporal information in videos and 2) the lack of labeled data for training. Unlike representation learning for static images, it is difficult to construct a suitable self-supervised task to effectively model both motion and appearance features. More recently, several attempts have been made to learn video representation through video playback speed prediction. However, it is non-trivial to obtain precise speed labels for the videos. More critically, the learned models may tend to focus on motion patterns and thus may not learn appearance features well. In this paper, we observe that the relative playback speed is more consistent with motion patterns and thus provides more effective and stable supervision for representation learning. Therefore, we propose a new way to perceive the playback speed and exploit the relative speed between two video clips as labels. In this way, we are able to effectively perceive speed and learn better motion features. Moreover, to ensure the learning of appearance features, we further propose an appearance-focused task, where we enforce the model to perceive the appearance difference between two video clips. We show that jointly optimizing the two tasks consistently improves the performance on two downstream tasks (namely, action recognition and video retrieval) w.r.t the increasing pre-training epochs. Remarkably, for action recognition on the UCF101 dataset, we achieve 93.7% accuracy without the use of labeled data for pre-training, which outperforms the ImageNet supervised pre-trained model. Our code, pre-trained models, and supplementary materials can be found at https://github.com/PeihaoChen/RSPNet.

Temporal action localization aims to accurately localize and recognize all possible action instances from an untrimmed video automatically. Most existing methods tackle this task by first generating a set of proposals and then recognizing each one independently. However, due to the complex structures and large content variations in action instances, recognizing them one by one can be difficult. Fortunately, some proposals often have shared information regarding one specific action. Such information, which is ignored in existing methods, can be used to boost the recognition performance. In this paper, we propose a novel mechanism, called relation attention, to exploit the informative relations among proposals according to their appearance or optical flow features. Specifically, we propose a relation attention module to enhance the representation power by capturing useful information from other proposals. This module does not change the dimensions of the original input and output and does not rely on any specific proposal generation methods or feature extraction backbone networks. Extensive experimental results show that the proposed relation attention mechanism is able to improve performance significantly on both Thumos14 and ActivityNet1.3 datasets upon several existing best-performing architectures. For example, relying on Structured Segment Networks (SSN), the proposed relation attention module helps to increase thejavascript:setNextPage(‘AUTHOR_VIEW_RESPOND_TO_COMMENTS’) mAP from 41.4 to 43.7 on the Thumos14 dataset and outperform the state-of-the-art results.

Driver behavior recognition has become one of the most important tasks for intelligent vehicles. This task, however, is very challenging since the background contents in real-world driving scenarios are often very complex. More critically, the difference between driving behaviors is often very minor, making it extremely difficult to distinguish them. Existing methods often rely only on RGB frames (or skeleton data), which may fail to capture the minor differences between behaviors and appearance information of objects simultaneously and thus fail to achieve promising performance. To address the above issues, in this paper, we propose a bidirectional posture-appearance interaction network (BPAI-Net), which simultaneously considers RGB frames and skeleton ( i.e. , posture) data for driver behavior recognition. Specifically, we propose a posture-guided convolutional neural network (PG-CNN) and an appearance-guided graph convolutional network (AG-GCN) to extract appearance and posture features, respectively. To exploit the complementary information between appearance and posture, we use the appearance features from PG-CNN for guiding AG-GCN to exploit the contextual information ( e.g. , nearby objects) to enhance posture features. Then, we use the enhanced posture features from AG-GCN to help PG-CNN focus on critical local areas of video frames that are related to driver behaviors. In this sense, we are able to use the interaction between two modalities to extract more discriminative features and thus improve the recognition accuracy. Experimental results on Drive&Act dataset show that our method outperforms state-of-the-art methods by a large margin (67.83% vs. 63.64%). Furthermore, we collect a bus driver behavior recognition dataset and yield consistent performance gain against baseline methods, demonstrating the effectiveness of our method in real-world applications. The source code and trained models are available at github.com/SCUT-AILab/BPAI-Net/.

Soft millirobot has attracted significant attention and demonstrated tremendous potential in human-robot interactions and safety inspections. Locomotion and perception are two crucial features for achieving effective gait and practical applications of robots. Inspired by nature, this research reports a magnetic soft millirobot that integrates locomotion and sensing capacities simultaneously. Microconical matrix with rich and regular surface morphologies are constructed directly inside the millirobot as both multilegged and triboelectric-enhanced sensing structures via cooperation of jet printing and magnetization-induction method with high-speed and high-precision. The robot can both recognize its current body state across various application scenarios and identify terrains through a machine learning strategy. Our work presents a customizable approach for smart millirobots to perform tasks in nonmagnetic structured environments and provides embedded sensing capability for next-generation soft robots.