31 December 2021 in IEEE Robotics and Automation Letters
The common paradigm of CNN-based action recognition modelsis to simply use the average of the dense predictions from every frame. However, these dense predictions are inefficient since all frames are evenly utilized regardless of the existence of the action. In real-time action recognition applications where the input video is untrimmed, the dense prediction is even more severely inefficient. Instead of these dense predictions, we propose a lightweight CNN-based sampler that selects frames that are relevant to the action in the video. The proposed sampler is trained by a self-supervised learning method that mixes one irrelevant frame in an input video clip and finds that irrelevant frame. With the trained sampler, we select frames to be used as inputs to the action recognition model. Our sampler can be combined with any action recognition models and works independently of them. Through various experiments, we demonstrate that the proposed sampler is not only time and memory-efficient but also significantly increases performance in the action recognition benchmarks, including HMDB51 and UCF101. Finally, we applied the self-supervised sampler to the real-time surveillance system in the subway station, and improved the action recognition performance by 7% with 80% reduction in computation. Code and dataset are available in https://github.com/seominseok0429/SelSupSampler and https://aihub.or.kr/aidata/34124 .
18 August 2024 in Remote Sensing Letters
6 February 2024 in IEEE Access
Semi-supervised semantic segmentation learns a model for classifying pixels into specific classes using a few labeled samples and numerous unlabeled images. The recent leading approach is consistency regularization by self-training with pseudo-labeling pixels having high confidences for unlabeled images. However, using only high-confidence pixels for self-training may result in losing much of the information in the unlabeled sets due to poor confidence calibration of modern deep learning networks. In this paper, we propose a class-wise adaptive strategy for semi supervised semantic segmentation (CASS) to cope with the loss of most information that occurs in existing high-confidence-based pseudo-labeling methods. Unlike existing semi-supervised semantic segmentation frameworks, CASS constructs a validation set on a labeled set, to leverage the calibration performance for each class. On this basis, we propose a calibration aware class-wise adaptive thresholding and class-wise adaptive over-sampling using the analysis results from the validation set. Our proposed CASS achieves state-of-the-art performance on the full data partition of the base PASCAL VOC 2012 dataset and on the 1/4 data partition of the Cityscapes dataset with significant margins of 83.0% and 80.4%, respectively.