The common paradigm of CNN-based action recognition modelsis to simply use the average of the dense predictions from every frame. However, these dense predictions are inefficient since all frames are evenly utilized regardless of the existence of the action. In real-time action recognition applications where the input video is untrimmed, the dense prediction is even more severely inefficient. Instead of these dense predictions, we propose a lightweight CNN-based sampler that selects frames that are relevant to the action in the video. The proposed sampler is trained by a self-supervised learning method that mixes one irrelevant frame in an input video clip and finds that irrelevant frame. With the trained sampler, we select frames to be used as inputs to the action recognition model. Our sampler can be combined with any action recognition models and works independently of them. Through various experiments, we demonstrate that the proposed sampler is not only time and memory-efficient but also significantly increases performance in the action recognition benchmarks, including HMDB51 and UCF101. Finally, we applied the self-supervised sampler to the real-time surveillance system in the subway station, and improved the action recognition performance by 7% with 80% reduction in computation. Code and dataset are available in https://github.com/seominseok0429/SelSupSampler and https://aihub.or.kr/aidata/34124 .

Mixup provides interpolated training samples and allows the model to obtain smoother decision boundaries for better generalization. The idea can be naturally applied to the domain adaptation task, where we can mix the source and target samples to obtain domain-mixed samples for better adaptation. However, the extension of the idea from classification to segmentation (i.e., structured output) is nontrivial. This paper systematically studies the impact of mixup under the domain adaptive semantic segmentation task and presents a simple yet effective mixup strategy called Bidirectional Domain Mixup (BDM). In specific, we achieve domain mixup in two-step: cut and paste. Given the warm-up model trained from any adaptation techniques, we forward the source and target samples and perform a simple threshold-based cut out of the unconfident regions (cut). After then, we fill-in the dropped regions with the other domain region patches (paste). In doing so, we jointly consider class distribution, spatial structure, and pseudo label confidence. Based on our analysis, we found that BDM leaves domain transferable regions by cutting, balances the dataset-level class distribution while preserving natural scene context by pasting. We coupled our proposal with various state-of-the-art adaptation models and observe significant improvement consistently. We also provide extensive ablation experiments to empirically verify our main components of the framework. Visit our project page with the code at https://sites.google.com/view/bidirectional-domain-mixup

Floods, exacerbated by climate change, necessitate timely and accurate situational awareness to support effective disaster response. While electro-optical (EO) satellite imagery has been widely employed for flood assessment, its utility is significantly limited under conditions such as cloud cover or nighttime. Synthetic Aperture Radar (SAR) provides consistent imaging regardless of weather or lighting conditions but it remains challenging for human analysts to interpret. To bridge this modality gap, we present diffusion-based SAR-to-EO image translation (DSE), a novel framework designed specifically for enhancing the interpretability of SAR imagery in flood scenarios. Unlike conventional GAN-based approaches, our DSE leverages the Brownian Bridge Diffusion Model to achieve stable and high-fidelity EO synthesis. Furthermore, it integrates a self-supervised SAR denoising module to effectively suppress SAR-specific speckle noise, thereby improving the quality of the translated outputs. Quantitative experiments on the SEN12-FLOOD dataset show that our method improves PSNR by 3.23 dB and SSIM by 0.10 over conventional SAR-to-EO baselines. Additionally, a user study with SAR experts revealed that flood segmentation performance using synthetic EO (SynEO) paired with SAR was nearly equivalent to using true EO–SAR pairs, with only a 0.0068 IoU difference. These results confirm the practicality of the DSE framework as an effective solution for EO image synthesis and flood interpretation in SAR-only environments.

Deep learning-based methods have achieved good performance in various recognition benchmarks mostly by utilizing single modalities. As different modalities contain complementary information to each other, multi-modal based methods are proposed to implicitly utilize them. In this paper, we propose a simple technique, called correspondence learning (CL), which explicitly learns the relationship among multiple modalities. The multiple modalities in the data samples are randomly mixed among different samples. If the modalities are from the same sample (not mixed), then they have positive correspondence, and vice versa. CL is an auxiliary task for the model to predict the correspondence among modalities. The model is expected to extract information from each modality to check correspondence and achieve better representations in multi-modal recognition tasks. In this work, we first validate the proposed method in various multi-modal benchmarks including CMU Multimodal Opinion-Level Sentiment Intensity (CMU-MOSI) and CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) sentiment analysis datasets. In addition, we propose a fraud detection method using the learned correspondence among modalities. To validate this additional usage, we collect a multi-modal dataset for fraud detection using real-world samples for reverse vending machines.