Changdae Oh

Changdae Oh

Auto-generated profile, based on publicly available information

Changdae Oh

University of Seoul

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 March 2024 in Pattern Recognition
Verified
Journal Article
Closed access
JKJinho Kang
Jinho Kang
TKTaero Kim
Taero Kim
YKYewon Kim
Yewon Kim
COChangdae Oh
Changdae Oh
JJJiyoung Jung
Jiyoung Jung
KSKyungwoo Song
Kyungwoo Song

12 April 2024 in Expert Systems with Applications
Verified
Journal Article
Closed access
COChangdae Oh
Changdae Oh
MPMinhoi Park
Minhoi Park
SLSungjun Lim
Sungjun Lim
KSKyungwoo Song
Kyungwoo Song

7 November 2023 in arXiv
Verified
Preprint
Open access
PDF file
COChangdae Oh
Changdae Oh
JSJunhyuk So
Junhyuk So
HBHoyoon Byun
Hoyoon Byun
YLYongTaek Lim
YongTaek Lim
MSMinchul Shin
Minchul Shin
KSKyungwoo Song
Kyungwoo Song
Pre-trained multi-modal models, such as CLIP, provide transferable embeddings and show promising results in diverse applications. However, the analysis of learned multi-modal embeddings is relatively unexplored, and the embedding transferability can be improved. In this work, we observe that CLIP holds separated embedding subspaces for two different modalities, and then we investigate it through the lens of uniformity-alignment to measure the quality of learned representation. Both theoretically and empirically, we show that CLIP retains poor uniformity and alignment even after fine-tuning. Such a lack of alignment and uniformity might restrict the transferability and robustness of embeddings. To this end, we devise a new fine-tuning method for robust representation equipping better alignment and uniformity. First, we propose a Geodesic Multi-Modal Mixup that mixes the embeddings of image and text to generate hard negative samples on the hypersphere. Then, we fine-tune the model on hard negatives as well as original negatives and positives with contrastive loss. Based on the theoretical analysis about hardness guarantee and limiting behavior, we justify the use of our method. Extensive experiments on retrieval, calibration, few- or zero-shot classification (under distribution shift), embedding arithmetic, and image captioning further show that our method provides transferable representations, enabling robust model adaptation on diverse tasks. Code: https://github.com/changdaeoh/multimodal-mixup

8 July 2023 in arXiv
Verified
Preprint
Open access
PDF file
COChangdae Oh
Changdae Oh
HHHyeji Hwang
Hyeji Hwang
HLHee-Young Lee
Hee-Young Lee
YLYongTaek Lim
YongTaek Lim
GJGeunyoung Jung
Geunyoung Jung
KSKyungwoo Song
Kyungwoo Song
With the surge of large-scale pre-trained models (PTMs), fine-tuning these models to numerous downstream tasks becomes a crucial problem. Consequently, parameter efficient transfer learning (PETL) of large models has grasped huge attention. While recent PETL methods showcase impressive performance, they rely on optimistic assumptions: 1) the entire parameter set of a PTM is available, and 2) a sufficiently large memory capacity for the fine-tuning is equipped. However, in most real-world applications, PTMs are served as a black-box API or proprietary software without explicit parameter accessibility. Besides, it is hard to meet a large memory requirement for modern PTMs. In this work, we propose black-box visual prompting (BlackVIP), which efficiently adapts the PTMs without knowledge about model architectures and parameters. BlackVIP has two components; 1) Coordinator and 2) simultaneous perturbation stochastic approximation with gradient correction (SPSA-GC). The Coordinator designs input-dependent image-shaped visual prompts, which improves few-shot adaptation and robustness on distribution/location shift. SPSA-GC efficiently estimates the gradient of a target model to update Coordinator. Extensive experiments on 16 datasets demonstrate that BlackVIP enables robust adaptation to diverse domains without accessing PTMs' parameters, with minimal memory requirements. Code: \url{https://github.com/changdaeoh/BlackVIP}