22 July 2026 in SSRN Electronic Journal
Vision foundation models based on the Vision Transformer (ViT) architecture have emerged aspowerful feature extractors deployed across diverse downstream tasks, yet they remain acutely vulnerable to adversarial perturbations that destabilize their representations even under imperceptible input modifications. Existing defenses-most notably adversarial training and certified randomized smoothing-achieve robustness at the cost of retraining or substantially modifying the backbone, impractical for large multi-task foundation models. We propose Robustness Tokens, a family of 𝜅 learnable token vectors appended to a frozen ViT backbone’s input sequence, optimized via a lightweight dual objective that preserves clean features while suppressing adversarial feature drift. Training takes under 300 gradient steps on a small ImageNet subset and updates a negligible fraction of parameters, leaving the backbone untouched. Across eleven architectures from two ViT families-DINOv2 and DeiT-III-Robustness Tokens raise feature-space cosine similarity under a non-adaptive PGD threat model above 0.82, versus undefended baselines as low as 0.011, with limited clean-performance degradation. Under an adaptive white-box threat model, cosine similarityfor DINOv2-B drops to 0.309, still well above the undefended baseline (0.044) but far from a full guarantee. Extending the adaptive evaluation to downstream tasks shows this gap is task-dependent: classification accuracy under adaptive attack collapses to 0.7%, indistinguishable from baseline, while segmentation retains a partial, order-of-magnitude benefit (27.1 vs. 5.2 mIoU). These results establish a parameter-efficient mechanism for hardening frozen foundation models against non-adaptive adversaries across two ViT pre-training paradigms, while leaving full white-box robustness,especially for classification, an open problem.