While Vision-Language Models (VLMs) based on large-scale models have shown revolutionary advancements across various vision-language tasks, research on improving VLM robustness remains underexplored. Existing studies primarily focus on attacking VLM after the pretrained visual or textual encoders, typically requiring obvious noise or long inference time. In this study, we look into VLM structure and highlight alignment module’s role as a protective filter that enhances VLM robustness against various perturbations. Motivated by these insights, we investigate VLM from both user and model developer perspectives and introduce the alignment perturbation strategy, which consists of multimodal, visual, and textual perturbations. Multimodal perturbation aims to achieve targeted textual output generation and is further utilized to enhance VLM robustness. Minimal perturbations to visual or textual inputs can lead to significant changes in the overall output of VLMs, revealing their sensitivity to both visual and textual input variations. Building on the alignment perturbation strategy, we propose alignment robust training, which efficiently improves VLM robustness by finetuning the parameters of alignment module without excessive resource consumption. Experiment results across various tasks and models demonstrate the effectiveness of the proposed alignment perturbation and alignment robust training. These methods deepen the understanding of VLM robustness, allowing for secure and reliable deployment towards diverse real-world scenarios. Codes are available at https://github.com/zhangconghhh/RobustVLMs.

Accurate pavement crack segmentation is important for large-scale road assessment, but dense pixel-wise annotations are expensive and subjective.This paper studies unsupervised pavement crack segmentation under a human-mask-free optimization setting, where no human-annotated pixel-wise crack masks are used as training targets. Existing reconstruction-based unsupervised methods usually infer cracks from input--reconstruction residuals, but crack-suppressed reconstruction is only indirectly constrained and residual thresholding is sensitive to complex pavement textures.To address these issues, we propose CrackDiffSeg, a reconstruction-guided segmentation framework without pixel-wise mask supervision.CrackDiffSeg uses conditional diffusion reconstruction as a crack-free background prior to suppress sparse crack structures while preserving stochastic pavement textures, and learns pixel-wise crack posteriors from paired input--reconstruction representations instead of directly thresholding residual maps.To provide crack-specific training signals without real paired crack-clean annotations, we introduce tip-growth-guided pseudo crack synthesis, which generates morphology-consistent crack--clean pairs from crack-free pavement images.The model is further adapted to real unlabeled crack images through reliability-gated self-training, and an adaptive few-candidate single-update strategy is designed to reduce diffusion inference cost.Experiments on public pavement crack benchmarks show that CrackDiffSeg achieves competitive unsupervised segmentation performance without using human-annotated pixel-wise crack masks during training.