Zirui Song, Bin Yan, Yuhan Liu, Miao Fang, Mingzhe Li, Rui Yan, Xiuying Chen. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.

Vision-Language-Action (VLA) models are advancing faster than the field can evaluate them reliably. Researchers use different metrics and lab-specific protocols, making it hard to tell whether reported gains reflect genuine progress or favorable evaluation choices. We present the first comprehensive benchmark-centric survey of VLA evaluation, covering 582 papers from 2023 to May 2026. We argue that a benchmark number supports a progress claim only if four conditions hold: the benchmark discriminates among top models; metrics capture the claimed capability; the procedure permits cross-paper comparison; and the inference from benchmark to deployment is valid. Current practice fails at all four. Benchmark choice is concentrated and saturated, with leading models clustering near the ceiling of dominant simulation suites. Metric reporting is one-dimensional, dominated by task success rate while efficiency, safety, and trajectory consistency remain underreported. Real-world evaluation is fragmented, with no widely adopted standard and few trials per task. Simulation scores are widely treated as evidence of real-world capability, yet standard task-centric suites correlate only weakly with real-robot performance unless explicitly calibrated to the target physical setup. These failures reveal a fundamental evaluation bottleneck: VLA models are advancing faster than our ability to measure that advance.