Distinguishing Alzheimer’s Disease (AD) from Mild Cognitive Impairment (MCI) is challenging due to their subtle morphological similarities in MRI, yet distinct therapeutic strategies are required. To assist junior clinicians with limited diagnostic experience, this paper proposes Vi-ADiM, a Vision Transformer framework designed for the early differentiation of AD and MCI. Leveraging cross-domain feature adaptation and task-specific data augmentation, the model ensures rapid convergence and robust generalization even in data-limited regimes. By optimizing a two-stage encoding module, Vi-ADiM efficiently extracts both global and local MRI features. Furthermore, by integrating SHAP and Grad-CAM++, the framework offers multi-granular interpretability of pathological regions, providing intuitive visual evidence for clinical decision-making. Experimental results demonstrate that Vi-ADiM outperforms the standard ViT-Base/16, improving accuracy, precision, recall, and F1 score by 0.444%, 0.486%, 0.476%, and 0.482%, respectively, while reducing standard deviations by approximately 0.06–0.29%. Notably, the model achieves these gains with a 48.96% reduction in parameters and a 49.65% decrease in computational cost (FLOPs), offering a reliable, efficient, and interpretable solution for computer-aided diagnosis.