Accurately diagnosing compound faults in gearboxes, where multiple fault modes co-occur, poses significant challenges, particularly when labeled training data is limited. This paper presents a novel gearbox fault diagnosis model integrating advanced signal processing, multi-scale feature extraction, and an optimized Support Vector Machine (SVM). The proposed model employs Variational Mode Decomposition (VMD) for signal denoising and decomposition, with parameters optimized using PSO and permutation entropy as the fitness function. A two-stage feature extraction process selects the most informative IMFs using Pearson correlation analysis and applies Wavelet Packet Decomposition (WPD) to extract multi-scale features. Energy and signal-to-noise ratio (SNR) analysis further refine the feature set by selecting the most discriminative wavelet packet component. Finally, a hybrid Genetic Algorithm-Particle Swarm Optimization (GA-PSO) approach is employed to optimize the SVM’s hyperparameters, enhancing its classification accuracy. Experimental evaluation using the PHM2009 Data Challenge dataset demonstrates that the proposed GA-PSO-SVM model achieves a higher fault identification accuracy (96.63%) than other SVM models with single optimization techniques and benchmark models like XGBoost and CNN. The results highlight the effectiveness of the hybrid optimization strategy and the model’s suitability for compound fault diagnosis with limited data. However, analysis of misclassifications reveals challenges in distinguishing fault classes with subtle differences and those involving highly complex, multi-component failures, suggesting avenues for future research.