Reliable detection of idler faults in belt conveyor systems is essential for minimising unplanned downtime, improving safety, and optimising maintenance. Conventional vibration- and acoustic-based approaches, which use fixed sensors or handheld microphones, often fail under non-stationary, noisy, or spatially variable conditions. To address these challenges, we propose a robust two-phase transfer learning framework leveraging acoustic drone data for cross-domain, non-invasive idler fault diagnosis. The framework combines pretrained Convolutional Neural Network (CNN) backbones, including ResNet-18, DenseNet-121, ShuffleNet-V2, and EfficientNet-B0, with Transformer encoders to capture local spectral and long-range temporal dependencies. Both Mel-spectrograms and MFCCs are employed to extract discriminative spectral-temporal fault features. A Mean-Covariance Alignment (MCA) mechanism is introduced to facilitate domain-invariant feature learning through two sequential phases: (i) source-domain pretraining on handheld recordings to establish robust acoustic representation and (ii) target-domain fine-tuning on drone data using progressive layer unfreezing and discriminative learning rates. Extensive experiments validate the effectiveness of the proposed framework. In Phase one, the ResNet-18–Transformer hybrid achieved a cross-fold mean accuracy on, with a 95% confidence interval ranging from 96.4% to 97.6%. handheld data, Ensemble learning further improved performance to 98.0%, indicating robust feature learning and effective transferability. In phase two, MCA outperformed the cross-entropy (CE) baseline when fully fine-tuning Mel-spectrogram features. The accuracy went from 84.7% (CE) to 86.0%, and the F1-score went from 83.4% to 85.6%. This was better than other domain adaptation methods like MMD, CMD, CORAL, and SWD. While MFCC features achieved higher baseline performance, they generally experienced drops under MCA adaptation. Across adaptation strategies and fine-tuning setups, Mel-spectrogram features demonstrated greater robustness, often maintaining or slightly improving performance, whereas MFCCs were more sensitive to domain adaptation. An ablation study further confirmed that aligning both first- and second-order statistics is crucial for achieving consistent improvements. These results indicate that MCA enhances CNN-Transformer transferability and highlight the reliability of Mel-spectrogram features for handheld-to-drone fault diagnosis in real industrial environments.