With the rapid development of Internet technology, information security has become increasingly important. Cryptographic analysis techniques, especially side-channel analysis (SCA), pose a significant threat to security systems. The latest SCA technology mainly utilizes the physical leakage signals generated during the operation of encryption devices, such as power consumption, temperature and electromagnetic radiation. These signals themselves carry the physical characteristics of the device, which are related to the encryption algorithm. Among them, the power consumption trace remains the main target of modern SCA research. However, such trajectories often bring about some analytical difficulties, such as the data sequence being too long, the feature points being distributed sparsely, and the internal relationships of the data being complex. These challenges hinder effective analysis. While Transformer architectures are good at capturing long-range dependencies in sequential data, their high computational complexity limits practical deployment. To address this, we propose replacing the self-attention (SA) module in Transformers with a focal modulation module. This modification significantly reduces computational complexity and reduces computational operations during feature extraction, enabling efficient and accurate side-channel attacks. Experimental results on benchmark datasets (ASCAD, AES_RD, AES_HD, DPAv4) demonstrate the superiority of our approach. The proposed method achieves a reduction in training time compared to standard Transformer models, and achieves superior key recovery performance, outperforming existing state-of-the-art models.