To address the challenges of weak model generalization and limited model capacity adaptation in traditional malware detection methods, this article presents a novel malware detection approach based on stacked depthwise separable convolutions and self-attention, termed CoAtNet. This method combines the strengths of the self-attention module’s robust model adaptation and the convolutional networks’ powerful generalization abilities. The initial step involves transforming the malicious code into grayscale images. These images are subsequently processed using a detection model that employs stacked depthwise separable convolutions and an attention mechanism. This model effectively recognizes and classifies the images, automatically extracting essential features from malicious software images. The effectiveness of the method was validated through comparative experiments using both the Malimg dataset and the augmented Blended+ dataset. The approach’s performance was evaluated against popular models, including XceptionNet, EfficientNetB0, ResNet50, VGG16, DenseNet169, and InceptionResNetV2. The experimental results highlight that the model surpasses other malware detection models in terms of accuracy and generalization ability. In conclusion, the proposed method addresses the limitations of traditional malware detection approaches by leveraging stacked depthwise separable convolutions and self-attention. Comprehensive experiments demonstrate its superior performance compared to existing models. This research contributes to advancing the field of malware detection and provides a promising solution for enhanced accuracy and robustness.