Ethnic minority embroidery from Guizhou is an important part of Chinese culture, reflecting the history, beliefs, and artistic traditions of the region’s diverse ethnic groups. However, challenges in automatic recognition arise due to data scarcity, complex textures, and the flexibility of handmade designs. This study constructs the Guizhou Province Intangible Cultural Heritage Embroidery dataset and proposes an improved MobileViT-DDC model to address the issues of complex textures and data scarcity. The model integrates Dilatefomer, Deformable Dilatefomer (DefDilatefomer), and Context Broadcasting Module (CBM) to capture local details and global information in embroidery patterns. Experimental results show that the MobileViT-DDC model achieves an accuracy of 98.40% on the Guizhou embroidery dataset (a 2.17% improvement over the original baseline model) with a 14% reduction in computational load; on the Pakistani National Dress Dataset, it reaches an accuracy of 79.07%, representing a 2.63% increase compared to the original baseline model of the same scale. This study is the first to apply a CNN-ViT hybrid model to ethnic embroidery recognition, providing a new solution for the digital preservation of cultural heritage. The model’s cross-cultural adaptability was further validated through its application to the Pakistani National Dress dataset.