Woven fabric composite structures are applied in a wide range of industrial applications. Composite structures are vulnerable to damage from working in complex conditions and environments, which threatens the safety of the in-service structure. Damage detection based on Lamb waves is one of the most promising structural health monitoring (SHM) techniques for composite materials. In this paper, based on guided Lamb waves, a lightweight deep-learning approach is proposed for identifying damaged regions in woven fabric composite structures. The designed deep neural networks are built using group convolution and depthwise separated convolution, which can reduce the parameters considerably. The input of this model is a multi-channel matrix transformed by a one-dimensional guided wave signal. In addition, channel shuffling is introduced to increase the interaction between features, and a multi-head self-attention module is designed to increase the model’s global modeling capabilities. The relevant experimental results show that the proposed SHM approach can achieve a recognition accuracy of 100% after only eight epochs of training, and the proposed LCANet has only 4.10% of the parameters of contrastive SHM methods, which further validates the effectiveness and reliability of the proposed method.