Structural damage identification based on structural health monitoring (SHM) data and machine learning (ML) is currently a rapidly developing research area in structural engineering. Traditional machine learning techniques rely heavily on feature extraction, where weak feature extraction can lead to suboptimal features and poor classification performance. In contrast, ML-based methods, particularly deep learning approaches like convolutional neural networks (CNNs), automatically extract relevant features from raw data, improving the accuracy and adaptability of the damage identification process. This study developed a time–frequency-based data-driven approach aiming to improve the effectiveness of traditional data-driven structural damage identification approaches for large complex structures. Firstly, the structural acceleration signals in the time domain were converted into two-dimensional images via the Gram angle difference field (GADF). Subsequently, the characteristic feature in the image data was studied by convolutional neural networks (CNNs) to predict the structural damage conditions. An experimental study on a scale model of a cable-stayed bridge was conducted to identify the damage of stay cables under the moving vehicle load on the main girders. The CNN was employed to extract the characteristic features from the time-varying monitoring data of vehicle–bridge interactions. The CNN parameters were optimized to conduct the structural damage classification task. The performance of the proposed method was evaluated by comparing it with various traditional pre-trained networks. The effect of environmental noise on the prediction accuracy was also investigated. The numerical results show that the ResNet model has the best performance in terms of damage identification accuracy and convergence speed, achieving higher accuracy and faster convergence compared to the other four traditional networks. The method can accurately identify damage on bridges using insufficient sensors on the bridge deck, which has valuable potential for application to real-world bridges with monitoring data. As the Signal-to-Noise Ratio (SNR) decreases from 20 dB to 2.5 dB, the prediction accuracy of ResNet decreases from 86.63% to 62.5%, which demonstrates the robustness and reliability in identifying structural damage.