For the purpose of maintaining and prolonging the service life of civil constructions, structural damage must be closely monitored. Monitoring the incidence, formation, and spread of damage is crucial to ensure a structure’s ongoing performance. This research proposes a unique approach for multiclass damage detection using acceleration responses based on synchrosqueezing transform (SST) together with deep learning algorithms. In particular, our pipeline is able to classify correctly the time series representing the responses of accelerometers placed on a bridge, which are classified with respect to different types of damage scenarios applied to the bridge. Using benchmark data from the Z24 bridge for multiclass classification for different damage situations, the suggested method is validated. This dataset includes labeled accelerometer measurements from a real-world bridge that has been gradually damaged by various conditions. The findings demonstrate that the suggested approach is successful in exploiting pre-trained 2D convolutional neural networks, obtaining a high classification accuracy that can be further boosted by the application of simple voting methods.