Accurate classification of rock fracture modes is essential for understanding rock mass instability mechanisms. To address the limitation of traditional acoustic emission (AE) classification methods that treat a single AE signal as a single fracture event, overlooking its composite nature from multiple fracture events and leading to misclassification, this study proposes a novel rock fracture mode classification method based on AE spectral analysis. This study details the development framework, theoretical model, classification criteria, application process, and experimental validation of the new rock fracture mode classification method. Uniaxial compression tests on granite, marble, and limestone, along with rockburst simulation tests on granite, were conducted to validate the classification of fracture modes. In rockburst simulations, shear fracture signals accounted for 48% on average, composite signals 40%, and tensile signals 12%. The method effectively distinguishes multiple fracture events within a single AE signal, accurately classifies fracture modes, and elucidates the dynamic evolution of fracture modes during the rockburst precursor stage, offering significant advantages for rock fracture mode classification and mechanistic insight.