Laser-induced breakdown spectroscopy (LIBS), limited by matrix effects, self-absorption in complex samples, and ambient atmospheric influences, still requires further improvement in detection sensitivity and signal stability. In this work, the excitation beam of LIBS is modulated into an optical vortex by an optical phase element, and optical vortex-induced LIBS is used to detect and analyze coal samples. Building on the uniform annular intensity distribution and orbital angular momentum (OAM) carried by the optical vortex, it is anticipated that spectral signal intensity can be enhanced by improving plasma ablation efficiency, reducing shielding effects, and increasing electron collision frequency, thereby reducing signal uncertainty and enhancing LIBS analytical performance. For the first time, a classification model combining principal component analysis (PCA) and support vector machine (SVM) is developed, integrating optical vortex-induced LIBS technology with machine learning. Using the PCA-SVM model, optical vortex-based LIBS attained a coal classification accuracy of 95%, significantly higher than the 88% achieved with Gaussian beams, thereby markedly improving classification performance for complex matrix samples. These results demonstrate that optical vortex-induced LIBS possesses strong potential for efficient detection of samples with complex matrices.