An Optical Burst Switching (OBS) network is vulnerable to Burst Header Packet (BHP) flooding attack. In flooding attacks, edge nodes send BHPs at a high rate to reserve bandwidth for unrealized data bursts, which leads to a waste of bandwidth, a decrease in network performance, and massive data loss. Machine learning techniques are utilized to detect this attack in the OBS network. In this paper, we propose a particle swarm optimization–support vector machine (PSO-SVM) model for detecting BHP flooding attacks, in which the PSO is used to optimize the parameters of the SVM. We use the dataset provided by the UCI warehouse to train and test the model. The experimental results show that the detection accuracy of the PSO-SVM model reaches 95.0%, which is 9.4%, 9.6%, 20.7%, 8% higher than naïve Bayes, SVM, k-nearest neighbor, and decision tree. Although DCNN outperforms our model, it requires more processing and training time. Collectively, our approach is effective and high-efficiency in detecting flooding attacks in optical burst switching networks and maintaining network stability and security.