This paper researches the recognition of modulation signals in underwater acoustic communication, which is the fundamental prerequisite for achieving noncooperative underwater communication. In order to improve the accuracy of signal modulation mode recognition and the recognition effects of traditional signal classifiers, the article proposes a classifier based on the Archimedes Optimization Algorithm (AOA) and Random Forest (RF). Seven different types of signals are selected as recognition targets, and 11 feature parameters are extracted from them. The decision tree and depth obtained by the AOA algorithm are calculated, and the optimized random forest after the AOA algorithm is used as the classifier to achieve the recognition of underwater acoustic communication signal modulation mode. Simulation experiments show that when the signal-to-noise ratio (SNR) is higher than −5dB, the recognition accuracy of the algorithm can reach 95%. The proposed method is compared with other classification and recognition methods, and the results show that the proposed method can ensure high recognition accuracy and stability.