Monitoring data outliers comprises isolated mode outliers, cluster mode outliers, and normal points. To identify and distinguish the data hopping problems caused by outliers and environmental mutations in the displacement monitoring data of concrete AMS, this paper proposes a method based on wavelet transform, DBSCAN clustering algorithm combined with isolated forest and reinforcement learning algorithm to identify outliers in concrete dam monitoring data. In this paper, the trend line of measuring point data are extracted by the wavelet transform algorithm, and the residual data are obtained by subtracting it from the original process line. Subsequently, the DBSCAN clustering algorithm divides the residual data according to density. Therewith, the outlier scores of different data clusters are calculated, the iterative Q values are updated, and the threshold values are set. The data exceeding the threshold are finally marked as outliers. Finally, the water level and displacement data were compared by drawing the trend to ensure that the water level change did not cause the final identified concrete dam displacement data outliers. The results of the example analysis show that compared with the other two outlier detection methods (“Wavelet transform combined with DBSCAN clustering” or “W-D method”, “Wavelet transform combined with isolated forest method” or “W-IF method”). The method has the lowest error rate and the highest precision rate, recall rate, and F1 score. The error rate, precision rate, recall rate, and F1 score were 0.0036, 0.870, 1.000, and 0.931, respectively. This method can effectively identify data jumps caused by an environmental mutation in deformation monitoring data, significantly improve the accuracy of outlier identification, reduce the misjudgement rate of outliers, and have the highest detection accuracy.