With the rapid development of the unmanned aerial vehicles (UAVs) industry, there is increasing demand for UAV surveillance technology. Automatic Dependent Surveillance-Broadcast (ADS-B) provides accurate monitoring of UAVs. However, the system cannot encrypt messages or verify identity. To address the issue of identity spoofing, radio frequency fingerprinting identification (RFFI) is applied for ADS-B transmitters to determine the true identities of UAVs through physical layer security technology. This paper develops an ensemble learning ADS-B radio signal recognition framework. Firstly, the research analyzes the data content characteristics of the ADS-B signal and conducts segment processing to eliminate the possible effects of the signal content. To extract features from different signal segments, a method merging end-to-end and non-end-to-end data processing is approached in a convolutional neural network. Subsequently, these features are fused through EL to enhance the robustness and generalizability of the identification system. Finally, the proposed framework’s effectiveness is evaluated using collected ADS-B data. The experimental results indicate that the recognition accuracy of the proposed ELWAM-CNN method can reach up to 97.43% and have better performance at different signal-to-noise ratios compared to existing methods using machine learning.