As the threat from malicious UAVs continues to intensify, accurate identification of individual UAVs has become a critical challenge in regulatory and security domains. Existing single-signal analysis methods suffer from limited recognition accuracy. To address this issue, this paper proposes a high-precision individual identification method for UAVs based on FFS-SPWVD and DIR-YOLOv11. The proposed method first employs a frame-by-frame search strategy combined with the smoothing pseudo-Wigner–Ville distribution (SPWVD) algorithm to obtain effective time–frequency feature representations of flight control signals. Building on this foundation, the YOLOv11n network is adopted as the baseline architecture. To enhance the extraction of time–frequency texture features from UAV signals in complex environments, a Multi-Branch Auxiliary Multi-Scale Fusion Network is incorporated into the neck network. Meanwhile, partial space–frequency selective convolutions are introduced into selected C3k2 modules to alleviate the increased computational burden caused by architectural modifications and to reduce the overall number of model parameters. Experimental results on the public DroneRFb-DIR dataset demonstrate that the proposed method effectively extracts flight control frames and performs high-resolution time–frequency analysis. In individual UAV identification tasks, the proposed approach achieves 96.17% accuracy, 97.82% mAP50, and 95.29% recall, outperforming YOLOv11, YOLOv12, and YOLOv13. This study demonstrates that the proposed method achieves both high accuracy and computational efficiency in individual UAV recognition, providing a practical technical solution for whitelist identification and group size estimation in application scenarios such as border patrol, traffic control, and large-scale events.