The widespread deployment of unmanned aerial vehicles has raised growing privacy and security concerns, creating an urgent need for effective drone detection and identification. To address these challenges, this paper proposes a semantic learning framework termed Drone Time–Frequency-Semantic Network (DroneTFS-SemNet) for drone recognition. The main contributions are as follows. A Temporal Enhancement module is designed to improve model adaptability and robustness through radio frequency (RF) signal noise simulation, feature pre-extraction, learnable frequency-band partitioning, and an attention mechanism. A Cross-Channel Recalibration Module is introduced to promote cross-channel interaction of RF features via dilated convolutions and Channel Shuffle, thereby enhancing the model’s learning capability. Third, Feature Fusion Attention is employed to weight, align, and fuse RF features from different levels, thereby improving the model’s ability to represent differences among drone categories and operation modes. A semantic construction and decision module is further developed to extract high-level semantic features and accomplish drone detection, classification, and recognition. Experimental results on the DroneRF dataset show that DroneTFS-SemNet achieves a drone classification accuracy of 99.73% and an operation mode recognition accuracy of 99.65%, outperforming multiple comparison methods. These results demonstrate that the proposed model has strong recognition capability on this public dataset and under controlled experimental conditions, providing an effective method for RF-based drone recognition research.