Real-time Unmanned Aerial Vehicle (UAV) detection is a growing research field centered on advanced computer vision and deep learning algorithms. However, the rise of unmanned aerial vehicles (UAVs) has sparked numerous concerns due to their potential for malicious use in illegal activities. To address these concerns, Vision-based object detection approaches for UAVs have recently been developed. Nonetheless, UAV detection in real-world scenarios, such as images with diverse backgrounds and various perspectives, remains underexplored. To fill this gap, we present a new UAV detection dataset called the real-world scenarios dataset (RWSD). This dataset leverages real-world footage and is constructed under challenging conditions, including complex backgrounds, varying UAV sizes, different perspectives, and multiple UAV types. It aims to support the development of robust UAV detection algorithms that can perform well in diverse and realistic conditions. YOLO, a popular one-stage object detection approach, is widely employed for UAV detection across different environments due to its efficiency and simplicity. However, this series of detectors encounters challenges in real-world scenarios, such as excessive computation and suboptimal detection rates. In this study, we propose a lightweight feature enhancement model (LFEM) to address these limitations. Specifically, we base our model on YOLOv5, introducing the Ghost module to improve UAV detection with fewer floating-point operations (FLOPs). Additionally, we incorporate the SIMAM module to enhance feature representation, particularly for real-world scenarios. Extensive experiments on the RWSD, UAVDT, and DOTAv1.0 datasets demonstrate the effectiveness of our approach. Our proposed LFEM achieves an impressive 93.2% mAP₅₀, outperforming baseline models while maintaining a lightweight profile. Comparative and ablation studies further confirm that our algorithm is a promising and efficient solution for practical UAV detection tasks.