In this paper, an object detection and recognition method based on improved YOLOv5 is proposed for application on unmanned aerial vehicle (UAV) aerial images. Firstly, we improved the traditional Gabor function to obtain Gabor convolutional kernels with better edge enhancement properties. We used eight Gabor convolutional kernels to enhance the object edges from eight directions, and the enhanced image has obvious edge features, thus providing the best object area for subsequent deep feature extraction work. Secondly, we added a coordinate attention (CA) mechanism to the backbone of YOLOv5. The plug-and-play lightweight CA mechanism considers information of both the spatial location and channel of features and can accurately capture the long-range dependencies of positions. CA is like the eyes of YOLOv5, making it easier for the network to find the region of interest (ROI). Once again, we replaced the Path Aggregation Network (PANet) with a Bidirectional Feature Pyramid Network (BiFPN) at the neck of YOLOv5. BiFPN performs weighting operations on different input feature layers, which helps to balance the contribution of each layer. In addition, BiFPN adds horizontally connected feature branches across nodes on a bidirectional feature fusion structure to fuse more in-depth feature information. Finally, we trained the overall improved YOLOv5 model on our integrated dataset LSDUVD and compared it with other models on multiple datasets. The results show that our method has the best convergence effect and mAP value, which demonstrates that our method has unique advantages in processing detection tasks of UAV aerial images.