Blueberries, as one of the more economically rewarding fruits in the fruit industry, play a significant role in fruit detection during their growing season, which is crucial for orchard farmers’ later harvesting and yield prediction. Due to the small size and dense growth of blueberry fruits, manual detection is both time-consuming and labor-intensive. We found that there are few studies utilizing drones for blueberry fruit detection. By employing UAV remote sensing technology and deep learning techniques for detection, substantial human, material, and financial resources can be saved. Therefore, this study collected and constructed a UAV remote sensing target detection dataset for blueberry canopy fruits in a real blueberry orchard environment, which can be used for research on remote sensing target detection of blueberries. To improve the detection accuracy of blueberry fruits, we proposed the PAC3 module, which incorporates location information encoding during the feature extraction process, allowing it to focus on the location information of the targets and thereby reducing the chances of missing blueberry fruits. We adopted a fast convolutional structure instead of the traditional convolutional structure, reducing the model’s parameter count and computational complexity. We proposed the PF-YOLO model and conducted experimental comparisons with several excellent models, achieving improvements in mAP of 5.5%, 6.8%, 2.5%, 2.1%, 5.7%, 2.9%, 1.5%, and 3.4% compared to Yolov5s, Yolov5l, Yolov5s-p6, Yolov5l-p6, Tph-Yolov5, Yolov8n, Yolov8s, and Yolov9c, respectively. We also introduced a non-maximal suppression algorithm, Cluster-NMF, which accelerates inference speed through matrix parallel computation and merges multiple high-quality target detection frames to generate an optimal detection frame, enhancing the efficiency of blueberry canopy fruit detection without compromising inference speed.