Reliable perception of young apple fruits in natural orchards is a prerequisite for automated thinning and intelligent orchard management, yet remains difficult in real field conditions due to small fruit size, dense distribution, branch–leaf occlusion, background similarity, and severe illumination degradation at night. This study presents FruitDet, a lightweight multi-module detector designed for robust day–night young apple fruit detection in complex orchard environments. A field dataset was established in a high-density apple orchard in Aksu, Xinjiang, covering daylight and low-light night-time scenes with diverse occlusion, scale, and illumination variations. To improve detection robustness without sacrificing computational efficiency, FruitDet combines three complementary mechanisms: an inverted-bottleneck-based multi-scale feature enhancement module for preserving small-fruit details, a channel–spatial attention module for suppressing foliage and illumination interference, and a lightweight Transformer-based context module for modeling long-range dependencies between fruits and surrounding orchard structures. In daytime scenes, FruitDet achieved 91.904% precision, 77.557% recall, 83.254% mAP50, and 66.427% mAP50–95; in night-time scenes, it maintained 90.107% precision, 75.135% recall, 80.544% mAP50, and 64.719% mAP50–95. Compared with mainstream detectors including YOLOv5n, YOLOv8n, YOLO11n, YOLO26n, Faster R-CNN, RT-DETR, and RT-DETRv2, FruitDet consistently delivered higher accuracy across lighting conditions. Ablation, visualization, public-dataset testing, and edge-deployment experiments verified that the proposed modules jointly improve small-object representation, background discrimination, low-light robustness, and real-time applicability. With 2.960 M parameters, 3.726 G FLOPs, and approximately 180 FPS, FruitDet offers a practical and efficient visual perception approach for Young fruit monitoring was conducted under both daytime and night-time orchard conditions covered in this study. All-weather orchard monitoring and robotic young-fruit thinning. The shareable data are available