Accurate detection and counting of open cotton bolls from unmanned aerial vehicle (UAV) RGB imagery are essential for organ-level cotton phenotyping and field monitoring, but remain challenging in high-density cotton fields because open bolls are small, densely distributed, partially occluded, and visually similar to plastic mulch, branches, senescent leaves, shadows, and drip-irrigation belts. To address these challenges, this study proposes OpenBoll-YOLO, a lightweight small-object detector designed for open cotton boll detection and counting in UAV nadir-view images. A UAV RGB dataset was collected at the boll-opening stage in Alar, Xinjiang, China, covering two cotton varieties and two acquisition dates. Based on YOLOv11s, OpenBoll-YOLO integrates three task-oriented components: a multi-kernel small-object enhancement pyramid (MSOEP) to preserve shallow spatial details and strengthen multi-scale feature fusion, a cross-stage partial block with a dynamic mixing layer (C2DML) to improve local structural discrimination under complex backgrounds, and a lightweight mixed aggregation network (LMANet) to enhance contextual representation with reduced model complexity. On the independent test set, OpenBoll-YOLO achieved 86.7% precision, 84.5% recall, 85.6% F1-score, 92.9% mAP@0.5, 81.5% mAP@0.75, and 71.7% mAP@0.5:0.95, with only 3.1 M parameters and an inference speed of 86 frames s⁻¹. Compared with YOLOv11s, it improved mAP@0.5 and mAP@0.5:0.95 by 2.4 and 5.9 percentage points, respectively, while reducing the parameter count by 67.0%. Counting evaluation further showed that OpenBoll-YOLO reduced the mean absolute error from 11.98 to 10.12 bolls image⁻¹ and increased R² from 0.87 to 0.91. These results demonstrate that OpenBoll-YOLO provides an accurate and lightweight solution for dense open cotton boll detection and counting in high-density field conditions.