Unmanned aerial vehicle (UAV) imagery poses persistent challenges for object detection, including dense small objects, large-scale variation, cluttered backgrounds, and stringent localization requirements, where conventional two-stage detectors often fall short in fine-grained small-object representation, efficient global context modeling, and classification–localization consistency. We specifically target low-altitude UAV-captured imagery with highly flexible viewpoints (near-nadir to oblique) and frequent platform-induced motion blur, which makes dense small-object localization substantially more challenging than in conventional remote-sensing imagery. To address these issues, we propose CASA-RCNN, a context-adaptive and scale-aware two-stage detection framework tailored to UAV scenarios. CASA-RCNN introduces a shallow-level enhancement module, ConvSwinMerge, which strengthens position-sensitive cues and suppresses background interference by combining coordinate attention with channel excitation, thereby improving discriminative high-resolution features for small objects. For deeper semantic features, we incorporate an adaptive sequence modeling module based on MambaBlock to capture long-range dependencies and support context reasoning in crowded or occluded scenes with practical computational overheadon a desktop GPU. In addition, we adopt Varifocal Loss for quality-aware classification to better align confidence scores with localization quality, and we design a ScaleAdaptiveLoss to dynamically reweight regression objectives across object scales, compensating for the reduced gradient contribution of small targets during training. Experiments on the VisDrone2021 validation benchmark show that CASA-RCNN achieves 22.9% mAP, improving Faster R-CNN by 9.0 points; it also reaches 36.6% mAP₅₀ and 25.7% mAP₇₅. Notably, performance on small objects improves to 12.5% mAP_s (from 6.9%), and ablation studies confirm the effectiveness and complementarity of the proposed components.