Reliable UAV detection from ground-based cameras remains difficult under changing illumination, complex backgrounds, and residual displacement between visible and infrared sensors. This paper proposes the Bidirectional Feature Alignment and Transfer-Gated Fusion Network (BATF-Net), a dual-stream detector that preserves separate RGB- and infrared-reference coordinate systems. At the C3, C4, and C5 stages of a two-stream YOLOX-s architecture, an offset estimator shared across transfer directions predicts RGB-to-infrared and infrared-to-RGB displacement fields. Bilinear sampling maps each source features into the corresponding reference coordinates, after which a spatial gate derived from local response magnitude and post-alignment cosine similarity controls residual feature transfer. The network is trained end to end with two detection-head losses and requires neither pixel-level registration labels nor an auxiliary geometric objective. Under the stated validation protocol, the RGB-/infrared-reference ${\mathrm{AP}}_{50:95}$ values are 41.9%/56.9% on MM-UAV and 66.3%/60.6% on Anti-UAV. Compared with the RGB–infrared detector preceding MMA-SORT, these values are higher by 0.8/0.4 and 0.4/0.5 percentage points, respectively. BATF-Net requires 50.0 G FLOPs and runs at 167 frames/s, compared with 54.2 G FLOPs and 125 frames/s for the MMA-SORT detector on the reported platform. These results show that bidirectional alignment can improve paired RGB–infrared UAV detection while preserving sensor-specific outputs and real-time throughput on the evaluated platform.