Accurate altitude estimation is critical for unmanned aerial vehicles (UAVs), yet barometric measurements are susceptible to atmospheric drift and dynamic disturbances. To address these limitations, this paper proposes a dual-layer, real-time differential barometric altimetry framework that integrates a ground reference station with an onboard fusion scheme based on Adaptive Weighted Averaging (AWA) and an Adaptive Extended Information Filter (AEIF). The ground reference station suppresses low-frequency atmospheric variations, while the onboard AEIF incorporates a physical pressure–height model and adaptive noise estimation to maintain a fast dynamic response. The proposed method is validated through numerical simulations, hardware-in-the-loop (HIL) experiments, and real flight tests. In a two-hour outdoor flight test, compared with barometric systems operating without a reference station, the proposed approach reduces the altitude RMSE from 4.05 m to 0.31 m, achieving an approximately order-of-magnitude improvement in representative scenarios and demonstrating decimeter-level altitude measurement accuracy.