Reliable altitude and vertical speed estimation are fundamental for unmanned aerial vehicle (UAV) autonomous flight, especially during low-altitude operations such as takeoff and landing. Barometric altimeters are widely used due to their low cost, high availability, and good long-term stability, providing smooth altitude trends over a wide operating range. However, barometric measurements are indirectly inferred from static pressure and are therefore sensitive to local airflow disturbances. In particular, rotor downwash and ground effect-induced pressure perturbations near the surface can introduce significant biases and short-term fluctuations in barometric altitude, which propagate into erroneous vertical speed estimates during critical flight phases. Time-of-flight (TOF) altimeters, such as radar or laser sensors, provide direct above-ground-level (AGL) measurements and are largely insensitive to ground effect-related pressure disturbances. Within their limited operational range, TOF altimeters typically offer higher accuracy and lower short-term noise compared with barometric altitude. Nevertheless, TOF sensors are characterized by a restricted valid measurement range and frequently exhibit non-ideal behaviors in real-world UAV operations, including out-of-range outputs, frozen measurements, and in-range biased readings. These anomalies violate the nominal sensor assumptions used in conventional Kalman filter-based fusion and can significantly degrade estimation performance if not properly handled. This paper proposes a hybrid Kalman–rule-based altitude estimation framework that fuses barometric and TOF altitude measurements to exploit their complementary characteristics while mitigating their respective limitations. A vertical dynamic state-space model is formulated to jointly estimate altitude, vertical velocity, accelerometer bias, and ground height offset. A rule-based anomaly detection and classification module is developed to identify multiple TOF altimeter failure modes observed in operational UAV flights. The detected anomaly states are incorporated into the Kalman filter to adaptively weight, accept, or reject TOF measurements, thereby improving robustness against sensor non-idealities. The proposed approach is validated using 39 real UAV flight logs covering diverse flight regimes, including low-altitude maneuvers, cruise, and autonomous landing. Experimental results show that the proposed framework provides more stable and robust altitude and vertical speed estimation under practical sensor anomaly conditions compared with conventional barometer-only and standard Kalman fusion configurations. These results demonstrate the practical effectiveness of the proposed method for fault-aware altitude estimation in UAV autonomous flight.