The positioning of civil aviation aircraft relative to a geographic reference point on Earth in a Cartesian frame is significant to detect the deviations from the desired path, especially for high-altitude airports or special airports based on performance-based navigation (PBN). To obtain these critical deviations during aircraft approach and landing, it is fundamental to estimate the continuous flight variables and discrete flight modes simultaneously with enough accuracy. With the coordinate conversion between the North, East, and Down (NED) frame and the geographic coordinate system based on World Geodetic System 1984 (WGS-84) considered, this study proposed a non-linear stochastic hybrid estimation algorithm with adaptive square-root unscented particle filtering (ASR-UPF) to estimate the true path. The probabilities of mode transition, represented by the normal cumulative density function of continuous states, determine whether to proceed with mode transitions. In addition, the adaptive update characterized by tracking variable noise and the importance sampling distributions based on the results of square-root unscented Kalman filtering (SR-UKF), as a comparative study of continuous system filtering, were used. The experiments illustrated the ASR-UPF is able to reduce the state estimation error more effectively, and more promptly track the error caused by incorrect mode estimation with adaptability compared to the SR-UKF. A further test with real flight data indicates that the proposed method gives the refined estimation of position and azimuth in NED frame.