In this work, the mechanism of RF loss up to 110 GHz for N-polar GaN has been studied. With the assistance of S-parameter characterization combined with secondary ion mass spectroscopy analyses, the incorporated oxygen impurity has been identified to be the main source bringing about the severe RF loss of N-polar GaN. The compensation of Fe-doping enables an effective reduction in RF loss. Moreover, the unintentionally doped (UID) GaN layer grown on top of the Fe-doped GaN buffer requires a careful design due to the distinct memory effect of Fe-doping in N-polar GaN. With an optimization of its thickness, a very low RF loss of 0.36 dB/mm at 94 GHz has been attained. Furthermore, by fitting the Fe concentration profile of UID-GaN according to the mass balance rate equation, it is found that the desorption of Fe on the N-polar GaN surface is significant. A bond-based model is introduced to elucidate the difference of the Fe memory effect between Ga-polar and N-polar GaN.