In order to achieve high precision from non-contact temperature measurement, the hardware structure of a broadband correlative microwave radiometer, calibration algorithm, and temperature inversion algorithm are innovatively designed in this paper. The correlative radiometer is much more sensitive than a full power radiometer, but its accuracy is challenging to improve due to relatively large phase error. In this study, an error correction algorithm is designed, which reduces the phase error from 69.08° to 4.02°. Based on integral calibration on the microwave temperature measuring system with a known radiation source, the linear relationship between the output voltage and the brightness temperature of the object is obtained. Since the metal aluminum plate, antenna, and transmission line will have a non-linear influence on the receiver system, their temperature characteristics and the brightness temperature of the object are used as the inputs of the neural network to obtain a higher accuracy of inversion temperature. The temperature prediction mean square error of a back propagation (BP) neural network is 0.629 °C, and its maximum error is 3.351 °C. This paper innovatively proposed the high-precision PSO-LM-BP temperature inversion algorithm. According to the global search ability of the particle swarm optimization (PSO) algorithm, the initial weight of the network can be determined effectively, and the Levenberg–Marquardt (LM) algorithm makes use of the second derivative information, which has higher convergence accuracy and iteration efficiency. The mean square error of the PSO-LM-BP temperature inversion algorithm is 0.002 °C, and its maximum error is 0.209 °C.