For systems of measurement, geometric errors such as manufacturing and assembly errors could have a significant impact on the accuracy of the angle encoders of the system. In this study, an error model of angular measurement with geometric errors of a torsional characteristic measurement system was developed based on multibody system theory, the aim of which was to reveal the impact of geometric errors on angular measurement and to compensate the measurement error. According to the principle of spatial error transfer, the decomposition of geometric errors is illustrated and the error matrix of geometric errors is constructed by the Denavit–Hartenberg (DH) method. Subsequently, an error compensation function can be obtained and the impact of geometric error on angular measurement is discussed. Finally, we demonstrate by the experimental results of an ultra-autocollimator that the proposed error compensation method reduced the angular measurement error from 3.7% to 0.7%, which shows that the proposed error model can effectively predict the angular measurement error. In addition, it can be seen from the measurement results of the RV reducer that the error of the torsional characteristic measurement system decreased significantly.