When magnetic matching aided navigation is applied to an underwater vehicle, the magnetometer must be installed inside the vehicle, considering the navigation safety and concealment of the underwater vehicle. Then, the interference magnetic field will seriously affect the accuracy of geomagnetic field measurement, which directly affects the accuracy of geomagnetic matching aided navigation. Therefore, improving the accuracy of geomagnetic measurements inside the vehicle through error compensation has become one of the most difficult problems that requires an urgent solution in geomagnetic matching aided navigation. In order to solve this problem, this paper establishes the calculation model of the internal magnetic field of the underwater vehicle and the geomagnetic measurement error model of the ship-borne magnetometer. Then, a compensation method for the geomagnetic measurement error of the ship-borne magnetometer, based on the constrained total least square method, is proposed. To verify the effectiveness of the method proposed in this paper, a simulation experiment of geomagnetic measurement and compensation of a ship-borne three-axis magnetometer was constructed. Among them, to be closer to the real situation, a combination of the geomagnetism model, the elliptic shell model and the magnetic dipole model was used to simulate the internal magnetic field of the underwater vehicle. The experimental results indicated that the root mean square error of geomagnetic measurement in an underwater vehicle was less than 5 nT after compensation, and the accuracy of geomagnetic measurement met the requirements of geomagnetic matching aided navigation.