An in situ method has been proposed for measuring all nonorthogonal angles of the beams and triaxial coils in a three-axis vector optically pumped magnetometer (OPM). Based on the combination and alignment of two pump beams, each of which can achieve biaxial transverse measurement in a time-sharing regime under modulated magnetic fields, a 3-D orthogonal coordinate measurement system is obtained, and the measurement models for three categories (six types) of nonorthogonal angles are established theoretically. On this basis, we experimentally measured the nonorthogonal angles along with related uncertainties at different modulated frequencies and strengths of the magnetic fields. The nonorthogonal angles were (3.65° ± 0.52°) between the two pump beams before compensation. They were (7.05° ± 0.23°) between the $X$ - and $Y$ -axis coils, (8.89° ± 0.21°) between the $Y$ - and $Z$ -axis coils, (5.23° ± 0.27°) between the $X$ - and $Z$ -axis coils, (7.24° ± 0.24°) between the $x$ -direction pump beam and $X$ -axis coil, and (9.10° ± 0.22°) between the $z$ -direction pump beam and the $Z$ -axis coil. This study measures the magnitudes of six types of nonorthogonality, without making an assumption of ideal orthogonality or using auxiliary calibration equipment. Moreover, this method can be extended to several other kinds of OPMs with simple modifications. This study is critical for evaluating and reducing the misalignment error to improve the accuracy of three-axis vector OPMs.

Triaxial uniform coils installed in the magnetically shielded cylinder are essential components of an atomic magnetometer to achieve ultra-sensitive measurements and are used to compensate for the ambient magnetic remanence in the magnetometer’s operating area. This paper proposes the new three-dimensional octagonal coils configured with a two-pair octagonal coil and two double-nested octagonal coils, which differs from the conventional square or circular shim coils. The spatially homogeneous distribution of the magnetic field of the three-dimensional shim coils based on this configuration is simulated and evaluated, and then the fabrication and experimental tests are completed. The theoretical calculation results show that the magnetic field uniformity of the z-direction two-pair octagonal coil is improved by about one order of magnitude compared with that of the conventional coil such as the Lee-Whiting coil and the Rubens coil. The magnetic field uniformity of x-direction or y-direction double nested octagonal coil is promoted by about two orders of magnitude compared with that of saddle coil. The experimental magnetic field distribution is consistent with the theoretical value. This configuration can reduce the outer diameter of the coil under the premise of meeting the requirement of setting the magnetic field uniform zone, and it is easier to install than the cylindrical coil assemblies.