To stabilize the frequency difference of the two-cavity dual-frequency Nd:YAG laser at 1064 nm, a scheme of the phase modulation of the orthogonally and linearly polarized dual-frequency laser using single electro-optic phase modulator has been proposed, and a frequency-difference stabilizing system for a two-cavity dual-frequency Nd:YAG laser based on the single-modulator quadrature-demodulated Pound–Drever–Hall (SM-QD-PDH) method has been designed and established. The characteristics of phase modulation, frequency discrimination, frequency stabilization, and frequency-difference stabilization of the orthogonally and linearly polarized dual-frequency laser with a frequency difference of ∼24 GHz at 1064 nm have been investigated experimentally. The experimental results have indicated that during a period of 1 h, the laser frequency stabilities of the linear and right-angle cavities are estimated by Allan variance to be better than 1.6 × 10 − 11 and 2.0 × 10 − 11, respectively, corresponding to a frequency-difference stability of better than 2.9 × 10 − 7. The designed SM-QD-PDH frequency-difference stabilizing system is not only simple but also has better performances than that of the double-modulator QD-PDH frequency-difference stabilizing system. Such a frequency-difference-stabilized two-cavity dual-frequency Nd:YAG laser can be used as an ideal light source for the synthetic-wave absolute-distance interferometric system.