High-speed ultrasonic vibration cutting (HUVC) applies an ultrasonic vibration to a cutting tool transverse to the cutting direction. The advantage of this method is that the high-speed cutting exceeds the critical cutting speed restrictions of ultrasonic vibration cutting. However, irregular dimple arrays are machined on the work-piece surface, increasing the surface roughness. In this paper, a phase shift control method is proposed, which aims to overlap dimples in a regular pattern to decrease the surface roughness for HUVC. First, this paper investigates the theory of phase shift control to guarantee the surface quality of HUVC. Then, a closed-loop direct digital synthesis (DDS) system is designed to keep the phase shift of the machined dimples stable and controllable. A contrast experiment is conducted to compare the performance of the closed-loop DDS and the open-loop DDS methods in HUVC process. The results show that the closed-loop DDS method can render the dimple phase stable and controllable during the HUVC process. The phase shift value influences the surface roughness. Setting the phase shift to 0.94π with an appropriate feed-rate, the surface roughness is reduced to 0.303, compared to a surface roughness of 0.785 when using the open-loop DDS method.