Existing machine vision-based roughness measurement methods cannot accurately measure the roughness of free-form surfaces (with large curvature variations). To overcome this problem, this paper proposes a roughness measurement method based on a generative adversarial network (GAN) and a BP neural network. Firstly, this method takes images and curvature of free-form surfaces as training samples. Then, GAN is trained for roughness measurement through each game between generator and discriminant network by using real samples and pseudosamples (from generator). Finally, the BP neural network maps the image discriminant value of GAN and radius of curvature into roughness value (Ra). Our proposed method automatically learns the features in the image by GAN, omitting the independent feature extraction step, and improves the measurement accuracy by BP neural network. The experiments show that the accuracy of the proposed roughness measurement method can measure free-form surfaces with a minimum roughness of 0.2 μm, and measurement results have a margin of 10%.