Weathering steel is a steel material that produces a protective rust layer on the surface of the steel that suppresses the propagation of corrosion. Periodic inspection of the rust condition is necessary to maintain this steel in good condition, and CNN-based methods for evaluating the appearance of rust have been studied. One of the problems in improving the classification accuracy of the CNN is how to prepare the rust images for training. Since it is difficult to collect rust images in the real world, it has been studied to train CNN with rust images generated by GANs. Conventional GANs have the problem that when a large number of images are generated, the generated images are biased toward similar ones. The scope of this paper is to improve the accuracy of CNN using GAN-generated images in classification tasks of rust conditions of weathering steel. The specific objectives of this study are to improve the diversity of images generated by GANs and thereby improve the accuracy of CNNs. To improve image diversity in GAN-based image generation, we propose to use Wasserstein GAN with Gradient Penalty (WGAN-GP), in addition to Conditional GAN (CGAN), and to introduce SSIM (Structural Similarity) and LBP (Local Binary Pattern) as loss functions, in addition to L1 and L2 losses. Then, the images generated by the proposed five types of GANs are applied to the training of CNNs. Comparison between the proposed GANs and the conventional CGAN shows that 1) the proposed GANs can reduce the SSIM similarity of the generated images by up to 1/4, and 2) in the classification of three classes of rust images, the use of the generated images of the proposed GANs can improve the F1 score by up to about 4%pt. Overall analysis shows that WGAN-GP-based is better than CGAN-based and that the newly introduced SSIM, LBP, L1 and L2 losses improve both similarity and classification accuracy.