This study proposes a new CycleGAN-based stain transfer model, called S3CGAN, equipped with a specialized color classifier structure. The specialized color classifier can assist the generative network to conquer the existing challenge in GANs, namely the instability of the network caused by the insufficient representativeness of the training data in the initial stage of network training. The color classifier is pretrained, hence it can provide correct color information feedback to the generator during the initial network training phase. The augmented information from color classification enables the generator to generate superior results. Owing to the CycleGAN architecture, the proposed model does not require representative paired inputs. The proposed model uses U-Net and a Markovian discriminator to enhance the structural retention ability to generate images with high fidelity.