In this study, the average grain size was evaluated from a microstructure image using a convolutional neural network. Since the grain size in a microstructure image can be directly measured and verified in the original image, unlike the chemical composition or mechanical properties of material, it is more appropriate to validate the training results quantitatively. An analysis of microstructure images, such as grain size, can be performed manually or using image analysis software; however, it is expected that the analysis would be simpler and faster with machine learning. Microstructure images were created using a phase-field simulation, and machine learning was carried out with a convolutional neural network model. The relationship between the microstructure image and the average grain size was not judged by classification, as the goal was to have different results for each microstructure using regression. The results showed high accuracy within the training range. The average grain sizes of experimental images with explicit grain boundary were well estimated by the network. The mid-layer image was analyzed to examine how the network understood the input microstructure image. The network seemed to recognize the curvatures of the grain boundaries and estimate the average grain size from these curvatures.