Grain moisture content (GMC) is a key indicator of the appropriate harvest period of rice. Conventional testing is time-consuming and laborious, thus not to be implemented over vast areas and to enable the estimation of future changes for revealing optimal harvesting. Images of single panicles were shot with smartphones and corrected using a spectral–geometric correction board. In total, 86 panicle samples were obtained each time and then dried at 80 °C for 7 days to acquire the wet-basis GMC. In total, 517 valid samples were obtained, in which 80% was randomly used for training and 20% was used for testing to construct the image-based GMC assessment model. In total, 17 GMC surveys from a total of 201 samples were also performed from an area of 1 m² representing on-site GMC, which enabled a multi-day GMC prediction. Eight color indices were selected using principal component analysis for building four machine learning models, including random forest, multilayer perceptron, support vector regression (SVR), and multivariate linear regression. The SVR model with a MAE of 1.23% was the most suitable for GMC of less than 40%. This study provides a real-time and cost-effective non-destructive GMC measurement using smartphones that enables on-farm prediction of harvest dates and facilitates the harvesting scheduling of agricultural machinery.