Classifying the quality of dates after harvesting the crop plays a significant role in reducing waste from date fruit production. About one million tons of date fruit is produced annually in Saudi Arabia. Part of this production goes to local factories to be produced and packaged to be ready for use. Classifying and sorting edible dates from inedible dates is one of the first and most important stages in the production process in the date fruit industry. As this process is still performed manually in date production factories in Saudi Arabia, this may cause an increase in the waste of date fruit and reduce the efficiency of production. Therefore, in our paper, we propose a system to automate the classification of dates fruit production. The proposed system focuses on classifying the quality of date fruit at the postharvesting stage. By automating the process of classifying date fruit at this stage, we can increase the production efficiency, raise the classification accuracy, control the product quality, and perform data analysis within the industry. As a result, this increases the market competitiveness, reduces production costs, and increases the productivity. The system was developed based on convolutional neural network models. For the purpose of training the models, we constructed a new image dataset that contains two main classes that have images of date fruit with excellent surface quality and another class for date fruit with poor surface quality. The results show that the used model can classify date fruit based on their surface quality with an accuracy of 97%.