Advances in machine learning and artificial intelligence have led to many promising solutions for challenging issues in agriculture. One of the remaining challenges is to develop practical applications, such as an automatic sorting system for after-ripening crops such as tomatoes, according to ripeness stages in the post-harvesting process. This paper proposes a novel method for detecting tomato ripeness by utilizing multiple streams of convolutional neural network (ConvNet) and their stochastic decision fusion (SDF) methodology. We have named the overall pipeline as SDF-ConvNets. The SDF-ConvNets can correctly detect the tomato ripeness by following consecutive phases: (1) an initial tomato ripeness detection for multi-view images based on the deep learning model, and (2) stochastic decision fusion of those initial results to obtain the final classification result. To train and validate the proposed method, we built a large-scale image dataset collected from a total of 2712 tomato samples according to five continuous ripeness stages. Five-fold cross-validation was used for a reliable evaluation of the performance of the proposed method. The experimental results indicate that the average accuracy for detecting the five ripeness stages of tomato samples reached 96%. In addition, we found that the proposed decision fusion phase contributed to the improvement of the accuracy of the tomato ripeness detection.