The physical features of fruit and vegetables make the task of vision-based classification of fruit and vegetables challenging. The classification of fruit and vegetables at a supermarket self-checkout poses even more challenges due to variable lighting conditions and human factors arising from customer interactions with the system along with the challenges associated with the colour, texture, shape, and size of a fruit or vegetable. Considering this complex application, we have proposed a progressive coarse to fine classification technique to classify fruit and vegetables at supermarket checkouts. The image and weight of fruit and vegetables have been obtained using a prototype designed to simulate the supermarket environment, including the lighting conditions. The weight information is used to change the coarse classification of 15 classes down to three, which are further used in AdaBoost-based Convolutional Neural Network (CNN) optimisation for fine classification. The training samples for each coarse class are weighted based on AdaBoost optimisation, which are updated on each iteration of a training phase. Multi-class likelihood distribution obtained by the fine classification stage is used to estimate a final classification with a softmax classifier. GoogleNet, MobileNet, and a custom CNN have been used for AdaBoost optimisation, with promising classification results.
The complex task of vision based fruit and vegetables classification at a supermarket self-checkout poses significant challenges. These challenges include the highly variable physical features of fruit and vegetables i.e. colour, texture shape and size which are dependent upon ripeness and storage conditions in a supermarket as well as general product variation. Supermarket environments are also significantly variable with respect to lighting conditions. Attempting to build an exhaustive dataset to capture all these variations, for example a dataset of a fruit consisting of all possible colour variations, is nearly impossible. Moreover, some fruit and vegetable classes have significant similar physical features e.g. the colour and texture of cabbage and lettuce. Current state-of-the-art classification techniques such as those based on Deep Convolutional Neural Networks (DCNNs) are highly prone to errors resulting from the inter-class similarities and intra-class variations of fruit and vegetable images. The deep features of highly variable classes can invade the features of neighbouring similar classes in a learned feature space of the DCNN, resulting in confused classification hyper-planes. To overcome these limitations of current classification techniques we have proposed a class distribution-aware adaptive margins approach with cluster embedding for classification of fruit and vegetables. We have tested the proposed technique for cluster-based feature embedding and classification effectiveness. It is observed that introduction of adaptive classification margins proportional to the class distribution can achieve significant improvements in clustering and classification effectiveness. The proposed technique is tested for both clustering and classification, and promising results have been obtained.
Image classification of fruit and vegetables at supermarket self-checkouts is a complex problem. Significant variations in the size, shape and colour of objects are involved, along with potentially large variations in the environmental conditions, must be accommodated to implement such a robust and effective system. Convolution Neural Networks (CNNs) have shown promising results for object classifications. However, the scarcity of training datasets due to the diversity of varieties and applications of fruit and vegetable classification is a significant limitation to the CNN implementation for this task. To overcome this, we propose the use of transfer learning and ensemble technique. Specifically, a transfer learning based weighted average weight optimisation ensemble technique is applied to the weights of GoogleNet and MobileNet by transfer learning the pre-trained CNNs using a custom dataset. Two hyperparameter optimisation techniques have been applied in a sequential way to identify the effective weights progressively. The optimised weights are used as an input to a normalised exponential softmax layer to estimate the final probability distribution for classification. A comparative evaluation among standalone GoogleNet, MobileNet and different levels of ensemble has been presented, which supports the adoption of this technique as a solution in a real-world supermarket environment.