Obtaining accurate food portion estimation automatically is challenging since the processes of food preparation and consumption impose large variations on food shapes and appearances. The aim of this paper was to estimate the food energy numeric value from eating occasion images captured using the mobile food record. To model the characteristics of food energy distribution in an eating scene, a new concept of “food energy distribution” was introduced. The mapping of a food image to its energy distribution was learned using Generative Adversarial Network (GAN) architecture. Food energy was estimated from the image based on the energy distribution image predicted by GAN. The proposed method was validated on a set of food images collected from a 7-day dietary study among 45 community-dwelling men and women between 21–65 years. The ground truth food energy was obtained from pre-weighed foods provided to the participants. The predicted food energy values using our end-to-end energy estimation system was compared to the ground truth food energy values. The average error in the estimated energy was 209 kcal per eating occasion. These results show promise for improving accuracy of image-based dietary assessment.
The ability to comprehensively assess the diet of infants is essential for monitoring adequate growth; however, it is challenging to assess dietary intake with a high level of accuracy. Infants rely on surrogate reporting by caregivers. This study aimed to determine if surrogate reporters (e.g., caregivers) could use an image-based mobile food record adapted (baby mFR) to record infants’ eating occasions, and via caregiver feedback, could assess the usability and feasibility of the baby mFR in recording infants’ diets. This was a cross-sectional study in which surrogate reporters (e.g., caregivers) recorded all food and beverage intake (including human milk) of the infant over a 4-day period. Trained research staff evaluated all images submitted during data collection for different indicators of quality. All surrogate reporters were asked to complete a usability questionnaire at the end of the 4-day data collection period. Basic descriptive analyses were performed on the infants 3–12 months of age (n = 70). A total of 91% (n = 64) of surrogate reporters used the baby mFR to record their infants’ eating occasions. The mean number of images submitted daily per participant via the mFR was 4.2 (SD 0.2). A majority of submitted images contained the fiducial marker and the food and/or beverage was completely visible. The mFR was found to be easy to use; however, suggestions were provided to increase utility of the application such as the inclusion of a bottle button and reminders. An image-based dietary assessment method using a mobile app was found to be feasible for surrogate reporters to record an infant’s food and beverage intake throughout the day.
Modern deep learning approaches have achieved great success in many vision applications by training a model using all available task-specific data. However, there are two major obstacles making it challenging to implement for real life applications: (1) Learning new classes makes the trained model quickly forget old classes knowledge, which is referred to as catastrophic forgetting. (2) As new observations of old classes come sequentially over time, the distribution may change in unforeseen way, making the performance degrade dramatically on future data, which is referred to as concept drift. Current state-of-the-art incremental learning methods require a long time to train the model whenever new classes are added and none of them takes into consideration the new observations of old classes. In this paper, we propose an incremental learning framework that can work in the challenging online learning scenario and handle both new classes data and new observations of old classes. We address problem (1) in online mode by introducing a modified cross-distillation loss together with a two-step learning technique. Our method outperforms the results obtained from current state-of-the-art offline incremental learning methods on the CIFAR-100 and ImageNet-1000 (ILSVRC 2012) datasets under the same experiment protocol but in online scenario. We also provide a simple yet effective method to mitigate problem (2) by updating exemplar set using the feature of each new observation of old classes and demonstrate a real life application of online food image classification based on our complete framework using the Food-101 dataset.