Multi-Task Learning of Dish Detection and Calorie Estimation

Abstract

In recent years, a rise in healthy eating has led to various food management applications, which have image recognition to automatically record meals. However, most image recognition functions in existing applications are not directly useful for multiple-dish food photos and cannot automatically estimate food calories. Meanwhile, methodologies on image recognition have advanced greatly because of the advent of Convolutional Neural Network, which has improved accuracies of various kinds of image recognition tasks, such as classification and object detection. Therefore, we propose CNN-based food calorie estimation for multiple-dish food photos. Our method estimates food calories while simultaneously detecting dishes by multi-task learning of food calorie estimation and food dish detection with a single CNN. It is expected to achieve high speed and save memory by simultaneous estimation in a single network. Currently, there is no dataset of multiple-dish food photos annotated with both bounding boxes and food calories, so in this work, we use two types of datasets alternately for training a single CNN. For the two types of datasets, we use multiple-dish food photos with bounding-boxes attached and single-dish food photos with food calories. Our results show that our multi-task method achieved higher speed and a smaller network size than a sequential model of food detection and food calorie estimation.