Background and Aims The Mediterranean diet (MD) is a dietary pattern that can lower the risk of non-communicable diseases, including diabetes. The MD Adherence (MDA) index determines how closely individuals follow MD, based on their consumed meals. The index can be automatically evaluated with a system which accurately recognises the food items that appear in a photo of a person’s meal. Methodology We propose a novel hierarchical algorithm to address the problem of multi-label automatic food recognition. The input of the system is an image of a meal, and the outputs are the MD-related food categories it contains. • Firstly, a convolutional neural network (CNN) is trained to recognise the food items that exist in an image. • The food categories are often confused by the CNN but are merged into coarse classes based on these errors. • Then, a newly introduced CNN following a hierarchical architecture, based on the work of [1], [2], learns to output from the coarse classes to the MD-related food categories. The architecture of the network can be seen in Figure 1. Results For the 31 MD-related food categories, the hierarchical model achieved a mean Average Precision (mAP) of 52.71%, outperforming the baseline model that had mAP of 46.55%. Figure 3 shows some results from the baseline and the hierarchical model. Conclusions The proposed algorithm can more accurately predict the food items that appear in an image than the baseline method and will be integrated into a smartphone application that estimates the weekly MDA on the basis of each consumed meal/drink. Figure 1: The hierarchical network architecture where Nc is the number of coarse classes. The network initially focuses more on predicting the coarse classes (upper part of the network) and then gives more weight to the fine, MD-related food categories (lower part). Figure 3: Example results of the baseline and the hierarchical network. Green indicates correct prediction and red indicates incorrect prediction. Red with grey background indicates missing prediction. References [1] Yan, Zhicheng, et al. "HD-CNN: hierarchical deep convolutional neural networks for large scale visual recognition." Proceedings of the IEEE international conference on computer vision. 2015. [2] Zhu, Xinqi, et al.. "B-CNN: branch convolutional neural network for hierarchical classification." arXiv preprint arXiv:1709.09890 (2017). Food Recognition in Assessing the Mediterranean Diet: A Hierarchical Approach I. Papathanail 1 , Y. Lu 1 , M. Vasiloglou 1 , T. Stathopoulou 1 , A. Ghosh 2 , D. Fäh 3 , S. Mougiakakou 1 ; 1 University of Bern, ARTORG Center for Biomedical Engineering Research, Bern, Switzerland, 2 Oviva S.A., Altendorf, Switzerland, 3 University of Zurich, Epidemiology, Biostatistics and Prevention Institute (EBPI), Zurich, Switzerland Contact: ioannis.papathanail@artorg.unibe.ch Acknowledgments This work was funded by Innosuisse under agreement n° 33780.1 IP-LS [www.innosuisse.ch]. Database We used a dataset that contains 5778 food images captured under free living conditions by the end users of Oviva. The images are annotated into 31 food categories of interest for MD, from which the MDA index is defined. The database was split into training and testing sets, with 5485 and 293 food images, respectively. Examples of images from the dataset are shown in Figure 2. Annotations: Vegetables Cheese Red Meat Annotations: Vegetables Red Meat White Rice Figure 2: Sample images from the training set (upper row) and the testing set (lower row). Annotations: Eggs Sweet Drink Annotations: Fruits Milky Coffee Sweets Nuts Ground Truth: Fruits Milk Processed Cereal Baseline Prediction: Fruits Milk Eggs Nuts Processed Cereal Hierarchical Prediction: Fruits Milk Processed Cereal Nuts Ground Truth: Eggs White Bread Tea Baseline Prediction: Eggs Tea Sweet Drink White Bread Hierarchical Prediction: Eggs White Bread Tea Ground Truth: Vegetables Red Meat Sweet Drink White Pasta Baseline Prediction: Vegetables Sweet Drink Red Meat White Pasta Hierarchical Prediction: Vegetables Red Meat Sweet Drink White Pasta Ground Truth: Eggs White Bread Coffee Baseline Prediction: Eggs White Bread Yoghurt Tea Coffee Hierarchical Prediction: Eggs White Bread Coffee Yoghurt Tea