Contents lists available at ScienceDirect Computers in Biology and Medicine journal homepage: www.elsevier.com/locate/compbiomed Personalized food consumption detection with deep learning and Inertial Measurement Unit sensor Lehel Dénes-Fazakas a,b,c ,1 , Barbara Simon b,1 , Ádám Hartvég b,1 , László Szilágyi a,b,d,1 , Levente Kovács a,b,1 , Amir Mosavi b,1 , György Eigner a,b,∗,1 a Physiological Controls Research Center, University Research and Innovation Center, Obuda University, Bécsi út 96/b, Budapest, 1034, Hungary b Biomatics and Applied Artificial Intelligence Institute, John von Neumann Faculty of Informatics, Obuda University, Bécsi út 96/b, Budapest, 1034, Hungary c Doctoral School of Applied Informatics and Applied Mathematics, Obuda University, Bécsi út 96/b, Budapest, 1034, Hungary d Computational Intelligence Research Group, Sapientia Hungarian University of Transylvania, Tirgu Mures, Romania ARTICLE INFO Keywords: Diabetes mellitus Recurent neural network Deep learning Gesture detection Personalized model ABSTRACT For individuals diagnosed with diabetes mellitus, it is crucial to keep a record of the carbohydrates consumed during meals, as this should be done at least three times daily, amounting to an average of six meals. Unfortunately, many individuals tend to overlook this essential task. For those who use an artificial pancreas, carbohydrate intake proves to be a critical factor, as it can activate the insulin pump in the artificial pancreas to deliver insulin to the body. To address this need, we have developed personalized deep learning model that can accurately detect carbohydrate intake with a high degree of accuracy. Our study employed a publicly available dataset gathered by an Inertial Measurement Unit (IMU), which included accelerometer and gyroscope data. The data was sampled at a rate of 15 Hz, necessitating preprocessing. For our tailored to the patient model, we utilized a recurrent network comprising Long short-term memory (LSTM) layers. Our findings revealed a median F1 score of 0.99, indicating a high level of accuracy. Additionally, the confusion matrix displayed a difference of only 6 s, further validating the model’s accuracy. Therefore, we can confidently assert that our model architecture exhibits a high degree of accuracy. Our model performed well above 90% on the dataset, with most results between 98%–99%. The recurrent networks improved the problem-solving capabilities significantly, though some outliers remained. The model’s average prediction latency was 5.5 s, suggesting that later meal predictions result in extended meal progress predictions. The dataset’s limitation of mostly single-day data points raises questions about multi-day performance, which could be explored by collecting multi-day data, including night periods. Future enhancements might involve transformer networks and shorter time windows to improve model responsiveness and accuracy. Therefore, we can confidently assert that our model exhibits a high degree of accuracy. 1. Introduction Food consumption, an inherent and basic human behavior, is closely related to the physiological well-being of the population [1–3], and most importantly for diabetes participants [4,5]. Recognizing food intake is critical to effective diabetes management. It is an essential part of a comprehensive approach to diabetes care that promotes better blood sugar regulation, informed nutritional choices, and the empowerment of participants to manage their condition [6–8]. Ef- fective treatment of diabetes has become a global concern. Treating diabetes includes monitoring food intake and physical activity. Tra- ditional methods like journaling lack accuracy due to manual entry ∗ Corresponding author. E-mail addresses: denes-fazakas.lehel@uni-obuda.hu (L. Dénes-Fazakas), simon.barbara@uni-obuda.hu (B. Simon), hartveg.adam@uni-obuda.hu (Á. Hartvég), kovacs@uni-obuda.hu (L. Kovács), amir.mosavi@uni-obuda.hu (A. Mosavi), eigner.gyorgy@uni-obuda.hu (G. Eigner). 1 All Authors equally contributed to the work. and reliance on memory [9]. Automated food consumption detection using wearable sensor systems is attracting attention as a solution [10]. Recent research shows that deep learning-based gesture recognition based on accelerometer and gyroscope data outperforms basic machine learning methods [11]. Traditional food consumption detection relies heavily on self-report data, is prone to human error, and is a time- consuming manual process. The solution is emerging as automated food consumption, which independently collects and processes data to complement traditional methods and minimize bias. These systems also hold potential for personal self-monitoring by providing tailored recommendations on food [12]. In addition to its basic nutritional https://doi.org/10.1016/j.compbiomed.2024.109167 Received 17 April 2024; Received in revised form 16 September 2024; Accepted 17 September 2024 Computers in Biology and Medicine 182 (2024) 109167 Available online 25 September 2024 0010-4825/© 2024 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by- nc/4.0/).