Prospective Independent Evaluation of the Carbohydrate Counting Accuracy of Two Smartphone Applications

Abstract

Introduction

Smartphone applications (apps) have been designed that help patients to accurately count their carbohydrate intake in order to optimize prandial insulin dose matching. Our aim was to evaluate the accuracy of two carbohydrate (carb) counting apps.

Methods

Medical students, in the role of mock patients, evaluated meals using two smartphone apps: Foodvisor^® (which uses automatic food photo recognition technology) and Glucicheck^® (which requires the manual entry of carbohydrates with the help of a photo gallery). The macronutrient quantifications obtained with these two apps were compared to a reference quantification.

Results

The carbohydrate content of the entire meal was underestimated with Foodvisor^® (Foodvisor^® quantification minus gold standard quantification = − 7.2 ± 17.3 g; p < 0.05) but reasonably accurately estimated with Glucicheck^® (Glucicheck^® quantification minus gold standard quantification = 1.4 ± 13.4 g; ns). The percentage of meals with an absolute error in carbohydrate quantification above 20 g was greater for Foodvisor^® compared to Glucicheck^® (30% vs 14%; p < 0.01).

Conclusion

The carb counting accuracy was slightly better when using Glucicheck^® compared to Foodvisor^®. However, both apps provided a lower mean absolute carb counting error than that usually made by T1D patients in everyday life, suggesting that such apps may be a useful adjunct for estimating carbohydrate content.

Key Summary Points

Why carry out this study?

Carb counting is a key aspect of flexible insulin therapy but it is too inaccurate in most T1D patients

Several carb counting smartphone applications are available but have rarely been evaluated

What was learned from the study?

Despite slight differences, these two apps were reasonably accurate, yielding carb counting errors lower than those usually made by patients

Digital Features

This article is published with digital features, including a summary slide, to facilitate understanding of the article. To view digital features for this article go to https://doi.org/10.6084/m9.figshare.14561859.

Introduction

The gold standard treatment for type 1 diabetes (T1D) is intensified insulin therapy, whether by multiple daily injections (MDI) or by continuous subcutaneous insulin injection (CSII), and preferably using insulin analogs in order to reduce hypoglycemic events [1]. In addition, T1D patients should be trained in how to match prandial insulin doses to carbohydrate intake, premeal blood glucose, and anticipated physical activity [1]. These skills are taught to patients at the onset of their diabetes or later in the course of their disease. The term “carbohydrate counting” or “carb counting” is widely used for describing the educational meal-planning tool based on meal carbohydrate recognition and quantification [2]. For patients using CSII, the bolus wizard—an advanced pump function—can be used to facilitate insulin dose tuning based on carb quantity and premeal blood glucose levels [3]. However, even with such an integrated tool, realizing the high carb counting accuracy needed to achieve good postprandial glucose control remains a challenge [4]. Despite several methods and reference booklets that have been developed by diabetes care teams, carb counting is often inaccurate, and can even be skipped by patients. Several smartphone applications to help patients with carb counting have been developed over the last decade [5]. However, there is virtually no available literature evaluating the clinical impact of these smartphone applications and their accuracy in quantifying carbohydrates. Our aim in this study was to perform an independent bicentric evaluation of the accuracy of carb counting by two smartphone applications, one relying on the manual entry of carbohydrates with the help of a photo gallery (Glucicheck^®) and the other using automated photography recognition technology (Foodvisor^®).

Methods

Overall Design

This protocol was approved by the Caen University Hospital Institutional Review Board (n°1671) and was performed in accordance with the Helsinki Declaration of 1964 and its later amendments. The medical students provided informed consent to participate in the study.

This prospective study was performed between October 2019 and March 2020. The study participants were medical students (MS) performing their clinical placements in diabetes care units at two French university hospitals. Five MS (three MS in Caen University Hospital and two MS in Strasbourg University Hospital) were asked to use a carbohydrate quantification smartphone application (app) to analyze, on a daily basis, one of the lunch meals served to hospitalized patients with diabetes on medical wards. Two carbohydrate quantification applications were assessed in this study: Foodvisor^® and GluciCheck^® (see the description below). MS received a short training session regarding the use of these apps and were asked to quantify at least one of their own meals with both apps before commencing data collection. MS were therefore considered to be mock patients: their dietary knowledge was considered to be at a similar level to that of patients with diabetes who had been informed by diabetes education programs. Thus, at this stage of their training, MS were able to identify different food groups as well as the main macronutrients in meals. These skills were considered adequate for the proper use of the smartphone apps. For each meal, MS were required to use the apps to determine the total quantity (g) of each course and its carbohydrate, lipid, and protein contents (g) (termed the app quantification). These results were collected and compared to those determined for the same meal by a dietician who weighed the food using laboratory scales to provide a reference assessment. The primary endpoint of the study was to evaluate the discrepancy in carbohydrate quantification between each app quantification and the reference quantification for each meal. Secondary endpoints were the discrepancy between the app and reference quantifications for the carb content of each food group or course—starters, meat/fish, starches (rice, pasta, potatoes, semolina, …), vegetables, bread, dairy, and desserts—and for the total lipid and total protein in the entire meal. The percentage of meals with a carbohydrate quantification discrepancy below 10 g was also compared between the Foodvisor^® and GluciCheck^® apps, as was the percentage with a discrepancy below 20 g.

Description of the Smartphone Applications

Foodvisor^® was created in 2015 and was designed to promote healthy eating and body weight control. The main feature of Foodvisor^® is its automated instant recognition of the different courses comprising a meal, using a single smartphone picture that is automatically analyzed by the app. Automated food recognition relies on a deep learning algorithm that is regularly updated with new images of meals contributed by Foodvisor^® users. The quantification of each course is performed by the app in a second step in which the area of the image occupied by food is estimated by the system, which then translates this information into the weight of the food and finally into the macronutrient content. The macronutrient content is summarized for each meal course by course and for the entire meal.

The GluciCheck^® app was created in 2013 (Roche Diabetes Care France, Meylan, France) and is designed specifically for patients affected by diabetes. Users are required to search for the different courses comprising their meal in a scroll-down menu or in a keyword search window. Courses appear in the form of three standardized images representing different amounts of the same food (the images are taken from a standardized image bank developed specifically for this app). Patients are then asked to determine the amount of food on their plate by visual comparison with these standardized images. Once the weight of food has been determined by the user, GluciCheck^® calculates and shows the macronutrient content of each course and of the entire meal.

Both of these apps use the French CIQUAL food composition table validated by the French Agency for Food, Environmental and Occupational Health & Safety (ANSES) to provide the macronutrient content once the weight of food has been determined, whether automatically (when using Foodvisor^®) or manually (when using GluciCheck^®).

Analysis Plan

All results were expressed as the mean ± standard deviation. The results for carbohydrate content obtained from app quantifications were compared to the reference assessments for each course and for the entire meal. Lipid and protein quantifications by the apps were compared to the reference quantifications for the entire meal. For these comparisons, Student’s t-tests for paired values and Pearson correlation coefficients were calculated. The amounts obtained from the apps were considered to be consistent with the reference amounts if Student’s t-test did not show a significant difference and/or if the Pearson correlation coefficient showed a significant correlation between the app data and reference data. In addition to these tests, the app and reference results were displayed as Bland–Altman representations (% difference (100 × (app result − reference quantification result)/average) vs. average), with the ± 1.96 SD line plotted on each graph.

The mean absolute error in carbohydrate quantification was compared between the apps using the unpaired t-test. Associations were then assessed with Pearson correlation tests. The percentage of meals with an absolute error in carbohydrate quantification of greater than 10 g or greater than 20 g was calculated and compared between apps using Fisher’s exact test.

Statistical analyses were performed and graphs were produced using Graph Pad Prism v8.4.3 (GraphPad Software, San Diego, CA 92108, USA). A p-value of < 0.05 was considered significant.

Results

Thirty meals were assessed using Foodvisor^®, and 28 meals using GluciCheck^®. Those 58 meals were generally composed of a starter (a raw vegetable salad most often) (4.5 ± 4.7 g of carbohydrate), a piece of meat or fish (0.8 ± 2.7 g of carbohydrate), starches (rice, pasta, potatoes, or semolina) (41.7 ± 16.5 g of carbohydrate), vegetables (4.8 ± 2.9 g of carbohydrate), bread (28.6 ± 7.1 g of carbohydrate), a dairy item (a piece of cheese or a yogurt) (4.1 ± 3.5 g of carbohydrate), and a dessert (stewed apple, fruit, or a slice of pie) (17.2 ± 7.3 g of carbohydrate). The mean macronutrient content of the entire meal was 94.0 ± 23.2 g of carbohydrate, 18.3 ± 9.6 g of lipid and, 45.8 ± 12.2 g of protein.

Compared to the reference quantification, the carbohydrate content of the entire meal was underestimated by Foodvisor^® (Foodvisor^® quantification minus reference quantification = − 7.2 ± 17.3 g; p < 0.05) but reasonably accurately estimated with Glucicheck^® (Glucicheck^® quantification minus reference quantification = 1.4 ± 13.4 g; not significant) (Table 1). A significant correlation was observed between the entire meal carbohydrate content from the reference quantification and that quantified by both apps, with the correlation being stronger for Glucicheck^® (Foodvisor^®: r = 0.57, p < 0.01; Glucicheck^®: r = 0.87, p < 0.001) (Fig. 1).

Table 1.

Macronutrient content of the entire meal and carbohydrate contents of the different courses as assessed by Foodvisor^® and Glucicheck^® and compared to the corresponding reference quantifications

	Foodvisor®				Glucicheck®
	Reference quantification	App quantification	App minus ref quantification (p)	Pearson correlation r (p)	Reference quantification	App quantification	App minus ref quantification (p)	Pearson correlation r (p)
Entire meal macronutrients
Carbs	95 ± 20	87.9 ± 16.9	− 7.2 ± 17.3 (0.031)	0.5713 (0.001)	97.1 ± 27	98.5 ± 25.1	1.4 ± 13.4 (0.5776)	0.8700 (< 0.0001)
Lipids	18.1 ± 10.8	12.4 ± 10.5	− 5.7 ± 9.2 (0.0022)	0.6259 (0.0002)	18.8 ± 6.8	18.5 ± 8.5	− 0.3 ± 5.9 (0.8640)	0.7247 (0.0022)
Proteins	43.3 ± 12.1	39.1 ± 15	− 4.2 ± 12.3 (0.1881)	0.6225 (0.0004)	50.7 ± 11.3	57.3 ± 19.2	6.5 ± 16.8 (0.1533)	0.4970 (0.0595)
Course carbs
Starters	3.3 ± 2.4	2.6 ± 2.2	− 0.7 ± 1.6 (0.0306)	0.7557 (< 0.0001)	5.8 ± 6.1	5.2 ± 5.1	− 0.6 ± 2.5 (0.2263)	0.9163 (< 0.0001)
Meat/fish	0.5 ± 1.5	2.7 ± 5.2	2.2 ± 4.4 (0.0112)	0.6307 (0.0003)	3.8 ± 13.4	4.1 ± 8.3	0.4 ± 6.3 (0.7777)	0.9413 (< 0.0001)
Starches	38.5 ± 16.5	31.1 ± 13.9	− 7.4 ± 13.3 (0.0065)	0.6290 (0.0003)	37 ± 22	36.3 ± 23.2	− 0.8 ± 6 (0.5221)	0.9657 (< 0.0001)
Vegetables	4.8 ± 3.4	5.5 ± 2.7	0.7 ± 2.2 (0.1371)	0.7670 (< 0.0001)	5.7 ± 4.8	5.7 ± 4.8	0 ± 0.9 (0.8246)	0.9804 (< 0.0001)
Bread	29.8 ± 6.3	28.7 ± 5.64	− 1.15 ± 6.9 (0.3717)	0.3315 (0.0735)	27.3 ± 7.8	27 ± 5.9	− 0.3 ± 4.2 (0.7243)	0.8444 (< 0.0001)
Dairy	4.3 ± 3.5	4.1 ± 4.4	− 0.3 ± 3 (0.6521)	0.7383 (< 0.0001)	3.9 ± 3.6	4.0 ± 3.5	0.1 ± 0.3 (0.6521)	0.9989 (< 0.0001)
Desserts	17.3 ± 8.2	16.5 ± 9.3	− 0.8 ± 6.8 (0.5152)	0.7080 (< 0.0001)	17 ± 6.3	19.4 ± 7.6	2.4 ± 4.7 (0.0130)	0.8731 (< 0.0001)

The quantification data presented are for carbohydrates unless otherwise indicated and are expressed as the mean ± SD, in grams (g). The “app minus reference quantification” columns present the differences (mean ± SD) between the app and reference quantifications and the results of Student’s t-test (p) (results in bold are significant differences). The “Pearson correlation r” columns present the correlation between the app and reference quantifications (results in bold are nonsignificant correlations)

Fig. 1.

Pearson correlation coefficient between the entire meal carbohydrate content (g) quantified by an app (left panell: Foodvisor^®; right panel: Glucicheck^®) and the corresponding reference quantification

Using Foodvisor^®, the carbohydrate content was underestimated for starters and starches (− 0.7 ± 1.6 and − 7.4 ± 13.3 g, respectively; p < 0.05) and overestimated for meat/fish (+ 2.2 ± 4.4 g; p < 0.05). The Foodvisor^® carbohydrate quantification was not correlated with the reference quantification for bread (r = 0.33; ns). The lipid content of the entire meal was underestimated with Foodvisor^® compared to the reference quantification (− 5.7 ± 9.2 g; p < 0.01) (Table 1).

With Glucicheck^®, the dessert carbohydrate content was overestimated (+ 2.4 ± 4.7 g; p < 0.05). All Glucicheck^® carbohydrate quantifications were correlated with the corresponding reference quantifications. Lipid and protein contents for the entire meal obtained with Glucicheck^® were not different from the corresponding reference quantifications (Table 1).

Bland–Altman graphs revealed narrower limits of agreement for Glucicheck^® compared to Foodvisor^® (Fig. 2).

Fig. 2.

Bland–Altman representations of carbohydrate quantifications of high-carb-content courses and the entire meal obtained with Foodvisor^® (left column) and Glucicheck^® (right column) as compared to the corresponding reference quantifications. The average of the carbohydrate quantification obtained with an app and the corresponding reference quantification is plotted on the x-axis while the percentage difference between those quantifications (100 × (app result − reference quantification result)/average) is plotted on the y-axis. thin horizontal lines represent ± 1.96SD for each data set

The mean absolute error in carbohydrate quantification for the entire meal was not significantly different between Foodvisor^® and Glucicheck^® (13.9 ± 12.4 vs 10.4 ± 8.6 g, respectively; p = 0.2255). The maximum absolute errors in carbohydrate quantification for the entire meal were 42.8 and 39.0 g with Foodvisor^® and Glucicheck^®, respectively. No significant correlation between the amount of carbs in the entire meal and the mean absolute error in the carb quantification was observed for either app (data not shown).

The percentage of meals for which the absolute error in carbohydrate quantification was greater than 10 g did not differ between the apps, but the percentage of meals with an absolute error in quantification above 20 g was higher for the Foodvisor^® app (30 vs 14%; p < 0.01) (Fig. 3).

Fig. 3.

Percentages of meals with an absolute error in app-based carbohydrate quantification below/above 10 g (a) or below/above 20 g (b). Apps: Foodvisor^® (black bars) and Glucicheck^® (gray bars). ** p < 0.01

Discussion

In this independent assessment of the carb counting accuracy of two meal analysis apps, we obtained better results with the Glucicheck^® app compared to Foodvisor^®. Carb quantification by Glucicheck^® was reasonably accurate for the entire meal and for most courses, whereas the Foodvisor^® app underestimated carb amounts for the entire meal and for starches. Despite similar mean absolute errors in carbohydrate quantification of approximately 10–15 g, the percentage of meals with an absolute error in carb count above 20 g was twofold higher with Foodvisor^® (30%) than with Glucicheck^® (14%). As these two apps use the same reference database for food composition, the superiority of Glucichek^® for carb quantification is probably due to Foodvisor^® being less accurate when determining food amounts, as it underestimated the amounts of starches, bread, and desserts by 15–30% (data not shown). These findings suggest that asking the user to determine the amount of food through visual comparison with standardized images of plated food, as performed by Glucicheck^®, leads to more accurate results than the automated assessment performed by Foodvisor^®. The human factor—which is more prominent with Glucicheck^®—was found to be beneficial in this study, as it allowed more accurate carbohydrate quantification than the totally automated system of Foodvisor^®, an application for which the only human factor is the way that the photo is taken with the smartphone.

Other carb counting applications have been previously evaluated. GoCARB^® is a computer vision-based smartphone system designed to estimate the plated meal carbohydrate content. GoCARB^® requires two smartphone photos of the same plate taken from two different viewing angles. With this information, GoCARB^® is able to recognize the type of food and to estimate its volume, allowing carb quantification from a reference table of food macronutrient composition [6]. Carb quantification by GoCARB^® showed a similar degree of accuracy to our experiments with Foodvisor^® and Glucicheck^®, with an absolute mean carb error of 14.8 ± 9.7 g [7]. This error was found to be higher for larger meals, whereas we found no correlation between the total amount of carbs in the entire meal and the error in the carb quantification with either of the apps used in our study. The accuracy of VoiceDiab^®, another carb counting system that relies on a vocal description of the meal, was also recently assessed, and outstanding results were reported: 96.3% of the meal evaluations performed by this system had an absolute carb error of less than 10 g [8]. However, it should be mentioned that GoCARB^® and VoiceDiab^® are not widely available, and that their accuracies were evaluated by physicians involved in their development.

Very accurate evaluation of the amount of carbohydrate in meals is of paramount importance for obtaining a postprandial profile within the target range, as demonstrated by Smart et al. several years ago [9]. Those authors monitored the 180-min postprandial continuous glucose monitoring (CGM) profiles of 34 T1D children and adolescents who ate five different breakfasts containing 40, 50, 60, 70, or 80 g of carbohydrate. The preprandial insulin dose was the same for each breakfast and was based on the subject’s usual insulin:carbohydrate ratio for 60 g carbohydrate. Both the 40 g and 50 g carb breakfasts, when consumed with the higher than required insulin dose, resulted in a significant increase in postprandial hypoglycemic events, whereas the 80 g carb breakfast, when consumed with the lower than required insulin dose, resulted in higher exposure to hyperglycemia [9]. On the other hand, in a study of children and adolescents by the same authors, there was no impact of a 10 g carb error on postprandial CGM profiles [10]. These two studies suggest that an absolute error of ≥ 20 g of carbohydrate had a significant impact on the postprandial glucose profile.

Such errors appear to be frequent in T1D patients, as seen in the GoCARB^® study [6], which reported a mean absolute carb counting error of 27.9 ± 38.2 g based on an analysis of 114 meals consumed by 19 T1D patients. Brazeau et al. evaluated the ability of 50 adults to accurately estimate the carb count in a real-life survey, and found that the carb count was underestimated for 63% of hundreds of analyzed meals, with the absolute error ranging from 4.0 to 38.3 g [11]. In the same study, the authors reported that the greater the absolute error observed, the greater the glucose variability, and the smaller the time in range (70–180 mg/dL) [11]. Also, in a T1D population of children and adolescents, it was demonstrated that under/overestimation of the meal carb content resulted in over/below-range postprandial glucose in 63–87% of the meals affected by such errors, respectively [4].

Several studies have explored the factors associated with carb counting errors. They showed that errors were more frequent and more pronounced for meals containing very high or, conversely, very low amounts of carbs, but also for high-calorie meals, regardless of their carb contents [12]. Other authors have demonstrated that the carbs in large meals were more frequently underestimated while those in small meals were overestimated [13]. In a real-life survey, a broad panel of T1D patients who were using carb counting were asked about their practices, perceptions, and expectations concerning this method. More than 90% of the patients considered that accurate carb counting was of paramount importance for their glucose control, but they also acknowledged that effective and accurate carb counting was difficult in everyday life. They reported that the main difficulties in applying carb counting were encountered when eating away from home, when eating unpackaged foods, and for large meals, but most of the respondents believed that new technologies should be helpful. Finally, a majority of patients pointed out that their postmeal glucose control was not always optimal, despite accurate carb counting [14].

The latter statement suggests that carb counting is not the only factor associated with postprandial glucose control. It has been clearly demonstrated that meals with high lipid and/or high protein contents often result in uncontrolled and/or prolonged postprandial hyperglycemia, despite the use of a precise insulin:carb ratio and accurate carb counting [15, 16]. Ryan et al. investigated postprandial glucose excursion after different meals with the same macronutrient content and a standardized rapid-acting analog insulin dose. They found that postprandial glucose excursion was higher with high glycemic index meals, even when an accurate insulin/carb ratio was applied [17]. Apart from these nutritional issues, the timing of rapid insulin injection also appears to be critical for optimal glucose control during the postprandial period; a lower postprandial glucose excursion is experienced when insulin injection is performed 20 min before meals compared to a postprandial injection or a “just before meal” injection [18]. This finding has recently been confirmed for ultra-rapid lispro, with optimal postprandial glucose control being obtained using the preprandial injection [19].

The accuracy of carb counting will remain a hot topic for hybrid closed-loop users, as all of these systems require meal announcement. Several studies of single-hormone artificial pancreas systems have also demonstrated that inaccurate carb counting is associated with poorer postprandial glucose control [20–22].

To our knowledge, this is the first independent study to evaluate the accuracy of two carb counting smartphone applications. However, our study does have several limitations. The use of medical students in the role of mock patients does not allow the extrapolation of these results to how real patients would use these applications. Also, the study only assessed the accuracy of these two apps for lunch meals, not for breakfast or dinner, which may have different meal compositions. In addition, only hospital meals were analyzed, which likely does not reflect the diversity of meals that patients may consume in their daily lives. Further, each meal was only analyzed once by a single medical student, which did not allow intra- nor interindividual reproducibility assessment. Finally, this accuracy study may not predict patient acceptance, compliance, or persistence in the use of such applications, nor the potential metabolic benefit provided by their daily use.

Conclusions

Carbohydrate quantification by Glucicheck^® resulted in an absolute carb counting error of below 20 g for more than 85% of meals. Carb counting accuracy was better when using the Glucicheck^® app compared to Foodvisor^®. However, both apps provided a mean absolute carb counting error lower than that commonly observed in real life, suggesting that these applications are of potential interest for better managing the prandial period. Further studies will be needed to determine whether these applications could be used in the long term to improve metabolic control.