DialBetics With a Multimedia Food Recording Tool, FoodLog

Abstract

Background: Diabetes self-management education is an essential element of diabetes care. Systems based on information and communication technology (ICT) for supporting lifestyle modification and self-management of diabetes are promising tools for helping patients better cope with diabetes. An earlier study had determined that diet improved and HbA1c declined for the patients who had used DialBetics during a 3-month randomized clinical trial. The objective of the current study was to test a more patient-friendly version of DialBetics, whose development was based on the original participants’ feedback about the previous version of DialBetics. Method: DialBetics comprises 4 modules: data transmission, evaluation, exercise input, and food recording and dietary evaluation. Food recording uses a multimedia food record, FoodLog. A 1-week pilot study was designed to determine if usability and compliance improved over the previous version, especially with the new meal-input function. Results: In the earlier 3-month, diet-evaluation study, HbA1c had declined a significant 0.4% among those who used DialBetics compared with the control group. In the current 1-week study, input of meal photos was higher than with the previous version (84.8 ± 13.2% vs 77.1% ± 35.1% in the first 2 weeks of the 3-month trial). Interviews after the 1-week study showed that 4 of the 5 participants thought the meal-input function improved; the fifth found input easier, but did not consider the result an improvement. Conclusions: DialBetics with FoodLog was shown to be an effective and convenient tool, its new meal-photo input function helping provide patients with real-time support for diet modification.

Ongoing diabetes self-management education of patients focused on behavioral and lifestyle modification is an essential element of diabetes care.¹ Systems based on information and communication technology (ICT) that support such self-management are promising tools to help patients better cope with their diabetes.² An ICT platform helps patients improve lifestyle-related behavior, improving their health status.³ And these tools should help limit medical costs somewhat by providing cost-effective support and intervention: if patients can improve diabetes self-management without undue time and attention from health care providers, the net effect should be containment of costs.⁴

Many health applications based on mobile phones have been developed and distributed worldwide. Previous studies have reported positive effects of these applications in providing ongoing support for diabetes self-management.⁵ However, most applications offer patients limited opportunity for feedback about satisfaction and usability; few applications are patient-oriented³ in that they fail to consider whether patients will find it easy or hard to use the application. Indeed, incorporation of technical advances into such applications can be challenging because developing integration between clinical practice and state-of-the-art technology is difficult. It helps if design of applications involves patients, letting them evaluate the system and incorporating their feedback in the development process.

We designed DialBetics as a smartphone-based application that supports patients’ improved self-management diabetes control. First, we tested the system with a pilot study, and made adjustments based on patient feedback.⁶ Then we conducted a 3-month randomized trial. The results were promising: HbA1c declined an average of 0.4% among the DialBetics group compared with the control group that did not use DialBetics.⁷ But input of diet and exercise declined over the study period, especially diet input (from 53.5 ± 35.7% the first 2 weeks to 26.5 ± 37.3% last 2 weeks with voice/text input; with photo input, 77.1% ± 35.1% first 2 weeks, 51.2 ± 42.0% last 2). End-of-study questionnaires rating participants’ assessment of DialBetics’ usability and their satisfaction with the system showed that the effort of inputting meals seemed unrewarding: they were not getting back enough information that would let them better understand problems related to dietary habits. They wanted the nutritional value of their whole meal, not just its main dish, which was all that the system evaluated then. An earlier DialBetics version had used only voice/text input; this version also allowed inputting a smartphone meal photo, and although that kept compliance relatively high, participants wanted nutritional evaluation and advice in time to modify their next meal. Because that feedback was not automated, it had usually taken 1 or 2 days to receive evaluation of each meal photo. Since our study had shown that diet evaluation by meal photo was reliable at group level, we next developed a DialBetics version assisted by FoodLog to automate dietary assessment. FoodLog is a food recording application that uses image processing to facilitate easier, faster input of meal photos.⁸

The objective of the current 1-week study was to test the DialBetics-assisted-by-FoodLog version that had resulted from participants’ feedback. This new version is the first such system—combining ICT technology and image processing—that performs real-time automated meal evaluation and diet advice to patients based on that evaluation. This pilot study would also determine if usability and compliance improved with the new, FoodLog-assisted meal-input function.

Methods

Design of DialBetics

The design of DialBetics to help improve diabetes self-management was detailed elsewhere.⁷ Briefly, DialBetics comprises 4 modules: (1) Data transmission: Patients’ blood glucose, blood pressure, body weight, and pedometer counts are measured at home and sent to the server twice a day, the first 3 upon waking, then blood glucose, blood pressure, and pedometer readings at bed time. (2) Evaluation: Data are automatically evaluated following the Japan Diabetes Society (JDS) guideline values—optimally, blood glucose <110 mg/dl before breakfast, <140 mg/dl at bed time; blood pressure <130/80 mmHg; and pedometer count >10,000. DialBetics determines if each reading satisfies guideline requirements, then immediately sends those results to each patient’s smartphone. Abnormal readings—blood glucose >400 mg/dl or <40 mg/dl, systolic blood pressure >220 mmHg—are checked by a physician who will intervene if necessary. (3) Exercise input: The patient’s text/voice messages about type and duration of exercise not counted by pedometer are sent to the server, with voice input converted to text and matched with text in the DialBetics database; advice on life-style modification, matched to the patient’s input, is sent back to each patient, using the output communication function that is part of the module. (4) Food recording and dietary evaluation: Patients input data on what they have eaten and drunk, and receive back nutritional evaluation and advice on diet modification. Improvement and automation of this process was accomplished by addition of FoodLog to the system.

Functioning of FoodLog

FoodLog is a smartphone-based application available in Japan to anybody who wants to keep a food diary.⁸ The user can describe meals by text. But the novel function of FoodLog is easy meal input by photo, facilitated by image processing. The user takes a photo of a meal, then extracts each dish by pressing its image on the smartphone screen. For ease of interaction, the dish-selection area is limited to a square shape whose size the user can change. The system searches the user’s personal database to find matches for the extracted dish, conducting image retrieval by the color tone of each food. Studies of food image recognition have reported color as the dominant factor.⁹ For the database search, we modified the previously reported spatial pyramid matching¹⁰ by using color histogram information for FoodLog. To ensure selection of the correct match, the system displays the 20 database images that most resemble the image the user had extracted, listed in order of probability of match. The user selects the best match and the system can then evaluate nutritional value. Because FoodLog makes use of the patient’s personal data, the top 20 candidates are almost always sufficient, and the user can always revert to text input if no match appears. Furthermore, the personal database is increased by addition of the newly identified and named food. Note that, because FoodLog utilizes user’s data, which puts constrains on the number of data, comparison is not possible with other state-of-the-art techniques, such as CNN, which uses data posted on the Internet by anonymous people.

DialBetics Assisted by FoodLog

The FoodLog function is what permits automation of nutritional evaluation and advice by DialBetics. When the patient selects the correct dish images comprising a meal in the process described above—which takes almost less effort to accomplish than to describe—the images are automatically sent to the DialBetics server and recorded with their calorie content and nutritional composition estimated from the standard tables of food composition in Dietary Reference Intakes, Japan (2010).¹¹ Since portion sizes have been specified by the patient, DialBetics can calculate the nutritional balance of the entire meal.

Each dish’s nutritional balance—including calories and totals of protein, fat, carbohydrate, fiber, salt, and cholesterol—is sent back almost as soon as patients have sent all the dish images (Figure 1). So is advice on dietary modifications for a more nutritionally well-balanced meal, defined by JDS guidelines as one whose calories come from about 50-60% carbohydrate, 20% protein, and 25% fat.¹² Accordingly, the suggestions from DialBetics reflect patients’ excess intake of calories, carbohydrate, etc., and others and poor fiber intake. The advice, chosen automatically from a sizable collection of database advice paragraphs, is keyed to each patient’s meal photo.

Figure 1.

Dietary evaluation of DialBetics assisted by FoodLog. According to the Japan Diabetes Society guidelines, to be well-balanced, a meal’s calories should come from about 50-60% carbohydrates, 20% protein, and 25% fat.

Three-Month Clinical Trial With Meal Photos

Details of the earlier 3-month clinical trial have been reported elsewhere.⁷ Briefly, 54 type 2 diabetes patients were enrolled and randomly divided: 27 in the DialBetics group (using DialBetics), and 27 in the non-DialBetics control group that continued their regular self-care regimens. The DialBetics dietary evaluation module recorded smartphone meal photos taken by patients. Meals’ total energy, macronutrients, dietary fiber, and salt were calculated by dieticians from those photos, the results sent to the patients to help modify their diet. Among the 27 patients in the DialBetics group, 22 actively used the dietary evaluation module; 3 of the 27 dropped out of the study almost immediately, so didn’t use DialBetics at all, and 2 used the system except for diet evaluation. The 22 who completed the 3 months, using the system fully, can be categorized as 15 in the HbA1c-improved group and 7 (whose HbA1c was already within guidelines) in the nonimproved group. Nutritional values in the first 2 and last 2 weeks of the study period were compared to examine how diet changed for each group.

Design of the Present Study

A 1-week pilot study was designed and approved by the Institutional Review Board. Participants could not have any severe complications—serum creatinine <1.5 mg/dl or proliferative retinopathy—and had to be able to exercise. Five patients diagnosed with type 2 diabetes more than 5 years ago were recruited; all had participated in the 3-month trial, and all provided informed consent. They were predominantly male (n = 4), average age 58.6 years (SD = 4.1), average HbA1c 7.1% (SD = 0.9).

Just before the study, they received a smartphone (Samsung Galaxy Note 1, Seoul, Korea), NFC-enabled glucometer (Terumo MS-FR201B, Tokyo, Japan) and Bluetooth-enabled BP monitor (Omron HEM-7081-IT, Kyoto, Japan), pedometer (Omron HJ-720IT, Kyoto, Japan) with adapter (Omron HHX-IT1), and weight scale (Omron HBF-206IT), all able to transmit measurement readings by wireless network to the DialBetics server. Following each measurement, the patient profile was updated on the server.

At the end of the study, we conducted a face-to-face interview with each participant to determine whether they felt that DialBetics was improved by the introduction of the FoodLog function, asking them to assess this version’s usability and their satisfaction with the new version.

Results

The 3-Month Study

Because the present study’s evaluation of adding FoodLog meal-photo input to DialBetics is meaningful only if DialBetics indeed improves diabetes patients’ self-management, here is the core result from our 3-month clinical study: HbA1c declined a significant 0.4% in the DialBetics group compared with the control group. And even among the 22 in the DialBetics group who fully used the system there were encouraging differences. The group included the 15 whose HbA1c improved and the 7 who remained nonimproved—but note that these 7 had already achieved an HbA1c below 7%, the treatment goal for HbA1c proposed by JDS guideline. Demographic characteristics, which are summarized in Table 1, were similar for both groups except for baseline HbA1c, which was higher in the 15-patient HbA1c-improved group (7.3 ± 1.0 vs 6.4 ± 0.5, P = .043 in the 7-patient nonimproved group). HbA1c values declined significantly in the HbA1c-improved group after 3 months: their HbA1c decreased an average of 0.6% (P < .01) compared with an average increase of 0.3% in the nonimproved group (P = .01). This accounts for the average 0.4% HbA1c reduction for the original 27-patient DialBetics group.

Table 1.

Characteristics of the Patients at the Baseline and the End of the Study by HbA1c Improvement.

	Improved			Nonimproved
	Start	End	Paired t test	Start	End	Paired t test
	Mean ± SD	Mean ± SD	P value	Mean ± SD	Mean ± SD	P value
n	15		－	7		－
Sex (M/F)	13/2		－	4/3		－
Age	55.9 ± 10.8	－	－	59.7 ± 9.9	－	－
BMI	26.5 ± 5.8	26.1 ± 5.6	.18	25.5 ± 7.1	25.4 ± 7.6	.82
SBP (mmHg)	135 ± 20	129 ± 19	.09	121 ± 11	128 ± 15	.34
DBP (mmHg)	77 ± 11	78 ± 12	.88	69 ± 9	75 ± 6	.30
HbA1c (%)	7.3 ± 1.0	6.7 ± 0.7	<.01	6.4 ± 0.5	6.7 ± 0.5	.01
FBG (mg/dL)	141 ± 41	136 ± 26	.58	127 ± 12	132 ± 20	.43
LDL_C (mg/dL)	118 ± 29	116 ± 28	.59	111 ± 21	111 ± 25	.97
HDL_C (mg/dL)	59 ± 15	61 ± 16	.65	67 ± 11	70 ± 14	.15
TG (mg/dL)	143 ± 93	142 ± 104	.79	106 ± 67	128 ± 61	.48

Per-meal intake of mean total energy and macronutrients, salt and dietary fiber was compared between the first and last 2 weeks for each group (Table 2). Intake of protein (P = .03) and dietary fiber (P < .01) significantly increased in the HbA1c-improved group, while intake of total energy and all nutrients remained the same in the HbA1c nonimproved group.

Table 2.

Change of Intake in Total Energy and Nutrients Over the Study Period.

	Improved			Nonimproved
	Start	End	t test	Start	End	t test
	Mean ± SD	Mean ± SD	p value	Mean ± SD	Mean ± SD	p value
Number of meals recorded	494	344		163	102
Energy (kcal)	576 ± 264	598 ± 256	.21	546 ± 290	536 ± 209	.77
Carbohydrate (g)	72.4 ± 32.2	71.7 ± 32.4	.73	67.6 ± 31.1	72.2 ± 29.5	.23
Fat (g)	20.1 ± 13.9	21.7 ± 13.9	.09	18.2 ± 12.7	16.6 ± 8.9	.26
Protein (g)	23.7 ± 14.2	25.8 ± 14.3	.03	22.0 ± 12.0	21.6 ± 10.8	.78
Dietary fiber (g)	4.8 ± 2.9	5.4 ± 3.4	<.01	5.0 ± 2.6	4.8 ± 2.3	.46
Salt (g)	3.2 ± 2.1	3.4 ± 2.0	.06	3.3 ± 2.1	3.6 ± 2.3	.34

The Present Study

The version of DialBetics used in the 3-month study offered both voice/text and photo input of diet. Despite the encouraging results, diet input had fallen off considerably even among those who used photo input. End-of-study questionnaires showed that participants thought voice/text took too much effort and that photo input took too long if they included all foods eaten at even 1 meal. We realized that taking 1 picture of the whole meal, then extracting each dish by 1-touch image processing, would dramatically reduce the effort required to input everything consumed at a meal. That is why the present version of DialBetics uses the FoodLog-based photo input image processing system, and why the present study was conducted: to determine if this upgraded version improved diet-input compliance. (These results do not involve HbA1c or diet since neither would be impacted in 1 week.)

Compliance

Because of the present study’s short period, compliance was relatively high both for health-data measurements and diet input. Morning measurements averaged over 90%, bedtime measurements nearly 85% (Table 3). Compliance was calculated by percentage: the number of times each health datum (ie, BP before breakfast) was measured divided by the number of participation days. This calculation method was also used for input of diet and exercise, although, since most exercise was counted by pedometer, exercise input remained low. Input of meal photos was higher than with the previous version of DialBetics (84.8 ± 13.2% vs 77.1% ± 35.1%, the first 2 weeks).

Table 3.

Compliance of the DialBetics Group for Health Data Measurements and Diet Input.

	Measurement (%)						Input (%)
	Morning blood glucose	Morning blood pressure	Morning body weight	Bedtime blood glucose	Bedtime blood pressure	Bedtime pedometer count	Photo of meals	Exercise
User 1	100.0	100.0	100.0	100.0	100.0	100.0	95.2	42.8
User 2	100.0	100.0	100.0	100.0	100.0	100.0	100.0	42.8
User 3	100.0	100.0	85.7	100.0	100.0	28.5	81.0	42.8
User 4	66.6	66.6	66.6	58.3	58.3	66.6	81.0	58.3
User 5	100.0	100.0	100.0	71.4	71.4	57.1	66.7	0.0

Diet Input

Although most participants took a picture of every meal, they did not always extract all food-item images from each photo, and the system sometimes could not recognize images because no database images were similar (Table 4). Between these 2 factors, the nutritional values of 17.5% of all foods participants consumed were not evaluated.

Table 4.

Diet Input.

	Number of photos input	Number of photos evaluated (%)	Number of foods in the photos input by patients	Number of foods input by users (%)
User 1	20	20 (100.0)	88	72 (81.8)
User 2	21	21 (100.0)	74	59 (79.7)
User 3	17	17 (100.0)	55	52 (94.5)
User 4	17	17 (100.0)	115	111 (96.5)
User 5	18	14 (77.8)	80	48 (60.0)

One participant’s input rate was low (77.8%); he later explained that he often did not extract meal images because DialBetics had already shown him what was wrong with his diet.

Interview With Patients

End-of-study interviews showed that 4 of the 5 participants considered the meal-input function improved (Table 5); the fifth found input easier, but did not think the result an improvement. Three participants found the advice very helpful for improving dietary habits while 2 thought it too long and redundant, including the one who “saw no need” after learning what was nutritionally wrong with his meals. One participant found it hard to incorporate the entire system into his daily routine. But even he agreed with the others that, objectively, using the system did not require too much time—35 minutes a day on average.

Table 5.

Usability of Diet-Input Function and Advice on Dietary Habits Compared with the Previous System.

Evaluation	Photo input (n)	Advice (n)	Patients’ comments
Very improved	2	2	It was helpful to think about what was an adequate portion to eat by inputting a portion size by myself.
			Extraction of image was useful to input all dishes. By saving a photo of a meal for later input, I didn’t need to worry about inputting my meal immediately when I was too busy to do that. I could know nutritional values of my meal immediately after I input extract images. Detailed advice was helpful to know how I could improve my dietary habit.
Slightly improved	2	1	I wanted to have my own food library in my smartphone, so I could refer to it when I input a meal.
No change	1	2	I could not add a new dish, which is not in the database.
			I didn’t know the portion size of my meal. Size is very subjective, so I needed to know a standard. What is it for one person?
			I wanted to have a holiday mode for advice; I was stressed when I received advice when I knew what I did wrong.
			Some advice was too long and redundant.
Reduced	0	0
Very reduced	0	0

Discussion

Evaluation of meal photos recorded by DialBetics isolated the factors that contributed to reduction of HbA1c during the 3-month clinical trial. Intake of protein and dietary fiber increased in patients whose HbA1c improved. HbA1c reduction in the DialBetics group may be due to that increase’s having resulted in a reciprocal reduction in carbohydrate intake. Using dietary photos instead of a food diary is much less time consuming and much more convenient for patients. They can receive more timely feedback to modify their diet habits based on the photos; equally, photo evaluation of diet lets patients and health care providers examine how diet changed or remained unchanged over time. This may result in more effective intervention. Our experience with the trial motivated us to revise DialBetics to automate diet evaluation using meal photos.

This novel system—DialBetics, now with FoodLog—is based on a smartphone platform giving patients remote real-time access to their health data and personal history of diet and exercise history, and advice on lifestyle modification. One of the most striking features of DialBetics is its use of image processing provided by FoodLog. This is the first application that calculates nutritional values and balance in detail to help patients improve diabetes self-management. Moreover, detailed follow-up and monitoring of patients’ dietary habits allows health care providers to understand each patient’s diet patterns and characteristics, which can be used to tailor diet consultations.

Although it is too early for conclusive judgments about the compliance rate and usability of the system after a small 1-week study, the system did have a high compliance rate, and its usability clearly was improved over that of the previous version. Better diet-input compliance was partially due to participants being able to save meal photos, extracting and inputting dish images at a more convenient time, which the previous version did not permit. Nevertheless, participants often neglected to extract images of some foods in a meal, so those foods were not included in diet evaluation. Previous research has shown that diet self-reports often underestimate patients’ real consumption.¹³ In the current study, participants often thought that small portions did not contain enough calories to affect total calorie intake—a little butter, a bite of ham or sausage, cream in coffee, and so on. The system proved useful for revealing diet patterns and the nutritional balance of patients’ meals.

Input of complete consumption may never be possible because of human fallibility and forgetfulness. But because, with our new, upgraded DialBetics, meal photos are automatically recorded, health care providers can analyze patients’ diet retrospectively, making objective and precise diet evaluation possible when needed.

The usability survey did not indicate that the use of FoodLog was burdensome. However, the daily time required to use the system increased from 23 to 35 minutes due to more detailed meal input: before, patients usually input only the main dish, specifically because inputting more took more effort; now, it was easier. Because the study lasted only 1 week, the time required was not an issue; however, it could affect long-term compliance. Since the current system underestimated nutritional values by about 10%—even with most dishes input—we need to reconsider how diet should be evaluated. A future study should evaluate the associations between amount of dietary input, nutritional balance, patient’s behavior, lifestyle advice and diabetes control to determine the ideal way to input diet.

A previous study showed that patients choose diabetes self-management programs in different ways, choosing timing and depth of engagement depending on their situation.¹⁴ One application let patients choose and adjust how often they received messages from the system and reported reduced body weights.¹⁵ The density of lifestyle advice that provides effective clinical outcomes with minimum stress differs from patient to patient, and should be different according to the stages of behavioral change. Future applications should use stepped or adaptive engagement strategies so users will not feel overwhelmed by continuous monitoring of health data and lifestyle, thus providing optimal choice for users.^4,16

Conclusions

In conclusion, DialBetics with a multimedia food record—FoodLog—was shown to be an effective and convenient tool, its new meal-photo input function providing patients with real-time support for diet modification. Based on the current study findings, we plan to continue improving the system, and conduct a study with a larger number of patients over a longer period.