Accuracy of Self-Report Versus Objective Smart Scale Weights During a 12-Week Weight Management Intervention

Abstract

Objective:

Greater frequency of self-weighing has been associated with greater weight loss in weight management interventions, but little is known regarding accuracy of self-report weight data.

Methods:

Agreement between objective smart scale and self-reported weight data was assessed in 74 adults (age=50.7 years, BMI=31.2 kg/m²) enrolled in a 12-week, Internet-based weight management program. Participants were asked to self-weigh daily using a study-provided smart scale and to self-report weights via the study website.

Results:

There was strong agreement between smart scale and self-reported weight values, r_icc=.982, but only moderate agreement between frequency of self-weighing assessed via each method, κ=.491, p<.0001. Greater self-weighing frequency was associated with greater weight loss across measures, ps<.001. Compared to days when they did both, weights were 0.66kg higher on days when participants self-weighed via smart scale but did not self-report weight (8% of days), and 0.58kg higher on days that they self-reported weight but did not self-weigh via smart scale (4% of days), ps<.0001.

Conclusions:

Results suggested that self-reported weight values are similar to smart scale measurements; however, either method alone may underestimate self-weighing frequency. Further, missing self-weighing data should not be treated as ignorable, as weights may be higher than those observed on non-missing days.

Introduction

Self-monitoring of body weight (“self-weighing”) has been emphasized as a key component of behavioral weight management programs.^1,2 Greater frequency of self-weighing has been associated with greater weight losses during these programs,^3–6 with this association mediated by greater adherence to caloric intake and physical activity goals.⁶

A key challenge with early research on self-weighing was dependence on self-report measures, such as retrospective questionnaires (e.g., asking participants the frequency at which they self-weighed, with categories ranging from daily to never) or participant self-monitoring records.⁴ Self-report data in general can be biased by social desirability^7,8 and retrospective questionnaire data may be impacted by additional recall biases.^9–12 More recently, the development of “smart” scales which directly transmit weight data via telephone/cellular networks or wireless internet has allowed researchers to collect objective data. Existing studies using these scales have replicated the association between higher frequencies of self-weighing and improved weight loss outcomes during intervention.^5,13–15

Taken together, existing results suggest that greater self-weighing frequency, as assessed via either smart scale or self-report, is associated with greater weight loss within weight management interventions; however, there has been limited research comparing these methods directly. One recent study found poor to moderate agreement between a retrospective, self-report questionnaire item assessing self-weighing frequency versus data collected via smart scale.¹⁶ To date, no studies have compared of self-reported weight data to smart scale data that was assessed concurrently. This has precluded the ability of researchers to assess both the agreement between values (e.g., do participants in a weight management program accurately self-report weight, compared to objective smart scale values?) and the relative validity of each method for measuring self-weighing frequency. It is unknown how often participants weigh themselves without self-reporting weight (and vice versa), and if this systematically biases data. The lack of direct comparison between measures has also limited the ability of researchers to disentangle the impact of self-weighing as a behavior from self-reporting of weight (e.g., whether the additional steps required to self-report weight has an impact on adherence to other weight-related behaviors and thus affects weight change beyond the influence of self-weighing itself).

The current study addressed these gaps by assessing agreement between weight data collected via in-home smart scale and weights self-reported on a study website during a 12-week, Internet-based behavioral weight management program. We proposed to investigate agreement between 1) weight values from the same day and 2) the frequency of self-weighing as assessed by each measurement. Further, we proposed to investigate the difference in weight values on days when participants both weighed themselves via smart scale and reported their weight on the study website, versus days when they used only one method.

Methods

The current study represents secondary data analyses of data collected during a 12-week, Internet-based weight loss program. This program was provided to employees or dependents of employees of Lifespan Corporation, a large healthcare corporation in Providence, Rhode Island, as part of a worksite health program. The Internet-based program had been implemented in previous studies^18,19 and was delivered in the context of a longer, observational study investigating trajectory of weight loss and regain over a full calendar year.²⁰ For this larger observational study, research tools were utilized that were unrelated to the intervention itself – namely, while participants were asked to self-report daily weights each week on the intervention website, the larger research study provided participants with “smart” scales that sent weights back to research servers. Smart scale weight data were not accessible by the intervention program or retrievable by participants. A full dataset containing all smart scale data was downloaded by the study team at the end of the year-long data collection period. This concurrent collection of self-report and smart scale weights throughout the 12-week intervention allowed for the direct comparison of these two methods in the current study.

Participants

Participants in the original study were 75 adults (age 18–70 years) with overweight or obesity (BMI ≥ 25 kg/m²). Full inclusion/exclusion criteria and information regarding recruitment, enrollment, and collection of written informed consent were published previously.¹⁷ Approval for the parent study was obtained from the Miriam Hospital Institutional Review Board, and approval for the current study was obtained from the University of Florida Institutional Review Board.

Intervention

A complete description of the 12-week, Internet-based intervention was published previously.¹⁷ Intervention content, delivered in weekly 12–15 minute lessons accessible via the study website, was adapted from the Diabetes Prevention Program²¹ and Look AHEAD.²² Prior to program start, participants attended one group-based education session which explained how to use the study website, provided caloric intake and physical activity goals consistent with current clinical recommendations,² and taught participants how to use study-provided tools to self-monitor their caloric intake (using a Calorie King reference book and paper self-monitoring records with room to record foods/drinks consumed, physical activity, and weight), physical activity (via a standard pedometer), and weight (via a study-provided smart scale). Participants were instructed to weigh themselves once daily (preferably in the morning before consuming any food/drinks) and to record each weight in paper self-monitoring records.

Participants were asked to log onto the study website each Sunday evening to self-report caloric intake, minutes of physical activity, and weight for each day of the week (participants could access the website as many times during the week as they wanted and enter this information on a more frequent basis if preferred). Every Monday morning, participants were provided with an automated, tailored feedback message based on submitted self-report data. Finally, as part of the larger observational research study, participants were given small financial incentives for completing a brief (11-item) weekly questionnaire and reporting self-monitoring data each week (ranging from $1–10 each week, for a total of $86 possible for submitting information every week for the 12 weeks). The incentive was provided specifically for reporting behavior, not for self-monitoring itself (i.e., participants could still receive the incentive if they reported that they did not self-monitor that week).

Measures

Participant demographic information was collected at baseline via self-report questionnaire. Height was measured to the nearest 0.1 cm at baseline by a trained research assistant using a wall-mounted stadiometer, with participant shoes removed.

Participant weight was measured in three ways. First, participants were weighed (in light indoor clothing, with shoes removed) at baseline and at a Month 3 follow-up assessment by trained research assistants, using calibrated digital Tanita BWB-800 scales (which have published accuracy to 0.1 kg).²³ Second, participant weights were recorded throughout the intervention from the study-provided smart scales (developed by BodyTrace, Inc.), which sent participant weights directly to BodyTrace servers via the cellular network. BodyTrace scales have stated accuracy to 0.1 kg²⁴ and have been demonstrated to have good concordance with weights measured during in-person assessment visits.²⁵ Weights from these scales were downloaded following the end of the one-year study period. Third, participants self-reported weights were collected via the intervention webserver. Weights from both the smart scale and the study website were collected in lb and converted to kg.

Analyses

Analyses were conducted using SAS version 9.4.²⁶ Data collected via smart scales were prepared by removing improbable outliers (most frequently representing another person in the household using the scale). For this, we fit generalized additive models of weight over time, by individual; points where residuals from this model were > 2.27 kg above or below predictive values were visually inspected, and removed if physiologically improbable. If multiple weights were recorded on the same day, the first (earliest) non-outlier weight was retained.

Agreement between smart scale and self-reported weight measurements was assessed via intraclass correlation and through calculation of the mean difference between measures and the 95% limits of agreement around this difference.²⁷ A Bland-Altman plot of agreement²⁷ was visually inspected to assess measurement bias.

To investigate agreement in the frequency of self-weighing assessed via each method, data were coded into binary categories indicating whether or not a participant weighed themselves via the smart scale and whether or not they self-reported their weight via the study website each day. Agreement in coding between measures was assessed using Cohen’s κ.²⁸ Counts of the frequency at which participants self-monitored weight via each method were calculated, and a McNemar’s test was used to investigate the difference in proportion of individuals self-weighing via each method.

The association between self-weighing frequency measured via each method and weight change during intervention was assessed using two univariate regressions and then a multivariate regression including both methods (to assess the effect of each method independent of potential issues with multicollinearity, and then to assess whether using both methods increased the variance explained by the model). Weight change over the course of intervention was defined as percent weight change from baseline, calculated using baseline and Month 3 assessment weights. Given that previously-published results from the parent study demonstrated a very small proportion of missing data at Month 3 (< 7%) and that weight changes from baseline to Month 3 calculated using both multiple imputation and baseline observation carried forward (BOCF) methods demonstrated nearly identical results,¹⁷ BOCF was used for missing weight data from the Month 3 assessment.

Finally, descriptive measures were used to report the number of days that individuals had scale data and but no self-report data, and vice versa. As exploratory analyses, two longitudinal multilevel models (nesting weights within individuals, controlling for the effect of time) were used to investigate whether weights were higher on days when participants only self-weighed via the study smart scale (and did not report their weight on the study website) or only self-reported weight via the study website (without self-weighing via the smart scale), compared to days when participants did both.

Results

Of the 75 participants enrolled in the parent study, one was removed from current analyses due to missing data (this participant had only one week of smart scale measurements and never logged into the study website). Table 1 provides demographic information and baseline characteristics of included participants. Over the 12-week intervention, participants self-weighed using the smart scale on an average of 74.55 ± 12.13 days (88.75 ± 14.44% of possible days) and self-reported weights on the study website on 70.65 ± 18.39 days (84.11 ± 21.90% of possible days).

Table 1.

Participant Demographics and Baseline Characteristics.

	Included Participants n=74
	M	SD
Age	50.65	10.41
Weight (kg)	86.65	16.76
BMI (kg/m2)	31.20	4.51
	n	%
Gender
Female	51	68.9
Male	23	31.1
Race*
American Indian or Alaska Native	1	1.4
Asian	2	2.7
Black or African American	7	9.5
White	64	86.5
Other	4	5.4
Ethnicity
Hispanic or Latino	2	2.7
Not Hispanic or Latino	72	97.3

^*Participants could self-select more than one race/ethnicity category, thus totals may exceed 100%

There was a very strong intraclass correlation between weights measured via smart scale and self-report, r_icc = .982, p < .0001. The average (mean ± SD) difference between measures (smart scale - self-report) was 0.07 ± 0.45 kg, with 95% limits of agreement ranging from −0.84 to 0.97 (of all observations outside of this range, 1.3% were below the lower bound and 3.8% were above the upper bound). A Bland-Altman plot of agreement (see Figure 1) did not demonstrate evidence of bias related to measurement (i.e., heavier individuals did not appear to systematically under- or over-report weight).

Figure 1.

Bland Altman Plot of Agreement Between Smart Scale Measured Weight and Self-Reported Weights.

In contrast to the very strong association between weight values, there was only a moderate association between measures when assessing the agreement of self-weighing frequency as calculated from the smart scale and self-report data, κ = .491, p < .0001 (see Table 2). Overall, participants weighed themselves via the smart scale on significantly more days (88.8% of days during the intervention) than they self-reported weights via the website (84.1% of days), p < .0001.

Table 2.

Frequency (in Days) of Weight Measured Via Smart Scale and Self-Reported Via Study Website.

	Weights collected via self-report
Weights collected via smart scale	Self-reported weight, days	Did not self-report weight, days
Had smart scale weight, days	5000	517
Did not have smart scale weight, days	228	471

Over the course of intervention (using assessment weights collected at baseline and Month 3), participants lost an average (mean ± SD) of −6.34 ± 4.86% of their baseline weight (−5.74 ± 4.94 kg). There was a significant association between percent weight change and self-weighing frequency as measured via smart scale, β = −0.189, SE = 0.042, p < .0001, model R² = .223, and self-report, β = −0.104, SE = 0.029, p < .001, model R² = .155, such that participants who self-weighed more frequently (via either method) lost significantly more weight during the course of intervention. Moreover, each additional day of self-weighing via smart scale was associated with an additional −0.19% weight change from baseline to Month 3, while each additional day of self-reporting weight was associated with an additional −0.10% weight change. While likely due to multicollinearity, when both variables were entered into a multivariate model, only self-weighing frequency as assessed via smart scale remained significant, p = .012 (frequency of self-reporting weight was no longer significant, p = .539). There was only minimal additional variance explained by this multivariate model, model R² = .227.

On 8.3% of intervention days (6.99 ± 10.98 days per participant), participants self-weighed via the smart scale but did not self-report their weight (see Table 2). Weights measured via the smart scale on these days were, on average, 0.66 kg higher compared to days that participants both self-weighed via smart scale and self-reported their weight, β = 0.660, SE = 0.078, t(5442) = 8.50, p < .0001. There was further an interaction between this effect and time, such that this tendency increased over the course of the intervention, β = 0.135, SE = 0.021, t(5440) = 6.35, p < .0001.

On 3.7% of intervention days (3.08 ± 4.29 days per participant), participants self-reported weights without having weighed themselves using the study-provided smart scale. Weights self-reported on these days were, on average, 0.58 kg higher than weights self-reported on days when participants both self-weighed via the smart scale and self-reported their weight, β = 0.583, SE = 0.091, t(5152) = 6.43, p < .0001, and this effect also increased over the course of intervention, β = 0.161, SE = 0.030, t(5151) = 4.46, p < .0001.

Discussion

Results demonstrated excellent agreement between weights measured via smart scale and those self-reported on a study website, and there was no evidence of systematic under- or over-reporting of weights by participant weight. Further, replicating results reported in previous studies,^{3,5,6,14–16} greater self-weighing frequency as assessed by either method was associated with greater weight loss during the course of the 12-week intervention. There was minimal benefit (in terms of increasing variance of weight change explained) of including both measurement methods in the same model. Despite the consistent finding across methods that self-weighing frequency was associated with weight loss (and despite the strong correlation between weight values measured on the same day), there was only moderate agreement between the frequency of self-weighing as assessed by each method. While participants both weighed themselves via the study smart scale and self-reported their weight on most days during the intervention, participants self-weighed via the smart scale without self-reporting weight on over twice as many days as they self-reported weights without using the study smart scale (an average of 7 versus 3 days per participant, respectively). Weights measured on days when participants completed only one method of self-weighing were, on average, higher than weights on days when participants did both.

These results have several important implications. First, the magnitude of associations between frequency of self-weighing and weight change observed in the current study were similar across measures (rs of −.47 and −.39 for smart scale and self-report, respectively), and similar to those found in previous studies (rs ranging from −.28 to −.47);^5,14–16 thus, both appear to represent valid measures for assessing self-weighing frequency in the context of weight loss. Given that both methods demonstrated missing data on days that data was recorded using the other method, however, it appears that both methods may underestimate the “true” frequency of self-weighing behavior.

As there was not substantial improvement in variance explained by the multivariate model compared to the univariate models, there may not be additional benefit, in terms of weight loss, to having participants self-report weight in addition to self-weighing via a smart scale. Thus, for studies that can provide smart scales to participants, integrating these tools within intervention such that participants are not required to also self-report weight may offer the benefit of decreasing participant burden while not negatively impacting intervention outcome. With this stated, we also consider it important to mention that implementation may require careful consideration. Smart scale data can be “messy;” while some smart scales can distinguish between users and only transmit data on a target user, others (e.g., the scale that we used in this study) transmit all data, including weights of other members in the household who use the scale. Even with a protocol asking participants to store the scale somewhere private and to discourage other members of the household from using the scale, our weight dataset included improbable weights likely resulting from other individuals stepping on the scale. The density of smart scale datasets make cleaning these data solely through visual checks impractical, and thus algorithms should be developed to detect outlier values.

Conversely, for studies that cannot feasibly use smart scales (e.g., due to funding restrictions), self-report weight values seem to be accurate compared to objectively-measured smart scale data, and self-report appears to adequately capture the frequency of self-weighing. Given that the current results suggest that individuals self-weigh more frequently than they self-report their weight, assessing frequency of self-weighing only via self-report may lead to an underestimation of actual self-weighing behavior. To improve accuracy of data collection in research studies utilizing self-report of self-weighing, it may be helpful to specifically emphasize the importance of accurately self-reporting these values at the start of the study.

Second, the finding that smart scale weights were higher on days that participants only self-weighed via the smart scale or only self-reported weights via the study website, compared to days when they did both, has important methodological implications. These results suggest that missing self-report weight data collected during intervention are unlikely to be ignorable (e.g., missing at random).²⁹ While we do not have data regarding the reasons that participants used only one method on a given day, there may be differences between measurement methods. Some researchers have argued that missing self-report weights demonstrate general program non-adherence;³⁰ while this may explain days on which participants did not self-weigh using either method, it may not adequately explain days on which participants self-weighed via smart scale but did not self-report. Previous research has also suggested that participants may avoid weighing themselves on days that they anticipate higher weights (e.g., after a day of higher-than normal caloric intake);³¹ it is possible that participants may also avoid self-reporting weights that are higher than expected/wanted after self-weighing (e.g., due to social desirability or feelings of frustration, shame, or embarrassment). Conversely, missing smart scale weights on days that participants self-reported weight values may reflect participant travel outside of the home (and thus lack of access to the study-provided smart scale). While we did not collect data on this pattern, study staff reported fielding questions from participants on how to continue intervention participation while traveling (e.g., for work or vacation). Weights on these days may be higher due to disruptions in regular eating/activity patterns and/or from self-weighing via a different scale at later times during the day (versus first thing in the morning, before having anything to eat or drink, as recommended protocol for using the study smart scale). Other researchers have noted technical difficulties with smart scales leading to missing data and requiring some participants to identify other locations (beyond their home, such as workplaces) to use the smart scale;¹³ however, these difficulties were not observed in the current study.

Interestingly, the trend for weights to be higher when only one method of self-weighing was used significantly increased over time. This may have important treatment implications given that declining adherence to program goals has been identified as the primary driver of weight-regain after initial weight loss.³² Combined with results from previous studies (including results indicating that missing weight values may be an early indicator of weight gain³⁰), current results highlight the potential for using non-adherence to self-weighing as treatment tailoring variable. In a study by Steinberg and colleagues,¹³ participants who received targeted feedback based on self-weighing frequency demonstrated greater adherence to self-weighing and greater weight loss compared to a wait-list control. Future research should investigate whether more proximal intervention tailoring may further improve outcomes (e.g., using an adaptive treatment design,^33,34 a study could investigate whether providing additional/tailored intervention to individuals after a missed day or two of self-weighing improves weight loss outcomes).

The current study had several important limitations. First, participants received small financial incentives for reporting self-monitoring data (caloric intake, physical activity, and weight) via the study website each week, which may have made participants more likely to self-weigh or to self-report weight. As discussed earlier, these incentives were provided for self-reporting data and not self-monitoring itself (i.e., participants could still receive the incentive for indicating that they did not self-monitor), which hopefully limited impact on self-weighing behavior. If anything, these incentives may have led to an inflation in the frequency of self-reporting weight; if a study were to replicate our methods without these incentives, even lower agreement between methods (e.g., a greater tendency for individuals to self-weigh using the smart scale but not self-report their weight) may be observed. Second, participants knew that researchers had access to smart scale data, which may have limited the tendency to over- or under-report weight values (potentially inflating agreement between weight values measured by each method). While participants were told during the informed consent process that the smart scales sent weight data directly to research servers, the use of these data to validate self-report weights was not emphasized. Further, smart scale features were not discussed or used during the intervention (indeed, lack of synchronization between the smart scales and our study website meant that participants had to self-report weights each week to receive feedback on progress; participants did not have access to the smart scale data during the intervention and it was not monitored by researchers). Finally, participants in the current sample were all employees or dependents of employees of a healthcare corporation and were predominately white and female, limiting generalizability of study findings.

The current study also had several notable strengths. This study was the first to investigate concurrent agreement between weights collected via smart scale and those self-reported via self-monitoring records during a behavioral weight management intervention. Further, the richness of our dataset allowed us to compare accuracy of both measures for assessing both weight values and self-weighing frequency, allowing us to replicate and extend existing literature on self-weighing and weight loss outcomes during behavioral weight management programs. The only other study to investigate agreement between self-weighing frequency assessed via smart scale and self-report used a single-item, retrospective question with categorical responses and was conducted in the context of a randomized trial assessing the impact of prescribing varying frequencies of self-weighing (daily versus weekly) on weight change.¹⁶ In contrast to the current results, which demonstrated that assessing frequency of self-weighing via self-report underestimated self-weighing frequency compared to smart scale data, this previous study found that this single-item self-report question overestimated self-weighing frequency compared to smart scale data. The authors stated that the categories used on the single-item question may have been insufficient at capturing the true frequency of self-weighing; however, they also noted potential contamination due to study design as participants may have felt compelled to self-report a frequency that matched their randomized group. The current study design, with self-weighing assessed via smart scale and self-report during a standard behavioral intervention (not specifically focused on self-weighing) heightens generalizability of the current results.

Conclusion

The current results suggest excellent agreement between weights measured via smart scales and via self-report. Although greater frequency of self-weighing was associated with greater weight loss across both methods, there was only moderate agreement between the frequency of self-weighing behavior assessed via each method. Participants self-weighed via smart scale on more days than weight was self-reported. Further, participant weights were higher on days that they had data from only one method compared to days that they had data from both (i.e., when they self-weighed using the smart scale but did not self-report their weight on the study website, or vice versa), suggesting that missing weight data should not be treated as ignorable. Future research should investigate whether interventions providing more proximal feedback delivered in response to self-weighing behavior can improve weight loss outcomes.

What is already known about this subject?

• Research has documented a consistent association between greater frequency of self-monitoring of weight (“self-weighing”) and improved weight loss outcomes for participants in behavioral weight management programs.

• Researchers have traditionally relied on self-report to assess self-weighing frequency; however, new “smart” scales (which send weights directly back to research servers) can be used to objectively assess participant weight and frequency of self-weighing.

• One recent study demonstrated moderate to poor agreement for self-weighing frequency between a retrospective self-report question and smart scale data; however, no studies have compared self-report self-weighing data and smart scale data collected concurrently during an intervention.

What does this study add?

• Results demonstrated a strong association between the weight values collected via smart scale and via self-report; however, there was only moderate agreement between measures when each was used to assess the frequency of self-weighing.

• Participants weighed themselves using the smart scale on more days than they self-reported their weights via a study website, and smart scale weights on these days were higher than weights measured on days when participants also self-reported their weights.

• Participant self-report weights were also higher on days that they did not self-weigh using the smart scale compared to days that they did both.