Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2012 Aug 20.
Published in final edited form as: Surg Obes Relat Dis. 2011 Mar 15;7(4):548–557. doi: 10.1016/j.soard.2011.02.009

Technology for Behavioral Assessment and Intervention in Bariatric Surgery

J Graham Thomas a, Dale S Bond a, David B Sarwer b, Rena R Wing a
PMCID: PMC3422738  NIHMSID: NIHMS284259  PMID: 21514246

Abstract

Patients’ behaviors have a substantial impact on postoperative outcomes following bariatric surgery. Thus, studying patients’ behaviors is essential to learning how to optimize postoperative outcomes. In order to be most effective, this research should employ the best tools available for assessing patient behaviors. Unfortunately, traditional methods of behavioral assessment (e.g., questionnaires and clinical interviews) rely primarily on patients’ retrospective self-report, which is often inaccurate. Despite their significant shortcomings, these types of assessments continue to predominate. However, technological advances now allow for much greater accuracy in the assessment of patient behaviors via devices such as accelerometers and palmtop computers. Accelerometers allow for patients’ physical activity to be measured objectively in great detail, in real-time, in patients’ natural environment. Ecological Momentary Assessment (EMA) via palmtop computer or mobile phone allows for assessment of important behaviors, such as eating and activity behaviors, to be measured with many of the same advantages. Furthermore, new computer-assisted technologies are in development that will further facilitate behavioral assessment. Technology also has the potential to play an important role in the delivery of behavioral interventions aimed at bariatric surgery patients, given that Internet-based treatments have already proven effective for non- surgical weight loss, are often highly cost-effective and easily disseminable. Future research should evaluate the efficacy of these programs for bariatric patients.

Keywords: technology, behavior, assessment, intervention, accelerometry, ecological momentary assessment, physical activity, diet, obesity, bariatric surgery

Importance of Patient Behaviors in Bariatric Surgery Outcomes

Although bariatric surgery is currently the most effective treatment for severe obesity, up to 25% of patients fail to achieve adequate weight loss(1-2), while others experience sizeable weight regain often within 2 years of surgery.(3) Although suboptimal outcomes can be partially attributed to surgical factors, patient behaviors are thought to play a prominent role as evidenced by the recent description of bariatric surgery as “behavioral surgery.”(4)

Despite the overall power and potential of bariatric surgery, patients often have difficulty adhering to recommended postoperative changes in eating and physical activity (PA) behaviors(5-8), which relate to poorer weight loss outcomes.(9-11) For example, Sarwer and colleagues found that poorer adherence to postoperative dietary recommendations was associated with smaller weight losses over time in 200 Roux-en-Y gastric bypass (RYGB) patients.(9) Bond et al. showed that among 122 RYGB patients who were inactive preoperatively, those who remained inactive at 1-year postoperatively lost significantly less weight compared to those who became active (or met national PA guidelines).(10) Similarly, Colles and colleagues found that smaller pre- to postoperative increases in leisure-time PA and more frequent recurrence of preoperative eating patterns were associated with poorer weight loss in 129 laparoscopic adjustable gastric banding (LAGB) surgery patients at 1-year postoperatively.(11) While the above studies appear to indicate the importance of assessing patient behaviors and the potential role of behavioral interventions in improving postoperative outcomes, they are limited by the use of subjective recall measures which may be prone to inaccuracies and bias.

In this review, we detail the limitations of using subjective recall measures to assess patient behaviors. We then discuss how newer measurement technologies such as pedometry/accelerometry and ecological momentary assessment (EMA) have recently been applied in the bariatric surgery population to provide real-time assessment of activity and eating behaviors, respectively. Additionally, we provide an example of how Internet technology in the form of interactive, individually-tailored online behavioral programs is being used to promote long-term postoperative weight loss success. Finally, we discuss potential future applications of technology for measurement and modification of patient behaviors and end with recommendations to improve the quality of behavioral assessment and intervention in the field of bariatric surgery.

Limitations of Using Subjective Recall Measures in Behavioral Bariatric Surgery Research

The measurement of patient behaviors in bariatric surgery research has relied largely on retrospective self-report measures such as questionnaires and clinical interviews, which depend on patients’ ability to recall behaviors, experiences, and/or environmental conditions over periods ranging from days to years. While it is assumed that patients are able to give an accurate report, this assumption has been rarely tested. A growing body of evidence suggests that individuals are highly accurate at recalling information about unique events (e.g., “the day I had surgery”), but highly inaccurate at recalling information about individual experiences that are “frequent, mundane, and irregular” (e.g., episodes of eating, physical activity, chronic pain, use of medication).(12) Given that behavioral researchers are frequently interested in the latter, and rarely interested in the former, these inaccuracies are, or should be, cause for concern.

The inaccuracies of retrospective self-reports are attributable to a combination of simple forgetting, a variety of mental “shortcuts” that individuals use to reconstruct their memory when forgetting has occurred, and a desire to please their health care providers by reporting “good behavior.”(12-14) For example, bariatric surgery patients may be asked about the frequency, size, and composition of their meals during follow-up visits with their surgeon and surgical support team (e.g., nurses, nutritionists). However, most individuals cannot remember the details of every meal and snack eaten within the last few days or weeks. Notably, when asked to recall their pattern of eating, patients will rarely give the accurate answer (i.e., “I can’t recall exactly.”). Instead, they will make assumptions about what their behavior must have been in order to produce an answer.(12, 15)

The assumptions that are used to reconstruct memory are susceptible to bias. For example, Consistency Bias may lead a patient is likely to assume that their past behavior is essentially the same as their current behavior, and so they will give a report about their behavior during the last month that actually only reflects their behavior during the most recent few days. Other examples of bias that negatively affect retrospective self report measures include Mood Congruent Memory Bias (i.e., patients are more likely to recall positive information when happy and negative information when unhappy), and the Peak Effect (i.e., patients are more likely to remember and generalize events of high intensity). For a more extensive discussion of biases that influence retrospective recall, see Schwarz, 2007 or Gorin & Stone, 2001.(13, 15)

Ecological validity is another cause for concern associated with retrospective self-report measures.(13-14, 16) Behavioral information that is collected in an artificial environment such as a research laboratory or physician’s office may not generalize to an individual’s natural environment. For example, as described in detail below, there is a substantial discrepancy between self-reported levels of PA obtained from bariatric surgery patients in a research setting, and levels of PA in the same individuals measured objectively via accelerometry in the natural environment.(17) Problems with generalizability are a well-known threat to the external validity of research findings, but like the other types of bias described above, this threat is often ignored by researchers and clinicians.

The problems associated with using self-report measures to assess health behaviors have been demonstrated empirically in a variety of studies. For example, studies using doubly-labeled water have shown that dietary recall measures are susceptible to systematic underreporting that is influenced by individual difference characteristics (e.g., body size, body dissatisfaction, dietary restraint, ethnicity).(18-20) Pencil-and-paper food diaries are subject to many of the same problems, partially because individuals tend not to complete them in real-time, as instructed.(21) Levels of adherence to self-care behaviors such as blood-glucose monitoring(22) and use of prescription medication(23) also differs substantially when measured objectively versus by self-report. Discrepancies between objective and self-report measures have also been demonstrated in bariatric surgery patients. For example, we have shown that accelerometer-determined changes in moderate-to-vigorous physical activity (MVPA) from pre- to 6-months postoperatively are much smaller than self-reported changes.(17)

Biased information collected via retrospective self-report measures may lead researchers to invalid conclusions that could influence postoperative outcomes. For example, a current controversy in the field of bariatric surgery concerns the role of binge eating in surgical weight loss outcomes. Early studies suggested that reports of preoperative binge eating may be associated with poor weight loss outcomes (24), while a more recent study indicates that patients with and without preoperative binge eating achieve similar weight losses at least within the first year after surgery.(25) The mixed conclusions of these studies may be due to inconsistent and/or inadequate assessment of binge eating behavior in this population. Thus, having valid and reliable data on patient behaviors is essential to resolving such controversies and developing appropriate behavioral guidelines for pre- and postoperative care. Recent advances in technology show great promise for addressing the limitations inherent in retrospectively measuring patients’ physical activity and eating behaviors, and assisting with patient care, as described in the following sections.

Technologies for assessing activity and eating behaviors in the bariatric surgery population

Real-time assessment of physical activity via objective monitors

To counter the biases and inaccuracies of PA questionnaires, researchers are increasingly using movement sensor technology in the form of pedometers and accelerometers to obtain an objective measure of habitual, free-living PA. Pedometers are small devices that detect hip movements during walking activity and provide a recording of total daily steps taken.(26) Given their low cost and ease of operation, pedometers are used in a variety of different research contexts including observational studies to determine the association between PA and various health parameters, and intervention studies as a motivational tool and/or outcome measure.(27) In addition, pedometers were recently used by Colles and colleagues to examine the relationship between PA changes and weight loss at 12-months after LAGB; findings showed that patients who recorded greater pre- to postoperative PA changes in their total daily step experienced greater weight loss.(11)

Accelerometers are small, battery-operated devices that attach to a certain part of the body (usually the waist, but also the arm and ankle) and use an integrated computer chip to continuously record movement (usually in 1-minute intervals). Although somewhat more costly, accelerometers provide distinct advantages over pedometers. First, most accelerometers are omnidirectional, indicating an ability to capture movement in multiple directions (i.e. vertical, as well as the anteroposterior and/or lateral planes), thus theoretically providing a more accurate assessment of total physical activity.(26) Second, data from accelerometers can be downloaded to a comuter and converted via software to activity counts and/or metabolic equivalents (METs) to determine intensity, frequency, and duration of activity. Additionally, newer technologies such as the SenseWear Pro Armband (SWA, Body Media, Pittsburgh, PA) provide potential for increased accuracy of PA measurement by combining accelerometry with measurement of physiological parameters such as heat flux, galvanic skin response and skin temperature (see Figure 1 for a depiction of the SenseWear Pro 2 Armband and example of data summary report).

Figure 1.

Figure 1

(A) SenseWear™ BMS armband accelerometer (B) Example of data output from a SenseWear™ BMS armband accelerometer in a study of physical activity in bariatric surgery patients. The participant was largely adherent to the measurement protocol, which required participants to wear the armband during waking hours only. As can be seen, on average each day, the participant wore the device for 13 hours and expended 2125 k/cal of which 182 k/cal were expended through 28 minutes of moderate intensity physical activity (i.e., ≥ 3 METs).

Accelerometers are increasingly used in population-based research to objectively examine patterns of PA and sedentary behaviors, adherence to national PA recommendations, and the relationship of PA and sedentary behaviors to obesity and associated comorbidities.(28-31) Accelerometers have also been employed within the context of large clinical trials involving behavioral weight loss interventions. For example, Jakicic and colleagues described the baseline PA patterns of 2,240 obese adults with type 2 diabetes in the Look AHEAD study.(32) Another study examined the PA levels of 1,648 overweight and obese individuals who were screened for the Weight Loss Maintenance trial.(33)

Although several studies have used accelerometers in non-surgical obese populations to quantify PA, few studies have used these devices to examine PA levels before and/or after bariatric surgery. King and colleagues used the ankle-worn StepWatch™ 3 Activity Monitor (SAM, OrthoCare Innovations, Washington, D.C.) to assess preoperative steps/day among 757 participants in a study from the Longitudinal Assessment of Bariatric Surgery (LABS) consortium.(34) A wide range of PA levels were observed, although the majority (80%) of participants were insufficiently active (<10,000 steps/day). Bond and colleagues have conducted three studies using two different devices, the waist-worn RT3 (StayHealthy, Monrovia, CA) and the SWA (BodyMedia, Inc., Pittsburgh, PA)(17, 35-36) The first study used the RT3 to compare moderate-to-vigorous physical activity (MVPA; ≥ 3 METs) levels in 38 preoperative patients and 20 normal-weight controls.(35) Findings showed that patients spent half as much time in MVPA compared to controls (26 min/day vs. 52 min/day) and that 68% of patients did not accumulate any weekly MVPA in ≥ 10 min bouts versus 13% of controls. In a subsequent study using the SWA, 42 preoperative patients were found to spend the vast majority (81%) of their time being sedentary.(36) The third study showed that objectively-measured daily MVPA minutes did not significantly change among 20 patients from pre- to 6-months postoperatively.(17)

It should be noted that using accelerometers to measure PA in bariatric surgery patients also involves special considerations and certain limitations. For example, the output generated by accelerometers requires additional processing to quantify time spent in different intensities of PA. Additionally, accelerometer count cut-offs for identifying different intensities of PA in the severely obese have not yet been established. Consequently, currently available thresholds that were determined in leaner samples may affect validity of PA estimates in the bariatric surgery population.

Real-time assessment of eating and activity behaviors via ecological momentary assessment (EMA)

Traditional measures of eating behavior such as pencil and paper food diaries and food intake questionnaires are notoriously unreliable because (a) participants are often non-adherent to the measurement protocol(37-38) (despite self-reports to the contrary)(21), (b) participants fail to give an accurate report of their eating behavior(39), and (c) the constructs measured by many questionnaires used to assess eating behavior are unclear and often not developed with the eating behaviors of bariatric surgery patients in mind.(40) Twenty-four-hour food recalls (often conducted by phone) may be somewhat more reliable, but tend to be expensive and time-intensive.(39) The National Cancer Institute (NCI) has developed the Automated Self-administered 24-Hour Recall (ASA24), an Internet-based dietary assessment tool which uses multi-level food probes to more accurately assess food types and amounts.(41) The ASA24 provides an animated audio character to guide respondents through the interview, and photographs to assist respondents in reporting portion sizes. The ASA24 is currently available to researchers in a beta form, and may provide a low-cost alternative for dietary assessment if accuracy and usability prove to be high.(41)

A variety of tools for the objective measurement of food intake are also in development.(42-43) In most cases, cameras in mobile phones or wearable devices are used to take photographs or videos of food that is consumed. The captured images are analyzed via computer, either automatically or in combination with a dietician, to determine the volume and composition of the food consumed. These novel technologies show promise for increasing the accuracy of dietary intake, but most are not fully mature and are not yet ready for widespread use.

Notably, all of the measures described above tend to focus on the composition of the diet, and do little to assess other important eating behaviors. However, there is a method of behavioral assessment (i.e., EMA) that is currently available, relatively inexpensive, able to measure a wide variety of behaviors, and thought to be less susceptible to bias.(12, 44) EMA typically employs palmtop computers or other handheld devices (e.g., mobile phones) to collect information about behaviors, experiences, and environmental conditions from individuals in real-time (i.e., as they occur) in their natural environment over multiple occasions. Participants may be instructed to answer questions via the device (a) when a pre-specified behavior or event occurs, such as an episode of eating or physical activity, (b) at regular intervals throughout the day, such as upon waking in the morning and before going to sleep at night, and/or (c) in response to a random prompt from the device, usually an audible tone. Random prompts are often used to avoid reactivity (i.e., alterations in behavior because of the EMA protocol). Each set of responses is time and date stamped so that it possible to know exactly when the device was used. This provides a significant advantage over traditional paper diaries, which are often completed retrospectively, despite instructions to complete them in real-time.(21)

Because EMA data are collected in real-time in the natural environment, they are thought to be less susceptible to the biases described above that negatively affect retrospective self-report measures, and measures occurring in artificial settings such as the research lab or clinic. Also, because individuals are assessed repeatedly over multiple occasions, it is possible to study dynamic processes such as the antecedents and consequences of behaviors.(12-13, 45) For example, one EMA study showed that negative affect tends to increase in the hours leading up to a binge eating episode, but decreases sharply immediately afterward in a sample of 131 patients with Bulimia Nervosa.(46) This study suggests (but does not prove) that binge eating may serve as a way to cope with unpleasant emotions for some individuals.

EMA is a common assessment strategy in behavioral medicine research, and has often been used to study health behaviors such as smoking(47), substance use and dependence(47-48), pain(49), mood states(50), and various other medical and psychiatric conditions. Furthermore, while EMA is commonly used to study eating disorders(14), it has very rarely been used to study obesity or obesity treatments. Some of the best examples of the use of EMA to study eating behavior are found in research of binge eating. For example, le Grange and colleagues used EMA to compare the mood and eating behavior of 42 overweight women with and without Binge Eating Disorder (BED).(51) Presence and absence of a BED diagnosis was made via clinical interview (i.e., the Structured Clinical Interview for the DSM IV) and a battery of self-report questionnaires. During the 2-week EMA protocol, participants were prompted to report on their eating behavior, mood, and circumstances at the time of eating. Surprisingly, the EMA protocol revealed equivalent rates of binge eating behavior in the two groups. Thus, the EMA revealed substantial binge eating behavior in the non-BED group that was not detected via the clinical interview or questionnaires. This study also showed that binge eating in both groups tended to be preceded by negative affect.

We are aware of only one published study that used EMA to assess the eating behaviors of bariatric surgery patients. Thomas and colleagues used a six-day EMA protocol to assess compliance with recommended eating behaviors of 21 bariatric surgery patients approximately six months after surgery.(52) Patients were highly compliant with certain eating behaviors, such as not eating and drinking at the same time. However, participants tended to eat meals and snacks less often than is recommended. More importantly, the meals and snacks that they eat were much larger and consumed more quickly than recommended, thus increasing the risk for complications and suboptimal weight losses.

In addition to the assessment of eating behaviors, EMA is also being used to assess PA behaviors. While accelerometers are currently the ideal way to measure most types of physical movement, EMA has the added advantage of providing a way to measure the context in which physical activity occurs, including behavioral antecedents and correlates of PA. For example, Dunton et al.(53) conducted an EMA study of the PA behaviors of 23 adults age 50+ and found that higher levels of self-efficacy and lower levels of negative affect prospectively predict, and having a positive social interaction was concurrently associated with, engagement in MVPA. Thus, increasing self-efficacy and teaching coping strategies for negative emotions may facilitate PA in older adults. EMA is also frequently used to evaluate the PA behaviors of adolescents (54-57), although it has never been used to study PA behaviors in a bariatric population.

The information obtained via EMA could have important implications for research and clinical practice. For example, the protocol described by Thomas et al.(52) could be used to determine which behaviors are most strongly associated with weight loss after bariatric surgery, in order to determine which behavioral recommendations should be prioritized. Additionally, it may be possible to determine environmental conditions and/or behavioral antecedents that may precede unhealthy eating behavior in patients with suboptimal weight loss. These antecedents would be logical targets for behavioral intervention delivered in person or via the Internet as described below.

EMA is not without limitations. For example, EMA incurs costs associated with acquiring and programming electronic devices. Participant burden can be high when individuals are asked to use the device to answer questions several times per day over several days or weeks. The complex data obtained via EMA often require sophisticated methods of data analysis, such as linear mixed models, which are able to accurately model multiple longitudinal observations, while accommodating missing data.

EMA is clearly not appropriate for every study. However, we believe that EMA should be routinely considered as a possible measurement strategy in behavioral bariatric surgery research, especially for large trials that could affect patient care. In these situations, it is especially important to obtain the most reliable and valid data possible, and the additional requirements of EMA are justified.

Technology for behavioral intervention in the bariatric surgery population

Internet-based behavioral interventions

Behavioral interventions have shown great promise for facilitating behavior change in related fields(58), and are just beginning to be used with bariatric surgery patients. For example, a recent study showed that a postoperative behavioral intervention administered to 30 vertical banded gastroplasty patients may improve excess weight loss (EWL) by as much as 25.7% at 36-months.(59) Despite being relatively new, behavioral interventions for bariatric patients have already begun to be delivered via the Internet. While Internet-based behavioral interventions for bariatric patients are largely unstudied, much has already been learned about how these Internet-based behavioral interventions can been applied to produce weight loss and successful maintenance in non-surgical populations.

Most Internet-based behavioral weight loss and weight maintenance interventions center on lifestyle modification.(60-62) The goal is to produce an energy deficit (or maintain energy balance in the case of weight maintenance) by modifying eating and physical activity habits. Several core strategies are used to facilitate the change in energy balance. First, patients set goals for weight loss, energy intake, and physical activity. Participants are then educated in the basic principles of nutrition and physical activity so that they may choose foods and exercises that will help them reach their goals. Self-monitoring of weight, eating, and activity behaviors helps patients track their daily progress towards their behavioral goals, and helps them to identify patterns in their behavior that may interfere with healthy weight control. Self-monitoring is perhaps the most important part of lifestyle interventions as it is strongly related to weight loss .(63) Participants are also taught to use stimulus control, which involves identifying cues that promote unhealthy eating and activity behavior (e.g., the availability of highly palatable food may cue overeating; the presence of a television in the home may cue sedentary behavior), modifying them (e.g., remove palatable foods from the home; setting a limit on television viewing), and adding cues that promote healthy behavior (e.g., setting fresh fruit or running shoes in a highly visible location in the home). When patients struggle and do not achieve desired outcomes, lifestyle modification provides the opportunity to identify barriers to success, challenge dysfunctional thoughts, and propose alternative strategies or different goals.

Internet-based weight control programs have several advantages when compared to traditional face-to-face treatment programs. Namely, treatment resources may be accessed as often as one likes, at any time, in any location that an Internet-connected computer is available. This eliminates costs and participant burden associated with clinic visits (e.g., travel, taking time off from working, childcare). The low cost and high reach of Internet-based interventions allows them to be made available to many more patients than could be accommodated via office visits.(64) Another notable advantage of Internet-based programs is the ability to tailor treatment to participants needs (e.g., via automated tailored feedback messages or by steering patients to pre-prepared materials that suit their particular needs). This can be especially powerful when paired with a robust self-monitoring system, which patients use to track important behavioral variables (e.g., weight, eating and activity).(65-66)

The outcomes of Internet-based behavioral weight loss interventions are highly variable, but several programs have achieved weight losses of 5.3% to 9.2% of initial body weight(67), which is similar to what is observed in traditional face-to-face treatment programs. The results seem to be enhanced when participants self-monitor important behavioral variables and receive personalized feedback from a human interventionist (although automated feedback is also effective and less costly).(65, 67) Providing resources to motivate and support participants (e.g., motivational tips, bulletin board, motivational mascot) may also improve outcomes.(68) As identified in a recent review, the largest problem associated with Internet-based weight control interventions is poor adherence to use of the website.(67) As many as 50% or more of participants may fail to use the website often enough to initiate the behavior changes that are necessary to produce health improvements.(69) Improving adherence to Internet-based interventions is an area of considerable research interest.

The design of Internet-based support programs for bariatric surgery patient should be informed by clinical practice guidelines(70), theory(71), and empirical research on Internet-based behavioral weight control interventions (described above). Specifically, patients should have access to information on recommended eating (e.g., meal frequency, dietary composition, portion size), and physical activity (e.g., type, frequency, intensity, duration) behaviors. They should also be encouraged to set goals for achieving compliance. Given that self-monitoring of behavior is one of the most important strategies for facilitating compliance, patients should be given tools for self-monitoring of their weight, diet, and physical activity. These tools should help patients track their progress towards their goals. Internet-based programs should also facilitate communication between patients and their surgical support teams. Ideally, the health providers should have access to patients’ self-monitoring records to facilitate treatment. Finally, Internet-based programs should contain information and support for long-term weight maintenance. Recent research shows that providing opportunities for peer interaction may help to promote motivation for sustained behavior change.(68)

Currently, there are no published studies that have evaluated the efficacy of Internet-based programs for support of the postoperative bariatric patient. However, there are at least two commercial programs offered by manufacturers of bariatric devices. Ethicon Endo-Surgery’s REALIZE mySUCCESS™ program, and Allergan’s My LAP-BAND Journey™, are examples of the application of behavioral modification theory and Internet technology specifically designed for bariatric surgery patients. These programs enable patients to personalize their behavioral weight loss strategy (both before and after surgery) while educating and training them to build healthy lifestyle routines. The websites provide patients with a number of features which allow them to monitor and track changes in their weight, food intake and physical activity. Recipes for healthy food to be consumed after bariatric surgery are available, as well as a variety of information to help patients make and sustain behavioral changes to optimize long term outcomes. In addition, the programs provide patient-reported data back to the bariatric surgery program, which give the bariatric team an opportunity to track patient progress, maintain contact and intervene when appropriate.

Additional research should be conducted to determine the best methods for using Internet-based behavioral interventions to improve bariatric surgery outcomes. This research should capitalize on developing technologies such as Ecological Momentary Intervention (EMI), which can be used to intervene with patients in real-time in their natural environment via a mobile phone or similar device.(72-73) EMI has already been used in a few studies to intervene on eating behaviors (74), weight loss (75), and physical activity (76) and the results are promising.

Summary

Behavioral research in bariatric surgery is important - identifying and intervening on specific eating and activity behaviors is necessary for optimizing surgical outcomes. However, previous research has largely relied on retrospective and often unvalidated measures to assess behavioral predictors of surgical outcomes. In this review, we presented technological advances in behavioral assessment that counter many of the limitations to previously used subjective recall measures, and discussed how two specific tools, accelerometry and EMA via palmtop computer, have been applied to assess PA and eating behaviors of bariatric surgery patients in real time and in their natural environment. In addition, we discussed how Internet-based behavioral interventions, which have been used successfully in non-surgical populations, are now being used to promote weight loss and maintenance after bariatric surgery.

Recommendations

Due to the limits of retrospective self-report instruments, we recommend that investigators conducting behavioral bariatric surgery research use measures that capture real-time performance of behaviors as they occur in the natural environment. Accelerometry provides an objective and relatively inexpensive measure of physical activity with minimal participant burden. Until validated objective measure of eating behavior can be implemented on a large scale, EMA affords researchers an improved capability to collect behavioral and environmental data in real-time in participants’ natural environment. Future research should employ accelerometry and EMA to examine the effect of PA, eating behavior, and environmental conditions on weight loss and other surgical outcomes within treatment control studies.

Recent research suggests that traditional face-to-face behavioral lifestyle interventions for weight control may be used to improve postoperative outcomes.(59) However, such interventions are costly, time intensive, and may have limited acceptability and reach for bariatric patients, who tend to exhibit poor attendance at follow-up visits.(9) Internet-based behavioral interventions show great promise for enhancing and maintaining postoperative outcomes. Several commercial programs already exist, and should be evaluated empirically in randomized clinical trials. The development of future Internet-based programs for bariatric patients should be informed by previous research on traditional Internet-based weight control programs, which have already identified effective methods of improving eating and physical activity behaviors.

Acknowledgements

This work was partially supported by a new investigator research grant from The Obesity Society (Thomas) and NIH grants DK083438 (Bond), R01-DK080738 (Sarwer) and R03-DK067885 (Sarwer). Dr. Sarwer also has additional grant funding from Ethicon Endo-Surgery.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Disclosures: Dr. Sarwer discloses that he has consulting relationships with Allergan, BaroNova, Enteromedics, and Ethicon Endo-Surgery. Authors Thomas, Bond, and Wing have no disclosures to report.

References

  • 1.Lufti R, Torquati A, Sekhar N, Richards WO. Predictors of success after laparoscopic gastric bypass: a multivariate analysis of socioeconomic factors. Surg Endosc. 2006;20:864–7. doi: 10.1007/s00464-005-0115-8. [DOI] [PubMed] [Google Scholar]
  • 2.Melton GB, Steele KE, Schweitzer MA, Lidor AO, Magnuson TH. Suboptimal weight loss after gastric bypass surgery: correlation of demographics, comorbidities, and insurance status with outcomes. J Gastrointest Surg. 2008;12:250–55. doi: 10.1007/s11605-007-0427-1. [DOI] [PubMed] [Google Scholar]
  • 3.Christou NV, Look D, Maclean LD. Weight gain after short- and long-limb gastric bypass in patients followed for longer than 10 years. Ann Surg. 2006;244:734–40. doi: 10.1097/01.sla.0000217592.04061.d5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Kral JG, Naslund E. Surgical treatment of obesity. Nat Clin Pract Endocrinol Metab. 2007;3:574–83. doi: 10.1038/ncpendmet0563. [DOI] [PubMed] [Google Scholar]
  • 5.Toussi R, Fujioka K, Coleman KJ. Pre- and post-surgery behavioral compliance, patient health, and postbariatric surgical weight loss. Obesity (Silver Spring) 2009;17:996–1002. doi: 10.1038/oby.2008.628. [DOI] [PubMed] [Google Scholar]
  • 6.Elkins G, Whitfield P, Marcus J, Symmonds R, Rodriguez J, Cook T. Noncompliance with behavioral recommendations following bariatric surgery. Obes Surg. 2005;15:546–51. doi: 10.1381/0960892053723385. [DOI] [PubMed] [Google Scholar]
  • 7.Welch G, Wesolowski C, Piepul B, Kuhn J, Romanelli J, Garb J. Physical activity predicts weight loss following gastric bypass surgery: findings from a support group survey. Obes Surg. 2008;18:517–24. doi: 10.1007/s11695-007-9269-x. [DOI] [PubMed] [Google Scholar]
  • 8.Bond DS, Phelan S, Leahey TM, Hill JO, Wing RR. Weight-loss maintenance in successful weight losers: surgical vs non-surgical methods. Int J Obes (Lond) 2009;33:173–80. doi: 10.1038/ijo.2008.256. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Sarwer DB, Wadden TA, Moore RH, et al. Preoperative eating behavior, postoperative dietary adherence, and weight loss after gastric bypass surgery. Surg Obes Relat Dis. 2008;4:640–6. doi: 10.1016/j.soard.2008.04.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Bond DS, Phelan S, Wolfe LG, et al. Becoming physically active after bariatric surgery is associated with improved weight loss and health-related quality of life. Obesity (Silver Spring) 2009;17:78–83. doi: 10.1038/oby.2008.501. [DOI] [PubMed] [Google Scholar]
  • 11.Colles SL, Dixon JB, O’Brien PE. Hunger control and regular physical activity facilitate weight loss after laparoscopic adjustable gastric banding. Obes Surg. 2008;18:833–40. doi: 10.1007/s11695-007-9409-3. [DOI] [PubMed] [Google Scholar]
  • 12.Stone AA, Shiffman S, Atienza AA, Nebeling L. Historical roots and rationale of Ecological Momentary Assessment (EMA) In: Stone AA, Shiffman S, Atienza AA, Nebeling L, editors. The Science of Real-Time Data Capture: Self-Reports in Health Research. Oxford University Press; New York: 2007. pp. 3–10. [Google Scholar]
  • 13.Schwarz N. Retrospective and concurrent self-reports: The rationale for real-time data capture. In: Stone AA, Shiffman S, Atienza AA, Nebeling L, editors. The Science of Real-Time Data Capture: Self-Reports in Health Research. Oxford University Press; New York: 2007. pp. 11–26. [Google Scholar]
  • 14.Smyth J, Wonderlich S, Crosby R, Miltenberger R, Mitchell J, Rorty M. The use of Ecological Momentary Assessment approaches in eating disorder research. Int J Eat Disord. 2001;30:83–95. doi: 10.1002/eat.1057. [DOI] [PubMed] [Google Scholar]
  • 15.Gorin AA, Stone AA. Recall biases and cognitive errors in retrospective self-reports: A call for momentary assessments. In: Baum A, Revenson T, Singer J, editors. Handbook of Health Psychology. Lawrence Erlbaum Associates; Mahwah: 2001. pp. 405–13. [Google Scholar]
  • 16.Schmuckler MA. What is ecological validity? A dimensional analysis. Infancy. 2001;2:419–36. doi: 10.1207/S15327078IN0204_02. [DOI] [PubMed] [Google Scholar]
  • 17.Bond DS, Jakicic JM, Unick JL, et al. Pre- to postoperative physical activity changes in bariatric surgery patients: self-report vs. objective measures. Obesity (Silver Spring) 2010;18:2395–7. doi: 10.1038/oby.2010.88. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Kimm SYS, Glynn NW, Obarzanek E, Aston CE, Daniels SR. Racial differences in correlates of misreporting of energy intake in adolescent females. Obesity (Silver Spring) 2006;14:156–64. doi: 10.1038/oby.2006.19. [DOI] [PubMed] [Google Scholar]
  • 19.Trabulsi J, Schoeller DA. Evaluation of dietary assessment instruments against doubly labeled water, a biomarker of habitual energy intake. Am J Physiol Endocrinol Metab. 2001;281:891–99. doi: 10.1152/ajpendo.2001.281.5.E891. [DOI] [PubMed] [Google Scholar]
  • 20.Westerterp KR, Goris AH. Validity of the assessment of dietary intake: problems of misreporting. Curr Opin Clin Nutr Metab Care. 2002;5:489–93. doi: 10.1097/00075197-200209000-00006. [DOI] [PubMed] [Google Scholar]
  • 21.Burke LE, Sereika SM, Music E, Warziski M, Styn MA, Stona A. Using instrumented paper diaries to document self-monitoring patterns in weight loss. Contemp Clin Trials. 2008;29:182–93. doi: 10.1016/j.cct.2007.07.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Wing RR, Epstein LH, Nowalk MP, Scott N, Koeske R. Compliance to self-monitoring of blood glucose: a marked-item technique compared with self-report. Diabetes Care. 1985;8:456–460. doi: 10.2337/diacare.8.5.456. [DOI] [PubMed] [Google Scholar]
  • 23.Garber MC, Nau DP, Erickson SR, Aikens JE, Lawrence JB. The concordance of self-report with other measures of medication adherence. Med Care. 2004;42:649–52. doi: 10.1097/01.mlr.0000129496.05898.02. [DOI] [PubMed] [Google Scholar]
  • 24.Hsu LK, Benotti PN, Dwyer J, et al. Nonsurgical factors that influence the outcome of bariatric surgery: a review. Psychosom Med. 1998;60:338–46. doi: 10.1097/00006842-199805000-00021. [DOI] [PubMed] [Google Scholar]
  • 25.Wadden TA, Faulconbridge LF, Jones-Corneille LR, et al. Binge eating disorder and the outcome of bariatric surgery at one year: a prospective, observational study. Obesity (Silver Spring) 2011 Jan 20; doi: 10.1038/oby.2010.336. [Epub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Corder K, Brage S, Ekelund U. Accelerometers and pedometers: methodology and clinical application. Curr Opin Clin Nutr Metab Care. 2007;10:597–603. doi: 10.1097/MCO.0b013e328285d883. [DOI] [PubMed] [Google Scholar]
  • 27.Bravata DM, Smith-Spangler C, Sundaram V, et al. Using pedometers to increase physical activity and improve health: a systematic review. JAMA. 2007;298:2296–304. doi: 10.1001/jama.298.19.2296. [DOI] [PubMed] [Google Scholar]
  • 28.Troiano RP, Berrigan D, Dodd KW, Masse LC, Tilert T, McDowell M. Physical activity in the United States measured by accelerometer. Med Sci Sports Exerc. 2008;40:181–8. doi: 10.1249/mss.0b013e31815a51b3. [DOI] [PubMed] [Google Scholar]
  • 29.Metzger JS, Catellier DJ, Evenson KR, Treuth MS, Rosamond WD, Siege-Riz AM. Patterns of objectively measured physical activity in the United States. Med Sci Sports Exerc. 2008;40:630–8. doi: 10.1249/MSS.0b013e3181620ebc. [DOI] [PubMed] [Google Scholar]
  • 30.Matthews CE, Chen KY, Freedson PS, et al. Amount of time spent in sedentary behaviors in the United States, 2003-2004. Am J Epidemiol. 2008;167:875–81. doi: 10.1093/aje/kwm390. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Sisson SB, Camhi SM, Church TS, Tudor-Locke C, Johnson WD, Katzmarzyk PT. Accelerometer-determined steps/day and metabolic syndrome. Am J Prev Med. 2010;38:575–82. doi: 10.1016/j.amepre.2010.02.015. [DOI] [PubMed] [Google Scholar]
  • 32.Jakicic JM, Gregg E, Knowler W, et al. Activity patterns of obese adults with Type 2 diabetes in the Look AHEAD study. Med Sci Sports Exerc. 2010;42:1995–2005. doi: 10.1249/MSS.0b013e3181e054f0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Young DR, Jerome GJ, Chen C, Laferriere D, Vollmer WM. Patterns of physical activity among overweight and obese adults. Prev Chronic Dis. 2009;6:A90. [PMC free article] [PubMed] [Google Scholar]
  • 34.King WC, Belle SH, Eid GM, et al. Physical activity levels of patients undergoing bariatric surgery in the Longitudinal Assessment of Bariatric Surgery study. Surg Obes Relat Dis. 2008;4:721–8. doi: 10.1016/j.soard.2008.08.022. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Bond DS, Jakicic JM, Vithiananthan S, et al. Objective quantification of physical activity in bariatric surgery candidates and normal-weight controls. Surg Obes Relat Dis. 2010;6:72–8. doi: 10.1016/j.soard.2009.08.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Bond DS, Unick JL, Jakicic JM, et al. Objective assessment of time spent being sedentary in bariatric surgery candidates. Obes Surg. 2010 Apr 16; doi: 10.1007/s11695-010-0151-x. [Epub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Baker R, Kirschenbaum D. Self-monitoring may be necessary for successful weight control. Behav Ther. 1993;24:377–94. [Google Scholar]
  • 38.Burke LE, Choo J, Music E, et al. Patterns of self-monitoring behavior affect weight change among weight loss study participants. Ann Behav Med. 2006;31(Suppl):S017. [Google Scholar]
  • 39.Lissner L, Potischman N. Measuring food intake in free-living populations: Focus on obesity. In: Allison DB, Baskin ML, editors. Handbook of Assessment Methods for Eating Behaviors and Weight-Related Problems: Measures, Theory, and Research. SAGE Publications; Los Angeles: 2009. pp. 255–81. [Google Scholar]
  • 40.Lowe MR, Thomas JG. Handbook of Assessment Methods for Eating Behaviors and Weight-Related Problems: Measures, Theory, and Research. SAGE Publications; Los Angeles: 2009. Measures of restrained eating: Conceptual evolution and psychometric update; pp. 137–85. [Google Scholar]
  • 41.Zimmerman TP, Hull SG, McNutt S, et al. Challenges in converting an interviewer-administered food probe database to self-administration in the National Cancer Institute Automated Self-administered 24-Hour Recall (ASA24) J Food Compost Anal. 2009;22:S48–S51. doi: 10.1016/j.jfca.2009.02.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Weiss R, Stumbo PJ, Divakaran A. Automatic food documentation and volume computation using digital imaging and electronic transmission. J Am Diet Assoc. 2010;110:42–4. doi: 10.1016/j.jada.2009.10.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Sun M, Fernstrom JD, Jia W, et al. A wearable electronic system for objective dietary assessment. J Am Diet Assoc. 2010;110:45–7. doi: 10.1016/j.jada.2009.10.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Glanz K, Murphy S. Dietary assessment and monitoring in real time. In: Stone AA, Shiffman S, Atienza AA, Nebeling L, editors. The Science of Real-Time Data Capture: Self-Reports in Health Research. Oxford University Press; New York: 2007. pp. 151–68. [Google Scholar]
  • 45.Shiffman S. Designing protocols for Ecological Momentary Assessment. In: Stone AA, Shiffman S, Atienza AA, Nebeling L, editors. The Science of Real-Time Data Capture: Self-Reports in Health Research. Oxford University Press; New York: 2007. pp. 27–53. [Google Scholar]
  • 46.Smyth JM, Wonderlich Sa, Heron KE, et al. Daily and momentary mood and stress are associated with binge eating and vomiting in bulimia nervosa patients in the natural environment. J Consult Clin Psychol. 2007;75:629–38. doi: 10.1037/0022-006X.75.4.629. [DOI] [PubMed] [Google Scholar]
  • 47.Shiffman S. Ecological momentary assessment (EMA) in studies of substance use. Psychol Assess. 2009;21:486–97. doi: 10.1037/a0017074. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48.Epstein DH, Marrone GF, Heishman SJ, Schmittner J, Preston KL. Tobacco, cocaine, and heroin: Craving and use during daily life. Addict Behav. 2010;35:318–24. doi: 10.1016/j.addbeh.2009.11.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Gendreau M, Hufford MR, Stone AA. Measuring clinical pain in chronic widespread pain: selected methodological issues. Best Pract Res Clin Rheumatol. 2003;17:575. doi: 10.1016/s1521-6942(03)00031-7. [DOI] [PubMed] [Google Scholar]
  • 50.Larsen RJ. Personality, mood states, and daily health. In: Stone AA, Shiffman S, Atienza AA, Nebeling L, editors. The Science of Real-Time Data Capture: Self-Reports in Health Research. Oxford University Press; New York: 2007. pp. 251–67. [Google Scholar]
  • 51.le Grange DL, Gorin A, Catley D, Stone AA. Does momentary assessment detect binge eating in overweight women that is denied at interview? Eur Eat Disord Rev. 2001;9:309–24. [Google Scholar]
  • 52.Thomas JG, Bond DS, Ryder BA, et al. Evaluation of postoperative adherence to recommended eating and activity behaviors in bariatric surgery patients’ natural environment via ecological momentary assessment. Surg Obes Relat Dis. 2010 doi: 10.1016/j.soard.2010.10.007. [Epub ahead of print] [DOI] [PubMed] [Google Scholar]
  • 53.Dunton GF, Atienza AA, Castro CM, King AC. Using ecological momentary assessment to examine antecedents and correlates of physical activity bouts in adults age 50+ years: a pilot study. Ann Behav Med. 2009;38:249–55. doi: 10.1007/s12160-009-9141-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Gorely T, Marshall SJ, Biddle SJ, Cameron N. Patterns of sedentary behaviour and physical activity among adolescents in the United Kingdom: Project STIL. J Behav Med. 2007;30:521–31. doi: 10.1007/s10865-007-9126-3. [DOI] [PubMed] [Google Scholar]
  • 55.Gorely T, Marshall SJ, Biddle SJ, Cameron N. The prevalence of leisure time sedentary behaviour and physical activity in adolescent girls: an ecological momentary assessment approach. Int J Pediatr Obes. 2007;2:227–34. doi: 10.1080/17477160701408833. [DOI] [PubMed] [Google Scholar]
  • 56.Dunton GF, Whalen CK, Jamner LD, Floro JN. Mapping the social and physical contexts of physical activity across adolescence using ecological momentary assessment. Ann Behav Med. 2007;34:144–53. doi: 10.1007/BF02872669. [DOI] [PubMed] [Google Scholar]
  • 57.Biddle SJ, Gorely T, Marshall SJ, Cameron N. The prevalence of sedentary behavior and physical activity in leisure time: A study of Scottish adolescents using ecological momentary assessment. Prev Med. 2009;48:151–55. doi: 10.1016/j.ypmed.2008.10.025. [DOI] [PubMed] [Google Scholar]
  • 58.Webb T, Joseph J, Yardley L, Michie S. Using the Internet to promote health behavior change: A systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res. 2010;12:1–18. doi: 10.2196/jmir.1376. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 59.Papalazarou A, Yannakoulia M, Kavouras SA, et al. Lifestyle intervention favorably affects weight loss and maintenance following obesity surgery. Obesity (Silver Spring) 2010;18:1348–53. doi: 10.1038/oby.2009.346. [DOI] [PubMed] [Google Scholar]
  • 60.Wadden TA, Crerand CE, Brock J. Behavioral treatment of obesity. Psychiatr Clin North Am. 2005;28:151–70. doi: 10.1016/j.psc.2004.09.008. [DOI] [PubMed] [Google Scholar]
  • 61.Wing RR. In: Behavioral approaches to the treatment of obesity. Bray GA, Bouchard C, editors. Informa Healthcare; New York: 2008. pp. 227–248. [Google Scholar]
  • 62.Bond DS, Wing RR. Behavioral interventions for the treatment of obesity in women with or at risk for cardiovascular disease. Current Cardiovascular Risk Reports. 2009;3:219–25. [Google Scholar]
  • 63.Burke LE, Wang J, Sevick MA. Self-monitoring in weight loss: a systematic review of the literature. J Am Diet Assoc. 2011;111:92–102. doi: 10.1016/j.jada.2010.10.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 64.Tate DF, Finkelstein EA, Khavjou O, Gustafson A. Cost effectivenessness of Internet interventions: Review and recommendations. Ann Behav Med. 2009;38:40–5. doi: 10.1007/s12160-009-9131-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 65.Tate DF, Jackvony EH, Wing RR. A randomized trial comparing human e-mail counseling, computer-automated tailored counseling, and no counseling in an Internet weight loss program. Arch Intern Med. 2006;166:1620–25. doi: 10.1001/archinte.166.15.1620. [DOI] [PubMed] [Google Scholar]
  • 66.Wing RR, Crane MM, Thomas JG, Kumar R, Weinberg B. Improving weight loss outcomes of community interventions by incorporating behavioral strategies. Am J Public Health. 2010;100:2513–9. doi: 10.2105/AJPH.2009.183616. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 67.Arem H, Irwin M. A review of web-based weight loss interventions in adults. Obes Rev. 2010 Aug 13; doi: 10.1111/j.1467-789X.2010.00787.x. [Epub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 68.Krukowski RA, Harvey-Berino J, Ashikaga T, Thomas CS, Micco N. Internet-based weight control: the relationship between web features and weight loss. Telemed J E Health. 2008;14:775–82. doi: 10.1089/tmj.2007.0132. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 69.Morgan PJ, Lubans DR, Collins CE, Warren JM, Callister R. The SHED-IT randomized controlled trial: evaluation of an Internet-based weight-loss program for men. Obesity (Silver Spring) 2009;17:2025–32. doi: 10.1038/oby.2009.85. [DOI] [PubMed] [Google Scholar]
  • 70.Blackburn GL, Hutter MM, Harvey AM, et al. Expert panel on weight loss surgery: executive report update. Obesity (Silver Spring) 2009;17:842–62. doi: 10.1038/oby.2008.578. [DOI] [PubMed] [Google Scholar]
  • 71.Ritterband LM, Thorndike FP, Cox DJ, Kovatchev BP, Gonder-Frederick La. A behavior change model for Internet interventions. Ann Behav Med. 2009;38:18–27. doi: 10.1007/s12160-009-9133-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 72.Heron KE, Smyth JM. Ecological momentary interventions: Incorporating mobile technology into psychosocial and health behaviour treatments. Br J Health Psychol. 2010;15:1–39. doi: 10.1348/135910709X466063. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 73.Tufano J, Karras B. Mobile eHealth interventions for obesity: a timely opportunity to leverage convergence trends. J Med Internet Res. 2005;7:e58. doi: 10.2196/jmir.7.5.e58. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 74.Atienza AA, King AC, Oliveira BM, Ahn DK, Gardner CD. Using hand-held computer technologies to improve dietary intake. Am J Prev Med. 2008;34:514–18. doi: 10.1016/j.amepre.2008.01.034. [DOI] [PubMed] [Google Scholar]
  • 75.Haapala I, Barengo NC, Biggs S, Surakka L, Manninen P. Weight loss my mobile phone: a 1-year effectiveness study. Public Health Nutr. 2009;12:2382–91. doi: 10.1017/S1368980009005230. [DOI] [PubMed] [Google Scholar]
  • 76.King AC, Ahn DK, Oliveira BM, et al. Promoting physical activity through hand-held computer technology. Am J Prev Med. 2008;34:138–42. doi: 10.1016/j.amepre.2007.09.025. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES