Accuracy of Parental Report and Electronic Health Record Documentation as Measures of Diet and Physical Activity Counseling

Abstract

Objective

To determine whether parental reports and electronic health record documentation of physician counseling on nutrition and physical activity reflect actual counseling provided.

Methods

Participants were parents of 198 children 2–12 years of age seen in a primary care pediatric clinic at an academic medical center for well child care and their 38 physicians. Parents completed a post-visit questionnaire to reported discussions on weight, nutrition and physical activity that occurred during the visit. Electronic health records were reviewed to measure documentation of these topics during the visit. Parental reports and records were compared to actual discussions based on coded audiotapes. Counseling was coded as having occurred if specific topics were mentioned during the encounter, however brief this mention was.

Results

Children were 48% female with a mean age of 5.4 years; 28% were overweight or obese. Sensitivity of parental report was high (70%–96%), but specificity was low (43%–78%), due to parents’ tendency to over-report counseling. Sensitivity of electronic health record documentation was generally low (40–53%) except for discussion of screen time (92%) and physical activity (88%); the specificity of this data source was also poor (42% and 21% respectively).

Conclusions

Electronic health record documentation may not be the most valid method of measuring physician counseling on weight, nutrition, and physical activity in pediatric primary care. Parental report using a questionnaire administered immediately after the visit is a better alternative in quality improvement or research studies when resources do not allow for direct observation, with the caveat that parents may over-report whether counseling was provided.

INTRODUCTION

Approximately 17% of children and adolescents between the ages of 2 and 19 years are obese and an additional 15% are overweight.¹ Pediatric obesity is a risk factor for the development of multiple physical and physiological problems in children, adolescents and adults.² Since the primary modifiable determinants of obesity include food intake and physical activity, the cornerstone of obesity prevention and management in children is behavior and lifestyle modification.^3–5

One measure of healthcare quality is adherence to evidence-based practice guidelines.⁶ The 2007 Expert Committee Recommendations on the Assessment, Prevention, and Treatment of Child and Adolescent Overweight and Obesity emphasize assessment of body mass index (BMI) and provision of diet and physical activity counseling during all well child visits.⁷ Counseling for nutrition and physical activity for children and adolescents is a quality measure in the Healthcare Effectiveness Data and Information Set (HEDIS) and a criterion for “meaningful use” of electronic health records (EHR) under the American Recovery and Reinvestment Act.^8,9 There is wide variation in how frequently clinicians adhere to these recommended practices. ^10–15

Rigorous assessment of interventions to increase clinician counseling on diet and physical activity requires validated counseling measures and instruments.¹⁶ Analysis of audiotapes is the most valid method of assessing counseling behaviors, but is time-consuming, expensive and requires rigorous staff training.¹⁷ Therefore, parental report or medical record documentation are often used as proxies for direct observation of clinician behavior. The accuracy of parental report varies by the acuteness and significance of the medical issue, length of time that has elapsed between the visit and reporting, and the child’s age.^18,19 Reliance on medical record documentation is also problematic because documentation shows low concordance with directly observed physician counseling, especially with respect to health behavior.^20–22

The goal of our study was to compare the sensitivity and specificity of parental report and EHR documentation with the “gold standard” of audiotape analyses of visits. To our knowledge, this study is the first to compare the accuracy of parental report and EHR documentation of diet and physical activity counseling with independent coders’ evaluation of audiotaped visits in pediatric primary care settings.

PATIENTS AND METHODS

Development of Parent Questionnaire

Questions in the parent questionnaire were derived from the Promoting Healthy Development Survey designed by The Child and Adolescent Health Measurement Initiative.²³ The parent questionnaire included demographic items and the following broad question: “Did your child’s doctor or other health provider talk with you about the following today”, followed by a list of topics. To the topics related to nutrition and television viewing already present, we added topics from the 2007 Expert Committee Recommendations, including sweetened drink intake, fruit and vegetable consumption, physical activity and weight.⁷ The parent questionnaire was developed in English and we determined from pilot testing that it would take approximately 15 minutes to complete. Five pediatricians in ambulatory practice reviewed the questionnaire for face validity; one pediatrician was a member of the committee that developed the Expert Committee Recommendations.

Recruitment of Parents and Physicians

Approval to conduct this study was obtained from the UC Davis Institutional Review Board. The study was conducted from April 2008-August 2009 at the University of California Davis Health System’s general pediatric clinic. Participants were physicians and parents of children between 2 and 12 years of age seen for well child care. A maximum of 24 patients were recruited from each physician’s panel and were eligible to participate regardless of their medical history. We identified eligible children the day prior to their visit and parents were invited to participate in the study by a research assistant in the waiting room before the visit. Parents were informed that the study would assess their perspective about the well child visit. Information on children’s age, sex, race/ethnicity, and parental education was collected from parental report. Physicians were informed that parent education would be assessed, but were not told the specific topics that would be evaluated, namely weight, nutrition and physical activity. All faculty physicians and residents in the practice were invited to participate.

Audiotaping and Coding of Visits

An audio-recorder was placed in the examination room in an inconspicuous location immediately after the parent and child entered the room. The complete visit was audiotaped with parents’ and physicians’ full knowledge. A coding form and codebook were developed with operational definitions for each discussion item and specific criteria for coding topics as having been discussed. The first 50 audiotapes were independently evaluated by two trained coders who were bachelor-level research staff members with training and experience in scoring audiotapes for similar research projects, as well as by the first author (U.S.), all of whom were blinded to parent questionnaire data. A consensus procedure was utilized using repetitive cycles of independent audiotape assessment, comparison of assessments, reaching consensus through discussion, revised operational definitions, and re-scoring until 95% agreement was reached. For most audiotapes this process was completed in two cycles. During this process, the codebook and coding form were revised to clarify issues where coders differed amongst themselves or with U.S. in their assessments. The remaining audiotapes were divided equally between the two coders for assessment. Coders reviewed all questions that arose with U.S. until coding was completed. We adapted these methods from those used by Pbert et al. for validation of a smoking cessation counseling exit interview.²⁴ Specific weight, nutrition and physical activity topics were noted as having been discussed if these topics were mentioned during the encounter, however brief this mention was.

Administration of Parent Questionnaire

A research assistant returned to the examination room immediately after the visit and provided parents with the questionnaire to complete in the clinic.

Electronic Health Record Review Process

Our Epic EHR system had been in place for approximately two years prior to the commencement of this study. Our clinic utilizes standardized well child visit templates that contain prompts to enhance data collection and documentation. There are separate templates for each age for annual well child visits. There is, however, overlap between well child visit templates in the two main content areas relevant to our study. The patient history section contains prompts for diet, child care, sleep, development and immunizations. The prompt for diet has a drop-down menu from which the user can select one or more of the following options: appropriate diet, picky eater, and an option to enter free-text. The anticipatory guidance section contains the text “Education Topics Reviewed” and a drop-down menu from which the user can select one or more of the following options: fluoride, TV/reading, diet, car seat, discipline/limit setting, exercise, safety, and an option to enter free-text. Our medical record review indicated that standardized templates were utilized for all patients included in this study.

Reviews of EHR were performed by a trained research assistant who is a doctoral student in epidemiology. Periodic audits were conducted by U.S. to maintain consistency. Data on height and weight were abstracted. Any characterization of patient weight or BMI in the visit note or problem list was considered a discussion of weight, irrespective of its extent or accuracy. Specific nutrition and physical activity topics were noted as having been discussed if these topics were documented in the EHR, however brief this mention was. We utilized definitions published by the Centers for Disease Control and Prevention to categorize weight status and corresponding BMI percentiles. Overweight was defined as BMI percentile between 85th to less than the 95th percentile. Obesity was defined as BMI percentile equal to or greater than the 95th percentile.²⁵

Statistical Analyses

Data analyses were conducted using survey data analysis procedures in SAS 9.2,²⁶ specifying visits from the same physician as a cluster to account for within-physician correlation when calculating standard errors and test statistics. The sensitivity and specificity of parental report and EHR documentation were calculated by considering audiotape to be the gold standard. The sensitivity of parental report (or EHR documentation) for a particular topic was defined as the number of visits in which the parent (or EHR documentation) correctly reported discussion of a particular topic divided by the total number of visits in which that topic was discussed according to the audiotape. Specificity was defined as the number of visits in which the parent (or EHR documentation) correctly reported that the topic was not discussed divided by the total number of visits in which the topic was not discussed. Paired comparisons between parental report and EHR documentation for sensitivity and specificity used Kish’s adaptation of McNemar’s test procedure for clustered survey data when the number of discordant observations was six or greater, and otherwise used an exact binomial McNemar test.²⁷ Statistical significance required a two-sided p-value <0.05.

Sample Size Calculation

Prior to data collection, we performed a sample size calculation with an anticipated enrollment of 30 physicians. We cautiously assumed that design effect might be as high as 1.5, inflating the target sample size for a sensitivity or specificity estimate. We targeted an enrollment of 204 visits and achieved an enrollment of 198 visits with usable audio recordings. However, better than expected physician recruitment resulted in smaller cluster-related design effects in our realized sample and narrower confidence intervals for many of our estimates.

RESULTS

Child, Parent and Physician Descriptive Variables

A total of 38 physicians and parents of 198 children (94% of eligible parents) were enrolled. Demographic information for participating children and their parents is presented in Table 1. Children were, on average, 5.4 years of age with 48% being female. Most children (63%) had public health insurance. Based on parental reports, 39% of children were White, 25% Black, 14% Asian, 2% Native American and 3% Pacific Islander; 34% of children were Hispanic. The parent who completed the questionnaire was most often the mother (76%). Approximately 90% of parents had completed high school. The majority of children were first-borns (51%). Approximately 12% of children in our sample were overweight and 16% were obese. Fifty six percent of children in the study were seen by faculty physicians alone, 23% by third year residents, 12% by second year residents, and 9% by first year residents.

Table 1.

Demographic information on children and parents/guardians, N=198

	Mean	SE
Child Demographics
Age (Years)	5.4	0.20
BMI percentile	58.9	2.4
Female	0.48	0.03
Public health insurance	0.63	0.08
Hispanic Ethnicity (All Races)	0.34	0.04
Race
African American/Black	0.25	0.05
Caucasian/White	0.39	0.04
American Indian/Alaska Native	0.02	0.01
Asian	0.14	0.04
Pacific Islander	0.03	0.01
Other Race	0.16	0.02
Parent/Guardian Demographics
Education
Less than 8th Grade	0.01	0.01
Some High School	0.09	0.02
High School Graduate	0.20	0.03
Some College / 2-year Degree	0.38	0.04
College Degree (4-year)	0.16	0.04
Graduate School	0.17	0.04
Relationship with Child
Mother	0.76	0.03
Father	0.19	0.02
Aunt/Uncle	0.01	0.001
Grandparent	0.03	0.01
Guardian	0.02	0.01

Discussions of Diet and Physical Activity by Audiotape Assessment

Table 2 compares the frequency of report of discussion topics by parent, EHR, and audiotape coding. Audiotape assessment, our standard of accuracy, indicated that weight was discussed in 87% of visits, fruits and vegetables were discussed in 77% of visits, and sugary beverages were discussed in 54% of visits. All other discussion topics appeared in fewer than half of visits, ranging from 7% for family meals to 45% for screen time.

Table 2.

Frequency of report of discussion of specific topics; % (95% confidence intervals), N=198

Topic	Parental Report (%, 95% CI)	Medical record documentation (%, 95% CI)	Audiotape assessment (%, 95% CI)
Weight	90 (86, 94)	39 (25, 53)	87 (82, 92)
Fruits/Vegetables	80 (71, 89)	44 (29, 59)	77 (66, 87)
Outside Food	46 (35, 57)	15 (7, 23)	21 (13, 30)
Family Meals	27 (18, 36)	7 (3, 11)	7 (2, 13)
Breakfast	37 (27, 47)	14 (2, 26)	18 (11, 24)
Sugary Beverages	54 (40, 67)	35 (21, 50)	54 (43, 65)
Screen Time	56 (40, 73)	73 (60, 87)	45 (27, 63)
Physical Activity	73 (65, 81)	83 (73, 92)	42 (32, 52)

Accuracy of Parental Report

Discussions on all items except sugary beverages were reported more frequently by the child’s parent/guardian than on audiotape assessment, indicating a high level of false positive reports. “False positive” is used here to refer to topics noted in the parent questionnaire that were not noted during audiotape assessment. The rate of false positive reports was high, ranging from 23% for discussions on family meals to 57% for discussions on physical activity. As described in Table 3, parental report was highly sensitive for discussions related to weight, fruits and vegetables, screen time, and physical activity (92% - 96%), but less sensitive for discussions of outside food, breakfast, and sugary beverages (63% – 88%). Parental report had fairly low specificity in general, ranging from 43% for physical activity to 77% for family meals.

Table 3.

Sensitivity, specificity, positive and negative predictive value of parental report of physician counseling on nutrition and physical activity, N=198 visits

Topic	Sensitivity (%)			Specificity (%)			Positive Predictive Value	Negative Predictive Value
	Est.	95%	CI	Est.	95%	CI	Probability	Probability
Weight	96	94	99	52	31	73	0.93	0.68
Fruits/Vegetables	92	87	97	59	41	78	0.88	0.69
Outside Food	88	78	98	66	56	76	0.41	0.95
Family Meals	77	56	98	77	69	85	0.20	0.98
Breakfast	63	47	79	69	58	79	0.30	0.90
Sugary Beverages	77	66	88	74	58	87	0.78	0.73
Screen Time	93	86	100	73	64	81	0.74	0.93
Physical activity	94	89	99	43	30	55	0.54	0.91

Accuracy of Electronic Health Record Documentation

For EHR documentation, the rate of false positives (i.e., topics recorded in the EHR as discussed but not noted on audiotape assessment) ranged from 4% for discussion of family meals to 79% for discussion of physical activity. Sensitivity of EHR documentation ranged from 42% to 92% depending on the topic, with documentation of consumption of sugary beverages and screen time having high sensitivities of 92% and 88% respectively (Table 4). The specificity of EHR documentation was particularly high for documentation of outside meals, family meals, breakfast, and sugary beverages (86%–96%). Of note, the specificity of EHR documentation was markedly low for physical activity and screen time, 21% and 42% respectively.

Table 4.

Sensitivity, specificity, positive and negative predictive value of medical record documentation on nutrition and physical activity, N=198 visits

Topic	Sensitivity (%)			Specificity (%)			Positive Predictive Value	Negative Predictive Value
	Est.	95%	CI	Est.	95%	CI	Probability	Probability
Weight	42	26	57	76	60	92	0.92	0.16
Fruits/Vegetables	51	36	67	80	64	97	0.90	0.33
Outside Food	43	24	62	93	85	100	0.62	0.86
Family Meals	43	16	69	96	92	100	0.43	0.96
Breakfast	40	18	62	91	81	100	0.50	0.88
Sugary Beverages	53	39	67	86	71	100	0.81	0.61
Screen Time	92	85	99	42	19	65	0.57	0.87
Physical activity	88	80	96	21	8	34	0.45	0.71

Sensitivity of Electronic Health Record Documentation and Parental Report

Parental report was significantly more sensitive than EHR documentation for weight, fruits/vegetables, outside food, and sugary beverages. Parental report was more sensitive for all other topics as well, but not significantly so. In general, sensitivity of parental report was highest for topics that were discussed most frequently, resulting in higher negative predictive value estimates on these topics for parental report compared to EHR documentation (Tables 2 to 4).

Specificity of Electronic Health Record Documentation and Parental Report

The specificity of EHR documentation was significantly higher for fruits/vegetables, outside food, family meals, and breakfast. Parental report had significantly better specificity for screen time and physical activity, topics that were discussed in about half of visits, leading to higher positive predictive value estimates for parent reports on these topics compared to EHR documentation.

DISCUSSION

Our study demonstrates that a parent questionnaire administered immediately after the clinical encounter has acceptable sensitivity in estimating whether discussions on weight, nutrition and physical activity occurred compared to a criterion measure of independent audiotape assessment of visits. However, parents tend to over-report counseling. Electronic health record documentation may not be the most valid method of measuring such counseling, since it resulted in underestimation of some discussions on weight and nutrition but markedly significant overestimation of discussions on screen time and physical activity, which may be related to nuances in EHR templates.

Rigorous assessment of the effectiveness of interventions to increase counseling on weight, nutrition and physical activity requires quantifying baseline levels of counseling and validating counseling measures and instruments. Our findings are applicable to clinical performance measurement and improvement since counseling for nutrition and physical activity are now nationally utilized quality measures.^8,9 It is therefore important that these measures accurately reflect what occurs during clinical care. Since EHR documentation underestimates the discussion of many topics related to weight and nutrition, and also has the potential to significantly overestimate counseling on certain topics specific to the design of local EHR templates, analyses based on such data may suffer from information bias. Our study indicates that, overall, parental report is a more sensitive source of information on counseling for nutrition and physical activity in pediatric primary care, when compared to EHR documentation.

Our findings are consistent with other studies that evaluated patient report as a measure of counseling on health behaviors. Pbert et al. found that that patient report correlated well with audiotape assessment of the clinical encounter to evaluate smoking cessation counseling and noted that discrepancies between patient reports and audiotapes were primarily attributable to over-reporting of counseling by patients.²⁴ Wilson et al found patient report to be an acceptable approach for measuring counseling on smoking and alcohol compared to audiotape assessment.²⁸ Specific to counseling topics addressed in our study, Sciamanna et al. demonstrated that adult patients’ report of physical activity counseling immediately following clinical encounters correlated well with audiotape evaluation.²⁹ Similarly, Pill et al found that adult patients’ recall of lifestyle counseling delivered during primary care was reasonably high.³⁰ Our study found that parents generally over-report discussion of topics related to weight, nutrition, and physical activity. It is possible that some parents may have recalled discussions that occurred at prior visits or provided desirable responses to please the interviewer or the interviewer’s perceived employer.

Regarding the accuracy of medical record documentation in measuring counseling, Wilson et al reported that medical record documentation to assess smoking and alcohol counseling delivered to adult patients significantly underestimated counseling.²⁸ Pill et al and DiMatteo et al also showed that medical record documentation significantly underestimated lifestyle counseling delivered to adult patients.^30,31 Wilson et al however noted that medical record documentation was an extremely specific measure of counseling, with 0% false positive rate.²⁸ We specifically assessed EHR documentation and found that it underestimated some discussions on weight and nutrition similar to the studies mentioned above, but markedly overestimated discussions on screen time and physical activity. An important issue related to utilizing EHR documentation as a measure of counseling is that nuances of specific systems, such as prompts, shortcuts or documentation templates, may contribute to over-documentation of certain topics. In our study, the low specificity of EHR documentation for screen time and physical activity could be partially explained by the common use of the phrase “television/ reading” in age-specific EHR templates utilized in our clinic that are incorporated into the patient instruction section. In many cases, it was clear from other parts of the visit documentation that reading had been discussed at length in a developmental or educational context and was the specific reason for use of this phrase; however, we could not exclude the possibility that discussions related to television had occurred. In abstracting those medical records, we therefore coded physical activity and screen time (a sub-category of physical activity) as having been discussed. We recommend that future studies utilizing medical record review ensure that EHR templates are designed to minimize overlap or ambiguity related to documentation of specific quality measures.

One limitation of our study that affects generalizability was that it was conducted at one site, namely an academic medical center’s outpatient clinic. Patients in our sample were seen by pediatric residents and faculty and approximately 40% of clinicians were pediatric residents. It is possible that parents at non-academic practices may have inherently different relationships and levels of continuity with their physicians, which may affect their recall of counseling. Our results may be relatively generalizable to other parent populations as there was substantial economic and educational diversity in our sample. However, our study was not powered to find significant variation in the accuracy of parental report and EHR documentation across patients with different characteristics, such as those with family history of obesity-related disease. All physicians in our study worked within one health system that utilized the EHR system, Epic. Documentation practices at clinical locations with non-EHR systems, other EHR systems, or variations in EHR tools and templates may yield different results. For example, it is possible that prompts in our EHR templates for counseling on screen time and exercise made it more likely that physicians in our study documented discussions on these topics.

Despite these limitations, strengths of our study include direct observation of physician-parent interactions, large sample size and specific focus on weight, nutrition and activity discussions during pediatric well child visits. Our study adds to existing knowledge regarding discrepancies between actual discussions occurring during clinical encounters, parental recall of discussions and EHR documentation. Nuances in EHR systems raise the potential for significant variation in documentation across sites. Our findings shed light on increasingly important issues related to how EHRs may contribute to accurate or inaccurate documentation. Since EHR documentation is increasingly being utilized for quality measurement, tools built into EHR systems must be designed to support accurate documentation and counseling consistent with recommended care, and to meet performance requirements for both care delivery and documentation. It is important to acknowledge that while our study primarily focused on comparing strategies to measure counseling, further research to determine the effectiveness of such counseling in improving patient outcomes is required.

In conclusion, EHR documentation may not be the most valid measure for performance assessment of counseling. Parental report using a questionnaire administered immediately after the clinical encounter is a better approach for measuring counseling for weight, nutrition and physical activity in research or quality improvement studies when resources do not allow for direct observation. Errors in parental report may be related to the specific questionnaire utilized as well as the context of administration, but these possibilities will need to be addressed in future research.

WHAT’S NEW

Parental report using a questionnaire administered immediately after the visit is a more valid method of assessing physician counseling on weight, nutrition, and physical activity in pediatric primary care compared to medical record documentation.

Abbreviations

BMI

body mass index

EHR

electronic health records