Abstract
Objectives The lack of feasible and meaningful measures of clinicians' behavior hinders efforts to assess and improve obesity management in pediatric primary care. In this study, we examined the external validity of a novel algorithm, previously validated in a single geographic region, using structured electronic health record (EHR) data to identify phenotypes of clinicians' attention to elevated body mass index (BMI) and weight-related comorbidities.
Methods We extracted structured EHR data for 300 randomly selected 6- to 12-year-old children with elevated BMI seen for well-child visits from June 2018 to May 2019 at pediatric primary care practices affiliated with Yale. Using diagnosis codes, laboratory orders, referrals, and medications adapted from the original algorithm, we categorized encounters as having evidence of attention to BMI only, weight-related comorbidities only, or both BMI and comorbidities. We evaluated the algorithm's sensitivity and specificity for detecting any attention to BMI and/or comorbidities using chart review as the reference standard.
Results The adapted algorithm yielded a sensitivity of 79.2% and specificity of 94.0% for identifying any attention to high BMI/comorbidities in clinical documentation. Of 86 encounters labeled as “no attention” by the algorithm, 83% had evidence of attention in free-text components of the progress note. The likelihood of classification as “any attention” by both chart review and the algorithm varied by BMI category and by clinician type ( p < 0.001).
Conclusion The electronic phenotyping algorithm had high specificity for detecting attention to high BMI and/or comorbidities in structured EHR inputs. The algorithm's performance may be improved by incorporating unstructured data from clinical notes.
Keywords: data validation and verification, primary care, pediatrics, specific conditions, quality, EHR systems
Background and Significance
Pediatric obesity is widespread and affects nearly one in five children in the United States. 1 2 3 Primary care providers (PCPs) provide a cornerstone of pediatric weight management by screening for elevated body mass index (BMI) and managing children with weight-related comorbidities including hypertension, hyperlipidemia, diabetes, and fatty liver disease. 4 5 Studies report that patients and families see their PCP as a reliable source of information and that the recommendations of PCPs can positively influence subsequent lifestyle modification attempts. 5 6 7 The U.S. Preventive Services Task Force recommends that clinicians screen all children 6 years and older for obesity using BMI, and the American Academy of Pediatrics issued recent guidelines for the management of childhood overweight and obesity to assist PCPs supporting BMI evaluation starting at age 2 years. 4 5
Despite the expanded use of electronic health records (EHRs) capable of automatically calculating BMI percentiles and providing electronic decision support to improve clinicians' recognition of elevated BMI, measures evaluating the diagnosis of children with overweight or obesity remain suboptimal. 8 9 10 11 Additionally, children with overweight are less likely than children with obesity to have documented diagnosis of overweight, diet and physical exercise counseling, and recommended screening studies. 9 10 12 Efforts to improve clinical practice are impeded by the lack of pragmatic measures of clinician adherence to these recommendations. Current quality metrics for weight screening and management used by the Healthcare Effectiveness Data and Information Set include BMI-related diagnosis codes that are not widely used by PCPs. 13 14 15 Intelligent use of EHRs to deliver decision support at the point of care and to track information about clinicians' recognition and attention to high BMI holds great potential for ensuring that clinicians do not miss opportunities to engage with families early to influence BMI trajectories and to provide consistent, evidence-based care.
Electronic phenotyping goes beyond simply identifying patients with disease characteristics to advancing population health by identifying patients with a disease who receive care consistent with expert recommendations. 11 16 17 18 19 In a 2018 study, Turer et al, published data describing the development and validation of an electronic phenotyping algorithm using structured data extracted from the EHR to detect evidence of pediatric clinicians' attention to elevated BMI and associated comorbidities consistent with the 2007 expert committee recommendation on childhood overweight/obesity. 20 The EHR data included diagnostic codes, laboratory studies, referrals to nutrition/weight management, and medications. Using manual chart review as the reference standard, the algorithm showed excellent sensitivity (94.8%) and specificity (97.9%) to detect attention to elevated BMI with or without attention to medical risk when applied to the EHR data of 6- to 12-year-old children with elevated BMI seen for primary care at 54 pediatric clinics in Dallas, Texas, United States ( Supplementary Table S1 , available in online version). 20 A follow-up study in 2019 found that children categorized by the algorithm as having attention to BMI alone or attention to BMI and medical risk were 20% more likely to improve relative BMI at follow-up. 21 Although these results are promising, the study included data from one geographic region in Texas and the generalizability of the algorithm to other geographic locations was unknown.
Objectives
In this study, we sought to replicate the electronic phenotyping algorithm published by the Texas group to assess its external validity in using structured EHR data to identify clinicians' attention to elevated BMI and weight-related comorbidities among pediatric primary care practices affiliated with the Yale Healthcare System. 20
Methods
Data Source
We examined randomly selected well-child encounters in the health records of 300 unique children aged 6 to 12 years with BMIs ≥85th percentile seen between June 1, 2018, and May 31, 2019, for ≥2 well-child visits at one of three pediatric primary care practices utilizing the Yale instance of Epic EHR (Epic Systems Corporation, Verona, Wisconsin, United States). If a child had multiple well-child encounters within the study period, the first well-child encounter was examined. Two of the primary care practices included in this study provide care to children with public insurance from predominately low-income and minority households, whereas the third practice is a private staff model health maintenance organization (HMO) that exclusively provides care to patients with managed health care from households with more variability in their socioeconomic status and racial/ethnic backgrounds. We excluded children with fewer than two separate visits at which the child's BMI was documented as ≥85 th percentile in the EHR to increase the likelihood that the child's BMI was truly elevated and not a measurement error. We also excluded children who were taking medications that impact weight (e.g., steroids) or had conditions that impact growth and nutrition (e.g., pregnancy, thyroid dysfunction, and hormonal abnormalities). This study was approved by the Yale Human Research Protection Program.
Measures and Data Collection
We extracted the following variables from the EHRs of eligible children:
Visit diagnoses, using International Statistical Classification of Disease and Related Health Problems, 10th Revision (ICD-10) codes, associated with the encounter.
Problem list ICD diagnosis codes entered on or up to 7 days after the date of the encounter.
Referrals associated with the encounter.
Procedures/lab orders associated with the encounter.
Medication lists queried for prescriptions written on the day of the encounter or up to 7 days after the date of the encounter.
Age at visit.
Height and weight measured at the visit to calculate BMI and BMI category: overweight (≥85th to < 95th percentile), obesity class 1 (≥95th to < 120% of the 95th percentile), and severe obesity class 2 and 3 (>120% of the 95th percentile). 22
Sex: male or female.
Race–ethnicity: non-Hispanic Black, non-Hispanic White, Hispanic, and non-Hispanic Asian/Other.
Health insurance: public (Medicaid), private (Blue Cross Blue Shield, managed care, or other commercial insurer), and uninsured/other (self-pay or missing).
Clinician type: nurse practitioner (NP), physician assistant (PA), physician (attendings and fellows), and resident physician.
Construction and Implementation of Algorithm
Using criteria for the electronic phenotype described in the 2018 Texas study, 20 we analyzed visit-associated diagnostic billing codes, problem-list codes, laboratory studies, medications, procedures, and referrals for our cohort to categorize visits into the following broad phenotypes: no attention, attention to BMI alone, attention to comorbidities alone, and attention to both BMI and comorbidities ( Fig. 1 ). We further subclassified attention to comorbidities into subtypes of attention to diabetes, to fatty liver disease, to hyperlipidemia, to hypertension, and to vitamin D deficiency. Detailed criteria for classifying visits into attention types are listed in Supplementary Table S2 (available in online version) for the broad attention-to-high-BMI phenotypes and in Supplementary Table S3 (available in online version) for the comorbidity subtypes. Criteria were enumerated by (1) reviewing the ICD-9 diagnostic codes used in the 2018 study and converting them to ICD-10 codes (using website http://www.icd10codesearch.com/ ), then (2) reviewing the laboratory studies, medications, referrals, and procedures described in the 2018 study and reconciling each with the corresponding data elements in the Yale cohort dataset.
Fig. 1.
Overview of phenotypes of attention assigned by the algorithm based on EHR data. Refer to Supplementary Tables S2 and S3 (available in the online version) for specific diagnostic codes, laboratory orders, medication prescriptions, and referrals for each attention type. BMI, body mass index; EHR, electronic health record.
Reference Standard
To evaluate the validity of the electronic phenotyping algorithm in the Yale cohort, we used independent chart reviews to identify the adapted algorithm's ability to detect attention to high BMI. Additionally, the chart review allowed detailed examination of both data elements included in the algorithm (e.g., diagnostic codes and laboratory studies) as well as unstructured documentation of the visit encounter to identify evidence of attention to high BMI or comorbidities that the algorithm was not designed to detect, such as free-text or uploaded/scanned media entries. This chart review process was performed in accordance with the methods published by Turer et al to maintain fidelity with the original chart review validation, with modifications described below. 20
The abstraction guide and questionnaire from the 2018 validation study were reviewed, converted from Word documents to an online Qualtrics survey (Qualtrics, Provo, Utah, United States) and adapted to allow chart review using the Yale Health System EHR to reflect institutional differences in EHR layout and note templates, and to direct chart reviewers to the location of specific information. Survey items that did not align with the study aims, such as questionnaire items in the original survey used to collect data for separate projects, were removed. We also modified and added questions to the chart abstraction questionnaire to improve the clarity of the questionnaire and to ensure the correct data were reconciled between the algorithm and the chart review. Examples of this included prompts to assess not only laboratory results but also incomplete laboratory orders placed by clinicians and to manually review visit and problem list diagnostic codes associated with an encounter. Supplementary Document S1 (available in online version) displays the final chart review guide and questionnaire.
Chart reviews included review of each child's BMI percentile on the day of the visit using the growth chart to verify that the child met criteria for overweight/obesity. Reviewers then systematically examined visit notes and visit-associated information in the contemporaneous problem list, medication list, laboratory studies, family history, and externally uploaded media. Each chart was reviewed systematically using the Qualtrics survey. First, two separate reviewers (A.G.B. and A.M.F.) independently reviewed a sample of 10% of the 300 charts, and responses to each survey question were compared. Discrepant responses were resolved using discussion among the reviewers and consultation with a third-party reviewer (M.S.) as needed. Interrater reliability was measured using the kappa statistic and interpreted using the guidelines outlined by Landis and Koch. 23
Our chart selection and attention assignment categories differed from the 2018 Texas study in a few ways to further reduce the potential for bias in chart review, to focus our scope on well-child care, and to retain additional granularity in attention types. The 2018 study included 100 randomly selected charts from each attention type (no attention, attention to BMI, attention to comorbidity), whereas, in this study, we sought to limit potential reviewer bias by including 300 randomly selected charts from our cohort without consideration of attention type. Attention types were assigned by the algorithm to each chart after completion of the chart review. In the Yale cohort, we focused on encounters that were exclusively coded as well-child visits, whereas in the Texas cohort, non-well-child visits were included in the analysis. Another difference between the original 2018 study and our validation study was the use of an “attention to comorbidities alone” category. While the Texas study did classify encounters that had attention to comorbidities alone, these encounters were ultimately adjudicated to “no attention” in both chart review and algorithm.
Statistical Analysis
Demographic information was summarized using mean and standard deviation for continuous variables and frequency and percent for categorical variables ( Table 1 ).
Table 1. Demographics of the sample of 300 encounters examined.
| Characteristics | Mean ± SD or N (%) |
|---|---|
| Age, y | 10 ± 1.9 |
| Race and ethnicity | |
| Asian/other | 33 (11.0%) |
| Hispanic/Latino | 128 (42.7%) |
| Non-Hispanic Black | 94 (31.3%) |
| Non-Hispanic White | 45 (15.0%) |
| BMI category | |
| Overweight | 130 (43.3%) |
| Obesity | 124 (41.3%) |
| Severe obesity | 46 (15.3%) |
| Clinician type | |
| Attending | 53 (17.7%) |
| Nurse practitioner | 75 (25.0%) |
| Physician assistant | 25 (08.3%) |
| Resident physician | 147 (49.0%) |
| Season | |
| Fall | 72 (24.0%) |
| Spring | 62 (20.7%) |
| Summer | 106 (35.3%) |
| Winter | 60 (20.0%) |
| Health insurance type | |
| Private | 44 (14.7%) |
| Public | 194 (64.7%) |
| Other/missing | 62 (20.7%) |
| Clinic | |
| Nonacademic practice | 35 (11.7%) |
| Staff model HMO | 74 (24.7%) |
| Academic practice | 191 (63.7%) |
Abbreviations: BMI, body mass index; HMO, health maintenance organization; SD, standard deviation.
Our primary aim was to determine the algorithm's sensitivity and specificity for detecting any attention to high BMI and/or comorbidities. Evidence of attention to high BMI and/or comorbidities in the encounter per chart review served as the reference standard.
In a secondary analysis, we used chi-square tests to examine the extent to which patient, clinician, and contextual factors (including baseline BMI category, clinician type, and clinical setting) were associated with differences in clinicians' attention to high BMI. The analysis was performed using attention type adjudicated by both chart review and the algorithm. We also examined the algorithm's sensitivity and specificity stratified by these factors. We hypothesized that the algorithm may have a higher sensitivity and specificity in children with more severe obesity (class 2 or 3), in encounters conducted by attending physicians versus other clinicians, and for visits in academic versus nonacademic clinical settings.
Kappa statistics were computed by hand for interrater reliability ( n = 2) of the chart review. All other analyses were completed using SAS version 9.4 (SAS Institute, Cary, North Carolina, United States). A p -value of 0.05 was considered statistically significant.
Results
Algorithm Validation
Of the 30 charts reviewed in duplicate by two reviewers (A.G.B., A.M.F.), 27 were included in the analysis; 3 did not meet the BMI threshold for inclusion. Kappa scores suggested substantial interobserver agreement: 0.73 (95% confidence interval: 0.52–0.94) on the category of attention assigned (no attention, attention to BMI alone, attention to comorbidities alone, or attention to BMI and comorbidities). Of the 300 charts reviewed by at least one reviewer, we categorized them as follows: (1) 102 visits as having evidence of attention to BMI, (2) 144 as attention to BMI and comorbidities, (3) four as attention to comorbidities alone, and (4) 50 as no attention. In contrast, the algorithm categorized (1) 93 visits as having evidence of attention to BMI, (2) 84 as attention to BMI and comorbidities, (3) 24 as attention to comorbidities alone, and (4) 99 as no attention. Thus, the prevalence of any attention to elevated BMI (alone, in combination with attention to comorbidities or as attention to comorbidities alone) using chart review was 83% (250/300) and using the algorithm was 67% (201/300). Comparison between chart review and algorithm assignments yielded a sensitivity of 79.2%, specificity of 94.0%, for the algorithm's ability to identify any attention to BMI and/or comorbidities ( Table 2 ).
Table 2. Attention assignments by the algorithm and by chart review.
| Algorithm-assigned attention type | Chart review attention type | ||||
|---|---|---|---|---|---|
| BMI | BMI and comorbidities | Comorbidities only | No attention | Total | |
| BMI | 66 | 27 | 0 | 0 | 93 |
| BMI and comorbidities | 2 | 82 | 0 | 0 | 84 |
| Comorbidities only | 0 | 18 | 3 | 3 | 24 |
| No attention | 34 | 17 | 1 | 47 | 99 |
| Total | 102 | 144 | 4 | 50 | 300 |
Abbreviation: BMI, body mass index.
In our examination of encounters in which the reviewers found evidence of attention and the algorithm did not (false negatives), discordance between the algorithm and chart review was frequently attributable to evidence of attention in the form of free text within the progress note ( Fig. 2 ). Of 52 false negative encounters, the majority had free-text evidence of attention to BMI in the “Subjective” (49%) and/or in the “Assessment and Plan” (83%) portions of the progress note. In examining false positive encounters, the algorithm identified three encounters as having evidence of attention to comorbidities due to the presence of laboratory studies, diagnosis codes, or referrals satisfying the algorithm's definition of attention to comorbidities, but chart review revealed that these inputs were not related to a clinician communicating about an elevated BMI or weight-related comorbidities. For example, there was an encounter at which a clinician ordered screening lipids and a vitamin-D level that was flagged as attention to comorbidities, yet chart review revealed that the lipids were ordered as part of universal screening at a 10-year-old's well-child and the vitamin D screen was ordered as part of routine screening for refugee patients.
Fig. 2.
Summary of overlap and discrepancies between algorithm and chart review.
Predictors of Attention to Body Mass Index/Comorbidities
Table 3 displays the prevalence of attention to BMI and/or obesity-related comorbidities stratified by patient, clinician, and contextual factors. BMI category, clinician type, and clinic type were associated with differences in the prevalence of attention to high BMI and/or comorbidities ( p < 0.001 for chart review and for algorithm).
Table 3. Identification of attention to body mass index and/or weight-related comorbidities by the chart review and by the algorithm and the sensitivity and specificity of the algorithm stratified by patients' body mass index category, clinician type, and primary care practice.
| Characteristic | Chart review | Algorithm | Any attention by algorithm vs. chart review | |||
|---|---|---|---|---|---|---|
| Attention a | p -Value a | Attention a | p -Value a | Sensitivity (95% CI) |
Specificity (95% CI) |
|
| Overall | ||||||
| BMI category | ||||||
| Overweight n = 130 |
95 (73.1%) | <0.001 | 70 (53.8%) | <0.001 | 71.6% (61.4, 80.4) | 94.3% (80.8, 99.3) |
| Obesity n = 124 |
110 (88.7%) | 93 (75.0%) | 83.6% (75.4, 90.0) | 92.9% (66.1, 99.8) | ||
| Severe obesity n = 46 |
45 (97.8%) | 38 (82.6%) | 84.4% (70.5, 93.5) | 100% (90.8, 100) | ||
| Clinician type | ||||||
| Attending n = 53 |
50 (94.3%) | <0.001 | 40 (75.5%) | <0.001 | 80.0% (66.3, 90.0) | 100% (29.2, 100) |
| Nurse practitioner n = 75 |
52 (69.3%) | 40 (53.3%) | 73.1% (59.0, 84.4) | 91.3% (72.0, 98.9) | ||
| Physician assistant n = 25 |
8 (32.0%) | 5 (20.0%) | 50.0% (15.7, 84.3) | 94.1% (71.3, 99.9) | ||
| Resident n = 147 |
140 (95.2%) | 116 (78.9%) | 82.9% (75.6, 88.7) | 100% (59.0, 100) | ||
| Practice | ||||||
| Nonacademic clinic n = 35 |
16 (45.7%) | <0.001 | 12 (34.3%) | <0.001 | 68.8% (41.3, 89.0) | 94.7% (74.0, 99.9) |
| Academic clinic n = 191 |
171 (89.5%) | 136 (71.2%) | 79.0% (72.1, 84.8) | 95.0% (75.1, 99.9) | ||
| Staff model HMO n = 74 |
63 (85.1%) | 53 (71.6%) | 82.5% (70.9,91.0) | 90.9% (58.7, 99.8) | ||
Abbreviations: BMI, body mass index; CI, confidence interval; HMO, health maintenance organization.
Attention to BMI, BMI and comorbidities, or comorbidities only.
Chi-squared test.
The algorithm's sensitivity and specificity for detecting attention to high BMI and comorbidity varied by BMI category, clinician type, and clinical setting, but our confidence intervals around these stratified estimates overlapped suggesting that any observed difference in magnitude was not statistically significant.
Discussion
In this study, we evaluated the external validity of an electronic phenotyping algorithm that uses structured data stored in EHRs to detect clinician attention to elevated BMI and associated comorbidities during pediatric primary care visits. Using manual chart review as the reference standard, our adapted electronic phenotyping algorithm's sensitivity to detect any type of attention (to high BMI, comorbidities, or both) was 79.2% and specificity 94.0%.
Factors associated with clinicians' attention to BMI and to comorbidities included BMI category (children with overweight had lower prevalence of attention, consistent with previous research 9 10 12 ), clinician type (children seen by NPs and PAs had lower prevalence of attention than resident physicians or attendings), and clinic site (one of the nonacademic clinics had less documented attention to BMI compared with the academic and staff model HMO clinics).
We found that the algorithm had a higher sensitivity and specificity for detecting any type of attention in children with severe obesity, and in encounters with certain clinician types and clinic settings. For example, the sensitivity and specificity of using structured data available in the EHR for identifying attention to high BMI and comorbidities using chart review was higher among children with more severe obesity and in encounters staffed by a physician. In terms of clinic setting, the algorithm had the highest observed sensitivity at academic clinics and the highest specificity at the staff model HMO clinic; however, the differences were numeric (based on magnitude) and not statistically significant (confidence intervals around these stratified estimates overlapped). When considering that the algorithm's performance depends on structured data in the EHR, these findings are compatible with those from other studies that have shown children with obesity are more likely to receive diagnoses denoting high BMI (e.g., obesity) than children with BMIs in the overweight range. 8 10
The sensitivity to detect attention in the Texas study was 94.8%, higher than our finding of 79.2%, and specificity was 97.9%, similar to our finding of 94.0%. The lower sensitivity we observed compared with the original study may be due in part to differences in how we defined attention to BMI and comorbidities. For example, in the 2018 study, children with evidence of attention to comorbidities but no evidence of attention to high BMI were assigned to the “no attention” group. In our external validation, we categorize these children as having “attention to comorbidities”—a separate category not examined by the original algorithm—and we assigned them to the “attention to BMI/comorbidities” group. The Texas group examined validity in both well-child and problem-focused (nonwell-child) visits and refined their algorithm using data from academic, community, and private practice clinics that purchased their EHR service through the Children's Health of Dallas. The algorithm's performance was improved through several iterations of code review and revision using data from one geographic region. Yet, clinical and documentation practices that form the basis for the algorithm's structured inputs (e.g., diagnosis codes and laboratories ordered as part of obesity-related comorbidities evaluations) likely vary geographically and temporally as coding systems change (e.g., ICD-9 to ICD-10 and beyond). One example of this is the treatment of vitamin-D deficiency as an obesity-related comorbidity. The 2018 study included a lipid panel or a vitamin D level as evidence of high-BMI comorbidity screening if either laboratory was completed on the exact same day as a laboratory study for a separate comorbidity (e.g., a hemoglobin A1c) or in association with a high-BMI visit diagnosis. However, in practice sites included in this external validation study, vitamin D deficiency is part of routine screening among a relatively large refugee population regardless of weight status. This contributed to the algorithm assigning attention to a comorbidity (yielding a false positive). Site-specific nuances in clinical practice such as this limit the generalizability of the original algorithm and suggest that consensus is needed regarding how to phenotype clinician health care delivery to patients with a disease. For example, our findings suggest that using screening for vitamin D deficiency performed within a time window that includes lipid screening is not specific for a clinician screening for one or more obesity-related comorbidities. Even though we did not require performance of both laboratory tests on the exact same day (as in the 2018 study), we would still recommend removing screening for vitamin D deficiency from the electronic phenotyping algorithm.
Despite our adapted algorithm's lower sensitivity than the original algorithm, the use of multiple EHR inputs to automate detection of clinicians' attention to high BMI/comorbidities holds promise for improving the detection of attention compared with the use of diagnosis codes alone. In a 2011 to 2015 study of Yale pediatric primary care clinics, the proportion of children with a visit diagnosis related to overweight or obesity was 11% among children with overweight, 37% among children with class 1 obesity, and 54% among children with severe obesity class 2 or 3. 24 Other studies examining the use of diagnosis codes for overweight and obesity have shown similarly low rates. 9 25 26 Thus, although assignment of a diagnosis of obesity or overweight may remain suboptimal, one can harness a clinician's use of other EHR functions (e.g., orders for studies and medications) to denote attention to BMI and weight-related comorbidities.
A notable limitation of the electronic phenotyping algorithm is that it currently uses only structured data and not the large amount of unstructured data in EHR encounters from free text in clinical notes. 27 Clinicians have suboptimal rates of entering BMI-related diagnosis codes that may reflect the historically poor reimbursement of BMI-related care, and clinicians variably use the problem list to document care over time. Yet, clinicians rely on visit progress notes to communicate care information both for their own use and that of their colleagues at future visits. Visit notes also serve as important medical–legal documents. We found that, of the encounters that were incorrectly labeled as no attention by the algorithm, the majority had evidence of BMI attention per documentation in the assessment and plan and/or in the subjective sections of the progress note. The exclusion of free-text elements in the progress note from the evaluation of clinical encounters excludes a key site that clinicians use to document BMI-related care. Future studies looking to continue use of electronic phenotyping to assess clinician behavior should include natural language processing to capture free-text/unstructured EHR data. Although this external validation replicated the application of the algorithm in a new setting, clinical context, and geographic location, our health system uses the same EHR and is likewise academically affiliated, which may limit the generalizability of our findings to the broader health care context. The algorithm's performance could vary in different regions, with diverse patient populations, or across varying EHR systems.
Conclusion
Our findings suggest that implementation of an electronic phenotyping algorithm without adaption for the local site may result in lower sensitivity than originally reported using data from a separate health system in which the algorithm is developed. Although still clinically useful and likely superior to approaches that rely on diagnosis codes only, the use of natural language processing could enhance the precision and accuracy of this electronic phenotyping algorithm as a pragmatic tool to detect clinicians' attention to elevated BMI, weight-related comorbidities, and potentially other aspects of obesity management.
Clinical Relevance Statement
The electronic phenotyping algorithm had high specificity for detecting attention to high BMI and/or comorbidities in structured EHR inputs. Algorithms that accurately and reliably identify recommended clinician practice from data entered or generated in the EHR as part of routine clinical care can serve as a “smaller, feasible automation opportunity,” which facilitates the efficient, automatic generation of information that can support quality improvement, streamline billing, and relieve clinicians' of documentation burden associated with low satisfaction with EHRs and burnout. 28 Specifically, such an algorithm could eliminate the need for clinician entry of counseling diagnosis codes (e.g., ICD-10 code Z71. 3 for Dietary Counseling and Surveillance) or clicking checkboxes to satisfy quality or billing requirements.
Multiple-Choice Questions
-
In which group was the electronic phenotyping algorithm most successful in identifying attention to overweight/obesity?
Patients seen by midlevel clinicians (PAs and NPs) compared with physicians
Patients seen in a community clinic, compared with a staff model HMO
Patients with severe obesity, compared with those who were overweight
Patients with incomplete EHR data, compared with those with full data
Correct Answer : The correct answer is option c. This study used EHR structured data to identify clinicians' attention to elevated BMI and associated comorbidities during pediatric primary care visits. In the population studied, children with obesity had a higher prevalence of attention compared with children with overweight. Since identification of attention depends on structured data in the EHR, these findings are compatible with those from other studies that have shown children with obesity are more likely to receive diagnoses denoting high BMI (e.g., obesity) than children with BMIs in the overweight range. Children seen by NPs and PAs had a lower prevalence of attention, as did those seen in a community clinic. As the algorithm depended on availability of EHR structured data, attention is more often identified in patients with complete data.
-
Why can't the EHR enhance identification of clinical care for overweight/obesity?
Clinicians more often enter data about obesity in specific obesity-related structured data fields compared to using free text
Clinicians frequently and reliably enter overweight or obesity as problems within the EHR, making these diagnoses rapidly identifiable
An algorithm can easily and appropriately parse the free text within progress notes used by clinicians to record this information
An algorithm can pull together a variety of data from different sections of the EHR that are entered by clinicians
Correct Answer : The correct answer is option d. Electronic phenotyping aggregates a variety of data entered in the chart in real time by clinicians to identify these clinicians' attention to specific clinical concerns, in this case, overweight or obesity. Free text has historically been difficult to electronically parse and was not included in this study; however, free-text elements in the progress note remain a key EHR location used by clinicians to document BMI-related care. As artificial intelligence algorithms grow more sophisticated, future studies may examine how these algorithms identify attention to obesity from within the clinical notes while maintaining patient privacy. The problem list, which is unfortunately not always accurately updated, is in any case not an accurate identifier of attention since it is static information that does not require regular updating.
Funding Statement
Funding This study was supported in part by Agency for Healthcare Research and Quality under Award Number grant K08HS024332, by the National Institute on Minority Health and Health Disparities of the National Institutes of Health (NIH) under Award Number R01MD014853, and by Clinical and Translational Science Awards Grant Number UL1TR001863 from the National Center for Advancing Translational Science, a component of the NIH.
Conflict of Interest None declared.
Data Availability Statement
To protect the privacy of individuals included in the study, the data underlying this article cannot be shared publicly. The data can be shared on reasonable request to the corresponding author.
Protection of Human and Animal Subjects
This study was approved by the Yale Human Research Protection Program.
Supplementary Material
References
- 1.Hales C M, Fryar C D, Carroll M D, Freedman D S, Ogden C L. Trends in obesity and severe obesity prevalence in US youth and adults by sex and age, 2007-2008 to 2015-2016. JAMA. 2018;319(16):1723–1725. doi: 10.1001/jama.2018.3060. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Skinner A CRS, Ravanbakht S N, Skelton J A, Perrin E M, Armstrong S C. Prevalence of obesity and severe obesity in US children, 1999-2016. Pediatrics. 2018;141(03):e20173459. doi: 10.1542/peds.2017-3459. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Hu K, Staiano A E. Trends in obesity prevalence among children and adolescents aged 2 to 19 years in the US from 2011 to 2020. JAMA Pediatr. 2022;176(10):1037–1039. doi: 10.1001/jamapediatrics.2022.2052. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.US Preventive Services Task Force . Grossman D C, Bibbins-Domingo K, Curry S J et al. Screening for obesity in children and adolescents: US Preventive Services Task Force recommendation statement. JAMA. 2017;317(23):2417–2426. doi: 10.1001/jama.2017.6803. [DOI] [PubMed] [Google Scholar]
- 5.Hampl S E, Hassink S G, Skinner A C et al. Clinical practice guideline for the evaluation and treatment of children and adolescents with obesity. Pediatrics. 2023;151(02):e2022060640. doi: 10.1542/peds.2022-060640. [DOI] [PubMed] [Google Scholar]
- 6.Ackard D M, Neumark-Sztainer D. Health care information sources for adolescents: age and gender differences on use, concerns, and needs. J Adolesc Health. 2001;29(03):170–176. doi: 10.1016/s1054-139x(01)00253-1. [DOI] [PubMed] [Google Scholar]
- 7.Kao T A, Ling J, Hawn R, Vu C. The effects of motivational interviewing on children's body mass index and fat distributions: a systematic review and meta-analysis. Obes Rev. 2021;22(10):e13308. doi: 10.1111/obr.13308. [DOI] [PubMed] [Google Scholar]
- 8.Imoisili O E, Goodman A B, Dooyema C A, Harrison M R, Belay B, Park S. Screening and referral for childhood obesity: adherence to the U.S. Preventive Services Task Force recommendation. Am J Prev Med. 2019;56(02):179–186. doi: 10.1016/j.amepre.2018.10.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Sharifi M, Rifas-Shiman S L, Marshall R et al. Evaluating the implementation of expert committee recommendations for obesity assessment. Clin Pediatr (Phila) 2013;52(02):131–138. doi: 10.1177/0009922812471712. [DOI] [PubMed] [Google Scholar]
- 10.Reyes I. An evaluation of the identification and management of overweight and obesity in a pediatric clinic. J Pediatr Health Care. 2015;29(05):e9–e14. doi: 10.1016/j.pedhc.2015.05.002. [DOI] [PubMed] [Google Scholar]
- 11.Lingren T, Thaker V, Brady C et al. Developing an algorithm to detect early childhood obesity in two tertiary pediatric medical centers. Appl Clin Inform. 2016;7(03):693–706. doi: 10.4338/ACI-2016-01-RA-0015. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Camp N L, Robert R C, Dawes C S, Kelly K P. Identification of overweight and obesity in low-income minority children by pediatric providers and child characteristics associated with underrecognition. J Pediatr Health Care. 2019;33(02):162–168. doi: 10.1016/j.pedhc.2018.07.007. [DOI] [PubMed] [Google Scholar]
- 13.Taveras E M, Marshall R, Kleinman K P et al. Comparative effectiveness of childhood obesity interventions in pediatric primary care: a cluster-randomized clinical trial. JAMA Pediatr. 2015;169(06):535–542. doi: 10.1001/jamapediatrics.2015.0182. [DOI] [PubMed] [Google Scholar]
- 14.Taveras E M, Marshall R, Sharifi M et al. Comparative effectiveness of clinical-community childhood obesity interventions: a randomized clinical trial. JAMA Pediatr. 2017;171(08):e171325. doi: 10.1001/jamapediatrics.2017.1325. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Weight Assessment and Counseling for Nutrition and Physical Activity for Children/Adolescents (WCC)Accessed May 20, 2020 at:https://www.ncqa.org/hedis/measures/weight-assessment-and-counseling-for-nutrition-and-physical-activity-for-children-adolescents/
- 16.Banda J M, Seneviratne M, Hernandez-Boussard T, Shah N H. Advances in electronic phenotyping: from rule-based definitions to machine learning models. Annu Rev Biomed Data Sci. 2018;1:53–68. doi: 10.1146/annurev-biodatasci-080917-013315. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Hewner S, Smith E, Sullivan S S. Identifying high-need primary care patients using nursing knowledge and machine learning methods. Appl Clin Inform. 2023;14(03):408–417. doi: 10.1055/a-2048-7343. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Joseph A, Mullett C, Lilly C et al. Coronary artery disease phenotype detection in an academic hospital system setting. Appl Clin Inform. 2021;12(01):10–16. doi: 10.1055/s-0040-1721012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Samal L, Wu E, Aaron S et al. Refining clinical phenotypes to improve clinical decision support and reduce alert fatigue: a feasibility study. Appl Clin Inform. 2023;14(03):528–537. doi: 10.1055/s-0043-1768994. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Turer C B, Skinner C S, Barlow S E. Algorithm to detect pediatric provider attention to high BMI and associated medical risk. J Am Med Inform Assoc. 2019;26(01):55–60. doi: 10.1093/jamia/ocy126. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Turer C B, Barlow S E, Sarwer D B et al. Association of clinician behaviors and weight change in school-aged children. Am J Prev Med. 2019;57(03):384–393. doi: 10.1016/j.amepre.2019.04.029. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Flegal K M, Wei R, Ogden C L, Freedman D S, Johnson C L, Curtin L R. Characterizing extreme values of body mass index-for-age by using the 2000 Centers for Disease Control and Prevention growth charts. Am J Clin Nutr. 2009;90(05):1314–1320. doi: 10.3945/ajcn.2009.28335. [DOI] [PubMed] [Google Scholar]
- 23.Landis J R, Koch G G. The measurement of observer agreement for categorical data. Biometrics. 1977;33(01):159–174. [PubMed] [Google Scholar]
- 24.Lydecker J A, Grilo C M. The missed diagnosis and misdiagnosis of pediatric obesity. Psychother Psychosom. 2017;86(03):173–174. doi: 10.1159/000452501. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Young E L. Increasing diagnosis and treatment of overweight and obese pediatric patients. Clin Pediatr (Phila) 2015;54(14):1359–1365. doi: 10.1177/0009922815581762. [DOI] [PubMed] [Google Scholar]
- 26.Reed M, Cygan H, Lui K, Mullen M. Identification, prevention, and management of childhood overweight and obesity in a pediatric primary care center. Clin Pediatr (Phila) 2016;55(09):860–866. doi: 10.1177/0009922815614350. [DOI] [PubMed] [Google Scholar]
- 27.Kennell T I, Jr, Willig J H, Cimino J J. Clinical informatics researcher's desiderata for the data content of the next generation electronic health record. Appl Clin Inform. 2017;8(04):1159–1172. doi: 10.4338/ACI-2017-06-R-0101. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Zayas-Cabán T, Okubo T H, Posnack S. Priorities to accelerate workflow automation in health care. J Am Med Inform Assoc. 2022;30(01):195–201. doi: 10.1093/jamia/ocac197. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
To protect the privacy of individuals included in the study, the data underlying this article cannot be shared publicly. The data can be shared on reasonable request to the corresponding author.