Abstract
We developed and validated an algorithm that uses combinations of extractable electronic-health-record (EHR) indicators (diagnosis codes, orders for laboratories, medications, and referrals) that denote widely-recommended clinician practice behaviors: attention to overweight/obesity/body mass index alone (BMI Alone), with attention to hypertension/other comorbidities (BMI/Medical Risk), or neither (No Attention). Data inputs used for each EHR indicator were refined through iterative chart review to identify and resolve modifiable coding errors. Validation was performed through manual review of randomly selected visit encounters (n = 308) coded by the refined algorithm. Of 104 encounters coded as No Attention, 89.4% lacked any evidence (specificity) of attention to BMI/Medical Risk. Corresponding evidence (sensitivity) of attention to BMI Alone was identified in 96.0% (of 101 encounters coded as BMI Alone) and BMI/Medical Risk in 96.1% (of 103 encounters coded as BMI/Medical Risk). Our EHR data algorithm can validly determine provider attention to BMI alone, with Medical Risk, or neither.
Keywords: body mass index, electronic health record, validation
Introduction
Advances in electronic-health-record (EHR) data analysis using code-based algorithms have created opportunities for characterizing patients with certain diseases and the quality of healthcare delivered to them without reliance on manual chart review.1,2 Prior studies have demonstrated that EHR data can be used to identify children with (1) overweight and obesity using weight and height data measured in clinics and stored in EHRs, and (2) overweight children with documented conditions known to be associated with overweight/obesity (including high blood pressure) using diagnosis codes.3,4 Yet, we previously have shown (using data from directly recorded visits with overweight children in primary care) that these measures alone are insufficient to denote provider communication regarding overweight/obesity and related conditions – clinical practices recommended by childhood obesity guidelines.5 Moreover, several studies document critically low rates of guideline-based screening and identification of obesity-related conditions such as hypertension.6,7 Thus, advances in EHR data analysis have not substantially improved our understanding of guideline-based healthcare delivered to children with overweight/obesity and related conditions.
Methods are needed to go beyond identifying patients with disease conditions – such as children with overweight/obesity – to determining delivery of guideline-based healthcare services to patients with the disease. This information is critical for (1) determining impact of guideline-recommended healthcare services on disease outcomes – for example, whether delivery of guideline-consistent weight-management and comorbidity care in primary care improves a child’s weight status – and (2) delivering next-generation decision support to promote clinician behaviors that improve disease outcomes. Therefore, the study aim was to use EHR data from primary-care visits of children with overweight and obesity to develop and validate an algorithm to identify clinician behaviors that denote delivery of widely recommended weight-management clinical practices to children meeting criteria for overweight and obesity, defined using Center for Disease Control definitions.8 The result will be an electronic phenotype for guideline-consistent clinician behaviors delivered to children with the disease.9–11
Methods
The study used EHR data from a cohort of 6-12 year-olds with overweight/obesity (≥2 measured body-mass-index (BMI) percentiles ≥85% documented in the EHR at ≥2 separate encounters) between October 2009 and May 2014 at 52 pediatric clinics in greater Dallas, TX.
This study, which was approved by the UT Southwestern Institutional Review Board, used de-identified data with a retained identifiable link (that allowed chart review) and waiver of informed consent.
To develop an algorithm to detect EHR evidence of recommended weight-management clinician behaviors during primary-care visits,9–11 we first developed a study protocol and generated a disease cohort using aggregated EHR data from the population of interest (Figure 1). Next, we outlined key features of guideline-recommended weight-management practice behaviors: identify high BMI, assess risk of certain obesity-related comorbidities (including high blood pressure, abnormal blood sugar, dyslipidemia, and fatty liver disease), and prevent and treat unhealthy weight gain and obesity-related comorbidities (Figure 2, Step 2). These clinical practices, first recommended by the Expert Committee in 2007, were adopted by the American Academy of Pediatrics, Bright Futures, and the National Heart, Lung, and Blood Institute, and are recommended by the US Preventive Services Task Force (who found Grade B evidence for identifying overweight and obesity at well-child visits by plotting BMI on growth charts).9–11
Figure 1.
Step 1: Develop protocol and generate disease cohort that will be used to develop an algorithm to determine widely recommended, guideline-consistent, clinician practice behaviors Figure 1 depicts the population, setting, and time frame used to develop the disease cohort, plus the intervention–guideline-recommended clinician practice behaviors – and comparison group. As shown, the time periods used to examine the unrefined variables during the EHR algorithm’s development were from one well-child visit to the next, approximately one year later. We also examined from one primary-care visit (not necessarily well-child) to the next visit.
Figure 2.
Steps 2–4: Develop algorithm that converts widely recommended clinical practices into an electronic phenotype for guideline-consistent clinician behaviors,8–10 then use disease cohort to test their association with disease outcomes Figure 2 depicts how widely recommended clinical practices for pediatric weight management for children with overweight and obesity were converted to potential electronic phenotypes and used to build guideline-consistent clinical practice behaviors – the weight-management clinical practice indicators examined in this analysis, including attention to high BMI Alone, attention to high BMI/Medical Risk, and No Attention (to BMI Alone or BMI/Medical Risk) – by detailing all possible evidence that could be extracted from an electronic health record (EHR), using the evidence to build unrefined clinical practice indicators, and examining these indicators for association with a key disease outcome (here, weight-status improvement).
To identify extractable EHR evidence that each practice behavior was delivered, we detailed diagnostic codes and orders (for laboratories, medicines, and procedures/referrals) that denoted attention to overweight/obesity/BMI alone (BMI Alone) or with attention to medical risk (BMI/Medical Risk) of hypertension, dyslipidemia, acanthosis/diabetes, fatty liver, or vitamin D deficiency (Figure 2, Step 3).
Then, using our disease cohort, we built unrefined electronic phenotypes and assessed associations of four categories (BMI Alone, Medical Risk Alone, BMI/Medical Risk, and No Attention) with weight-status improvement (Figure 2, Step 4a-b). Step 4b suggested that attention to Medical Risk Alone was not associated with weight-status improvement. To verify this, we reviewed evidence in a second disease cohort – 100 video-recorded primary-care visits in 6–12-year-olds with overweight and obesity, and found providers almost never discuss obesity-related comorbidities without discussing BMI. Therefore, attention to Medical Risk Alone was not included as a separate clinician behavior.
Next, we used iterative manual chart reviews to refine specific evidence used to define attention to BMI Alone, BMI/Medical Risk, or neither (No Attention) (Figure 3, Step 5). Chart reviews included reading progress notes, text-based fields, and scanned documents that had been uploaded to the EHR. We started by reviewing 30 algorithm-coded visit encounters (10 visits per indicator) to detect and resolve modifiable coding errors. We repeated cycles of reviewing visit encounters (30 algorithm-coded visits at a time), reviewing text-based fields in visit encounter-related progress notes, and revising codes (adding codes we had missed and removing incorrect codes) until we detected and resolved all modifiable coding errors.
Figure 3.
Steps 5–6: Finalize refined algorithm, then use it to examine validity in a random sample of the disease cohort. Figure 3 depicts that the final algorithm included two major recommended clinical practices for pediatric weight management, and that the final step in the study was to examine validity of the EHR algorithm compared to extensive chart review.
To validate the EHR algorithm, we used an additional 300 visit encounters (100 visits per indicator) randomly drawn from the overall cohort to examine each indicator’s ability to detect attention to BMI Alone, BMI/Medical Risk, or neither (No Attention) (Figure 3, Step 6). Some visit encounters (drawn at random) were duplicates of previous ones reviewed, and we replaced them by randomly drawing additional visit encounters. This resulted in review of 308 visit encounters total.
The primary study outcome was the algorithm’s ability to detect No Attention—that is, absence of evidence that a provider addressed BMI Alone or BMI/Medical Risk (for example, no text-based notation or evidence elsewhere in the visit encounter indicating a provider attended/discussed overweight/obesity/BMI or risk/presence of any obesity-related comorbidity). Secondary outcomes were the algorithm’s sensitivity to detect attention to high BMI Alone and BMI/Medical Risk. Analyses were conducted using SAS, version 9.4.
Results
The overall cohort included 7192 children at 52 clinics. The validation cohort was comprised of 298 children with a total of 308 visit encounters at 47 clinics. Of the validation cohort, 48% were female, 68% had obesity, 32% were overweight; 54% were Latino and 20% were Black or African American; 81% were publicly insured. Just over half (52%) were followed at the academic/teaching clinic, 29% at the community-based clinics, and 19% in private practices.
Coding errors we detected and resolved included: (a) adding overlooked diagnostic codes, particularly “V-codes” indicating observation, evaluation, or counseling (Table 1), (b) excluding medications unrelated to BMI/Medical Risk (eg clonidine and guanfacine were used to treat attention deficit rather than hypertension), and (c) requiring two laboratory screenings to increase likelihood that the labs were drawn to investigate obesity-related medical risk. After 10 iterations of chart review (300 total charts reviewed in development phase), no further modifiable coding errors were detected.
Table 1.
Electronic health record evidence (diagnostic codes, laboratory tests, and referral/procedure orders) used to determine provider attention to high BMI Alone or high BMI/Medical Risk in a cohort of children with overweight and obesity
| Clinical Practice | Diagnosis Code (ICD-9) | Medicines and Laboratory Tests | Referrals and Procedures |
|---|---|---|---|
|
Attention to high BMI Alone Defined as primary-care visit with ≥1 diagnosis code OR medicine or laboratory test OR referral/procedure Specific medicines and Common Procedural Term (CPT) codes available upon request. |
|
|
|
|
Attention to BMI/Medical Risk Defined using evidence of:
|
|
|
|
| Specific diagnostic codes, medications, and CPT codes available upon request |
Manual progress-note/text review from the randomly selected charts determined 89.4% of 104 charts coded by the algorithm as No Attention indeed lacked text evidence of attention to BMI Alone or BMI/Medical Risk (specificity) (Table 2). Similarly, corresponding evidence was found in 96.0% of 101 charts coded as BMI Alone and 96.1% of 103 charts coded as BMI/Medical Risk.
Table 2.
Validity of algorithm vs. comprehensive chart review for detecting No Attention, attention to high BMI Alone, or attention to both high BMI/Medical Risk
| Electronic Phenotype | Chart Review Evidence |
Total | |||
|---|---|---|---|---|---|
| No Attention | BMI Alone | BMI/Medical Risk | |||
| No Attention | Frequency | 93 | 8 | 3 | 104 |
| Row Percent | 89.4% = TN | 7.7% | 2.9% | ||
| Column Percent | 97.9% = TP | 7.4% | 2.9% | ||
| BMI Alone | Frequency | 1 | 97 | 3 | 101 |
| Row Percent | 1% | 96.0% = TP | 2.9% | ||
| Column Percent | 1.05% | 89.8% = TN | 2.8% | ||
| BMI/Medical Risk | Frequency | 1 | 3 | 99 | 103 |
| Row Percent | 1.0% | 2.9% | 96.1% = TP | ||
| Column Percent | 1.05% | 2.8% | 94.2% = TN | ||
| Total | Frequency | 95 | 108 | 105 | 308 |
Abbreviations: BMI, body mass index; TN, true negative (specificity); TP, true positive (sensitivity).
Once the algorithm was finalized, the proportion of the EHR cohort (N = 7192) with algorithm-determined evidence of No Attention, attention to high BMI Alone, and attention to high BMI/Medical Risk at well-child visits was: 55%, 26%, and 19%, respectively.
Discussion
Our findings indicate that it is valid to apply an algorithm to EHR primary-care data to determine delivery of guideline-consistent weight-management clinician behaviors (attention to high BMI alone, to high BMI with medical risk, or neither) to children with overweight and obesity. Indicators built using combinations of diagnostic codes, orders, and visits have very good specificity and excellent sensitivity for capturing corresponding text-based documentation.
Application of the BMI/Medical Risk algorithm revealed that nearly half of 6-12-year-old children with overweight/obesity lack evidence at well-child visits of any clinician attention to high BMI alone or in combination with any attention to medical risk from hypertension, diabetes, fatty liver disease, or vitamin D deficiency. We now are poised to use the algorithm to test the cross-sectional associations and prospective impact of the clinician-behavior variables on BMI and comorbidity improvement over time.
The algorithm developed and described in this study advances use of EHR data from identifying diseases—such as overweight and obesity—to developing and validating electronic phenotypes denoting guideline-consistent practice behaviors delivered to those with the disease – here, primary-care attention to high BMI/overweight/obesity alone and in combination with (any) medical-risk assessment. The electronic phenotypes described here performed well in our disease cohort. To determine generalizability in other populations, their performance should be tested in separate disease cohorts, along with (at least a cursory) chart review to identify and incorporate any geographic or medical-center specific nuances in diagnosis codes or referral patterns. The critical importance of replicating these methods for children with overweight/obesity and other diseases is that, if such electronic phenotypes of guideline-consistent practice behaviors perform well, it may be possible for knowledge engineers to work with guideline-writing bodies to encode computer-interpretable guidelines at each guideline update. Such advances would aid in ensuring medical centers can maintain knowledge bases for guideline-based clinical decision-support systems – an enormous and costly task that is required for decision-support systems each time guidelines change.12
Prior studies have documented that, because EHRs automatically use measured height and weight to calculate and plot BMI and BMI percentile, EHR data can be used to identify children meeting criteria for overweight and obesity;3,4 however, although this EHR function is helpful, a plotted BMI alone is insufficient for determining whether clinicians communicate the information to patients and families. Neither, for many reasons (including lack of insurance reimbursement), does use of diagnostic billing codes for overweight/obesity alone identify whether providers communicate with patients and families about high BMI and medical risk. Our study demonstrates that additional data elements stored in the EHR (beyond BMI or diagnostic billing codes) can be used to validly understand primary-care attention to overweight/obesity and (any) medical-risk assessment. Similar data elements exist in national data networks, including PCORnet’s Common Data Model domains, and may prove useful for enhancing primary-care delivery of guideline-based weight-management care via clinical decision support systems.13
A notable strength of our study is use of common EHR data elements to derive a relatively simple rule-based algorithm.14 Rule-based logic lends transparency and ease of updating rules (for example, when guidelines change) in the context of EHR-enabled decision support. A limitation of our algorithm uncovered by manual chart extraction was that its specificity might be improved by adding automated text-based extraction using natural language-processing or machine-learning algorithms.15,16 Even these approaches, though, are improved by using iterative chart review and annotation.16 Additional algorithms are needed to detect evidence of provider delivery of each step along the care pathways for guideline-based screening, evaluation, and management of hypertension, diabetes, hyperlipidemia, and fatty liver disease. Guidelines for each of these comorbidities are complex, and overweight and obesity are the main indications (in childhood) to screen for them. Upon screening, each comorbidity has its own guideline-based decision tree. Busy primary-care providers need support to efficiently identify past actions taken (by them and other healthcare providers) and next actions needed for overweight/obesity and each associated comorbidity. Specific decision support should guide what to do after an abnormal screen and, when a screen returns normal, when to rescreen. All of this can be automated through the EHR.
Conclusion
In this study, we report the first step toward making next-generation decision support possible—development and validation of an algorithm to detect EHR evidence of clinician attention to recommended weight-management clinical practices during primary-care visits.
Acknowledgments
We would like to thank Children’s Medical Center Dallas, the flagship hospital of Children’s HealthSM; Children’s Health Pediatric Group practices; Nancy Kelly, MD and Children’s Medical Center’s Continuity-of-Care Clinic; Christopher Menzies, MD; and Taylor Gheen for their assistance obtaining the study data. For assistance with data management, indicator coding, and identifying visit-encounters for chart review, we would like to thank Brian Adamson and Joanne Sanders. For assistance with chart review, we would like to thank Marco (Tony) Fierro and Emily Marks.
Funding
Supported in part by Award # K23HL118152 from the National Heart, Lung, and Blood Institute (NHLBI; to Dr Turer). The content is solely the responsibility of the authors, and does not necessarily represent the official views of the NHLBI or NIH.
Contributors
Christy Turer designed the study, obtained funding, supervised electronic-health-record (EHR) data collection and chart review, assisted with analyses and interpretation of data, and drafted the manuscript. Both Celette Skinner and Sarah Barlow oversaw the final study design, assisted with analyses and data interpretation, and critically revised the manuscript. All authors approved the final manuscript as submitted.
Competing interests
None.
References
- 1. Blumenthal D, Tavenner M.. The “meaningful use” regulation of electronic health records. N Engl J Med 2010; 3636: 501–4. [DOI] [PubMed] [Google Scholar]
- 2. Ancker JS, Kern LM, Edwards A, et al. ; with the HITEC Investigators. Associations between healthcare quality and use of electronic health record functions in ambulatory care. J Am Med Inform Assoc 2015; 224: 864–71. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Boone-Heinonen J, Tillotson CJ, O’Malley JP, et al. Characterizing a “big data” cohort of over 200, 000 low-income U.S. infants and children for obesity research: the ADVANCE early life cohort. Matern Child Health J 2017; 213: 421–31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Bailey LC, Milov DE, Kelleher K, et al. Multi-institutional sharing of electronic health record data to assess childhood obesity. PLoS One 2013; 86: e66192.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Turer CB, Barlow SE, Montano S, Flores G.. Discrepancies in communication vs. documentation of weight-management benchmarks: analysis of recorded visits with Latino children and associated health-record documentation. Glob Pediatr Health 2017; 4: 2333794X16685190.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Brady TM, Solomon BS, Neu AM, Siberry GK, Parekh RS.. Patient-, provider-, and clinic-level predictors of unrecognized elevated blood pressure in children. Pediatrics 2010; 1256: e1286–93. [DOI] [PubMed] [Google Scholar]
- 7. Kaelber DC, Liu W, Ross M, et al. Comparative effectiveness research through collaborative electronic reporting consortium. Diagnosis and medication treatment of pediatric hypertension: a retrospective cohort study. Pediatrics 2016; 1386: e20162195.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Kuczmarski RJ, Ogden CL, Guo SS, et al. 2000 CDC growth charts for the United States: methods and development. Vital Health Stat 11 2002: 1–190. [PubMed] [Google Scholar]
- 9. Barlow SE; Expert Committee. Expert committee recommendations regarding the prevention, assessment, and treatment of child and adolescent overweight and obesity: summary report. Pediatrics 2007; 120 Suppl 4: S164–92. [DOI] [PubMed] [Google Scholar]
- 10. Dennison BA. Bright futures and NHLBI integrated pediatric cardiovascular health guidelines. Pediatr Ann 2012; 411: e31–6. [DOI] [PubMed] [Google Scholar]
- 11. O’Connor EA, Evans CV, Burda BU, Walsh ES, Eder M, Lozano P.. Screening for obesity and intervention for weight management in children and adolescents: evidence report and systematic review for the US Preventive Services Task Force. JAMA 2017; 31723: 2427–44. [DOI] [PubMed] [Google Scholar]
- 12. Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 106: 523–30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.PCORnet, the National Patient-Centered Clinical Research Network. PCORnet Common Data Model (CDM). http://www.pcornet.org/pcornet-common-data-model/. Accessed October 20, 2018.
- 14. Lingren T, Thaker V, Brady C, et al. Developing and algorithm to detect early childhood obesity in two tertiary pediatric medical centers. Appl Clin Inform 2016; 0703: 693–706. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Savova GK, Masanz JJ, Ogren PV, et al. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. J Am Med Inform Assoc 2010; 175: 507–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Chen Y, Carroll RJ, McPeek Hines ER, et al. Applying active learning to high-throughput phenotyping algorithms for electronic health records data. J Am Med Inform Assoc 2013; 20 (e2): e253–9. [DOI] [PMC free article] [PubMed] [Google Scholar]