Catch Me if You Can: Acute Events Hidden in Structured Chronic Disease Diagnosis Descriptions Show Detectable Recording Patterns in EHR

Abstract

Our previous research shows that structured cancer DX description data accuracy varied across electronic health record (EHR) segments (e.g. encounter DX, problem list, etc.). We provide initial evidence corroborating these findings in EHRs from patients with diabetes. We hypothesized that the odds of recording an “uncontrolled diabetes” DX increased after a hemoglobin A1c result above 9% and that this rate would vary across EHR segments. Our statistical models revealed that each DX indicating uncontrolled diabetes was 2.6% more likely to occur post-A1c>9% overall (adj-p=.0005) and 3.9% after controlling for EHR segment (adj-p<.0001). However, odds ratios varied across segments (1.021<OR<1.224, .0001<adj-p<.087). The number of providers (adj-p<.0001) and departments (adjp<.0001) also impacted the number of DX reporting uncontrolled diabetes. Segment heterogeneity must be accounted for when analyzing clinical data. Understanding this phenomenon will support accuracy-driven EHR data extraction to foster reliable secondary analyses of EHR data.

Introduction

The development of learning healthcare systems depends on reliable secondary use of Electronic Health Record (EHR) data.^1,2 Research endeavors including comparative effectiveness research^3,4 and precision medicine^5–7 also rely on such practices. One of the pillars of secondary analyses of clinical data is patient selection for electronic cohort development,^8,9 which often relies on diagnosis (DX) data among other EHR information.^10,11 Thus, the accurate assignment of structured DX data within the EHRs is crucial to ensure reliable research outcomes in clinical data reuse.¹² However, code-based structured DX data recording has data quality (DQ) limitations due to coding system designs and their implementation in EHR systems.^13–18 For example, some EHR implementations provide multiple textual descriptions per DX code to facilitate search and selection, which could be leveraged as an additional EHR data source.^19,20 However, a large number of DX descriptions may complicate the selection of the most accurate version containing all features describing the patient’s disease.

Existing informatics research has focused on providing evidence of DX data unreliability and developing methods to reduce reliance on structured DX data alone.^21–23 Yet, there is much less published work devoted to understanding the root causes of poor DQ, the interplay between clinical processes producing the resulting data and how to reliably use imperfect data reliably. On one hand, DX inaccuracy rates have been studied for decades^12,24,25 and error rates have improved over time (e.g. ICD code inaccuracy rates from 20-70% in the 1970s to 20% in 1980s),²⁴ but DX data reliability remains questioned.^3,26 The field offers little help to remedy this beyond augmentation with alternative EHR data to ensure reliability.^22,23 On the other hand, the field has privileged overcoming DX data limitations using natural language processing for DX extraction from clinical notes and EHR phenotyping.^23,27–31 Though these methods improve the precision and recall in electronic cohort selection,8 they may also introduce additional uncertainty to secondary analyses.^21,28 Additionally, they may fail to take advantage of knowledge available in the clinical setting where the data is generated.^32–34 Thus, the state of the art provides limited resources to leverage this rich data source.

Though DQ limitations are a healthcare-wide issue, research to understand how clinical workflows and intra-EHR data sources (i.e., EHR segment of data entry such as problem list or encounter DX) affect EHR data entry processes and DQ in light of secondary use has been explored in oncological DX data almost exclusively.^{20,26,35–37} This is likely due to two methodological advantages: (1) A chart containing a biopsy report, which can be considered a partial gold standard and (2) The chronic and stable nature of a cancer DX, which will most likely remain on the patient’s chart. However, this setup is unlikely to inform secondary users of clinical data on reliable recording of acute phenomena not required for billing purposes, which would advance EHR-based chronic disease management research in the context of learning health systems.^6,33,38 For example, identifying periods of uncontrolled glucose levels in diabetes patients from DX data, whether a lab value is available or not in the EHR, would open a new exploitable source of data for such research.

To explore this possibility, we examined DX data for patients treated for diabetes to assess whether acute phenomena such as uncontrolled diabetes were recorded in structured EHR data at rates that would be exploitable using statistical methods. We aimed to uncover differences in DX data reporting rates before and after evidence of uncontrolled diabetes was available in the EHR. In concordance with published findings,^20,37 we hypothesized that recording new lab information about acute complications of patient treated for chronic disease increases the odds of recording structured DX data indicating this complication, but that the odds vary across EHR data sources (e.g., encounter DX, problem list, order DX, etc.). To test this hypothesis, we compiled patient DX sequences (i.e., a chain of chronologically-ordered DX records in a patient’s EHR) containing all diabetes DX in a period of 90 days before or after an hemoglobin A1c (HbA1c) laboratory result higher than 9% was reported, in accord with current diabetes care guidelines.³⁹ Each DX sequence contained specific ICD-10 codes for Type 1 or 2 diabetes (i.e., ICD-10 DX code, E10.* and E11.*) that might mention lack of control (e.g. “uncontrolled diabetes”, “diabetes with hyperglycemia”). We built statistical models predicting the odds of these sequences occurring before or after an elevated HbA1c based on the number of DX records reporting uncontrolled diabetes. We selected diabetes-related DX data not only due to its public health significance⁴⁰ but also due to its chronic and variable nature over time (e.g., glucose level flow, controlled vs. uncontrolled levels, complex comorbidities). We present descriptive and summary statistics for our EHR-phenotyped cohort^8,41 of patients followed by our statistical modelling results for hypothesis testing. This analysis provides a new understanding of how acute occurrences recorded within DX descriptions can be identified and how the use of numerous DX descriptors impacts logging practices. It also provides evidence of the generalizability of prior findings uncovered in oncology EHR data^20,36,37 and demonstrates that this effect is large enough to be detectable without the need for hand-curated patient cohorts.²⁰ These findings underscore the need for closer attention to intra-EHR data sources for secondary use of clinical data to support analysis reliability in the context of learning health systems.

Methods

We extracted structured DX data and relevant covariates across five intra-EHR data sources corresponding to five EHR segments (i.e., primary encounter DX, encounter DX, billing DX and order DX and problem list DX) from Wake Forest Baptist Medical Center’s Translational Data Warehouse. We extracted data for patients treated for diabetes including HbA1c values and DX description selected by clinicians during charting. We used these descriptions to identify diabetes DX descriptions reporting uncontrolled diabetes using both string matching and natural language processing methods. We tested our hypothesis by fitting logistic regression models to predict the odds of patient-based 90-day DX sequences to be recorded after an HbA1c value above 9% was entered into the EHR. We selected an odds ratio to reflect the presence to absence ratio of uncontrolled diabetes DX data recorded after an elevated HbA1c. We first built and validated an overall regression for all our EHR data. Then, we re-built the overall model including data EHR segment as a predictor variable to detect statistically-significant differences across intra-EHR data sources. Finally, we rebuilt the model for each EHR segment separately to detect potential qualitative differences across EHR segments. This study was approved by Wake Forest University School of Medicine’s Institutional Review Board (IRB#: IRB00062976).

We extracted DX data for all patients with at least one HbA1c above 9%, indicating uncontrolled diabetes status valid for the following 90 days. Our data included date of first HbA1c result above 9%.³⁹ To be included, patients had to be adults (i.e., 18 years of age when first data were recorded) and have at least one HbA1c value greater than 9%. We selected this last inclusion criteria based on a phenotyping algorithm feature developed by the New York City Health Department.⁴² This phenotype has a reported positive predictive value of 0.97 at the 6.7% level for both types of diabetes after validation with a gold standard⁴¹. Patients with only one clinical encounter within 90 days before or after their first high HbA1c were excluded. This measure screened for charts that did not contain enough DX data after the HbA1c results were returned, reducing potential noise that would dampen statistical effects in our raw data.

Our dataset contained 4,160 distinct diabetes DX descriptions that could be selected by clinicians at the time of entering a DX into the EHR. We processed descriptions using a multi-stage approach to identify descriptions of uncontrolled diabetes. First, we used string matching techniques to identify the terms “uncontrolled”, “inadequately controlled”, “out of control”, “poorly controlled” or “hyperglycemia”, or “not at goal” according to past and current DX coding guidelines.⁴³ Second, we used natural language processing (i.e., semantic concept extraction) to identify other concepts revealing uncontrolled diabetes. We extracted all medical concepts mentioned in these descriptions using NOBLE Coder,⁴⁴ an NLP named entity recognition tool for biomedical text. This tool is based on the NCI thesaurus terminology,⁴⁵ which contains concepts spanning multiple healthcare domains (i.e., clinical care, translational and basic research). We then reviewed the resulting concepts including the same terms cited for string matching. Third, we compared concepts found via direct string matching and concept extraction finding similar number of DX descriptions returned (755 and 715 for string matching and concept extraction, respectively). The percent agreement between the two methods was 99.3%. We also found that all concepts identified via concept extraction were contained in the set identified by string matching. We calculated two-rater Cohen’s Kappa⁴⁶ to verify inter-rater reliability across methods using the irr R package.⁴⁷ We found high agreement between both approaches (Kappa=0.977, p<.0001). Finally, a clinician reviewed 10% of all DX descriptions to estimate classification accuracy. We found that percent agreement between our best-performing method and our expert was 98.1% and the inter-rater reliability Kappa=0.931 (p<.0001), revealing the high accuracy of our DX feature extraction process.

Our final analytical dataset extracted from our EHR contained 158,660 diabetes DX recordings for 11,179 patients, recorded between November 1^st 2015 and January 1^st, 2020. This time frame was defined to ensure ICD coding version consistency (i.e., include DXs after ICD-10 implementation). Our initial dataset consisted of DX records with corresponding timestamp and patient identifier. Each DX had a specific DX description and was associated with an ICD-10 code (i.e., E10.* and E11.* codes corresponding to type 1 and type 2 diabetes). Each patients’ initial HbA1c result higher than 9% recorded date was added to each DX record with the ‘PostHbA1c’ indicator that served at the dichotomous independent variable for our regressions. Additional variables were calculated to differentiate sequences of DX entered 90 days before and after the first high HbA1c value. These variables included the number of DX entered in each sequence, the number of unique DX descriptions entered, the DX sequence length in days (i.e., maximum number of days between DX entry and the HbA1c), the number of distinct visit providers, the number of departments or clinical units treating the patient (i.e., care units involved in patient treatment such as internal medicine, family medicine and endocrinology) and the number of DX referring to uncontrolled diabetes. Summary statistics such as mean, median and extreme values were employed to screen the data for outliers, missing values and erroneous input. Dates were also reviewed for potential errors such as values outside the study’s time window. We verified the normality of continuous variables using histograms.

To test our hypothesis, we built binomial regressions⁴⁸ to predict the number of accurate and inaccurate DX across patient charts using R’s generalized linear model (GLM).⁴⁹ We elected to build binomial regressions to accommodate for our dichotomous outcome variable, ‘PostHbA1c’. This type of regression allowed us to keep a consistent model type throughout the analysis as a means to improve cross-model comparison by avoiding overdispersion⁴⁸ and zero inflation⁵⁰ issues presented by the counts of uncontrolled diabetes DX in our dataset. Our models predicted the odds of a patient-based 90-day DX sequences of appearing before or after a 9%+ HbA1c lab value was recorded in the EHR on all calculated variables available in our dataset. These variables were the number of DX referring to uncontrolled diabetes, number of DX entered, number of unique DX descriptions entered, DX sequence length (i.e., maximum number of days since the HbA1c result), number of distinct visit providers and number of departments or clinical units treating the patient. We used a stepwise backward elimination model building strategy⁵¹ exploring models including interactions up to the second degree (i.e., 2 variable interactions). This approach was used to maximize model fit, ensure the inclusion of all relevant covariates and also validate the predictive power of the number of DX reporting uncontrolled diabetes on the pre-post HbA1c dichotomous variable. In cases where the number of uncontrolled DX was not included in the final model, we rebuild the regression to assess the presence of an independent effect; such effect would be considered lesser due to poorer model fit. We maximized goodness of fit using Akaike’s information criterion⁵², using the stepAIC function from R’s MASS package.⁵³ Statistical significance was set at p=0.05 for all models and adjustments for multiple comparison were made using R’s p.adjust function⁵⁴ using Holm’s correction method⁵⁵. We refer to the odds ratios as OR and adjusted p values as adj-p.

Multiple software tools were used to carry out this analysis. Data extraction and preprocessing was executed using a DataGrip software client (version 2019.1, JetBrains s.r.o., Prague, Czech Republic). Visual exploration and analyses were performed using Tableau (version 2019.3.2, Tableau Software, Inc., Seattle, WA). All statistical analyses and data manipulation such as data scrubbing and reshaping were done in R version 3.6.230 and RStudio (version 1.2.5033, RStudio, Inc., Boston, MA).

Results

Our dataset contained 158,660 diabetes DX recordings for 11,179 patients across five clinical EHR segments (Table 1). The dataset contained 550 distinct DX descriptions entered by 2,636 providers across 646 departments or clinical units. The average number of days from HbA1c result was 25.9±29.8 (Mean±Std.Dev.). Pre-HbA1c data contained 43,805 diabetes DX recordings for 6,448 with 427 distinct DX descriptions entered by 1,811 providers across 492 departments or units. The average number of days from the pre-HbA1c result was 30.9±30.4. Post-HbA1c data contained 114,855 diabetes DX recordings for 10,398 with 505 distinct DX descriptions entered by 2,358 providers across 612 departments or units. The average number of days from the post-HbA1c result was 24.1±29.3. The percentage of DX reporting uncontrolled diabetes was 24.9% overall, 23.8% pre-HbA1c and 25.2% post-HbA1c. Within this overall dataset, the data contained 15,725 (28.4% uncontrolled diabetes) primary encounter DX for 6,917 patients, 34,097 (25.7% uncontrolled diabetes) non-primary encounter DX for 9,690, 8,849 (23.2% uncontrolled diabetes) problem list DX for 2,810 patients, 68,633 (25.2% uncontrolled diabetes) DX attached to procedure orders for 10,052 and 31,356 (21.6% uncontrolled diabetes) billing DX for 6,968 within the same time frame. Distinct DX description numbers showed similar levels for both encounter DX and order DX but showed much lower numbers for problem list DX with 155 and billing DX with 138 distinct descriptions, compared to close to 400 for other EHR segments. The number of providers was similar to the overall numbers except for problem list DX (617) and order DX (78) compared to over one thousand in most cases. Department/unit numbers showed comparable numbers for all segments. Number of days was also stable across segments except for problem list and order DX (23.4±27.8 and 20.9±29.5).

Table 1.

Dataset Descriptive Statistics by EHR Segment.

	All Segments		Primary Encounter DX	Encounter DX	Problem List DX	Order DX	Billing DX
Measures\Timeframe	Overall	Pre-HbA1c	Post-HbA1c	Overall	Overall	Overall	Overall	Overall
Distinct Patients	11,179	6,448	10,398	6,917	9,690	2,810	8,630	6,968
Number of DX Records	158,660	43,805	114,855	15,725	34,097	8,849	68,633	31,356
Distinct DX Descriptions	550	427	505	356	435	155	402	138
Number of Uncontrolled Diabetes DX Records	39,435	10,433	29,002	4,469	8,772	2,054	17,354	6,786
Distinct Providers	2,636	1,811	2,358	1,357	1,815	617	78	1,644
Distinct Hospital Department/Unit	646	492	612	315	508	200	441	394
Days from HbA1c Result (Mean±Std.Dev.)	25.9±29.8	30.9±30.4	24.1±29.3	30.5±29.1	28.7±29.8	23.4±27.8	20.9±29.5	32.5±29.2

We found a statistically-significant relationship between post-HbA1c DX recording and the number of DX reporting uncontrolled diabetes in the overall data (Table 2). Our model revealed that a 90-day DX sequence was 2.6% more likely to appear in the post-HbA1c time frame for each uncontrolled diabetes DX included (adj-p=.0005). Our model controlled for the total number of DX entries (adj-p<.0001), the number of distinct DX descriptions per patient (adjp<.0001), the number of providers (adj-p<.0001), the number of department/units (adj-p=.0052) and DX sequence length (i.e., max days from HbA1c, pre/post) (adj-p<.0001). The number of providers had the largest odds ratio (OR=2.487, adj-p<.0001), reflecting the likelihood of multi-provider follow-up after a high HbA1c value. The total number of DX in each sequence had the smallest difference from a no-effect ratio (OR=0.981, adj-p<.0001), revealing the large number of DX per patient before and after a high HbA1c result likely due to their chronic disease. Our stepwise model building approach did not include significant variable interactions.

Table 2.

Overall Regression Results. Uncontrolled diabetes DX included in DX sequence makes it 2.6% more likely be recorded post-HbA1c (adj-p=.0005).

Term	Odds Ratio (exp(ß))	Estimate (ß)	Confidence Interval (95%)		Std. Error	p-value	Adjusted
Number of Uncontrolled	1.026	0.025	0.012	0.039	0.007	0.0002	0.0005
Diabetes DX Number of DX	0.981	-0.020	-0.027	-0.013	0.004	<.0001	<.0001
Number of Distinct DX Number of Providers	1.115	0.109	0.080	0.139	0.015	<.0001	<.0001
Number of	2.487	0.911	0.849	0.975	0.032	<.0001	<.0001
Departments/Units	1.130	0.123	0.037	0.209	0.044	0.0052	0.0052
DX Sequence Length (Days)	0.938	-0.064	-0.066	-0.062	0.001	<.0001	<.0001

We were able to find a similar, yet stronger statistically-significant effect of the number of DX reporting uncontrolled diabetes on post-HbA1c DX recording after controlling for data EHR segment (Table 3); this model also uncovered statistically significant differences across EHR data entry segments. A DX sequence was 3.9% more likely to appear in the post-HbA1c time frame per uncontrolled diabetes DX recorded (adj-p<.0001). Our model controlled for the number of distinct DX descriptions per patient (adj-p<.0001), the number of providers (adj-p<.0001), the number of departments/units (adj-p<.0001) and DX sequence length in days (adj-p<.0001). The number of departments/units had the strongest predictive effect (OR=2.575, adj-p<.0001), reflecting the likelihood of multi-department follow-ups. DX sequence length in days had the smallest difference from a no-effect odds ratio of 1 (OR=0.968, adj-p<.0001). This model also revealed statistically significant differences across EHR segments including encounter DX (OR=.61, adj-p<.0001), problem list DX (OR=1.288, adj-p<.0001), order DX (OR=.658, adj-p<.0001) and billing DX (OR=.795, adj-p<.0001).These terms show differences in diabetes DX sequences across EHR data entry segments, including a bias in favor of post-HbA1c DX entries in the problem list compared to primary encounter DX data. Our stepwise model building approach did not include any significant variable interaction terms. Further exploration of this EHR segment effect, we found that modeling the number of DX reporting uncontrolled diabetes and EHR segment with an interaction term returned high levels of statistical significance, though it presented a slightly poorer fit (AIC=114834 vs. AIC=129333 for stepwise model and interaction model, respectively). Each DX reporting uncontrolled diabetes doubled the chances of post-HbA1c recording (OR=2.045, adj-p<.0001). There were significant differences between the primary encounter DX sequences and encounter DX sequences (OR=.933, adj-p=.0058) as well as problem list DX sequences (OR=1.128, adj-p<.0001). Interactions between these variables were also statistically significant (.596<OR<.842, all adj-p<.0001). This model revealed that there is heterogeneity in the entry of diabetes DX and that the entry of uncontrolled diabetes DX is EHR segment-dependent. Specifically, primary encounter DX appears to have the largest difference in uncontrolled diabetes DX post-HbA1c, compared to pre-HbA1c; this difference seems weaker with order and billing DX.

Table 3.

EHR Segment-Controlled Regression Results. Uncontrolled diabetes DX included in DX sequence makes it 3.9% more likely be recorded post-HbA1c (adj-p<.0001), after controlling for EHR segment. Compared to primary encounter DX, other EHR segments had different odds ratio estimates (all adj-p<.0001).

Term	Odds Ratio (exp(ß))	Estimate (ß)	Confidence Interval (95%)		Std. Error	p-value	Adjusted p-value
Number of Uncontrolled Diabetes DX	1.039	0.039	0.024	0.053	0.008	<.0001	<.0001
Number Distinct of DX	1.086	0.083	0.054	0.112	0.015	<.0001	<.0001
Number of Providers	1.244	0.218	0.168	0.268	0.025	<.0001	<.0001
Number of Departments/Units	2.575	0.946	0.897	0.996	0.025	<.0001	<.0001
DX Sequence Length (Days)	0.968	-0.032	-0.033	-0.031	0.000	<.0001	<.0001
EHR Segment - Primary Encounter DX (Ref.)	1	0	-	-	-	-	-
Encounter DX	0.610	-0.494	-0.539	-0.448	0.023	<.0001	<.0001
Problem List DX	1.288	0.253	0.211	0.296	0.022	<.0001	<.0001
Order DX	0.658	-0.418	-0.465	-0.372	0.024	<.0001	<.0001
Billing DX	0.795	-0.230	-0.274	-0.186	0.022	<.0001	<.0001

We found further evidence for these differences building individual regression models for each EHR segment (Table 4). Specifically, the influence of the number of uncontrolled diabetes DX presented different odds ratios across data from different EHR data entry segments. In concordance with our interaction model findings, primary encounter DX revealed the largest odds ratio (OR=1.139, adj-p=.0065) even when controlling for covariates pre-selected using our stepwise selection approach. Encounter DX had the second largest odds ratio under the same conditions (OR=1.091, adj-p=.0073). The other EHR segments presented weaker effects or no effect at all. The number of uncontrolled DX was not included in our stepwise models for problem list or order DX. Number of uncontrolled diabetes DX seemed to have a weak statistically-significant relationship with post-HbA1c DX sequences for problem list DX (adj-p=0.087) but a strong one for billing DX (OR=1.224, adj-p<.0001). Order DX was included in stepwise model, showing a relevant effect but no statistical significance (adj-p=.130). Re-building the model with no covariates revealed a statistically-significant effect of moderate size for order DX (OR=1.064, adj-p<.0001).

Table 4.

Odds Ratios of Post-HbA1c DX Sequence Appearance Per Number Diabetes DX Reporting Uncontrolled Diabetes Across EHR Segments. The number of uncontrolled diabetes DX presented different odds across data from different EHR segments (1.021<OR<1.224, .0001<adj-p<.087); primary encounter DX revealed the highest odds ratio (OR=1.139, adj-p=.0065).

Regression	In Stepwise Model	Odds Ratio (exp(ß))	Estimate (ß)	Confidence Interval (95%)		Std. Error	p-value	Adjusted p-value
Primary Encounter DX	Yes	1.139	0.131	0.044	0.218	0.044	0.0033	0.0065
Encounter DX	Yes	1.091	0.087	0.032	0.145	0.029	0.0024	0.0073
Problem List DX	No	1.186	0.171	0.015	0.350	0.085	0.044	0.087
Order DX	Yes	1.021	0.021	-0.003	0.045	0.012	0.0877	0.1305
Billing DX	No	1.224	0.202	0.154	0.250	0.025	<.0001	<.0001

Discussion

We used statistical modelling to evaluate whether an HbA1c value above 9% had an effect on the number of diabetes DX recordings reporting uncontrolled diabetes. Our regressions revealed that an increase in the number of DX reporting uncontrolled diabetes after a high HbA1c value is detectable in EHR data. We also found that this increase varied across EHR segments (i.e., intra-EHR data sources). Controlling for these EHR segments, we found that intra- EHR data sources impacted the odds of DX sequences being recorded pre or post high HbA1c result. We also found the number of DX reporting uncontrolled diabetes was linked to the source EHR segment via a variable interaction. This revealed that the recording of uncontrolled diabetes DX may depend on the data entry context, EHR segment where DX are recoded and the clinical workflow. All point estimates consistently predicted an increase in the number of uncontrolled diabetes DX after the HbA1c value despite the heterogeneity across EHR segments corresponding distinct intra-EHR data sources (Figure 1), despite uncontrolled diabetes being recorded only about 25% of the time overall. These results reveal that DX descriptions recorded in EHRs may contain additional disease process information that may be exploitable and require minimal supplemental informatics work.

Figure 1.

Odds Ratios Across EHR Segments and Overall. Odds ratio estimates consistently predict uncontrolled diabetes DX increases after high HbA1c values despite the heterogeneity across EHR segments and corresponding intra-EHR data sources.

Our study explores aspects beyond usual DX code assignment accuracy research^1,17,56 and contributes to the existing body of knowledge on DX DQ for secondary use in EHR systems in three ways. First, this study confers generalizability of prior methods and results first demonstrated in oncological EHR studies to other disease processes. Specifically, prior work has shown that though the number of DX representing a disease process increase after a biopsy report is charted, the effect varies across EHR data sources.^20,35–37 Our analysis shows that these effects are present in other disease processes beyond cancer EHRs and that they may also be valid for disease sub-processes (i.e., controlled vs. uncontrolled glucose in diabetes management DX entries). These findings open new research avenues that may lead to the reliably leveraging DX descriptions that contain information generated during patient care when additional resources for confirmation are available for accurate charting.^36,57 Our analysis also provides further evidence of DX incompleteness and lack of concordance reported in prior studies.^26,35,36 Second, our results provide further evidence of the impact of EHR segment differences, workflow differences and intra-EHR data source on clinical DQ.^58–61 According to current diabetes management guidelines, any patient with an HbA1c value higher than 9% should be diagnosed and treated for uncontrolled diabetes.³⁹ Though billing guidelines do not require documentation of uncontrolled diabetes⁴³, a fully-accurate and complete DX description responding to a high standard of DQ should, in principle, contain this information.^36,57 Our study shows that although there is an increase in the number of uncontrolled diabetes DX after an elevated HbA1c, the proportion varies depending on the EHR segment in which the DX is recorded. It is possible that clinicians may choose DX codes to facilitate downstream processing rather than documenting care with the highest degree of precision.⁶² This is concordant with prior findings.^36,57 Finally, this study provides evidence that should encourage the development of methods for the exploitation of DX descriptions. Extraction of this additional disease process information supports electronic cohort development for clinical data reuse.^8,9,11 Our findings show that secondary disease information is contained in textual DX descriptions and is recorded at statistically-detectable levels. These descriptions may provide a supplementary source of information to identify disease processes and conditions in patient charts beyond structured data, which can improve EHR phenotyping. Such technique may also be used for the detection of time-dependent disease sub-processes, which are often difficult to detect in EHR data.^3,63

Variable data entry across EHR segments hints at a major shortcoming in data logging systems.^36,57 Specifically, EHR users enter data differently for different EHR segments. Though this may be driven by clinical need, it greatly hinders secondary use of clinical data unless there is specific and granular understanding of these differences. Data entry variability may signify that though clinical data entry systems may have the capability of recording DX descriptions that go well beyond the capabilities of billing codes (e.g., ICD-10), there seems to be no support or encouragement for systematically recording more precise and accurate structured DX descriptions in EHRs. This is in line with prior findings.^36,37,57 In clinical practice, this issue may be tightly linked to challenges in EHR interface design and usability.^64–66 We believe that future EHR improvements to support DX data entry could potentially address these problems. Entering the most precise and comprehensive DX for each patient in every data entry context may not support clinical care needs and constraints^62,67 and may be impossible to achieve for all users. However, our data shows that some users already appear to be logging precise DX. This high-quality logging seems to be stable according to our findings and may be exploitable. Further research is needed to confirm this but having such a source of data would unlock a new way of extracting additional DX information from EHR data.

In the context of learning health systems, this study offers preliminary evidence to support the potential viability of a new source of DX information from EHRs extending beyond primary billable DX. Although ICD DX codes have been deemed unreliable for secondary analysis for decades,^13–18 textual DX descriptions may provide additional information that can be easily extracted with minimal informatics work. This may support quick data reuse for clinical decision support and population health management within learning health systems,^6,32 A concrete example would be semi-automated cohort development for preliminary automated association detection from clinical data using textual DX entries in structured EHR data. Such processes would surely accelerate data reuse and evidence generation by leveraging operational data within learning systems³⁸ without the need to deploy complex methods requiring months of overhead work. One may argue that the most accurate DX information is always within clinical notes and thus leveraging structured DX data is unnecessary. However, extracting information from clinical notes reliably for a large number of patients remains a challenge.^27,68 Phenotyping,^8,22,31 the use of complex algorithms²² and other technological solutions,⁶⁹ are other ways of bypassing the use of structured DX data but these technologies often introduce a degree of uncertainty in analyses and are still in development. It is possible that augmenting these approaches with the use of structured textual DX information may enable improvement by focusing the textual input of natural language processing methods to extract specific information.

Our analysis presents four limitations mostly related to its preliminary nature. First, our study relied on an EHR-phenotyped patient cohort rather than a hand-curated cohort of patients. Previous studies in this line of research have relied on clinician-developed cohorts, which are expensive and time consuming to develop. In our case, such cohort was not available, so we used EHR phenotyping techniques to select a patient population from our EHR database. We used a phenotyping algorithm feature developed by the New York City Health Department⁴² that has been reported to have a positive predictive value of 0.97 after validation with a gold standard.⁴¹ Our screening conditions were more stringent than those used for evaluation (i.e., HbA1c of 9% rather than 6.7% used for validation), so our positive predictive value may be closer 1. We relied on this phenotyping approach aimed to provide preliminary evidence and pave the way toward future work that will leverage hand-curated cohorts along with DX gold standard information. Second, the odds ratios were relatively small (i.e., 1.026<OR<1.224) compared to other predictors. However, our models consistently predicted increases in DX reporting uncontrolled diabetes after a high HbA1c value for all point estimates. To ensure result transparency and reduce potential bias/censoring, we made minimal assumptions that were justified by reliable clinical knowledge and data needs. We made three assumptions: (1) including adult patients with HbA1c>9% (required by our research question), (2) patients had to have 2 or more encounters (to ensure data enough data available for analysis) and (3) application of a time window of 90 days around lab results (driven by HbA1c test specificities). Such assumption-light approach is most likely to lead to the detection of a true effect in EHR data.⁶³ It is likely that the weak, yet consistent effect described in this paper may be due to the heterogeneity in our data (i.e., data from multiple EHR segments, workflows, clinics, departments and providers) and in our patients due to our phenotyped patient cohort.⁸ Our study has teased out the heterogeneity in EHR data and we will address patient heterogeneity in future work by developing and leveraging hand-curated diabetes patient cohorts. Three, we employed clinical data from a single healthcare system. However, this is our first attempt to uncover such an effect. Multi-site data will be used for generalizability and external validity analyses in future work. Finally, we only evaluated the effect of one abnormal lab value on DX recording of acute disease sub-processes for a single chronic disease. Given the preliminary nature of this analysis and the lack of other literature covering this topic, we compiled this initial series of statistical models providing evidence of the existence of this potentially-exploitable phenomenon and its variability across EHR segments and corresponding intra-EHR data sources. Generalizability of these initial findings will be tested among other chronic diseases in future work.

Future work will be divided into two segments: confirmatory analyses to verify the robustness of our conclusions and the evaluation of methodologies to leverage this new source of DX information reliably. On one hand, we will reproduce this analysis for a hand-curated cohort of diabetes patients to confirm the validity of our conclusions. We will also verify the external validity of our findings by reproducing our analysis for a cohort of patients diagnosed with a different chronic disease and presenting an acute sub-process (e.g., uncontrolled hypertension). On the other hand, we will employ informatics methods such as natural language processing and fuzzy string matching to extract other information from these DX descriptions and conduct chart reviews to verify the validity and accuracy of extracted information. Validation of these methods may lead to the development of new source of DX data in EHRs for clinical research and chronic disease management within learning health systems.

Conclusion

We consistently found increased odds of uncontrolled diabetes DX counts after a 9%+ HbA1c value, despite the heterogeneity across EHR segments and despite uncontrolled diabetes being recorded roughly one out of four times, overall. This consistent effect provides strong preliminary evidence that recording new lab information in EHRs of patients treated for chronic diseases increases the odds of recording structured DX data indicating such a complication at statistically-detectable levels. Our findings provide preliminary evidence that there is additional information embedded in DX description data within EHRs that may be exploitable for patient identification based on non-billable acute events. This analysis also provides evidence of the EHR segment-dependent nature of clinical data recording beyond oncology EHR data. Even though ICD codes have been deemed unreliable sources of DX information for decades,^13–18 textual DX descriptions may be an alternative source of information for agile cohort development and the development of clinical decision support interventions that reliably leverage real-time clinical data in the context of learning systems.