Abstract
Prior studies have demonstrated that misclassification of study variables due to electronic health record (EHR)-discontinuity can be mitigated by restricting EHR-based analyses to subjects with high predicted EHR-continuity based on a simple algorithm. In this study, we compared EHR continuity in populations covered by Medicare, Medicaid, or commercial insurance.
Using claims linked EHRs from a multi-center network in Massachusetts, including Medicare (MA EHR-Medicare cohort) and Medicaid (MA EHR-Medicaid cohort) claims data; and TriNetX (TriNetX cohort) claims linked EHR data from 11 US based healthcare organizations, we assessed (1) EHR-continuity quantified by proportion of encounters captured by EHR (capture proportion, CP); (2) area under receiver operating curve (AUROC) of previously validated model to identify patients with high EHR-continuity (CP>0.6); (3) misclassification of 40 patient characteristics, quantified by average standardized absolute mean difference (ASAMD). Study participants were ≥65 years (Medicare) or ≥18 years (Medicaid, TriNetX) with ≥365 days of continuous insurance enrollment overlapping with an EHR encounter. We found that the mean CP was 0.30, 0.18 and 0.19 and AUROC of the prediction model to identify patients with high EHR-continuity was 0.92, 0.89 and 0.77 in the MA EHR-Medicare, MA EHR-Medicaid and TriNetX cohorts, respectively. Restricting to patients with predicted EHR-continuity percentile of top 20%, 50%, and 50% in MA EHR-Medicare, MA EHR-Medicaid, and TriNetX cohorts resulted in acceptable levels of misclassification (ASAMD <0.1). Using a prediction model to identify cohorts with high EHR-continuity can improve validity, but cut-offs to achieve this goal vary by populations.
Keywords: Electronic medical records, comparative effectiveness research, information bias, data continuity
Introduction
Electronic health record (EHR) databases are increasingly being used as a primary data source for comparative effectiveness research (CER) studies.1, 2 However, the main drawback of using this data source is the misclassification of exposures, outcomes and covariates 3, 4 due to a lack of EHR-continuity or, EHR-discontinuity, which has been previously defined as “having medical information recorded outside of the study EHR system”.4 Such information, recorded outside a study network’s EHR, is “invisible” or unavailable to researchers and can lead to misclassification of key variables and bias study results. Our prior work has demonstrated that such misclassification can be mitigated by restricting EHR-based analyses to subjects with high predicted EHR-continuity on the basis of a simple algorithm.4,5 This algorithm was further expanded to include demographic variables, with strongest overall predictors of high EHR-continuity being having BMI recorded in the EHR, having 5 or more physician office visits, and having 10 or greater distinct drugs recorded.6
However, this algorithm was developed and validated using data from Medicare beneficiaries aged 65 years and older, from one EHR system in MA and its generalizability to other EHR systems in different regions is unknown. Additionally, the optimal threshold of predicted EHR-continuity to define a cohort of subjects with high EHR-continuity to achieve an acceptable degree of misclassification has not been established outside the original study cohort.
In this study, we aimed to 1) measure and compare observed EHR-continuity, quantified by Capture proportion (CP), using EHR and claims data from 3 different EHR cohorts linked with Medicare, Medicaid and mixed commercial insurance claims databases (MA EHR-Medicare cohort, MA EHR-Medicaid cohort and TriNetX [TNX] cohort respectively); 2) measure predicted EHR-continuity using the previously developed EHR-continuity prediction algorithm and compare the performance of the algorithm in these three cohorts using data from EHR only; 3) determine the threshold of predicted EHR-continuity to distinguish between subjects with high vs low EHR-continuity in the three cohorts; and 4) compare misclassification of 40 selected variables within levels of predicted EHR-continuity in the three cohorts using data from EHR only vs. data from claims and EHR.
Methods
Data source
For this study, we created three different cohorts. The first cohort contained data from a Massachusetts based EHR system linked to Medicare claims data (MA EHR-Medicare cohort). The second cohort contained data from the same Massachusetts based EHR system but linked to Medicaid claims data (MA EHR-Medicaid cohort). The third cohort used data from the TriNetX Linked Claims Network (TriNetX cohort).
For the MA EHR-Medicare cohort, we used EHR data from a care delivery network that consists of 2 tertiary hospitals, 3 community hospitals, and >35 primary care centers, spanning from 1/1/2000 to 12/31/2017.6 While the tertiary centers are based in an urban center in Massachusetts, the community hospitals and primary care centers are in suburban centers. The EHR data were linked longitudinally with Medicare fee-for-service (FFS) administrative claims data and the linkage spanned from 1/1/2007 to 12/31/2017. Medicare FFS is a federally funded insurance program that covers legal residents in the U.S. aged 65 years and older and individuals with certain disabilities.7 For the MA EHR-Medicaid cohort, we used EHR data from the same network as the MA EHR-Medicare cohort, but the EHR data were linked longitudinally with Medicaid administrative claims data and the linkage spanned from 1/1/2007 to 12/31/2014. Medicaid is a joint federal and state program designed for low to moderate income legal residents in the U.S without a specific age requirement.8 The data for the TriNetX cohort were drawn from the TriNetX Linked Claims Network from approximately 3,309,640 patients from 11 healthcare organizations across the United States that include both rural and urban areas and contains information from insurance claims linked EHR (approximately 67.1% covered by a commercial insurance, and 31.8% by Medicaid), spanning from 01/01/2008 to 01/19/2022.
Both the Massachusetts and TriNetX networks include medical facilities across the full spectrum of care continuum. Each EHR database contains information on patient demographics, medical diagnoses, procedures, medications, laboratory results and vital signs. The claims data contain information on demographics, enrollment start and end dates, dispensed medications and performed procedures, and medical diagnoses.9 There was no overlap between the MA EHR cohorts and the TriNetX cohort.
Study population
The study subjects were required to be at least 65 years of age at cohort entry for the MA EHR-Medicare cohort and at least 18 years of age at cohort entry for the MA EHR-Medicaid and TriNetX cohorts; have at least 365 days of continuous enrollment in the respective insurance plans prior to cohort entry; and have at least one encounter recorded in the EHR system during the 365 days of continuous enrollment. We excluded patients who had differently recorded sex and age in EHR and claims data. The index or cohort entry date was the first date on which all inclusion criteria were met. Follow-up started on the first day after the index date and lasted until the first of the following events: (1) end of insurance enrollment; (2) death; or (3) end of claims data availability. The data were segmented into 365-day intervals following cohort entry and were truncated at the beginning of the interval at which censoring occurred (Table S1, Figure S1).
Measurement of observed EHR-continuity using EHR-claims linked data
Observed EHR-continuity, using claims and EHR data, was defined as the Capture Proportion (CP) in the three study cohorts and computed using the formula:
We did not distinguish between outpatient and inpatient encounters in the TriNetX dataset, because >60% of encounters in this dataset were missing information on type of encounter. We considered this is an acceptable limitation because the sensitivity analyses in our prior study demonstrated that the relative weights applied to outpatient vs. inpatient were not influential on our findings.4 We also conducted additional analysis for capture proportion in the MA EHR-Medicare and MA EHR-Medicaid cohorts without distinguishing between outpatient and inpatient encounters to evaluate any differences.
Calculation of predicted EHR-continuity
Predicted EHR-continuity was computed based on the coefficients of the EHR-continuity prediction algorithm with demographics6 using only data available in the EHR (Table S2). Lower predicted EHR-continuity corresponds to greater EHR incompleteness.
Validation of the expanded EHR-continuity prediction model using EHR data alone
Informed by literature, observed EHR-continuity or CP (calculated above) ≥0.6 was defined as high EHR-continuity.10 Performance of the EHR-continuity prediction algorithm in identifying patients with high EHR-continuity was determined by area under ROC curve (AUROC) in the three cohorts.
Study variables
Based on literature, we selected 40 variables commonly used in the comparative effectiveness research (CER). 4, 5 This included 15 drug exposures (use of antiplatelet medications, oral hypoglycemics, antihypertensive medications, non-steroidal anti-inflammatory drugs, opioid medications, antidepressants, antipsychotic medications, anticonvulsants, proton pump inhibitors, antiarrhythmics, statins, dementia medications, hormonal medications, antibiotics, and oral anticoagulants) and 25 co-morbidities (dementia, atrial fibrillation, chronic lung, liver or kidney disease, cancer, diabetes, hypertension, anemia, psychosis, depression, pneumonia, human immunodeficiency virus infection, fracture, rheumatoid arthritis, ischemic stroke, intracranial hemorrhage, myocardial infarction, acute kidney injury, hepatotoxicity, gastrointestinal bleeding, deep vein thrombosis, pulmonary embolism, and congestive heart failure).
Determination of cut-off to define patients with high vs. low predicted EHR-continuity:
To determine cut-off, we used standardized absolute mean difference (SAMD), which is the difference of group means standardized by their (pooled) standard deviations11, between the classification based on only EHR data (subject to missing data due to EHR-discontinuity) and that based on EHR-claims data (the reference standard). Because we required the study population to have the respective insurance coverage, we assume claims data record medical information generated from reimbursable care and not subject to data incompleteness due to EHR-discontinuity. A SAMD of less than 0.1 is generally chosen as an acceptable level of balance in the context of achieving adequate confounding control.12 Each cohort was divided into deciles based on the predicted EHR-continuity score such that higher deciles had higher EHR-continuity. Nine iterations of analysis were conducted using nine possible cut-off points to distinguish between subjects with high vs low EHR-continuity based on the decile of predicted EHR-continuity. For example, in the first iteration, subjects in the top decile were considered as having high EHR-continuity and those in the lower nine deciles were considered as having low EHR-continuity. In the second iteration, subjects in the top two deciles of EHR-continuity were considered as having high EHR-continuity and those in the lower eight deciles were considered as having low EHR-continuity, and so on. At each iteration, the average standardized absolute mean difference (ASAMD) of all the 40 variables, was computed. The highest iteration at which the lower limit of the 95% confidence interval of the ASAMD was less than 0.1 was identified as the cut-off point to define the sub-population with high EHR-continuity.
Assessment of variable misclassification patients with high vs. low predicted EHR-continuity
Using the cut-off determined by the previous step to distinguish between patients with high vs. low predicted EHR-continuity, ASAMD for the 40 CER relevant variables listed above was calculated among the two groups of patients in all the three cohorts. The analysis was conducted for every year of follow up (up to year 7) to ensure consistency in performance of the EHR-continuity prediction algorithm.
Assessment of representativeness of the high EHR-continuity cohort
The combined comorbidity scores (CCS)13 of all patients in each of the three cohorts were calculated using claims data (therefore not subject to misclassification due to EHR-discontinuity). We quantified the difference in comorbidity profiles across study cohorts by the ASAMD of distributions of the comorbidity score among patients with high vs. low EHR-continuity.
This study was approved by the Institutional Review Board of the Brigham and Women’s Hospital, Boston, Massachusetts. The statistical analyses were conducted using SAS 9.4 (SAS Institute Inc., Cary, NC).
Results
Study population and capture proportions
There were 319,740 patients in the MA EHR-Medicare (mean age=74 years [SD=7.6], female%= 59.2, Black race%=2.9, mean CCS=2.1 [SD=3]); 95,113 patients in the MA EHR-Medicaid (mean age=39.2 years [SD=13.9], female%= 65.7, Black race%=15.9, mean CCS=0.5 [SD=1.4]); and 1,319,218 patients in the TriNetX (mean age=44.7 years [SD=16.4], female%= 58.4, Black race%=23.4, mean CCS=0.8 [SD=1.9]) cohorts. (Table 1). 46% of subjects in the MA EHR-Medicare cohort had a diagnosis code for cancer at baseline compared to 10% and 12% in the MA EHR-Medicaid and TriNetX cohorts. Similarly, a higher proportion of subjects in the MA EHR-Medicare cohort had hypertension, diabetes or chronic kidney disease or a prescription claim for antihypertensive or lipid lowering medications compared to those in the MA EHR-Medicaid and TriNetX cohorts. Higher proportion of subjects in the MA EHR-Medicaid cohort (44%) had a prescription claim for opioid medications (38% and 33% in the MA EHR-Medicare and TriNetx cohorts respectively). CP in the first year of follow up was 0.30, 0.18 and 0.19 in the MA EHR-Medicare, MA EHR-Medicaid and TriNetX cohorts respectively (Figure 1a). 52%, 67%, and 58% subjects in the MA EHR-Medicare, MA EHR-Medicaid and TriNetX cohorts respectively had a CP of <0.1. In all the three cohorts, the CP remained constant in the subsequent years (Figure 1b). Additional calculation of CP in the MA EHR-Medicare and MA EHR-Medicaid cohorts without distinguishing between outpatient and inpatient encounters showed consistent results compared with CP calculated making this distinction (Table S3, Figure S2).
Table 1:
Patient characteristics
| Patient characteristic | MA EHR-Medicare | MA EHR-Medicaid | TriNetX |
|---|---|---|---|
| N=319740 | N=95113 | N=1319218 | |
|
| |||
| Age in years, mean (sd) | 74 (7.6) | 39.2 (13.9) | 44.7 (16.4) |
| Female, n (%) | 189387 (59.2) | 62655 (65.7) | 770267 (58.4) |
| Race, n (%) | |||
| Black | 9205 (2.9) | 15106 (15.9) | 308599 (23.4) |
| White | 266342 (83.3) | 50375 (52.9) | 915788 (69.4) |
| Others, including patients with missing race | 44193 (13.8) | 29830 (31.3) | 94831 (7.2) |
| CCS, mean (sd) | 2.1 (3.0) | 0.5 (1.4) | 0.8 (1.9) |
| Dementia, n (%) | 31623 (9.9) | 1404 (1.5) | 26231 (2.0) |
| Atrial fibrillation, n (%) | 52169 (16.3) | 1569 (1.6) | 35416 (2.7) |
| Chronic lung disease, n (%) | 65003 (20.3) | 11709 (12.3) | 172106 (13.1) |
| Chronic liver disease, n (%) | 26488 (8.3) | 8196 (8.6) | 78553 (6.0) |
| Chronic kidney disease, n (%) | 40824 (12.8) | 1869 (2.0) | 51302 (3.9) |
| Cancer, n (%) | 146777 (45.9) | 9501 (10.0) | 157296 (11.9) |
| Diabetes, n (%) | 70232 (22.0) | 7430 (7.8) | 138227 (10.5) |
| Hypertension, n (%) | 254514 (79.6) | 26852 (28.2) | 472751 (35.8) |
| Anemia, n (%) | 85134 (26.6) | 9763 (10.2) | 138294 (10.5) |
| Psychosis, n (%) | 15167 (4.7) | 3800 (4.0) | 42170 (3.2) |
| Depression, n (%) | 61942 (19.4) | 18214 (19.1) | 192072 (14.6) |
| Pneumonia, n (%) | 28071 (8.8) | 4697 (4.9) | 40894 (3.1) |
| HIV, n (%) | 440 (0.1) | 1477 (1.5) | 10966 (0.8) |
| Fracture, n (%) | 9264 (2.9) | 1427 (1.5) | 673 (0.1) |
| RA, n (%) | 9872 (3.1) | 964 (1.0) | 15941 (1.2) |
| Ischemic stroke, n (%) | 4488 (1.4) | 427 (0.4) | 3188 (0.2) |
| ICH*, n (%) | 1531 (0.5) | 348 (0.4) | 2052 (0.2) |
| MI*, n (%) | 3876 (1.2) | 482 (0.5) | 2233 (0.2) |
| AKI*, n (%) | 15737 (4.9) | 1230 (1.3) | 10429 (0.8) |
| Hepatotoxicity*, n (%) | 822 (0.3) | 290 (0.3) | 1349 (0.1) |
| GI Bleeding*, n (%) | 5588 (1.8) | 611 (0.6) | 2776 (0.2) |
| Major bleeding*, n (%) | 16409 (5.1) | 1712 (1.8) | 9435 (0.7) |
| DVT*, n (%) | 1999 (0.6) | 416 (0.4) | 1311 (0.1) |
| PE*, n (%) | 1693 (0.5) | 303 (0.3) | 1350 (0.1) |
| CHF*, n (%) | 29799 (9.3) | 1957 (2.1) | 31569 (2.4) |
| Antiplatelet medications, n (%) | 62418 (19.5) | 6487 (6.8) | 101143 (7.7) |
| Oral hypoglycemic agents, n (%) | 54407 (17) | 7523 (7.9) | 129338 (9.8) |
| Antihypertensive medications, n (%) | 233439 (73.0) | 24403 (25.6) | 337673 (25.6) |
| NSAIDs, n (%) | 66101 (20.7) | 36198 (38.0) | 389986 (29.6) |
| Opioid medications, n (%) | 119650 (37.4) | 41848 (43.9) | 432273 (32.8) |
| Antidepressants, n (%) | 90972 (28.5) | 29155 (30.6) | 282972 (21.5) |
| Antipsychotic medications, n (%) | 15113 (4.7) | 10804 (11.3) | 72490 (5.5) |
| Anticonvulsants, n (%) | 42994 (13.2) | 13928 (14.6) | 161270 (12.2) |
| PPIs, n (%) | 95062 (29.7) | 15259 (16.0) | 197114 (14.9) |
| Antiarrhythmics, n (%) | 9084 (2.8) | 493 (0.5) | 9138 (0.7) |
| Statins, n (%) | 171037 (53.5) | 9090 (9.5) | 220931 (16.8) |
| Dementia medications, n (%) | 12571 (3.9) | 230 (0.2) | 5376 (0.4) |
| Hormonal medications, n (%) | 18077 (5.7) | 9928 (10.4) | 113237 (8.6) |
| Antibiotics, n (%) | 204389 (63.9) | 51851 (54.4) | 589294 (44.7) |
| Oral anticoagulants, n (%) | 39942 (12.5) | 2181 (2.3) | 33732 (2.6) |
MA, Massachusetts; EHR, Electronic health records; CCS, Combined comorbidity score; CHF, congestive heart failure; HIV, human immunodeficiency virus; RA, rheumatoid arthritis; AKI, acute kidney injury; ICH, intracranial hemorrhage; MI, myocardial infarction; PE, pulmonary embolism; DVT, deep vein thrombosis; NSAIDs, nonsteroidal anti-inflammatory drugs; PPI, Proton pump inhibitors.
Note:
All variables except “Race” measured in EHR + Claims data. Race variable measured using EHR data due to lesser missingness in EHR.
Analysis of variance (ANOVA) and chi-square tests comparing characteristics between populations was <0.001 for all variables.
Figure 1:
Capture proportions (CP) among patients in MA EHR-Medicare, MA EHR-Medicaid and TriNetX cohorts
Performance of the EHR-continuity prediction model
The AUROC of the EHR-continuity prediction algorithm predicting good EHR-continuity (observed CP≥0.6) in the MA EHR-Medicare, MA EHR-Medicaid and TriNetX cohorts was 0.92, 0.89 and 0.77, respectively. There was no change in the performance of the algorithm when observed CP was calculated without distinguishing between inpatient and outpatient encounters,
Cut-off to identify patients with high vs. low EHR-continuity
Based on the lower bound of the 95% CI for ASAMD being less than 0.1, top 2 deciles vs lower 8 deciles of predicted EHR-continuity was defined as the cut-off to identify subjects with high vs low EHR-continuity in the MA EHR-Medicare cohort, with subjects in the top 2 deciles having high EHR-continuity (Figure 2). Similarly, in the MA EHR-Medicaid and TriNetX cohorts, top 5 deciles vs lower 5 deciles of predicted EHR continuity was defined as the cut-off to identify subjects with high vs low EHR-continuity, with subjects in the top 5 deciles having high EHR-continuity.
Figure 2:
Average standardized absolute mean difference of 40 CER relevant variables
Evaluation of misclassification
In the first year of follow up, those with low EHR-continuity in the MA-EHR Medicare cohort (lower 8 deciles of predicted EHR-continuity) had an increase in misclassification of 3.73 (95% CI 2.38–6.25) times that of those with high EHR-continuity (top 2 deciles of predicted EHR-continuity) (Table 2). In the MA EHR-Medicaid and TriNetX cohorts, the increase in misclassification among those with low EHR-continuity (lower 5 deciles of predicted EHR-continuity) was 2.58 (95% CI 1.79–4.17) and 3.08 (95% CI 2.09–5.00) times more than those with high EHR-continuity respectively, in the first year of follow up. In each cohort, the pattern of misclassification was consistent in the both groups across the years of follow up, indicating consistency in the performance of the EHR-continuity prediction algorithm.
Table 2:
Difference in Measuring 40 Selected Variables in EHR vs. EHR + Claims Data
| MA EHR-Medicare | |||||||
|---|---|---|---|---|---|---|---|
|
| |||||||
| Year after cohort entry | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|
| |||||||
| ASAMD* (95% CI) | |||||||
|
| |||||||
| Low EHR-continuity | 0.41 (0.31–0.52) | 0.46 (0.33–0.58) | 0.46 (0.34–0.58) | 0.46 (0.33–0.58) | 0.45 (0.33–0.57) | 0.44 (0.33–0.56) | 0.44 (0.33–0.55) |
| High EHR-continuity† | 0.11 (0.09–0.14) | 0.11 (0.09–0.14) | 0.11 (0.09–0.14) | 0.11 (0.08–0.13) | 0.10 (0.08–0.13) | 0.09 (0.07–0.11) | 0.09 (0.07–0.11) |
| Increased misclassification among those with low EHR-continuity (95% CI) ‡ | 3.70 (2.44–4.97) | 4.05 (2.62–5.48) | 4.10 (2.65–5.55) | 4.20 (2.71–5.69) | 4.46 (2.87–6.04) | 4.84 (3.10–6.57) | 4.88 (3.10–6.66) |
|
| |||||||
| MA EHR-Medicaid | |||||||
|
| |||||||
| Year after cohort entry | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|
| |||||||
| ASAMD* (95% CI) | |||||||
|
| |||||||
| Low EHR-continuity | 0.31 (0.22–0.40) | 0.34 (0.24–0.43) | 0.35 (0.25–0.45) | 0.35 (0.26–0.45) | 0.35 (0.26–0.45) | 0.36 (0.26–0.46) | 0.37 (0.27–0.47) |
| High EHR-continuity† | 0.12 (0.09–0.16) | 0.14 (0.10–0.18) | 0.14 (0.10–0.19) | 0.14 (0.10–0.18) | 0.14 (0.10, 0.17) | 0.13 (0.10–0.17) | 0.13 (0.09–0.16) |
| Increased misclassification among those with low EHR-continuity (95% CI) ‡ | 2.54 (1.48–3.59) | 2.43 (1.44–3.42) | 2.42 (1.46–3.39) | 2.54 (1.55–3.54) | 2.60 (1.61–3.60) | 2.72 (1.68–3.76) | 2.94 (1.83–4.04) |
|
| |||||||
| TriNetX | |||||||
|
| |||||||
| Year after cohort entry | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|
| |||||||
| ASAMD* (95% CI) | |||||||
|
| |||||||
| Low EHR-continuity | 0.40 (0.31–0.50) | 0.44 (0.34–0.54) | 0.45 (0.35–0.55) | 0.47 (0.36–0.57) | 0.46 (0.36–0.57) | 0.47 (0.37–0.58) | 0.49 (0.38–0.59) |
| High EHR-continuity† | 0.13 (0.10–0.17) | 0.16 (0.11–0.20) | 0.16 (0.11–0.20) | 0.17 (0.12–0.21) | 0.16 (0.12–0.20) | 0.16 (0.12–0.20) | 0.17 (0.13–0.22) |
| Increased misclassification among those with low EHR-continuity (95% CI) ‡ | 3.04 (1.95–4.13) | 2.86 (1.82–3.89) | 2.88 (1.84–3.91) | 2.80 (1.81–3.79) | 2.94 (1.91–3.96) | 3.00 (1.97–4.03) | 2.84 (1.87–3.81) |
MA, Massachusetts; EHR, Electronic health records; ASAMD, Average standardized absolute mean difference; CI, Confidence interval.
Lower ASAMD indicates less misclassification among 40 selected variables when using EHR data alone vs. linked EHR-claims data (only the former is subject to misclassification due to EHR-discontinuity).
Defined as top 20%, 50% and 50% of predicted EHR-continuity in the Medicare, Medicaid, and TriNetX populations, respectively
Defined as the ratio of ASAMD among those with low EHR- continuity divided by the ASAMD among those with high EHR-continuity.
Evaluation of representativeness
The comorbidity profiles of those with high vs. low EHR-continuity did not differ substantially. The ASAMD for all comorbidity scores in those with high vs. low EHR-continuity was 0.03, 0.03, and 0.04 in the MA EHR-Medicare, MA EHR-Medicaid, and TriNetX cohorts, respectively. (Figure 3).
Figure 3:
Representativeness
Discussion
Based on 3 large US cohorts that consist of individuals with different demographic and insurance coverage backgrounds, we found that the proportion of study EHR capturing all reimbursable encounters recorded in the claims data was generally low and varied substantially by EHR system (30%, 18% and 19% in the MA EHR-Medicare, MA EHR-Medicaid, and TriNetX cohorts, respectively). The predicted EHR-continuity cut-off to achieve satisfactory variable classification (i.e., ASAMD<0.1 compared to the classification of the reference standard) varied substantially by study cohorts (i.e., top 2 deciles in the MA EHR-Medicare cohort vs. top 5 deciles in the MA EHR-Medicaid and TriNetX cohorts). Using these cut-offs to define “high EHR-continuity” in the respective cohorts, we observed a consistent reduction in variable misclassification by 59 to 80% in all three cohorts across 7 years. This pattern is consistent with our previous study which compared Medicare cohorts in health systems based in Massachusetts and North Carolina.5 Findings of our current study further indicate a similar pattern in the younger MA EHR-Medicaid and TriNetX cohorts, but at a different percentile-based cut-off of the predicted continuity.
In clinical practice, medical information often gets repeated across encounters. For example, a discharge diagnosis often gets recorded in the subsequent follow-up appointments in the primary care setting as well as specialist visits if the patient needs such care. A prescription that is ordered by one provider can also be recorded during subsequent encounters during the medication reconciliation process.14 Therefore, researchers theoretically do not need all records to achieve a satisfactory level of variable classification. To determine what values of EHR-continuity correspond to acceptable levels of misclassification, we examined the predicted EHR-continuity alongside the observed misclassification of 40 selected variables commonly used in comparative safety and effectiveness research.4 Using an uniform requirement of variable classification quality informed by literature,12 and after ranking people by their predicted EHR-continuity, our findings suggest researchers would need to restrict to the top 2 deciles of predicted EHR-continuity in the MA EHR-Medicare cohort (older, mean age 74 [SD=7.6]) but could include the top 5 deciles in the MA EHR-Medicaid and TriNetX cohorts (younger, MA EHR-Medicaid mean age 39 [SD=13.9]; TriNetX mean age 45 [SD=16.4]) to achieve the same level of study variable ascertainment quality. The influence of EHR-continuity appears to be greater in older than in younger populations. It is possible that the older population with more comorbidities may require care from different EHR systems (e.g., due to unforeseen emergency room visit or specialty care) and a higher capture proportion of encounters is required to record all these diagnoses whereas there may be more regular routine office visit in the younger population (i.e., for annual physical exam rather than for illness). It is possible that in the regular check-up visit, there is no medical diagnosis made and missing such encounters in the healthier population does not lead to missing information on medical diagnosis.
The differences in EHR-continuity distributions and EHR-continuity prediction model performance should be interpreted in the context of different populations as well as the EHR data acquisition process. The original EHR-continuity prediction model was developed in a study cohort of Medicare Fee for Service (FFS) beneficiaries4, 6, so the Medicaid and TriNetX are considered external validation cohorts. The study datasets for the MA EHR-Medicare and MA EHR-Medicaid cohorts were established by linking EHR from one multi-center delivery network in Massachusetts with the Medicare and Medicaid claims data, respectively. In contrast, the TriNetX cohort consisted of EHR from eleven different delivery networks across variable regions in the US who were enrolled in commercial, Medicaid, or other insurance plans. The multi-network feature of the TriNetX data can add heterogeneity and complexity that could affect the correlation between EHR-continuity and variable assessment accuracy. For example, different EHR systems may have various sources of drug information (e.g., prescribing, electronic administration, vs. medication reconciliation data)5.
Our three study cohorts also represented three different demographic backgrounds related to medical insurance. Our MA EHR-Medicare cohort was restricted to those aged 65 years and older. Since Medicaid is a US healthcare program designed for low-to-moderate income individuals, our MA EHR-Medicaid cohort over-represented people with low social-economic status.15 In contrast, the majority of the TriNetX cohort were commercially insured. The mean age was 74.0, 39.2, and 44.7 years and non-white proportion was 16.7, 47.1, and 30.6% in the MA EHR-Medicare, MA EHR-Medicaid, and TriNetX cohorts. The prevalence of comorbidities also varied substantially across the three cohorts (Table 1). Older adults generally have more comorbidities and prescriptions, and their complex medical needs may sometimes not be met within a single health system (and EHR network).16 Additionally, since the demographic factors and comorbidities are closely associated with medical seeking behavior17, it is not surprising to observe differences in EHR-continuity distribution and model performance across the study populations. However, it is reassuring to find similar patterns of information bias (variable misclassification) reduction within the high EHR-continuity cohorts using cut-offs determined in the current study.
We observed similar comorbidity profiles assessed based on claims data (not subject to misclassification due to EHR-discontinuity) in patients with high vs. low EHR-continuity. It is important to note that while the EHR-continuity predictors are related to healthcare utilization, they represent utilization recorded in the study EHR, not in the entire healthcare system. In other words, EHR data can be fragmented, and people can seek care in other systems. Those considered to have “low EHR-continuity” (hence low healthcare utilization) in a given system can have “high EHR-continuity” (hence high healthcare utilization) in another system. This can explain why the patients with different predicted EHR-continuity determined by EHR utilization factors have similar comorbidity profiles in the claims data (Figure 3).
Our work has some limitations. First, though varied, all patients in the three cohorts were covered by an insurance plan during the study period. The validity of the model in the uninsured population remains unassessed. Second, while it is reasonable to assume that all reimbursable healthcare encounters are captured by the claims data, there may still be some atypical encounters (e.g., telephone encounters, medication refill note, etc.) not captured by claims data that could render meaningful healthcare data. Third, some factors that were designed in the original model, such as “seeing the same provider more than once” cannot be assessed in some datasets (like the TriNetX used in this study). Also, most of the patients in the TriNetX cohort had missing information on encounter type (e.g., inpatient vs. outpatient) and we had to ignore the distinction when calculating observed EHR-continuity. These challenges represent practical limitations in various data settings when comparing EHR-continuity and the lower model performance observed in TriNetX cohort should be interpreted with caution.
When using EHR data in epidemiologic studies, the EHR-continuity algorithm can be used to calculate the predicted EHR-continuity at baseline. To define a subpopulation with high EHR-continuity, based on the findings of our study, we suggest researchers use a cut-off of top 2 deciles of predicted EHR-continuity when creating a cohort of patients ≥65 years and a cut-off of top 5 deciles of predicted EHR-continuity when creating a cohort of patients ≥18 years. At these cut-off points, we have demonstrated that there is a reduction in misclassification of variables commonly used in CER studies.
Conclusions
We compared the influence of EHR-continuity on information bias in 3 large cohorts with linked claims-EHR data in the US. Compared to the MA EHR-Medicare fee-for-service (older) cohort, the MA EHR-Medicaid and TriNetX (younger) cohorts have a lower average EHR-continuity. The EHR-continuity prediction model performance is fair in all three cohorts but the cut-off of predicted values to achieve satisfactory quality in variable classification are different: restricting to the top 20% for the MA EHR-Medicare cohort and restricting to the top 50% for the MA EHR-Medicaid and TriNetX cohorts. With such a restriction, the misclassification of the key variables can be reduced by 60 to 80%. The comorbidity profiles based on claims data are similar in those with high vs. low EHR-continuity.
Supplementary Material
Study highlights.
What is the current knowledge on the topic?
Prior studies have shown that misclassification of variables commonly used in comparative effectiveness research when using electronic health record (EHR) data, can be mitigated by restricting EHR-based analyses to subjects with high predicted EHR-continuity based on a simple algorithm that was developed in a cohort of Medicare beneficiaries.
What question did this study address?
Though the EHR-continuity prediction algorithm has demonstrated reliable performance in the original dataset, its performance in other data sources is unknown.
What does this study add to our knowledge?
Our study validated the algorithm in two separate cohorts: Medicaid and TriNetX. We found that the cut-off to identify those with high EHR-continuity varied by population.
How might this change clinical pharmacology or translational science?
In comparative effectiveness research using EHR data, the continuity prediction algorithm can be used to identify those with high EHR data continuity to reduce misclassification and bias. The cut-off to achieve such a goal varies by population.
Source of Funding:
This project was supported by NIH Grant R01LM012594 and NIH R01LM013204.
Footnotes
Conflict of Interest: The authors declared no competing interests for this work.
References
- 1.Randhawa GS. Building electronic data infrastructure for comparative effectiveness research: accomplishments, lessons learned and future steps. J Comp Eff Res. Nov 2014;3(6):567–72. doi: 10.2217/cer.14.73 [DOI] [PubMed] [Google Scholar]
- 2.Corley DA, Feigelson HS, Lieu TA, McGlynn EA. Building Data Infrastructure to Evaluate and Improve Quality: PCORnet. J Oncol Pract. May 2015;11(3):204–6. doi: 10.1200/jop.2014.003194 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Cowie MR, Blomster JI, Curtis LH, et al. Electronic health records to facilitate clinical research. Clin Res Cardiol. Jan 2017;106(1):1–9. doi: 10.1007/s00392-016-1025-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Lin KJ, Singer DE, Glynn RJ, Murphy SN, Lii J, Schneeweiss S. Identifying Patients With High Data Completeness to Improve Validity of Comparative Effectiveness Research in Electronic Health Records Data. Clin Pharmacol Ther. May 2018;103(5):899–905. doi: 10.1002/cpt.861 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Lin KJ, Rosenthal GE, Murphy SN, et al. External Validation of an Algorithm to Identify Patients with High Data-Completeness in Electronic Health Records for Comparative Effectiveness Research. Clin Epidemiol. 2020;12:133–141. doi: 10.2147/clep.S232540 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Merola D, Schneeweiss S, Jin Y, Lii J, Lin KJ. Advancing an Algorithm for the Identification of Patients with High Data-Continuity in Electronic Health Records. Clin Epidemiol. 2022;14:1339–1349. doi: 10.2147/clep.S370031 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Scala-Foley MA, Caruso JT, Ramos R, Reinhard SC. Medicare eligibility, enrollment, and coverage. Am J Nurs. Feb 2004;104(2):81–3. doi: 10.1097/00000446-200402000-00028 [DOI] [PubMed] [Google Scholar]
- 8.Hennessy S Use of health care databases in pharmacoepidemiology. Basic Clin Pharmacol Toxicol. Mar 2006;98(3):311–3. doi: 10.1111/j.1742-7843.2006.pto_368.x [DOI] [PubMed] [Google Scholar]
- 9.National Center for Environmental Health. Medical Claims Data. Center for Disease Control and Prevention. May 17, 2019. Accessed November 11, 2022. https://www.cdc.gov/asthma/data-analysis-guidance/medicaid-claims-data.htm [Google Scholar]
- 10.Lin KJ, Glynn RJ, Singer DE, Murphy SN, Lii J, Schneeweiss S. Out-of-system Care and Recording of Patient Characteristics Critical for Comparative Effectiveness Research. Epidemiology. May 2018;29(3):356–363. doi: 10.1097/ede.0000000000000794 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Franklin JM, Rassen JA, Ackermann D, Bartels DB, Schneeweiss S. Metrics for covariate balance in cohort studies of causal effects. Stat Med. May 10 2014;33(10):1685–99. doi: 10.1002/sim.6058 [DOI] [PubMed] [Google Scholar]
- 12.Austin PC. Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Stat Med. Nov 10 2009;28(25):3083–107. doi: 10.1002/sim.3697 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Gagne JJ, Glynn RJ, Avorn J, Levin R, Schneeweiss S. A combined comorbidity score predicted mortality in elderly patients better than existing scores. J Clin Epidemiol. Jul 2011;64(7):749–59. doi: 10.1016/j.jclinepi.2010.10.004 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Steinkamp J, Kantrowitz JJ, Airan-Javia S. Prevalence and Sources of Duplicate Information in the Electronic Medical Record. JAMA Netw Open Sep 1 2022;5(9):e2233348. doi: 10.1001/jamanetworkopen.2022.33348 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Norris L What’s the difference between Medicare and Medicaid? Medicareresources.org. February 14, 2022. Accessed November 11, 2022. https://www.medicareresources.org/faqs/whats-the-difference-between-medicare-and-medicaid/
- 16.Elliott J, Stolee P, Boscart V, Giangregorio L, Heckman G. Coordinating care for older adults in primary care settings: understanding the current context. BMC Fam Pract. Aug 7 2018;19(1):137. doi: 10.1186/s12875-018-0821-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Fiscella K, Franks P, Doescher MP, Saver BG. Disparities in health care by race, ethnicity, and language among the insured: findings from a national sample. Med Care. Jan 2002;40(1):52–9. doi: 10.1097/00005650-200201000-0000 [DOI] [PubMed] [Google Scholar]
- 18.Birman-Deych E, Waterman AD, Yan Y, Nilasena DS, Radford MJ, Gage BF. Accuracy of ICD-9-CM codes for identifying cardiovascular and stroke risk factors. Med Care. May 2005;43(5):480–5. doi: 10.1097/01.mlr.0000160417.39497.a9 [DOI] [PubMed] [Google Scholar]
- 19.Wahl PM, Rodgers K, Schneeweiss S, et al. Validation of claims-based diagnostic and procedure codes for cardiovascular and gastrointestinal serious adverse events in a commercially-insured population. Pharmacoepidemiol Drug Saf. Jun 2010;19(6):596–603. doi: 10.1002/pds.1924 [DOI] [PubMed] [Google Scholar]
- 20.Andrade SE, Harrold LR, Tjia J, et al. A systematic review of validated methods for identifying cerebrovascular accident or transient ischemic attack using administrative data. Pharmacoepidemiol Drug Saf. Jan 2012;21 Suppl 1(Suppl 1):100–28. doi: 10.1002/pds.2312 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Tamariz L, Harkins T, Nair V. A systematic review of validated methods for identifying venous thromboembolism using administrative and claims data. Pharmacoepidemiol Drug Saf. Jan 2012;21 Suppl 1:154–62. doi: 10.1002/pds.2341 [DOI] [PubMed] [Google Scholar]
- 22.Waikar SS, Wald R, Chertow GM, et al. Validity of International Classification of Diseases, Ninth Revision, Clinical Modification Codes for Acute Renal Failure. J Am Soc Nephrol. Jun 2006;17(6):1688–94. doi: 10.1681/asn.2006010073 [DOI] [PubMed] [Google Scholar]
- 23.Cushman M, Tsai AW, White RH, et al. Deep vein thrombosis and pulmonary embolism in two cohorts: the longitudinal investigation of thromboembolism etiology. Am J Med. Jul 1 2004;117(1):19–25. doi: 10.1016/j.amjmed.2004.01.018 [DOI] [PubMed] [Google Scholar]
- 24.Cunningham A, Stein CM, Chung CP, Daugherty JR, Smalley WE, Ray WA. An automated database case definition for serious bleeding related to oral anticoagulant use. Pharmacoepidemiol Drug Saf. Jun 2011;20(6):560–6. doi: 10.1002/pds.2109 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Myers RP, Leung Y, Shaheen AA, Li B. Validation of ICD-9-CM/ICD-10 coding algorithms for the identification of patients with acetaminophen overdose and hepatotoxicity using administrative data. BMC Health Serv Res. Oct 2 2007;7:159. doi: 10.1186/1472-6963-7-159 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.