Concordance of Number of Chronic Conditions estimated from Electronic Health Record or Self-Report

Abstract

Purpose:

Self-report or electronic health records can be used to calculate number of chronic conditions for study participants. Although agreement for specific conditions can be found in the literature, there is a lack of information on how the total number of conditions compares between the two sources.

Methods:

Using a long-standing cohort study, the number of chronic conditions was estimated for 351 participants using self-report and data from their electronic health record. Agreement for each condition, and for total number of conditions was estimated using a weighted kappa with 95% Confidence Interval (CI), and using Lin’s concordance coefficient, and a Bland-Altman plot. Predictors of discord were identified using generalized linear models with concordant pairs as reference.

Results:

Discord varied by condition but ranged from 3.7% (diabetes, kappa=0.86) to 33.2% (hyperlipidemia, kappa=0.31). While the mean number of chronic conditions was similar, it showed poor agreement [weighted kappa=0.46 (95% CI: 0.41, 0.52)], and did not show pattern of discrepancy as the average number of chronic conditions increased. Discrepancies were related to condition, male sex, and lower education.

Conclusion:

There are opportunities to ensure that electronic health records are complete, and that patients receive adequate health education to become aware of the conditions that they have.

1. Introduction

A person’s number of chronic conditions (nCCs) is often used in clinical outcomes research as a proxy for general wellness, wellbeing, or burden of health care [1–5]. Additionally, nCCs have been linked with poorer health related quality of life within many specific diseased-defined populations, including diabetes, metabolic health syndrome, chronic obstructive pulmonary disease (COPD), cancer, COVID-19, low back pain [6–11] among others. Multiple sources exist for collecting nCCs including patient reported health surveys (both validated and not), medical chart abstraction, and administrative data; yet no gold standard has been established. There has been extensive research comparing administrative or electronic health record (EHR) data with patient self-report (SR), but most have been within specific populations or for individual conditions [12–19]. To our knowledge, there has been no study to examine overall agreement between nCCs as estimated by EHR and by SR in a general population.

Although many studies use EHR data, it has been noted that the quality of the data therein is limited at best, with discrepancies between primary care records and hospital records, and erroneous records of basic demographics such as race and ethnicity [20–22], let alone more complicated traits. In fact, several studies that compared EHR to SR performed electronic chart abstractions by doctors or other trained health professionals to obtain more accurate information than an EHR data pull alone (adjudicated EHR). SR of co-morbid conditions, often asked as a matrix of conditions preceded by “Has a doctor or other health professional ever told you that you had…” are also often fraught with inconsistencies, which have been associated with patient demographics such as age, sex, self-reported health, and health literacy [23, 24]. Additionally, even single-source data can be discrepant over time. While it is not easy to detect or reconcile those errors in cross-sectional studies, some methods of adjudication have been proposed for longitudinal studies [25, 26].

The impact of these discrepancies could influence epidemiological research conclusions and public health interventions. For example, in the Longitudinal Aging Study of Amsterdam [23], according to self-reported data, there was an increase in cardiac disease (CAD) from 1992–1993 to 2008–2009, whereas prevalence of CAD reported by general practitioners failed to detect an increase. Hypothetically, if SR data was accurate, then reports generated from general practitioners would miss an opportunity to implement public health interventions, leading to an increase in CAD prevalence over time. Conversely, if reports were based on mistaken SR, resources may be mis-allocated by initiating a needless public health intervention. Of note in that study was that agreement on CAD diagnosis between self-report and practitioner rates were moderate and did not change much over time with kappa estimates of 0.72 and 0.75, respectively.

While comparisons of SR and EHR for specific diseases and, to a limited extent discrepancies in nCCs have been shown, few studies have examined what patient demographics may impact specific discrepancies. In fact, when comparing data from these two sources, there are 8 potential situations that may occur, despite that without knowing the truth, only 4 are distinct (Table 1). Specifically, 2 are discrepancy-free (we denote those states 1 and 2), and 6 involve different types of errors (states 3–8). These errors likely have different underlying root causes (Table 1). States 3 and 4 both show disease in SR, but not EHR. This is commonly referred to as “over-reporting;” however, state 3 (erroneous SR of disease) and state 4 (data not captured in the EHR) likely have very different causes. States 5 and 6 are often described as “under-reporting” where a condition is recorded in the EHR, but not SR. Again, state 5 (erroneously in EHR) and State 6 (erroneous denial by SR) likely have very different causes. States 7 (under-detection of existing condition) and 8 (mis-diagnosis) reflect errors that we assume to be small in prevalence in research populations, although likely higher in populations with limited access to care. Unfortunately, unless a true state of disease is known these are indistinguishable from the correctly identified states 1 and 2, respectively; state 3 is indistinguishable from 4; similarly state 5 from 6.

Table 1.

Eight Potential combinations of data reports from electronic health records (EHR) or Self Report (SR) and disease status

Data Reports	Referred to as	True State: No disease	True state: Disease
Condition in SR and EHR	Concordant	State 8: Error (Mis- or Over-diagnosed )	State 2: Correct
Condition in SR, not EHR (SR only)	Over-reported	State 3: Error in SR	State 4: Error in EHR
Condition Not SR, in EHR (EHR only)	Under-reported	State 5: Error in EHR	State 6: Error in SR
Not in SR and not in EHR	Concordant	State 1: Correct	State 7: Error ( Undetected Disease)

These errors may be present for one or many diseases, and the underlying mechanisms that may help predict coding errors may vary by disease or condition. The purpose of this analysis is twofold. First, to compare the nCCs as estimated using SR or using the EHR. Second, to determine if there are patient factors that may predict discrepancies or predict conditions recorded in SR only or conditions found only in the EHR. The first aim will help determine if an assessment of health using SR is adequate to describe a population, or if extracting data from an EHR is necessary. The second aim, although we will not be able to distinguish errors, per se, will enable us to compare our cohort with other studies that have examined discrepancies between SR and EHR, as well as expand this research further by looking at the type of discrepancy.

2. Methods

2.1. Data

The Health Literacy and Cognitive Function among Older Adults (LitCog) study includes a cohort of primary care patients that has been prospectively followed since 2007 [27]. Participants were recruited from either a general medicine clinic or one of six federally qualified health centers in the Chicago area from 2008–2015. At recruitment, participants were English-speaking and between 55–74 years of age. Data was collected from in-person interviews and for the participants recruited from the general medicine clinic, permission to access their EHR was obtained. Due to the manner of data collection on chronic conditions, this analysis focuses on wave three (performed between 2014–2017) and was restricted to participants with EHR data.

2.2. Measures

In wave three, the participants were asked if they ever had been diagnosed with a matrix of eleven of the most common chronic conditions including: hypertension, arthritis, hypercholesterolemia (high cholesterol), cancer, asthma, diabetes, cardiovascular disease (CVD), chronic heart failure (CHF), bronchitis/emphysema/COPD (COPD), stroke, and chronic kidney disease (CKD). ICD codes corresponding to diagnosis or treatment for these conditions (Supplemental Table A.1) were extracted from the EHR for any visit date between a year prior to study enrollment and their wave three interview, per participant consent. The nCCs for each participant were estimated by the number of these eleven conditions that were present, as assessed by SR or EHR separately. We considered “EHR only” reports as any condition being present in the EHR, but not on SR (regardless of the true disease state); and “SR only” as any condition being on the SR, and not in the EHR.

Demographic characteristics were obtained by SR at wave three, as were a series of health behavior questions. Busyness (the pace of daily events for an individual) and routine (the predictability of daily events) were measured by the Martin and Park Environmental Demands (MPED) questionnaire. [28] The scores range from 7–35 and 4–20, respectively, with higher values corresponding to higher levels of self-reported busyness and routine. The Mini-mental state exam (MMSE) was given as a screener to assess cognitive function [29] scores ranging from 0–30 with low scores indicating cognitive impairment. Physical functioning was measured using the Patient-Reported Outcomes Measurement Information System (PROMIS) physical function short-form 10a [30]. Raw scores were calculated with high scores indicating greater functioning. Raw scores were then translated into a corresponding t-score, which rescales the raw score into a standardized score with a mean of 50 and standard deviation (SD) of 10.

2.3. Statistical analysis

We first present descriptive statistics for our sample, and estimated prevalence of each of the 11 conditions by SR and in the EHR, comparing each using McNemar’s test of paired proportions to determine if any were more or less likely to be present in SR only or EHR only. Additionally, we estimated a kappa statistic to assess the level of agreement for each condition. To assess the overall level of agreement and compare the SR nCCs to the EHR nCCs, three methods were implemented: 1) A weighted kappa to estimate the amount of agreement; 2) Lin’s concordance coefficient, to assess the amount of linear agreement adjusted for any mean differences; and 3) A Bland-Altman plot, which graphs the mean of the two against the difference in the two estimates to determine if there was any pattern to the discrepancy (e.g., if there is greater difference in estimates as the average nCCs increases). Potential predictors of discord were identified using generalized linear regression (marginal) models [log-binomial regression to obtain the Relative Risks (RR)], and unstructured covariance structure to account for multiple conditions within participant. This was done three times; first to predict discord, as that analysis is common in most of the existing literature; second, to predict SR only reports; and third, to predict EHR only reports. All models used concordant pairs as reference, and the second and third eliminate the other discrepancy (as it would be impossible to both have SR only and EHR only reports for the same condition). A two-step backwards selection approach was taken. First to explore potential associations with self-reported demographics [age, sex, race, body mass index (BMI), education, and employment status], then considering health-related covariates (PROMIS physical functioning, anxiety, and depression, busyness and routine, a 5-point Likert-scale for general health, and MMSE). RRs and 95% confidence intervals (CIs) are presented, and for condition were calculated with a contrast, estimating the risk for a certain condition relative to all others.

3. Results

3.1. Sample Description

There were 506 individuals who participated in wave three of LitCog, 351 (69.4%) of whom had available EHR data. Sample demographics for the 351 are presented in Table 2. Most of the population self-identified as white (64%), female (71%), with a mean age of 69 (SD=5), and a median nCCs of 3 (range 0–9). Figure 1 shows the prevalence of each condition under consideration by source of information, as well as by maximum (seen in either SR or in EHR) and minimum (seen in both SR and EHR) estimates of prevalence. Between 60–76% of participants were regarded as having hypertension, 41–68% as having arthritis, and 40–74% as having hyperlipidemia.

Table 2.

Characteristics of Health Literacy and Cognitive Functioning among Older Adults at Wave 3 (2014–2017).

Characteristic	Full cohort	With EHR data
N	506	351
Age, mean (SD)	68.6 (5.2)	69.2 (5.4)
Female, N (%)	362 (71.5%)	250 (71.4%)
Race, N (%)
Black	203 (40.1%)	88 (25.1%)
White	255 (50.4%)	225 (64.1%)
Other	48 (9.5%)	38 (10.8%)
BMI (m/kg2), mean (SD)	28.5 (6.3)	27.9 (5.9)
PROMIS physical functioning	47.5 (9.1)	48.5 (8.9)
MPED Busyness, range 7–35	17.3 (5.1)	17.4 (5.0)
MPED Routine, range 4–20	13.5 (3.0)	13.7 (2.8)
MMSE, range 0–30	28.1 (2.1)	28.5 (1.8)
Education, N (%)
High school or less	118 (23.4%)	78 (22.4%)
Some College/technical school	112 (22.2%)	75 (21.5%)
College graduate	101 (20.0%)	77 (22.1%)
Graduate degree	174 (34.5%)	119 (34.1%)
Income, N (%)
< $10,000	46 (9.5%)	34 (10.2%)
$10,000 – $24,999	114 (23.5%)	75 (22.4%)
$25,000 – $49,999	99 (20.4%)	69 (20.6%)
> $50,000	227 (46.7%)	157 (46.9%)
Employment Status, N (%)
Full-time	59 (11.7%)	39 (11.2%)
Part-time	89 (17.7%)	64 (18.4%)
Not working	355 (70.6%)	244 (70.3%)
Number of chronic conditions (nCCs) Median (minimum, maximum)
By Self-report	3 (0, 9)	3 (0, 9)
By EHR	--	3 (0, 9)

Figure 1:

Estimated prevalence of Chronic Conditions by Source of information and estimated kappa and 95% confidence interval for agreement of Self Report (SR) and Electronic health record (EHR).

Prevalence is presented for conditions reported either in the EHR or by SR (yellow), by SR (green), in EHR (red), and those found both in SR and EHR (blue).

The amount of discord varied by condition but ranged from 3.7% (diabetes) to 33.2% (hyperlipidemia), and agreement, as measured by a kappa coefficient, varied from 0.31 for hyperlipidemia to 0.86 for diabetes (Figure 1). Of the discrepancies, cancer, arthritis, CVD, and high cholesterol were more likely to be in SR, whereas diabetes, hypertension, stroke, and CKD were more likely to be in EHR (all McNemar p ≤0.02). Although there was 8.3%, 8.1% and 7.2% discord for CHF, COPD, and asthma, respectively, there were no appreciable differences in rates of reports in SR only or EHR only.

3.2. Number of chronic conditions

The mean and SD for the nCCs was 2.9 (1.7) by SR and 2.8 (1.5) by the EHR. The mean difference (SR-EHR) was 0.07, with a SD of 1.28. Only 72 (20.5%) participants had complete agreement between EHR and SR for all 11 conditions, 140 (39.9%) had one discrepancy, 90 (25.6%) had two, 38 (10.8%) had three, and 11 (3.1%) had 4 or 5. There was no “preference” towards being reported in SR only or EHR only as the number of discrepancies increased. For example, of the 90 participants who had 2 discrepancies, 27 (30%) had two different conditions in the EHR only, 40 (44%) had one condition in the EHR and another only in SR, and 23 (24%) had two different conditions only in SR. Figure 2 shows the number of overall discrepancies by participant, and if those discrepancies were more likely from SR only or EHR only or a combination of discrepancies. The weighted kappa to quantify the degree of agreement was poor 0.46 (95% CI: 0.41, 0.52), as was Lin’s Concordance coefficient 0.68 (0.62, 0.73). The Bland-Altman plot indicates the amount of discrepancy did not vary by the average nCCs, suggesting no trend to the amount or direction of disagreement as the average nCCs increases (Figure 3).

Figure 2:

Number of discrepancies found in SR and EHR by study participant. Neutral (gray) color represents an equal number of discrepancies found in EHR and SR; darker hues of green represent more EHR discrepancies, darker shades of purple represent more SR errors. (e.g. for those with 3 discrepancies, dark green represents those who have 3 conditions listed in the EHR only, lighter green represents those participants with 2 conditions in the EHR only and 1 SR only, light purple represents 1 condition EHR only and 2 conditions SR only, and dark purple represents those with 3 conditions in SR only.

Figure 3:

Bland-Altman plot of number of chronic conditions as estimated by Self-Report (SR) and by Electronic Health Record (EHR), shown with a jitter so that multiple observations can be seen. A blue line represents a mean difference of 0. A pink line represents the observed mean difference (0.07), dotted red line represents two standard deviations (SD) from the mean, and dotted green line represents 3 SDs from the mean (1SD=1.28).

3.3. Predicting Discrepancies

Generalized linear models show that a higher likelihood of discrepancies between SR and EHR was associated with condition, specifically Arthritis [RR and 95%CI: 3.57 (2.71, 4.70)], diabetes [1.72 (1.23, 2.40)], and hyperlipidemia [4.99 (3.83, 6.51)], and male sex [1.43 (1.16, 1.77)]; whereas a lower likelihood was associated with hypertension [0.43 (0.26, 0.72)], and having a college education or higher [0.73 (0.60, 0.90)]. More specifically, having a chronic condition in SR only was associated with arthritis [5.84 (4.19 8.13)], CVD [1.81 (1.15, 2.85)], cancer [2.14 (1.39, 3.28)], and hyperlipidemia [6.61 (4.80, 9.11)], and male sex [1.62 (1.30, 2.03)]; whereas a lower likelihood of SR only was associated with diabetes [0.12 (0.03, 0.48)] and CKD [0.12 (0.03, 0.50)], having a college degree or higher [0.65 (0.52, 0.81)], and working for pay [0.71 (0.56, 0.910]. A higher likelihood of reports in the EHR only was associated with condition, specifically arthritis [RR and 95%CI: 1.99 (1.34, 2.96)], hyperlipidemia [3.49 (2.49, 4.91)], and hypertension [3.08 (2.21, 4.29)] whereas lower likelihood was associated with CVD [0.49 (0.25, 0.95)] and diabetes [0.50 (0.26, 0.99)]. A lower risk of EHR only reports was also associated with a higher routine of everyday demands [RR and 95%CI: 0.95 (0.92, 0.99)] (Table 3).

Table 3:

Relative Risk and 95% Confidence Interval estimates from General Linear Models with logit link predicting any discrepancy, self-reported only and EHR only discrepancies – with reference being concordance.

Predictor	Any discrepancy	SR only	EHR only
	RR and 95% CI	RR and 95% CI	RR and 95% CI
Condition
Arthritis	3.57 (2.71, 4.70)	5.84 (4.19, 8.13)	1.99 (1.34, 2.96)
Asthma	0.67 (0.44, 1.02)	1.01 (0.57, 1.78)	0.66 (0.37, 1.18)
CHF	0.73 (0.49, 1.10)	1.04 (0.60, 1.81)	0.91 (0.53, 1.53)
COPD	0.77 (0.52, 1.15)	1.41 (0.87, 2.29)	0.53 (0.28, 1.01)
CVD	0.90 (0.62, 1.32)	1.81 (1.15, 2.85)	0.49 (0.25, 0.95)
Cancer	1.08 (0.75, 1.57)	2.14 (1.39, 3.27)	0.54 (0.28, 1.05)
High Cholesterol	4.99 (3.83, 6.51)	6.61 (4.80, 9.11)	3.49 (2.49, 4.91)
Diabetes	1.72 (1.23, 2.40)	0.12 (0.03, 0.48)	0.50 (0.26, 0.99)
Hypertension	0.43 (0.26, 0.72)	0.58 (0.29, 1.20)	3.08 (2.21, 4.29)
Kidney	0.78 (0.52, 1.16)	0.12 (0.03, 0.50)	0.89 (0.52, 1.51)
Stroke	0.26 (0.14, 0.48)	0.58 (0.28, 1.19)	1.25 (0.82, 1.92)
Sex (male)	1.43 (1.16, 1.77)	1.62 (1.30, 2.03)	--
College degree or higher	0.73 (0.60, 0.90)	0.65 (0.52, 0.81)	--
Working for pay	--	0.71 (0.56, 0.91)	--
Routine	--	--	0.95 (0.92, 0.99)

4. Discussion

While nCCs provide important information and serve as a valid surrogate for overall health or health care burden, it is important to consider the source of the data used to estimate it. SR is an easy way to obtain information but is subject to recall bias, and may be confounded by sex, education, or work status; a finding reported here and in other studies [23, 31–33]. Information from the EHR is more difficult to obtain, may be limited as it is not collected for research purposes, is subject to coding errors, and as we found may be confounded by level of routineness of the patient, which may indicate certain patients are more likely to either discuss more with their health care providers or ensure their EHR records are up to date. To our knowledge, this is the first study that has examined any association between discrepancies and the MPED scales, and we believe this finding may warrant further investigation. At least one other study [14] has hypothesized “The accurate capture of patient information by EHRs depends on patient awareness and the documentation practices of providers; however, the effectiveness with which patient information is captured is unknown.”

Discrepancies between the two sources also depend on the nature of the comorbidities considered. For example, we speculate that hypertension and diabetes, two conditions often seen only in EHR data, may not be included in SR if they are controlled by medication. Arthritis, a condition often included in SR may not be included in the EHR if the patient does not discuss the matter with their health care team believing that over the counter medication may be the only recourse available. Lastly, we hypothesize that conditions that have clear diagnostic criteria, are easily identified, or need consistent treatment, such as asthma or diabetes, have a higher degree of concordance than conditions that are less defined such as arthritis or high cholesterol. This, too, has been seen in other studies [13–17, 34].

Although researchers are recognizing data quality problems associated with EHRs [20], manual chart abstraction is often considered the closest thing to a gold-standard, but even with experienced chart reviews, the discrepancies with SR persist [18]. We encourage researchers to consider the various types of errors as discussed in the introduction and aim to minimize data collection errors by using validated questionnaires in assessing SR of conditions, as well as a complete, up-to-date list of ICD-9 or ICD-10 codes such as those found at the Chronic Conditions Data Warehouse in data pulls from the EHR. Additionally, extra care should be used when using EHRs to calculate other indices of health, such as the Charlson Comorbidity Index [35], which weights certain conditions, and could potentially exacerbate errors.

This study does have limitations. As this is a secondary analysis of an existing cohort, we could not use all available data. Wording of several of the conditions originated as “During the past 12 months, have you had any of the following” as modeled from prior national health research studies. These included cancer, stroke, HTN, high cholesterol, and peripheral artery disease (included as a CVD diagnosis). It was not until wave three that it was consistently asked “Has a doctor or nurse ever told you that you have.” As the EHR data extraction included patient history, we felt it was more appropriate to use the more historical version of SR. Additionally, permission to access EHR records was only obtained from individuals recruited from the general medicine clinics, and not the federally qualified health centers, which may potentially bias our sample. Full demographics of wave three are included in Table 2, and the subsample with EHR data, apart from race, appear to be representative of the full sample, but there may be unknown characteristics of the patients or the providers that may bias this analysis.

Additionally, all patients were older adults from a single geographic location. However, Lix, et al [32] examined predictors of agreement in SR and administrative data in an adult Canadian population (age 19 years and older), specifically for Arthritis, Asthma, DM, heart disease, HTN, and Stroke. They, too, found differences in agreement by chronic condition, more discord among older persons, found higher concordance among males for arthritis, asthma, and diabetes, but more discordance for heart disease, HTN, and stroke. As they looked at individual conditions, and not nCCs, per se, this finding may have been impacted by the fact that the prevalence of HTN was high in our sample. Additionally, they reported differences in agreement by income for some conditions, a finding that we did not detect. As that study looked at specific conditions, and not nCCs, per se, some of the results that they reported may have been impacted by the prevalence of conditions once summed.

To determine true disease conditions is beyond the scope of this project. Indeed, researchers have developed a PheKB knowledgebase, a collaboration between the NIH Collaboratory and the eMERGE Network, and have dedicated many resources to determine true disease status based on data mining each EHR [36]. However, to obtain the high sensitivity and specificity needed to then use data to help in genetic phenotyping, many individuals are considered “indeterminant of disease status” and removed from further analyses. When trying to apply some of these algorithms to our data, a similar situation occurred. With the increased interest in artificial intelligence (AI) in health care research, and the natural dependence of AI on the EHR, caution should indeed be exercised before determining disease status for any research.

The PheKB knowledgebase is one example of how the research question might determine what measures should be considered in measuring covariates. When trying to determine the prevalence rates in our sample, we considered what might be the ranges by using concordance to determine disease status. That is, we considered an individual has a condition if both the EHR and SR indicated such, we anticipate this would be quite specific in identifying those with the condition. Conversely if the condition was not in either EHR or SR, they were determined to not have the condition – this definition having a higher specificity. While this provided a range, more strict determinations (both sources) may be better in surveillance or screening studies, whereas the more lenient (either source) may be better in etiologic research.

5. Conclusion

In light of these findings, we urge researchers to consider why the use of nCCs is important for the line of research undertaken, to consider multiple sources of information, and to determine potential adjudication methods for conflicting information. Further, there needs to be more effort put forth in assessing true states of disease for individuals and ensuring the individuals are informed as to the state of their own health. This would include improvements in defining disease states, improvements in consistency of EHR for recording past and current conditions, and patient health literacy education so that patients are self-aware and have the opportunity to develop self-efficacy to control their health.

Highlights.

• Self-report and electronic health records vary in reports of chronic conditions.

• Neither self-report nor electronic health records can be considered a gold-standard

• Predictors of discord are both condition and individual specific

• Prognostic models that use only electronic health records, risk false conclusions.

• More work should be done to diagnose and document true disease status.

List of Abbreviations:

nCCs

number of chronic conditions

COPD

chronic obstructive pulmonary disease

EHR

electronic health record

SR

patient self-report

CAD

cardiac disease

LitCog

Health Literacy and Cognitive Function among Older Adults

CVD

cardiovascular disease

CHF

chronic heart failure

COPD

bronchitis/emphysema/COPD

CKD

chronic kidney disease

MMSE

Mini-mental state exam

PROMIS

Patient-Reported Outcomes Measurement Information System

RR

Relative Risks

CI

Confidence Interval

BMI

body mass index