Missing clinical and behavioral health data in a large electronic health record (EHR) system

Abstract

Objective Recent massive investment in electronic health records (EHRs) was predicated on the assumption of improved patient safety, research capacity, and cost savings. However, most US health systems and health records are fragmented and do not share patient information. Our study compared information available in a typical EHR with more complete data from insurance claims, focusing on diagnoses, visits, and hospital care for depression and bipolar disorder.

Methods We included insurance plan members aged 12 and over, assigned throughout 2009 to a large multispecialty medical practice in Massachusetts, with diagnoses of depression (N = 5140) or bipolar disorder (N = 462). We extracted insurance claims and EHR data from the primary care site and compared diagnoses of interest, outpatient visits, and acute hospital events (overall and behavioral) between the 2 sources.

Results Patients with depression and bipolar disorder, respectively, averaged 8.4 and 14.0 days of outpatient behavioral care per year; 60% and 54% of these, respectively, were missing from the EHR because they occurred offsite. Total outpatient care days were 20.5 for those with depression and 25.0 for those with bipolar disorder, with 45% and 46% missing, respectively, from the EHR. The EHR missed 89% of acute psychiatric services. Study diagnoses were missing from the EHR’s structured event data for 27.3% and 27.7% of patients.

Conclusion EHRs inadequately capture mental health diagnoses, visits, specialty care, hospitalizations, and medications. Missing clinical information raises concerns about medical errors and research integrity. Given the fragmentation of health care and poor EHR interoperability, information exchange, and usability, priorities for further investment in health IT will need thoughtful reconsideration.

INTRODUCTION

In a push to improve patient safety and reduce medical costs, as of 2015, over 500 000 US physicians and almost 6000 hospitals should have operating electronic health records (EHRs) and health information technology systems, or face penalties in Medicare reimbursement.^1–3 These rules were in the 2009 The Health Information Technology for Economic and Clinical Health Act, part of the economic stimulus that year.^2,3 However, the fragmentation of US health care and the lack of interoperability and information exchange among the hundreds of EHR systems currently in use^4–6 threaten the achievement of the underlying safety goals of this legislation, because the medical information in most EHRs is incomplete, which can result in medical errors.⁷

Insured individuals in the US are typically assigned to a primary care provider site, which may have an EHR, but individuals frequently receive specialty care at other locations which do not (and usually cannot) share data with that EHR. Computer decision support systems in EHRs are intended to protect patients at the time of prescription by guiding drug selection and dosing, and alerting physicians about dangerous drug-drug or drug-disease interactions. However, increased physician reliance on computer decision support in the present context of fragmentation and incomplete data may lead to poor-quality care and medical injury.⁸ For example, duplicated drug orders (in different EHRs within the same hospital) resulted in excess administration of insulin, leading to the death of a patient.⁹ A study of recent malpractice filings identified 147 cases where EHRs contributed to patient harm; 46 of those resulted in patient death.^9,10

We hypothesized that fragmentation might be especially common in mental health care, because patients may protect their privacy by seeking behavioral care at a separate location from their somatic care.^11,12 Primary care physicians then not only run the risk of medication errors, but also miss opportunities to encourage adherence to mental health visits and medications. Treatment adherence is particularly poor among mentally ill outpatients and can lead to adverse outcomes, including hospitalization.^13–15

While fragmentation and incomplete clinical data in EHRs are recognized problems, almost no published data estimate their extent. Since health insurers maintain claims data on almost all drugs and health services received by a covered population, insurance claims can validate the completeness of mental health care data in provider EHRs. In this report, we compared the information recorded in a typical EHR at a large premier interdisciplinary practice with data from insurance claims during the same period. We focused on diagnoses, outpatient and emergency department visits, and hospitalizations for depression and bipolar disorder, common conditions that may be treated with a combination of psychotropic drugs and ongoing outpatient treatment. These mood disorders affect roughly 10% of the adult population and impose major social costs due to functional impairment, suicidality, health care use, and lost work productivity^16–19; for the year 2000, the economic burden of depression alone in the US was estimated to be $83 billion.²⁰ We also included overall health services in our analyses of missingness, as these results have important implications for both patients with mental illness and other patients.

METHODS

Study Population and Variables

Our study population included members of Harvard Pilgrim Health Care (HPHC) aged 12 and over who were assigned for primary care throughout 2009 to Harvard Vanguard Medical Associates (HVMA), a multispecialty medical practice serving over 300 000 in Massachusetts. HPHC is the second largest health insurer in New England with over 1 million members, roughly 15% of whom are assigned to HVMA. Both organizations are national health care quality leaders^21,22 while typifying the fragmented US health system, in that HPHC contracts with multiple provider organizations and HVMA cares for patients with a mix of insurance arrangements. HMVA was an early pioneer in the use of EHRs and since 2000 has relied on Epic™, the most popular EHR system in the US.^23–25

Our 2 study cohorts included members with at least 1 claims diagnosis^26,27 of depression (but no diagnosis of bipolar disorder, N = 5140), or diagnosis of bipolar disorder (N = 462) in 2009. Together these comprised 13% of continuously enrolled HPHC/HVMA adolescents and adults.

Study outcomes included depression or bipolar diagnosis in the EHR from a 2009 clinical encounter; claims-based counts of outpatient care days (total and behavioral), hospitalizations, and emergency department visits; and comparable counts of face-to-face clinical care days from the EHR. Outpatient and inpatient care days in the EHR were combined because outpatient hospital visits could not be classified automatically to outpatient or inpatient status. We excluded laboratory-only and imaging-only events from counts in both data sources. Descriptive study measures were obtained from insurer enrolment files (age, sex), US census data linked to member address (neighborhood-level educational attainment), and pharmacy claims (psychotropic medication use).

Analyses

Study data were thoroughly checked for internal and external consistency.^28–30 We calculated the proportion of each cohort whose diagnosis of interest appeared in 2009 structured EHR data. We compared service utilization results using several approaches. First, within the insurance claims alone, we calculated the average number of distinct days of outpatient care (total and behavioral, separately) at the EHR site versus days with care only received elsewhere.

Next, we identified outpatient care days, hospitalizations, and emergency department visits in claims (total and behavioral, separately) and calculated the proportion of these that could not be found in the EHR data (by matching on patient ID, service date, and type). Finally, we tallied all days of care (inpatient or outpatient combined, total and behavioral separately) in the EHR, and calculated the proportion of these that could not be found in the claims data.

RESULTS

Table 1 shows the characteristics of the study cohorts. Both patient cohorts were more likely to be female when compared to members overall. A majority of the study population received antidepressant medication during the observation year, and roughly 3 out of 4 patients with bipolar illness received a mood stabilizer (anticonvulsant or lithium). Nineteen percent of patients with bipolar illness experienced an acute psychiatric event at a hospital during the year. Only 72.7% of patients with depression and 72.3% of those with bipolar disorder had their diagnosis recorded in a 2009 EHR clinical encounter.

Table 1:

Selected characteristics of the study cohorts and the prevalence of service use in the study year, according to insurance claims and the primary site EHR

Characteristics and Service Use	All insured members assigned to EHR practice site for primary care	Cohort having depression diagnosis in claims	Cohort having bipolar disorder diagnosis in claims
	N = 43 582	N = 5140	N = 462
Characteristic (%)
Aged 12–19 years	14.0	9.0	16.5
Aged 20–39 years	27.0	25.7	30.1
Aged 40–64 years	54.2	60.5	50.6
Aged 65 + years	4.7	4.8	2.8
Female sex	54.4	67.1	61.5
Low-education neighborhood	23.5	21.1	20.3
Prevalence of any service use, according to claims (%)
Overall
ED visit (w/o hospital admission)	–	23.5	35.7
Hospital admission	–	11.8	23.2
Either an ED or a hospital admission	–	27.2	39.8
Behavioral
Mental health specialist visit		85.8	97.2
Psychiatric ED visit (w/o hospital admission)	–	2.8	13.4
Psychiatric hospital admission	–	2.4	16.7
Either psychiatric ED or psychiatric hospital admission	–	3.7	18.8
Medication dispensing, according to claims
Any antidepressant use (%)	–	64.7	59.7
Avg months supplied among antidepressant users	–	10.0	10.6
Any antipsychotic use (%)	–	4.9	47.4
Avg months supplied among antipsychotic users	–	6.3	8.1
Any mood stabilizer use (%)	–	29.6	76.0
Avg months supplied among mood stabilizer users	–	5.6	11.6
Prevalence, according to EHR (%)
Any depression diagnosis	–	72.7	–
Any bipolar disorder diagnosis	–	–	72.3
Any mental health specialist visit	–	58.0	74.0

All individuals in the table had HPHC insurance and HVMA primary care assignment throughout 2009. A “low-education” neighborhood had >50% of adults aged 25+ who had attained no more than high school–level education. Individuals in the depression cohort had 1 or more ICD-9 code for depression (296.2x, 296.3x, 300.4x, 301.12, 309.0x, 309.1x, 309.28, or 311.x) in an inpatient or outpatient claim in 2009, and no diagnosis of bipolar disorder; individuals in the bipolar disorder cohort had 1 or more bipolar codes (296.0x, 296.4x, 296.5x, 296.6x, 296.7x, or 296.8x) in an inpatient or outpatient claim in 2009. We defined acute services as psychiatric using either the institutional code or primary/admitting diagnosis. We included EHR diagnoses from structured fields in outpatient or inpatient encounters, not other encounter types (eg, telephone contacts, letters) or free-text fields. Hyphens indicate outcomes that were not measured.

In general, service events appearing in the EHR also appeared in claims data. However, a substantial majority of behavioral service events observed in claims for patients with depression and bipolar disorder were missing from the EHR.

Sites of Outpatient Care Identified in Insurance Claims

Based on claims data only, patients with depression had an average of 3.3 behavioral visits (ie, outpatient care days) with a provider at the EHR site (HVMA) in 2009, and another 5.1 visits (or 60% of total behavioral) outside of the EHR site (Figure 1). Patients with bipolar disorder had an average of 6.4 behavioral visits at the EHR site versus 7.6 (54%) at external provider sites.

Figure 1:

Average number of days with outpatient care per patient in 2009, total and behavioral, and percentage of care days occurring at external provider sites, according to insurance claims. In the analyses below, we defined “behavioral” as having any mental health specialist provider. We conservatively classified days that included care at both the EHR site and external providers (∼3% of total) as occurring at the EHR site. Although all subjects were assigned to HVMA for their primary care, we conservatively classified care occurring beyond HVMA in the broader Atrius provider network as occurring at the EHR provider site, because Atrius providers share a single EHR. For all categories of care shown below, average days at the EHR site as derived from claims were identical to average days at the EHR site as derived from EHR data (average EHR counts not shown).

Use of external services was not limited to mental health care. Our claims analyses indicated that patients with depression averaged 11.2 outpatient visits at the EHR site and 9.3 additional visits outside of the EHR site; that is, 45% of outpatient visits of all types were received at external sites (generally not captured by the EHR). Patients with bipolar disorder had 13.6 outpatient visits at the EHR site plus 12.2 visits externally (ie, 46% of total outpatient visits were external; Figure 1).

Cross-Linkage of Specific Care Events between the 2 Data Sources

When we linked data from both sources to match specific events across systems, the results were very similar: roughly half of the outpatient care days in claims could not be matched to clinical contacts recorded in the EHR (Table 2, Claims events). Consistent with our data in Table 1, the extent of missingness observed in these matched analyses was greater for behavioral services as compared to overall outpatient care.

Table 2.

Matched analyses showing service events as enumerated in 1 data source and the percentage of those events that were missing in the second data source, for claims events and EHR events

Data source and type of service	Depression		Bipolar Disorder
	(N = 5140)		(N = 462)
	Count in claims data	Percentage of events in claims data that were missing from EHR data (%)	Count in claims data	Percentage of events in claims data that were missing from EHR data (%)
(a) Claims events
Outpatient care days (cohort mean)
Total	20.5	45	25.0	43
Behavioral	9.4	57	15.6	53
Emergency department visits (cohort total)
Total	1646	47	262	57
Behavioral	146	88	71	92
Hospitalizations (cohort total)
Total	1174	26	210	53
Behavioral	229	90	148	88
	count in EHR data	Percentage of events in EHR data that were missing from claims data (%)	count in EHR data	percentage of events in EHR data that were missing from claims data (%)
(b) EHR events
Days with clinical care (cohort mean, inpt or outpt)
Total	11.2	7	14.0	6
Behavioral	4.2	2	7.5	3

We defined the percentage missing as the proportion of events in the first data source that did not have matches in the second data source. Matching required the same patient ID, event type, and service date (±1 day), and for the behavioral subset, we required the presence of behavioral coding. For outpatient care days in claims and all care in the EHR, “behavioral” was defined as having any mental health diagnosis, procedure, or provider specialty code (ie, a broader definition than in the figure). For ED visits and hospitalizations in claims, “behavioral” was defined as having a psychiatric institutional code or psychiatric primary/admitting diagnosis. EHR events may be missing from claims as a result of duplicate coverage, limits on coverage, partial encounters, or other reasons.

Hospital-based events were also substantially underrepresented in the EHR. Among all acute psychiatric services in claims (594 hospital admissions or ED visits), 89% were missing from the EHR. Overall, 43% of all hospital-based events (hospital or ED, psychiatric and non-psychiatric) were missing from the EHR. By contrast, clinical events appearing in the EHR could be matched to events in claims in 93–98% of cases (Table 2, EHR events).

DISCUSSION

Very little published data exists on the completeness of medical information in primary care EHRs.^31–33 We assessed EHR completeness for patients with depression or bipolar disorder by quantitatively comparing diagnoses and service use in a major primary care site EHR to those in claims data during the same period. While a broad range of specialty services is available at the primary site studied and some specialist care was documented in their EHR, roughly a quarter of current depression and bipolar diagnoses and more than half of behavioral visits were missing. Data missingness was similarly high for non-behavioral care, both inpatient and outpatient. Nearly 90% of acute psychiatric services at hospital facilities, representing more severe exacerbations of mental illness, were not captured in the EHR.

The US Centers for Medicare and Medicaid Services (CMS) defines EHR “meaningful use” with detailed criteria, including requirements that an EHR contain some diagnostic information on each patient and checks for medication interactions, and is capable of exchanging clinical data with other providers.³⁴ Higher stages of meaningful use are generally defined by the presence of additional functions. However, the CMS criteria do not specify the quality of the data or the practical usability of any functions. Missing data undermines many central EHR functions. Published reports touting the anticipated benefits of the recent rapid adoption of EHRs^35–38 (eg, improved safety and quality of care) should be tempered by frank examinations of EHRs as they currently exist. Above all, individual providers and health system leaders need to be fully cognizant of the information gaps and disconnects that lie behind the screen. Features that are intended to improve care and protect patients from harm may be inadequate in typical fragmented health systems, offering false comfort.

Missing clinical information is likely to result in medication errors and other patient harms.³⁹ Concurrent psychiatric treatment has multiple implications for the delivery of safe, high-quality somatic care. For example, use of antipsychotic agents should be accompanied by enhanced monitoring for cardiometabolic complications,⁴⁰ and many commonly used pain relievers carry extra risks in the presence of SSRI or lithium use.⁴¹ Primary care practitioners with knowledge of their patients’ previous depression treatment will be more likely to care for important conditions such as postpartum depression⁴² or emergent bipolar disorder.⁴³ EHRs are often credited with improvements in safety when compared with earlier paper and manual systems.^44,45 However, to the extent that medical decision-making grows increasingly automated and reliant on new information technologies,^46–48 care will suffer if there is overreliance on data that are in fact incomplete.³⁹ We found no studies assessing data completeness in paper charts or whether completeness improved or deteriorated after a shift to electronic records.

Given extensive information missingness, epidemiological and evaluative studies that rely on data from typical provider EHRs^49–51 will undercount diagnosed patient populations and their use of services and medications. With appropriate privacy safeguards, EHRs can assist in clinical research: rich narrative data and test results, for example, may be useful in case identification and measurement of severity for small, nonrepresentative studies. However, for population-level studies, insurance claims are more complete and systematic, providing a foundation for reliable research.^52–58

Our findings should be understood in the context of several limitations. We focused on patients with mental illness and their behavioral care services, and behavioral care may represent a unique case. It is often segregated by location, though the multispecialty EHR site in our study (HVMA) has behavioral clinics in almost all locations. Insurers and employer sponsors frequently offer and manage behavioral care benefits separately from other care. All study subjects had behavioral coverage through the insurer HPHC, which then carves out coverage to a managing vendor; this arrangement did not affect our access to behavioral claims data. At HVMA, a privacy firewall in the EHR prevents other clinicians from viewing psychiatric diagnoses entered by mental health specialists. Thus, our study greatly underestimates the missingness of behavioral data experienced by primary care providers who use the EHR. However, all diagnoses were available in the de-identified data abstracted for this study.

While behavioral health care is unique, it is important to emphasize that our findings demonstrate that the problem of incomplete clinical data in the EHR is not limited to behavioral care. Rates of missingness were high among both behavioral events and overall events, both in and outside the hospital. Specialist care of all types is particularly likely to be underrepresented in a primary care EHR.⁵⁹ HVMA is a multispecialty provider group; we expect that in many other simpler primary care settings, the extent of missing specialist care in the EHR would be far higher than at HVMA.

Our study did not include medication data from the EHR; this would include initial prescription orders written by internal providers, dispensing at in-house pharmacies, and patient-reported medications. Given the extent of outside service use we observed and the well-documented problem of primary nonadherence (ie, not filling prescriptions),⁶⁰ an attempt at measuring drug utilization based on data in the EHR would underestimate use, as compared to essentially complete insurer-paid pharmacy claims.^58,61,62

Our analyses focused on current service encounters and the diagnoses made at these encounters, both of which were underrepresented in the EHR. EHR data often contain other potentially useful information about mental and non-mental health concerns that are not available in claims, such as diagnoses from current letters, telephone calls, and electronic correspondence; diagnoses entered in earlier years on the patient’s “active problem” list; and free-text clinician comments. These supplementary internal data may be available for clinicians using the EHR during encounters even if current diagnoses and services from off-site utilization are missing. However, these supplementary items represent “softer” data, which typically do not feed into patient safety and management functions like Clinical Decision Support. Similarly, it is possible for providers at HVMA to access patient information at some external partnering institutions, but such inquiries must be conducted individually; this information does not flow into the EHR. Finally, HPHC provides claims data extracts to HVMA for a minority of members in managed care-style financial risk–sharing arrangements; these claims are used in some HVMA case management activities but, again, do not feed into the EHR or its safety functions. The general attributes of the HVMA EHR have not changed since the time of our study and the EHR meets CMS meaningful use benchmarks.

We present findings from one large multisite, multispecialty setting that may not be generalizable to other settings. For example, in a rural setting, patients may have fewer choices for specialist care and may receive a higher proportion of their care at their primary site. Nevertheless, the continuing fragmentation of US health care⁵⁹ ensures that incomplete clinical data in primary site EHRs is a widespread problem.

Better interoperability could be facilitated with national technical standards. Federal policies to date have tilted too far in accommodating EHRs vendors’ desire for flexible, voluntary standards. The incompatible products that result undermine public health goals and can lock providers in to proprietary systems that cannot easily share data. Other barriers to information exchange between entities include provider concerns about costs and liability and patient concerns about confidentiality. Future progress may be possible through more accountable insurer-provider arrangements which include the sharing of data from claims with EHRs and clinicians, patient-driven data-sharing mechanisms, and regulations requiring more complete, timely, and usable data flows where access already exists.^4,5,63,64 EHRs have long been touted as a technology that will advance patient safety in the United States, and enormous public and private investment has been funneled into EHR development in recent years.^65,66 Adoption of EHRs among US office-based physicians increased from 48% in 2009 to 83% in 2014,⁶⁷ though the extent of their actual use and effectiveness is unknown. In this research, we found that the lack of integration, interoperability, and exchange in US health care resulted in a major EHR missing roughly half of the clinical information. Policymakers should put more focus on the quality and utility of health information and ways these can be improved, instead of simply tallying up EHR purchases and supposed capabilities.

FUNDING

This research was an infrastructure core activity of the Mental Health Research Network and was funded by a supplement from the National Institute of Mental Health to the existing Cancer Research Network, funded by the National Cancer Institute, and through a 3-year cooperative agreement with the NIMH (U19 MH092201, “Mental Health Research Network: A Population-Based Approach to Transform Research”).

COMPETING INTERESTS

The authors affirm that they have no competing interests.