Evaluating the Accuracy of Existing EMR Data as Predictors of Follow-up Providers

Abstract

In order to evaluate the accuracy of existing EMR data in predicting follow-up providers, a retrospective analysis was performed on six months of data for inpatient and ED encounters occurring at two hospitals, and on related outpatient data. Sensitivity and Positive Predictive Value (PPV) were calculated for each of eight predictors, to determine their effectiveness in predicting follow-up providers. Our findings indicate that access to longitudinal patient care records can improve prediction of which providers a patient is likely to see post-discharge compared to simply using Primary Care Provider data from admissions records. Of the predictors evaluated, a patient's past appointment history was the best predictor of which providers they would see in the future (PPV = 48% following inpatient visits, 35% following emergency department visits). However, even the best performing predictors failed to predict more than half of the follow-up providers and might generate many “false” alerts.

Introduction

In the increasingly complex world of modern medicine, keeping track of patients across care settings can be a difficult task. Patients receive treatment from numerous providers and experience numerous transitions of care. These care transitions are a critical juncture in the healthcare process and are susceptible to poor and incomplete communication. ^1,2

An important handoff occurs when patients are discharged from the hospital after inpatient or emergency department (ED) encounters. Primary Care Providers (PCPs) are often unaware of treatment that patients have received from other providers and in other settings unless informed by the patient themselves. Numerous researchers have described the importance of studying and improving the transition from hospital to outpatient care. ^3–7 Several studies have indicated that poorly handled transitions of care at hospital discharge are responsible for post-discharge adverse events and have suggested that improving this transition would reduce the likelihood of adverse events. ^8–10

To improve communication across this transition, a system of automated notifications was implemented at Intermountain Healthcare (Intermountain), to proactively notify PCPs of the existence of documentation for hospital visits involving their patients. While these providers already have electronic access to this documentation, often they do not access it in a timely manner, if at all, because they are not aware of its existence.

Because Intermountain does not maintain a database of patient-PCP relationships, which provider(s) ought to be notified following a patient's hospital visit can be unclear. The initial design of the system only sends notifications to the provider named as the PCP in the patient's hospital admission record. These data are incomplete and at times inaccurate, and rely on both the patient knowing who their PCP is, and on the admissions clerk asking for the information and then recording it accurately. In addition, follow-up care is often provided by specialists who are not a patient's PCP. ^6,11 These providers might also benefit from being notified about a patient's hospital care.

The purpose of this case report was to find out if existing medical data from electronic medical records (EMRs) could be used effectively to predict with which providers a patient would have appointments for follow-up treatment. To date, a few studies have been published looking at ways to use electronic data to predict who a doctor will consider as “their” patient. ^12,13 These studies do not, however, answer the question of who the patients consider to be “their” doctors, or who they are likely to see for follow-up care.

Methods

This study was approved by the University of Utah and Intermountain Healthcare Institutional Review Boards.

This study analyzed six months of clinical and administrative data from two Intermountain hospitals, LDS Hospital in Salt Lake City, Utah, and McKay-Dee Hospital in Ogden, Utah, and from numerous Intermountain clinics.

Intermountain has developed its own EMR ¹⁴ that is used by both inpatient and outpatient providers for storing and retrieving patient data. Data for this study were gathered from the Enterprise Data Warehouse (EDW), a large database that combines medical records from the EMR with other relevant records, including administrative data.

This analysis compared the accuracy of seven alternative criteria for predicting which providers a patient was likely to have an appointment with after an inpatient or ED encounter to the current method, using the PCP named in the admission record. All eight predictors are listed in ▶.

Table 1.

Table 1 Performance of Predictors on Inpatient and ED Encounters (95% CI)

	Inpatient		Emergency
Predictor	Sensitivity	PPV	Sensitivity	PPV
1 PCP named by patient in admission record	20%	26%	19%	22%
	(19%–21%)	(25%–27%)	(19%–20%)	(21%–23%)
2 Last provider who had a scheduled appointment with the patient	28%	48%	24%	35%
	(27%–29%)	(47%–49%)	(24%–25%)	(34%–36%)
3 Provider(s) who had the most scheduled appointments with the patient in the previous 180 days	31%	38%	28%	25%
	(31%–32%)	(37%–39%)	(28%–29%)	(25%–26%)
4 All providers who had scheduled appointments with the patient in previous 180 days	48%	34%	42%	27%
	(47%–49%)	(33%–35%)	(41%–43%)	(26%–27%)
5 Last provider to order medication for the patient	15%	39%	14%	32%
	(14%–15%)	(38%–41%)	(13%–14%)	(31%–33%)
6 All providers who ordered medication for the patient in the previous 180 days	20%	29%	18%	25%
	(19%–20%)	(28%–30%)	(17%–19%)	(24%–26%)
7 All providers who ordered laboratory tests for the patient in the previous 180 days	26%	12%	18%	13%
	(25%–27%)	(12%–13%)	(17%–19%)	(13%–14%)
8 All providers who dictated clinical documents about the patient in the previous 180 days	48%	7%	33%	5%
	(47%–49%)	(6%–7%)	(32%–34%)	(5%–5%)

CI = confidence interval; ED = emergency department; PCP = primary care provider; PPV = positive predictive value.

Hospital data for encounters classified as either inpatient or ED encounters occurring at either LDS Hospital or McKay-Dee Hospital, between January 1, 2006 and June 30, 2006, were included for analysis. Only the first encounter for each patient was included in the analysis. No encounters were excluded on the basis of age or race.

Each hospital encounter that met the inclusion criteria was compared against all patient appointments found in the EDW for the 60 days after the discharge date. The EDW patient schedule table stores information about outpatient appointments scheduled at all Intermountain and Intermountain-affiliated outpatient clinics. A follow-up appointment was defined as a scheduled appointment found in the patient schedule with a start date within 60 days of the hospital discharge date. Our assumption is that the providers who would be most benefited by a notification about a patient's hospital care are those that treat the patients in the 60 days following their discharge.

While not every scheduled appointment actually occurs, we chose to base our measure on all scheduled appointments rather than only completed appointments. The reason behind an uncompleted appointment is not recorded in the EDW, so it is not possible to tell when a cancellation might have been due to an appointment being mistakenly scheduled with the wrong provider, was simply rescheduled, or was cancelled because it was deemed unnecessary. We are looking for relationships between providers and patients and while a completed appointment may indicate a stronger relationship than an uncompleted one, we still feel that a scheduled appointment is a strong indicator of an existing relationship.

The current predictor, PCP named in the admission record, was recommended by a committee of outpatient providers as the best way to ensure that the alerts go to providers who will find them useful. The other seven predictors were recommended by informaticians who have extensive experience working with EMRs, including work on the problem of determining existing patient-provider relationships based on available electronic data.

For each predictor, data were grouped into a 2x2 contingency table. To view these tables, as well as a brief discussion of how true and false positives and negatives were defined, please see Appendix 1, available as an online resource at www.jamia.org.

For each of the predictors, Sensitivity and Positive Predictive Value (PPV) were calculated. In the context of this analysis, Sensitivity is the percentage of providers who had follow-up appointments scheduled with the patient that were correctly indicated by the predictor, and can be thought of as the percentage of actual follow-up providers who would be alerted by an alerting system triggered by that predictor.

The PPV of each predictor is the percentage of follow-up providers indicated by the predictor that actually had follow-up appointments scheduled with the patient. If that predictor were used as the triggering criterion for an alerting system, 1-PPV is the percentage of alerts that would be sent to providers who did not have a follow-up appointment scheduled with the patient.

In any notification system, it is important not only that the notifications go to as many providers who will treat the patient as possible, but that the number of alerts sent to providers who will not see the patient is limited. It has been demonstrated that too many false positive alerts can cause providers to ignore more relevant alerts, ¹⁵ and this concern was also voiced by the Intermountain outpatient provider committee.

Results

In the first six months of 2006, there were 12,413 inpatient encounters and 26,055 ED encounters at the two hospitals in the study. Some 53% of the inpatient encounters and 33% of the ED encounters had at least one follow-up appointment scheduled at an Intermountain outpatient facility in the next 60 days. For more details about these encounters and the availability of associated prediction criteria please see Appendix 2, available as an online resource at www.jamia.org.

Sensitivity and PPV for the eight predictors are given in ▶. “Last provider who had an appointment with the patient prior to their hospital encounter” had the highest PPV, 48% for inpatient encounters and 35% for ED encounters.

“All providers who had scheduled appointments with the patient in the 180 days preceding their hospital encounter” had the highest Sensitivity, 48% for inpatient encounters and 42% for ED encounters. “All providers who dictated clinical documents about the patient in the last 180 days” also had a Sensitivity of 48% for inpatient encounters, but only 33% for ED encounters. Raw numerical data for all eight of the predictors can be found in Appendix 1, available as an online resource at www.jamia.org.

Discussion

There are two key findings that merit discussion here. First, the predictors that performed best in the analysis were those based on a patient's previous appointment history. It seems logical that past appointments were the best predictor of future appointments—that the providers a patient met with previously would likely be seen again. Each of the three predictors based on the past appointment schedule significantly outperformed the current standard of using the PCP named in the patient's hospital admission record.

The PCP identified in the patient's hospital admission record was not the worst predictor, but neither was it the best. This does not necessarily mean that the identified PCP should not be alerted. In instances where PCP data are collected accurately, it is likely that this provider will appreciate being notified about their patient's treatment regardless of whether or not a follow-up appointment occurs.

The second key finding from the analysis is that none of the data we evaluated were highly accurate in their prediction of follow-up providers. None of the predictors successfully predicted more than half of follow-up providers, and the best predictor still would generate more false alerts than correct ones. Certainly, this analysis was fairly simple in nature and further refinement of the prediction criteria would likely lead to improved performance. Possibilities for improved prediction might include the use of other clinical or administrative data, experimentation with varying windows of time, or the combination of multiple prediction criteria.

No matter how the predictors are created, it is also true that prior medical data will be unable to predict all follow-up appointments. For example, consider that patients being treated at the hospital for a new condition are likely to begin receiving treatment from providers with whom they have no past medical history. Thus, it is also vital to research and provide mechanisms that allow outpatient providers to “pull” information from electronic medical record systems in addition to improving on existing “push” methods.

The results of this study are limited by the fact that the Intermountain EDW only contains data for encounters and appointments taking place at Intermountain facilities. While Intermountain does provide about half of all medical treatment in the region, ¹⁴ the EDW does not contain data for encounters and appointments occurring with providers and facilities outside of the Intermountain network. Some patients that did not appear to have preceding or subsequent outpatient appointments may well have had appointments with providers outside of the Intermountain network. This seems especially likely for ED encounters.

Additionally, the generalizability of this study is limited in that many healthcare networks do not have record systems that provide easy access to both inpatient and outpatient medical records. Not having access to records of previous outpatient care would limit an organization's ability to implement notifications using some of the criteria analyzed here.

Conclusions

Coded, electronic medical data are a rich resource that can be exploited to improve current medical and administrative practices. Our findings indicate that longitudinal electronic patient records, including past appointment records, can improve on current methods of predicting follow-up providers. While none of the predictors we evaluated produced overwhelming results, several outperformed the current prediction method, using the PCP name provided by a patient and recorded by an admissions clerk. Simply substituting one of these predictors for the current method would somewhat improve the accuracy of the automated notification system, but, given the simplistic nature of this study, it is likely that further research would lead to refined prediction criteria that improve upon the sensitivity and accuracy of the predictors evaluated here.