Developing and Validating a Pediatric Potentially Avoidable Transfer Quality Metric

Abstract

We aimed to evaluate a quality metric that identifies pediatric potentially avoidable transfers from diagnosis and procedure codes. Using physician medical record review as the gold standard, the following steps were used: (1) develop the initial metric definition; (2) estimate initial metric definition operating characteristics; (3) refine this definition to optimize the c-statistic; (4) validate this optimized metric definition using a separate sample. The initial metric using Sample A patient transfers had a c-statistic of 0.63 (95%CI 0.53–0.73). Following 22 revisions, the optimized metric definition was a transfer discharged within 24 hours that did not receive any of a select list of 60,268 specialized diagnoses or procedures. The optimized metric on Sample B demonstrated a sensitivity of 80.6%, specificity of 85.7%, and c-statistic of 0.83 (95%CI 0.75–0.91). The quality metric developed and validated in this study demonstrated satisfactory operating characteristics, providing a feasible means to measure this important outcome.

INTRODUCTION:

Many hospitals have insufficient resources to provide definitive care to pediatric patients.^1–3 Pediatric patients thus often require transfer to a specialty hospital. Transition of care research has largely focused on hospital-to-home and intra-facility transitions.^4–6 Inter-facility transfers, however, are a relatively understudied process.⁷ Such transfers, whether for adult or pediatric patients, are known to be fraught with communication breakdowns, diagnosis discordance, and delays in care.^8–11 Transfers can be arduous for providers, burdensome on families, risky for patients, and an inefficient use of resources.^8,12,13 Some transfers are unnecessary. Indeed, potentially avoidable transfers (PAT) are estimated to be relatively common, based on current definitions. Among pediatric patients, 27% to 39% of transferred patients are discharged from the receiving hospital’s emergency department (ED) or hospitalized for less than 24 hours without receiving a specialized intervention,^12,14,15 suggesting that many transfers may be avoidable. Each PAT costs an estimated additional $909 per transfer.¹² Pediatric transfers are increasing over time, with the largest increase occurring among common conditions.¹⁶ PAT are thus possibly increasing over time, and because many PAT represent a poor tradeoff between benefits and harms, it is important to develop valid and feasible metrics for use in measuring the incidence of PAT.

A validated PAT metric using readily available data (e.g., electronic medical record (EMR) data, administrative data) would provide a cost-effective, practical means to compare health systems, facilitating further research to target interventions, improve performance, and evaluate progress. Currently, the definition of PAT varies across studies, and we lack validation studies to support available PAT metrics.^12,14,15,17

The objective of this study was to evaluate the criterion validity of quality metrics that identify pediatric PAT from diagnosis and procedure codes, using multiple physician medical record review as the gold standard. We aimed to develop an improved PAT metric with a discriminative capacity (c-statistic) of at least 0.80 when tested against physician medical record review. This metric is intended for research and for improving quality and value; very high discrimination (>0.95) is unlikely to be achievable without incorporating unmeasurable contextual and relationship factors that properly influence clinical decision-making.

METHODS:

Using independent, blinded medical record review by multiple physicians as the gold standard, we used the following steps to develop, refine, and test the PAT metric: (1) developed the initial PAT metric definition based on existing literature; (2) estimated the sensitivity, specificity, likelihood ratios (LR) positive and negative, and c-statistic of the initial PAT definition; (3) refined this definition to optimize the c-statistic; and (4) then validated this optimized PAT definition using a separate sample.

Data Source and Study Population:

The primary data source was EMR data from a 129-bed urban non-freestanding children’s specialty hospital. The data were queried using the Epic Clarity database, which contains inpatient and ED records on all discharged patients. Variables of interest included ED and/or inpatient length of stay, procedure codes, diagnosis codes, and disposition. International Classification of Diseases, 10th Revision, Procedure Classification System (ICD-10-PCS) codes were used to capture procedures performed in the inpatient setting. Current Procedural Terminology (CPT) codes were used to capture procedures and professional services furnished in the ED. International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM) codes were used to capture diagnoses. Finally, Epic order codes were queried, because these codes capture the most comprehensive list of procedures, including procedures that might not be represented in billing data.

From all discharges that occurred from October 1, 2015 to September 30, 2016, we randomly sampled 300 patients who met inclusion criteria. A sample of 150 patient transfers was randomly selected among the 300 to be used for the initial analysis and refinement of the PAT metric (Sample A). The remaining 150 patient transfers were used as a separate sample to validate the PAT metric (Sample B). Eligible patients included children aged 8 days to 18 years admitted by transfer from an acute care hospital’s ED or inpatient unit, including patients who arrived to the specialty hospital’s ED and those who were directly admitted to an inpatient unit. To restrict the sample to patients at moderate-to-high risk of PAT, we included only patients discharged from the receiving hospital within 24 hours from inpatient admission, if applicable, or arrival to the ED. Although it is possible that a true PAT might not be discharged within 24 hours, such events are extremely rare based on our clinical experience. Restricting the sample to those at moderate-to-high risk of PAT was important to enrich the sample with true PATs, maximizing our ability to fine-tune the PAT metric definition. Using a timeframe of 12 to 24 hours is also common practice among researchers studying PAT.^12,14,17 Neonates <8 days of age were excluded given their different transfer patterns in comparison to non-neonates.^12,18 To understand characteristics of the study sample, the following data were obtained from Epic: patient age, sex, race, ethnicity, insurance status, post-transfer arrival type (ED versus direct admission), and post-transfer hospital location (ED, non-intensive care unit, or intensive care unit). We used percentages to summarize these patient-level characteristics; categorical variable data was compared between Sample A and Sample B using Pearson’s chi-square test.

Physician Medical Record Review:

For validation purposes, two or three physicians reviewed each patient’s records from Epic, the EMR system, to categorize transfers as PAT or not PAT. A trained research assistant printed and de-identified relevant Epic documents from the post-transfer hospital. Documents included transfer summaries, admission notes, consultation notes, study and procedure results, and discharge summaries. Four pediatric hospitalists with clinical experience working at both a specialty hospital and a community hospital independently evaluated the documents to categorize each transfer as potentially avoidable or not. These physicians did not use a standardized assessment tool, because we did not want to influence their perceptions and thus categorization of the patient transfers. Rather, we wanted this gold standard to be guided by clinical logic. Each patient transfer was initially categorized by two physicians who were blinded to the identities of the patient, referring hospital, and treating physicians. Independent physician categorizations were compared. In case of disagreement, a third physician independently categorized the case. The respective triad of reviewing physicians for each discrepant case then met to share their individual interpretations of each discrepant case and then independently re-categorize the case. This process was performed first for patient transfers from Sample A and then for patient transfers from Sample B. Reliability of the record review categorizations was estimated between the two initial physicians prior to discussing discrepant results as well as among the three physicians after their discussions using the intraclass correlation coefficient (ICC).¹⁹ Categorizations were reconciled using majority vote.

Initial PAT Metric Definition:

The initial PAT metric definition, based on existing literature,^12,14,15,17 was a transferred patient discharged from the receiving hospital’s ED or hospitalized for less than 24 hours without receiving a specialized diagnosis or procedure. This definition based on diagnosis and procedure codes categorized transfers as PAT or not PAT. The list of specialized diagnoses and procedures that fed into PAT metric categorization were developed by four pediatricians and a professional coding analyst by applying lists from prior publications. The PAT categorization followed a hierarchy that defaulted towards “not PAT”. For example, a patient with a radiograph (not specialized procedure), cough (not specialized diagnosis), and electroencephalogram (specialized procedure) was deemed “not PAT” due to the presence of a specialized procedure.

Analysis of Initial PAT Metric:

Using reconciled physician medical record review categorization as the gold standard, we used Sample A patient transfers to estimate the operating characteristics (sensitivity, specificity, and LR positive and negative) of the initial PAT definition. We calculated 95% confidence intervals (CIs) for sensitivity and specificity using the Wilson score method.²⁰ We applied the method by Simel et al²¹ to calculate intervals for the LR.

Refining the PAT Metric:

We performed an iterative revision process to refine the PAT metric definition. This process involved reviewing cases where the gold standard disagreed with the PAT metric categorization to inform the re-categorizing of individual diagnosis and procedure codes. By adding and removing codes considered to be specialized diagnoses and procedures, we created PAT metric definitions with different combinations of ICD-10 codes, CPT codes, and Epic procedure order codes. For each new PAT metric definition, we re-estimated the operating characteristics to inform ongoing revisions.

To provide a basis for comparison among revised PAT metric definitions, we used logistic regression to estimate the area under the receiver operating characteristic (AUROC) curve or c-statistic for each definition, as the c-statistic provides the most useful single-number summary of the discriminative capacity of an ordinal classification rule.^22,23 CIs for c-statistics were derived from the algorithm by DeLong et al²⁴ and asymptotic normal CIs. The PAT definition with the highest c-statistic (under two-sided testing) was determined to be the refined metric.

To enhance the refined PAT metric definition, we used code sets and classification algorithms to further expand the list of predictors. Specifically, we referenced 2016 CPT and ICD-10-CM/PCS code books to identify additional specialized diagnoses and procedures. We also incorporated additional specialized diagnosis and procedure codes based on classifications from the Agency for Health Research and Quality (AHRQ) Clinical Classification Software²⁵ and the list of therapeutic and diagnostic procedures from the AHRQ Procedure Classes for ICD-10-PCS.²⁶ This process was performed with a professional coding analyst and in consultation with a pediatric neurologist, pediatric orthopedist, pediatric surgeon, and pediatric neurosurgeon. After expanding the list of specialized codes feeding into the PAT metric definition, we re-estimated the operating characteristics to ensure that the process of expanding the list of diagnosis and procedure codes to create the optimized PAT metric did not decrease the c-statistic.

Testing the Optimized PAT Metric:

We tested the initial metric and the optimized metric in a separate sample of 150 patient transfers (Sample B). This separate sample was used for validation testing to minimize overfitting biases. We identified the diagnoses of the false positive and false negative cases to gain information on limitations of the PAT metric. Analyses was performed using Stata 15.²⁷ This study was reviewed and approved by the University of California Davis IRB.

RESULTS:

Patient-level demographic and utilization characteristics for the 300 patient transfers included in this study are presented in Table 1. Mean patient age was 6.3 years. The majority of patients had public insurance (64.0%). Fifty-three percent of patient transfers were directly admitted to an inpatient unit, rather than arriving to the ED at the post-transfer hospital. Among those direct admissions, 36.2% were directly admitted to an intensive care unit. The patient-level characteristics were distributed similarly between the Sample A and Sample B groups.

Table 1:

Patient-Level Characteristics Presented by Sample Group

	Sample A N = 150	Sample B N = 150	P-value
Age group, years			0.68
0 – 1	33 (22.0)	37 (24.7)
2 – 5	50 (33.3)	42 (28.0)
6 – 11	31 (20.7)	37 (24.7)
12 – 18	36 (24.0)	34 (22.7)
Sex			0.48
Female	67 (44.7)	61 (40.7)
Male	83 (55.3)	89 (59.3)
Race			0.48
White	63 (42.0)	75 (50.0)
Black	18 (12.0)	10 (6.7)
Asian	12 (8.0)	12 (8.0)
Other	52 (34.7)	48 (32.0)
Unknown	5 (3.3)	5 (3.3)
Ethnicity			0.84
Hispanic or Latino	38 (25.3)	38 (25.3)
Not Hispanic or Latino	110 (73.3)	111 (74.0)
Unknown	2 (1.3)	1 (0.7)
Insurance status			0.56
Private	50 (33.3)	54 (36.0)
Public	97 (64.7)	95 (63.3)
Other	3 (2.0)	1 (0.7)
Post-transfer arrival type			0.36
Emergency department	74 (49.3)	66 (44.0)
Direct admission	76 (50.7)	84 (56.0)
Post-transfer hospital location			0.64
Emergency department	74 (49.3)	66 (44.0)
Non-ICU	49 (32.7)	53 (35.3)
ICU	27 (18.0)	31 (20.7)

Data presented as n (%). Categorical variable data was compared between Sample A and Sample B using Pearson’s chi-square test.

ICU – intensive care unit

The physicians’ medical records review of transfers demonstrated good inter-rater reliability.²⁸ As shown in Table 2, the Sample B pre- and post-discussion ICCs (95% CI) were 0.62 (0.47–0.72) and 0.82 (0.65–0.91), respectively. Reconciled categorizations for the gold standard found 14.7% of Sample A and 20.7% of Sample B to be PAT (p-value = 0.17).

Table 2:

Reliability of the Physician Medical Record Review

	ICC (95% CI)
	Prior to Discussions of Discrepant Cases	After Discussions of Discrepant Cases
Sample A (n=150)	0.64 (0.51–0.74)	0.85 (0.72–0.93)
Sample B (n=150)	0.62 (0.47–0.72)	0.82 (0.65–0.91)

ICC – intraclass correlation coefficient

CI – confidence interval

The initial PAT metric using Sample A patient transfers had low sensitivity (31.8% (95% CI 16.4–52.7%)) but high specificity (93.8% (95% CI 88.2–96.8%)), with a c-statistic of 0.63 (95% CI 0.53–0.73).

The iterative revision process to refine the PAT metric definition involved 22 revisions. The final list of specialized diagnosis and procedure codes that comprised the optimized PAT metric definition included 53,039 ICD-10-PCS codes, 225 CPT codes, 69 Epic procedure order codes, and 6,935 ICD-10-CM codes. The majority of the codes included in the optimized PAT metric definition were major therapeutic or diagnostic procedures from the list of AHRQ Procedure Classes for ICD-10-PCS.²⁶ The Stata macro program code to apply the final PAT metric to user data is provided as supplemental material (Program S1). Table 3 shows example specialized diagnoses and procedures that comprise the PAT metric.

Table 3:

Examples of the PAT Metric Specialized Diagnoses and Procedures

Example Diagnoses	Example Procedures
• Epidural hemorrhage • Crushing injury of hand • Crohn’s disease with intestinal obstruction • Appendicitis • Acute respiratory failure • Sickle-cell with acute crisis • Diabetes mellitus with ketoacidosis • Drowning • Poisoning	• Fluoroscopy • Electroencephalogram • Computed tomography head • Echocardiography • Excision of appendix • Surgical pathology • Respiratory ventilation • Intubation • Phototherapy

The examples provided in this table are only a few select diagnoses and procedures. The list of specialized diagnoses and procedures that comprise the PAT metric is much more extensive.

The optimized PAT metric using Sample A patient transfers had higher sensitivity (90.9% (95% CI 72.2–97.5%)) with adequate specificity (84.4% (95% CI 77.1–89.6%)). The c-statistic improved to 0.88 (95% CI 0.81–0.94). The operating characteristics of the initial and optimized PAT metric using Sample A are presented in Table 4.

Table 4:

Operating Characteristics of Initial and Optimized PAT Metric Using Sample A: The Refinement Process

	Sensitivity	Specificity	LR+	LR-	c-statistic
Initial PAT Metric	31.8% (16.4–52.7%)	93.8% (88.2–96.8%)	5.09 (2.05–12.62)	0.73 (0.54–0.97)	0.63 (0.53–0.73)
Optimized PAT Metric	90.9% (72.2–97.5%)	84.4% (77.1–89.6%)	5.82 (3.81–8.89)	0.11 (0.03–0.40)	0.88 (0.81–0.94)

PAT – potentially avoidable transfer

LR – likelihood ratio

CI – confidence interval

Each characteristic is reported with (95% CI). CI for sensitivity and specificity calculated using the Wilson score method. CI for likelihood ratios calculated using the method by Simel et al. CI for c-statistics derived from the algorithm by DeLong et al.

In the validation Sample B, the optimized PAT metric demonstrated a sensitivity of 80.6% (95% CI 63.7–90.8%), specificity of 85.7% (95% CI 78.3–90.9%), positive LR of 5.64 (95% CI 3.52–9.06), and negative LR of 0.22 (95% CI 0.11–0.46). The c-statistic was 0.83 (95% CI 0.75–0.91), which was above the goal c-statistic of 0.80. The operating characteristics of the initial and optimized PAT metric using Sample B are presented in Table 5.

Table 5:

Operating Characteristics of Initial and Optimized PAT Metric Using Sample B: The Validation Testing Process

	Sensitivity	Specificity	LR+	LR-	c-statistic
Initial PAT Metric	22.6% (11.4–39.8)	92.4% (86.2–96.0)	2.98 (1.21–7.38)	0.84 (0.69–1.02)	0.58 (0.50–0.65)
Optimized PAT Metric	80.6% (63.7–90.8%)	85.7% (78.3–90.9%)	5.64 (3.52–9.06)	0.22 (0.11–0.46)	0.83 (0.75–0.91)

PAT – potentially avoidable transfer

LR – likelihood ratio

CI – confidence interval

Each characteristic is reported with (95% CI). CI for sensitivity and specificity calculated using the Wilson score method. CI for likelihood ratios calculated using the method by Simel et al. CI for c-statistics derived from the algorithm by DeLong et al.

During the validation testing, the optimized PAT metric categorized 28.0% of Sample B as PAT. There were 17 false positive cases misclassified as PAT by the optimized metric. Among these false positive cases, the principal diagnoses were mostly respiratory illnesses, representing 10 of the 17 cases. Specifically, 7 patients had asthma, 2 had bronchiolitis, and 1 had croup. Although patients with asthma as the principal diagnosis represented only 12 of the 150 patient transfers in Sample B (8.0%), this group represented 41.2% of the false positive cases. There were 6 false negative cases misclassified as not PAT by the optimized metric. The diagnoses and procedures for each of the false positive and false negative cases are provided in Supplemental Table 6.

DISCUSSION:

We aimed to evaluate quality metrics that identify pediatric PAT from diagnosis and procedure codes. Our goal was to develop an improved PAT metric with adequate discrimination to support its use in research to target interventions, improve hospital performance, and evaluate progress. Our improved PAT metric demonstrated satisfactory operating characteristics, including a sensitivity of 80.6%, specificity of 85.7%, and c-statistic of 0.83, which achieved our goal c-statistic of at least 0.80. To our knowledge, this study is the first to validate a quality metric that identifies pediatric PAT. The improved PAT metric uses EMR diagnosis and procedure codes, so application of this quality metric is very feasible.

Our research builds on prior studies that have quantified and described pediatric transfers with rapid discharges and no specialized interventions. These prior studies found that 27% to 39% of transferred patients were discharged from the receiving hospital’s ED or hospitalized for less than 24 hours without receiving a specialized intervention.^12,14,15 Another study refined this approach by adapting the list of specialized interventions in order to stratify pediatric transfers as “probably avoidable” and “possibly avoidable,” finding 19% to be probably avoidable and an additional 30% to be possibly avoidable.²⁹ The use of physician medical record review in this study suggests that the PAT problem is possibly not as common as previously indicated. Rather, based on physician medical record review, pediatric PAT represents about 15% to 21% of transferred children who had short stays at the receiving hospital. Nonetheless, given the risks (e.g., safety risks, delays in care, burdens on families)^8–11,13 and high cost of transfers,¹² the relatively common occurrence of PAT found by physician medical record review in our study supports the importance of targeting efforts to address this problem.

Our improved PAT metric was developed with the intention of being used in future research and quality improvement efforts to evaluate and track performance. This metric is not intended to be used as a clinical decision-making tool, as such application of the metric is not appropriate. This metric uses data from the post-transfer hospital in order to categorize transfers as potentially avoidable or not potentially avoidable for the purpose of measuring the incidence of PAT. Because the metric uses post-transfer hospital data, it does not account for the many pre-transfer hospital factors that might influence the decision to transfer a patient. Medical decision-making during pediatric transfers is a complex process that involves various factors such as physician skills and comfort, availability of resources, and policies. For example, the majority of pediatric ED visits occur in community hospitals that see fewer than 15 children per day;³⁰ this low pediatric volume can contribute to lack of preparedness in providing definitive management for common pediatric conditions. Many ED providers do not have access to necessary pediatric specialists and resources.^1–3,31 The use of telemedicine as a means of providing expert consultation prior to initiating a transfer can offer assistance and reassurance to local physicians, thus preventing some unnecessary transfers.^32–34 However, not all healthcare facilities have access to the telemedicine technology.³⁵ Finally, research suggests that hospitals with written transfer protocols are less likely to transfer pediatric patients.³⁶ In our present study, by categorizing pediatric transfers as PAT based on having an early discharge and not receiving a specialized diagnosis or procedure, our PAT metric does not consider these many other factors.

As this PAT metric is intended to be used for research and quality improvement, perfect accuracy is not necessary. With a sensitivity of 80.6% and specificity of 85.7%, our PAT metric will sufficiently be able to identify outlier hospitals with high or low PAT proportions. Our PAT metric will also sufficiently allow monitoring of trends over time to evaluate the impact of interventions. Now that it has been validated in a single hospital, an important next step is to validate the improved PAT metric at multiple medical centers, with the view to establishing its generalizability. To support its use as a quality of care indicator, the operating characteristics, especially the specificity, would need to improve.

Respiratory illnesses contributed the majority of false positive cases (i.e., incorrectly classified as PAT) in this study. An explanation for this finding is that many severe respiratory illnesses are managed with supportive care alone and can have a rapid recovery process. Common respiratory illnesses (e.g., acute bronchiolitis, asthma with status asthmaticus, croup) may have a broad range of illness severities. Therefore, children with respiratory illnesses who require inter-facility transfer might not receive a specialized diagnosis or procedure. As a result, such necessary patient transfers will be categorized as potentially avoidable by the PAT metric. One strategy to address this issue and thus improve the specificity of the PAT metric is to include medications in the PAT metric definition. In this way, for example, a patient who receives multiple hours of continuous albuterol nebulization could be identified and categorized as not PAT. Incorporating medications into the PAT metric definition was not used in this study, because we wanted to keep the PAT definition as simple as possible.

Limitations:

We included Epic procedure order codes in the PAT metric definition because the ICD-10-PCS and/or CPT codes appeared to be incomplete. Epic procedure order codes captured the comprehensive list of procedures that were performed for transferred patients including some that were not represented in ICD-10-PCS or CPT codes. One consequence of including Epic procedure order codes in the PAT metric definition is that the metric cannot be used on claims data. However, Epic is the most commonly installed EMR system for inpatient hospital use.³⁷ Despite this limitation, this study provides the first validated metric that captures pediatric PAT.

Limitations of this study also include the use of a single site. Although the study sample includes multiple pre-transfer hospitals, the post-transfer hospital was one medical center. The transfer practices at this single site may differ from other sites. Importantly, this study builds the foundation for a subsequent multi-center study. Another limitation is that the sample was limited to patients discharged within 24 hours of transfer. We assumed that any transfers that remained at the post-transfer hospital for more than 24 hours were not PAT. Although longer stays may occasionally follow avoidable transfers, this unusual scenario is likely to reflect idiosyncratic social circumstances, such as lack of transportation, which would have limited generalizability across sites. Another limitation is that physician record reviewers may have had difficulty ascertaining some reasons for unavoidable transfers, such as referring physician discomfort or lack of experience. Furthermore, the difference between Sample A and Sample B in the proportion of patient transfers categorized as PAT for the gold standard (14.7% versus 20.7%) highlights potential limitations in using physician chart review as the gold standard. Since the patient transfers from Sample A were categorized before those from Sample B, the physicians’ perceptions of what was considered to be potentially avoidable could have adapted during their categorization process.

CONCLUSION:

In summary, the improved PAT metric developed and validated in this study demonstrated satisfactory operating characteristics that support its use for research and quality improvement purposes. By identifying pediatric PAT using EMR diagnosis and procedure codes, this metric provides a feasible means to measure and monitor the frequency of this important outcome across hospitals and over time.