The Impact of Enhanced Critical Care Training and 24/7 (Tele‐ICU) Support on Medicare Spending and Postdischarge Utilization Patterns

Abstract

Objective

To estimate the effect of implementing a tele‐ICU and a critical care residency training program for advanced practice providers on service utilization and total Medicare episode spending.

Data Sources/Study Settings

Medicare claims data for fee‐for‐service beneficiaries at 12 large, inpatient hospitals in the Atlanta Hospital Referral Region.

Study Design

Difference‐in‐differences design where changes in spending and utilization for Medicare beneficiaries eligible for treatment in participating ICUs was compared to changes in a comparison group of clinically similar beneficiaries treated at similar hospitals’ ICUs in the same hospital referral region.

Extraction Methods

Using Medicare claims data from January 2010 through June 2015, we defined measures of Medicare episode spending during the ICU stay and subsequent 60 days after discharge, and utilization measures within 30 and 60 days after discharge.

Principal Findings

Implementation of the advanced practice provider residency program and tele‐ICU was associated with a significant reduction in average Medicare spending per episode, primarily driven by reduced readmissions within 60 days and substitution of home health care for institutional postacute care.

Conclusions

Innovations in workforce training and technology specific to the ICU may be useful in addressing the shortage of intensivist physicians, yielding benefits to patients and payers.

Treatment in hospital intensive care units (ICUs) is costly: Inpatient admissions with an ICU stay have an average cost of $61,800—roughly 2.5 times higher than an admission with no ICU stay (Barrett et al. 2014). Due to the complexity and clinical severity of cases in the ICU, specially trained intensivist physicians achieve the best outcomes (Pronovost et al. 2002; Wilcox et al. 2013). However, there is a shortage of board‐certified intensivist physicians (Halpern et al. 2013); only 37 percent of all ICU patients in the United States are currently covered by intensivist physicians (Lois 2014). In many ICUs, there are no intensivist physicians present on‐site at night or on weekends (Gajic and Afessa 2009). Recent research suggests that 7 to 10 percent of Medicare beneficiaries will require ICU admission each year (Barnato et al. 2004). Therefore, the aging U.S. population is expected to exacerbate this physician shortage, while incurring higher aggregate costs. This has motivated efforts to improve care quality and/or reduce costs in the ICU.

Two approaches have succeeded in addressing these aims. The first is the integration of nurse practitioners and physicians’ assistants (henceforth “advanced practice providers” [APPs]) into the ICU team. Integrating APPs is demonstrably cost‐effective (Fry 2011) and may maintain or decrease length of stay (LOS) and mortality (Hoffman et al. 2005; Kleinpell, Ely, and Grabenkort 2008; Fry 2011; Gershengorn, Johnson, and Factor 2012). The second is the implementation of a tele‐ICU that allows continuous off‐site monitoring of patients, including intensivist physician oversight at night and on weekends for “additional clinical surveillance and support” (Goran 2010). Tele‐ICU implementation is correlated with reductions in ICU and hospital LOS and mortality (Rosenfeld et al. 2000; Breslow et al. 2004; Zawada et al. 2006, 2009; McCambridge et al. 2010; Franzini et al. 2011; Lilly et al. 2011, 2014; Young et al. 2011; Goran 2012; Kohl et al. 2012; Wilcox and Adhikari 2012; Willmitch et al. 2012; Kumar et al. 2013; Sadaka et al. 2013). Evidence on cost reductions from tele‐ICUs for hospitals has been mixed (Franzini et al. 2011; Goran 2012; Coustasse et al. 2014), although more recent research has identified potentially large savings to hospitals (Lilly et al. 2017).

The literature has not considered potential savings to payers nor effects on subsequent medical care after discharge, resulting from either type of intervention. Our study fills gaps in previous research by addressing whether highly trained APPs, supported by 24/7 tele‐ICU monitoring and intensivist physician oversight, can yield savings for payers by reducing health care utilization after hospital discharge for episodes of care in the ICU.

Emory University Hospital (“Emory”) received a Healthcare Innovation Award in the amount of $10.75 million from the Centers for Medicare & Medicaid Innovation (CMMI). They received the award in the summer of 2012 to fund implementation of two ICU programs: a critical care APP residency training program, which began placing graduates in ICUs in January 2013, and a tele‐ICU that began in April 2014 (Buchman et al. 2017). We used a difference‐in‐differences (DD) strategy and Medicare claims data to estimate changes in outcomes among Medicare beneficiaries treated in ICUs connected to the tele‐ICU, relative to Medicare beneficiaries treated in standard ICUs at local comparison hospitals. We examined changes in Medicare spending, inpatient length of stay (LOS), discharge destination, inpatient readmissions, and postdischarge emergency department (ED) visits.

Intervention and setting

Eight ICU/coronary care units (CCUs) in three participating Emory hospitals in Atlanta, Georgia, were connected to the tele‐ICU (Table 1). In general, the staffing plan for each participating ICU included daytime coverage by one to three APPs and an intensivist physician, while one or two APPs covered each ICU at night with intensivist support through the tele‐ICU. Hardware included high‐resolution cameras and monitors and high‐fidelity audio equipment, at every patient bed in the tele‐ICU, for real‐time face‐to‐face communication. The audio–visual communication function was embedded in the tele‐ICU software, which also provided advanced physiologic alerting and redisplay of laboratory, pharmacy, and vital sign data from the hospital information system and bedside physiologic monitors. All participating ICUs went online in April/May 2014, except one CCU that went online in March 2015.

Table 1.

Tele‐ICU Coverage at Emory Hospitals

Hospital	Connected (Participating) ICUs	Tele‐ICU Coverage Start Date	Unconnected (Nonparticipating) ICUs
Emory Saint Joseph's Hospital	1 Medical/Surgical ICU,	4/25/2014	None
	1 Coronary critical unit (CCU),
	1 Cardiothoracic (CT) Surgery ICU
Emory University Midtown Hospital	1 Medical/Surgical ICU,	4/30/2014,	1 Neuro‐ICU
	1 CCU, 1 CT Surgery ICU	3/25/2015 for the CCU	1 General ICU
Emory University Hospital	2 CT Surgery ICUs	5/1/2014	2 Neuro‐ICUs
			1 Surgical ICU
			2 General ICUs

The tele‐ICU was staffed 24/7 by experienced critical care nurses. Tele‐ICU software and nurses monitored patient vital signs and alerted bedside staff to deviations from best practice guidelines. Tele‐ICU nurses also offered “eyes on” monitoring of ICU patients when bedside nurses were occupied with other patients. On night and weekend shifts, both tele‐ICU nurses and bedside nurses consulted with tele‐ICU intensivist physicians, who placed orders to address emerging patient needs.

The residency program course was 6–12 months, with the 12‐month program focused on additional leadership and mentoring skills. APP residents rotated between different ICUs across the Emory system and received training in physical competencies such as chest tube insertion, feeding tube placement, and intravascular access. The knowledge‐based curriculum focused on critical thinking rather than rote memorization. Once the tele‐ICU was operational, all clinicians learned to interface with tele‐ICU staff, but APP residents were not trained to staff the tele‐ICU itself. The first residents graduated in January 2013 and by February 2015, the program had 19 graduates, most of whom remained working in Emory ICUs.

The combined Emory programs allowed residency‐trained APPs to perform many ICU procedures with oversight from the off‐site tele‐ICU intensivist physicians. One tele‐ICU physician covered all eight ICUs during a night or weekend shift and rarely needed an in‐person (bedside) physician to deliver hands‐on patient care. Emory expected the 24/7 monitoring, trained APPs, and off‐site intensivist physician oversight would improve care quality by reducing delays and errors. They expected patients would be discharged sooner (shorter LOS) and in better condition, with less need for institutional care after discharge (fewer readmissions and ED visits, fewer discharges to institutional postacute care [IPAC]), yielding savings for Medicare.

A broad range of patients were admitted to the eight participating ICUs during the study period, from general medical cases to surgical cases. There was a preponderance of cardiac patients because six of the eight units were CCUs or cardiothoracic surgery (CT) ICUs.

Methods

Analytic Design and Sample Selection

The Emory program and our independent evaluation were funded by CMMI, and the population of interest was Medicare and Medicaid beneficiaries (CMMI, 2016).1 Due to lags in Medicaid data from Georgia, and lack of timely and detailed data about beneficiaries enrolled in Medicare Advantage plans during the study period, the analysis was limited to beneficiaries enrolled in traditional fee‐for‐service (FFS) Medicare.

The Emory intervention was not randomly assigned to hospitals or beneficiaries. We therefore utilized a DD design to estimate the impact of the combined interventions (critical care APP residency and the tele‐ICU). This required us to define a sample of beneficiary inpatient stays with ICU services from the eight participating ICUs, and similar inpatient‐ICU stays from comparison hospitals that were not participating in the Emory program. Such samples were defined for the baseline period before the program began, and for the intervention period during which the program was active.

The first step in sample creation was selecting comparison hospitals. The large Emory hospitals are subject to Atlanta, Georgia's unique array of services and competitors, and macroeconomic and regulatory environments. We selected as comparisons all nine acute care hospitals in the Atlanta Hospital Referral Region with at least 250 beds and at least one ICU or CCU similar to those at a participating hospital. These comparison hospitals should experience trends over time that parallel those experienced by the participating hospitals.

We next linked ICU patient registries from Emory to Medicare claims to define intent‐to‐treat criteria for the sample. The registries identified all patients treated in the eight participating ICUs after the tele‐ICU began. From claims, we identified the ICU revenue center codes corresponding to the types of participating ICUs (e.g., CCU). However, the revenue center codes were not specific enough to perfectly distinguish participating from nonparticipating ICUs. Therefore, we also identified the primary and secondary diagnosis for each beneficiary in the registry. The ICU revenue center code, in conjunction with the primary and secondary diagnoses, served as criteria for the intended treatment population. We applied these exact criteria to both the Emory and comparison hospitals, in both the intervention and baseline periods, to define the analytic sample. Any inpatient stay that had a relevant ICU revenue center code (0200 or 021X) and an exact combination of primary and secondary diagnoses as those for at least one patient in the Emory registry was included in the analytic sample. Beneficiaries under 18 were excluded from the sample.

Based on this intent‐to‐treat (ITT) analytic sample, we interpret outcome estimates as the effect of the joint programs on Medicare beneficiaries who received care in the type of ICU covered by the tele‐ICU, and whose primary and secondary diagnoses made them eligible for Emory's programs. To the extent that we included episodes of care that were not actually covered by the tele‐ICU or by residency graduates, estimates of the impact of the two programs will be downward biased.

As the ITT criteria were based on the tele‐ICU registry, and the tele‐ICU went live in April 2014, we treated April 2014 as the start of the intervention period. The analysis used Medicare Part A and B claims and Medicare enrollment data from January 1, 2010, through June 30, 2015. Each inpatient admission to an Emory or comparison hospital by a Medicare FFS beneficiary during this period that met the ITT criteria initiated an episode of care, which included the inpatient stay and subsequent 60 days. To ensure that two (or more) episodes for the same beneficiary were clinically independent, we required a minimum of 120 days to elapse between the inpatient discharge date of one episode and the start of another. Thus, if a beneficiary was discharged on January 1, 2010, any additional episodes she had prior to April 30, 2010 were omitted from the analysis. We did this because the program might influence whether a beneficiary has another ICU stay. If so, the distribution of beneficiaries in the intervention period would differ from the distribution in the baseline period, yielding an invalid comparison. We also removed episodes where the beneficiary died in the hospital since deceased beneficiaries cannot subsequently incur costs or receive additional care after discharge. We revisit this exclusion in section IV. The final analytic file contained 30,360 episodes: 6,129 Emory baseline, 3,093 Emory intervention, 17,136 comparison baseline, and 4,002 comparison intervention episodes.

Cost and Utilization Measures

We calculated total Part A and B Medicare payment per episode, including the hospitalization during which the ICU stay occurred through 60 days following hospital discharge, which captured spending for postacute care (PAC) and subsequent inpatient readmissions or ED visits. We also measured changes in all‐cause readmissions to any hospital within 30 or 60 days after discharge, ED visits to any hospital within 30 or 60 days after discharge, and discharge destination. Although it does not influence Medicare spending due to Medicare's prospective payment system, we explored inpatient LOS, which may suggest changes in quality of care or potential savings to the Emory hospitals.

The 30‐day window for readmissions is consistent with CMS quality measures, while 60 days incorporates extended PAC. These timeframes were applied to ED visits for consistency. Discharge destination outcomes included home without home health agency care; home health agency care; institutional postacute care (IPAC) such as skilled nursing facilities; and “other” destinations such as federal or psychiatric hospitals, or hospice.

Statistical Analyses

We estimated changes in cost and utilization outcomes between the baseline and performance periods for intervention and comparison hospitals, using a regression‐adjusted DD approach. For each of the Emory hospitals, the baseline extended from January 1, 2010, through spring 2014 (see Table 1). The baseline for the comparison hospitals was January 1, 2010, through April 25, 2014. Data were pooled across all comparison hospitals and compared with pooled data for all intervention hospitals to yield a single estimate of program effect over the full intervention period observable in our data set. Limited sample sizes prevented estimating effects for individual ICUs or hospitals.

Despite careful selection of representative hospitals and the common clinical criteria imposed on the intervention and comparison groups, the parallel trends assumption would fail if the two groups experienced differential changes over time in factors correlated with the outcomes of interest. To mitigate this possibility, we controlled for beneficiary‐ and episode‐level factors that could influence cost and utilization. We obtained beneficiary age, sex, race, and Medicaid enrollment from the Medicare Master Beneficiary Summary Files. From the claims data, we identified whether the beneficiary qualified for Medicare due to age or disability, and whether the episode began with a transfer from another inpatient hospital or health care institution. Diagnosis‐related group (DRG) codes assigned to the clinical index episodes were mapped to 25 “major diagnostic categories” (MDCs) using a crosswalk from the National Bureau of Economic Research (NBER, 2014).2 MDCs reflect the primary clinical reason for an inpatient admission. ICD‐9 codes from claims were used to calculate Charlson comorbidity index (CCI) scores, a measure of disease burden and case mix originally developed to predict patient mortality (Charlson et al. 1987). Hierarchical condition category (HCC) risk scores for 2010–2015 were obtained from the Chronic Conditions Warehouse. HCC scores are a separate measure of disease burden and comorbidities developed by the Center for Medicare & Medicaid Services (CMS) (Pope et al. 2000). In both cases, a higher score indicates more conditions and poorer health. Although the CCI and HCC scores are correlated, they do not perfectly overlap, and together control for a wide range of potential differences in case mix between the Emory and comparison groups. In addition to these covariates, the regression model included quadratic terms for age, HCC score, CCI, and an indicator for missing HCC score. Quarterly fixed effects were included to account for changes over time affecting both Emory and comparison hospitals, and hospital fixed effects were included to adjust for time‐invariant differences between the Emory and comparison hospitals.

Results

Descriptive Analysis

Table 2 presents descriptive statistics for outcomes and key covariates. It shows baseline means and differences between the Emory and comparison groups, as well as the unadjusted differential change in outcomes and episode‐level characteristics between the baseline and intervention periods.

Table 2.

Summary Statistics

	Emory Baseline Period (N = 6,129)	Comparison Baseline Period (N = 17,136)	Difference in Baseline Period	Differential Change in Intervention Period
Independent variables
Female (%)	44.4	49.4	5.1***	−2.3*
Nonwhite (%)	41.0	28.5	12.5***	−0.5***
Age	70.2 ± 14.0	72.2 ± 12.8	−2.0***	0.0
Hierarchical Condition Category Score	2.9 ± 3.1	2.7 ± 2.7	0.3***	0.3***
Medicaid (%)	64.1	66.8	−2.7***	0.0
Nondisabled (%)	63.5	64.8	−0.1*	0.2
Hospital Transfer (%)	22.2	6.8	15.4***	−8.7***
Other Health Transfer (%)	3.3	5.4	−2.1***	5.0***
Charlson Comorbidity Index Score	3.1 ± 2.1	2.9 ± 1.9	0.2***	−0.2***
Dependent variables
Total 60‐day Medicare Spending ($)	12,744 ± 18,187	10,969 ± 16,234	1,175***	−1,882***
30‐Day Inpatient Readmissions (%)	20.6	19.4	1.2**	−1.3
60‐Day Inpatient Readmissions (%)	30.4	28.6	1.8**	−2.7**
30‐Day Emergency Department Visits (%)	26.2	24.4	1.8***	−0.5
60‐Day Emergency Department Visits (%)	36.6	35.2	1.4**	−1.3
Length of Stay (days)	11.6	8.6	3.0***	−0.6***
Discharge Destination: Home (%)	46.6	52.2	−5.6***	2.4**
Discharge Destination: Home Health (%)	21.5	17.1	4.5***	4.2***
Discharge Destination: Institutional Postacute Care (%)	22.3	21.2	1.1*	−7.4***
Discharge Destination: Other (%)	9.5	9.5	0.0	0.9

Notes: ± indicates standard deviation for continuous measures.

Differential changes for (%) measures are reported as percentage point changes.

*p < .10, **p < .05, ***p < .01.

Although the differences were nearly all statistically significant, means of episode‐level covariates in the baseline were similar between the Emory and comparison groups. Exceptions include the proportion of nonwhite beneficiaries treated and proportion of episodes starting with a transfer from another hospital, which were both substantially higher for the Emory population than the comparison. All differences remained fairly constant throughout the baseline and intervention periods, except for the rates of transfer from hospitals or other facilities, which declined more for the Emory group than the comparison group.3

In the baseline period, rates of readmissions and ED visits after discharge were slightly higher for the Emory group, while the proportion of episodes with a discharge home without home care (rather than to IPAC) was lower. This was despite an average LOS nearly 3 days longer for the Emory group. Consequently, average Medicare spending per episode was $1,775 higher at Emory in the baseline period.

We note several significant differences in the unconditional change from the baseline to intervention period between the Emory and comparison groups. Relative to the comparison group, the rate of readmissions within 60 days declined for the Emory group, and the average spending per episode of care fell by $1,882. This was accompanied by a reduction in LOS and a shift of discharge destination from institutional IPAC to home (with or without home health care).

Regression Results

Table 3 shows the regression‐adjusted DD estimates for all outcomes. Total spending for the inpatient admission and subsequent 60 days after discharge decreased by an average of $1,486 per episode at Emory relative to the comparison group (p < .01). The relative rate of 60‐day readmissions fell by 2.1 percentage points for Emory relative to the comparison group, a decrease of 7.1 percent (p < .10). These results were accompanied by significant changes in the pattern of discharge destinations. The rate of discharge to home health care for the Emory group increased by 4.9 percentage points (22.53 percent; p < .01) relative to the comparison group, while the rate of discharge to “other” destinations increased by 1.9 percentage points (19.5 percent; p < .05). This was largely offset by a 6.9 percentage point decrease in the rate of discharge to institutional LTPAC, a reduction of 31.0 percent (p < .01) relative to the comparison group. We found no significant changes in the rate of 30‐day readmissions, 30‐ or 60‐day postdischarge ED visits, or inpatient LOS.

Table 3.

Regression‐adjusted Difference‐in‐Difference Estimates

Measure	Average Treatment Effect [95% Confidence Interval]	Standard Error	Percent Change
Cost
Total 60‐day Medicare Spending ($)	−1,486.27*** [−2,385.22, −587..32]	458.65	−11.66
Utilization
30‐Day Inpatient Readmissions	−0.89 [−3.12, 1.34]	1.14	−4.29%
60‐Day Inpatient Readmissions	−2.14* [−4.65, 0.37]	1.28	−7.05%
30‐Day Emergency Department Visits	0.21 [−2.26, 2.68]	1.26	0.60%
60‐Day Emergency Department Visits	−0.54 [−3.23, 2.15]	1.37	−1.46%
Length of Stay (days)	−0.08 [−0.51, 0.35]	0.22	−0.66%
Discharge Destination
Discharge Home	0.19 [−2.38, 2.76]	1.31	0.41%
Home Health	4.85*** [2.40, 7.30]	1.25	22.53%
Institutional Postacute Care	−6.90*** [−8.74, −5.06]	0.94	−30.98%
Other	1.86** [0.15, 3.57]	0.87	19.50%

Notes: 95% confidence interval reported in brackets.

Changes in cost estimated using linear regression; those in utilization rates were estimated using binary logistic regression. Changes in length of stay estimated using a negative binomial regression model. Discharge destination changes estimated using a multinomial logistic regression model. Utilization and discharge destination results presented as estimated average treatment effects (ATE) rather than coefficient estimates because regression models are nonlinear. The ATE reflects the change in outcomes for episodes of care meeting the intent‐to‐treat criteria after the start of the tele‐ICU program, relative to the change that would have occurred in the absence of the two interventions.

Changes in spending reported in dollar terms, changes in LOS reported in days, and all other measures as percentage point changes. All inferences are based on Huber–White robust standard errors.

*p < .10, **p < .05, ***p < 0.01.

These results were mainly consistent with the unadjusted descriptive results in Table 2 in both direction and magnitude. Relative changes in LOS and discharge home were no longer significantly different, while the differential change in discharge to “other” facilities became statistically significant, highlighting the importance of the regression risk adjustment. The general consistency of the unadjusted and adjusted results lends credence to the validity of the comparison group, as relatively little of the differential changes in outcomes between the Emory and comparison groups can be attributed to changes in observable beneficiary or episode characteristics. Nonetheless, we conducted additional sensitivity analyses of the validity of the comparison group.

Sensitivity Analyses

Sensitivity analyses focused on total Medicare spending as this was a primary outcome of interest. Causal attribution of the results to the intervention requires that the parallel trends assumption holds. If differences between the two groups were constant over time prior to the start of the intervention, this strengthens the assumption that the differences would have remained constant in the intervention period as well, in the absence of the intervention. To test this assumption, we compared trends in total spending between the intervention and comparison groups during the baseline period. As additional tests of this assumption, we also estimated placebo interventions, which omit the true period of performance (all episodes April 25, 2014 or later) and treat January 1, 2012, January 1, 2013, and January 1, 2014, as pseudo‐start dates for the tele‐ICU intervention. As the periods for these placebo interventions are all in the baseline period of the true intervention, the estimated effect of the placebo interventions should be close to zero. Large and statistically significant effects for placebo interventions may indicate a failure of the parallel trends assumption.

Estimating regression‐adjusted linear time trends for the Emory and comparison groups in the baseline period, we find that the cost of comparison episodes trended upward at an insignificant $39 per quarter (p = .11), while the cost of Emory episodes trended upward at an insignificant $2 per quarter (p = .97). The $37 difference between the two trends was not statistically significant (p = .46). Even if the trend had been significant, the cumulative effect of the difference (assuming five full quarters of the intervention) would have been just $185: equivalent to roughly 12 percent of the estimated difference attributed to the Emory program by the main model.

Falsely assigning the tele‐ICU start date as January 1, 2012, and allowing the program to run through April 25, 2014, we estimated a $115 increase in spending attributable to the placebo program (p = .82). When we assigned January 1, 2013, as a false start date, we estimated a $635 reduction in spending attributable to the Emory program, a result that is statistically insignificant (p = .22) and coincides with the entry of residency graduates into Emory ICUs, which may be driving the reduction. Assigning a false start date of January 1, 2014, yielded an estimated reduction of only $69 (p = .94), suggesting that the slight discrepancy in trends between the two groups had virtually disappeared prior to the start of the intervention.

As a second sensitivity analysis, we omitted all observations from January 1, 2013, through the tele‐ICU implementation date in 2014, so that the baseline period ended in 2012 and there was a gap or clean period before the intervention began. This eliminated any “contamination” of the baseline period from the entry of residency graduates into Emory ICUs prior to the start of the tele‐ICU. If the residency graduates reduced average spending in the baseline period, then the estimated baseline difference between the two groups would be smaller than the true difference that existed prior to either intervention. This would exert an upward bias (in direction not magnitude) on the DD estimate. Using these data, we found a regression‐adjusted average reduction of $1,874 per episode (p < .01), suggesting that our primary regression‐adjusted model was producing upward biased (and therefore more conservative) estimates of savings.

It is possible that the intervention increased mortality at the margin, in which case certain episodes that were observed in the baseline (beneficiaries who survived) would not be observed in the performance period. If episodes most likely to end in death were more expensive than average, an increase in mortality would censor the most expensive episodes in the intervention period, creating the illusion that average spending fell. We therefore redefined the sample to include episodes during which the beneficiary died in the hospital, adding roughly 3,000 additional episodes to the analytic sample. We estimated an insignificant decrease in mortality of 1.1 percentage points among Emory hospitals relative to the comparison group (p = .16). Although insignificant, the magnitude was substantial, equal to a 12.0 percent reduction in mortality from the baseline. However in this analysis, average Medicare spending fell by $1,226 per episode (p < .01), indicating that the reduction in spending was not attributable to increased mortality.

Lastly, we tested whether results were due to the specification of the model. Because spending outcomes were by definition non‐negative and potentially skewed by high‐cost episodes, linear regression could produce inconsistent estimates. To test for this possibility, we ran a generalized linear model (GLM) with a log link, which is similar to running a linear model on log‐transformed data. However, unlike the log‐linear model, GLM yields consistent estimates of the average treatment effect in the presence of heteroskedastic errors (Manning 1998; Blackburn 2007). With this approach, we found a reduction of $1,627 per episode in the Emory group relative to the comparison group (p < .01), indicating that the estimated reduction in spending was not attributable to the choice of linear specification.

Conclusion and Limitations

Regression‐adjusted estimates indicate that Emory's combined critical care residency and tele‐ICU programs significantly reduced average Medicare spending per episode of care by $1,486 relative to a comparison group. Aggregating these savings over the 3,093 episodes at Emory during the performance period, we estimate that the program saved Medicare approximately $4.6 million over 14 months (95% confidence interval of $1.8 to $7.4 million). This reduction was likely driven by significant declines in the relative rate of 60‐day readmissions, and a substantial shift from institutional PAC to home health care or other institutional care. These shifts in utilization patterns indicate that the interventions improved care quality, reducing the need for subsequent readmissions or high‐intensity postacute care.

Estimated changes in spending do not appear to be driven by a failure of the parallel trends assumption, by the exclusion of episodes in which the beneficiary died in the hospital, or by the linear regression specification.

While our findings are robust to numerous sensitivity checks, we acknowledge several counterintuitive findings. First, the estimated decrease in inpatient LOS was not significant, contrary to previous literature. Second, the estimated decline in 60‐day readmissions was significant, while the estimated decline in 30‐day readmissions was not. The reduction in 30‐day readmissions was, however, substantial (4.3 percent) and consistent with the estimated reduction for 60‐day readmissions (7.1 percent). Additionally, institutional PAC is usually complete within 30 days, but home health episodes last 60 days under Medicare's home health prospective payment system. The shift to more home health care meant longer oversight by home nurses, which may have reduced readmissions during the 60‐day episodes. Lastly, estimated changes in the rate of ED visits within 30 or 60 days after inpatient discharge were small and insignificant. We are uncertain why changes in the rate of ED visits were not consistent with changes in readmissions. However, the general congruence of estimates (reduced PAC intensity, reduced inpatient readmissions, and reduced Medicare spending), coupled with rigorous sensitivity analyses, lends credence to our overall conclusions regarding the benefits of the Emory programs. As we were only able to define a sample of episodes for beneficiaries who could have received treatment in a participating or comparison ICU, rather than those who actually did, the estimated average effects should be viewed as conservative.

The most important limitation of this study was the inability to define an analytic sample based solely on care received in relevant ICUs. Instead, we used criteria reflecting a hospital unit‐based ITT definition, which excluded episodes in which beneficiaries had diagnoses that never caused admission to a participating ICU. The sample was also limited to Medicare FFS beneficiaries. Although this limits the generalizability of results, research suggests that Medicare FFS populations are similar to Medicare managed care populations (Mirel et al. 2012; Haberman 2013). Medicare beneficiaries accounted for 37 percent of ICU days in 2005 (Halpern and Pastores 2010) and this number will likely rise as the U.S. population ages. Our sample, although incomplete, therefore represents an important and growing population of high‐acuity hospital patients.

We were not able to separately distinguish the effect of the APP residency training program from that of the tele‐ICU. Although the criteria defining the analytic sample were based on the Emory tele‐ICU registry, we cannot determine the extent to which these episodes did or did not involve care by residency graduates because we do not know which beneficiaries received care from the graduates. However, estimating a falsification treatment period from January 2013 through April 2014, the period during which beneficiaries may have been treated by residency graduates before the tele‐ICU began, resulted in an estimated savings of $635 per episode. Subtracting this from the full estimated effect for the combined programs leaves a residual of $851 in savings attributable to the tele‐ICU—a result that would have been significant at the 10 percent level with the standard error reported in Table 3.

An additional, minor limitation is that our control variables for clinical severity were less precise than those frequently used in the literature (e.g., Acute Physiology and Chronic Health Evaluation [APACHE] scores) because this information is not available from Medicare claims. However, so long as differences in average clinical severity (conditional on the control variables) remained constant over time, the DD estimator should account for these differences. As our parallel trends assumption appears to hold, we can assume that any differences in average clinical severity between the intervention and comparison groups were constant over time. While the failure to include more precise clinical controls may have increased the standard errors of our estimates, it did not bias the estimated ATEs. The validity of the parallel trends assumption through the baseline period also lends credence to the selected comparison hospitals and beneficiaries.

These results add new information to the literature about tele‐ICU interventions. Prior studies clearly demonstrated improvements in mortality and LOS due to tele‐ICU programs, and we show that combined tele‐ICU and APP residency training programs have the potential to reduce episode cost to payers without the need for additional intensivist physicians. Although this analysis encompassed only a few hospitals located in a single metropolitan area, and only one type of public payer (traditional Medicare), the magnitude of the effects suggests that implementing these interventions in similar environments may yield substantive savings to Medicare and potentially to other payers. The results also suggest previously unexplored benefits for beneficiaries, such as reduced out‐of‐pocket expenses and the negative experience of additional hospitalizations. Future research could determine whether these benefits can accrue among other populations, or can be realized in other hospitals.

One implication of these findings is that total investment in such programs, in particular costly tele‐ICU systems, may be suboptimal. To the extent that benefits accrue to payers, while hospitals bear the cost of the interventions, there will be an underinvestment in such interventions. The Emory P.I. (co‐author Timothy Buchman) notes that sustaining the costly tele‐ICU has been difficult, because ICU admissions covered by the tele‐ICU are reimbursed at the same rate as standard ICU stays. Our results suggest that payers might want to consider whether alternative reimbursement strategies could foster the adoption of similar interventions.

Supporting information

Appendix SA1: Author Matrix.