The Effects of a Multifaceted Intervention to Improve Care Transitions Within an Accountable Care Organization: Results of a Stepped-Wedge Cluster-Randomized Trial

Abstract

BACKGROUND

Transitions from hospital to the ambulatory setting are high risk for patients in terms of adverse events, poor clinical outcomes, and readmission.

OBJECTIVES

To develop, implement, and refine a multifaceted care transitions intervention and evaluate its effects on postdischarge adverse events.

DESIGN, SETTING, AND PARTICIPANTS

Two-arm, single-blind (blinded outcomes assessor), stepped-wedge, cluster-randomized clinical trial. Participants were 1,679 adult patients who belonged to one of 17 primary care practices and were admitted to a medical or surgical service at either of two participating hospitals within a pioneer accountable care organization (ACO).

INTERVENTIONS

Multicomponent intervention in the 30 days following hospitalization, including inpatient pharmacist-led medication reconciliation, coordination of care between an inpatient “discharge advocate” and a primary care “responsible outpatient clinician,” postdischarge phone calls, and postdischarge primary care visit.

MAIN OUTCOMES AND MEASURES

The primary outcome was rate of postdischarge adverse events, as assessed by a 30-day postdischarge phone call and medical record review and adjudicated by two blinded physician reviewers. Secondary outcomes included preventable adverse events, new or worsening symptoms after discharge, and 30-day nonelective hospital readmission.

RESULTS

Among patients included in the study, 692 were assigned to usual care and 987 to the intervention. Patients in the intervention arm had a 45% relative reduction in postdischarge adverse events (18 vs 23 events per 100 patients; adjusted incidence rate ratio, 0.55; 95% CI, 0.35-0.84). Significant reductions were also seen in preventable adverse events and in new or worsening symptoms, but there was no difference in readmission rates.

CONCLUSION

A multifaceted intervention was associated with a significant reduction in postdischarge adverse events but no difference in 30-day readmission rates.

Transitions from the hospital to the ambulatory setting are high-risk periods for patients in terms of adverse events, poor clinical outcomes, and readmission. Processes of care during care transitions are suboptimal, including poor communication among inpatient providers, patients, and ambulatory providers1,2; suboptimal communication of postdischarge plans of care to patients and their ability to carry out these plans3; medication discrepancies and non-adherence after discharge4; and lack of timely follow-up with ambulatory providers.5 Healthcare organizations continue to struggle with the question of which interventions to implement and how best to implement them.

Interventions to improve care transitions typically focus on readmission rates, but some studies have focused on post-discharge adverse events, defined as injuries in the 30 days after discharge caused by medical management rather than underlying disease processes.2 These adverse events cause psychological distress, out-of-pocket expenses, decreases in functional status, and caregiver burden. An estimated 20% of hospitalized patients suffer a postdischarge adverse event.1,2 Approximately two-thirds of these may be preventable or ameliorable.

The advent of Accountable Care Organizations (ACOs), defined as “groups of doctors, hospitals, and other health care providers who come together voluntarily to give coordinated high quality care to their patients,” creates an opportunity for improvements in patient safety during care transitions.6 Another opportunity has been the advent of Patient-Centered Medical Homes (PCMH), consisting of patient-oriented, comprehensive, team-based primary care enhanced by health information technology and population-based disease management tools.7,8 In theory, a hospital-PCMH collaboration within an ACO can improve transitional interventions since optimal communication and collaboration are more likely when both inpatient and primary care providers (PCPs) share infrastructure and are similarly incentivized. The objectives of this study were to design and implement a collaborative hospital-PCMH care transitions intervention within an ACO and evaluate its effects.

METHODS

This study was a two-arm, single-blind (blinded outcomes assessor), stepped-wedge, multisite cluster-randomized clinical trial (NCT02130570) approved by the institutional review board of Partners HealthCare.

Study Design and Randomization

The study employed a “stepped-wedge” design, which is a cluster-randomized study design in which an intervention is sequentially rolled out to different groups at different, prespecified, randomly determined times.9 Each cluster (in this case, each primary care practice) served as its own control, while still allowing for adjustment for temporal trends. Originally, 18 practices participated, but one withdrew due to the low number of patients enrolled in the study, leaving 17 clusters and 16 sequences; see Figure 1 of Appendix 1 for a full description of the sample size and timeline for each cluster. Practices were not aware of this timeline until after recruitment.

Study Setting and Participants

Conducted within a large Pioneer ACO in Boston and funded by the Patient-Centered Outcomes Research Institute (PCORI), the Partners-PCORI Transitions Study was designed as a “real-world” quality improvement project. Potential participants were adult patients who were admitted to medical and surgical services of two large academic hospitals (Hospital A and Hospital B) affiliated with an ACO, who were likely to be discharged back to the community, and whose PCP belonged to a primary care practice that was affiliated with the ACO, agreed to participate, and were designated PCMHs or on their way to being designated by meeting certain criteria: electronic health record, patient portal, team-based care, practice redesign, care management, and identification of high-risk patients. See Study Protocol (Appendix 2) for detailed patient and primary care practice inclusion criteria.

Patient Enrollment

Study staff screened participants from a daily automated list of patients admitted the day before, using medical records to determine eligibility, which was then confirmed by the patient’s nurse. Exclusion criteria included likely discharge to a location other than home, being in police custody, lack of a telephone, being homeless, previous enrollment in the study, and being unable to communicate in English or Spanish. Allocation to study arm was concealed until the patient or proxy provided informed written consent. The research assistant administered questionnaires to all study subjects to assess potential confounders and functional status 1 month prior to admission (Medical Outcomes Study 12-Item Short Form Health Survey [SF-12]).10 Patients were recruited between March 2013 and October 2015.

Intervention

The intervention was based on a conceptual model of an ideal discharge11 that we developed based on work by Naylor et al,12 work by Coleman and Berenson,3 best practices in medication reconciliation and information transfer according to our own research,13^–15 the best examples of interventions to improve the discharge process,12,16,17 and a systematic review of discharge interventions.18 Some of the factors necessary for an ideal care transition include complete, organized, and timely documentation of the patient’s hospital course and postdischarge plan; effective discharge planning; coordination of care among the patient’s providers; methods to ensure medication safety; advanced care planning in appropriate patients; and education and “coaching” of patients and their caregivers so they learn how to manage their conditions. The final multifaceted intervention addressed each component of the ideal discharge and included inpatient and outpatient components (Table 1 and Table 1 of Appendix 1).

TABLE 1.

Components of Multifaceted Intervention to Achieve Ideal Hospital Discharge

Inpatient pharmacist–led medication reconciliation and patient counseling Patient education and coaching from an inpatient discharge advocate (either a registered nurse or nurse practitioner) and primary care responsible outpatient clinician (either a licensed practical nurse, registered nurse, or physician assistant) Communication between discharge advocate and responsible outpatient clinician Structured postdischarge phone call by the responsible outpatient clinician Structured postdischarge follow-up visit in the primary care practice Health information technology enhancements Optional programs for selected patients ○ Structured visiting nurse services ○ Home pharmacist program ○ Referral to palliative care

Patient and Public Involvement in Research

As with all PCORI-funded studies, this study involved a patient-family advisory council (PFAC). Our PFAC included six recently hospitalized patients or caregivers of recently hospitalized patients. The PFAC participated in monthly meetings throughout the study period. They helped inform the research questions, including confirmation that the endpoints were patient centered, and provided valuable input for the design of the intervention and the patient-facing components of the data collection instruments. They also interviewed several patient participants in the study regarding their experiences with the intervention. Lastly, they helped develop plans for dissemination of study results to the public.19

We also formed a steering committee consisting of physician, nursing, pharmacy, information technology, and administrative leadership representing primary care, inpatient care, and transitional care at both hospitals and Partners Healthcare. PFAC members took turns participating in quarterly steering committee meetings.

Evolution of the Intervention and Implementation

The intervention was iteratively refined during the course of the study in response to input from the PFAC, steering committee, and members of the intervention team; cases of adverse events and readmissions from patients despite being in the intervention arm; exit interviews of patients who had recently completed the intervention; and informal feedback from inpatient and outpatient clinicians. For example, we learned that the more complicated a patient’s conditions are, the sooner the clinical team wanted them to be seen after discharge. However, these patients were also less likely to feel well enough to keep that appointment. Therefore, the timing of follow-up of appointments needed to be a negotiation among the inpatient team, the patient, any caregivers, and the outpatient provider. PFAC members also emphasized that patients wanted one person to trust and to be the “point person” during a complicated transition such as hospital discharge.

At the same time, the intervention components evolved because of factors outside our control (eg, resource limitations). In keeping with the real-world nature of the research, the aim was for the intervention to be internally supported because incentives were theoretically more aligned with improvement of care transitions under the ACO model. By design, the PCORI contract only paid for limited parts of the intervention, such as a nurse practitioner to act as the discharge advocate at one hospital, overtime costs of inpatient pharmacists, and project manager time to facilitate inpatient-outpatient provider communication. (See Table 1 of Appendix 1 for details about the modifications to the intervention.)

Lastly, in keeping with PCORI’s methodology standards for studies of complex interventions,20 we strove to standardize the intervention by function across hospitals, units, and practices, while still allowing for local adaptation in the form. In other words, rather than specifying exactly how a task (eg, medication counseling) needed to be performed, the study design offered sites flexibility in how they implemented the task given their available personnel and institutional culture.

Intervention Fidelity

To determine the extent to which each patient in the intervention arm received each intervention component, a project manager unblinded to treatment arm reviewed the electronic medical record for documentation of each component implemented by providers (eg, inpatient pharmacists, outpatient nurses). Because each intervention component produced documentation, this provided an accurate assessment of intervention fidelity, ie, the extent to which the intervention was implemented as intended.

Outcome Assessment

Postdischarge Follow-up

Based on previous studies,2,21 a trained research assistant attempted to contact all study subjects 30 days (±5 days) after discharge and administered a questionnaire to identify any new or worsening symptoms since discharge, any healthcare use since discharge, and functional status in the previous week. Follow-up questions used branching logic to determine the relationship of any new or worsening symptoms to medications or other aspects of medical management. Research assistants followed up any positive responses with directed medical record review for objective findings, diagnoses, treatments, and responses. If patients could not be reached after five attempts, the research assistant instead conducted a thorough review of the outpatient medical record alone for provider reports of any new or worsening symptoms noted during follow-up within the 30-day postdischarge period. Research assistants also reviewed laboratory test results in all patients for evidence of postdischarge renal failure, elevated liver function tests, or new/worsening anemia.

Hospital Readmissions

We measured nonelective hospital readmissions within 30 days of discharge using a combination of administrative data for hospitalizations within the ACO network plus patient report during the 30-day phone call for all other readmissions.22

Adjudication of Outcomes

Adverse events and preventable adverse events

All cases of new or worsening symptoms or signs, along with all supporting documentation, were then presented to teams of two trained blinded physician adjudicators through application of methods established in previous studies.4,21 Each of the two adjudicators independently reviewed the information, along with the medical record, and completed a standardized form to confirm or deny the presence of any adverse events (ie, patient injury due to medical management) and to classify the type of event (eg, adverse drug event, hospital-acquired infection, procedural complication, diagnostic or management error), the severity and duration of the event, and whether the event was preventable or ameliorable. The two adjudicators then met to resolve any differences in their findings and come to consensus.

Preventable readmissions

If patients were readmitted to either study hospital, we conducted an evaluation, based on previous studies,23 to determine if and how the readmission could have been prevented including (a) a standardized patient and caregiver interview to identify possible problems with the transitions process and (b) an email questionnaire to the patient’s PCP and the inpatient teams who cared for the patient during the index admission and readmission regarding possible deficiencies with the transitions process. As with adverse event adjudications, two physician adjudicators worked independently to classify the preventability of the readmission and then met to come to consensus. Conflicts were resolved by a third adjudicator.

Analysis Plan

To evaluate the effects of the intervention on the primary outcome, the number of postdischarge adverse events per patient, we used multivariable Poisson regression, with study arm as the main predictor. A similar approach was used to evaluate the number of new or worsening postdischarge signs or symptoms and the number of preventable adverse events per patient. We used an intention-to-treat analysis: If a practice did not start the intervention when they were scheduled to, based on our randomization, we counted all patients in that practice admitted after that point as intervention patients. We adjusted for patient demographics, clinical characteristics, month, inpatient unit, and primary care practice as fixed effects. We clustered by PCP using general linear models. Intervention effects were expressed as both unadjusted and adjusted incidence rate ratios (IRRs). We also conducted a limited number of subgroup analyses, determined a priori, to determine whether the intervention was more effective in certain patient populations; we used interaction terms (intervention × subgroup) to determine the statistical significance of any effect modification.

To evaluate the effects of the intervention on nonelective readmissions and preventable readmissions, we used a similar approach, using multivariable logistic regression. Postdischarge functional status, adjusted for status prior to admission, was analyzed using multivariable linear regression and random effects by primary care practice. The general linear mixed model (GLIMMIX) procedure in the SAS 9.3 statistical package (SAS Institute) was used to carry out all analyses.

Power and Sample Size

We assumed a baseline rate of postdischarge adverse events of 0.30 per patient.21 We conservatively assumed an effect size of a change from 0.30 in the control group to 0.23 in the intervention group (a relative reduction of 22%, which was based on studies of preventability rates23 and close to the minimum clinically important difference). Based on prior studies,4,22 we assumed an intraclass correlation coefficient of 0.01 with an average cluster size of seven patients per PCP. Assuming a 10% loss to follow-up rate and an alpha of 0.05, we targeted a sample size of 1,800 patients to achieve 80% power, with one-third of the patients in the usual care arm and two-thirds in the intervention arm.

RESULTS

We enrolled 18 PCMH primary care practices to participate in the study, including 8 from Hospital A (out of 13 approached), 8 from Hospital B (out of 11), and 2 from other ACO practices (out of 9) (plus two pilot practices). Reasons for not participating included not having dedicated personnel to play the role of the responsible outpatient clinician, undergoing recent turn-over in practice leadership, and not having enough patients admitted to the two hospitals. One practice only enrolled 5 patients in the study and withdrew from participation, which left 17 practices.

Study Patients

We enrolled 1,679 patients (Figure 1). Reasons for nonenrollment included being unable to complete the screen prior to discharge, not meeting inclusion criteria or meeting exclusion criteria, being assigned to a pilot practice, and declining informed written consent. The baseline characteristics of enrolled patients are presented in Table 2. Differences between the two study arms were small. About 47% of the cohort was not reachable by phone after five attempts for the 30-day phone call, but only 69 (4.1%) were truly lost to follow-up because they were unreachable by phone and had no documentation in the electronic medical record in the 30-days after discharge.

FIG 1.

CONSORT Flow Diagram Illustrating Flow of Patient Subjects Through the Trial

TABLE 2.

Patient Characteristics

Characteristic	Usual care N = 679	Intervention N = 978
Age, No. (%), y
18-49	157 (23)	209 (21)
50-59	152 (22)	229 (23)
60-69	168 (25)	258 (26)
70 or greater	202 (30)	278 (28)
Female, No. (%)	364 (54)	540 (55)
Service, No. (%)
Medicine	403 (59)	556 (57)
Surgery	276 (41)	422 (43)
Elixhauser Comorbidity Index scorea, No. (%)
<1	219 (32)	331 (34)
1-5	159 (23)	232 (24)
6-10	105 (15)	155 (16)
≥11	196 (29)	260 (27)
Language, No. (%)
English	637 (94)	947 (97)
Spanish	42 (6)	31 (3)
Race/ethnicity, No. (%)
White non-Hispanic	500 (74)	728 (74)
Dementiab, No. (%)	66 (10)	96 (10)
Health literacy,c No. (%)
Adequate	463 (68)	665 (68)
Inadequate or marginal	59 (9)	91 (9)
NA	157 (23)	222 (23)
Caregiver, No. (%)
No	486 (72)	746 (76)
Yes, helps with IADLs only	128 (19)	171 (17)
Yes, helps with basic ADLs	64 (9)	61 (6)
ED visits in past 6 mo, No. (%)
0	433 (64)	639 (65)
1	90 (13)	124 (13)
≥2	156 (23)	215 (22)
HOSPITAL Score,d median (IQR)	3.0 (2.0-4.0)	3.0 (2.0-4.0)
Baseline SF-12 score (1 mo prior to admission), median (IQR)	45.0 (20.0-59.5)	46.1 (20.0-59.5)
Income by ZIP Code, median (IQR), US $	71,809 (51,723-88,829)	72,461 (53,795-88,829)
Education level, No. (%)
Less than HS graduate	53 (8)	44 (4)
HS graduate or GED	155 (23)	241 (25)
At least some college	361 (53)	510 (52)
Other/not reported	110 (16)	183 (19)
Marital Status, No. (%)
Single	177 (26)	292 (30)
Married or domestic partner	369 (54)	514 (53)
Separated, widowed, or divorced	114 (17)	138 (14)

^avan Walraven C, Austin PC, Jennings A, Quan H, Forster AJ. A modification of the Elixhauser comorbidity measures into a point system for hospital death using adminitrative data. Med Care. 2009;47(6):626-633.

^bPatients’ dementia status was assessed using Mini-Cog© score (<3 considered consistent with dementia).

^cPatients’ health literacy was assessed using the Short-Test of Functional Health Literacy in Adults (s-TOFHLA) score (≥23: adequate, 17-22: marginal, <17: inadequate), or NA: did not attempt for various reasons (eg, visual impairment, patient declined to take test, patient unable to take test because of medical condition/surgery).

^dHOSPITAL Score can help predict risk of 30-day potentially avoidable readmissions.30,31

Abbreviations: ADLs, activities of daily living; ED, emergency department; GED, general education development certification; HS, high school; IADLs, instrumental activities of daily living; IQR, interquartile range; SF-12, Medical Outcomes Study 12-Item Short Form Health Survey.

Intervention Fidelity

The majority of patients did not receive most intervention components, even those components that were supposed to be delivered to all intervention patients (Table 3). A minority of patients were referred to visiting nurse services and to the home pharmacy program. However, 855 patients (87%) in the intervention arm received at least one intervention component.

TABLE 3.

Intervention Fidelity

Intervention component	Hospital A, No. (%) (N = 531)	Hospital B, No. (%) (N = 447)
Inpatient pharmacist counseling	195 (37)	152 (34)
DA note in EMR for Hospital A; reminder email for DA-ROC communication at Hospital Ba	208 (39)	385 (86)
Postdischarge phone call with patient from ROC within 14 business days	206 (39)	235 (53)
Postdischarge clinic visit within 14 business days	156 (29)	197 (44)
Visiting nurse services
Network VNA (structured intervention)	88 (17)	87 (19)
Nonnetwork VNA	59 (11)	93 (21)
Not referred	384 (72)	267 (60)
Outpatient pharmacist visit
Patient accepted service	55 (10)	30 (7)
Patient declined service	48 (9)	27 (6)
Not referred	428 (81)	390 (87)

^aEmail sent by project manager to both DA and ROC the day after admission to open up lines of communication and remind them to exchange information with each other about the patient’s history of self-management, the current hospital course, and possible needs for postdischarge care.

Abbreviations: DA, discharge advocate; EMR, electronic medical record; ROC, responsible outpatient clinician; VNA, Visiting Nurses Association representative.

Outcome Measures

The intervention was associated with a statistically significant reduction in several of the outcomes of interest, including the primary outcome, number of postdischarge adverse events (45% reduction), and new or worsening postdischarge signs or symptoms (22% reduction), as well as preventable postdischarge adverse events (58% reduction) (Table 4). There was a nonsignificant difference in functional status. There was no significant effect on total nonelective or on preventable readmission rates. When analyzed by type of adverse event, the intervention was associated with a reduction in adverse drug events and in procedural complications (Table 2 of Appendix 1). Of note, there was no significant difference in the proportion of patients with at least one adverse event whether the outcome was determined by phone call and medical record review (49%) or medical record review alone (51%) (P = .48).

TABLE 4.

Outcomes

Outcome	Usual care N = 679	Intervention N = 978	Unadjusted rate (95% CI), P value	Adjusted rate (95% CI), P value
Postdischarge adverse events, rate per 100 patients (95% CI)	23 (20-27)	18 (15-21)	IRR, 0.76 (0.61-0.95), .01	IRR, 0.55 (0.35-0.84), .006a
New or worsening postdischarge signs or symptoms, rate per 100 patients (95% CI)	92 (85-97)	90 (86-101)	IRR, 0.98 (0.88-1.08), .64	IRR, 0.78 (0.64-0.95), .01a
Preventable postdischarge adverse events, rate per 100 patients (95% CI)	10 (8-13)	6 (5-8)	IRR, 0.62 (0.44-0.87), .005	IRR, 0.42 (0.21-0.84), .01a
30-day postdischarge SF-12 Score, mean (SD)	46.14 (8.11)	47.25 (7.85)	Difference, 1.11 (−0.02 to 2.26), .05	Difference, 0.08 (−1.29 to 1.45), .91b
Nonelective 30-day readmission, No. (%)	78 (11.5)	107 (10.9)	OR, 0.95 (0.69-0.29), .73	OR, 1.08 (0.64-1.85), .77a
Preventable readmissions, No. (%)	12 (1.8)	22 (2.2)	OR, 1.31 (0.64-2.67), .46	OR, 1.86 (0.61-5.62), .27a

^aRate was adjusted for study month, Elixhauser Comorbidity Index score, age, sex, language, race/ethnicity, cognitive status, health literacy, functional status 1 month prior to admission, caregiver status, median income by ZIP Code, emergency department visits in the previous 6 months, HOSPITAL score, inpatient unit, and primary care practice.

^bRate was adjusted for study month, Elixhauser Comorbidity Index score, age, sex, language, race/ethnicity, cognitive status, health literacy, functional status 1 month prior to admission, caregiver status, median income by ZIP Code, emergency department visits in the previous 6 months, HOSPITAL score. Patients were clustered by inpatient unit, and primary care practice was treated as a random effect.

Abbreviations: IRR, incidence rate ratio; OR, odds ratio; SF-12, Medical Outcomes Study 12-Item Short Form Health Survey.

In subgroup analyses, there was no evidence of effect modification by service, hospital, patient age, readmission risk, health literacy, or comorbidity score (Table 3 of Appendix 1). Table 4 of Appendix 1 provides examples of postdischarge adverse events seen in the usual care arm that might have been prevented in the intervention.

DISCUSSION

This intervention was associated with a reduction in postdischarge adverse events. The relative improvement in each outcome aligned with the hypothesized sensitivity to change: the smallest improvement was seen in new or worsening signs or symptoms, followed by postdischarge adverse events and then by preventable postdischarge adverse events. The intervention was not associated with a difference in readmissions. The lack of effect on hospital readmissions may have been caused by the low proportion of readmissions that are preventable, as well as low intervention fidelity and lack of resources to implement facets such as postdischarge coaching, an evidence-based intervention that was never adopted.16,23 One lesson of this study is that it may be easier to reduce postdischarge injury (still an important outcome) than readmissions.

Putting this study in context, we should note that the literature on interventions to improve care transitions is mixed.18 While there are several reports of successful interventions, there are many reports of unsuccessful ones, often using similar components. Success is often the result of adequate resources and attention to myriad details regarding implementation.24 The intervention in our study likely contributed to improvements in patient and caregiver engagement in the hospital, enhancements of communication between inpatient and outpatient clinicians, and implementation of pharmacist-led interventions to improve medication safety. Regarding the latter, several prior studies have shown the benefits of pharmacist interventions in decreasing postdischarge adverse drug events.4,25,26 Therefore, even an intervention with incomplete intervention fidelity can reduce postdischarge adverse events, especially because adverse drug events make up the majority of adverse events.1,2,21

Perhaps the biggest lesson we learned was regarding the limitations of the hospital-led ACO model to incentivize sufficient up-front investments in transitional care interventions. By design, we chose a real-world approach in which interventions were integrated with existing ACO efforts, which were paid for internally by the institution. As a result, many of the interventions had to be scaled back because of resource constraints. The ACO model theoretically incentivizes more integrated care, but this may not always be true in practice. Emerging evidence suggests that physician group–led ACOs are associated with lower costs and use compared with hospital-led ACOs, likely because of more aligned incentives in physician group– led ACOs to reduce use of inpatient care.27,28

An unresolved question is whether the ideal implementation approach is to protect the time of existing clinical personnel to carry out transitional care tasks or to hire external personnel to do these tasks. We purposely spread the intervention over several clinician types to minimize the additional burden on any one of them, minimize additional costs, and play to each clinician’s expertise, but in retrospect, this may not have been the right approach. By providing additional personnel with dedicated time, interest, and training in care transitions, the intervention may be delivered with higher quantitative and qualitative fidelity, and it could create a single point of contact for patients, which was considered highly desirable by our PFAC.

This study has several limitations. A large proportion of patients (44%) were unavailable for postdischarge phone calls. However, we were able to perform medical record review for worsening signs (eg, lab abnormalities) and symptoms (as reported by patients’ providers) in the postdischarge period and adjudicate them for adverse events for all but 69 of these patients. Because all these patients had ACO-affiliated PCPs, we would expect most of their utilization to have been within the system and, therefore, to be present in the medical record. The proportion of patients with at least one adverse event did not vary by the method of follow-up, which suggests that this issue is an unlikely source of bias. Assessment of readmission was imperfect because we do not have statewide or national data. However, our combination of administrative data for Partners readmissions plus self-report for non-Partners readmission has been shown to be fairly complete in previous studies.29 Adjudicators could not be fully blinded to intervention status due to the lack of blinding of admission date. We did not calculate a kappa value for interrater reliability of individual assessments of adverse events; rather, coming to consensus among the two adjudicators was part of the process. In only a handful of cases was a third adjudicator required. Lastly, this study was conducted at two academic medical centers and their affiliated primary care clinics, which potentially limits generalizability; however, the results are likely generalizable to other ACOs that include major academic medical centers.

CONCLUSION

In conclusion, in this real-world clinical trial, we designed, implemented, and iteratively refined a multifaceted intervention to improve care transitions within a hospital-based academic ACO. Evolution of the intervention components was the result of stakeholder input, experience with the intervention, and ACO resource constraints. The intervention reduced postdischarge adverse events. However, across the ACO network, intervention fidelity was low, and this may have contributed to the lack of effect on readmission rates. ACOs that implement interventions without hiring new personnel or protecting the time of existing personnel to conduct transitional tasks are likely to face the same challenges of low fidelity.