The Effect of AIDS Clinical Trials Group Protocol 5164 on the Time From Pneumocystis jirovecii Pneumonia Diagnosis to Antiretroviral Initiation in Routine Clinical Practice: A Case Study of Diffusion, Dissemination, and Implementation

Although early initiation of antiretroviral therapy in HIV-infected patients after a diagnosis of Pneumocystis pneumonia increased after ACTG 5164 (7.4%–50.0%), a subsequent implementation and dissemination initiative optimized uptake of early antiretroviral therapy as clinically routine (50.0%–80.3%).

Abstract

Background. Diffusion, dissemination, and implementation of scientific evidence into routine clinical practice is not well understood. The Adult AIDS Clinical Trials Group (ACTG) Protocol 5164 showed that early antiretroviral therapy (ART; ie, within 14 days) after diagnosis of an opportunistic infection improved clinical outcomes, compared with later initiation. Subsequently, the San Francisco General Hospital (SFGH) HIV/AIDS Service performed the SFGH 5164 Initiative to disseminate and implement the findings of ACTG 5164.

Methods. We evaluated patients who received a diagnosis of Pneumocystis jirovecii pneumonia (PCP) from 1 January 2001 through 30 March 2011. Survival analyses were used to assess changes in the time to initiation of ART after PCP, and logistic regression was used to evaluate changes in the odds of early ART (ie, within 14 days) because of ACTG 5164 and SFGH 5164 Initiative.

Results. Among 162 patients, the adjusted hazard of ART initiation increased by 3.05 (95% confidence interval [CI], 1.86–5.02) after ACTG 5164 and by 4.89 (95% CI, 2.76–8.67) after the SFGH Initiative, compared with before ACTG 5164. When compared with before ACTG 5164, the proportion of patients who received ART within the 14 days after PCP diagnosis increased from 7.4% to 50.0% (P < .001) after ACTG 5164 and from 50.0% to 83.0% (P = .02) after the SFGH 5164 Initiative.

Conclusions. Diffusion of findings from of a randomized trial changed practice at an academic medical center, but dissemination and implementation efforts were required to establish early ART at acceptable levels. Early ART initiation can be achieved in real-world patient populations.

Implementation and dissemination sciences are motivated by the recognition that the translation of clinical evidence into everyday practice is suboptimal in many areas of health care, including human immunodeficiency virus (HIV) medicine [1, 2]. This process of translation is based on 3 overlapping but distinct processes. First, diffusion is the passive transfer of knowledge through existing channels, such as conferences, publications, and other established forms of scientific communication [3, 4]. Dissemination is more active than diffusion and is the managed and strategic spread of information [5]. Implementation involves processes to adapt scientific knowledge to local circumstances and address barriers to execution [3, 5]. Scientific evaluation of these 3 processes in translation has emerged as a national priority [6, 7] and as an important basis for improving effectiveness of health care services.

Although HIV medicine has benefited from a robust number of randomized trials and a strong evidence base, to date, studies evaluating the translation of this evidence into routine practice are relatively few, and those that exist are cross-sectional and derived mainly from administrative data [5]. Cross-sectional studies (such as completeness of use of sulfamathoxazole-trimethoprim prophylaxis in patients with CD4 cell counts <200 cells/mL) are unable to assess changes in practice over time [2]. However, the rate and extent of this change is a crucial dimension of understanding the diffusion, dissemination, and implementation of scientific knowledge. Studies using administrative data are not linked to local clinical practice initiatives or patterns [8]. Without this contextual knowledge, inferences about the mechanisms of translation are limited. Understanding of the anatomy of how evidence becomes practice requires longitudinal analyses in a specific clinical practice as new scientific evidence emerges.

The AIDS Clinical Trials Group (ACTG) Protocol 5164 was a randomized trial that found that early antiretroviral therapy (ART) in the context of an acute opportunistic infection (OI) reduced AIDS progression and mortality by 50% [9]. The study was first presented in February 2008 at the 14th Annual Conference on Retroviruses and Opportunistic Infections and then published in May 2009. In October 2009, the leadership of the University of California at San Francisco/San Francisco General Hospital (SFGH) HIV/AIDS Division formally adopted a recommendation of early ART after an OI diagnoses and performed the SFGH 5164 Initiative. This initiative was composed of (1) establishing new local guidelines for starting ART in the context of OI in both inpatient and outpatient settings; (2) disseminating education materials for nonspecialty providers; (3) meeting with opinion leaders in the hospital and community and with medical and family practice house-staff; (4) acceleration of inpatient insurance application processes to cover ART; and (5) liaison with an existing inpatient HIV/AIDS linkage team to identify patients with OIs and to reinforce early ART messaging to inpatient providers.

This analysis sought to understand the effects of ACTG 5164 and the SFGH 5164 Initiative on ART initiation in a clinic-based cohort of HIV-infected patients with Pneumocystis jirovecii pneumonia (PCP) as a case study of diffusion, dissemination, and implementation. We considered changes in ART initiation after ACTG 5164 and before the SFGH 5164 Initiative to represent the process of diffusion, whereas changes in practice after the SFGH 5164 Initiative represent the effect of dissemination (ie, strategic distribution of information) and implementation (ie, adaptation of local conditions to facilitate early ART). We restricted the analysis to patients with PCP, because patients with this OI comprised the largest subgroup in the ACTG 5164 trial.

METHODS

Patients

We evaluated adult, HIV-infected patients who presented for ≥2 primary care visits to Ward 86, the outpatient HIV clinic at San Francisco General Hospital, from 1 January 2001 through 30 March 2011, with a diagnosis of PCP and who were not receiving ART at the time of PCP diagnosis (as defined by no ART in the preceding 90 days). Patients were observed until ART initiation or were censored at their last clinic visit. Sociodemographic characteristics (eg, age, sex, race, and risk factor for HIV acquisition) and clinical characteristics (eg, presenting CD4 cell counts and HIV loads, ART start date, diagnosis dates of PCP infection, primary HIV care visit dates, and deaths) were obtained in the course of routine clinical care and recorded in the clinic’s electronic medical records system. Both clinical and microbiologic diagnoses of PCP were included. ART initiation was defined by both a prescription for antiretroviral medications and a note in the patient chart documenting subsequent use of ART for at least 1 week. All dates of PCP diagnosis and ART initiation were confirmed by manual chart review.

AIDS Clinical Trials Group 5164 and San Francisco General Hospital 5164 Initiative

ACTG 5164 was presented at Conference on Retroviruses and Opportunistic Infections on 8 February 2008, and we considered this date as the first temporal threshold across which we evaluated changes in practice potentially because of diffusion of findings. From 1 October 2009 through 30 December 2009, the SFGH 5164 Initiative was conducted to disseminate the results of ACTG 5164 to key clinical staff at our hospital and in the community and also to adapt program conditions to facilitate early ART initiation. We considered the conclusion of this initiative (30 December 2009) as the second temporal threshold across which to evaluate changes in practice potentially because of dissemination and implementation.

The SFGH 5164 Initiative was composed of a number of activities to transfer knowledge. First, providers in the HIV/AIDS Division participated in drafting local guidelines to summarize the findings of ACTG 5164 and provide practical answers to anticipated frequently asked questions. For example, the local guidelines provided recommendations on selection of antiretroviral medications when initiating ART before the results of genotypic assays for HIV drug resistance become available. Second, the document was disseminated to key providers outside the HIV/AIDS Division. To this end, the director of the HIV/AIDS Consult Service presented this document to internal medicine and family medicine staff at resident teaching sessions. In addition, individual meetings with leadership of inpatient services were held to directly communicate with local opinion leaders. Meetings were also held with providers at community clinics outside SFGH but who admit patients to SFGH.

The SFGH 5164 Initiative also involved adaptation of local administrative and clinical procedures to enable early ART after diagnosis of an OI. The Positive Health Access to Services and Treatment (PHAST) program is a federally funded multidisciplinary team that assists HIV-infected hospitalized patients who have psychiatric diagnoses, substance-dependence, and/or housing instability to connect to outpatient care after hospital discharge. As part of the SFGH 5164 Initiative, the PHAST team had 2 additional responsibilities. First, PHAST began to initiate applications for the California AIDS Drug Assistance Program to cover the costs of outpatient antiretroviral medications before patient discharge—a task previously usually initiated by outpatient social services. Second, to sustain the messaging about early ART initiation, the PHAST team began to maintain surveillance of inpatients eligible for ART and notified the inpatient HIV consult service of these patients.

Analysis

We used 2 analytic approaches to evaluate the effect of ACTG 5164 on ART initiation in patients with PCP. Analyses that consider the rate of initiation as a continuous outcome (ie, days from PCP diagnosis to ART initiation) are potentially able to detect small but significant changes in practice that represent an effect of interest. Analyses that consider early ART as a binary outcome (ie, receipt of ART within 14 days or not), however, provide a more direct measure of whether the findings of ACTG 5164 were actually met. We therefore used both survival analyses (ie, Kaplan–Meier and Cox proportional hazards) to evaluate changes in the time to ART initiation after PCP diagnosis by period and logistic regression to assess changes in the odds of early ART (ie, receipt within 14 days). For Kaplan–Meier estimates, we used the log rank test to evaluate whether differences in ART initiation were statistically significant. In both cases, we conducted single-predictor analyses and included factors significant with a P value <.10 in the multivariable model. In addition, we examined the occurrence of virologic suppression (defined as HIV RNA level <400 copies/mm³) during each of the periods of interest to evaluate whether earlier ART initiation was associated with poorer virologic response, potentially because of inadequate preparation about adherence. This study was approved by the Committee on Human Research at UCSF as an analysis of existing data collected in the context of routine clinical care (informed consent was therefore waived). All analyses were performed using Stata, version 11.0 (StataCorp).

RESULTS

From 1 January 2001 through 30 March 2011, 162 patients with PCP and not receiving ART were seen at the Ward 86 HIV practice at SFGH (Table 1). The median age was 41 years (interquartile range, 35–46 years), and 148 (91.4%) were men. Seventy-four patients (45.7%) were white, 36 (22.2%) black, 16 (9.9%) Asian/Pacific Islander, 24 (15.7%) other, and 10 (6.2%) unknown race. Thirty-eight patients (23.4%) were Hispanic. The median CD4 cell count at the time of PCP diagnosis was 49 cells/μL (interquartile range, 15–105 cells/μL). One hundred eighteen patients received a diagnosis of PCP before ACTG 5164, 26 between ACTG 5164 and the SFGH 5164 Initiative, and 18 patients after the SFGH 5164 Initiative.

Table 1.

Patient Characteristics

Factor	No. (n = 162)	Percentage or IQR
Age at PCP diagnosis, years, median (IQR)	41	(35–46)
Male sex, n (%)	148	91.4
Race, n (%)
Asian/Pacific Islander	16	9.9
Black	36	22.2
Other	24	15.7
White	74	45.7
Unknown	10	6.2
Hispanic ethnicity	38	23.4
CD4 cell count at diagnosis, cells/μL, median (IQR)	49	15–105
HIV risk factor
Men who have sex with men	91	56.2
Heterosexual	37	22.8
Injection drug user	21	13.0
Other	13	8.6
HIV RNA level at PCP, log10 copies/mm3, median (IQR)	4.84	3.58–5.33
Interval
Before clinical trial ACTG 5164	118	72.8
After clinical trial ACTG 5164, before SFGH 5164 Initiative	26	16.1
After SFGH 5164 Initiative	18	11.1

Abbreviations: HIV, human immunodeficiency virus; IQR, interquartile range; PCP, Pneumocystis jirovecii pneumonia; SFGH, San Francisco General Hospital.

Before ACTG 5164, Kaplan–Meier estimates (Figure 1) of the cumulative incidence of ART initiation at 30, 60, and 90 days after PCP diagnosis were 20.3% (95% confidence interval [CI], 14.2%–29.3%), 45.8% (95% CI, 37.3%–55.2%), and 70.9% (95% CI, 62.4%–78.8%), respectively. After ACTG 5164 but before the SFGH 5164 Initiative, estimates were 79.2% (95% CI, 61.5%–92.4%), 91.7% (95% CI, 76.7%–98.5%), and 91.7% (95% CI, 76.7%–98.5%), respectively. After the SFGH Initiative, estimates of ART initiation at these times were 82.4% (95% CI, 61.7%–95.6%), 88.2% (95% CI, 68.8%–98.0%), and 94.1% (95% CI, 76.5%–100%). The unadjusted rate ratio in the period between ACTG 5164 and the SFGH Initiative, compared with before ACTG 5164, was 2.81 (95% CI, 1.77–4.46) (Table 2). The unadjusted rate ratio of ART initiation after the SFGH Initiative, compared with before ACTG 5164, was 4.27 (95% CI, 2.53–7.22). When the period after the SFGH 5164 Initiative was compared with the interval between ACTG 5164 and the SFGH 5164 Initiative, the rate ratio of ART initiation was 1.52 (95% CI, 0.81–2.86). In a multivariable Cox model, ACTG 5164 and the SFGH 5164 Initiative were the only statistically significant factors associated with ART initiation. After adjusting for HIV risk factor and race and compared with the period before ACTG 5164, the rate ratio for ART initiation was 3.05 (95% CI, 1.86–5.02) between ACTG 5164 and the SFGH 5164 Initiative and 4.89 (95% CI, 2.76–8.67) after the SFGH 5164 Initiative. The rate of ART initiation after the SFGH 5164 Initiative, compared with the period between ACTG 5164 and the SFGH Initiative was 1.60 (95% CI, 0.82–3.11).

Figure 1.

Time to antiretroviral therapy initiation after diagnosis of Pneumocystis jirovecii pneumonia (PCP).

Table 2.

Multivariable Cox Proportional Hazards Regression of the Effect of AIDS Clinical Trials Group 5164 on Antiretroviral Therapy Initiation

	Single predictor			Multivariable
Factor	HR	95% CI	P value	HR	95% CI	P value
Age (per 10 years)	0.99	0.82–1.20	.95
Male sex	1.06	0.59–1.72	.98
Race
White	ref			ref
Asian	1.24	0.72–2.14	.44	1.32	0.75–2.34	.33
Black	0.67	0.43–1.02	.06	0.69	0.42–1.15	.15
Other	1.42	0.89–2.30	.14	1.00	2.77–8.67	.71
Hispanic ethnicity	1.28	0.88–1.88	.20
CD4 level at PCP diagnosis (per 50 cells/μL)	0.93	0.85–1.02	.15
HIV risk factor
MSM	ref			ref
Heterosexual	0.93	0.63–1.38	.72	0.86	0.56–1.33	.50
IDU	0.60	0.35–1.00	.05	0.65	0.39–1.08	.09
Other	0.40	0.20–0.80	.01	0.62	0.32–1.21	.16
Interval
Before ACTG 5164	ref			ref
Between ACTG 5164 and the SFGH 5164 Initiative	2.81	1.77–4.46	<.01	3.05	1.86–5.02	<.01
After the SFGH 5164 Initiative (as compared with before ACTG 5164)	4.27	2.53–7.22	<.01	4.89	2.76–8.67	<.01
After the SFGH 5164 Initiative (as compared with between ACTG 5164 and the SFGH 5164 Initiative)	1.52	0.81–2.86	.20	1.60	0.82–3.11	.16

Abbreviations: ACTG, AIDS Clinical Trials Group; CI, confidence interval; HIV, human immunodeficiency virus; HR, hazard ratio; IDU, injection drug user; MSM, men who have sex with men; OR, odds ratio; SFGH, San Francisco General Hospital.

The probability of early ART increased over the 3 calendar periods (Figure 2). Before ACTG 5164, 7.4% of patients started ART within 14 days after PCP diagnosis. Between ACTG 5164 and the SFGH 5164 Initiative, 50.0% of patients started early. After the SFGH Initiative, 83.0% of patients started ART early (P < .001 when comparing this period with the period before ACTG 5164 and P = .02 when comparing this period with the interval between ACTG 5164 and the SFGH 5164 Initiative). In unadjusted analyses (Table 3), age, HIV risk factor, and the dates of ACTG 5164 and the SFGH 5164 Initiative were associated with early ART initiation. Multivariable analyses (adjusting for age, race, and HIV risk factor) found that, compared with the period before ACTG 5164, both the interval between ACTG 5164 and the SFGH 5164 Initiative (OR, 15.49; 95% CI, 4.87–49.27) and the period after the SFGH 5164 Initiative (OR, 124.71; 95% CI, 20.9–734.27) were associated with early ART initiation. The odds of early ART initiation after the SFGH 5164 initiative, compared with the period between ACTG 5164 and the SFGH 5164 Initiative, was 8.04 (95% CI, 1.34–48.3).

Figure 2.

Proportion of patients per calendar year who initiated antiretroviral therapy (ART) early (defined as <14 days after Pneumocystis jirovecii pneumonia diagnosis). The total number of patients per year is shown across the top of the figure.

Table 3.

Logistic Regression Model of the Effect of AIDS Clinical Trials Group (ACTG) 5164 on Early Antiretroviral Therapy (ART) Initiation (Defined as ART Initiation ≤14 Days After Pneumocystis jirovecii Pneumonia Diagnosis)

	Single predictor			Multivariable
Factor	OR	95% CI	P value	OR	95% CI	P value
Age (per 10 years)	1.68	1.05–2.70	.03	1.04	0.98–1.11	.17
Male sex	4.9	0.63–38.32	.13
Race
White	ref
Asian	1.12	0.31–3.92	.86
Black	0.54	0.18–1.61	.86
Other	1.77	0.67–4.69	.24
Hispanic ethnicity	1.17	0.49–2.78	.72
CD4 cell count at diagnosis (per 50 cells/μL)	0.93	0.75–1.14	.48
HIV risk factor
MSM	ref			ref
Heterosexual	0.37	0.13–1.05	.06	0.37	0.10–1.42	.15
IDU	0.26	0.06–1.21	.09	0.08	0.01–0.78	.03
Other	0.18	0.02–1.46	.11	0.12	0.01–2.04	.14
Interval
Before ACTG 5164	ref			ref
Between ACTG 5164 and the SFGH 5164 Initiative	15.85	5.37–46.87	<.01	15.49	4.87–49.27	<.01
After SFGH Initiative (as compared with before ACTG 5164)	79.29	18.49–340.04	<.01	124.71	20.9–743.27	<.01
After SFGH Initiative (as compared with between ACTG 5164 and the SFGH 5164 Initiative)	5.00	1.16–21.5	.03	8.04	1.34–48.3	.02

Abbreviations: CI, confidence interval; HIV, human immunodeficiency virus; IDU, injection drug user; MSM, men who have sex with men; OR, odds ratio; SFGH, San Francisco General Hospital.

The incidence of suppression of HIV RNA level to <400 copies/mm³ (Figure 3) at 180 days was 79.3% (95% CI, 71.1%–86.5%) before ACTG 5164, 83.9% (95% CI, 67.4%–95.0%) between ACTG 5164 and the SFGH 5164 Initiative, and 93.1% after the SFGH 5164 Initiative (95% CI, 73.1%–99.0%; P = .93).

Figure 3.

Time to first HIV RNA level <400 copies/mm³ after antiretroviral therapy initiation.

CONCLUSIONS

We found that ACTG 5164 led to a change in the time to ART initiation after PCP diagnosis in routine practice at an academic medical center. Diffusion, represented by the changes in practice after the ACTG 5164 and before the local SFGH 5164 Initiative, was associated with an ∼3-fold increase in the rate of ART initiation and an increase in the probability of early ART from a rare occurrence (7%) to relatively common (50%). Nevertheless, targeted information transfer and modification of local conditions were needed to maximize the uptake of early ART to desirable levels (83% after the SFGH Initiative). ART initiation was driven largely by temporal thresholds representing availability of new evidence and local practice initiatives rather than patient characteristics, suggesting that early ART after an OI is applicable to real-world patient populations as well as those in the original randomized trial.

Although diffusion is considered to be an inadequate mechanism of translation (pessimistic estimates suggest that 17 years are typically required for evidence to become widely practiced [10]), in this case, we observed notable and rapid changes in ART initiation even before targeted dissemination and implementation efforts. The academic nature of San Francisco General Hospital may partially explain this observation. Many providers attend scientific conferences, and new data are frequently discussed by providers in clinical care and case conferences. Furthermore, HIV medicine may differ from other fields. Over the past 2 decades, treatment improvements have emerged almost every year, and practitioners of HIV medicine may therefore be particularly accustomed to incorporating new evidence quickly [11]. Further evaluation of the roles of the academic environment and specialty may be an important area for research in implementation and dissemination sciences.

Although diffusion had a notable effect, our analysis supports the notion that dissemination and implementation are still required to ensure that evidence from a randomized trial is systematically applied in practice at an acceptable level. Leadership represents a novel area of inquiry as a determinant of clinical effectiveness [12, 13]. Although we have not formally measured these characteristics in our institutional environment, the local SFGH 5164 Initiative was indeed supported by local leadership who energized faculty members who, in turn, affected nonspecialty providers with face to face conversations. Adaptations to overcome locally specific barriers (such as accelerating insurance applications) were also necessary for routine early ART initiation. Although the health care system on a larger scale is complicated, disorganized, and often opaque, the infrastructural adaptations made to facilitate early ART in this case largely occurred in a single and effective microsystem. Our findings support empirical research to understand effectiveness through characterizing the functional, front-line units of health care [13, 14].

A perennial challenge to the translation of evidence into practice lies with concerns that research patients in clinical trials systematically differ in clinical, behavioral, and social characteristics from real-world patients [14]. Effectiveness, the success of practice in the real world, depends on the engagement of patients and providers [15]. Indeed, there was concern that uptake of early ART after PCP by patients at SFGH would be limited by patient-based factors, such as psychiatric comorbidities or housing instability, even if providers wanted to start ART earlier. Although we found that injection drug users had relatively lower odds of early ART in adjusted analysis, the strongest influence on ART start was the temporal intervals representing the effect of ACTG 5164 and the SFGH 5164 Initiative. Early ART is indeed feasible in real-world patients presenting with OIs.

A notable limitation to this analysis is that we cannot evaluate institutional determinants of implementation and dissemination because this is a single site study. We plan to pursue multi-site analysis in the future to evaluate site-level determinants of successful translation. Second, we do not have precise and comprehensive assessment of mental health and substance abuse diagnoses, which are common in our patient population. Our clinic, however, has always been a public health clinic and has served a similar patient population over the intervals assessed in this analysis, and therefore, changes in these factors are unlikely to explain changes in ART initiation. Furthermore, we find it unlikely that any confounder, whether measured or unmeasured, could change the proportion of patients receiving early ART after an OI from 7% (before ACTG 5164) to 83% (after the SFGH Initiative). Third, successful dissemination and implementation require not only uptake but also sustained practice change. Longer periods of observation will be needed to observe whether early ART after OI remains a component of routine practice in our setting. Finally, the patient population in this study was derived from a clinic database. We therefore were unable to assess outcomes for patients who received a diagnosis in the hospital but who never linked to primary care. We believe, however, that although this may affect the estimates of the magnitude of ART initiation, this selection process has not changed over time and, thus, may have a limited effect on the relative changes in ART initiation across the temporal thresholds of interest (our central estimate of interest).

In summary, we found that, at an academic medical center, ACTG 5164 changed clinical practice through passive diffusion but that active dissemination and implementation led by the HIV/AIDS Division were required to bring the application of early ART into a range that could be considered to be successful adoption. Initial evaluation did not find early ART practices to be associated with poorer initial virologic outcomes. Our findings support the notion that early ART after an OI is feasible in routine practice even in patient populations with more behavioral, psychiatric, and social comorbidities than those typically included in randomized trials. Multi-site studies are needed to further define the role of leadership, institutional effectiveness, and the specific elements of institutional culture on the success of implementation and dissemination efforts, as determined by changes in routine practice.