Abandonment of High-Dose Chemotherapy/Hematopoietic Cell Transplants for Breast Cancer Following Negative Trial Results

Abstract

Objective

In 1999, three randomized controlled trials concluded that high-dose chemotherapy followed by autologous hematopoietic stem cell transplantation (HDC/HCT) is no better than conventional chemotherapy for women with breast cancer. This study documents the impact of the trials on use of HDC/HCT and describes how hospitals reacted to the trials.

Data Source

We used patient-level data on 15,847 HDC/HCTs reported to the Center for International Blood and Marrow Transplant Research between 1994 and 2005.

Study Design

We report trends in total HDC/HCT procedure volume, compare the time to hospitals' exit from the HDC/HCT market between research and nonresearch hospitals, and document trends in hospital-specific volumes in the 2 years before exit.

Principal Findings

HDC/HCT volume declined from 3,108 in 1998 to 1,363 the year after trial results were released. In 2002, only 76 procedures were performed. Teaching hospitals and the hospitals that participated in the trials were no slower to discontinue the procedure compared with nonteaching, nonparticipating hospitals. At the hospital level, volume declined steadily in the months before abandonment.

Conclusion

The results suggest that comparative effectiveness research studies that report negative results can reduce spending, but specialists may be reluctant to relinquish cutting-edge technologies.

There is an extensive literature on the adoption and diffusion of new health care technologies, but relatively few studies examine the abandonment of technologies found to be ineffective (Rye and Kimberly 2007). The general perception is that abandonment occurs slowly (RAND 2010). For example, a 1983 National Heart, Lung, and Blood Institute clinical trial reported that intermittent positive pressure breathing therapy did not reduce mortality or improve quality of life for patients with chronic obstructive pulmonary disease (The IPPB Trial Group 1983). By 1986, use remained at about 50 percent of the 1983 level (Duffy and Farley 1992). Of the 37 hospitals that offered intermittent positive pressure breathing therapy in 1893, 23 continued to offer it in 1986.

In this paper we examine the abandonment of high-dose chemotherapy followed by autologous hematopoietic stem cell transplantation (HDC/HCT) for women with breast cancer. Initially, technology evaluations by the Blue Cross and Blue Shield Association and other groups raised concerns about the value of the procedure, but matters remained unsettled through the 1990s until randomized clinical trials demonstrated that the procedure was no better than standard outpatient chemotherapy. Although results from the trials were released over 10 years ago, it is useful to revisit this example in light of the current debate over comparative effectiveness research and its potential impact on practice patterns and costs. The objectives of this study were to understand the impact of the trials on use of HDC/HCT, the relationship between hospital characteristics and abandonment of HDC/HCT, and the abandonment process.

A BRIEF HISTORY OF HDC/HCT

High-dose chemotherapy followed by autologous blood and marrow transplantation entails (1) harvesting and storing hematopoietic stem cells (HSC) obtained directly from the patients' bone marrow or peripheral blood after hematopoietic growth factor±chemotherapy mobilization, (2) administering high doses of chemotherapy with the intent of killing malignant cells but with the recognition that healthy marrow cells will also be destroyed, and (3) transplanting the stored HSC back into the patient to regenerate hematopoiesis. The story of how HDC/HCT came to be widely used as a treatment for breast cancer despite the absence of evidence of efficacy has been recounted previously in Rettig et al.'s (2007) excellent False Hope. Briefly, the first HDC/HCT in women with breast cancer were performed in the late 1970s, but the procedure did not come into widespread use until the late 1980s and early 1990s (Antman and Gale 1988). Initially, HDC/HCT was used in women with metastatic disease, but, by the mid 1990s about half of HDC/HCTs were performed in women with less advanced (stages II and III) malignancies.

For women with few curative treatment options, HDC/HCT offered the hope of long-term survival, but, because of its cost, side effects (including gastrointestinal toxicity, nausea, vomiting and diarrhea, infection, and organ toxicity), the large number of potentially eligible patients with this common malignancy, and limited evidence of efficacy, the procedure was subject to a high degree of scrutiny from insurers. Many initially refused to pay for the procedure on the grounds that it was investigational. Noncoverage was the subject of a number of lawsuits in the 1990s brought by patients against insurers (Mello and Brennan 2001), and 11 states eventually mandated that insurers cover HDC/HCT.

Table 1 shows a timeline of the major studies and technology evaluations of HDC/HCT as a treatment for breast cancer from the early 1980s to the end of the 1990s. Uncontrolled studies reporting tumor response rates provided the first evidence of the potential for HDC/HCT, but the Blue Cross and Blue Shield Association concluded there was insufficient evidence to recommend HDC/HCT over conventional therapy (Eddy 1992; Rettig et al. 2007). A 1993 study comparing outcomes among HDC/HCT patients to historical controls (Peters et al. 1993) and a 1995 South African randomized controlled trial (Bezwoda, Symour, and Dansey 1995) reported positive results. However, a Dutch trial, published in 1998, found no difference in progression free and overall survival between patients treated with HDC/HCT versus conventional therapy (Rodenhuis et al. 1998).

Table 1.

Timeline of the Major Evaluations of HDC/HCT Through 1999

Date	Description	Assessment*
1980s	Phase 2 studies	+
1988	BCBSA technology evaluation	−
1990	BCBSA technology evaluation	−
1992	Literature review by David Eddy (1992)	−
1993	Peters et al. (1993) report benefits of HDC/HCT relative to historical controls	+
1994	BCBSA technology evaluation	−
1995	ECRI Institute: HDC/HCT “no better” than conventional therapy	−
1995	South African study showing benefits for HDC/HCT †	+
1996	BCBSA technology evaluation	+
1996	NCCN guideline: HDC/HCT should be evaluated in clinical trials but is not a recommended treatment	−
October 1998	Dutch RCT published (Rodenhuis et al. 1998)	−
February 1999	Closed-door NCI meeting to review RCT results	−
March 1999	Media reports of negative RCT findings	−
May 1999	Reporting of RCTs at the annual ASCO meeting	−

^*+HDC/HCT improves outcomes, −HDC/HCT no better than conventional care.

^†A 2000 audit found that the trial was fraudulent.

ASCO, American Society of Clinical Oncology; BCBCA, Blue Cross Blue Shield Association; HDC/HCT, high-dose chemotherapy/hemapoetic cell transplantation; NCCN, National Comprehensive Cancer Network; RCTs, randomized controlled trials.

Matters remained unsettled until May 1999, when investigators presented results showing no overall survival advantage for women receiving HDC/HCT from three randomized controlled trials—two from the United States and one from Sweden—at the annual meeting of the American Society of Clinical Oncology (ASCO). A fourth trial from South Africa, which was also presented at the May 1999 meeting, reported a survival advantage for patients in the HDC/HCT arm. A subsequent investigation, which began in early 2000 and published findings in the March 18, 2000 Lancet (Weiss et al. 2000), determined that the principal investigator had manufactured the data for this study and the previously published South African trial.

Initially, many oncologists and professional groups reacted cautiously to the negative trial results. For example, the American Cancer Society stated (Schellenbach 1999),

The American Cancer Society strongly supports reimbursement for bone marrow transplantation by insurance carriers for the treatment of appropriate hematologic malignancies. We believe that there is currently insufficient evidence to determine the efficacy of bone marrow transplantation for breast cancer, and support further analysis of clinical data obtained from carefully controlled peer-reviewed clinical trials.

Some oncologists continue to believe that the trials did not conclusively establish the equivalence of HDC/HCT and conventional therapy and that HDC/HCT may offer benefits for certain categories of breast cancer patients. A retrospective analysis of trial results and Center for International Blood and Marrow Transplant Research (CIBMTR) data (Berry et al. 2002) found that while short-term survival rates were similar between women with metastatic tumors treated with standard chemotherapy versus HDC/HCT, women treated with HDC/HCT “might have a modestly higher long-term probability of survival.” However, the prospect of performing additional trials of HDC/HCT is dim.

METHODS

Data and Sample Selection

We analyzed the rate and pattern of abandonment of HDC/HCT for breast cancer using data from the CIBMTR. The CIBMTR is a voluntary consortium involving more than 500 transplant centers in 54 countries. For this analysis, we used patient-level data on 15,847 HDC/HCT performed in the United States for women with a primary diagnosis of breast cancer that were reported to the CIBMTR from 200 U.S. transplant centers between 1994 and 2005. The CIBMTR estimates that it collected data on 60 percent of autologous HSC transplant procedures (all diagnoses) performed in the United States during this time period. It probably captured a larger share of HDC/HCT procedures, because these were mostly performed at the high volume research centers that make up a disproportionate share of participating CIBMTR centers.

We analyzed discontinuation rates in the 122 hospitals that performed one or more HDC/HCT procedures in women with breast cancer between 1995 and 1997, performed more than 10 HCT procedures overall between 1995 and 2005, and were performing HDC/HCT procedures in women with breast cancer in 1998.

Variable Construction

We considered a hospital and its physicians to have abandoned HDC/HCT if it did not perform any procedures in women with breast cancer for six consecutive months. We classified the last month in which a hospital performed HDC/HCT in women with breast cancer as the exit month. For the sake of explication, we characterize abandonment as a decision undertaken by hospitals. In reality, the abandonment decision is a joint decision by hospital administrators, specialists, referring physicians, and patients.

We measured hospital characteristics using publicly available sources (e.g., a list of the National Cancer Institute's [2010] Comprehensive Cancer Centers) and the CIBMTR data. We measured each hospital's HDC/HCT and HCT volume (including autologous and allogenic procedures and transplants for women with breast cancer and patients with other diagnoses) based on transplants performed between 1995 and 1997 and classified hospitals as above or below the median.

Concern about a lack of external validity may discourage physicians from abandoning treatments following a negative trial result. Providers with better-than-average outcomes may believe that their patient selection and treatment protocols are superior to those tested in the trial. We tested this hypothesis in the context of HDC/HCT by comparing exit rates based on historical, hospital-level outcomes. We measured hospital-level outcomes for HDC/HCT by calculating 1-year survival rates for women treated between 1995 and 1997, standardized by cancer stage (metastatic versus other).

Statistical Analysis

We depict hospitals' abandonment of HDC/HCT using Kaplan–Meier curves and test for unadjusted differences between groups using the log-rank test. Because of the high degree of overlap in the hospital characteristics and the small sample size, we did not perform a multivariate analysis.

We posit that if hospitals consciously decided to discontinue offering HDC/HCT, then procedure volume will drop suddenly in the month before abandonment. By contrast, if hospitals abandoned HDC/HCT only after demand for the procedure falls, then volume in the months before abandonment will decline slowly. We estimated hospital-level trends in monthly HDC/HCT volume before abandonment using a negative binomial regression model with a time trend variable and an indicator variable for the month before the hospital stopped performing HDC/HCT (see Appendix SA2). The sample consists of each hospital's monthly HDC/HCT volume in the 24 months before abandonment. We used the model to predict the path of hospital-level volume before abandonment.

RESULTS

Procedure Volume

Monthly counts of the number of women undergoing HDC/HCT and reported to the CIBMTR registry are displayed in Figure 1. The graph shows volume through January 2002, after which volume was near 0. The solid line represents a 5-month moving average. After rising rapidly through 1997, the number of cases started to decline in early 1998, dropping rapidly after the May 1999 ASCO meeting. The decline in the use of HDC/HCT beginning in early 1998 may have been a reaction to the August 1998 publication of the Dutch trial. Results of the trials presented at the May 1999 ASCO meeting were publicized before the meeting—NBC news ran a story about the studies on March 9, 1999—but it is unlikely that the results were known before 1999. The incidence of breast cancer was increasing during this period.

Figure 1.

Monthly Counts of HDC/HCT Procedures in Women with Breast Cancer

By May 2000, the number of HDC/HCTs was <20 percent of the March 1998 peak. However, physicians did not discontinue HDC/HCT overnight. Over 1,500 women received HDC/HCT in the 2 years after May 1999. Some or many of these women were enrolled in ongoing clinical trials.

Hospital-Level Abandonment

Hospitals' abandonment of HDC/HCT is represented in Figure 2. The height of the curve (the solid line) indicates the proportion of hospitals continuing to perform HDC/HCT in women with breast cancer. About 20 percent of hospitals performing HDC/HCT exited the market before May 1999.

Figure 2.

Kaplan–Meier Curve of Hospital-Level Abandonment of HDC/HCT

The pace of exit increased in May 1999, as indicated by the divergence between the Kaplan–Meier curve and the dotted line, representing the pre-May 1999 trend. By May 2001, only a handful of hospitals were performing HDC/HCT in women with breast cancer on a regular basis.

Table 2 shows the association between selected hospital characteristics and the median time to exit in months. Note that there is substantial overlap between the categories (e.g., 12 of the 19 phase III trial sites are comprehensive cancer centers). The median time to exit, measured from January 1999, among all 122 hospitals was 15 months. Among the 19 hospitals that enrolled patients in the phase III trials presented at the May 1999 ASCO meeting the median time to exit was 21 months. The difference in abandonment rates is depicted in Figure 3.

Table 2.

Impact of Hospital Characteristics on Abandonment of HDC/HCT

		Months to Exit (from January 1, 1999)
Hospital Characteristic	Number of Hospitals	Median	p-Value*
All hospitals	122	15	N/A
Phase III trial site	19	21	.098
Comprehensive cancer center	33	18	.048
Teaching hospital	71	17	.295
HDC/HCT mandate state	29	14	.530
High HDC/HCT 1-year survival rate†	61	15	.833
High HDC/HCT volume (i.e., women with breast cancer)†	61	17	.044
High HCT volume (i.e., breast cancer and other diagnoses)†	61	18	.011
High HDC/HCT volume as a share of total HCT volume†	61	15	.897

^*Statistical significance was assessed using the log-rank test.

^†These outcomes were measured using transplants performed during the period January 1, 1995–December 31, 1997. The sample was split based on the median of each variable. Hospitals in the “High” group have values above the median. The medians are 38% for survival, 42 for HDC/HCT volume, 95 for HCT volume, and 54% for HDC/HCT share.

HCT, hemapoietic cell transplant; HDC, high-dose chemotherapy; N/A, not applicable.

Figure 3.

Kaplan–Meier Curve of Hospital-Level Abandonment of HDC/HCT, Trial Sites versus Other Hospitals

National Cancer Institute Comprehensive Cancer Centers and teaching hospitals (as measured by membership in the Council of Teaching Hospitals) were slower to abandon HDC/HCT, though differences in median exit times are not significant. Hospitals located in one of the 11 states with insurance benefit mandates for HDC/HCT abandoned HDC/HCT at similar rates to hospitals in nonmandate states. Exit rates were similar between hospitals with low and high survival rates.

Hospitals that performed more HDC/HCT procedures and more HCT procedures overall were slower to abandon HDC/HCT. Hospitals that were more dependent on HDC/HCTs for their overall HCT volume, as indicated by the share of HCTs performed in breast cancer patients, abandoned HDC/HCT at similar rates to hospitals with below-median HDC/HCT shares.

We used Cox's proportional hazard models to perform a multivariate analysis (see the Appendix SA2 for a table of result). Survival and volume measures were entered as continuous variables. Only the variable measuring HDC/HCT volume as a share of total HCT volume was significantly related to exit rates, with odds ratios ranging from 0.15 to 0.18 depending on the specification. In general, the univariate and multivariate analyses fail to reject the hypothesis that abandonment rates were similar between the hospitals that did and did not participate in phase III clinical trials and academic and nonacademic medical centers.

The Abandonment Process

The previous section presumes that hospitals' medical staff offices and transplant specialists actively decided when to abandon HDC/HCT. However, it is possible that hospitals and specialists are passive actors in the process, with the decision to abandon a procedure occurring only after demand by patients and referring physicians has evaporated. Consider the different patterns of abandonment displayed in Figure 4. The top dashed line, labeled “Active,” illustrates the trend in procedure volume we would expect to see if the abandonment decision was made exclusively by hospitals. In this scenario, volume is steady and then drops precipitously once the hospital decides to stop offering HDC/HCT. The bottom dashed line, labeled “Passive,” illustrates the trend we would expect to see if patients and referring physicians, rather than hospitals, decide to abandon the procedure. In this scenario, volume declines steadily, reflecting diffusion of the negative results in the community, until it reaches zero, at which time the hospital has effectively, if not purposely, abandoned the procedure.

Figure 4.

Pattern of Hospital-Level Disadoption

Using results from the negative binomial model, we predicted the path of the decline in volume for the typical hospital. The dotted line in Figure 4 depicts the predicted values from the regression model. The observed pattern is somewhere between the two scenarios, but it is probably closer to the “Passive” scenario. Twenty-four months before abandonment, average volume was 1.6 cases per month. One month before abandonment, volume was about 0.6 cases per month, or <40 percent of the volume 2 years prior.

DISCUSSION

Comparative effectiveness research is often touted as a tool to reduce costs. The hope is that studies will uncover examples of widely used, costly technologies that are no better than less expensive alternatives. However, if studies that report negative results do not influence practice patterns, cost savings may fail to materialize. Analyses by the Congressional Budget Office (2007) and RAND (2010) have struggled to arrive at reasonable estimates of the budgetary impact of comparative effectiveness research. While it may be impossible to rigorously estimate savings, studies of historical examples of comparative effectiveness research and their impact on practice patterns can help to inform the debate. We found that the release of clinical trials showing that HDC/HCT was equivalent to conventional chemotherapy led to a rapid decline in the use of the procedure.

A handful of studies have documented the abandonment of medical therapies following trials reporting equivalence results (as opposed to studies that expose previously unknown drug side effects). The federally sponsored National Emphysema Treatment Trial (National Emphysema Treatment Trial Research Group 2003), which enrolled patients between January 1998 and July 2002, tested lung-volume reduction surgery versus medical therapy. The trial found that surgery improved exercise capacity but did not extend survival. Annual procedure volume declined from 800 pretrial (Huizenga et al. 1998) to 254 posttrial (Naunheim 2007). Medicare covered the procedure with restrictions following the trial, but during the trial it covered surgery only for trial participants, which may have hastened the decline. By contrast, practice patterns responded slowly to negative results from a trial testing the effectiveness of intermittent positive pressure breathing therapy for patients with chronic obstructive pulmonary disease (Duffy and Farley 1992).

Studies of the impact of trial results on practice patterns are useful for prioritizing comparative effectiveness research monies. The National Institutes of Health and other funders may consider the potential impact of comparative effectiveness research on practice patterns when deciding which grant applications to fund (VanLare, Conway, and Sox 2010), and so it is important to understand how the uptake of research recommendations varies across conditions, technologies, and clinical settings.

The main limitation of this study is that we cannot observe abandonment patterns at hospitals and cancer clinics that did not report data to the CIBMTR registry. However, procedure counts from the CIBMTR registry are very close to procedure counts from the Nationwide Inpatient Sample, suggesting that CIBMTR captured a large share of the inpatient HDC/HCT procedures performed in women with breast cancer (see Table 5.1 in Rettig et al. 2007). Another limitation is that we do not observe underlying demand for the procedure by patients and referring physicians. Our inferences are based only on the realized volume of HDC/HCT procedures.

Throughout its brief history, HDC/HCT was carefully scrutinized by insurers. However, it is unlikely that changes in coverage policy played a large role in the abandonment process. Rates of abandonment were similar in states with and without HDC/HCT coverage mandates, and Aetna did not withdraw coverage for HDC/HCT until May 2000 (Rettig et al. 2007). Insurers' internal review processes and inability to change benefits between annual contract renewals make it difficult to rapidly incorporate negative study results into coverage policies.

Hospitals that produce new medical knowledge, that is, teaching hospitals and National Cancer Institute Comprehensive Cancer Centers, did not abandon HDC/HCT more rapidly than nonteaching, non-Comprehensive Cancer Center hospitals. These findings are somewhat surprising given that these hospitals produce, and thus ought to be more responsive to, new medical knowledge and research. Some of these centers may have continued to enroll patients in clinical trials that were opened before May 1999, believing that these studies would address questions left unanswered by the results presented at the ASCO meeting. Additionally, hospitals' decisions may have been influenced by the incentives presented by fee-for-service reimbursement, which generally rewards hospitals and physicians for increasing procedure volume, and by physicians whose careers were closely tied to the continuation of research on HDC/HCT in women with breast cancer. In general, our results are consistent with, but do not prove, the hypothesis that hospitals did not consciously decide to abandon the procedure. Rather, they suggest that hospitals passively abandoned HDC/HCT as demand among patients declined precipitously.

Our findings indicate that comparative effectiveness research has the potential to reduce costs. Of course, every therapy has unique features that may speed up or retard abandonment. Women undergoing HDC/HCT typically experienced more side effects than women undergoing conventional chemotherapy (Farquhar et al. 2005a, b). Although most were temporary and reversible, some patients and referring physicians may have viewed HDC/HCT as inferior to conventional chemotherapy once it became clear that HDC/HCT did not improve survival.

Another unique feature of the HDC/HCT experience was the degree of publicity surrounding the trials. The findings were widely covered in the popular press. Obviously, decision makers have to be aware of comparative effectiveness results to act on them. It remains an open question whether knowledge of trial results by specialists is sufficient to elicit a change in practice patterns, or whether wider dissemination to referring physicians and even patients is necessary. Specialists may be reluctant to “give up” on cutting-edge procedures, and negative results often prompt physicians to improve upon, but not abandon, technologies perceived to be innovative or cutting edge. The cycle of innovation, analysis, and refinement is a critical element of medical progress, but it should not be allowed to continue indefinitely in the absence of evidence of effectiveness.

SUPPORTING INFORMATION

Additional supporting information may be found in the online version of this article:

Appendix SA1: Author Matrix.

Appendix SA2. Coefficients froma Cox-Proportional Hazards Model of the Time to Exit (N5122).

Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.