Benefit, Risk, and Outcomes in Drug Development: A Systematic Review of Sunitinib

Abstract

Background:

Little is known about the total patient burden associated with clinical development and where burdens fall most heavily during a drug development program. Our goal was to quantify the total patient burden/benefit in developing a new drug.

Methods:

We measured risk using drug-related adverse events that were grade 3 or higher, benefit by objective response rate, and trial outcomes by whether studies met their primary endpoint with acceptable safety. The differences in risk (death rate) and benefit (overall response rate) between industry and nonindustry trials were analyzed with an inverse-variance weighted fixed effects meta-analysis implemented as a weighted regression analysis. All statistical tests were two-sided.

Results:

We identified 103 primary publications of sunitinib monotherapy, representing 9092 patients and 3991 patient-years of involvement over 10 years and 32 different malignancies. In total, 1052 patients receiving sunitinib monotherapy experienced objective tumor response (15.7% of intent-to-treat population, 95% confidence interval [CI] = 15.3% to 16.0%), 98 died from drug-related toxicities (1.08%, 95% CI = 1.02% to 1.14%), and at least 1245 experienced grade 3–4 drug-related toxicities (13.7%, 95% CI = 13.3% to 14.1%). Risk/benefit worsened as the development program matured, with several instances of replicated negative studies and almost no positive trials after the first responding malignancies were discovered.

Conclusions:

Even for a successful drug, the risk/benefit balance of trials was similar to phase I cancer trials in general. Sunitinib monotherapy development showed worsening risk/benefit, and the testing of new indications responded slowly to evidence that sunitinib monotherapy would not extend to new malignancies. Research decision-making should draw on evidence from whole research programs rather than a narrow band of studies in the same indication.

Numerous analyses have chronicled high costs, attrition, and lengthy delays in drug development (1–3). In one recent study of US Food and Drug Administration (FDA)-approved drugs, the median time between start of clinical development and regulatory licensure was 6.5 years (4). In another, only one in 10 drugs entering clinical development is ultimately licensed for clinical application; the fraction of drugs receiving licensure in areas where clinical need is especially pressing—cancer and neurological drugs—is lower still (5,6). Such costs and attrition exact a heavy toll on health care systems, as expenses associated with drug development are ultimately absorbed into the cost of pharmaceuticals. They also impose burdens on patients participating in trials.

Little is known about the total patient burden associated with clinical development and where burdens fall most heavily during a drug development program. Clinical development is characterized by a range of different investigations: Drug developers conduct multiple trials in parallel with each other, continue testing long after a drug is licensed, and search for responding secondary clinical indications alongside lead indications. Thus, many of the aggregate figures about translation success and cost obscure patterns of success and trends in risk and benefit within a whole drug development program.

Many recent policy and research initiatives aim at accelerating the pace and improving success rates in clinical translation (7–10). Understanding patterns of research activity and burden in drug development can help identify opportunities for directing research resources more efficiently. It can also determine activities where burden is greatest, thus providing an evidence base for human protections.

To better understand the total risk/benefit associated with developing a drug and trends in risk/benefit spanning clinical development, we conducted a systematic review of monotherapy cancer trials for a successful drug, sunitinib, across 10 years of clinical development. We used objective response rate as a measure of benefit and grade 3 or above drug-related adverse events as a measure of harm.

Methods

The primary aim of this study was to capture a portfolio of published clinical trials for a single drug, sunitinib, and to quantify the total amount of patient burden and benefit encountered in clinical development. We defined a portfolio as all monotherapy trials testing sunitinib in oncology. Our secondary goals were to track the evolution of risk/benefit over the course of development and the relationship between risk/benefit with funding. Because sunitinib is licensed only as monotherapy and to limit the scope of this research effort, we focused our analysis on monotherapy trials.

Literature Search

We selected the drug sunitinib for analysis because it afforded 10 years of published trials in a variety of disease indications and a large but manageable number of trials. Trials were captured by searching Embase and Medline on March 11, 2015 for trials using the terms “sunitinib,” “Sutent,” and variations on “SU11248” and combined results with the exploded MeSH terms, including “randomized controlled trials,” “controlled clinical trials,” “random allocation,” “double-blind,” “single-blind,” “placebo,” “feasibility,” “pilot,” “proof of principle,” “open label,” “non-random,” “clinical trials” and “phase 1” to “phase 4” (Supplementary Methods, available online).

Trial publications were then screened by BC for the following inclusion criteria: 1) primary report, 2) final report (where a trial’s results were published more than once), 3) interventional trials, 4) human subjects, 5) phase 1 to phase 4, and 6) monotherapy trials. We excluded publications that were: 1) secondary reports, 2) interim results, 3) meta-analyses, 4) retrospective or observational studies, 5) laboratory analyses of ex vivo human tissues, 6) reviews, 7) preclinical studies, and 8) letters, editorials, guidelines, interviews, etc. During full-text analysis, abstract-only publications and poster presentations were excluded.

To explore a second dimension of drug development, we repeated the above for a second portfolio of sunitinib studies: combination studies involving the most intensively researched indication in our sample, renal cell carcinoma (RCC).

Extraction

We devised an extraction template that captured variables in the following domains: trial demographics, methods, hypothesis, trial design, measures of patient risk/burden and benefit (including adverse events, treatment duration, response, and investigator conclusions). Criteria for extraction were prespecified in a codebook, and coders underwent training before data collection.

We scored the number of grade 3 or 4 adverse events (defined by Common Terminology Criteria for Adverse Events [CTCAE] criteria [11]) in a trial conservatively, based on the highest number of events in a single category per trial. We based median duration of treatment on the length of time a patient must be enrolled in the trial to collect the primary endpoint. When not specified, duration of treatment was imputed by the product of the median number of cycles and the period of each cycle. Toxicities were only scored if they were defined as probably or definitely treatment related. Benefit was calculated in terms of the overall response rate (ORR; sum of complete and partial responses using RECIST criteria [12]) in individuals receiving sunitinib on an intent-to-treat basis. We used ORR because it is a widely used surrogate endpoint in cancer trials and allowed us to compare the magnitude of effect across diverse indications and trial phases. Indications were divided into meaningful categories identified in consultation with an oncologist (the exact indication for any node can be reviewed by reference to the numbered citation in our bibliography).

The success or failure of a study was determined based on whether the study reached its prespecified primary endpoint (usually ORR) and whether the regime was deemed tolerable by the authors. All studies were extracted by two independent coders using Numbat software—an open source meta-analysis management tool developed by BC (Numbat available from: http://bgcarlisle.github.io/Numbat/). Disagreements were reconciled by discussion between coders. In cases where opening and closure dates were unavailable, we contacted the corresponding authors for these details, with a reply rate of 35%. Our codebook is available upon request.

Analysis and Statistics

The differences in risk (death rate) and benefit (ORR) between industry and nonindustry trials were analyzed in R version 3.1.3 (13) using an inverse-variance weighted fixed-effect meta-analysis implemented as a weighted regression analysis. The outcome was regressed on a 0/1 indicator variable to identify industry trials. The individual trial results were weighted by sample size, as the variance of a rate estimate is directly proportional to sample size. We defined P values of less than .05 as statistically significant. All statistical tests were two-sided.

Results

Study Characteristics

Our literature searches captured 115 primary publications of interventional trials of sunitinib (see Figure 1 for a PRISMA flow diagram and Supplementary Table 1, available online, for per-trial details). There were 103 monotherapy studies (14–116) and 12 combination therapy trials (117–128) spanning 11 different combination therapies. Properties of trials in our sample are described in Table 1 (see Supplementary Figure 1, available online).

Figure 1.

PRISMA flow diagram.

Table 1.

Properties of extracted trials in our sample

Trial characteristics	Monotherapy (14-116) (n = 103) No. (%)	Combination therapy (117-128) (n = 12) No. (%)
Phase
1	15 (15)	8 (67)
1-2	1 (1)	2 (17)
2	80 (78)	2 (17)
3	7 (7)
Sponsor
Industry only	43 (42)	8 (67)
Nonindustry support	54 (52)	3 (25)
Not stated	6 (6)	1 (8)
Number of centers
Single center	36 (35)	3 (25)
Multicenter	63 (61)	8 (67)
Not stated	4 (4)	1 (8)
Randomization
Randomized	19 (18)	1 (8)
Nonrandomized	84 (82)	11 (92)
Trials with a non-sunitinib comparator arm	15 (15)	2 (17)
Average duration, wk	20.0	18.5
Results
Positive	48 (47)	1 (8)
Inconclusive	14 (14)	2 (17)
Negative	41 (40)	9 (75)
Mean sample size	88.3	20.1
Location of corresponding author
N America	58 (56)	9 (75)
Europe	32 (31)	3 (25)
Asia	11 (11)
S America	1 (1)
Australia	1 (1)

Total Patient Benefit and Burden

The 103 trials included in the sunitinib monotherapy analysis represent 9092 patients and 3991 patient-years of involvement over 10 years. Corresponding authors were based in 17 different countries. In the portfolio we identified, seven different doses, seven schedules, and 32 separate malignancies were tested. In total, 1052 patients in sunitinib monotherapy experienced objective tumor response (15.7% of intent-to-treat population, 95% confidence interval [CI] = 15.3% to 16.0%) and 98 died from drug-related toxicities (1.08%, 95% CI = 1.02% to 1.14%). There was a minimum of 1245 grade 3–4 drug-related toxicities (13.7%, 95% CI = 13.3% to 14.1%). In Figure 2A, we present the sequence of monotherapy and select combination therapy trials using an Accumulating Evidence and Research Organization (AERO) diagram (129). Based on start dates for these trials, an average of 10 new monotherapy trials were launched each year.

Figure 2.

Accumulating Evidence and Research Organization (AERO) graphs for sunitinib therapy. A) monotherapy stratified by indication. Nodes represent all available sunitinib cancer trials testing anticancer activity, arranged according to first publication date horizontally and stratified by indication. Larger square nodes are phase 1 trials in the designated indication. Small squares represent presence of patients with a particular indication in a “mixed malignancy” phase 1 trial. Circular nodes are phase 2 trials. Triangles are phase 3 trials. White nodes indicate acceptable toxicity and positive effect, gray represents inconclusive results, and black nodes indicate negative results for their prespecified primary endpoint. Number of nodes may not sum to figures represented in Table 1 because of different classificatory schema used in AERO graph. B) AERO graph for sunitinib combination therapy. Nodes represent sunitinib cancer trials, stratified by combination therapy, arranged according to publication year. Number of nodes may not sum to data in Table 1 because of different classificatory schema used in AERO graph.

The phase 1 acute myeloid leukemia (AML) trial is white, despite not being followed up in phase 2 because it reports five (33%) partial responses; but these were short-lived and the authors of this report recommended against further development. There are three phase 2 trials in pancreatic adenocarcinoma, prostate cancer, and hepatocellular carcinoma (HCC) that met prespecified primary endpoints other than ORR (eg, disease control rate [DCR] or 12-month progression-free survival [PFS-12]). However, the authors of the pancreatic adenocarcinoma trial “argue against any usefulness of sunitinib in this patient population” (55). The prostate cancer trial authors qualified their use of PFS-12 as a “soft endpoint” (35), and the authors of the HCC trial admit that PFS-12 “should no longer be used as a primary endpoint for trials testing new compounds in HCC” (47). In each of these cases, the trials had very few objective responses (1 [1.4%], 2 [5.6%], and 1 [2.2%], respectively) and no licensure was obtained for these indications. Later trials in mesothelioma, adrenocortical carcinoma, oligometasteses, uveal melanoma, endometrial cancer, meningioma, and thymoma in Figure 2A appear to show evidence of success in identifying new responding indications. However, the positivity represents a switch in primary endpoint from response rate to progression-free survival. With the exception of endometrial cancer, all of these “positive” trials would have been considered negative had response rate been chosen for primary endpoint (ie, ORR was 5% or lower).

Figure 2B shows the exploration of 11 combination therapy regimes for RCC—an indication that responded and received licensure as monotherapy (117–128). In all, 241 patients and 111 patient-years of involvement in combination therapy studies over five years with corresponding authors based in four different countries, testing 11 different combination therapies, were sampled. In total, 64 patients experienced objective tumor response (32.2% of intent-to-treat population, 95% CI = 29.5% to 34.8%) and four patients died from treatment-related toxicity (1.7%, 95% CI = 1.4% to 1.9%). There was a minimum of 69 grade 3–4 drug related toxicities (28.6%, 95% CI = 25.2% to 32.1%).

Sunitinib was first licensed by the FDA in 2006 for second-line gastrointestinal stromal tumor (GIST) and RCC and was licensed for pancreatic neuroendocrine tumors (pNET) in 2011. The relationship between achievement of milestones in sunitinib development and patient burden as measured by treatment-related deaths in the whole portfolio is depicted in Figure 3. Very few patient deaths occurred before appropriate dose, schedule, and initial responding indications were defined (3 deaths before FDA approval). However, burden accumulated before a third indication was established—81 treatment-related patient deaths in total.

Figure 3.

Cumulative treatment-related deaths in trials of sunitinib for key milestones. The total number of deaths in the whole portfolio is charted against time. Dates are based on trial closure.

The above analysis suggests that most of the clinically useful applications of sunitinib were discovered very early on in clinical development, and our AERO diagram suggests a worsening risk/benefit balance with further testing of new malignancies. To characterize the evolution of risk and benefit, we plotted cumulative rates of objective response and drug-related life threatening morbidity and mortality for all monotherapy trials testing indications that were not yet FDA approved (Figure 4). Cumulative death rates increased, while cumulative ORR rates diminished with time (see Supplementary Figure 2, available online, for noncumulative risk and benefit proportions by year for studies in non–FDA approved indications).

Figure 4.

Cumulative risk and benefit proportions for studies in non–US Food and Drug Administration–approved indications. All events were classified by report authors as probably or definitely drug related. Dates are based on trial closure. ORR = overall response rate.

Patterns of Research, Coordination, and Outcomes

Our AERO diagram reveals 30 phase II trials testing indications that had been previously tested in phase I. In three instances, positive signal in phase I (defined as an objective response) led to positive phase II studies; in eight instances, absence of signal in phase I was followed by negative phase II studies of the same indication in a larger population. In 18 instances, indications were explored in phase II without prior reported testing of the same indication in phase I trials. We found many instances where single indications were tested in monotherapy studies before the results of earlier studies in the same indication were available. For RCC and GIST, phase II studies were initiated before closure to enrollment in phase I and phase III trials were initiated before closure to enrollment in phase II. In two instances, inconclusive initial phase II studies in a given indication were followed by negative trials in the same indication. We found 16 instances where negative phase II trials in an indication were followed by other, negative phase II trials in the same indication. We calculate that 22 patients died and 155 grade 3–4 events occurred because of drug toxicity in these potentially replicative studies. Based on inception and closure dates in trial reports, however, no replicative phase II trials were initiated after the first closed to recruitment. The mean time between closure to recruitment and publication was 26 months. Mean time to publication for negative studies in our cohort of trials was similar to that for positive studies (26 vs 27 months).

Industry would be expected to vigorously pursue malignancies that show the greatest prospect of approval. Since industry is the single largest funder, it would also be better positioned to synthesize the totality of evidence and coordinate investigations accordingly. We therefore tested whether industry-funded studies showed a more favorable risk/benefit profile for trials of indications not yet FDA approved. Industry-funded studies showed no statistically significant difference in ORR (10.0% vs 8.5%, P = .62) or in drug-related death rate (1.54% vs 1.53%, P = .98) when compared with nonindustry.

Discussion

We found striking discontinuities as sunitinib advanced toward postlicensure trials, with success rates and patient benefit diminishing as the research program matured. Our findings suggest potential deficiencies in the way information was used for planning studies and have implications for the planning and review of trials for new interventions.

Our analysis reveals patterns of risk and outcomes that would not be apparent from aggregate figures regarding success rates in translation. Though sunitinib clearly counts as a success story for drug development, the number of negative trials for indications not yet FDA approved far exceeded the number of positive trials, and risk/benefit worsened with time. Indeed, the total risk/benefit for the whole monotherapy portfolio, which included phase II and II studies leading to licensure, was similar to that reported for phase I monotherapy cancer studies in general (130).

Key elements for unlocking the clinical utility of sunitinib (responding indications, schedule, and dose) were established early on—and at a very modest burden for patients. The almost uniform inability to detect a clinically promising signal after the first three indications suggests that after initial discovery of responding indications researchers were unable to marshal preclinical evidence, knowledge of pathophysiology, or biomarkers to select new indications for further testing.

Policy discussions of clinical translation often tacitly portray the process of clinical translation as an orderly and methodical process, whereby early-phase studies beget safer late-phase studies that match test conditions (131). Our analysis of the whole portfolio of sunitinib monotherapy trials—and a slice of combination therapy studies—reveals a pattern that runs contrary to this portrayal. Phase III and additional phase II trials are launched before initial phase II trials in the same indication are completed, and trials exploring new indications are run concurrently and in rapid succession; the latter makes it difficult for researchers to use outcomes to plan subsequent studies. Exploration of new indications continued despite mounting evidence that responding indications had been saturated or that criteria for launching tests of new indications were not bearing fruit. The patient burden associated with this approach to indication exploration—at least for sunitinib—was considerable.

Our identification of potentially replicative studies, and a rapid fire approach to exploring indications, suggest that some burdens of translation, in particular, adverse events for participants and costs associated with unsuccessful primary endpoint attainment, arise not merely from inherent uncertainties, but also from poor coordination and information flow across the full research portfolio.

The patterns we observed—if they hold for other drugs—have implications for ethical review and data monitoring of trials and what is disclosed to prospective subjects during informed consent. For instance, patients might be told whether they are participating in a study that is preceded by studies with positive signal or not and what this might entail in terms of risk. Data monitoring committees should remain abreast of and respond to the outcomes of other new indication studies in a translation trajectory. When a study in an indication fails, institutional review boards and data safety monitoring boards should require explicit justification for initiating new or continuing underway studies in the same indication.

Our findings also have implications for initiatives addressing inefficiencies in drug development. Studies in our sample were funded and pursued by various actors and sponsors, and no single entity is synthesizing findings across different indications and coordinating further testing. The highly favorable risk/benefit profile of studies early in development represents the effect of incentives for drug developers to sprint to market with an agent that can begin to recoup the costs of development. Industry quickly capitalized on low-hanging fruit. After clear demonstration of activities in several different malignancies, we see a process of exploration in which the search for additional indications appears to have been poorly coordinated and may have involved a relaxation of evidentiary criteria.

These results should be interpreted in light of the following limitations. First, trends for sunitinib monotherapy may not generalize to other drugs. Sunitinib is a multikinase inhibitor; the prospect of many potential applications for a relatively novel drug—and strong signal of activity early on—may have driven an unusual level of indication exploration; many off-target effects likely drove the heavy burden. Second, to our knowledge, all of the studies enrolled patients who had exhausted established effective therapy. Given the absence of effective therapy for these life-threatening diseases, there are good reasons for vigorous research programs testing possible therapies. Nevertheless, patients with advanced disease are entitled to having their interests protected by proper planning, coordination, and risk minimization. Moreover, studies of second-line therapy and beyond still entail opportunity costs and draw on scarce research resources. Third, any single malignancy may have seemed—and may continue to appear—a plausible candidate for sunitinib monotherapy. Nevertheless, knowledge of how sunitinib monotherapy had fared against a host of other malignancies might have tempered the rationale for further exploration of indications and combination therapies. Fourth, some apparent redundancy reflected in our AERO diagram reflects important differences in trial hypotheses. For example, variation in trial outcomes in the RCC stratum in part reflects that some trials tested RCC subtypes, such as non–clear cell RCC, that are less responsive to sunitinib. Fifth, had investigators chosen different endpoints (eg, progression-free survival), the pattern of success and failure and evolution of risk/benefit might appear different. Last, our study relied on fully published research reports. According to one analysis, 61% of phase II studies are never published (132), and we found that 36% of registered sunitinib trials were not published within four years of the trials’ registry closure dates. However, we suspect that unpublished studies, if added to our analysis, would not improve the risk/benefit balance of the entire portfolio.

Our findings reinforce the suggestion, offered by others (133), that systematic review and research decision-making should draw on evidence from whole research programs rather than particular tranches within them. Any one protocol may appear to have a sound basis, but when the sunitinib monotherapy development portfolio is viewed as a whole there was a trend of worsening risk/benefit, and the testing of new indications responded slowly to accumulating evidence that sunitinib monotherapy would not extend to new malignancies. Most major codes of ethics prescribe the minimization of risk; a view of the entire portfolio and an assessment of cumulative risk/benefit offer a resource for achieving these ends.

Funding

This work was funded by the Canadian Institutes of Health Research (EOG111391).

Without intending to suggest their endorsement of our analysis, we thank Dean Fergusson, Abe Fuks, and Charles Weijer for various discussions; we also thank Janet Dancey for consultations. Faults remain our own.

The authors declare no competing interests related to the content of this paper.