Challenges in the Design and Implementation of The Multicenter Uveitis Steroid Treatment (MUST) Trial – Lessons for Comparative Effectiveness Trials

Abstract

Background

Randomized clinical trials are an important component of comparative effectiveness (CE) research because they are the optimal design for head-to-head comparisons of different treatment options.

Purpose

To describe decisions made in the design of the Multicenter Uveitis Steroid Treatment (MUST) Trial to ensure that the results would be widely generalizable.

Methods

Review of design and implementation decisions and their rationale for the trial.

Results

The MUST Trial is a multicenter randomized controlled comparative effectiveness trial evaluating a novel local therapy (intraocular fluocinolone acetonide implant) versus the systemic therapy standard of care for noninfectious uveitis. Decisions made in protocol design in order to broaden enrollment included allowing patients with very poor vision and media opacity to enroll and including clinical sites outside the US. The treatment protocol was designed to follow standard care. The primary outcome, visual acuity, is important to patients and can be evaluated in all eyes with uveitis. Other outcomes include patient-reported visual function, quality of life, and disease and treatment related complications.

Limitations

The trial population is too small for subgroup analyses that are of interest and the trial is being conducted at tertiary medical centers.

Conclusion

CE trials require greater emphasis on generalizability than many RCTs but otherwise face similar challenges for design choices as any RCT. The increase in heterogeneity in patients and treatment required to ensure generalizability can be balanced with a rigorous approach to implementation, outcome assessment and statistical design. This approach requires significant resources that may limit implementation in many RCTs, especially in clinical practice settings.

Introduction

A primary objective of comparative effectiveness research (CER) is to generate evidence that compares the benefits and harms of alternative treatment options in order to provide clinicians and consumers with evidence to inform treatment choices.[1] Randomized clinical trials (RCTs) are an important component of CER because they can provide unbiased head-to-head evaluations of alternative treatment options. Depending on the treatment information gaps, the questions asked by CE RCTs can be quite different – encompassing both good and bad outcomes, or some subset thereof, but must provide information on outcomes that are important to patients and physicians that can be generalized to a broad set of clinical circumstances.[2]

For decades, treatment of non-infectious uveitis involving the posterior segment of the eye (intermediate, posterior or pan-uveitis) has relied on systemic corticosteroids supplemented with immunosuppressive drugs when indicated.[3] In 2005 an intraocular corticosteroid implant for non-infectious uveitis, the fluocinolone acetonide implant (Retisert®, Bausch & Lomb, Rochester, NY) was licensed in the United States (US). The fluocinolone acetonide implant was the first treatment approved by the FDA specifically for non-infectious uveitis, and pivotal trials suggested it was effective in controlling uveitis for up to three years and avoided potential systemic side effects of alternative therapies but had a high rate of local ocular side effects.[4] Disease experts judged that it had the potential to become a first-line treatment for uveitis, thereby changing treatment practices from a rheumatological to a surgical paradigm and substantially increasing the up-front cost of treatment (each implant is priced at $18,250, not including the costs of implant surgery).[5] Trials of the implant, including the ones leading to approval, have used time to re-activation of uveitis activity after initiation of treatment.[6,7,8] Uveitis activity reflects the immediate effect of therapy, not long-term control of disease. Using uveitis activity as a primary outcome may bias the results against systemic therapy because systemic regimens rely on tapering medications until uveitis activity returns and then increasing the medication to the threshold needed to control inflammation. Therefore, protocols that require tapering at set time points are expected to show a favorable treatment effect for the implant because the implant provides sustained treatment for 2 to 3 years. This bias is the key deficiency of the industry sponsored trial purported to evaluate effectiveness of the implant compared to standard therapy.[7] Hence, the National Eye Institute (NEI) agreed to sponsor and awarded funding to conduct the Multicenter Uveitis Steroid Treatment (MUST) Trial as a comparative effectiveness trial.

The specific aims of the MUST Trial are to compare the fluocinolone acetonide intravitreal implant to standard systemic corticosteroid therapy supplemented, when indicated, by immunosuppressive therapy, with respect to visual outcomes, control of uveitis, ocular and systemic side effects of therapy, and patient-reported quality of life. We completed enrollment of 255 patients with uveitis in December 2008. Details about trial procedures and the baseline characteristics of enrolled patients have been published.[9] The two year results of the trial have also been published.[10] Because our goal was to inform clinical practice about the relative merits and risks of local therapy with the implant versus systemic therapy, key decisions regarding the primary hypothesis, eligibility criteria, clinical centers, comparator treatment, and outcomes were evaluated with an emphasis on their effect on the generalizability of the results.

Choice of a superiority versus a non-inferiority design

A fundamental issue we faced is whether a superiority or a non-inferiority design was more appropriate. A non-inferiority paradigm was appealing because the two treatments entail very different administration procedures (daily oral drugs versus a surgically administered intraocular implant that lasts for 2.5 years or longer) and are associated with different risk profiles.[3,7] Hence, establishing the implant as non-inferior to systemic therapy would be of interest because treatment choice could then be recommended based on other considerations, such as side effects and convenience. However, there was uncertainty about the appropriate margin for equivalence. Furthermore, because of the known risks associated with the implant, such as almost certain cataract in phakic eyes and high rates of ocular hypertension requiring intervention [6,7], and the substantial up-front cost differential [5] between the implant and oral corticosteroids, we concluded that the implant would need to show a clear advantage over systemic therapy in order for it to be adopted as a first-line therapy. This consideration led to our choice of a superiority design as the most informative for clinical practice.

Eligibility criteria

Non-infectious uveitis includes a diverse group of syndromes, sometimes associated with systemic inflammatory diseases that themselves may require systemic corticosteroid therapy.[3] Despite heterogeneity in the clinical presentation of these syndromes, the standard approach for more severe uveitis cases is systemic corticosteroids [3] with the fluocinolone acetonide implant now as an alternative treatment option. Hence, we constructed our eligibility criteria to include all types of non-infectious intermediate, posterior or pan-uveitis and to exclude only patients with contraindications to either treatment regimen (e.g., high risk of glaucomatous damage should a large increase in intraocular pressure occur) or other safety considerations (e.g., pregnancy or children younger than 13 years of age).[9] We also excluded patients who required systemic corticosteroid therapy for associated systemic disease, in order to ensure the validity of the results by minimizing crossover for patients randomized to implant therapy.

The goal of the study was to evaluate alternative treatments for active uveitis, but both treatment approaches require initial quieting of inflammation using corticosteroids. Therefore, we enrolled patients who had quiet uveitis that had been active within the past 60 days and those with currently active uveitis. Even though our primary outcome is visual acuity (see discussion below on choice of primary outcome), we allowed patients with very poor vision (hand motion or better) in one or both eyes to enroll. While patients with severe vision loss sometimes are incapable of improvement in vision, most uveitis patients are able to improve, and those unable to improve would be distributed randomly between the comparison groups. There is a similar problem for patients enrolled with good vision; their vision cannot improve but they can lose vision. This loss in efficiency was offset by including all eyes with uveitis in the primary analysis, which increased power, since there was relatively low correlation (rho~0.4) between visual acuity in eyes of patients with bilateral uveitis. Although many of our secondary outcomes rely on evaluating images of the eye to assess structural changes (slit lamp photographs of the lens, color fundus photographs, fluorescein angiograms and optical coherence tomograms), we did not exclude patients with severe media opacity that precluded obtaining good images because these patients could contribute information toward the primary outcome; furthermore, media opacity is likely to be a common clinical situation. To preserve validity, we developed minimum imaging protocols for patients with severe media opacities or occluded pupils so that as many protocol images as possible were obtained and missing data were minimized.[9]

Choice of clinical centers

We enrolled patients at 23 centers, 21 in the US, one in London, England and one in Melbourne, Australia, all of which were tertiary care centers. The choice of centers was a pragmatic decision; each center had to provide evidence to show that reasonable numbers of eligible patients were followed and that the clinic was equipped with the appropriate tools for assessing outcomes, e.g., visual acuity lanes and imaging equipment.

Choice of control group

The obvious control group was systemic therapy with oral corticosteroids, which is the established standard approach compared to less established treatment regimens such as intraocular injections of corticosteroids. While the implant therapy readily lent itself to a clear treatment protocol, there can be heterogeneity in how oral corticosteroids are administered, as well as in how supplementary immunosuppressive agents are chosen and administered. The treatment protocol for systemic therapy relied on established treatment guidelines [3] that allowed flexibility in approach. Flexibility was essential, given that some patients entered the trial with active inflammation requiring a high dose of systemic corticosteroids while others already had been receiving treatment and had quiet uveitis, in which case tapering of corticosteroids was the next reasonable step. Likewise, for some patients a given immunosuppressive drug might be contraindicated (e.g., methotrexate in a patient with liver disease). The protocol specified an appropriate initial dose of systemic corticosteroids for patients with active disease and provided a tapering regimen to be used for those with quiescent disease. In cases for which corticosteroid-sparing immunosuppression was indicated, we permitted choice of immunosuppressive agents based on best medical judgment, but required that immunosuppressive agents be administered in accordance with the expert guidelines.[3] This approach of establishing standards for usual care is recommended for “real-world” effectiveness trials.[11]

Implementing a treatment regimen that allowed heterogeneity presented challenges for both education of clinical investigators and monitoring for quality assurance. Education of clinical investigators was performed via a training meeting, a detailed protocol, and manual of operations with user-friendly guides. In addition, we organized a group of study physicians as a quality assurance team to provide continued education on the protocol and formal review of treatment implementation, the Medical Therapy Quality Assurance Committee (MTQAC). MTQAC members visited each of the 23 clinical centers approximately annually to review the medical records of study patients in order to ensure that the individual treatment regimens were in accordance with the protocol guidelines for corticosteroid dose and tapering, and for ensuring appropriate initiation of immunosuppression. As the trial proceeded the MTQAC functions expanded to ensure that participants assigned to implant therapy were treated in accordance with the protocol as well, the main issues being tapering of oral steroids and timing of implant surgery. At each visit, the MTQAC site visitor reviewed the findings and any concerns with study physicians.

One aspect of systemic treatment is that it affects both eyes. Therefore, patients (and not eyes) were randomized. Hence, patients with bilateral uveitis of sufficient severity to require corticosteroid treatment for both eyes randomized to the implant arm, 64% of enrolled patients, were assigned to receive implants in both eyes. Overall 89% of patients had bilateral disease at enrollment, but uveitis was not always of sufficient severity in both eyes to require bilateral implants. The implant protocol specified insertion of an implant whenever uveitis flared seriously in the fellow eye during follow-up. This approach to treatment of bilateral disease allowed flexibility that reflects clinical practice while preserving validity of our comparison by limiting cross-overs from implant to systemic therapy.

We did not consider masking of treatments to be possible because of ethical considerations regarding the use of sham surgery and the allowable diversity in the treatment regimen for the systemic arm, which included tapering of corticosteroids and, when required, a choice of corticosteroid-sparing agents. Furthermore, masking of treatments is less desirable in CE trials because patients are typically aware of their treatment in the clinical setting.[11]

Choice of primary outcome and secondary outcomes

The primary outcome for the trial was visual acuity because of its importance to patients and the longstanding view that protection of visual acuity is the goal of therapy. Visual acuity as the primary outcome has the advantage of being a definitive clinical outcome rather than a surrogate one. Finally, it is possible for one treatment to provide superior control of inflammation but result in inferior visual acuity because of ocular complications of treatment, such as cataract or glaucoma.

Typical approaches used to characterize visual acuity include threshold events, e.g., a 15 letter loss or gain or loss of visual acuity to a level worse than 20/40.[12] However, the effects of uveitis on visual acuity are variable and, in many instances, vision loss is reversible.[13] Patients with active uveitis present with a wide range of visual acuity in affected eyes, some with significant vision loss that may be reversed if uveitis is controlled before permanent structural damage occurs. An event-type outcome would require restricting enrollment to patients with relatively good vision and would be less inclusive for evaluating effectiveness of a therapy, especially since uveitis is a disease that often lasts for decades. We chose change in visual acuity as the primary outcome so that both gains and losses in visual acuity could be studied and treatment effects over time could be evaluated.

Visual acuity typically is evaluated as an eye-specific outcome, whereas our treatment protocol was designed to evaluate the better treatment choice for a patient; we randomly assigned treatments to patients, not eyes. We chose to evaluate visual acuity in all eyes with uveitis as opposed to choosing a single “study eye” because a key difference in the treatment strategies is that systemic therapy affects both eyes whereas an implant does not. Of the patients with bilateral disease (89% of patients enrolled in the MUST Trial [9]) the minority (29%) did not have active disease in both eyes and were assigned to receive an implant in one eye if randomized to the implant arm. These eyes were eligible for implant if they developed worsening in the second eye at a later date. In such cases, treatment of the initially more severely affected eye with an implant may lead to poor outcomes in the other eye that systemic treatment may have prevented. Hence, we believe including all eyes with uveitis in the final analysis is the fairest comparison, along with an analysis of visual acuity in the better eye regardless of uveitis status. Because we include all eyes with uveitis in the primary outcome analysis, our statistical models must incorporate techniques to account for correlation between eyes from the same patient and among repeated measurements made on the same eye. We use general estimating equations [14] to account for the longitudinal correlation within an eye and bootstrapping to account for two eyes from the same patient. For patients with bilateral disease, it was possible for one eye to show improvement in visual acuity while the companion eye with uveitis lost vision, regardless both eyes were included in the analysis of all eyes with uveitis.

Our choice of change in visual acuity as the primary outcome required that we define a clinically meaningful difference in visual acuity. We first designed the trial to have a detectable difference of 5 letters (equivalent to 1 line on a logarithmic visual acuity chart). However, we were criticized by the FDA and others that this difference was too small to be clinically relevant; furthermore, it required a large sample size. We subsequently revised the sample size of the trial to detect a larger difference, 7.5 letters, which is a difference associated with acceptance and use of new therapies for neovascular age-related macular degeneration. [15,16] Based on the data from the first 100 patients enrolled, we also adjusted our sample size to account for the higher rate of bilateral disease (almost 90% instead of the 67% originally estimated) and lower between-eye correlation of change in visual acuity at 1 year of follow-up (correlation coefficient of 0.4 instead of 0.6). Even with the reduced sample size, i.e., from 400 to 250 participants, the trial is well powered to detect a 7.5 letter difference; ~90% with a 10% inflation factor to account for heterogeneity and losses to follow-up. Including all eyes with uveitis in the primary analysis had an overall beneficial impact on sample size because most patients had bilateral disease (89%) and the between-eye correlation of visual acuity change is relatively low.

In addition, we had to ensure that visual acuity was measured according to a standard protocol and not influenced by knowledge of treatment assignment. Masking was difficult because it was not possible to conceal that an eye had undergone surgery during the early postoperative period and neither the patient nor the ophthalmologist could be unmasked to treatment assignment. We used a well-established protocol for assessing visual acuity [17] and appointed a Visual Function Quality Assurance Committee (VFQAC) to oversee the training and monitoring of visual acuity assessments and facilities according to well-established standards. We also required that there be at least two certified visual acuity examiners at each center – one who did not know patients treatment assignment and another one who could be unmasked to assignment. The unmasked examiner performed assessments during the first six months of follow-up when the immediate effects of implant surgery were more pronounced. All subsequent visual acuity assessments were performed by the visual acuity examiner masked to treatment assignment. This approach allowed the primary outcome, change in visual acuity from baseline to two years, to be ascertained in a masked fashion.

We followed patients for several other protocol defined outcomes that are related to possible adverse effects of treatment (e.g., cataract, glaucoma, diabetes, hypertension, osteopenia or osteoporosis). We also collected data on overall and vision-specific quality of life as well as health utility in order to be able to evaluate patient-centered outcomes and cost-effectiveness of treatments. We selected validated questionnaires that addressed vision specific functioning [18,19] and overall health-related functioning [20], or were established tools for cost-effectiveness analysis. [21,22]

We continue to follow patients beyond their two year anniversary in order to better understand the long term benefits and risks of the two treatment strategies. For example, the implant is associated with increased rates of ocular hypertension requiring surgical intervention, and if not controlled, may lead to glaucoma and vision loss over the longer term.[6,7] Including many protocol-defined secondary outcomes will allow rigorously assessment of the benefits and risks of treatments as well as the cost, which is an important component of CE research.[1]

Access to Implants

Our initial plan for the MUST Trial was to collaborate with the manufacturer of the implant, with our results available to support the NDA for the implant. However, the implant was approved in 2005 based on data from other studies, converting the study to a post-FDA approval study in the United States, but not the UK or Australia. At that point, the company changed its production process in order to scale up production in anticipation of product launch, which resulted in a delay in study implementation for over a year because implants were unavailable. Also, following FDA approval, the company preferred to sell implants to study participants who could afford them rather than to provide them to all participants free of charge. This decision by the manufacturer presented an important dilemma for several reasons. First, the newly approved implant was priced at $18,250, [5] not including the fees associated with surgery, and because the implant was a newly approved product there was uncertainty about insurance or other third-party coverage. There was no possibility that the study funds could cover the cost of all the implants required (conservatively estimated at $4.4 million), which also would have violated NEI policies. Second, the implant was not approved in Australia or the United Kingdom, so it was not available commercially at those sites. Third, enrolling patients in a trial that involves randomization to a drug or a surgical intervention is a difficult proposition and adding the complication of a huge initial cost differential between treatments and the uncertainty about insurance coverage made the task even more formidable. Finally, to assure the validity of the results, we had to be fairly certain prior to randomization that patients assigned to implants actually would be able to receive them, i.e., insurance or some other mechanism would be identified to cover the cost.

We addressed this problem with a multi-pronged approach. First, we negotiated successfully with the company to supply implants without charge for up to 20 patients per year who were unable to obtain them through routine channels (either because of underinsurance, residence in the UK or Australia, or enrollment at the NEI intramural center, which was not allowed to charge patients for treatment). Second, the central resource centers provided support in getting the implants onto hospital formularies and connecting patients who could not afford implants with the indigent patient program run by the manufacturer. Finally, NEI provided us with a “safety net” fund of $50,000 per year. We were allowed to use this fund to cover excessive and unexpected costs associated with the assigned treatment for uveitis that were not covered by either the patient’s insurance or the patient capitation funds. Patients and/or clinical centers were reimbursed for excessive out-of-pocket expenses (the guideline was that only expenses more than $2,000 could use this mechanism). To gain access to these funds, clinics had to get institutional review board (IRB) approval and provide documentation of the expenses. We were not allowed to buy implants with these funds, but could pay for the expenses of implant surgery for patients whose insurance denied coverage. The safety net fund also could be used to cover expenses associated with systemic therapy, which sometimes is costly, in order to promote adherence to the assigned treatment.

Lessons Learned

Including foreign clinics required an additional regulatory task of ensuring and sometimes facilitating the registration of their respective IRB’s with the Office of Human Research Protections. Furthermore, re-importing the implants into the UK required additional labeling and review by a Qualified Person (QP) according to European Union and Medicines and Healthcare products Regulatory Agency (MHRA) regulations. Satisfying this requirement involved contracting with a pharmaceutical services company, providing documents supportive of a QP declaration, and having the clinic submit a Clinical Trials Authorization (CTA) amendment listing the pharmaceutical service company as the importer of the implants. We had to plan shipments of implants well in advance of anticipated need and budget for the additional costs required to comply with local regulatory guidelines. Had the implant been approved in the UK, the process likely would have been easier.

Implementing a defined standard care treatment protocol required training and monitoring. We were fortunate that widely accepted treatment guidelines were in place. Regardless, we devoted substantial resources via the MTQAC site visits to ensure that the patients received the assigned treatments according to the protocol. Usual care of patients with uveitis can vary substantially, sometimes even relying upon repeated corticosteroid injections as a form of local therapy. High standards for systemic therapy, which allowed some local therapy, were necessary in order to maximize the interpretability the results and to ensure a fair comparison. The biggest issue, ironically, had to do with appropriately implementing discontinuation of steroids in the group assigned to implants. We also changed the protocol during the trial to allow more local therapy in the systemic arm based on changing practice. These issues were brought to our attention via data monitoring and MTQAC visits.

Visual acuity was selected as the primary outcome because the objective of treatment is to improve and preserve vision and vision is important to patients. There is an emerging consensus among the ophthalmology research community that treating visual acuity as a continuous outcome is appropriate in many situations.[23] Similar to monitoring therapy, we devoted significant resources to training, certifying, masking and monitoring visual acuity examiners in order to obtain accurate and objective measurement of vision. In retrospect, it may have been preferable to use the Electronic-Visual Acuity system developed by Beck and colleagues.[24] That system would have required more upfront expenditures to equip all clinics but would have reduced training and monitoring requirements and risk of bias due to unmasking. However, having spent the resources to ensure validity of our primary outcome, we are confident of our results.

We evaluated data collected in the beginning of the trial to re-estimate our sample size in an admittedly ad hoc fashion, but without regard to treatment assignment. We adjusted our sample size for a larger detectable difference and updated other key parameters (i.e., rate of bilateral disease and between-eye correlation of change visual acuity) based on observed data. In retrospect, we should have used an adaptive design that included re-assessment of these parameters in the protocol. The potential for such adjustments should be considered in the design phase for all types of trials. By far, the largest contributor to change in the sample size from 400 to 250 was the increase in the detectable difference from 5 to 7.5 standard letters.

Acceptance of participants with missing baseline data increased the need to ensure and document treatment concealment. An astute Data and Safety Monitoring Committee member noted a differential in the proportion of images classified as cannot grade at baseline between treatment groups. On investigation, the imbalance proved most likely due to chance as we could find no evidence of anomalies in the treatment assignment process. The limited data we had on why data were missing also were helpful.

Publically sponsored studies often do not have the funds to provide the treatments under evaluation in CE trials and rely on drug donations from manufacturers or coverage of treatment costs by third-party payers. We had to develop several methods to ensure that patients were able to receive their assigned treatments. One avenue was that the NEI allowed us to distribute a limited supply of monies (safety net) to help participants who otherwise, for financial reasons, could not receive their assigned treatment. We consulted the Research Ethics Consulting Service associated with the Johns Hopkins Institute for Clinical and Translational Research about the safety net fund. Their opinion (paraphrased) was that the safety net was not only ethical, but was favorable from a research ethics perspective because it gave more people the choice to enroll in the trial without being coercive. Nevertheless, we discouraged enrollment of patients without health insurance, because the grant funds would not be able to cover all of their medical expenses related to the treatment of uveitis. This decision was pragmatic.

Discussion

All clinical trials require many decisions to be made about design and procedural questions that affect the validity of results. For the most part, CE trials do not require new types of decisions, but they may require somewhat different decisions depending on the ramifications for generalizability. We have described our rationale for some of the more challenging decisions we made and discussed their effects on generalizability.

Our question was essentially one of CE, the intravitreal implant versus standard care for noninfectious uveitis. Our trial enrolled a more heterogeneous patient population that included patients with more severe disease, had a more rigorous standard care protocol for systemic regimens and more clinically relevant, longer term outcomes than the licensure trials. Despite allowing some variability in treatment, our approach has been to bring similar rigor to the trial execution as a licensure trial in order to preserve validity and limit sample size. This approach allows us to provide unbiased outcomes for a number of important events, both good and bad, under conditions that ensure both groups received treatment that met the standard of care and had objective follow-up assessments.

This type of CE trial is important for establishing the relative merits of available treatment options since licensure trials are usually focused on establishing efficacy and safety, rather than effectiveness. Pharmaceutical manufacturers may have few incentives to conduct trials of new products versus standard care.[25, 26] In our situation the relative merits of different treatment paradigms for an infrequent disease, which may last decades, were not well addressed in the approval process for the new treatment approach, and the initial cost-differential between the approaches is huge. Hence, it was appropriate for the NEI to sponsor a CE trial that was conducted like most licensure trials, e.g., with careful, detailed data collection and standardized procedures, but included a more complete evaluation of clinical outcomes.

There were compromises we had to make that could limit the generalizability of the trial results. Our clinical centers are primarily tertiary care centers and academic medical centers because uveitis is an uncommon disease; it would have been logistically difficult and prohibitively expensive to conduct the MUST Trial at general ophthalmology practices. We trained study ophthalmologists on the treatment protocols and monitored their implementation, which is typically not done in clinical care.[2] However, the treatment guidelines for systemic treatment were those established by an expert panel for uveitis and allowed for flexibility with regard to prednisone tapering and choice of immunosuppressive drugs.[3] We did not allow patients to receive systemic treatment for one eye and an implant for the other, but that approach is likely an unusual one since it makes little sense clinically to expose patients to the risks of both types of treatment. We believe these and other compromises were essential to provide interpretable results that credibly compared the two treatment paradigms, local versus systemic treatment.