PRAGMATISM OF RANDOMIZED CLINICAL TRIALS ON RANIBIZUMAB FOR THE TREATMENT OF DIABETIC MACULAR EDEMA

Randomized clinical trials evaluating ranibizumab for DME were found to be more pragmatic than explanatory. Patients in a “real-life” clinical setting who would be ineligible for landmark trials achieve similar outcomes to eligible patients.

Abstract

Purpose:

To evaluate the pragmatism and generalizability of randomized clinical trials (RCTs) on ranibizumab for diabetic macular edema and determine whether clinical outcomes would differ based on whether or not patients fulfill the eligibility criteria of these RCTs.

Methods:

Pragmatism and generalizability of three RCTs on ranibizumab for diabetic macular edema (DRCRnet Protocols I and T, and RESTORE) were rated using the PRECIS-2 tool. A cohort of consecutive patients with diabetic macular edema was assessed to determine whether clinical outcomes differed based on whether or not patients met the RCT eligibility criteria. Univariable and multivariable regression analyses, adjusted for baseline best-corrected visual acuity, central retinal thickness and number of injections received, were used.

Results:

All RCTs were rated as being more pragmatic than explanatory, with DRCRnet trials being the most pragmatic. Of the 216 eyes (176 patients) included in the cohort, 63% would have met eligibility criteria for Protocol T, 61% for Protocol I, and 56% for RESTORE. When adjusted for best-corrected visual acuity, central retinal thickness, and number of ranibizumab injections received, there were no statistically significant differences in best-corrected visual acuity or central retinal thickness found between “eligible” and “ineligible” patients.

Conclusion:

Randomized clinical trials evaluating ranibizumab for diabetic macular edema were more pragmatic than explanatory. “Ineligible” patients still benefited from ranibizumab therapy.

Visual loss due to diabetic macular edema (DME) is a common complication of diabetic retinopathy, with an estimated global prevalence of 6.8% in all individuals with diabetes.^1–3 Several randomized clinical trials (RCTs) found intravitreal injection of ranibizumab to be effective for the treatment of DME^4–6 and more efficacious than macular laser photocoagulation.^7,8 The National Institute for Health and Care Excellence in the United Kingdom appraised ranibizumab for the treatment of DME.⁹ The National Institute for Health and Care Excellence found ranibizumab to be cost-effective only for patients with DME and central retinal thickness (CRT) of ≥400 μm. In patients with CRT <400 μm, laser prevailed, and macular laser is therefore recommended and remains the treatment of choice for this group of patients with DME in the United Kingdom.

Properly designed and adequately conducted RCTs are considered to provide the best-available evidence to guide patient care, only surpassed by meta-analysis and individual participant data meta-analysis of randomized trials. Randomized trials can be designed as pragmatic or explanatory.¹⁰ Pragmatic RCTs are primarily designed to determine the effectiveness of an intervention under the usual conditions in which this intervention will be applied.¹¹ By contrast, explanatory trials are designed to determine the efficacy of an intervention under ideal circumstances; known biases and confounders are meticulously controlled for to maximize the ability to reveal any therapeutic effect of the agent being evaluated. As a result, it is difficult to determine whether the results obtained in explanatory trials could be reproduced in a “real-life” clinical setting.

Many trials have both pragmatic and explanatory elements. Thorpe et al devised the pragmatic-explanatory continuum indicator summary (PRECIS) tool to assist trial design by clarifying whether a given research methodology is pragmatic or explanatory, and to ensure design decisions matched the intended trial purpose.¹¹ This was later refined to create PRECIS-2, developed with the help of more than 80 international trialists, clinicians, and policymakers.¹² The PRECIS-2 has 9 domains, including eligibility, recruitment, setting, organization, flexibility: delivery, and flexibility: adherence, follow-up, primary outcome, and primary analysis (Table 1), which are scored on a 5-point Likert scale, with one being very explanatory and 5 very pragmatic according to guidelines provided.¹² Although created to assist trialists to design trials that would align with the trial's intended purpose (pragmatic or explanatory), the PRECIS-2 tool has been used also to rate the pragmatism of already conducted RCTs with good interrater reliability.¹³

Table 1.

PRECIS-2 Tool: Domains and Their Description

When making decisions involving treatment, clinicians need to consider available evidence and determine whether this evidence is relevant to the particular patient to be treated. However, patients evaluated in clinical practice and in need of treatment will not always fulfill the criteria set in the trials providing the evidence. In this case, the efficacy of the particular treatment under consideration is unknown. The more pragmatic the trials from which the evidence has been gathered, the more likely it is that the trials' outcomes would be applicable to patients seen in clinical practice.

The current study aimed at assessing the pragmatism of landmark RCTs evaluating ranibizumab for the treatment of DME, using the PRECIS-2 tool, determining what proportion of patients seen in clinic meet the eligibility criteria set in these RCTs, and evaluating whether outcomes differ based on whether or not patients fulfill the eligibility criteria.

Methods

Ranibizumab trials were chosen for evaluation, as this drug was the first licensed anti–vascular endothelial growth factor agent approved for the treatment of DME, and subsequently, most patients in our clinics would have received this treatment during the study period.

Three published landmark RCTs investigating the efficacy of intravitreal ranibizumab for the treatment of DME were selected, as follows:

1. The Protocol I of the Diabetic Retinopathy Clinical Research Network (DRCRnet) comparing ranibizumab plus prompt or deferred laser or triamcinolone plus prompt laser for the treatment of DME.⁷

2. The Protocol T of the DRCRnet comparing aflibercept, bevacizumab, and ranibizumab for DME.⁶

3. The RESTORE study, comparing ranibizumab monotherapy or combined with laser versus laser monotherapy for DME.⁸

RISE and RIDE were also RCTs evaluating ranibizumab for the treatment of DME. In RISE and RIDE, injections of ranibizumab were given monthly to all patients for a period of 2 years. This strategy differs greatly from that used in clinical practice, and thus, it was decided not to include these RCTs in the current study.

The PRECIS-2 tool was used to score the three selected RCTs; 2 independent graders (N.L. and G.V.) scored each trial based on the nine PRECIS-2 domains (Table 1). The average of the scores given by each of the graders to each of the included trials where plotted in the PRECIS-2 wheel. Scores were allocated considering the guidance provided by PRECIS-2.¹² If the scores given by the graders differed considerably (i.e., larger than two points), discrepancies were discussed and scores were moderated accordingly.

For all three trials, “flexibility: adherence” was deemed not to be relevant, as the intervention was being administered directly by the health care provider, rather than being dependent on patient's self-administration. For this reason, this domain was not scored, and the maximum possible overall score was 40 rather than 45.

A cohort of patients was established to address the second objective of the study, namely the proportion of patients meeting the eligibility criteria set on the selected RCTS and whether outcomes after treatment with ranibizumab differ based on whether or not patients fulfill eligibility criteria set in the three selected RCTs. This patient cohort consisted of all consecutive eligible patients evaluated in a specialized DME clinic at the Belfast Health and Social Care Trust, Belfast, Northern Ireland, who had received their first dose of ranibizumab between March 1, 2014, and March 31, 2015. Patients were identified through an electronic diagnostic database. To be eligible, they should have received the minimum loading dose (considered to be three injections) and be followed for a minimum period of 12 months. Patients were excluded if they received any other intravitreal agent, such as aflibercept or dexamethasone, during the follow-up period. The study was registered with the Audit Department of the Belfast Health and Social Care Trust (audit approval number 5170) and was deemed exempt from full ethical review. The authors confirm that data collection conformed to the local policy at the Belfast Health and Social Care Trust. Patients considered eligible for this audit were given a study number to ensure confidentiality was maintained.

In keeping with procedures of standard clinical practice, all patients had an ocular examination, which included refraction at baseline and best-corrected visual acuity (BCVA) measured with Early Treatment Diabetic Retinopathy Study (ETDRS) visual acuity charts, intraocular pressure check, and slit-lamp biomicroscopy. Spectral domain optical coherence tomography scans were routinely obtained by trained ophthalmic photographers at every clinic visit using commercially available equipment (Heidelberg Spectralis; Heidelberg Engineering, Heidelberg, Germany). Intravitreal injections were performed in a clean room; topical anesthesia and povidone 5% were used in all cases before the procedure. The treatment protocol consisted of three ranibizumab injections at 4-weekly intervals followed by a pro re nata regimen thereafter.

All three RCTs selected provided clear eligibility (inclusion and exclusion) criteria. Based on these inclusion and exclusion criteria, which included ocular and systemic factors, each patient in the established cohort was evaluated by two investigators (S.S. and J.L.Y.) to determine whether or not they would have been eligible to be included in each of these trials. A patient was considered to have met the inclusion criteria only if all selection criteria (ocular and systemic) were fulfilled at the time of initiation of ranibizumab treatment. For patients who did not meet the eligibility criteria of the landmark trials, the reasons for exclusion were recorded.

For all patients, the following information was extracted: age, sex, diagnosis (Type I or Type II diabetes mellitus), visual acuity (ETDRS letter score) at baseline and at 12 months, number of injections by 12 months, and CRT at baseline and at 12 months.

The following outcomes were determined and compared between “eligible” (fulfilled all eligibility criteria) and “ineligible” (did not fulfill one or more of the eligibility criteria) patients: change in BCVA from baseline to Month 12 and change in CRT from baseline to Month 12.

Univariable and multivariable regression analyses, taking into account within-subject correlation as random effects in linear mixed models, were undertaken to assess differences in outcomes between “eligible” and “ineligible” patients of the patient cohort. Multivariable analyses were adjusted by progressively adding baseline BCVA, baseline CRT, and number of injections received to the univariable model that included eligibility status only. All analyses were conducted using Stata 15.1 (StataCorp, College Station, TX).

Results

Overall, the three RCTs selected were scored as more pragmatic than explanatory (Table 2 and Figure 1). The DRCRnet Protocol T study⁶ was scored as the most pragmatic of all the RCTs selected, with the RESTORE study⁸ being rated as least pragmatic overall. All three trials were rated as rather pragmatic or very pragmatic for eligibility, recruitment, setting, organization, primary outcome, and primary analysis.

Table 2.

PRECIS-2 Scores for Landmark Trials

Fig. 1.

PRECIS-2 wheels illustrating the pragmatism of the RCTs included in the study, generated through www.precis-2.org. A. Protocol I, DRCRnet.⁷ B. Protocol T, DRCRnet.⁶ C. RESTORE study.⁸.

Flexibility in delivery of the intervention and follow-up was felt to be rather explanatory for all three trials, as monthly appointments are difficult or not feasible in many public health care systems, such as those established in Europe. The treatment and retreatment criteria set in the RESTORE study⁸ was considered to be less stringent than those set out by the DRCR network; therefore, this domain (flexibility: delivery) was scored as more pragmatic than that of the DRCRnet trials.

With respect to subject eligibility, the graders made note of the lower tolerance for blood pressure (BP) levels in the RESTORE study; BP levels for included patients had to be <160/101, compared with <181/111 for the DRCRnet trials, which was felt to be a stricter limit than that applied in most clinical settings.

A cohort of 176 patients (216 eyes) was established, based on the criteria set and presented in the “Methods” section (see above). Demographics and baseline characteristics of the patients included in this cohort have been summarized in Table 3.

Table 3.

Demographics and Characteristics of the Study Cohort

Almost three-quarter of the eyes in the cohort were found to fulfill the eligibility criteria set in at least one of the three trials included (154/216, 71.3%); just under half (102/216, 47.2%) were found to be eligible for all three (Table 4). The greatest proportion of eyes met eligibility criteria set in DRCRnet Protocol T (63.0%), with least number of eyes fulfilling the eligibility criteria set in the RESTORE study (55.6%). Approximately 60.6% of eyes were eligible for DRCRnet Protocol I.

Table 4.

Proportion of Study Cohort Eyes and Patients Meeting Eligibility Criteria for Entering DRCRnet Protocols I and T and the RESTORE Study

Overall, 74 (34.3%), 69 (31.9%), and 73 (33.8%) eyes did not fulfill eligibility criteria to be included in DRCRnet Protocols I and T and the RESTORE study, respectively, for ocular reasons. The main reasons for exclusion were low or high BCVA (for Protocols I and T, DRCR.net: n = 26, 10%; for RESTORE: n = 44, 20.4%) and panretinal photocoagulation undertaken within 4 to 6 months of inclusion (for Protocols I and T, DRCR.net: n = 31, 14.5%; for RESTORE: n = 27, 12.5%). For both the DRCRnet trials, eyes were most often excluded because of ocular factors, particularly recent laser (Table 5). No eye was excluded due to inflammation or infection or other ocular comorbidity.

Table 5.

Most Frequent Reasons for Patients in the Study Cohort Not to Meet Eligibility Criteria Set in DRCRnet Protocols I and T and the RESTORE Study

Overall, 15 (6.9%), 15 (6.9%), and 45 (20.8%) eyes [of 11 (6.3%), 11 (6.3%), and 33 (48.8%) patients] did not fulfill eligibility criteria set in DRCRnet Protocols I and T and the RESTORE study, respectively, because of systemic factors. The RESTORE study had a more stringent BP requirement (<160/100), and this alone excluded 13.9% (30/216) of eyes from the study cohort.

The entire patient study cohort had an average baseline ETDRS letter score of 60.3 (±16.3) and gained an average of 6.4 letters (4.9–7.9; 95% confidence interval [CI]) by 12 months.

Univariable regression analysis demonstrated that eligible patients, on average, gained more letters by Month 12 than ineligible patients (Tables 6 and 7). This was most pronounced for the DRCRnet Protocol T study in which patients meeting eligibility criteria gained on average 7.5 letters when compared with 4.6 letters in those that did not fulfill them.

Table 6.

Baseline Characteristics of the Study Cohort and Univariable Regression Analysis Comparing Outcomes of “Eligible” and “Ineligible” Patients Based on Eligibility Criteria Set in DRCRnet Protocols I and T and the RESTORE Study

Table 7.

Multivariable Regression Models Comparing Patients Fulfilling (“Eligible”) and Not Fulfilling (“Ineligible”) Eligibility Criteria Set in the DRCR.net Protocol I and T Studies and the RESTORE Study

The average baseline CRT for the study cohort was 484 μm (±134), with an average change in CRT of −152 μm (−178 to −125; 95% CI) by 12 months. No statistically significant difference in CRT change was detected between eligible and ineligible subjects for any set of eligibility criteria.

Differences in BCVA or CRT showed modest changes in multivariable models progressively adjusting for baseline BCVA, CRT, or the number of injections, with no statistically significant differences detected.

Considering the potential effect of a broader range of baseline BCVA in ineligible patients compared with eligible patients (5–85 letters [Snellen equivalent 20/800–20/20] for the study cohort versus 24 to 78 letters [Snellen equivalent 20/320–20/32] for Protocol I; 24 to 78 letters [Snellen equivalent 20/320–20/32] for Protocol T; and 39 to 78 letters [Snellen equivalent 20/160–20/32] for RESTORE study), which may have been difficult to control for in regression analyses, we then calculated the mean difference in BCVA gain in a group of 179 eyes of 151 patients who had baseline BCVA between 34 (Snellen equivalent of 20/200) and 78 letters (Snellen equivalent 20/32). For this group also, results remained unchanged, and no statistically significant differences in BCVA gain were observed between eligible and ineligible eyes, when adjusted for baseline BCVA, CRT, and number of injections received (−2.2 letters, 95% CI: −5.4 to −0.9 letters, P = 0.167 for DRCRnet Protocol I, −3.2 letters, 95% CI: −6.4 to 0.1 letters, P = 0.056 for DRCRnet Protocol T, and −2.4 letters, 95% CI: −5.5 to 0.7 letters, P = 0.133 for the RESTORE study).

In a linear mixed model, adjusting for baseline BCVA and baseline CRT, exclusion due to any ocular criterion was not associated with visual change for any study (P > 0.1), ruling out effects larger than ± 3 letters. However, exclusion due to any systemic criterion was associated with −12 letters worse outcome in DRCRnet Protocols I and T (P < 0.001 for both) but not in the RESTORE study (−2.4 letters, P = 0.160) in ineligible patients. This finding remained when adjusting for the number of injections received.

Among ocular criteria considered simultaneously in a regression model, only the single individual with visual gain of 52 letters in whom DME was not the primary cause of visual loss (Table 5) significantly influenced overall change of vision (P < 0.001). After the exclusion of this subject, among systemic criteria with at least one count in the model and after adjusting for baseline BCVA and CRT, according to DRCRnet Protocols I and T, previous cardiovascular events accounted for worse BCVA change by −15 letters (95% CI: −27 to −7, P < 0.001) and elevated systemic BP accounted by −9 letters (95% CI: −16 to −3; P = 0.007). For the RESTORE study, there was no exclusion due to recent cardiovascular events, and the threshold criterion for exclusion due to BP criteria was lower and was not associated with BCVA change (−3 letters, 95% CI: −6.8 to +0.7 letters, P = 0.092).

Discussion

All three selected landmark RCTs evaluating ranibizumab for DME, namely DRCRnet protocols I and T and the RESTORE study, were found to be more pragmatic than explanatory using the PRECIS-2 tool, with Protocol T being the most pragmatic of all. The RESTORE study was felt to be less pragmatic overall, mainly due to stricter eligibility criteria, excluding patients with BP >160/100 mmHg, recent change in antihypertensive medications or a previous history of stroke. Both DRCRnet Protocols I and T were considered to be very pragmatic in terms of their criteria for patient eligibility.

Despite the fact that all selected RCTs were graded as being more pragmatic than explanatory, it was found that only 55.6% to 63.0% of eyes fulfilled the eligibility criteria of the three RCTs selected. Nearly 30% of eyes did not meet eligibility criteria for any of these RCTs. Around a quarter of eyes failed to meet the inclusion criteria set in these RCTs because they either had a level of vision outside the established limits in the RCTs or because they had recently received panretinal photocoagulation, indicating that there was proliferative diabetic retinopathy present in addition to DME.

The difference in visual acuity gain between eligible and ineligible patients consistently favored eligible patients by about two ETDRS letters but never reached statistical significance. Of interest, point estimates were unchanged after progressively adjusting by baseline BCVA, CRT, and number of intravitreal injections received. The precision of these estimates ruled out a difference in mean BCVA gain by no more than 5 letters favoring eligible patients, a limit that is acceptable to rule out clinically important differences in visual change.¹⁴ However, differences in CRT between eligible and ineligible patients were negligible, and their precision excluded maximum differences by no more than 50 μm. All this considered, we conclude that trial ineligibility in our clinical setting did not seem to have a meaningful or large effect on visual and anatomical outcomes in patients with DME undergoing of ranibizumab treatment.

The subgroup of patients with ocular exclusion criteria, mainly high or low visual acuity, did not show differences in visual outcomes when baseline BCVA was accounted for. On the contrary, patients with systemic exclusion criteria, particularly high BP or a change in antihypertensive medication, showed significantly less gain in vision than the others. This subgroup analysis, however, was exploratory and should be considered with caution and confirmed in further studies, given the relatively small number of patients in this subgroup. Nonetheless, there is strong evidence from cohort studies that uncontrolled BP increases the chances of progression of diabetic retinopathy.^15–17 Our study suggests that a sizeable fraction of patients in need of anti–vascular endothelial growth factor injections for DME may suffer from visual loss despite appropriate treatment due to systemic cardiovascular disease and uncontrolled BP. This finding underlines the importance of an integrated approach by both diabetologists and ophthalmologists to patients with vision-threatening diabetic retinopathy.

Despite the high proportion of treated patients in the clinical cohort presented herein who did not meet the eligibility criteria set in the DRCR.net Protocols I and T and RESTORE, no statistically significant differences in outcomes were found between “eligible” and “ineligible” patients. This finding is reassuring for clinicians. Some of the reasons for excluding patients from clinical trials may not relate to a predicted lack of efficacy in groups of patients excluded but more so to their potential confounding effect. It has been shown that people with a higher baseline visual acuity will be expected to have higher final vision after treatment,^18–20 which would support initiating early treatment, when vision is still good. However, there would be less chance for an improvement in visual acuity, given the good level of vision at baseline. In the ETDRS, macular laser photocoagulation was shown to prevent visual loss but did not improve vision; 85% of patients included in this trial had visual acuity of ≥20/40 at baseline.²¹ Similarly, patients in the Protocol T study with an initial visual acuity score between 69 and 78 letters (Snellen equivalent 20/40–20/32) gained significantly fewer letters than those with an initial letter score of 69 letters (≤20/50) or less.⁶ A DRCRnet trial on anti–vascular endothelial growth factor (anti-VEGF) therapy in people with very good levels of vision is now underway and would provide information in the future on this group.²²

This study has several limitations, including the retrospective nature of the study cohort and the short-term follow-up at which outcomes were tested (12 months). Although we included a large number of eyes in the study, an even larger cohort would have increased the power of our study. A post hoc power calculation considering 72 ineligible versus 144 eligible eyes with an SD of 16 letters, as we observed at baseline, would have had 80% power to detect a difference of 6.5 ETDRS letters and 90% power to detect a difference of 7.5 letters. Thus, the sample size included should have given us enough power to detect clinically relevant differences if these had existed.

Strengths include the identification of consecutive eligible cases through an electronic database, the relatively high number of eyes included in the analysis (n = 216) and the very detailed evaluation of inclusion and exclusion criteria set in each of the three selected RCTs for each patient in the study cohort. When examining the differences in outcomes between eligible and ineligible patients, it may have been interesting to compare the influence of HbA1c as marker of glycemic control. However, HbA1c has not been found to affect visual outcomes in patients with DME treated with intravitreal anti-VEGF factors.^19,20,23

Many patients with DME treated with ranibizumab in the clinical setting did not meet the eligibility criteria set in the RCTs demonstrating the efficacy of this treatment. Despite this, no apparent differences in outcomes (BCVA and CRT) were found between individuals fulfilling eligibility criteria and those that did not. This information may be used by clinicians to decide whether or not to initiate treatment with ranibizumab in patients with DME in their clinical practice and for the counseling of these patients.