Abstract
Objective
To evaluate the success of masking study participants to treatment allocation using sham intravitreal injections.
Methods
Eyes were randomized to prompt laser plus sham injections, prompt laser plus intravitreal ranibizumab injections, deferred laser plus intravitreal ranibizumab injections, or prompt laser plus intravitreal triamcinolone injections up to every 16 weeks with sham injections intermittently. All eyes could receive treatment or sham as often as every 4 weeks. Participants with 2 study eyes had one eye randomized to sham+laser and one eye randomized to a real injection group. Sham injections were performed by pressing the syringe hub against the conjunctiva to mimic a real injection. Laser treatment was not masked. At the 1 year visit, participants were asked if they believed the injections received during the study were real, sham, or sometimes real and sometimes sham.
Results
Among 423 participants with one study eye, the correct assignment was stated by 10% of the sham+prompt laser group, 88% of the ranibizumab+prompt laser group, 90% of the unmasked ranibizumab+deferred laser group, and 44% of the triamcinolone+laser group. Among the 112 participants with 2 study eyes, the correct assignment was stated for 24% of the sham+prompt laser eyes.
Conclusions
Successful masking of an intravitreal injection can be accomplished when a sham injection procedure carefully mimics a true injection procedure. Masking appears less successful when one eye is receiving a real injection and the other eye is receiving a sham injection or an individual eye sometimes receives a real and sometimes a sham injection.
Introduction
Minimizing bias in a randomized clinical trial is a critical consideration during study design. Randomization alone does not ensure an unbiased experiment. One important factor to consider in designing a randomized trial is whether to mask the participant to the treatment group assignment. The subjectivity of the outcome, type of treatment, and disease should be considered when determining whether study participants should be masked. For example, visual acuity, which is the primary outcome measure for many ophthalmic trials, is a subjective measurement where the results can be influenced by subject effort. Therefore, subject knowledge of treatment assignment could bias the measurement. Masking the study participants to the treatment group assignment is also important when knowledge of treatment group could affect a participant’s behavior in such a way that it could influence the course of the disease or have an effect on the trial’s primary outcome.1 Although usually desirable, masking of participants is not always feasible and in some cases, is impossible.
Numerous randomized clinical trials recently have compared intravitreal injections with standard treatment in several retinal diseases including diabetic macular edema, age-related macular degeneration, and macular edema from retinal vein occlusions. 2–4 In studies evaluating intravitreal injections, sham injections are sometimes used to attempt to mask the study participant to treatment assignment.5, 6 In general, a sham intravitreal injection is a procedure that mimics a real intravitreal injection but does not penetrate the eye. In contrast, some studies use a placebo intravitreal injection, in which an inert substance like saline is injected into the vitreous.7 Although a sham injection may be less effective at masking a study participant compared with a placebo injection, a sham injection is more often used since it decreases the risk of injection related complications. Sham intravitreal injections have been used frequently YET, the masking success of the sham injections has not been reported to our knowledge. The Diabetic Retinopathy Clinical Research Network (DRCR.net) conducted a clinical trial evaluating intravitreal ranibizumab or intravitreal triamcinolone for diabetic macular edema (DME) in which the success of masking of the participants was evaluated. The evaluation of the masking is reported herein.
Methods
The success of masking of sham injections was assessed in the DRCR.net trial “Intravitreal Ranibizumab or Triamcinolone Acetonide in Combination with Laser Photocoagulation for Diabetic Macular Edema” which was a phase 3 randomized clinical trial conducted at 52 clinical centers across the United States. Study participants with one study eye were randomized to 1 of 4 different treatment groups: (1) sham injection plus prompt focal/grid photocoagulation (sham+prompt laser), (2) 0.5 mg intravitreal ranibizumab plus prompt focal/grid photocoagulation (ranibizumab+prompt laser), (3) 0.5 mg intravitreal ranibizumab with deferred focal/grid photocoagulation (ranibizumab+deferred laser), and (4) 4 mg intravitreal triamcinolone plus prompt focal/grid photocoagulation with triamcinolone injections up to every 16 weeks with sham injections intermittently (triamcinolone +prompt laser). Study participants with 2 study eyes were randomly assigned to receive either ranibizumab+prompt laser, ranibizumab+deferred laser or triamcinolone+prompt laser in one eye and sham+prompt laser in the other. The details of the trial have been reported previously.8 Of note, a prior real intravitreal injection was not an exclusion criteria provided treatment was at least 4 months prior to randomization.
Treatment and Masking
To minimize potential bias on measurement of the study outcome, study participants were masked through the 1-year primary outcome except for those with an eye assigned to the ranibizumab+deferred laser group. This group was not masked since at the time the study was developed, the investigators believed that a reliable sham laser treatment to mask study participants was not feasible and unlikely to be successful. Visual acuity (primary outcome) examiners were masked to treatment allocation for all groups at the 1 year primary outcome visit. Investigators were not masked to treatment assignment.
Study drug or sham injections were given as frequently as every 4-week study visit. When retreatment with a study drug or sham injection was indicated, eyes assigned to one of the ranibizumab groups could receive ranibizumab as often as every 4-week visit; eyes assigned to intravitreal triamcinolone could receive triamcinolone as often as every fourth 4-week study visit (i.e. approximately every 16 weeks) with sham injections as often as every 4-week visit in between triamcinolone injections; eyes assigned to sham+prompt laser could receive sham injections as often as every 4-week visit. There were 13 maximally possible sham or intravitreal injections prior to the 1-year primary outcome visit. The median (25th, 75th percentile) number of injections through the first year was 11 (8, 13) sham injections in the sham+prompt laser group, 8 (6, 10) ranibizumab injections in the ranibizumab+prompt laser group, 9 (6, 11) ranibizumab injections in the ranibizumab+deferred group, and 5 (3, 7) sham injections and 3 (2, 4) triamcinolone injections in the triamcinolone+prompt laser group.
Injection Procedure
All pre- and post-injection procedures were identical for both sham and intravitreal injections. Both procedures required the use of a lid speculum and were preceded by a povidone iodine prep of the conjunctiva directly over the site to receive the injection. For a sham injection, the hub of a syringe without a needle was pressed against the conjunctival surface to simulate the force of an actual injection. Indirect ophthalmoscopy was required post-injection to confirm that the central retinal artery was perfused and to assess any complications. Use of antibiotics in the pre-, peri-, or post-injection period was at investigator discretion. The frequency of pre and post injection antibiotic use was similar when a real injection was performed compared with when a sham injection was performed. Both post-injection and pre-injection antibiotics were given 33% vs. 25% of the time, only pre-injection antibiotics were given 10% vs. 12% of the time, and only post-injection antibiotics were given 21% vs. 15% of the time for a real injection compared with a sham injection respectively. Post-injection intraocular pressure (IOP) measurement was also at the discretion of the investigator. Intraocular pressure measurement was performed 41% of the time with real injections and 35% of the time with sham injections.
Assessing Masking Success
At the completion of the 52-week primary outcome visit, each study participant was asked a question to assess the effectiveness of the masking technique. Study coordinators or investigators were instructed to read the following question aloud to the subject: ‘Do you think the injections you have been getting in the [right/left] eye during the study have been: (1) real injections into the eye; (2) sham injection, meaning that a needle has not been injected into the eye, (3) sometimes real and sometimes sham.” Each study participant was to be instructed to make his or her best guess as to treatment allocation if the study participant was uncertain. If the study participant refused to choose one of the three choices, this was recorded. The validity of the masking question has not been established. The masking question was implemented in April 2008; 55 participants with 71 study eyes completed their primary outcome visits prior to implementation of the masking question and therefore did not have an opportunity to respond and 38 study participants with 46 study eyes did not complete the 1 year visit and therefore did not complete the masking question. Additionally, 15 participants with 21 study eyes were excluded where the study participant received a non-study treatment for DME in at least one eye.
Results
Of 565 randomized participants eligible for this analysis, 125 had 2 study eyes totaling 690 study eyes. The treatment group assignments for participants with one study eye are shown in Table 1 and for participants with two study eyes in Table 3. Overall, 44% of the eligible participants were women, with an average age of 63 years (±10) and average visual acuity letter score of 63 (±12) which was an approximate Snellen equivalent of 20/63. Seventy-three percent of the eligible cohort was white, 16% was African American, and 9% was Hispanic or Latino. One-hundred Sixty (23%) of the eyes had received a prior intravitreal injection for DME. Baseline characteristics of the full study cohort have been reported previously.8
Table 1.
Study Masking Questionnaire Response – Participants with 1 Study Eye
| Sham + laser | Ranibizumab+ prompt laser | Ranibizumab+ deferral laser | Triamcinolone + laser | |
|---|---|---|---|---|
| Participant Response | N = 105 | N = 105 | N = 112 | N= 118 |
| Refused to Provide One of the Responses Below | 4 (4%) | 5 (5%) | 6 (5%) | 2 (2%) |
| Participant Response | N=101 | N=100 | N=106 | N=116 |
| Always Real Injection | 73 (72%) | 88 (88%) | 95 (90%) | 64 (55%) |
| Always Sham Injections | 10 (10%) | 0 | 4 (4%) | 1 (1%) |
| Sometimes real sometime sham | 18 (18%) | 12 (12%) | 7 (7%) | 51 (44%) |
Table 3.
Study Masking Questionnaire Response – Participants with 2 Study Eyes
| Ranibizumab+ prompt laser | Ranibizumab + deferral laser | Triamcinolone + laser | |
|---|---|---|---|
| Participants Questioned | N = 46 | N = 42 | N= 37 |
| Refused to Answer | 2 (4%) | 9 (21%) | 2 (5%) |
| Participant Response for Injected Eye | N=44 | N=33 | N=35 |
| Correct Response Injection Eye | 37 (84%) | 29 (88%) | 11 (31%) |
| Incorrect Response Injection Eye | 7 (16%) | 4 (12%) | 24 (69%) |
| Participant Response for Sham Eye | N=44 | N=33 | N=35 |
| Correct Response - Sham Eye | 13 (30%) | 9 (27%) | 5 (14%) |
| Incorrect Response - Sham Eye | 31 (70%) | 24 (73%) | 30 (86%) |
| Participant Response Overall | N=44 | N=33 | N=35 |
| Both Eyes Correct | 13 (28%) | 8 (19%) | 1 (3%) |
| Sham Eye Correct/Injection Eye Incorrect | 0 | 1 (2%) | 4 (11%) |
| Sham Eye Incorrect/Injection Eye Correct | 24 (52%) | 21 (50%) | 10 (27%) |
| Neither Eye Correct | 7 (15%) | 3 (7%) | 20 (54%) |
Masking Success
At the 1-year primary outcome visit, study participants were asked to indicate which treatments the participant believed he/she had received. Seventeen study participants with one study eye (4%) and 13 study participants with two study eyes (10%) refused to guess the treatment assignment (Table 1). Among the 423 study participants with one study eye who completed the masking question and who had only received the randomized treatment (Table 1), 10% of the sham+prompt laser group believed sham was given all the time (meaning 90% thought they received real injections), while the study participants believed a real injection was always given for 88% of the ranibizumab+prompt laser group and 90% of the ranibizumab+deferred laser group (correct response), and a real injection was sometimes given in 44% of the triamcinolone+prompt laser group (correct response). Across all 4 treatment groups, only 4% of participants indicated a belief that sham injections were always given. Correctness of response did not appear different for any of the following subgroups within treatment group: prior injection for DME, prior laser for DME, baseline visual acuity, age, gender, race and ethnicity, whether treatment was given at the 48 week visit (visit prior to masking assessment), whether the participants received 3 or more injections between the 28–48 week visit (inclusive), or whether the site was an academic- or community-based center (Table 2).
Table 2.
Percentage of Correct Response by Baseline Subgroups – Participants with 1 Study Eye
| Baseline Subgroup | N | Sham + laser Correct Response | Ranibizumab+ prompt laser Correct response | Ranibizumab+ deferral laser Correct Response | Triamcinolone + laser Correct Response |
|---|---|---|---|---|---|
| Percentage with a Correct Response | |||||
| Prior Intravitreal Injection for DME | |||||
| Prior Injection | 24,25,33,29 | 4% | 92% | 88% | 41% |
| No Prior Injection | 77,75,73,87 | 12% | 87% | 90% | 45% |
| Age | |||||
| <65 | 59,56,60,72 | 12% | 89% | 92% | 46% |
| >=65 | 42,44,46,44 | 7% | 86% | 87% | 41% |
| Visual Acuity | |||||
| <=65 | 53,56,49,57 | 11% | 89% | 88% | 42% |
| >=66 | 48,44,57,59 | 8% | 86% | 91% | 46% |
| Gender | |||||
| Woman | 43,47,46,52 | 5% | 91% | 91% | 44% |
| Man | 58,53,60,64 | 14% | 85% | 88% | 44% |
| Race/Ethnicity | |||||
| White | 71,68,81,85 | 13% | 84% | 93% | 47% |
| African American | 19,21,13,18 | 5% | 100% | 85% | 33% |
| Hispanic/Lati no | 10,8,10,8 | 0% | 88% | 70% | 38% |
| Center Type | |||||
| Academic Based | 32,30,27,33 | 13% | 93% | 96% | 33% |
| Community Based | 69,70,79,83 | 9% | 86% | 87% | 48% |
| Number of Injections between 28–48 weeks (inclusive) | |||||
| ≤ 2 | 28,55,46,64 | 4% | 91% | 83% | 38% |
| ≥ 3 | 73,45,60,52 | 12% | 84% | 95% | 52% |
| Study Treatment at 48 Week Visit | |||||
| No | 35,72,57,80 | 6% | 88% | 90% | 43% |
| Yes | 66,28,49,36 | 12% | 89% | 90% | 47% |
Among the 112 participants with 2 study eyes (Table 3), 24% of the study participants believed a sham injection was always given in the eye that received only sham injections (meaning 76% thought a real injection was received in the sham eye). Among the 79 participants with 2 study eyes from the masked treatment groups (i.e. excluding participants with either eye in the ranibizumab+deferred laser group), 23% of the participants believed a sham injection was always given in the eye that received only sham injections. For the eye receiving a real injection at least sometimes, the correct assignment was stated for 84% of the ranibizumab+prompt laser eyes, 88% of the ranibizumab+deferred laser eyes, and 31% of the triamcinolone eyes.
Relationship of Masking and Visual Acuity Outcome in the Sham+Laser Group
In participants with one study eye in the sham+laser group, a positive visual acuity response to treatment did not significantly impact the percentage of correct responses (p-value = 0.63). The participants believed real injections were given always or sometimes 90% of the time when visual acuity improved 5 or more letters (N=60), 93% of the time when visual acuity changed by 4 or fewer letters (N=29), and 83% of the time when visual acuity worsened by 5 or more letters (N=12). Thus, even for the 41 eyes with no apparent improvement who had sham+laser, 37 (90%) still believed a real injection had occurred. Similarly the mean visual acuity letter score change for an eye in the sham+laser group was 6.0±9.4 for participants who thought that real injections had been received and 3.6±8.1 for participants who thought that only sham injections had been received (p-value=0.45).
Discussion
Study participant masking of treatment assignments is an important factor to consider when assessing potential bias in a trial. This study evaluated how well a sham intravitreal injection would mask participants as to whether a real or sham injection was given. The results indicate that very few participants believed they received only sham injections in a randomized trial comparing sham intravitreal injections with real intravitreal injections. Furthermore, in the treatment group that at some visits received a real injection and other visits a sham injection, a majority of participants appeared to believe that they received a real injection at times when they received a sham injection. Even when visual acuity during the course of the study did not improve or worsened, most participants thought that they had received real injections
Compared with the participants who received sham injections only, participants in the treatment group that received both real and sham injections in the same eye less often thought that real injections were always given. This may imply that a participant is more likely to identify a sham injection when also receiving a real injection in the same eye. However, the visual disturbances caused by the opaqueness of the triamcinolone may have contributed to this decreased masking. However, since over half of the participants in the group receiving both real and sham injections believed real injections were always given, it does appear that masking was successful even in this circumstance.
Across all treatment groups masked per protocol, these results suggest that it may be more likely that a participant with 2 study eyes would believe sham injections were always given in the sham eye than participants in the sham group with only 1 study eye (23% vs. 10% respectively, p-value = 0.02). Nevertheless, 77% of the participants with 2 study eyes thought that the eye receiving only sham injections received real injections.
Just because masking is successful does not mean it is necessary, i.e. that unmasking of the study participants will result in bias. The effect of masking intravitreal injections on visual acuity and patient-reported visual function outcomes has been evaluated in subfoveal choroidal neovascularization secondary to age-related macular degeneration.9, 10 Hawkins et al showed in some scenarios that visual acuity outcomes in an unmasked cohort were similar to a matched masked cohort from a subsequent trial. It is unknown, however, if the same results would apply to other diseases or other trial designs. In fact, considering the possible effect that treatment group knowledge may have on participant’s behavior during the study, it seems unlikely that an AMD participant’s behavior during the course of the study could have much influence on the disease progression. In contrast, with diabetes, improving glycemic control could have an effect on DME and subsequently have an effect on visual acuity outcomes related to the DME. It is important to note that this study did not assess the potential bias in visual acuity outcome that may occur when study participants receiving intravitreal injections are not masked to treatment assignment.
In summary, this study has demonstrated that a sham intravitreal injection can successfully mask study participants to treatment group assignment when a detailed procedure is followed which attempts to carefully mimic a true injection procedure. Masking may be less successful when one eye receives real injections and the other eye receives sham injections or an individual eye receives both real and sham injections.
Acknowledgments
Financial Support: Supported through a cooperative agreement from the National Eye Institute and the National Institute of Diabetes and Digestive and Kidney Diseases, National Institutes of Health, U.S. Department of Health and Human Services EY14231, EY14229, EY018817
Footnotes
An address for reprints will not be provided.
Financial Disclosure: The funding organization (National Institutes of Health) participated in oversight of the conduct of the study and review of the manuscript but not directly in the design or conduct of the study, nor in the collection, management, analysis, or interpretation of the data, or in the preparation of the manuscript. A complete list of all DRCR.net investigator financial disclosures can be found at www.drcr.net
References
- 1.Beck RW. To mask or not to mask. Arch Ophthalmol. 2009;127(6):801–2. doi: 10.1001/archophthalmol.2009.109. [DOI] [PubMed] [Google Scholar]
- 2.Scott IU, Ip MS, VanVeldhuisen PC, et al. A randomized trial comparing the efficacy and safety of intravitreal triamcinolone with standard care to treat vision loss associated with macular Edema secondary to branch retinal vein occlusion: the Standard Care vs Corticosteroid for Retinal Vein Occlusion (SCORE) study report 6. Arch Ophthalmol. 2009;127(9):1115–28. doi: 10.1001/archophthalmol.2009.233. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Campochiaro PA, Heier JS, Feiner L, et al. Ranibizumab for macular edema following branch retinal vein occlusion: six-month primary end point results of a phase III study. Ophthalmology. 2010;117(6):1102–12. e1. doi: 10.1016/j.ophtha.2010.02.021. [DOI] [PubMed] [Google Scholar]
- 4.Nguyen QD, Shah SM, Heier JS, et al. Primary end point (six months) results of the ranibizumab for edema of the macula in diabetes (READ-2) study. Ophthalmology. 2009;116(11):2175–81. e1. doi: 10.1016/j.ophtha.2009.04.023. [DOI] [PubMed] [Google Scholar]
- 5.Mitchell P, Bandello F, Schmidt-Erfurth U, et al. The RESTORE Study Ranibizumab Monotherapy or Combined with Laser versus Laser Monotherapy for Diabetic Macular Edema. Ophthalmology. 2011;118(4):615–25. doi: 10.1016/j.ophtha.2011.01.031. [DOI] [PubMed] [Google Scholar]
- 6.Rosenfeld PJ, Brown DM, Heier JS, et al. Ranibizumab for neovascular age-related macular degeneration. N Engl J Med. 2006;355(14):1419–31. doi: 10.1056/NEJMoa054481. [DOI] [PubMed] [Google Scholar]
- 7. NCT00996437. [Accessed Accessed May 12, 2011];An evaluation of intravitreal ranibizumab for vitreous hemorrhage due to proliferative diabetic retinopathy. http://clinicaltrials.gov/ct2/show/NCT00996437?term=NCT00996437&rank=1.
- 8.The Diabetic Retinopathy Clinical Research Network. Randomized trial evaluating ranibizumab plus prompt or deferred laser or triamcinolone plus prompt laser for diabetic macular edema. Ophthalmology. 2010;117(6):1064–77. e35. doi: 10.1016/j.ophtha.2010.02.031. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Hawkins BS, Bressler NM, Reynolds SM. Visual acuity outcomes among sham vs no-treatment controls from randomized trials. Arch Ophthalmol. 2009;127(6):725–31. doi: 10.1001/archophthalmol.2009.101. [DOI] [PubMed] [Google Scholar]
- 10.Hawkins BS, Bressler NM, Reynolds SM. Patient-reported outcomes among sham vs no-treatment controls from randomized trials. Arch Ophthalmol. 2011;129(2):200–5. doi: 10.1001/archophthalmol.2010.359. [DOI] [PubMed] [Google Scholar]