Double-blinding of an acupuncture randomized controlled trial optimized with clinical translational science award resources

Abstract

Background:

Clinical trial articles often lack detailed descriptions of the methods used to randomize participants, conceal allocation, and blind subjects and investigators to group assignment. We describe our systematic approach to implement and measure blinding success in a double-blind phase 2 randomized controlled trial (RCT) testing the efficacy of acupuncture for the treatment of vulvodynia.

Methods:

Randomization stratified by vulvodynia subtype is managed by Research Electronic Data Capture (REDCap) software’s Randomization Module adapted to achieve complete masking of group allocation. Subject and acupuncturist blinding assessments are conducted multiple times to identify possible correlates of unblinding.

Results:

At present, 48 subjects have been randomized and completed the protocol resulting in 87 subject and 206 acupuncturist blinding assessments.

Discussion:

Our approach to blinding and blinding assessment has the potential to improve our understanding of unblinding over time in the presence of possible clinical improvement.

Background

The efficacy of acupuncture is difficult to demonstrate in the absence of a double-blind milieu. We are currently testing a double-blind treatment protocol for the treatment of vulvodynia. Vulvodynia is a chronic pain condition characterized by vulvar pain and painful sexual intercourse (dyspareunia) and has no consistently effective treatments.¹ We provide details of our procedures to achieve an internally valid, double-blind acupuncture efficacy trial with careful attention to blinding of the treatment condition, concealment of treatment allocation, and measurement of blinding success.

Acupuncture experiments have a range of methodological challenges. Most are performed unblinded or single-blinded (subject blind, acupuncturist unblind) and use a range of control conditions including no acupuncture conditions, which may vary in intensity (e.g., physical therapy, usual care, no other care, or attention control), or sham acupuncture using a variety of methods. Evidence shows stronger effects for acupuncture compared to no acupuncture,² yet these designs do not establish that the therapeutic effect is more than a placebo or expectancy effect, a common criticism of acupuncture experiments.^3–5 Sham acupuncture control conditions may use shallow needle insertion at non-acupuncture points or non-penetrating needles, but have shown more similar results to therapeutic acupuncture, possibly because needling or skin stimulation at the alternative sites has some therapeutic effect.² Others have argued that acupuncture in single blind controlled studies is less effective than routine clinical care due to the use of a base unit that hides needle penetration.⁶ We have developed a rigorous test of acupuncture for reduction of the pain and dyspareunia of vulvodynia using penetrating (verum) and placebo needles that are identical in appearance and are designed to feel the same to subjects and acupuncturists, i.e., double blind needles (Figure 1). The validity of these double-blind needles has been supported in multiple studies, in single use applications,^7–12 but this is the first study to use this methodology in a multi-needle, multi-session experiment. Our study is designed to test the efficacy of the standardized vulvodynia acupuncture protocol beyond placebo effect and, secondarily, to learn about correlates of blinding.

Figure 1.

A set of double-blind acupuncture needles (DBNs)

Treatment allocation in the context of a randomized trial is the process of assigning subjects to either a treatment or control condition and the allocation sequence is the randomized order of each subject assignment. Selection bias may occur when research staff create conditions that influence which subjects are enrolled and the sequence of their assignment to a treatment arm. To prevent this bias, methodologists recommend that subjects are randomized after enrollment and that the allocation sequence is concealed from staff.¹³ In a double-blind placebo controlled trial, the blinding and concealment of treatment allocation can occur through the same process and both are critical to the integrity of the study. In accordance with the consolidated standards of reporting trials (CONSORT),¹⁴ we present our process for blinding and treatment allocation as well as the level of blinding for persons involved in this research. These details have rarely been reported in methodology sections of published RCTs.^14–16

While it is accepted that double blinding is ideal in an experimental design, the measurement of blinding success is controversial.^17–19 For many conditions and therapies, blinding to treatment arm may be confounded with treatment efficacy. As a subject improves, the subject and clinician may correctly assume the subject received an active treatment. Measurement of blinding success was part of the CONSORT checklist but was removed in the 2010 revision due to this dilemma.^14,19 Our study dually tests an acupuncture treatment protocol and the use of double-blind needles in a multi-session trial. Therefore, we have taken a discovery approach to understanding blinding success through a multi-pronged assessment for subjects and acupuncturists that will allow us to learn about possible correlates of unblinding. We present an overview of our protocol, methods to conceal treatment allocation and maintain a double-blind protocol, and our approach to measuring the success of blinding.

Methods

Study overview

Our study was designed to achieve equipoise for all non-specific effects of our treatment protocol. All 80 subjects expected to complete the study are acupuncture naïve at time of enrollment. Subjects randomized 1:1 to either penetrating or placebo needles, receive a 13-needle, twice-weekly, 10-session, 5-week acupuncture treatment protocol. Subjects and acupuncturists are instructed to speak only if safety is compromised in an attempt to remove the therapeutic effects of their relationship. A research associate monitors the fidelity of the acupuncture treatment protocol at all treatment sessions. Data collection and management are facilitated using the Research Electronic Data Capture (REDCap), hosted at the University of Illinois at Chicago.^20,21 This study was approved by the University of Illinois at Chicago Institutional Review Board. Successful completion of our research will allow us to study the methodology of double-blinding in a multi-session multi-needle efficacy trial and to test the efficacy of an acupuncture protocol for the treatment of vulvodynia pain.

Randomization protocol

We developed our randomization and blinding procedures to support the internal validity of our study. The on-site statistician oversees the labeling of needle packages, which arrive from the manufacturer in Japan tagged with the treatment arm identifier (1 or 2; Figure 2). The statistician removes the tag and labels the packet with a unique code, described below.

Figure 2.

Needle package contents for each subject

Randomization is implemented using the REDCap randomization module for its ability to manage stratified randomization and because we could adapt it to mask group allocation. We chose to randomize within strata defined by diagnostic subtype, generalized vulvodynia and provoked vestibulodynia, due to its prognostic importance.²² We wanted to achieve complete masking of group allocation to ensure that our clinical and research staff did not observe patterns across subjects that might unblind them. For example, if staff were aware of group allocation (e.g., treatment arm 1) they might notice that subjects in that arm seem to improve compared to the other arm and deduce it is the penetrating needles arm. If this occurs, staff have the potential to introduce an expectancy bias in their interactions with future subjects assigned to arm 1. Therefore, we went beyond masking that arm 1 might be the penetrating needle arm, thus preventing staff from ascertaining any treatment arm identifiers entirely.

The REDCap randomization module executes a user-defined randomization model. It requires the project statistician to upload an allocation sequence developed using other software and generates assignments accordingly with the click of a button. Typically, in a randomized controlled trial subjects are assigned to 2 or 3 groups, but since we adapted the module to mask allocation we created 176 “groups”, one for each possible subject accounting for attrition and unknown strata proportions. Instead of REDCap assigning one of the two treatment arms, we had it assign one of the 176 “groups” which represent unique codes for each individual enrolled. We called this code a randomization identifier (ID) that is linked to treatment arm in a spreadsheet stored outside of the REDCap system. This randomization ID serves as the study ID for the remainder of the study.

We developed the allocation sequence using Stata²³ with permuted blocks of consistent size including twice as many potential assignments than our target sample size. After copying this sequence into a spreadsheet, we added a diagnostic subtype indicator in equal proportions and a randomly ordered list of IDs. This master worksheet, which included treatment arm allocations (1 and 2), was saved in a secure location with access restricted to the two study statisticians. An example of this master worksheet is shown in Table 1. We used the diagnostic subtype and ID columns from our master worksheet to set up the randomization module in REDCap. We used ID instead of the treatment arm indicator, which functioned as desired because we set up 176 “groups”. Thus, when a staff person randomizes a subject, the ID code is revealed rather than the treatment arm. The staff selects the needle package with the matching code, which contains the subject’s needles for their full course of acupuncture.

Table 1.

Master worksheet sample for randomization allocation schedule.

		Generated by Stata		Manually added*
arm	block	Random number	Sequence number	Generalized subtype	Randomization ID
2	1	0.0238564	1	1	140
1	1	0.1413895	2	1	173
1	1	0.1841138	3	1	116
2	1	0.971364	4	1	145
2	2	0.0461595	1	1	180
1	2	0.4007082	2	1	136
2	2	0.7773784	3	1	141
1	2	0.9007107	4	1	115
2	43	0.2526355	1	0	183
1	43	0.3804519	2	0	134
2	43	0.6840113	3	0	104
1	43	0.9718139	4	0	150
1	44	0.1008327	1	0	133
2	44	0.5935789	2	0	179
2	44	0.754083	3	0	160
1	44	0.8610591	4	0	181

^*Columns used for REDCap allocation template.

Note: Two Arm 1 and two Arm 2 assignments were randomly shuffled within each block by creating a column with random numbers, then sorting the random number from smallest to largest within each block. Half of the blocks were assigned to each diagnostic subtype. A randomization ID was inputted to be linked to each treatment arm assignment.

By design, the principal investigator, research staff, and acupuncturists are blinded to any treatment arm identifier and allocation of assignment is concealed. The statistician is aware of a treatment arm identifier (1 or 2) but blinded to its meaning. Our off-site statistician is the only person not blinded to treatment arm. Consequently, he manages communication with study subjects following completion of the study (e.g., emails to inform the subject about treatment received and to offer free acupuncture sessions to those receiving placebo).

Measurement of blinding success

Both subjects and acupuncturists are asked to guess the type of needles used without a “don’t know” option and to rate their confidence on a 0 to 10 scale. Subjects answer these questions twice, after their first and after their final acupuncture session via a self-administered survey using a tablet computer. The first assessment allows us to examine subjects’ blinding before sustained treatment effects are evident. The second assessment, after 10 acupuncture sessions, will allow us to examine the association of treatment effects with beliefs about treatment received.

Our approach for the assessment of acupuncturist blinding capitalizes on our 10-session protocol. We use multiple acupuncturists to administer the study treatments. Due to the logistics of scheduling 10 sessions per subject, multiple acupuncturists may treat each subject over the 5-week course of treatment. Acupuncturists are asked the blinding questions after the subjects’ first and tenth sessions and each time they see a subject for the first time.

Statistical analyses

Once data collection is completed, statistical methods will be used to assess blinding. We will compute a weighted guess score coding a correct guess as +1 and an incorrect guess as −1, which will be multiplied by the confidence rating yielding a possible range of −10 to +10, where 0 indicates no confidence in the guess regardless of its accuracy. For subject responses we will calculate a separate blinding index for each treatment arm using Bang’s method, adapted for our guess score, which yields a value from −1 interpreted as opposite guessing (100% incorrect guesses) to +1, which is complete unblinding (100% correct guessing).²⁴ Bang’s Blinding Index is a descriptive measure that can be compared with findings from other literature reviews published using this approach.^25,26 Mixed effect linear regression models will be used to understand correlates of the guess score measure for subjects and acupuncturists in separate analyses.²⁷ Our data will contain multiple records per subject and random coefficients will be used to account for respondents’ influence on their repeated measurements over time. Treatment arm by time interactions will be a key predictor in our modeling. Other candidate variables for these analyses are shown in Table 2.

Table 2.

Possible predictors of unblinding for exploratory analyses.

Variable	Description
Treatment arm	Penetrating vs. non-penetrating placebo needles
Session number	Session 1 – 10 of treatment protocol that is associated with the guess, treated as categorical fixed effect
Recent pain rating	*Patient’s worst pain in past 24 hours. For 10th session this rating will be indicative treatment response.
Acupuncturist’s experience	Years of experience
Protocol experience	Number of treatments acupuncturist provided within the study in total
Session # of acupuncturist- subject pair	*Number of sessions acupuncturist has treated the subject
Body mass index	Acupuncturist may note difference with needle insertion if subject has low versus higher body fat.
Month	*Months of data collection over the duration of the study
Adverse Events	Subject feeling like they will “pass out” or the treatment is too strong

^*Time varying measures that will be tested as average and deviation scores as appropriate.

Results

We are currently 22 months into our data collection period and have screened 238 women and enrolled and randomized 58 to participate in the study. We have completed the protocol with 48 subjects and 9 dropouts. We employ 6 part-time acupuncturists. The results presented below include the lessons learned from our randomization approach, our adherence to the blinding measurement protocol, and the current numbers of blinding assessments associated with the 48 subjects who have completed the study.

Randomization

Our randomization model was designed to assign subjects stratified by two diagnostic subtypes of vulvodynia. This function has been implemented without issue. So far, one randomization-related problem arose on one occasion during the study period. The REDCap system was unavailable when a new subject entered the study and needed to be randomized. Our solution was to randomly choose a needle package so that treatment could be initiated as usual. Afterwards, the statistician reassigned the package chosen to the next randomization identifier in the sequence once REDCap was back online and reallocated the original packet number for future use. This required careful documentation and correction to the master worksheet. A back-up plan was also made should this problem recur. In our study, the outage and our work-around for randomization was more easily corrected than if we had used REDCap to assign treatment arm directly. It should be noted that the randomization sequence in REDCap cannot be edited once the module is set up and in production.

Blinding assessments

We examined the blinding assessment data for missing values to judge adherence with our protocol for the 48 subjects who completed the protocol. We recognized in the early months of enrollment that the collection of the blinding assessment after the first session was overlooked for several subjects. In contrast, no blinding assessment was missing from the 10^th session. Assessing the circumstances, we determined that all missing assessments were not filled out due to the complexity of intake tasks to be completed at the first visit. The research specialist self-corrected and developed a reminder system so that the questions were asked routinely. We assume these missing assessments are consistent with the missing at random mechanism because they are associated with a procedural error in the first months of data collection. Consequently, we will estimate our mixed effect models using full-information maximum likelihood, which will utilize all available data and produce less biased than complete case analysis.^28,29

At present, we have 39 (81%) of subjects with two blinding assessments and 9 (19%) with one. Our acupuncturist assessments were collected following the appropriate acupuncture sessions: 1^st, 10^th, and every session when an acupuncturist treated a subject for the first time. These data were collected with 94% adherence to the protocol. For the same 48 subjects described above we have 206 assessments distributed across the 10 sessions as shown in Figure 3.

Figure 3.

Number of acupuncturist blinding assessments by visit (48 subjects, 9 acupuncturists, 206 assessments)

Discussion

The methods described here reflect a team effort to carefully design and implement a rigorous and internally valid acupuncture double-blind RCT. Although we encountered minor issues along the way, we are confident that our study will be useful in testing the efficacy of our acupuncture protocol for the relief of vulvodynia pain compared to a valid placebo condition. Our use of the REDCap randomization module to assign a unique randomization code has worked well to conceal the allocation schedule and mask treatment arms from staff. These procedural steps have been easy to implement and work reliably. We believe that we have created the best conditions to maintain blinding to treatment allocation and group assignment, but to achieve acupuncture double-blinding and to successfully measure it over a 10-session, 5-week treatment may be difficult. Statistical approaches to quantify the degree of unblinding have been developed and used.^{18,24,25,30–33} Other researchers have used methods for analysis and interpretation when blinding has been breached.^34,35 Our approach is to discover if there are limits to blinding that occur over time, with experience, and with symptom relief.

We recognize some limitations in our approach. Our research specialist, though blind to condition, is not asked blinding questions. Anecdotally, she has observed situations that are potentially unblinding (e.g., reddening of the skin at the needle insertion site, more pain reaction with needle insertion, needles that fall over after placement). These situations are infrequent and do not occur consistently with the same subjects or acupuncturists. Time will tell if they lead to unblinding due to experience.

In sum, the methods employed here, and lessons learned may be helpful to others in developing rigorous acupuncture studies. In addition to our primary aim, to evaluate efficacy for the treatment of vulvodynia pain, this study has promise for adding to our understanding about acupuncturists’ blinding with non-penetrating placebo needles. It will also provide a longitudinal opportunity to look at subject blinding before and after a clinical effect may occur.