Selecting a control in IVF and acupuncture RCTs: How sham controls may unnecessarily complicate the RCT evidence base

Abstract

Background

Acupuncture has typically been tested in trials that evaluate subjective, patient-reported outcomes such as pain. Ratings of pain and similar subjective states can be strongly influenced by respondents’ pre-judgments, preferences, and expectations about treatment benefits. Therefore, controlling for these expectations or “placebo effects” by using a sham acupuncture control group is critically important in trials of acupuncture for pain-related conditions. This need for sham acupuncture controls in trials of acupuncture for pain-related conditions may have led to the belief that sham acupuncture is always the most “rigorous” control, and that it should therefore be used for all acupuncture trials, including trials of adjuvant acupuncture for IVF.

Objective

To examine the theoretical and methodological rationales for the use of sham acupuncture controls in trials of adjuvant acupuncture for IVF, and to identify the drawbacks of using a sham acupuncture control that may have its own effects on the pregnancy outcome.

Conclusions

In trials of adjuvant acupuncture for IVF, the outcome is pregnancy, which is entirely objective and unlikely to be affected by a patient’s expectations of a benefit of acupuncture. Because it seems unlikely that an IVF patient’s knowledge of whether or not she was receiving adjuvant acupuncture would affect her ability to become pregnant from IVF, using sham acupuncture to control for expectation/placebo effects seems unnecessary in this context. Even if adjuvant acupuncture were to increase IVF success rates only through a psychosomatic effect mechanism, such as by reducing stress, this stress reduction effect would be integral to the working mechanism by which adjuvant acupuncture increases IVF pregnancy rates, and therefore, it seems inappropriate to control for and separate out any such stress-reduction effect by using a sham control. Because of the risk that the sham is not an inert placebo but rather an active treatment that may affect the pregnancy outcome, using sham acupuncture as the control may unnecessarily confuse rather than clarify the interpretation of the effects of IVF adjuvant acupuncture. Using both theoretical concerns and epidemiological evidence, researchers should carefully weigh the benefits and drawbacks of using sham acupuncture to blind patients in adjuvant acupuncture for IVF trials, and should question, rather than automatically accept, whether “placebo effects” are an important risk of bias in this context.

Acupuncture has been used in China for centuries to regulate the female reproductive system (1). Several RCTs have tested whether adding acupuncture to the embryo transfer procedure increases the IVF pregnancy success rate. A randomized trial by Paulus et al (2) was the first to test this hypothesis, by administering a session of acupuncture, both before and after the embryo transfer, and comparing pregnancy rates of this acupuncture group with rates of a usual care control group. Paulus et al (2) found that the odds of achieving a pregnancy in the acupuncture group were more than two times the odds in the usual care control group. Several subsequent trials, including some with sham control groups and others with usual care control groups, have been conducted in an attempt to validate the results of Paulus et al (2). Most of these trials have used a protocol similar to that used by Paulus et al.

The first systematic review and meta-analysis on this topic, published in February 2008 (3), included 7 robust RCTs published from 2002 through 2006. This meta-analysis (3) found that complementing the embryo transfer procedure with acupuncture was associated with statistically significant and clinically relevant improvements in clinical pregnancy, ongoing pregnancy, and live birth. These improvements were higher than the reported improvements for any other adjuvant procedure for IVF (4). This could be an important development in IVF research because a safe and low-cost adjuvant procedure that could increase IVF success rates would be valuable in improving patient outcomes, reducing overall costs, and reducing the length and stress of the IVF procedure.

Since this February 2008 meta-analysis (3), several new RCTs have been published, and many of these RCTs have included “sham” acupuncture as the control group. These sham-controlled RCTs have had heterogeneous designs (i.e. using various types of sham acupuncture as the control), and have had inconsistent results, with some RCTs showing the true acupuncture to be superior to the sham (5–7) and others showing the sham to be superior to the true acupuncture (8–11), and in one case the sham was statistically significantly superior to the true acupuncture (8).

The purpose of this methodological review is to question why sham controls were used in these acupuncture for IVF trials; to provide an evidence-based analysis of whether sham acupuncture is the most appropriate control in this context; to review the potential problems with using a sham control that may not be inert; and finally to propose methods of pooling and analyzing the resulting database of trials to derive a clinically important message that could be useful for IVF practitioners and patients. The choice of control in these RCTs is not only of interest to IVF-acupuncture researchers, but is also of general relevance to researchers in assisted conception because it gets to the reasoning and evidence behind what constitutes “best evidence” and what is the most rigorous design for RCTs in IVF.

Why have sham acupuncture controls been used in acupuncture trials?

In acupuncture trials, sham acupuncture has often been used as the control. This is because acupuncture has typically been evaluated in RCTs using subjective, patient-reported outcomes such as pain. Such patient-reported, subjective outcomes can be largely influenced by prejudgments, preferences, and expectations about acupuncture and therefore controlling for these “placebo effects” (12) by using a sham acupuncture control is critically important in this context. For example, when pain is the outcome in an acupuncture trial, a trial participant’s awareness of whether or not she was receiving acupuncture, a treatment that she may expect or prefer, can influence her later reported assessments of level of pain. Sham acupuncture controls are critical to blind patients to whether or not they received true acupuncture in order to control for such placebo effects and to thereby estimate the true biological effects of acupuncture. Because of the critical need to avoid “placebo effects” (i.e. defined as “the impact of expectation on subjective outcomes” (12)), the fundamental challenge of acupuncture for pain trials has been developing a sham control that is sufficiently believable to patients as to be indistinguishable from true acupuncture, and yet at the same time not so similar to true acupuncture that the sham has a therapeutic effect of its own, and is therefore not an inert placebo (13). This need for sham acupuncture controls in RCTs of acupuncture for pain-related conditions may have led to the belief that shams are the most “rigorous” control, and that they should therefore be used for all acupuncture trials (14).

Is the rationale for using sham acupuncture controls in IVF trials supportable on theoretical grounds?

However, it is arguable whether the use of sham controls to blind patients to the treatment they receive and to guard against the placebo effect is relevant in the context of RCTs of acupuncture for IVF because IVF RCTs have entirely objective outcomes. That is, for RCTs with subjective, patient-assessed, self-reported outcomes such as pain, outcomes can be largely affected by judgments and expectations (12), but it seems much less likely that a patient’s knowledge of whether or not she was receiving acupuncture would affect her ability to become pregnant.

Some have argued that women in IVF trials need to be blinded to treatment assignment by using a sham control because the knowledge that a woman is receiving acupuncture may relax her, reduce her stress and anxiety levels, and thereby improve her chances of pregnancy (14, 15). A sham acupuncture control, it is argued, is necessary to equalize this stress reduction effect in both groups. The effect of stress on IVF outcomes is an extensively researched area, but findings have been inconsistent and difficult to interpret. Although some studies have suggested a possible association between stress levels and infertility (16), other studies (17) have found no evidence that psychological stress had any influence on the outcome of IVF treatment. Suffice it to say that the association between stress and IVF success is controversial and unresolved; there is no conclusive experimental evidence that lower psychological stress levels result in improved IVF outcomes (18), and it is even less well established that psychological stress reduction interventions would either affect IVF outcomes, or affect the biological processes (e.g. endogenous hormone release, blood flow to the uterus) that may impact on IVF outcomes. Even if a patient’s belief that she was receiving acupuncture improved her expectations and thereby reduced her psychological stress levels, it seems unlikely that a belief or expectation of a benefit of acupuncture would have such a large physiological stress-reducing effect (distinct from any biological effects of the acupuncture needles) as to increase her chances of pregnancy.

The comparison between acupuncture and sham acupuncture on pregnancy outcomes, though perhaps of some scientific interest, has little clinical relevance. That is, even if acupuncture were to cause pregnancy only through a psychosomatic mechanism, such as by reducing stress, the end result would still be a pregnancy or live birth, which is not just a placebo effect, but a true clinically relevant benefit. Indeed, when a sham has effects that are part and parcel of the working mechanism of acupuncture (e.g. reduced anxiety), but without being a feasible alternative in clinical practice, you can learn little from sham-controlled trials. For clinically relevant conclusions, we need to compare realistic alternative, like adjuvant acupuncture versus no adjuvant to IVF.

Blinding is used not only to prevent biased responses by patients (i.e. response bias), but also to prevent biased performance of healthcare providers (i.e. performance bias) (19). In this context, performance bias would mean that the embryo transfer physicians would perform the embryo transfer procedure better on the patients who were receiving acupuncture in order for the trial to find that acupuncture was a successful adjuvant procedure. However, considering the cost of the embryo transfer procedure and the importance of successful embryo transfers in maintaining high pregnancy rates at clinics, it would seem that the embryo transfer physicians would be motivated primarily to perform a successful embryo transfer for all patients, rather than to show that acupuncture, a non-proprietary therapy, is an effective adjuvant procedure. Furthermore, the embryo transfer physicians and other researchers involved in the trial would not have a financial interest in the trial’s results. That is, positive results would be unlikely to result in direct financial gain to the physicians or individual clinics supporting and conducting the studies because acupuncture, unlike pharmaceuticals and medical devices, is not a proprietary therapy and therefore it cannot be patented and sold by an individual, clinic, or company. Although blinding of embryo transfer physicians is probably not critical for reducing bias in IVF trials, it still seems fairly simple to do, and it should not be discouraged. However, blinding of physicians does not require the use of a sham control (20), which, as described below, can increase the risk of bias of a trial’s results.

Blinding is also useful to prevent outcomes assessment bias. That is, blinding is useful to prevent the outcomes assessors from knowing which treatment groups patients were assigned to, as this knowledge can influence assessment of subjective outcomes. However, the issue of outcomes assessor blinding seems irrelevant in this context (21, 22), where the outcomes are entirely objective and clear-cut (i.e. pregnancy is either present or not). That is, the likelihood of a pregnancy diagnosis would not be affected by knowledge of the treatment assignment (acupuncture versus no acupuncture/sham).

Blinding can also be important in RCTs to ensure that the randomized groups received an equal amount of attention, care, and ancillary treatment (19). However, in this context, most of the RCTs randomize patients on the day of embryo transfer, which is the final step in the IVF cycle. Because the IVF cycle is completed at the point of embryo transfer, there is little opportunity for differential treatment or attention to be provided to patients after embryo transfer.

Finally, blinding can be important to avoid different levels of follow-up across treatment groups due to a greater enthusiasm among patients for a given treatment (e.g. acupuncture) relative to a different treatment/usual care control (23–26). However, in these IVF trials there are few to no losses to follow-up because the IVF clinic setting provides a captive patient population, in which all women, regardless of randomization assignment or treatment received, are available to be examined to determine whether or not a pregnancy is present.

Do existing methodological and meta-epidemiological studies on the placebo effect suggest that shams/placebos are necessary for trials with entirely objective outcomes, such as RCTs of acupuncture for IVF?

Based on the theoretical considerations described above, we would not expect important differences in acupuncture’s effects on pregnancy outcomes depending on whether or not a sham acupuncture control group was used because all outcomes are entirely objective (i.e. pregnancy and births), and would a priori not be expected to be largely affected by patient expectations and placebo effects.

However, ideally one would like to base the choice of control group on evidence from empirical research studies rather than a priori expectations. In that regard, several leading methodologists within the internationally respected Cochrane Collaboration (27) have conducted meta-epidemiological studies that have examined the evidence for the placebo effect, for both subjective and objective outcomes. These studies suggest that the placebo effect is important for subjective, but not objective, outcomes measures. Namely, Hrobjartsson and Gotzsche conducted a systematic review of RCTs in which patients were assigned to either placebo or no treatment, (as well as an active treatment arm in some of the RCTs reviewed) in order to estimate the magnitude of the placebo effect (28, 29). The reviewers found that a placebo can demonstrate a benefit over a ‘no treatment’ control in studies with subjective outcomes such as pain, but that the placebo demonstrates no significant effect over a no treatment control in studies with objective outcomes. More recently, Wood and colleagues (22) conducted a meta-epidemiological study to examine whether lack of blinding (i.e. no placebo control) was associated with biased estimates of interventions’ effects in trials, and whether the association between lack of blinding and biased estimates varied depending on whether the outcome investigated was subjective or objective. These investigators found that there was indeed evidence of bias, suggested by exaggerated effect estimates, when there was a lack of blinding in trials assessing subjective outcomes, but that there was little evidence of bias associated with lack of blinding in trials assessing all-cause mortality or other objectively assessed outcomes. For the outcomes of pregnancy and birth, which are probably the most objective of all outcomes, with the possible exception of mortality, we believe that there is a low risk of bias due to lack of blinding.

Kaptchuk and colleagues conducted a methodological study (30) to specifically examine the placebo effect in the context of an acupuncture RCT. Their study provides empirical data on the placebo effect of acupuncture, on both subjective and objective outcomes. Specifically, Kaptchuk and colleagues randomized patients to placebo oral pills or sham (placebo) acupuncture, and compared these 2 groups on several different outcomes. Their study found that patients randomized to sham acupuncture reported less pain than patients randomized to placebo oral pills, suggesting that acupuncture has an enhanced placebo effect compared to oral placebo pills on subjective outcomes (i.e. patient-rated pain outcomes). This finding supports the idea that it is particularly critical to control for placebo effects in acupuncture trials that evaluate subjective outcomes, such as pain. However, Kaptchuk and colleagues’ study also found that the placebo effect was confined to self-reported, subjective outcomes (e.g. pain) and that there was no placebo effect (i.e. no improvement from baseline), for either the placebo acupuncture or placebo pill, on the completely objective outcome that they measured (i.e. grip strength). Their findings suggest that an enhanced placebo effect of acupuncture, or indeed any placebo effect of acupuncture, is confined to subjective outcomes. Indeed, in their Discussion, Kapchuk and colleagues concluded “That the differential placebo effect was confined to self reported measures (and not to grip strength) suggests an effect that may be confined to subjective outcomes.”

Are shams/placebos used in other trials of subfertility and have leading subfertility researchers and methodologists considered a sham as a critical design feature for reducing bias in these trials?

In subfertility trials in general, where outcomes are entirely objective, blinding of either patients or physicians is “infrequently attempted” (31). More specifically, Arce and colleagues point out in their review article on the methodological issues in the design of efficacy trials of IVF that “Double-blinding in assisted reproductive technology trials is infrequently attempted” because of the difficulties of implementing blinding in an IVF context (31). Arce and colleagues describe this in terms of gonadotropic trials, and explain that “double-blinding in practice remains very difficult” because it would require that the “investigational drug would need to have indistinguishable primary packaging material compared to the approved comparator.” Yet developing similar packaging seems like a relatively minor difficulty compared to developing a sham procedure for acupuncture.

Leading IVF methodologists have not judged blinding to be a critical element for reducing bias in IVF trials (32). In Vail and Gardener’s review of common errors in the design and analysis of trials of subfertility (32), they did not even include lack of blinding as one of the 9 pre-specified design and analysis errors they evaluated. Vail and Gardener’s review found that only 2 out of 39 RCTs published in the two leading subfertility journals were free of important errors in design or analysis. Considering the considerable room for improvement in subfertility trial design, IVF trialists should focus future efforts on correcting design flaws which can truly bias results, rather than being concerned with blinding status, which is unlikely to bias results. Other commentators (33, 34) have also not mentioned blinding or the need for a placebo as an important quality criterion for subfertility trials. Given that leading IVF methodologists have not considered blinding to be an important quality criterion for IVF trials in general, why should acupuncture for IVF trials be held to a different standard?

What problems can be presented by the use of shams?

If sham acupuncture were completely inert, then the use of a sham acupuncture control would not complicate the interpretation of the results of acupuncture for IVF trials. Indeed, for IVF adjuvant medications for which an inert placebo is easily developed (e.g. a placebo pill), blinding should be easy to accomplish, should not increase the risk of bias, and should not be discouraged. However, sham acupuncture may not be inert, and therefore the use of a sham control for IVF trials with purely objective outcome (i.e. pregnancy, birth) may increase the risk of bias, and may therefore unnecessarily confuse rather than clarify interpretation of the effects of IVF adjuvant acupuncture. Indeed, if the sham is not an inert placebo but rather a different form of acupuncture with its own effects on the pregnancy outcome, then results of sham-controlled trials will systematically deviate from the true value of what the trials are attempting to estimate.

How do we determine whether or not different types of sham acupuncture can affect trial outcomes? No consensus exists on how best to assess the effects of different types of sham acupuncture on outcomes, and no methods have been validated. Therapeutic effects of different types of sham acupuncture controls cannot be reliably estimated from meta-epidemiological research. The reason for this is that if sham-controlled trials using shams that involve more active stimulation (i.e. needle insertion; stimulation of true acupuncture points) showed smaller benefits of acupuncture, there could be 2 possible interpretations for this finding: 1) “active” or intensive shams have physiological activity that influences the outcome and therefore biases the effects of the trial towards the null or 2) shams that involve more active stimulation are the only shams that are believable to patients and only these shams fully and appropriately control for placebo effects and biased responses (25, 35). Therefore, although it would be difficult or impossible to generate conclusive epidemiological evidence to show that shams have therapeutic effects on a pregnancy outcome or that different types of shams have different effects, we provide below a justification from acupuncture research and theory of how the different types of shams used in acupuncture for IVF RCTs may have an effect on pregnancy.

One type of sham that has been used in 4 of the sham-controlled RCTs (7–9, 11) involved non-insertive but pricking “sham” needles placed at true acupuncture points, such that these sham needles gave patients a pricking or penetrating sensation on their skin indistinguishable from that of a true acupuncture needle (8, 9, 11, 36), throughout the duration of the acupuncture session. Such sham needles may be likely to influence the pregnancy outcome because the type of stimulation these shams apply is comparable to applying acupressure to the acupuncture points. Acupressure is a traditional form of acupuncture that has been shown in RCTs and systematic reviews to be an effective treatment for various conditions (37–39). A recent positron emission tomography study has indicating that non-insertive but pricking sham needles placed at true acupuncture points can stimulate regions of the brain associated with natural opiate and neurotransmitter production (40). These pricking but non-insertive sham needles may also have effects on the postulated acupuncture points similar to the effects of superficial needle penetration (i.e. a common technique in many authentic traditional Japanese acupuncture styles (41)) since both superficially inserted needles and non-insertive but pricking sham needles would both be likely to stimulate the acupuncture points. If the research question of interest focused on the potential working mechanism of acupuncture, and whether insertion of needles at acupuncture points or applying pressure at acupuncture points resulted in greater efficacy, than this sham control would be appropriate. However, the research question of greatest interest is the effectiveness of acupuncture in everyday practice, and in this context, using this type of sham control does not seem most appropriate.

Other acupuncture for IVF sham-controlled trials have used shams that involved real needle insertion at true acupuncture points, but at points traditionally used to treat a non-fertility related condition (e.g. back pain) (5). The reason that such shams would be likely to influence the pregnancy outcomes is because any type of true acupuncture point selection traditionally used for treating a specific condition may have not only a specific effect on the targeted condition but also a generalized adaptogenic or salutary effect, and therefore may also be beneficial for other conditions, according to acupuncture theory (1, 42).

Finally, other sham-controlled trials used shams that involved needle insertion not directly on, but near, the true acupuncture points (10). These shams may have an effect on the pregnancy outcome because the location of the acupuncture points may not be as precisely defined as acupuncture tradition suggests (43), and stimulation in areas near the “true” points may result in therapeutic activity. Indeed, physiological effects of needle penetration, even if the needles are inserted at non-acupuncture points, is suggested by several lines of research in human beings (42, 44, 45) as well as in animal studies which have shown that needle insertion can have nonspecific analgesic effects through a postulated mechanism of “diffuse noxious inhibitory control” (46). The physiological effects of this non-acupuncture point needle insertion sham may influence the pregnancy outcome through 2 of the postulated mechanisms through which acupuncture may influence IVF success: 1) the needles may stimulate natural opiate production, which may inhibit the central nervous system outflow and the biological stress response (47) and thereby promote a successful embryo transfer, and 2) the needles may stimulate the release of neurotransmitters (48), which may in turn stimulate secretion of gonadotrophin releasing hormone, thereby influencing the menstrual cycle, ovulation, and fertility (49).

How can the current evidence base be interpreted and how should it be analyzed?

As discussed above, in many of the RCTs of acupuncture for IVF, the shams that have been used may not be inert placebos, but rather different forms of acupuncture or acupressure. These trials may in fact be comparing 2 different forms of acupuncture, and are not appropriate for addressing the question of interest, which is whether IVF adjuvant acupuncture is helpful in increasing pregnancy rates. The complications in analyzing these trials is compounded by the fact that the different “sham” controls may have different effects on pregnancy, with some shams affecting the pregnancy outcome and others not. A systematic review of these trials may not be helpful in clarifying the potential role of acupuncture in IVF.

However, there is also a substantial database of no adjuvant treatment controlled trials, and pooling the data from these trials may be useful to estimate the average effect of acupuncture on IVF success rates, and to examine the impact of potential trial-level modifiers (e.g. number of acupuncture treatments in the trial, baseline pregnancy rate in the trial etc) on this effect. To study patient-level effect modifiers, for example, to assess whether there are different effects of adjuvant acupuncture in different populations of women (e.g. women aged > 35 versus ≤35) or different effects with different IVF-related procedures (e.g. single versus double embryo transfer), we would need to conduct an individual patient data (IPD) meta-analysis. An IPD meta-analysis involves the central collection, validation, and re-analysis of “raw data” from all trials relevant to a given research question (50). Such an IPD meta-analysis could possibly answer some of the unresolved questions about the value of acupuncture as an adjuvant to the embryo transfer of IVF. An IPD could address the effects of patient-level moderator variables on acupuncture’s success rate, and thereby facilitate an individualized approach to the use of acupuncture in IVF (51). An IPD will also provide a high quality, international evidence base to better inform practice, research, and debate.

Conclusion

The purpose of this paper is certainly not to argue against blinding and the use of placebos in RCTs with patient reported outcomes. Indeed, in the context of acupuncture for pain trials, we have considered blinding to be the most important interval validity criterion for reducing the risk of bias (25). Nor is it our purpose to argue against blinding of embryo transfer physicians, which is unlikely to introduce bias and may even slightly reduce bias. Rather, we argue only that researchers should carefully weigh the benefits and drawbacks of using sham acupuncture to blind patients in IVF trials, using both theoretical concerns and epidemiological evidence, and should question, rather than automatically accept, whether “placebo effects” are an important risk of bias in this context, as many RCTs and systematic review publications have suggested (8–10) and concluded (52). The question about the need for sham controls may also apply to other invasive, difficult to blind adjuvant procedures evaluated in IVF RCTs (53). Only by having access to RCTs without critical errors in design (32) can systematic reviewers reliably analyze and interpret these RCTs to draw evidence-based conclusions that can be useful to physicians treating patients undergoing IVF.