Factors that Influence Practitioners’ Interpretations of Evidence from Alternative Medicine Trials: A Factorial Vignette Experiment Embedded in a National Survey

Abstract

Background

Clinical trial evidence in controversial areas such as complementary and alternative medicine (CAM) must be approached with an open mind.

Objective

To determine what factors may influence practitioners’ interpretation of evidence from CAM trials.

Research Design

In a mailed survey of 2400 U.S. CAM and conventional medicine practitioners we included two hypothetical factorial vignettes of positive and negative research results for CAM clinical trials. Vignettes contained randomly varied journal (Annals of Internal Medicine vs. Journal of CAM) and CAM treatment type (acupuncture, massage, glucosamine, meditation, and reiki). Response items also included randomly varied patient circumstances -- chronic refractory symptoms and the patient requesting CAM.

Measures

All practitioners rated the effectiveness and their willingness to recommend the therapy for a described patient. We used logistic regression to determine the independent influence of the four factors on respondents’ effectiveness and legitimacy judgments.

Results

1561 practitioners responded (65%). Relative to Reiki, conventional medicine practitioners were most willing to recommend glucosamine (OR=3.0; 95% CI [1.6–5.4]), than massage (1.9 [1.1–3.3]), acupuncture (1.3 [0.8–2.2]), and meditation (1.2 [0.7–2.0]). CAM practitioners rated acupuncture as effective more than other CAM therapies (OR=5.8 [2.6–12.8]) compared to Reiki), and were more willing to recommend acupuncture (OR=12.3 [4.8–31.8]). When presented evidence of inefficacy, CAM practitioners were most willing to recommend acupuncture relative to other CAM therapies (OR=15.5 [9.0–26.9]).

Conclusions

Practitioners’ judgments about CAM trial evidence depend on the type of treatments reported. Confirmation bias may play a role in the clinical translation of new evidence from clinical trials.

Over the past decade, the National Institutes of Health has sponsored rigorous scientific research on complementary and alternative medicines (CAM) (1). Yet little is known about how data from that research have been translated into clinical care, which depends on healthcare practitioners, both CAM and conventional medicine, taking an interest in and keeping an open mind about new research data that may be of clinical consequence. Keeping an open mind may be particularly difficult for practitioners whose training and experience have strongly influenced their beliefs about what constitutes a legitimate treatment.

Prior cross-sectional studies suggest that physicians’ self-reported willingness to recommend a specific CAM practice depends on their prior beliefs about the efficacy of the practice, the reputation of journals in which data on that practice are reported, whether the practice is requested by a patient, and whether or not the patient’s condition is refractory to other more conventional medicine measures.(3–5) Physicians may be more open to herbs, vitamins, supplements and other “biologically based” CAM modalities than they are to energy therapies and other practices whose effects are less explicable from the biomedical paradigm.(6) Resch et al demonstrated the potential for reviewer bias against CAM in scientific review manuscripts.(7) They found that reviewers’ ratings of the importance of the manuscript and their verdict about acceptance or rejection were significantly higher for the “orthodox” remedy. However, little is known about how physicians’ prior beliefs may influence their interpretations and clinical translation of scientific data from clinical trials of CAM. (8)

This study uses experimental vignettes to examine the impact of contextual features on conventional medicine and CAM practitioners’ interpretations of research evidence from hypothetical clinical studies of CAM. It was designed to test four hypotheses that arose from our assessment of physician survey data, from commentaries surrounding the publication of several major CAM trials in 2005–2006, as well as from focus groups we conducted: First, conventional medicine practitioners would be less likely to either rate a particular CAM practice as effective, or be willing to recommend it, if the practice is less obviously “biologically based”—even after being presented with new scientific evidence for the efficacy of the CAM practice. Second, conventional medicine practitioners’ willingness to recommend CAM therapies would be determined in part by contextual factors such as the prestige of the journal in which positive research findings are reported, whether a patient requests the therapy, and whether a patient has suffered chronic, refractory symptoms. Third, these factors would exert little influence on physicians’ interpretations of evidence showing inefficacy, because of strong prior propensity to reject such therapies. Finally, we hypothesized that CAM practitioners in our study (acupuncturists and naturopaths) would be much more likely than physicians to rate different CAM practices as effective and legitimate, even after considering evidence for the inefficacy of those practices.

METHODS

Participants

We mailed a self-administered, 12-page questionnaire to a random sample of 1200 US physicians (600 internists and 600 rheumatologists) as well as to 1200 licensed CAM practitioners (600 acupuncturists and 600 naturopaths) drawn from commercially available data sets. Overall, 1561 out of 2400 practitioners responded (65% response rate). Survey items were developed through focus groups and cognitive testing with 8 practitioners, and a pilot mailing. Details on survey methods and initial results are reported elsewhere. (9)

In this survey we presented respondents with two vignettes, each of which summarized the results of a clinical research study described as hypothetical. These vignettes contained randomly varied information that we hypothesized would influence physicians’ judgments about the research results. Following each vignette, respondents were asked to rate the effectiveness of the treatment studied from “very effective” to “not at all effective”, and its legitimacy from “very likely to recommend” to “would definitely not recommend”.

Factorial Vignette Design

Because several factors may influence a person’s real world decisions, factorial vignette design helps define the independent contribution of several variables to a respondent’s judgments without any given respondent needing to rate every possible permutation of a vignette scenario (10). By randomly spreading multiple permutations of the vignettes across the entire study sample, factorial survey methods allow for testing of the independent contribution of each informational factor on a given criterion measure. This approach allows for balance between covariates such that no factor becomes over-represented in the sample.

Vignette Elements

In this study, we created two vignettes that were structurally identical across respondents except for the randomly varied elements. The first vignette (referred to hereafter as the “efficacy vignette”) reported a hypothetical recent clinical trial of treatments for osteoarthritis of the hand. The trial was described as published in one of two medical journals (Annals of Internal Medicine [AIM] or Journal of Complementary and Alternative Medicine [JCAM]) and found significant improvement in pain ratings, without significant side effects, for those who received one of five possible adjunctive CAM therapies (acupuncture, Reiki, meditation, glucosamine, or massage) compared to those receiving conservative medical therapy alone. These CAM therapies were selected based on qualitative data regarding what both alternative and conventional medicine practitioners could generally recognize and make reasonable judgments about and represent the five major categories of CAM (whole medical systems, energy medicine, mind-body, biologically-based therapies, and manipulative therapies) as well as a spectrum of biological plausibility from within a biomedical paradigm. (We assumed respondents would use their own working definition of these therapies to answer questions, rather than asking them to struggle with the potential cognitive dissonance of reconciling their concepts with definitions we might give them.) The second vignette (referred to hereafter as the “inefficacy vignette”) followed the same structure as the first but reported on a trial of treatments for carpel tunnel syndrome in which no benefit was found for the adjunctive CAM therapy. We constrained the distribution of parameters so that no subject received the same journal type or treatment in the second vignette as he or she had received in the first.

Patient Circumstances

To test the effects that specific patient circumstances may have on physicians’ willingness to recommend a therapy, we randomly varied whether the hypothetical patients were described as having chronic, refractory symptoms and whether or not the patient explicitly requested the CAM therapy. This was based on our hypothesis that conventional medicine practitioners may be more flexible in recommend CAM, even if unproven, when most other options have been exhausted.

This approach of varying four factors across two vignettes led to 640 possible unique survey permutations. A 25% subset of these permutations was selected at random in order to reduce the number of permutations to a manageable set of 160, such that the main effects of each factor and two-way interactions between the factors could be estimated.

Statistical Analysis

All data were double-entered and 100% verified. In unadjusted comparisons of respondents from each practitioner group, we performed Pearson’s chi-square tests for categorical variables (gender, race, and practice setting) and analysis of variance for age. Practitioners’ judgments of effectiveness and their willingness to recommend CAM therapies presented in the vignettes served as the outcome variables of interest. Practitioners’ judgments about the effectiveness of the CAM therapies were dichotomized as “very or moderately effective” versus “only slightly or not at all effective.” Willingness to recommend was dichotomized as “very or moderately likely” versus “unlikely or would definitely not recommend.” We then used Chi-Square tests (or Fisher’s exact tests where appropriate) to examine the unadjusted association between the criterion measures and the different predictors.

Using logistic regression, we tested for the independent effect of each explanatory variable on the two outcome variables of interest. The explanatory variables were journal type, type of CAM treatment, as well as two patient circumstances -- patient request and whether the hypothetical patient was described as having chronic, refractory symptoms. The results from the multivariable logistic regression models are reported with odds ratios and 95% confidence intervals (two-sided tests of significance, α=0.05). We performed all analyses using SAS version 9 software (Cary, NC).

Human Subjects Approval

This study was approved by institutional review boards of the National Institutes of Health Office of Human Subjects Research, Mayo Clinic, and the University of Massachusetts-Boston.

Role of the Funding Source

The funding source, the National Center for Complementary and Alternative Medicine at the U.S. National Institutes of Health was not involved in data collection, analysis, or writing of the manuscript.

RESULTS

Among the 1561 respondents, 679 (43%) were physicians, and 882 (57%) were CAM practitioners. Practitioner groups varied considerably in their personal and professional characteristics (Table 1). Within practitioner groups, respondents and non-respondents did not differ with respect to age, gender, race, or practice type. Overall, randomization was effective. Similar numbers of respondents returned surveys for all 160 permutations. With respect to the factorial design, randomization was effective. Age, gender, ethnicity, practice setting, baseline familiarity of CAM treatments, and baseline perceived efficacy of CAM treatments did not differ significantly across the levels for any of the randomized factors.

Table 1.

Characteristics of 1561 Respondents Shown by Conventional and Complementary and Alternative Medicine (CAM) Practitioner Group

	CAM Practitioners (N=882)	Conventional Medical Practitioners (N=679)	Total (N=1561)	p value
Age				<.0011
N	846	647	1493
Mean (SD)	47.8 (10.0)	50.6 (8.2)	49.0 (9.4)
Range	(29.0–81.0)	(28.0–88.0)	(28.0–88.0)
Gender	N (%)	N (%)	N (%)	<.0012
Male	313 (37%)	477 (73%)	790 (53%)
Female	538 (63%)	175 (27%)	713 (47%)
Ethnicity/Race				0.012
Asian or Asian-American	130 (15%)	88 (14%)	218 (15%)
Black or African-American	5 (1%)	14 (2%)	19 (1%)
Hispanic	26 (3%)	27 (4%)	53 (4%)
Other	29 (3%)	12 (2%)	41 (3%)
White or Caucasian	654 (78%)	509 (78%)	1163 (78%)
Practice Setting				<.0012
Solo, Private	565 (65%)	186 (28%)	751 (49%)
Group, Private	227 (26%)	334 (50%)	561 (36%)
Institutional, Private	25 (3%)	28 (4%)	53 (3%)
Academic	17 (2%)	96 (14%)	113 (7%)
Other	29 (3%)	24 (4%)	53 (3%)
Multiple checked	9 (1%)	3 (0.4%)	12 (1%)

¹ANOVA F-Test

²Chi-Square

³Percentages based on actual numbers of respondents. N counts do not all sum to 1561 because of missing data and partial non-response, and percentages may not sum to 100 due to rounding. Respondents answered most questions.

Effectiveness and Willingness to Recommend Ratings

In the efficacy vignette, conventional medicine practitioner ratings of effectiveness were split; overall, about half (48–54%) rated the treatment as effective in light of evidence for efficacy. The percentage of physicians who were willing to recommend the range of CAM therapies (65–84%) was somewhat higher compared to their effectiveness ratings. Most CAM practitioners (76–95%) rated the CAM practice as effective for all types of CAM and journal source. CAM practitioners expressed a strong degree of willingness to recommend CAM (71–97%) across treatment types.

For the inefficacy vignette, almost all conventional medicine practitioners rated the CAM practice as ineffective (2–5% “effective”) and were unwilling to recommend them (3–9% “willing to recommend”). CAM practitioners differed sharply from physicians, with many rating CAM therapies as effective (11–65%) and being willing to recommend CAM therapies (20–77%), notwithstanding new evidence to the contrary. (Figure 1)

Figure 1.

Proportions of CAM and Conventional Medicine Practitioners’ Ratings of Effectiveness and Likelihood to Recommend in Hypothetical Vignettes Reporting Efficacy and Inefficacy of CAM Therapies

Factors Influencing Effectiveness Judgments

Physicians did not differ significantly in their judgments of effectiveness for efficacy or inefficacy vignettes by journal source (p=.09 for positive), or type of CAM (p=.94 for positive). Physicians’ effectiveness ratings after the inefficacy vignette were so low they did not permit inferential testing.

In contrast, in unadjusted analyses, CAM practitioners’ judgments of effectiveness were significantly influenced by type of CAM therapy in both the efficacy and inefficacy vignettes (p<.001). Throughout these ratings, acupuncture was viewed most favorably. For the positive vignette, 95% of CAM practitioners rated acupuncture as effective 6% higher than the next highest treatment type (massage) and 19% higher than the lowest rated treatment (Reiki). CAM practitioners continued to rate the effectiveness of acupuncture considerably higher than other CAM treatments even after being presented with evidence showing no benefit of acupuncture (Vignette 2, Table 2). These difference persisted after adjustment for other covariates [OR=5.8 (95% CI: 2.6–12.8) for the positive vignette and OR=15.0 (95% CI:8.3–27.4) for the negative], using Reiki as the referent category. Acupuncture ratings were significantly higher than those for each of the other therapies (p<0.05 for each comparison, for each vignette). Similar to physicians, CAM practitioners’ judgments about the effectiveness of the CAM therapy was not significantly influenced by journal source for either vignette (Table 2).

Table 2.

CAM and Conventional Medicine Practitioners’ ratings of the effectiveness of CAM therapies, stratified by the journal in which research data is presented and the type of CAM studied.

Vignettes and Factors		CAM Practitioners					Conventional Medical Practitioners
		N	“Effective” %	P-value† (unadj.)	OR‡ (adj.)	95% C.I.	N	“Effective” %	P- value† (unadj.)	OR‡ (adj.)	95% C.I.
Research Vignette #1: Clinical Trial of a CAM therapy for osteoarthritis of hand showing improvement in pain scores
Journal Source	Annals of Int. Medicine	439	85	0.4	1.0	--	329	54	0.09	1.0	--
	Journal of CAM	420	83		0.9	(0.6, 1.3)	317	48		0.8	(0.6, 1.0)
CAM Therapy	Reiki (ref)	174	76	<.001	1.0	--	135	49	0.94	1.0	--
	Acupuncture	159	95		5.8	(2.6, 12.8)	122	49		1.0	(0.6, 1.7)
	Glucosamine	156	83		1.5	(0.9, 2.7)	128	52		1.2	(0.7, 1.9)
	Massage	182	89		2.5	(1.4, 4.5)	126	53		1.2	(0.7, 1.9)
	Meditation	188	78		1.1	(0.7, 1.8)	135	52		1.2	(0.7, 1.9)
Research Vignette #2: Clinical Trial of a CAM therapy for carpal tunnel syndrome showing no improvement
Journal Source	Annals of Int. Medicine (ref)	403	25	0.27	1.0	--	319	3
	Journal of CAM	414	28		1.3	(0.9, 1.8)	331	3
CAM Therapy	Reiki (ref)	153	11	<.001	1.0	--	132	2
	Acupuncture	163	65		15.0	(8.3, 27.4)	122	3
	Glucosamine	171	16		1.6	(0.8, 3.0)	125	2
	Massage	169	25		2.7	(1.4, 4.9)	139	5
	Meditation	161	17		1.7	(0.9, 3.2)	132	2

^†Unadjusted p-value from Chi-Square test comparing percent rating “effective” across levels of factor.

^‡Odds ratio from multivariable logistic regression model estimating odds of rating a practice as “effective”, if given a specific factor when controlling for all other factors.

Factors Influencing Willingness to Recommend

Conventional medicine practitioners’ willingness to recommend a CAM therapy in light of new evidence of efficacy was not affected by journal source, patient request or chronic refractory symptoms (Table 3), although there was a trend toward being more likely to recommend a practice supported by positive evidence if a patient requested it [unadjusted p=.05; adjusted OR=1.4 95% CI(1.0 – 2.0)]. However, conventional medicine practitioners’ willingness to recommend CAM was influenced by treatment type (unadjusted p=.003) in the efficacy vignette. As hypothesized, relative to Reiki, the odds ratio associated with conventional medicine practitioners’ willingness to recommend CAM therapies was highest for glucosamine (OR=3.0; 95% CI[1.6–5.4]), followed by massage (1.9 [1.1–3.3]), acupuncture (1.3 [0.8–2.2]), and meditation (1.2 [0.7–2.0]). In subsequent pair-wise comparisons between treatment types, conventional medicine practitioners’ willingness to recommend glucosamine was significantly greater than for acupuncture (p=0.009) and meditation (p=0.004). The other pair-wise comparisons did not reach statistical significance. Conventional medicine practitioners’ willingness to recommend any CAM therapy was so low for the inefficacy vignette that inferential testing was not possible.

Table 3.

CAM and Conventional Medicine Practitioners’ likelihood of being willing to recommend a CAM therapy, stratified by the journal in which research data is presented, the type of CAM studied, whether a patient directly requests the therapy, and whether the patient has suffered chronic, refractory symptoms.

Vignettes and Factors		CAM Practitioners					Conventional Medicine Practitioners
		N	“Likely to Rec.” %	P-value† (unadj.)	OR‡ (adj.)	95% C.I.	N	“Likely to Rec.” %	P-value† (unadj.)	OR‡ (adj.)	95% C.I.
		Research Vignette #1: Clinical Trial of a CAM therapy for DJD of hand showing improvement in pain scores
Journal Source	Annals of Int. Medicine (ref))	437	87	0.94	1.0	--	331	76	0.16	1.0	--
	Journal of CAM	420	87		1.0	(0.7, 1.6)	320	71		0.8	(0.6, 1.1)
CAM Therapy	Reiki (ref)	174	71	<.001	1.0	--	136	65	0.003	1.0	--
	Acupuncture	159	97		12.3	(4.8, 31.8)	125	70		1.3	(0.8, 2.2)
	Glucosamine	156	91		4.0	(2.1, 7.7)	128	84		3.0	(1.6, 5.4)
	Massage	182	90		3.6	(2.0, 6.5)	127	78		1.9	(1.1, 3.3)
	Meditation	186	86		2.5	(1.5, 4.2)	135	69		1.2	(0.7, 2.0)
Patient Request	No	435	85	0.08	1.0	--	341	70	0.045	1.0	--
	Yes	422	89		1.4	(0.9, 2.1)	310	77		1.4	(1.0, 2.0)
Chronic, Refractory Sx	No	421	85	0.13	1.0	--	322	73	0.80	1.0	--
	Yes	436	89		1.9	(0.9, 2.0)	329	74		1.1	(0.8, 1.6)
Research Vignette #2: Clinical Trial of a CAM therapy for carpal tunnel syndrome showing no improvement
Journal Source	Annals of Int. Medicine (ref)	407	38	0.32	1.0	--	320	5
	Journal of CAM	420	41		1.3	(0.9, 1.7)	332	5
CAM Therapy	Reiki (ref)	154	18	<.001	1.0	--	133	3
	Acupuncture	167	77		15.5	(9.0, 26.9)	123	4
	Glucosamine	173	21		1.2	(0.7, 2.0)	124	3
	Massage	172	46		3.8	(2.3, 6.4)	139	9
	Meditation	161	35		2.4	(1.4, 4.0)	133	5
Patient Request	No (ref)	421	40	1.0	1.0	--	330	5
	Yes	406	40		1.0	(0.7, 1.4)	322	5
Chronic, Refractory Sx	No (ref)	415	38	0.44	1.0	--	340	6
	Yes	412	41		1.1	(0.8, 1.6)	312	4

^†Unadjusted p-value from Chi-Square test (Vignette 1) or Fisher’s exact test (Vignette 2) comparing percent “willing to recommend” across levels of factor.

^‡Odds ratio from multivariable logistic regression model estimating probability of “willing to recommend” with journal source, CAM therapy, patient request, and chronic symptoms as predictors.

In unadjusted analysis of factors influencing CAM practitioners’ willingness to recommend a treatment, acupuncture was again associated with higher ratings than were other treatment types (Table 3). Most expressed willingness to recommend acupuncture in the presence of evidence showing benefit (97%) or no benefit (77%). Reiki received the lowest ratings from CAM practitioners: 71% were willing to recommend Reiki when shown the efficacy vignette, but only 18% when shown the inefficacy vignette.

Relative to Reiki, CAM practitioners’ willingness to recommend CAM therapies from the efficacy vignette were highest for acupuncture (OR=12.3; 95% CI[4.8–31.8]), followed by glucosamine (4.0) [2.1–7.7]), massage (3.7 [2.0–6.5]), and meditation (2.5 [1.5–4.2]). In subsequent pair-wise comparisons between treatment types, the ratings of willingness to recommend were significantly greater for glucosamine than they were for acupuncture (p=0.009) or meditation (p=0.004).

When presented with the inefficacy vignette, substantial proportions of CAM practitioners remained willing to recommend most CAM therapies, but the order of preference changed somewhat. Relative to Reiki, CAM practitioners were most willing to recommend acupuncture (OR=15.5; 95% CI[9.0–27], followed by massage (3.8 [2.3–6.4]), meditation (2.4 [1.4–4.0]), and glucosamine (1.2 [7–2.1]). No other factors – journal type, patient request, or chronic, refractory symptoms – influenced CAM practitioners’ judgments for either the efficacy or inefficacy vignettes (Table 3)

DISCUSSION

In this randomized experimental factorial vignette study embedded in a national cross-sectional survey, we found that whether CAM and conventional medicine practitioners believe and are willing to apply the findings from CAM clinical trials appears to depend on which CAM treatments are reported and practitioners’ prior beliefs about those treatments. Physicians rated the effectiveness of different CAM therapies similarly when shown evidence for their respective efficacy. However, conventional medicine practitioners’ willingness to recommend a CAM therapy followed our hypothesized hierarchy of biological plausibility. CAM practitioners showed a strong belief in the effectiveness of acupuncture even when presented with evidence of inefficacy, being willing to recommend it more than any other CAM therapy. We failed to confirm the influence of 3 other factors (journal source, patient request and the existence of chronic refractory symptoms) in the clinical translation of CAM research evidence.

If results of CAM research are to be translated into clinical practice in accordance with evidence-based standards (11), they must be approached with an open mind. Yet, it is reasonable to assume that all practitioners interpret evidence through an interpretive frame that likely combines elements of their education, experience, professional commitments and culture, and clinical experience. These data suggest that in the controversial area of CAM research, practitioners’ prior beliefs and attitudes about specific CAM therapies may constitute part of an a priori interpretive frame through which they view new evidence.(12) And thus, these beliefs may shape whether and how emerging evidence translates into clinical practice. Confirmation biases that reflect preconceptions about the merits of various treatments may work in opposite directions.(13) For instance, if conventional medicine practitioners are at least familiar with and favorably disposed toward a treatment that they find reasonable and plausible, such as massage or glucosamine, they are more likely to interpret new research evidence about that treatment generously. Conversely, if these same practitioners are not familiar with a therapy or cannot make sense of how it might work, as may be the case with Reiki, they are likely to view with skepticism any new evidence for the efficacy of that therapy. Although these data do not definitively establish a confirmation bias, they at least raise that possibility.

Prior studies have demonstrated that conventional medicine practitioners and CAM practitioners hold different attitudes about alternative therapies.(3–5,7,13) These new experimental data suggest that practitioners both—CAM and conventional—may have their own confirmation biases toward new evidence about CAM based on different interpretive frames. Pre-existing attitudes about specific CAM therapies seemed to shape the judgments of both CAM and conventional medicine practitioners, though in different ways.

In the minds of practitioners with a long-held favorable view of a therapy, extensive experience may serve as its own kind of countervailing evidence that in the end is more compelling than the evidence offered by clinical research. Indeed, it may be reasonable to expect that interpreters of new evidence give up their prior beliefs only if the countering evidence is overwhelming in its power, since prior beliefs are more settled and are often accompanied by theoretical explanations that shape broader patterns of thought.

Moreover, in the case of the acupuncturists we surveyed, negative evidence relating to the efficacy of acupuncture, may be resisted because acupuncture is the primary therapeutic tool that these practitioners employ. Alternatively, not only does clinical experience suggest that acupuncture is an effective treatment, but evidence from recent trials indicates that acupuncture is better than no treatment and usual care for relieving pain in various conditions, despite typically being no better than a sham intervention. In short, CAM practitioners may believe acupuncture works despite isolated “negative evidence” because it really does work in many circumstances.(15–17)

A similar dynamic may be manifest in the way conventional medicine practitioners rate the efficacy of and their willingness to recommend different CAM therapies. As hypothesized, the more biologically plausible treatments such as glucosamine received higher ratings (based on willingness to recommend) than did therapies such as meditation and Reiki. However, unlike the CAM practitioners, when presented with evidence of inefficacy, conventional medicine practitioners almost universally reported believing the presented therapy was not effective and were not willing to recommend it to patients.

Why should conventional medicine practitioners seem to have a relatively higher appraisal of acupuncture’s benefit in particular? Many practitioners may be more familiar with proposed plausible mechanisms of acupuncture’s effects involving activating endogenous opioids or more recent neuroimaging studies suggesting that acupuncture has regionally specific, quantifiable effects on relevant brain structures.(18) Or, because acupuncture has been more extensively studied, they may be familiar with the evidence demonstrating the beneficial effects of acupuncture in other chronic pain settings.(15–17)

Can evidence from null or negative CAM trials influence clinical practice? It is possible that attitudes in favor of certain CAM therapies among CAM practitioners – some of which may be justified by emerging scientific evidence -- are sufficiently entrenched that only overwhelming empirical evidence to the contrary could change their minds as demonstrated by CAM practitioners’ preference for acupuncture in this survey. Physicians seem to interpret evidence for more biologically plausible CAM therapies more generously. These experimental results mirror larger debates about physicians’ belief in certain plausible therapies in the face of new evidence (e.g. with respect to studies of pulmonary artery catheterization, knee arthroscopy, and hormone replacement therapy).(19) For both conventional medicine and CAM practitioners, prior beliefs and experiences may hinder deliberate, appropriate, and incremental accumulation and modification of knowledge about CAM therapies.

Study results did not confirm our hypotheses concerning the impact of journal type, patient request, and having chronic refractory symptoms on practitioners’ judgments. Several possible explanations exist. First, these factors may in fact be irrelevant in practitioners’ judgments in the range of conditions we presented. Secondly, we may have failed to present the hypothetical vignettes in a sufficiently realistic manner to ascertain their actual influence on practitioners’ judgments. Thirdly, patients likely seek CAM and CAM practitioners in order to receive particular interventions, making specific information about patient requests less relevant. Finally, if practitioners pay attention to scientific publication at all, they may be much more concerned that a study is published in a peer-reviewed journal with good study design and sound results rather than being concerned about the prestige of the particular journal in which those results are published.

This study has several limitations. First, we examined hypothetical vignettes, which only indirectly support inferences about what practitioners would actually do and what informational factors they would use to make actual clinical decisions. Second, social desirability bias may have influenced respondents. For instance, conventional medicine practitioners who have strong favorable attitudes toward CAM even when the evidence does not support the treatment may be less willing to admit that belief in a survey sponsored by the government research institute. We also did not survey all types of CAM practitioners, making inferences about those groups or all CAM practitioners impossible. Furthermore, this study does not account for the fact that evidence accumulates over time, and that practitioners ideally would be constantly revising their best estimates about the clinical utility of various treatment as new evidence emerges. Our approach cannot capture such subtleties. Nevertheless, this study with rigorous design and systematic sampling suggests that practitioners’ prior attitudes about particular CAM therapies will influence the clinical translation of new research evidence about those therapies. Future investigations may prove fruitful in further characterizing the factors that influence the clinical translation and application of research evidence.(19)

CONCLUSIONS

Practitioners’ judgments about the efficacy of CAM treatments and their willingness to recommend them based on trial evidence depend on the type of treatments reported but not on the source of information, patient request or having chronic refractory symptoms. Physicians are quick to reject CAM therapies when presented with evidence of inefficacy. CAM practitioners are unwilling to give up their belief in the benefit of CAM therapies even when presented with new evidence of inefficacy. Clinical translation of randomized trial data in the area of CAM faces considerable challenges. Clinicians need to be alert to the potential influence of bias in translating new evidence into clinical practice. Research to assess barriers to the translation of clinical trial evidence relating to CAM should enhance the value of this important research.