A TRAIN THE TRAINER MODEL FOR INTEGRATING EVIDENCE-BASED MEDICINE INTO A COMPLEMENTARY AND ALTERNATIVE MEDICINE TRAINING PROGRAM

Abstract

Background:

Public interest in complementary and alternative medicine (CAM) has grown over the past decade, accompanied by increased demand for evidence-based approaches to CAM practice. In order to define the role evidence-based decision making has in CAM practice, CAM professionals must have a full understanding of evidence-based medicine (EBM) concepts.

Objective:

This paper describes the design, implementation, and evaluation of a week-long intensive EBM short course for CAM faculty at a naturopathic and classical Chinese medicine institution.

Intervention:

This 20-hour course, entitled Principles of EBM for CAM Professionals, teaches participants how to access and appraise biomedical literature, apply it to their work, and teach these concepts to their students.

Results:

Results from precourse and postcourse evaluations suggest that, in a small group of participants, there were significant changes in EBM practice attitudes, self-appraised skills, and objectively assessed skills as a result of this course. Participants indicated they were committed to increasing their use of EBM in practice, enhancing EBM skills, using EBM in teaching, and working to change the culture at their institution to support use of EBM. At six months, 80% of participants had fully or partially followed through on their commitment to change plans.

INTRODUCTION

Over the last 10 to 15 years, public interest in complementary and alternative medicine (CAM) has surged, with 38% of adults using some form of CAM therapy by 2007.¹ Accordingly, interest in CAM careers is growing, with rapidly escalating enrollment in the nation’s naturopathic medical schools (personal communication with K. Howard, Executive Director and CEO of the American Association of Naturopathic Physicians, October 2009). Over a similar time frame, the concept of evidence-based medicine (EBM) has evolved from a novel idea to a widely accepted approach to clinical decision making in allopathic medicine. In EBM, medical decisions are made by identifying, critically evaluating, and applying relevant information to patient care.² This information is derived from a variety of sources, including clinical experience, clinical and laboratory diagnostic measures, patient preference, and medical literature. It is important to note that the medical literature is a component of the evidence used to make a clinical decision, but not the only source of information.

Many allopathic training programs have incorporated curricula to educate students in EBM, and this training is required for accreditation in allopathic undergraduate and graduate programs.³ However, EBM is a relatively new concept in CAM training and practice. Evidence-based medicine–related accreditation standards for naturopathic medical schools have only recently been adopted,⁴ and requirements for Chinese medicine schools are limited to doctoral programs, not master’s programs.⁵ Complementary and alternative medicine providers are inundated with new information on diagnostic and therapeutic treatment options by the press, patients, and peers. Most practitioners and teachers of CAM disciplines receive little training in EBM, thus expertise in EBM and research methodology is limited.⁶ Critical first steps in encouraging communication between allopathic and CAM providers include improving allopathic provider understanding of CAM therapies and educating CAM professionals in research and EBM.^7–9

In order to define the role evidence-based decision making has in CAM and integrative practice, CAM educational leaders must have a full understanding of EBM concepts and process. From this foundation, these leaders can determine how EBM concepts should be implemented using a CAM frame of reference and develop EBM-related undergraduate training accreditation standards. In recognition of this knowledge gap, the National Center for Complementary and Alternative Medicine announced a program to “enhance CAM practitioners’ exposure to, understanding of, and appreciation of the evidenced-based biomedical research literature and approaches to advancing scientific knowledge.”¹⁰

In 2007, research faculty at the National College of Natural Medicine (NCNM), a naturopathic and classical Chinese medicine school in Portland, Oregon, along with investigators at the Oregon Health & Science University (OHSU) successfully competed for a Research in Complementary and Alternative Medicine Program grant. A fundamental goal of this program is to train a team of selected faculty (Vanguard faculty) in research content, EBM, and research teaching strategies. This core faculty group is then responsible for training additional faculty, developing and implementing curricular change institution wide, and disseminating content more broadly. The Vanguard faculty initiative has four components: intensive training in EBM, professional skills enhancement, peer and mentored support, and, ultimately, the utilization of these skills to incorporate EBM into the curriculum. In this paper, we describe our experience in developing and implementing an intensive EBM course to train the Vanguard faculty.

METHODS

We assembled a multidisciplinary team to develop an intensive EBM curriculum for CAM professionals. The team consisted of two faculty from NCNM with expertise in CAM research and CAM clinical practice, and two faculty from OHSU with expertise in EBM education and research methodology. We also obtained external guidance from the Society of General Internal Medicine EBM Taskforce. Funding for this project was provided through a National Institutes of Health, National Center for Complementary and Alternative Medicine educational training grant.

Needs Assessment

Five NCNM faculty attended a national EBM workshop (9th Annual Rocky Mountain Workshop on How to Practice Evidence-Based Medicine, Colorado Springs, Colorado, August 2007). During two debriefing sessions held upon return from this workshop, faculty members were asked to describe their perceptions and experiences with the workshop, future recommendations for workshops in EBM, perceptions about EBM, ideas on how to incorporate EBM concepts into their clinical or class curricula and recommendations on what they would need to do so, and perceived barriers to incorporating these concepts. Two of the most frequently discussed concepts in these sessions were the importance of including CAM-specific topics and literature in future EBM workshops and the benefit of having a small teacher-to-student ratio in these workshops.

In addition, we conducted a literature review to explore the larger experience in conducting EBM training programs for CAM providers and identification of learning needs. Content from both the debriefing and the literature review guided our curriculum development process.

Description Of Educational Intervention andTarget Audience

The participants consisted of core faculty members at NCNM. Invitations were extended to faculty members with active involvement in teaching in classroom and clinical venues throughout the four-year training programs.

The 20-hour course, entitled Principles of EBM for CAM Professionals, teaches participants how to access and appraise biomedical literature, how to apply it to their work, and how to teach these concepts to their students. The purpose of the course is to provide a short-term, intensive introduction to EBM concepts for CAM faculty and encourage them to think about how they can incorporate these concepts into their curricula. The EBM course is structured to occur over five half days, allowing the faculty to maintain routine clinic and education roles during the afternoon. The specific learning objectives and content of each half day is presented in Tables 1 and 2.

Table 1.

Learning Objectives for the EBM in CAM Course

	By the end of the short course, participants should be able to:
General	Define the role of evidence in making clinical decisions Identify unique issues related to use of evidence-based medicine in a complementary and alternative medicine practice
Framing questions Resources	Formulate clinical questions using a framework that facilitates finding answers Understand how the type of clinical question influences choice of resource Become familiar with evidence-based resources, including CAM-specific resources, and understand the strengths and limitations of each Categorize resources into four levels: studies, syntheses, synopses, and summaries Navigate one resource from each level of evidence
Validity assessment	Critically appraise the validity of an article about therapy and interpret the results Critically appraise the validity of a systematic review and interpret the results Critically appraise the validity of a practice guideline and interpret the results
Application Using and teaching EBM	Apply the evidence to decision making for individual patients Identify opportunities to incorporate EBM into their own practice Identify opportunities to role model use of EBM techniques Identify opportunities and venues to teach EBM content

EBM, evidence-based medicine; CAM, complementary and alternative medicine.

Table 2.

Short Course Schedule

Day One	Definitions/steps in EBM process
	Asking clinical questions
	Resources: hierarchy of medical evidence, demonstration of primary data sources
Day Two	Critical appraisal: therapy
	Resources: navigating secondary resources
	Discussion: CAM-specific issues in identifying, interpreting, and applying evidence from therapy articles
Day Three	Critical appraisal: systematic review/meta-analysis
	Resources: CAM specific
	Discussion: CAM-specific issues in identifying, interpreting, and applying evidence from systematic reviews/meta-analyses
Day Four	Incorporating EBM in day-to-day practice
	Role modeling and EBM teaching strategies
	Critical appraisal: clinical practice guidelines
Day Five	Small group teaching presentation: therapy
	Small group teaching presentation: systematic review
	Postcourse assessment

EBM, evidence-based medicine; CAM, complementary and alternative medicine.

The course content was created with the principles of adult learning theory in mind,¹¹ with an emphasis on experiential learning.¹² The content includes formal lectures, open discussions, small group work, and real-time internet searches. The course is team taught, with instructors from NCNM, OHSU and the Portland VA Medical Center. After learning how to appraise articles from both CAM and non-CAM peer-reviewed journals during the course, participants form groups and lead their peers in an article appraisal exercise on the last day of the course. This exercise is designed to give them experience teaching this content in a classroom setting.

Outcome Assessment

Prior to starting the course, all participants were asked to complete a survey assessing degree of prior exposure to EBM, attitudes toward EBM, current application of EBM in practice, use of EBM resources, and skills self-assessment (Table 3). We used a survey previously piloted and validated on a group of Canadian general internists.¹³ We also conducted an EBM skills test consisting of sets of six multiple choice questions, each requiring three potential responses.^14,15 This instrument has established content validity, internal consistency, discriminative power (ability to discriminate between different levels of expertise), and responsiveness.¹⁶ Postcourse assessments included attitudes¹³ and skills self-assessment.^14,15 Participants also were instructed to complete a “commitment to change” document, where they listed goals for incorporating EBM concepts into both classroom and clinic teaching, as well as their own clinical and scholarly work. Six months later, participants were asked if they were successful in implementing these goals and, if not, what the barriers to change were. The commitment to change model has been established as a valid predictor of actual change in practice^17,18 and has been used to evaluate changes in medical teaching practice.¹⁹

Table 3.

Sample Evaluation Measures

Evaluation Scale	Sample Questions and Statements
Research attitudes  Participants were asked to indicate their level of agreement with statements regarding clinical research and practice (strongly agree to strongly disagree).	“Research evidence is more important than previous clinical experience in choosing the best treatment for a patient.” “Because most clinical research articles report results for groups of patients rather than individuals, their applicability to the care of an individual patient may be unclear.”
Clinical practice attitudes  Participants were asked to indicate their attitudes toward EBM and the use of information resources in guiding clinical decisions (very positive to very negative).	“How would you rate your attitude toward the potential role of evidence-based medicine in clinical practice?” “How often do you use the following information sources to help guide your clinical decisions?” Information sources include clinical experience, review articles in medical journals, articles from focused searching of databases, clinical practice guidelines, and others.
EBM attitudes  Participants were asked to indicate their level of agreement with statements regarding EBM (strongly agree to strongly disagree).	“EBM devalues clinical experience and intuition.” “EBM leads to more cost-effective practice.” “Physicians must be able to distinguish methodologically sound from poor research.”
Skills self-appraisal  Participants were asked to rate their skills in EBM and research literacy (very competent to not at all competent).	“Formulating a clear question based on a patient problem.” “Evaluating the methodology of published studies.” “Teaching others to conduct literature searches.”
Knowledge test  Participants were asked questions to gauge their knowledge and understanding of EBM and research literacy concepts.	Questions evaluate learners’ knowledge and understanding of EBM and research literacy concepts such as RCTs, meta-analysis, odds ratios, confidence intervals, and systematic reviews.

EBM, evidence-based medicine; RCT, randomized controlled trial.

Statistical Analysis

Baseline and follow-up scores were created for the various attitudinal items so that high scores represented agreement with an EBM focus. In the self-assessed skills and attitudes survey,¹³ the five “practice attitudes” items and the eight “EBM attitudes” questions were summed. Due to their strong correlation, the two single-question attitude items “attitude toward EBM in clinical practice” and “attitude toward EBM clinical practice guidelines” were combined to create a single value.

The seven items from the skills self-appraisal were summed. High scores indicated a participant felt very competent using their EBM skills and applying them in teaching. Baseline and follow-up scores were created for the 18 objective knowledge items.^14,15 A paired t test was used to compare the baseline to follow-up scores. Commitment to change goals were categorized by course administrator group consensus. After six months, participants reported whether their goals had been achieved.

RESULTS

Eleven NCNM faculty members participated in the course, including department chairs and a NCNM librarian. Course participants had a range of professional backgrounds, including naturopathic physicians, Chinese medicine practitioners, doctors of philosophy and medical doctors.

Table 4 shows the comparison between baseline and follow-up for the self-assessed EBM knowledge and attitudes questions. Table 4 also includes the reliability (Cronbach α) for each attitudinal scale. The three attitude scales all had modest reliability, and the self-appraisal scale had high reliability. All outcome measures changed in the desirable direction. Practice attitudes toward EBM were significantly more favorable after training than before. Self-appraised ratings of competence at applying EBM knowledge and using it in teaching were significantly greater after training.

Table 4.

Baseline to Follow-up Comparison for Attitude and Knowledge Questions

Scale (Range)	Baseline	SD	Follow-up	SD	t Test	P Value	Reliability
Research attitudes (1–7)	4.4	1	4.9	0.8	2.8	.02	.65
Clinical practice attitudes (1–5)	3.4	0.8	4	0.8	2	.04*	.69
EBM attitudes (1–5)	3.7	0.4	3.9	0.5	1.6	.07*	.59
Skills self-appraisal (1–7)	3.6	1.2	4.3	1.2	3.5	.01	.90
Knowledge test (0–18)	11	3.5	14.2	1.8	2.8	.02

All df = 9. High means indicate greater agreement with evidence-based medicine (EBM) values, greater feelings of EBM competence, or correct responses to EBM knowledge questions.

^*One-tailed.

Participants’ commitment to change goals sorted into four broad categories: increasing use of EBM in clinical practice, developing EBM skills such as searching and appraising articles, incorporating EBM principles into classes, and helping to create and sustain an EBM-friendly culture at NCNM by bringing in outside speakers, improving faculty access to resources, and encouraging colleagues in their EBM work. Six months later, participants reported that 80% of the goals had been achieved. Reported barriers to achieving these goals included lack of time, EBM resource inaccessibility, and the need for more hands-on practice with EBM concepts. Although EBM knowledge was good prior to the course, it was nonetheless significantly enhanced through training.

CONCLUSION

The use of evidence-informed practice is gaining momentum in the CAM field but there is no consensus on the best way to educate CAM clinicians, academic faculty, and students in EBM concepts. There are a number of nationally based workshops (McMaster University and the Rocky Mountain EBM workshops are two examples), whose short-term, intensive classes fit in well for faculty schedules, but these workshops are not targeted to a CAM audience. Developing a CAM-focused EBM curriculum is further challenged by the small body of peer-reviewed literature on CAM therapies, as they are practiced by CAM practitioners.^20,21 Nevertheless, a number of groups have been developing EBM curricula in CAM settings with varying strategies.^6,9,22–25 In this manuscript, we have demonstrated that a week-long, intensive course targeted toward CAM professionals effectively enhanced EBM knowledge in this community, and improved participant EBM practice attitudes, self-appraised, and objectively assessed EBM skills. Future evaluations will investigate broader outcomes of this course, including change in participant behaviors, wider organizational changes at NCNM, and change in student learning and performance.^26,27

In developing and administering the course curriculum, four key issues arose. First, based on needs assessments, it was important to adapt the course to the interests of CAM professionals. Consequently, the topics chosen to prompt development of well-framed clinical questions and demonstrate literature searches had to be relevant to naturopathic physicians and Chinese medicine practitioners. Likewise, articles chosen to illustrate literature appraisal methods had to be CAM related.

Secondly, course administrators found it challenging to find articles that were both relevant to course participants and conducive to teaching EBM concepts. When introducing concepts in study validity assessment, it is more straightforward to start with high-quality, randomized controlled trials. Studies used for instruction of EBM concepts like “number needed to treat” and “relative risk,” need to include dichotomous outcomes. Although there are many such articles in the medical literature, very few studies of this quality and type have been conducted on CAM therapies. The paucity of randomized controlled trials also means that few conclusive systematic reviews have been done on CAM therapies, and there are few CAM-specific EBM reviews and summary resources available. In addition, the studies that have been conducted on CAM therapies rarely reflect the individualized, multimodality nature of how CAM interventions are actually delivered by practitioners.²⁸

Thirdly, we encountered varying opinions among course participants about the relevance of EBM in their work. As described by Mills et al²¹ in a 2002 paper, many CAM professionals maintain that the individuality of treatments and the philosophical differences inherent in CAM approaches to disease are often incompatible with standardized research protocols like randomized controlled trials.²¹ As such, it was important to include time in the course to explore barriers to application of EBM principles in CAM. Participants were encouraged to recognize where evidence in the literature was sparse and weigh available evidence in other forms, such as ancient texts or personal clinical experience.

Finally, course leaders recognized the importance of having participants practice teaching EBM content. Thus, we reserved the last day of the course for participants to lead critical appraisal sessions for their peers. With only a week of instruction, participants were tentative about their readiness to teach EBM but ultimately felt this best prepared them for the classroom.

There are several limitations to our curricular design and course evaluation. Course participants were primarily naturopathic physicians or Chinese medicine practitioners based at one educational institution. As such, our findings may not extend to other types of CAM professionals in other practice settings. By design, participant numbers were small, thus limiting our ability to draw broader conclusions from the evaluation of change in attitudes toward EBM and acquisition of EBM knowledge.

To ensure sustainability of the EBM curriculum at NCNM, several steps are being taken. Former participants will teach modules in upcoming short courses and will provide seminars and one-on-one sessions for their peers. Course leaders are developing an EBM teaching resource guide for CAM faculty. In addition, the short course curriculum will be expanded into a 12-week class for NCNM students and presented as workshops at national naturopathic and Chinese medicine conferences.

Improving CAM faculty understanding of EBM, enhancing skills in EBM techniques, and changing attitudes toward this practice approach is a critical first step in building a CAM educational infrastructure and curriculum supportive of incorporating evidence in decision making. Complementary and alternative medicine providers who can interpret evidence and understand the role of evidence in clinical practice will be better equipped to participate in clinical research, work with their allopathic counterparts to integrate care, and take steps to improve patient care.