Training the Evidence-Based Practitioner

Abstract

An important goal of chiropractic clinical education should be to teach specific evidence-based practice (EBP) skills to chiropractic students, interns, and doctors. Using a nominal group process, the authors produced a document similar to the Council of Chiropractic Education standards for clinical competencies that can be used to drive an EBP curriculum. Standard texts and journal articles were consulted to create the standards for this program and each standard and corresponding learning objective was discussed in detail and was then graded by the committee in terms of importance and the level of competency that should be attained. Six standards and 31 learning objectives were generated with the learning objectives being further divided into lists of specific competencies. It is the hope of these authors that by sharing this document it can serve as a comprehensive and detailed seed document for other institutions.

Introduction

Over the past 10 years, evidence-based medicine has had an increasing influence on clinical education in allopathic medicine.^{1, 2} A number of population-based outcome studies have documented that patients who receive evidence-based therapy have better outcomes than patients who do not.^3–7 Chiropractic educators have also recognized that an important goal of chiropractic clinical education should be to teach specific evidence-based practice (EBP) skills to chiropractic students, interns, and doctors.⁸ Unfortunately, a 2000 survey on the prevalence of EBP teaching revealed that only 18% of chiropractic colleges worldwide required interns to routinely generate clinical research questions or conduct literature searches.⁹ This situation is likely changing, however, due in part to NIH funding of four U.S. chiropractic colleges to strengthen their research/ EBP curricula. A key strategy in moving chiropractic practice onto a firmer foundation of accepted scientific principles is to train a generation of graduates who are literate in the EBP model. Another reason for this evolution in chiropractic education becomes apparent in light of the changes in health care dating back to the 1990s. These changes include the landmark inclusion of chiropractic care by the Veterans Administration, as well as a growing trend of cooperation between allopathic and chiropractic physicians. Although health care models, language, and attitudes may differ, an appreciation of evidence-based outcomes and EBP literacy offers the best common ground to help cement this burgeoning cooperation. Continued cooperation between the two disciplines better serves the needs of the patient and makes for good public policy.

Sackett defined evidence-based medicine as the integration of the best available research evidence in conjunction with clinical expertise and consideration of patient values.⁷ Although a strong platform of evidence does not exist for many aspects of chiropractic or allopathic medicine, clinical decision making should take place in an evidence-informed environment.

The World Federation of Chiropractic identity statement describes chiropractors as having a specialized approach to patients “based on best available and clinical evidence.” If we embrace this premise, then the graduate from a chiropractic college must be able to do the following: formulate a searchable clinical question, rapidly access the best evidence available, assess the quality of that evidence, determine whether it is applicable to a particular patient or group of patients, and decide whether and how to incorporate the evidence into the care being offered (ie, integrate evidence with clinical experience, patient preference, and complexity of the specific case). These skills, proposed by Sackett⁷ and adopted by the Sicily conference on evidence-based medicine,¹⁰ can be used as the foundation for establishing a set of standards and competencies to guide chiropractic education.

We produced a document similar to the Council of Chiropractic Education standards for clinical competencies that we have used to guide the design of an EBP curriculum. This document outlines the standards, general learning objectives, and specific competencies related to educating University of Western States, College of Chiropractic (UWS) students in this discipline. It has served as the blueprint for developing a special EBP curriculum nested within our general curriculum.

A number of basic pedagogical principles helped to shape the needs and content of the document. The focus of the curriculum should be to help students develop the ability to access the best evidence and then evaluate its quality, generalizability, and application to the individual patient. Practicing chiropractors need to utilize these skills at two levels. For literature within their area of expertise, they need to be able to perform an assessment of the quality and methodology of a primary study. Without this ability, they become vulnerable to the assessments of others who may be uniformed, biased, or unskilled. They must also be able to rapidly access, assess, and digest high-quality evidence from prefiltered sources. This skill is necessary for the efficient application of health care information to patient management. Without this skill, it is likely that practitioners will find the whole process of EBP too time consuming and quickly abandon it. A chiropractic college curriculum will need to address both skill sets.

Another guiding principle was that the skills, knowledge, and attitudes necessary to be a practicing evidenced-based doctor should not be learned in isolation, sequestered in a few designated courses or a journal club, but rather should penetrate the entire curriculum. Otherwise EBP behaviors will likely also be sequestered to those same courses and the essential competencies may not be mastered. EBP principles have more impact on acquired knowledge and critical appraisal skills when they are introduced early rather than later in medical and complementary and alternative education.^{11, 12} Therefore, EBP instruction should begin in the preclinic courses, as early as the first term. To encourage EBP as a behavior, not just a knowledge base, the program also needs to be integrated into “the clinical setting and routine care.’′¹³ Randomized trials have demonstrated that learning and applying these skills are achievable goals.^14–16

Methods

In 2005 an EBP curricular committee was formed to construct a new program that would be vertically and horizontally integrated throughout the college. The committee was composed of faculty and administrators. It combined the primary grant investigators with key teaching personnel from the clinics, clinical science division, research division, and library science. We recognized that it was critical to engage representatives from each of the main divisions of the college that would be affected by this project. In addition, a content expert from Oregon Health Science University (OHSU) participated. The ad hoc EPB curricular committee did not include students. It was decided that whereas student stakeholders could offer valuable input regarding how the curriculum would be implemented, they would be less able to contribute to decisions regarding core content. A student representative was brought into the process later when portions of the proposed curriculum moved on to be reviewed by the college's institutional curriculum committee.

The task began with a review of the literature. Keywords used were EBM, evidence-based medicine, EBP, evidence-based practice, teaching, education, curriculum, curriculum guidelines, chiropractic, CAM, complementary and alternative medicine, medical. The databases searched were Academic Search Premier, ERIC, Medline, PubMed, CINAHL. Google Scholar, and Dog Pile; Complete Planet search engines were also employed. McMaster University, Oxford's Center for Evidence-Based Medicine, and Duke University websites were explored. Although some articles reviewed teaching methodology and there were a few small outcome studies, none presented an adequate set of specific competencies or outlined a curriculum as such. One set of standards and learning objectives was published by the nursing school at the University of Texas Health Science Center in San Antonio, but the focus and level of detail was judged to be inadequate for our purposes. The literature review failed to identify a set of specific, detailed, measurable learning objectives in the realm of EBP.

Consequently, it was decided to write an original document entitled “Training the EBP Practitioner: Standards, Learning Objectives, and Specific Competencies.” The goal was to generate a document with sufficient detail to plan a 4-year curricular program and to be amenable to direct inclusion into specific course syllabi. Our next question was what classic content should comprise a curricular-wide EBP program? We are aware of no evidence-based literature that establishes the ideal content for an EBP curriculum, or how to weight this information, or how to establish an appropriate depth of subject. There is no evidence-based pyramid to exploit indicating what content should be taught and in what proportions. The process is essentially built on convention and expert opinion. From an EBP perspective, questions can be divided into background and clinically oriented foreground questions. Our question essentially was a background question. For a background question, current textbooks are acknowledged as a reasonable source. In fact, for our particular question, the classic textbooks by Straus and Guyatt and others represented the most appropriate sources.

A preliminary draft document was constructed by one of the committee members (RL) relying heavily on the basic content of classic textbooks in EBP, especially Straus et al's updated edition of Sackett's seminal text Evidence-Based Medicine: How to Practice and Teach EBM.¹⁷ Another useful resource was Guyatt and Rennie's User's Guide to the Medical Literature.¹⁸ The major teaching issues in these texts were merged and further revised over time drawing from additional text books^{13, 19, 20} and journal articles.^{10, 21–37} While these sources supplied the bulk of the content, the work of the committee was to reorganize the information to mirror the five steps of EBP and reformat it to specific standards and learning objectives.

Revision and Review

The document was then heavily revised using a nominal group process by the EBP Curricular Committee. Each learning objective was discussed in detail. The editing, content, and intent of the learning objectives were influenced by the varied perspectives of the participants. Participants were faculty and consultants, who were already teaching various components of EBP, practiced EBP, taught research methodology courses and participated in research, or contributed information literacy expertise. The teaching experiences and personal opinions of the participants significantly shaped the document. For example, some learning objectives are directed at understanding the dangers of overreliance on preappraised literature because of potential biases of the reviewers. Others target the behavior of reading only abstracts because abstracts do not always accurately reflect the data or the conclusions found in the paper. These specific learning objectives are not readily found in the classic textbooks but resonated with the experience of the panel. Finally, the standards and learning objectives were weighted by the committee in terms of importance and the level of competency that should be attained by students progressing through the program.

The development of the EBP Standards and Learning Objectives was committee-based and intensive. The committee met on a weekly basis from August 2005 through May 2006 to develop the document. It has continued to meet from August 2007 to the present. The rollout of the new curriculum began in the summer of 2007 and was completed in January 2009. We are continuing to gather data as each new cohort of students moves through the program. All major EBP changes went through the college's curriculum committee, which required approval from a larger group of faculty, cutting across all of the academic divisions, and included student representation. Many of the curricular changes proposed in the EBP plan were expedited because the changes constituted internal changes within existing research methodology courses and clinical thinking courses. Because the instructors of these courses were already committee members, it was judged to be within their purview to alter the emphasis and some of the content of their own courses without a full review by the larger UWS curriculum committee.

Grading the Learning Objectives

The standards have gone through a vetting process in which committee members have assigned a point value to the role or importance of each item (Table 1). A scale was developed with each grade operationally defined, blending student performance goals (eg, “minimal competency necessary”) with practical implementation descriptors (eg, “very likely the focus of test questions”). Each objective was assigned a weight of 1 to 3 by five members of the EBP Curricular Planning Committee. The Director of Library Services graded only EBP standards 1 and 2, bringing the tally to six individuals for those two standards. The average score is listed after each competency. It can be interpreted as follows. A 1.0 indicates that all of the members who graded the item felt that the competency/objective is an important basic skill. A grade of 1.3, for example, indicates that there was majority agreement; however, not all thought it was of first-level importance and some relegated it to a lower level. A grade of or near 3 represents uniform agreement that the objective is for expert-level competency and should not be a competency for UWS graduates. The most useful grades are those closest to 1 and 3 because they represent clear consensus. A grade of 2 can indicate either a relative consensus or, instead, a wide disparity of opinions. In either case, an intermediate grade was assigned. An actual breakdown of each vote for each item is available upon request. Each category includes practical descriptors, such as the likelihood that the information would appear as a test question or would likely be referred to in non-EBP courses.

Table 1.

Grades for the learning objectives

Value	Operational Definition	Descriptors
1	Basic/minimal competency for the graduate	Minimal competency necessary for graduates to apply EBP to their practice. Very likely the focus of test questions and practice assignments. Student must demonstrate consistent mastery. More likely to permeate teaching in diagnosis and management courses.
2	Advanced competency	Very useful but not absolutely necessary to master in order to apply EBP in practice. May appear in test questions and some assignments. Students may be asked to gain some experience with the knowledge or skill but will not be assessed for consistent mastery. Less likely to permeate teaching in diagnosis and management courses.
3	Expert level competency	More appropriate for one interested in becoming involved in research. Intended more as exposure or enrichment and may be sacrificed in the interest of time in a course. Seldom or never the object of test questions or focused assignments. Not likely to permeate the material in other courses; more likely found only in specialty courses on research or EBP.

Results

The Sicily consensus group¹⁰ identified five main principles which we have expanded into six standards, adding a standard regarding an understanding of EBP in general, its strengths, weaknesses, and process. There are six standards in our document:

1. 1. The EBP competent practitioner can present an overview of EBP.

2. 2. The EBP competent practitioner can translate an issue of clinical uncertainty into an answerable question.

3. 3. The EBP competent practitioner can efficiently and effectively search for and retrieve useful and up-to-date health care information and evidence.

4. 4. The EBP competent practitioner critically appraises the evidence for validity and clinical importance.

5. 5. The EBP competent practitioner applies the relevant evidence to practice.

6. 6. The EBP competent practitioner engages in self-evaluation of his or her process for accessing, appraising, and incorporating new evidence into practice.

Each standard is divided into general learning objectives. In turn, the 31 learning objectives have been further elucidated by lists of specific competencies. For standard 1, for example, learning objective 1 focuses on one's ability to describe EBP. Competencies include being able to define EBP and other essential terms and outlining the steps of applying EBP.

The standards and competencies were designed to serve as a foundation for an entire program and not just a single course or two. Some learning objectives have been added that are not usually emphasized in classic EBP, but were judged to be important components of information literacy. These topics include the role of disease-oriented research, expert opinion, evidence-based textbooks, and motivational interviews. The complete document can be accessed by contacting the authors and a 35-page abbreviated version can be found online as a free separate electronic appendix for this issue (volume 25, number 1) at www.journalchiroed.com.

Discussion

The standards document was used to drive the design of our EBP curriculum. The purpose of this document was not to generate ground-breaking content, but rather to organize the content of EBP as reflected in a number of well regarded EBP textbooks. Our 2-year experience in organizing and editing our learning objectives left us with the realization that if we had had access to such a seed document, our large-scale curricular planning would have proceeded much more rapidly and efficiently. The subject matter as reflected in the standards document is much more complete and usable for curricular planning and course syllabi than as is currently found in any single textbook. The document blends input from multiple sources—something useful when planning a curriculum, but very time consuming to create “from scratch.” In addition, some readers might find the rating system a unique tool and the ratings themselves of interest if only to reflect the status of the individual objectives at another institution. Many of the items can serve as draft statements for further refinement into more focused and measurable competencies. Until such a standards document is created on a truly multiorganizational consensus basis, individual institutions are better off refining these learning objectives to conform with their own goals, time limitations, and teaching requirements. Some of the items in the document are already at an acceptable stage of development to be judged both actionable and measurable. The document is very thorough and its completeness has been a virtue in that it remains a convenient and, more important, a comprehensive menu from which to select content and to prioritize skill acquisition. We offer it in its complete form on the JCE website so others can follow its evolution, access a comprehensive set of potential EBP learning objectives, and select those judged most appropriate for their circumstances. We have also found the granular level of detail to be useful in our attempt to construct a coordinated program across multiple courses, various instructors, several academic divisions, and 4 academic years. Very explicit content in the learning objectives allows for more effective communication and coordination. Having been extensively engaged in EBP curricular building, we think that this document would be valuable to anyone involved in a similar endeavor.

Although the standards and learning objectives were intended to drive a 4-year curriculum, the learning objectives in this document and their weightings remain overly ambitious. In short, the new curriculum put in place does not and will not fulfill all of the learning objectives envisioned in the document. In addition, many of the individual learning objectives will need to be revised to more accurately and realistically embody exactly what a student is expected to be able to do—many of the items suffer from vagueness in this respect, serving more as a content listing as opposed to a demonstrable, measurable objective.

Another significant limitation is that the standards and competencies presented here have not yet been thoroughly tested. In fact, the effectiveness of the learning objectives cannot be measured directly, only the effectiveness of the curriculum which is based on them. Ultimately, success must be measured as changes in behavior and clinical practice. We are currently gathering data on the implementation of the competencies throughout the curriculum, as well as student EBP knowledge, attitudes, and practice behaviors. We are also developing an alumni survey to measure if EBP is applicable to their patient populations and if they incorporate the use of evidence in their clinical decision-making process. The process of curricular design implementation, faculty development, and outcomes will be published elsewhere.

A final limitation is that the sense of what is an appropriate learning objective is itself a moving target. As we teach this material and coordinate the overall EBP curriculum and become more experienced and sophisticated, the expectation of competency acquisition is changing. Although the knowledge and experience base of the committee ranged from research to EBP, the center of expertise tilted somewhat more in the direction of the researchers than the clinical users. Since the beginning of the process, however, key committee players have received additional training through both the Oxford and McMaster EBP programs for teachers as well as focused EBP training in the Human Investigation Program at OHSU. This evolving expertise coupled with the experience of implementing the new curriculum will trigger further revisions and refinements. It has become clear that a “2.0” version of the document is needed.

The process of evolving changes and constructive revision in our EBP standards and curriculum can be illustrated by the following example. One specific competency required students to be able to calculate odds ratios (ORs) from data found in research articles. This competency was delegated to a public health course and an assigned Harm article. In the practical classroom setting, it quickly became clear that the ORs often reported in a study have already been filtered through sophisticated formula (to account for a variety of confounders), making it impossible to reconstruct simple OR calculations from the raw data cited. We have concluded that it is more useful to simply demonstrate the calculations so that students comprehend the general concept. Our expectations have shifted to an emphasis on the ability to interpret the weight and ramifications of odds ratios rather than being able to do the calculations.

Whereas this particular change has already been made, many other learning objectives are expected to undergo further review and revision. One area in particular that will continue to undergo the most radical shifts in emphasis is biostatistics. It is likely that the amount of biostatistics content, level of detail, and performance expectations will be recalculated and de-emphasized.

Conclusion

The initial phase of our curricular planning project resulted in generating a set of standards, learning objectives, and specific competencies that we have used as a blueprint to strengthen EBP teaching and learning within our curriculum. Although intended primarily for internal purposes, we realized that if such a document had already been available in the literature, it would have saved an enormous amount of time during our planning stage and circumvented the need to create such a document. Its strengths are that it is comprehensive, combines much of the content found in well regarded EBP textbooks as well as other articles, organizes the material into the five-step rubric for EBP, and begins the difficult process of turning this material into learning objectives and specific competencies. Its weaknesses are that it reflects the biases and limitations of the committee who constructed it, it needs to be evaluated for effect on student learning and practice, and it requires much additional work to further craft each learning objective into language which is clearly attainable and measurable. It is not offered as a proposed set of standards generalizable to the field, but rather as a potentially valuable seed document for educators planning either large curricular changes or creating more specific competencies for a particular course. Over the next several years we will be revising this document based on faculty and student input and other outcomes that we have been measuring as part of the R-25 grant during the rollout of the new curriculum. We are hoping that its publication will ignite discussion and similar projects at other institutions, which ultimately may result in a generalizable, validated curricular blueprint with broad appeal.

Funding Sources

This study was supported by the National Center for Complementary and Alternative Medicine, National Institutes of Health, and Department of Health and Human Services.

Conflicts of Interest

The authors reported no conflicts of interest.