Principles and Practice of Clinical Research course for surgeons: an evaluation of knowledge transfer and perceptions

Abstract

Background

Knowledge and training in evidence-based medicine (EBM) and clinical research is under-represented in most surgical training programs in North America. To address a lack of resources for surgeons, trainees and related specialties, we developed a Principles and Practice of Clinical Research (PPCR) course. The current study evaluated transfer of knowledge and perceptions about the course.

Methods

The course was an intensive 2.5-day workshop consisting of interactive lectures and small group breakout sessions. Pre- and postcourse tests were completed by participants. The Fresno test, questions from the Centre of Applied Medical Statistics (CAMS) test and questions developed by the course chairs were used to determine if participants’ knowledge of EBM, clinical research methodology and statistics improved. We also elicited participant perceptions of the course.

Results

Overall participant knowledge about EBM and clinical research methods improved significantly from the pre- to the postcourse test (mean improvement in score 13.5%, relative increase 35.3%, p < 0.001). Specifically, improvements were demonstrated on the Fresno test (mean improvement 13.5%, relative increase 36.1%, p < 0.001) and the CAMS test (mean improvement 11.4%, relative increase 20.1%, p = 0.001). Participants showed the greatest improvement in general knowledge about clinical research (mean improvement 15.4%, relative increase 46.5%, p < 0.001). The PPCR course was well received; 30 (81.1%) participants who completed the course evaluation gave it a positive rating.

Conclusion

Participants in a short course focusing on EBM and clinical research methodology had significant improvements in scores on tests of knowledge gained. Widespread implementation of similar courses may bridge knowledge gaps for surgeons, surgical trainees and health professionals. Whether shorter knowledge gains are retained in the longer term remains unknown.

Evidence-based medicine (EBM) is the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.¹ To facilitate EBM in clinical practice, physicians and allied health professionals require a fundamental understanding of the hierarchy of research evidence, study methodology and critical appraisal of health research literature.

Although the concepts collectively referred to as EBM have become popular, many physicians lack the adequate education to effectively appraise and conduct clinical research.² A survey of active hospital physicians found that of the 225 respondents to a self-administered survey regarding their knowledge of EBM methodologic terms, only 10 physicians (4.4%) were able to answer all 12 questions correctly.³ Most physicians surveyed in that study indicated interest in additional EBM education.³ Similarly, Poolman and colleagues⁴ surveyed Dutch surgeons on their knowledge of, competence in and attitudes toward EBM and found that improved knowledge of EBM was associated with younger age and working in an academic or teaching setting (p = 0.004). These authors also found that surgeons who completed the survey were receptive to EBM.⁴

A key requirement in the training of physicians as primary care practitioners, internists or surgeons is the development of skills necessary to critically appraise available literature. CanMEDS guidelines⁵ outline 7 competency areas in which medical students, specialists and medical personnel are required to be proficient. The Scholar role, which requires competency in knowledge and science, includes the ability to assess medical information critically, which makes courses on EBM crucial not only for medical students, but also for other health professionals.⁶ Similarly, beginning in July 2002, the Accreditation Council for Graduate Medical Education (ACGME) instructed all residency programs to require their residents to demonstrate competency in 6 core areas: patient care, interpersonal and communication skills, medical knowledge, professionalism, practice-based learning and systems-based practice.⁷ A recent national survey of orthopedic program directors and selected orthopedic residents reported that barriers to implementation of the core competencies included low priority compared with clinical duties, lack of faculty or resident education and lack of formal orthopedic core competencies.⁷ In light of the survey findings, courses in EBM and surgical research methodology are essential for surgical trainees to meet the core competency of practice-based learning.

Whereas multiple courses exist to enhance EBM and clinical research knowledge, most are targeted toward non-surgical specialties and are not led by clinician–researchers. Resources directed at surgeons have largely included EBM-focused literature series in journals, preappraised resources in journals and short instructional lectures at major surgical meetings. In addition, a recent survey of organizations offering postgraduate EBM courses for physicians concluded that learning opportunities were insufficient to ensure widespread dissemination of knowledge and skills.⁸

To address the lack of educational resources for surgeons and surgical trainees, we developed the Principles and Practice of Clinical Research (PPCR) course, an intensive 2.5-day course designed to educate surgeons, surgical trainees and allied health professionals on the principles of EBM and research methodology. The course faculty predominantly comprised surgeon–researchers, and the course content was primarily surgery-focused. To understand the educational impact of the course, we conducted a prospective study to evaluate whether attendees improved knowledge in EBM and clinical research methods.

Methods

Course concept and design

The 2 course chairs first conceptualized the PPCR course in 2005. They held multiple meetings and informal focus groups and discussions with content experts, including surgeons, surgical trainees and allied health professionals, to develop the course content and select the appropriate course faculty. After multiple reiterations, the course chairs developed content for a PPCR pilot course, which took place in 2008. Feedback from this course was used to make the necessary improvements to the content and format of the 2009 PPCR course.

Course description

The PPCR course aimed to improve the knowledge of its participants on the principles and practice of EBM and surgical research methodology. Over 2.5 days, attendees participated in a series of lectures and small group breakout sessions (Box 1). The plenary sessions spanned a spectrum of issues relevant to surgical research. The structure of the course included lectures given by experienced surgical researchers in sessions focusing on the principles of evidence-based surgery, study designs in clinical research, how to prepare a study protocol, practical aspects of randomized controlled trials (RCTs), practical aspects of non-randomized trials, statistical basics, and writing papers and presenting findings. The major component topics of the PPCR course included the principles of clinical research, research study design, practical aspects of clinical trials, health statistics and disseminating clinical research. The course chairs believed that this program provided surgical residents, clinicians and associated research staff a comprehensive foundation of the present state of health research methodology by covering evidence-based surgery principles, levels of evidence, research ethics, funding opportunities and data management and collection.

Box 1. Principles and Practice of Clinical Research course program.

Plenary sessions

• What is evidence-based surgery?

• Grades of recommendation (GRADE approach)

• Discussion of various study designs, including randomized control trials, clinical case series, case–control studies, prospective cohort studies, in vitro and in vivo study designs

• How to limit bias in clinical research

• Study protocols

• Need for a trial: building the case for doing a randomized clinical trial

• Study design: understanding outcomes assessment, measuring outcomes in surgery, necessary sample size

• Is my trial feasible? Can I actually pull it off?

• Data management — modern approaches

• Ethical considerations/research ethics board approval/consent forms

• Budgeting for a clinical trial

• Trial organization: research coordinators and committees

• Practical statistics for surgeons

• Getting your research published: the well-constructed paper

• Writing your paper for publication

• Finding funding to make your research a success

Session topics

Basic science sessions

• Preclinical proposals: the good, the bad and the ugly

• Bias detective 101

• Getting your research published: the well-constructed paper

• Critical appraisal

Clinical trial sessions

• Research proposals: the good, the bad and the ugly

• Critical appraisal: randomized controlled trial

• Bias detective 101

• Getting your research published: the well-constructed paper

• Critical appraisal: observational study

Course evaluations

To evaluate knowledge transfer to the course participants, we distributed pre- and postcourse tests. The aim of the tests was to measure the course participant’s increase in knowledge of EBM, clinical research methodology and statistical knowledge before and after participating in the course. Attendees completed the precourse tests on the eve of the course. We distributed the postcourse tests at the end of the course and required completion before departure.

Participants completed a questionnaire on demographic characteristics, including age, previous epidemiology training, medical or surgical specialty and university affiliation. Additional questions included the adapted Fresno test,⁹ several questions from the Centre of Applied Medical Statistics (CAMS) test of the University of Cambridge (www.phpc.cam.ac.uk/cams/cams-introductory-statistics-courses/check-your-knowledge-of-statistics/), and several questions developed by the course chairs.

The Fresno test was designed to assess the effectiveness of an EBM curriculum of the family practice residency program at the University of California.⁹ The test comprises 12 open-ended questions, including the presentation of 2 scenarios (modified to include principles of surgical decision-making) requiring the participant to formulate a focused research question, identify the most appropriate research design for answering the question, show knowledge of electronic database searching, identify issues important for determining the relevance and validity of a given research article and discuss the magnitude and importance of research findings.⁹

For example, our postcourse test included the following scenario: “You have just seen Hilda, an 80-year-old woman admitted to hospital with a hip fracture. She has a remote history of myocardial infarction but was otherwise well and ambulating independently prior to her slip and fall. She is terribly concerned about getting her hip ‘fixed’ soon. Given other emergency operating priorities, it does not appear as though Hilda will receive surgery for the next few days. Her daughter asks you if there is any risk to delaying her surgery.” Seven questions about the scenario required that the participant review the given information and write a focused clinical question that would help to organize a search of the clinical literature. The test asked the participant to further develop their approach to the scenario as though they were treating the patient, asking them to determine where a clinician might go to find answers to such questions, what strategy they might use to approach the literature search and what type of study might be best to address the question. The test also asked the participant to specify, upon finding a report of original research, what characteristics of the study they would use to determine if the study was relevant to their patient and if the study was valid.

A recent study validating the Fresno test involved 96 individuals, 43 of whom were residents or faculty members of the Fresno University residency program and 53 of whom were self-identified experts who responded to online advertising.⁹ The validity testing found that the Fresno test showed a wide range of questions and a large difference between novice and expert scores, proving that the test requires a high level of knowledge and understanding of EBM.⁹

In addition, we used several relevant questions from the CAMS test to measure change in statistical knowledge. We adapted 5 questions (total 8 points) from the CAMS test about measures of central tendency and spread, factors that affect sample size calculations and the most effective way to assess the degree of association between 2 continuous variables. The CAMS test also provides a life-like scenario that forces the participant to apply the most appropriate statistical test to a real-life situation, verifying the participants’ understanding of the application of such statistical measures.

Finally, 8 additional statistical questions that requested identification of the most applicable analysis for a specific data set were included. Questions concerned factors that would most affect sample size calculations and the most correct test to compare proportions between 2 surgical treatments. The 8 additional questions were in multiple-choice format and were marked out of a maximum of 8 points.

Combining all 3 tests, the maximum score that a participant could achieve on the precourse test was 140 points; similarly, the maximum score for the postcourse test was 128 points. The pre- and postcourse tests were each divided into a clinical scenario section scored out of a possible 80 points and a research methodology section scored out of a possible 60 (precourse test) or 48 (postcourse test) points. Aside from the clinical scenarios and 2 questions in the research methodology section, the pre- and postcourse tests consisted of the same questions. Two separate members of the course faculty marked the tests.

Participants were also asked to complete a course evaluation to assess their perception of knowledge gained. The evaluation comprised 15 questions that asked participants to rate the course on a scale of 1–5 (1 being the lowest score, 5 being the highest) based on a number of different aspects: organization, completeness, course faculty, effectiveness of the panel discussions and lectures, ability to achieve learning objectives and overall presentation of the course. We considered a score of 4 or 5 to indicate of an overall positive rating. Additionally, 5 short-answer questions further evaluated participant perceptions about the course. These questions asked participants for suggestions on improving the course and their overall appreciation of the course.

Statistical analysis

Data were analyzed using SPSS version 17.0 and were stratified according to the level of training of the course participants, at which point descriptive statistics were reported based on level of training and overall. Dichotomous data were reported as number of participants and proportions. Continuous data were presented as means or medians with standard deviations (SDs). We used descriptive statistics for participant demographic characteristics and for previous experience with EBM and clinical and basic science research. We also reported the mean pre-and postcourse test scores and the mean change in score across each level of training and overall. We scored the tests according to the developers’ guidelines. We used Student t tests to determine if there were differences in the mean pre- and postcourse test scores and relative increases in overall scores across each level of training. Two different people (P.P. and E.K.) independently analyzed the results of all statistical tests to ensure accuracy.

Results

Participant characteristics

Sixty-two individuals participated in the PPCR course. Of these, 50 (80.6%) participants completed the precourse test, and 62 (100%) completed the postcourse test. The mean age of participants was 31.3 (SD 5.3) years, and 29 (46.8%) participants were men (Table 1). Twenty-six (41.9%) participants were surgical residents (postgraduate year [PGY] 1–4). Half of the participants indicated that they did not have any previous epidemiology training. The attendees were from 13 different universities.

Table 1.

Demographic characteristics of course participants

Characteristic	No. (%)*
No.	62
Sex
Male	29 (46.8)
Female	21 (33.9)
Not reported	12 (19.4)
Age, mean (SD) yr	31.30 (5.335)
Previous epidemiology training
None	31 (50.0)
Limited
Course work	8 (12.9)
Bachelor of Science	3 (4.8)
Medical Science	1 (1.6)
FRCSC	1 (1.6)
Moderate to Advanced
MSc	4 (6.5)
PhD	1 (1.6)
Not reported	13 (21.0)
Specialty
Medical resident	26 (41.9)
PGY1	13 (21.0)
PGY2	10 (16.1)
PGY3	2 (3.2)
PGY4	1 (1.6)
PGY5	0
Medical student	2 (3.2)
Surgeon or fellow	15 (24.2)
Research coordinator	7 (11.3)
Not reported	12 (19.4)
University affiliation
McMaster University	19 (30.6)
University of Calgary	6 (9.7)
University of Toronto	5 (8.1)
University of Western Ontario	3 (4.8)
University of Alberta	3 (4.8)
Laval University	2 (3.2)
University of British Columbia	1 (1.6)
University of Manitoba	1 (1.6)
Queen’s University	1 (1.6)
Memorial University	1 (1.6)
University of Montréal	1 (1.6)
University of Ottawa	1 (1.6)
University of Saskatchewan	1 (1.6)
Not reported	17 (27.4)

FRCSC = Fellow of the Royal College of Surgeons of Canada; PGY = postgraduate year; SD = standard deviation.

^*Unless otherwise indicated.

Precourse test scores

For the 50 participants who completed the precourse test, the mean score was 38.2% (range 1.4%–85.0%, SD 21.3). Participants with prior education or self-declared EBM and research knowledge scored higher than those without previous training (53.2% v. 28.3%, p < 0.001).

Postcourse test scores

Overall, the attendees showed a significant improvement in postcourse test scores (absolute increase 13.5%, relative increase 35.3%, p < 0.001; Table 2). Residents and medical students, key targets for the course, demonstrated substantial improvements in knowledge (absolute increase 13.7%, relative increase 33.8%, p = 0.003; Table 2). Participants improved across component scores making up the overall test, including the Fresno test (mean improvement 13.5%, relative increase 36.1%, p < 0.001; Table 3), the CAMS test (mean improvement 11.4%, relative increase 20.1%, p = 0.002; Table 4) and the additional statistical questions (mean improvement 15.4%, relative increase 46.5%, p < 0.001; Table 5).

Table 2.

Change in the overall pre- and postcourse test scores

Participants	Measure; mean (SD) %			Relative increase, %	p value
	Precourse test score	Postcourse test score	Change in score
All attendees, n = 50*	38.2 (21.3)	51.7 (19.1)	13.5 (20.0)	35.3	< 0.001
Surgeons/fellows, n = 15	37.7 (27.3)	49.8 (29.1)	12.1 (16.9)	32.1	0.003
Residents/medical students, n = 28	40.5 (44.5)	54.2 (26.3)	13.7 (40.0)	33.8	0.003
Research coordinators, n = 7	30.3 (16.3)	46.3 (16.7)	15.9 (23.8)	52.5	0.13

SD = standard deviation.

^*Only the attendees who completed both the pre- and postcourse tests are included.

Table 3.

Change in scores on Fresno test of evidence-based medicine questions on the pre- and postcourse tests

Participants	Measure; mean (SD) %			Relative increase, %	p value
	Precourse test score	Postcourse test score	Change in score
All attendees, n = 50*	37.4 (22.0)	50.8 (21.3)	13.5 (21.5)	36.1	< 0.001
Surgeons/fellows, n = 15	37.3 (27.2)	48.5 (31.4)	11.2 (19.2)	30.0	0.009
Residents/medical students, n = 28	39.7 (46.6)	53.3 (28.8)	13.5 (45.7)	34.0	0.006
Research coordinators, n = 7	28.1 (16.4)	46.0 (17.0)	17.9 (24.5)	63.7	0.10

SD = standard deviation.

^*Only the attendees who completed both the pre- and postcourse tests are included.

Table 4.

Change in scores on the Centre of Applied Medical Statistics test questions on the pre- and postcourse tests

Participants	Measure; mean (SD) %			Relative change, %	p value
	Precourse test score	Postcourse test score	Change in score
All attendees, n = 508	56.8 (25.5)	68.1 (21.7)	11.4 (24.3)	20.1	0.002
Surgeons/fellows, n = 15	54.2 (33.3)	70.0 (17.5)	15.8 (34.2)	29.2	0.006
Residents/medical students, n = 28	57.8 (29.7)	70.1 (17.4)	12.3 (62.7)	21.3	0.005
Research coordinators, n = 7	58.0 (21.9)	56.3 (32.3)	−1.8 (40.8)	3.1	0.91

SD = standard deviation.

^*Only the attendees who completed both the pre- and postcourse tests are included.

Table 5.

Change in scores on the course chairs’ questions on the pre- and postcourse tests

Participants	Measure; mean (SD) %			Relative increase, %	p value
	Precourse test score	Postcourse test score	Change in score
All attendees, n = 50*	33.1 (22.8)	48.5 (15.6)	15.4 (25.0)	46.5	< 0.001
Surgeons/fellows, n = 15	26.7 (48.3)	47.9 (21.1)	21.3 (28.7)	79.8	0.003
Residents/medical students, n = 28	35.5 (39.5)	51.1 (36.4)	15.6 (34.4)	43.9	0.002
Research coordinators, n = 7	37.5 (16.1)	39.3 (24.4)	1.8 (28.3)	4.8	0.87

SD = standard deviation.

^*Only the attendees who completed both the pre- and postcourse tests are included.

Participant perspectives

Overall, the PPCR course was very well received by participants; 30 (81.1%) participants gave it an overall positive rating (Table 6). Of the 37 participants who completed the course evaluation, 29 (78.4%) stated that they felt they gained sufficient knowledge to perform and produce impactful research, and 32 (86.5%) indicated that they would recommend this course to others.

Table 6.

Perceptions of course participants derived from course evaluations^*

Statement	Positive scores, no. (%)
The course material was informative, at the appropriate level and well presented	29 (80.6)
The learning objectives of the course were clearly stated and achieved	28 (75.7)
The combination of breakout sessions and panel discussions created an interactive atmosphere	28 (75.7)
The course faculty was easily approachable and accessible	33 (89.2)
There was adequate opportunity to interact with course faculty	33 (89.2)
The course faculty overall	34 (91.9)
The course overall	30 (81.1)
Gained sufficient knowledge to perform and produce impactful research	29 (78.4)
Recommend this course to others	32 (86.5)

^*Thirty-seven respondents.

Discussion

Our study evaluating the knowledge transfer and perception among participants of a novel course directed at surgeons suggests that a short, intensive exposure to clinical research principles and EBM with surgical faculty resulted in a significant improvement in scores on tests of knowledge acquired. The magnitude of increase in our study test scores, greater than 10%, was large and statistically significant.¹⁰

Several other studies have found improvements following a course on the principles of EBM and health research methodology. Straus and colleagues¹¹ evaluated the effectiveness of teaching EBM skills to attending physicians and medical residents on their real-world practice in a community hospital. Participants were given a precourse test, completed 7 training courses on EBM (1 hour each), received a textbook on EBM and completed a postcourse test.¹¹ The study found that participants significantly increased their use of high-quality RCTs and systematic reviews.¹¹ Dorsch and colleagues¹² recently evaluated the impact of an EBM course on medical students’ self-perceptions of EBM skills and performance in applying EBM skills in a simulated case scenario. The authors found a statistically significant improvement in the students’ perception of their skills in formulating a clinical question, finding the best clinical evidence and appraising the validity and applicability of an article.¹² In addition, the authors found statistically significant improvements from the pre- to the postcourse test in structuring a research question; completing a literature search; assessing the validity of an article; understanding the concepts of chance, bias and confounding variables; and describing the adequacy of the components of the study.¹² A similar study measured the effectiveness of an EBM course taught 2 hours per week for 7 consecutive weeks to internal medicine residents.¹³ These authors found a significant improvement in the ability of residents to formulate research questions and search the literature.¹³ They also found a significant improvement in the residents’ quantitative understanding of research.¹³

Our study had several strengths, including the use of a pre- and postcourse test study design, the utilization of 2 previously developed and validated questionnaires and the measurement of participant perceptions. The 3 questionnaires used in the testing covered all of the key aspects of the course content. The study was also strengthened by the timing of the administration of the questionnaires, limiting the risk of contamination.

Limitations

About 80% of the course participants completed the pre-course test, whereas 100% completed the postcourse test. It remains plausible that participants who initially felt uncomfortable with the course material neglected to complete the precourse test. However, the relatively high completion rate limits the extent of this bias. Whereas components of the overall test have been previously used in other settings, the overall aggregate test has not been used or formally validated. Although formal construct validation was not undertaken, we are confident that the test had adequate face and content validity and was sufficiently sensitive to detect improvements in knowledge. For example, the aggregate questionnaire distinguished well between attendees with previous research experience and novices.

Another limitation that should be noted is the wide range of SDs found in participant test scores (Tables 2–5). These findings are a result of extreme test scores; whereas some participants excelled (obtained a score of 80% or greater), others did not (obtained a score less than 5%). This, partnered with a small sample size, accounts for the wide range of SDs we reported. Given that the tests did not impact the success of a participant’s completion and certification of attendance, some may have been less motivated to put effort into the tests, which may have affected their scores. Participants were not anonymous, as tests asked participants for their names to allow us to adequately compare pre- and postcourse test scores. Finally, we cannot generalize the short-term improvements that we observed to sustained longer-term knowledge about clinical research and EBM.

Conclusion

Participants of a short course focusing on EBM and clinical research methodology experienced significant improvements in scores on tests of knowledge acquired. Widespread implementation of similar courses may bridge knowledge gaps for surgeons, surgical trainees and health professionals. Further research will determine the long-term educational effectiveness of surgically focused EBM courses.