Problem-Based Learning in Anesthesiology: An Evaluative Study of a Medical Student Clerkship

Abstract

Background

Problem-based learning (PBL) in medical education has enjoyed widespread acceptance in recent years, particularly in the basic sciences. However, it has been used less frequently in the clinical education of medical students, and quite infrequently in medical student education on anesthesiology. Critics of PBL emphasize cost factors and a lack of evidence of superior educational outcomes.

Methods

This study reports evaluation data on the use of PBL with fourth year medical students who rotated through a required clerkship in clinical anesthesiology and pharmacology over the course of three academic years. Students were asked to complete evaluation questionnaires concerning the clerkship, with particular attention to PBL and the performance of PBL preceptors. Preceptors were also asked to self-rate their performance by using identical questionnaires. Standardized items across all three years facilitated analysis of student and faculty satisfaction as one outcome of the PBL process.

Results

Results indicate that student response to PBL was mixed. Students rated PBL preceptors more favorably than the PBL process itself, and rated resident physician preceptors more favorably than attending physician preceptors. All preceptors rated their own performance lower than did their students. Significant differences between student and faculty opinion of PBL were also noted. Interpretation of results are limited by less than ideal response rates; nevertheless, valuable insight was gained into the perception of PBL in an anesthesiology clerkship.

Conclusions

The success of this educational method may be dependent upon certain factors related to “educational context”. Further study is needed of contextual and other factors related to the use of PBL in medical education.

Introduction: What is Problem-Based Learning?

Problem-based learning (PBL) is an educational methodology which places emphasis on inductive, problem-oriented education. Since its initial introduction into medical education at McMaster University (Canada) in the mid-1960s, PBL has gained widespread acceptance among medical educators. It is now being used in over sixty medical schools worldwide, with many others in the process of implementing PBL as part of their curricula (1).

As applied in medical education, PBL is described as a learning process which involves a clinical problem serving as a focus or stimulus for the application of problem-solving or reasoning skills. This clinical problem (often in the form of a real or fictitious patient case) stimulates the search for information needed to understand the mechanisms responsible for the problem and how it might be resolved. Students involved in PBL generally work in small groups of six to ten students, under the guidance of a tutor or preceptor, whose role is to facilitate learning by group members in a non-directive way.

A related feature of PBL is an emphasis upon integrated learning, that is, an attempt to combine basic and clinical sciences across the entire curriculum. Such integration might be horizontal (i.e., within a given curricular year), vertical (i.e., between curricular years), or both. It is felt that an integrated curriculum enabling medical students to learn factual information within the context of patient care (where it will eventually be used) will result in higher achievement by those students. It is also suggested that integrated learning will train physicians to retain knowledge and practice lifelong learning (2-3).

Two other issues associated with PBL are: 1) the likelihood that students will achieve greater levels of competence in so-called “soft” areas of medical practice (e.g., patient communication, ability to function as part of a health care team, possession of ethical/humanistic qualities); and, 2) the increased level of student satisfaction with PBL as a more humane approach to training future physicians. A major application of PBL in medical education has been its use as a vehicle to evaluate students’ non-cognitive traits, most often measured as faculty ratings of students on attitudinal and process-oriented variables (4-5).

PBL has not, however, been without its critics within medical education. It has been described as too labor-intensive, too expensive, and not justifiable in light of the actual learning outcomes achieved by medical students (2). Some medical school faculty have reservations about the merits of PBL (6). The medical education literature is replete with differing opinions regarding the effectiveness of PBL in such areas as academic achievement, student and faculty satisfaction, students’ clinical performance, students’ study behaviors, specialty choice of PBL-trained students and/or resident physicians, and overall implementation issues (7-9). Indeed, the authors of three literature review articles concerning PBL reach different conclusions about the outcomes of PBL (4).

In summary, problem-based learning involves a new, still-evolving paradigm which was developed as a result of general dissatisfaction with the more traditional, lecture-based approach to medical education (10). PBL attempts to incorporate the principles of adult learning theory in a manner which increases the effectiveness of medical education by emphasizing learning processes and non-cognitive characteristics of professionalism.

Materials and Methods

Problem-based learning became a formal part of curricular reform at our medical school in 1992 with a major educational grant from a prominent private foundation (11-12). Acceptance by a majority of students and faculty indicate that PBL will remain a successful and permanent part of the Kentucky curriculum, rather than what one author has called a “passing fad” (10).

We first introduced PBL into the 4th year medical curriculum during the 1994-1995 academic year, within the context of a new, required course entitled “Advanced Clinical Pharmacology and Anesthesiology”. All 4th year medical students are enrolled in this 4-week clerkship, which generally takes place in early spring and emphasizes practical experiences in both therapeutics and anesthesia. A major goal of the course is to facilitate the development of students as independent thinkers who can capably research specific topics and present them to their peers in both small and large group settings. During the study period, a total of three courses in the fourth year of our medical curriculum (including ours) used the PBL format.

In our course, students were randomly divided into small groups and asked to work through a total of four paper cases (one per week), using the PBL format. Case were written to emphasize basic anesthesiology principles and applied pharmacology in patients receiving perioperative care. Other major experiences during the course included didactic presentations, student preparation of a drug formulary, and clinical experiences in chronic pain, acute pain, intensive care units, and operating rooms (13). Students were also asked to give a brief oral presentation to the entire class on an assigned learning issue from their PBL group during specially scheduled “Guided Student Teaching” (GST) sessions (14). PBL participation and the GST presentation together comprised approximately 30% of the students’ course grade.

We were specifically interested in the evaluation of the PBL portion of this course, for a number of reasons. It was hypothesized that 4th year medical students might have preconceived notions about this learning format, given their previous exposure to it within the Kentucky curriculum and their status as senior students just a few months away from beginning their residency training. A related question was whether students would make a distinction between rating the PBL experience itself and the faculty preceptors who guided them in those experiences. We were also interested in how our PBL preceptors viewed their own performance, as compared to student perceptions of preceptor performance. Finally, we were also interested in contributing to the theoretical knowledge base for medical education by examining three issues (cultural influences, types of learning experiences, and assessment methods) pertaining to the “educational context” of our course.

In order to gain insight into these issues and further our understanding of how PBL can be used in the clinical education of fourth year medical students, we gathered data from students and faculty participating in the course by use of both quantitative ratings of designated items and qualitative comments. We collected data in a uniform manner over the three academic years, using the same ratings items with all three iterations of the course (see Table 1).

Table 1.

Ratings Items for Course Evaluation &PBL Preceptor Evaluation 4th Year Clinical Pharmacology & Anesthesiology Course (3 Iterations) University of Kentucky College of Medicine

RATING SCALE:
4	Outstanding
3	More than Adequate
2	Adequate
1	Less than Adequate
Course Evaluation Items
1	The overall quality of this course was:
2	My understanding of what was expected of me during this course (e.g. as specified in the syllabus) was:
3	Overall, the degree to which the evaluation processes/methods matched course content was:
4	The feedback I received on my performance during the course was:
5a	Active learning experiences (e.g., labs, computer-assisted instruction) were*:
5b	Active learning experiences (e.g., labs, ANY size group, computer-assisted instruction) were++:
6	My PBL exeprience during this course was*:
7	Overall, the learning materials used in this course (e.g., text, handouts, slides, overheads) were:
8	The integration of basic science and clinical concepts was:
PBL Preceptor Evaluation Items
1	Overall impression of this preceptor’s performance
2	Established ground rules or expectations for the group
3	Kept the group focused/on track
4	Helped the group set learning issues without arbitrarily defining them
5	Asked open-ended questions
6	Challenged students on their statements (e.g., asked for supporting evidence) in group
7	Interacted appropriately with students in group
Response Rates	Forms/Pct Returned
Course Evaluation	257/291 = 88%
PBL Preceptor Evaluation - Students	421/1152 = 37%
PBL Preceptor Evaluation - Faculty self=ratings	38/144 = 26%

^*Item was used during first two academic years of the study

⁺⁺Item was used during third academic year of the study

During all three years studied, we administered course evaluation forms on the final day of the course. These forms were provided by the College and used standardized items, thus enabling us to make comparisons of our course ratings across the curriculum.

PBL ratings forms wherein students were asked to rate their preceptors’ performance are also part of a standardized system; however, not all courses using PBL have implemented this ratings system. Students were asked to rate their PBL preceptors at the conclusion of each PBL case. Identical ratings forms were used for each case.

We also asked faculty to rate their own performance as PBL preceptors; identical ratings forms were used for this purpose. We were interested in how student ratings of PBL preceptors would compare with preceptor self-ratings. We were also interested in how preceptors viewed their performance during the course and whether they were confident in their own abilities as preceptors. The authors are not aware of similar comparisons between student and self-ratings of PBL preceptors during a 4th year clinical course.

All ratings forms (course forms completed by students and PBL preceptor performance forms completed by students and faculty) used an identical four point rating scale where 4=outstanding, 3=more than adequate, 2=adequate, 1=less than adequate. Response rates during our study varied, ranging from a high of eighty-eight percent (88%) for course ratings to a low of twenty-six percent (26%) for preceptor self-ratings (see Table 1). Qualitative comments from students and faculty were received on the same forms as quantitative ratings.

One change was made to the course evaluation ratings form during the study. During the most recent academic year, the PBL item (item 6, Table 1) was dropped as a separate item in the College-wide evaluation system. In its place, students were asked to rate “active learning experiences” (item 5b, Table 1) during all courses, including ours. It was explained, both on the ratings form and in person at the time when forms were administered, that this item referred to all small group learning experiences including PBL.

Results

The overall response of students to the course as a whole was positive. Course evaluations by students on a standardized, College-wide rating system were consistent with other clinical course offerings (mean course ratings of 2.50 for our course, compared to mean rating of 2.54 for all other clinical courses).

Mean ratings for the PBL ratings item (item 6, Table 1) in our course were 2.33 and 2.22 for the first two academic years respectively. These ratings placed PBL in our course below the mean compared to ratings of this item for other clinical courses within the College during the same time period (mean of 2.66 for all other clinical courses). During the third academic year of our study, students in our course were asked to rate a modified item (item 5b, Table 1) which reflected any type of small group learning activity, including PBL. The mean rating for this modified “active learning” item was 2.11 for the most recent academic year, and was again below the mean for all other clinical courses (mean of 2.50).

Via this course evaluation process, we also asked for and received qualitative comments from students regarding the PBL process. These comments were viewed by the course directors as explanatory in nature, i.e., the students’ written comments provide insight into why they gave a high or low rating to a particular item (15). A consistent theme of these comments was that the students did not wish to use a PBL learning format during their 4th year of medical school, due to the fact that they had been involved in PBL extensively in prior curricular years. Many students felt that learning was better accomplished through clinical, patient care experience than by classroom-oriented PBL; they preferred taking care of “real patients” to spending time in PBL sessions.

In addition to rating the course itself, students were also asked to rate their PBL preceptors using a separate ratings form. The PBL ratings form contains seven items pertaining to PBL preceptor performance (see Table 1). This PBL ratings system has been used in a number of College courses, both basic science and clinical, that feature PBL as a primary learning method. However, these items are not part of an overall College-wide system and, thus, comparisons between ratings of the PBL preceptors in our course and ratings of preceptors in other courses across the curriculum are not possible. Therefore, we compared mean item ratings on variables to be described below only for our course, using SPSS/PC (a standard statistical analysis software package). Results are reported as mean (M) and standard error of mean (SEM). We tested for mean item differences using an independent samples t-test procedure with a .05 level of significance.

A specific interest was whether students would rate preceptors differently based on type of preceptor. During the course, all PBL preceptors were physicians from the Department of Anesthesiology; some preceptors were attending faculty and others were senior level resident physicians. Mean ratings of resident preceptors were higher on all seven ratings items, and these differences were significantly higher on 5 of the 7 items (see table II). There were more ratings of attending faculty preceptors than for senior resident preceptors (total # of ratings of faculty preceptors=350; total # of ratings of resident preceptors=71).

Table 2.

Student Ratings of Attending Faculty and Senior Resident PBL Preceptors 4th Year Clinical Pharmacology & Anesthesiology Course (3 Iterations) University of Kentucky College of Medicine

Item	Faculty Mean+	Faculty SEM*	Resident Mean++	Resident SEM*	P-Value**
Overall impression of preceptor performance	3.43	0.04	3.65	0.010
Established ground rules/expectations for group	3.32	0.04	3.49	0.09	0.083
Kept group focused/on track	3.35	0.04	3.59	0.08	0.007
Helped group set learning issues	3.30	0.04	3.58	0.08	0.003
Asked open-ended questions	3.37	0.04	3.66	0.07	0.001
Challenged students on their statements in group	3.44	0.04	3.58	0.08	0.129
Interacted appropriately with students in group	3.50	0.04	3.74	0.07	0.003
Average all items	3.39	0.04	3.61	0.08	0.034

⁺Total number of ratings for faculty preceptors: 350

⁺⁺Total number of ratings for resident preceptors: 71

^*Standard Error of Mean

^**Based on independent T-test with .05 level of significance, unequal variance assumed (emboldened are significantly different)

We compared all student ratings of preceptor performance with all preceptor self-ratings on the same seven items (see table III). On five of the seven items, students rated preceptor performance higher than preceptors rated their own performance, with the lone exception being item four (“ability to keep the group ‘on track’/focused”). Mean ratings of students were significantly higher on three items (see table III). In regard to each of these three aspects of preceptor performance, the students as a group apparently felt more positive about PBL preceptor performance than did the preceptors themselves.

Table 3.

Student Ratings of Attending Faculty PBL Preceptors versus Faculty Self-Ratings 4th Year Clinical Pharmacology & Anesthesiology Course (3 Iterations) University of Kentucky College of Medicine

Item	Faculty Mean+	Faculty SEM*	Resident Mean++	Resident SEM*	P-Value**
Overall impression of preceptor performance	3.16	0.13	3.47	0.04	0.030
Established ground rules/expectations for group	3.00	0.14	3.35	0.04	0.020
Kept group focused/on track	3.41	0.13	3.39	0.04	0.908
Helped group set learning issues	2.97	0.12	3.34	0.04	0.007
Asked open-ended questions	3.16	0.17	3.42	0.04	0.146
Challenged students on their statements in group	3.27	0.17	3.42	0.04	0.193
Interacted appropriately with students in group	Not rated by faculty; no comparison made
Average all items	3.16	0.14	3.43	0.04	0.217

⁺Total number of ratings for faculty preceptors: 38

⁺⁺Total number of ratings for resident preceptors: 424

^*Standard Error of Mean

^**Based on independent T-test with .05 level of significance, unequal variance assumed (emboldened are significantly different)

We also tested the internal consistency of the seven PBL preceptor ratings items by analyzing both sets of ratings (one set wherein students rated their preceptors, the other set wherein preceptors rated themselves). Cronbach’s alpha reliability results were .951 for faculty ratings and .941 for students’ ratings.

Discussion

The results of our study of problem-based learning in a 4th year clinical course for medical students show that student opinion of this learning format was mixed. Our students did not rate PBL very highly, as compared with student ratings of PBL in other College courses (both basic science and clinical). This fact, supplemented by the qualitative comments of students who completed our course, leads us to conclude that PBL was perceived less enthusiastically by our students.

It is interesting to speculate as to why our students did not rate their PBL experiences as highly. Advanced medical students may not be as enthusiastic about PBL, particularly in those instances where they are nearing the completion of the undergraduate (M.D.) degree program. Further research is needed to determine whether the effectiveness of PBL as an educational methodology for medical students is affected by level of student, curricular year, and/or previous participation in this learning format.

It may also be true that our use of PBL in a very intense manner (i.e., four cases in a four week clerkship) may have been perceived by the students as too demanding. PBL is very labor intensive and requires a great deal of work outside small group meetings on the part of the students. There are other courses in the College that run for longer periods of time and yet do not use as many PBL cases as we did (e.g., a typical curricular block might last approximately twelve weeks and use three cases). Perhaps the negative ratings of PBL in our course reflect student opinion that we are attempting to do too much in too short a time frame. Qualitative comments from students indicated that the course was too “crowded” with PBL and other requirements. We plan to test this hypothesis in the next iteration of the course, where we will use only two PBL cases instead of four.

Regarding the results of faculty ratings, we noted a more positive attitude toward the PBL preceptors than toward the PBL process itself. This result is consistent with previous work done at our institution (16). Students rated the performance of both attending physician and senior resident physician preceptors highly, but apparently felt that the senior residents performed better in the role of PBL preceptor than did attending faculty. In addition, students in our course generally rated preceptor performance more highly than preceptors rated their own performance.

Because PBL is a relatively new methodology for medical schools, there is little research-based information about what constitutes “ideal” preceptor performance. It was not possible to compare the performance of our PBL preceptors with a “gold standard” for PBL preceptors in other clinical courses.

The careful use of student ratings information by educational decision-makers has been widely discussed (17-19). We agree with the view that decision-makers should be cautious about the interpretation of student ratings of PBL, given the lack of reliable standards against which preceptor performance may be compared (20). Before student ratings can be used as a factor in decisions about individual faculty members’ teaching performance, further research must be done to establish reliable and valid standards for optimal PBL preceptor performance. In the interim, student ratings information should be viewed as one method among many to provide formative feedback to faculty regarding how their teaching behavior(s) in PBL groups did or did not contribute to student learning. Students should be encouraged to provide this feedback in a timely and serious manner, in order that further contributions can be made to our overall understanding of preceptor skills and the PBL process itself.

It is also interesting to speculate about why our students rated the performance of resident preceptors higher than the performance of attending faculty preceptors. Perhaps this group of 4th year students more quickly identified with the residents, given that they were on the verge of joining the ranks of graduate medical education trainees themselves. A somewhat similar result was found in one study at our College wherein first year medical students were asked to rate the performance of 4th year medical students and attending faculty who served as preceptors for small group learning activities (21).

Possible reasons for the lower self-ratings of preceptor performance by faculty (as compared to student ratings) include the following: preceptors could have been resistant to PBL as a viable educational method and thus rated all aspects of PBL (including their own performance in PBL sessions) lower; preceptors lacked both experience and confidence in their own abilities as preceptors; student ratings of PBL preceptors may have been inflated due to the fact that students (more than faculty) enjoyed the close interaction with faculty that is part of the PBL approach.

While the tendency of medical students to rate PBL as highly enjoyable has been well documented (2; 7), the results of our study do not lend support to this notion. Student opinion is not, of course, the final determinant of the value of a given educational method; nevertheless, it is important. Our students gave high ratings to PBL preceptors, and yet did not rate the overall PBL experience itself as highly as we had anticipated.

Self-assessment by both students and faculty in medical education has often been advocated (22; 5; 23). However, little has been done to research the self-assessment of PBL students and preceptors and how perceptions of their own performance may impact the PBL process. One study examining the self-ratings of student performance compared with PBL preceptors’ ratings of student performance found that ratings tended to increase over time, and that students’ self-evaluation skills are difficult to achieve given the “halo effect” of subjective ratings (Hay 1995). Further research is needed into the perceptions of PBL preceptor performance, based not only on student ratings but also on self-ratings by PBL preceptors.

Within the educational literature lies an important theoretical construct known as “educational context” which could be helpful in interpreting the results of our study. This construct is defined rather broadly in the literature, and encompasses some or all of the following conditions related to how a given educational experience occurs: cultural influences (24); students’ learning styles and/or approaches to studying (25-26); learning environment and type of learning experiences (e.g., conventional didactic versus problem-oriented, case based experiences) (27); assessment methods used (27-28); presence or absence of a strong social support system for students (29).

Given the paucity of theory-based approaches to education in the professions (30), the construct of educational context is a useful one and raises a number of questions about the use of PBL in medical education. We designed our study in a manner which relied on student and faculty ratings, assuming that such ratings are an appropriate means of examining selected contextual issues, namely the cultural influences, types of learning experiences, and assessment methods at work in our course. We offer this study as an attempt to foster further discussion and study of “educational context” as applied to medical education, acknowledging that we have barely “scratched the surface” of operationalizing this theoretical construct.

PBL has been used extensively at our medical school, and research on its educational effectiveness is ongoing. As part of this research, we sought to examine PBL within the educational context of a required 4th year clinical course. Results indicate that the success or failure of PBL may be highly impacted by a multitude of factors, including certain issues related to the educational context in which PBL is used. Our study demonstrates the importance of the following contextual factors:

1. Cultural influence - as judged by student ratings results, PBL at our College is perceived more positively by students who are earlier in their medical school careers; 4th year students may view PBL in a less enthusiastic manner, particularly as they near completion of their studies and prepare for residency training. However, our results appear to support the use of qualified PBL preceptors who are resident physicians, at least insofar as their effectiveness is perceived by the students themselves.

2. Type(s) of educational experiences - such things as type of PBL preceptor (i.e., attending or resident physician faculty), preceptor enthusiasm and preparedness, and self-perception of preceptors as to their own performance may have an impact on how PBL is perceived by medical students and its overall effectiveness in medical education. Preceptors in this study rated their own performance differently than did students; the importance of this difference in terms of its actual or potential impact on PBL as an educational method is unknown. Preceptor performance was generally viewed as positive by both faculty and students, yet there were significant differences occurring on certain items which represented preceptor teaching behaviors. Our study lends limited support to the independent evaluation of preceptor performance in distinction from the overall PBL experience; preceptor teaching behaviors were rated more positively than was the actual PBL experience itself. We believe it is important to distinguish between these two aspects of evaluating PBL.

3. Assessment methods - we assigned nearly one-third of the total course grade to students’ PBL experience based on the use of ratings forms completed by PBL preceptors. This approach to grading in PBL, wherein the preceptor assigns numerical ratings to selected aspects of student performance within the PBL group, has enjoyed widespread use. However, we consider this approach to be somewhat subjective and difficult to use. Given the lack of experience and confidence in the use of ratings of student performance as a viable method of grading in PBL, further study of this issue is needed (31-32). It is apparent that medical educators are grappling with a number of pragmatic aspects of PBL; assessment of student learning is a prime example. Conventional methods of assessment are of limited value (32), at least insofar as judging student performance within the PBL group. In our judgment, further development of appropriate assessment methods are needed if PBL is to sustain credibility.

The interpretation of our data is subject to important methodological limitations. First, our results are limited by the small sample sizes and low response rates for PBL preceptor ratings (see Table 1). Particularly with the ratings of PBL, the opinion of the non-respondents could be different from those who completed ratings forms, thus affecting the resulting conclusions. The complexity of the educational issues associated with PBL require a larger sample and better data. We plan to continue this research in future iterations of the course in hopes of building a larger data set and increasing our confidence in the results of this research.

Also, the fact that during the study period (three academic years) we did not use the same set of faculty preceptors meant that it was not possible to compare changes in ratings between course years as the experience levels of the preceptors changed. Although the same ratings items were used across all three years, the fluctuation in ratings sample size and composition of faculty preceptors present problems in data interpretation.

Finally, we were unable to compare ratings of PBL in our course with ratings of PBL in the other two fourth year courses in the curriculum; the other courses did not require students to rate PBL using the standardized ratings system we used. It is possible that these students did not rate PBL highly in any of the fourth year courses which used PBL during the study period. It is difficult to make judgments regarding the curricular level of the course as an explanation for why fourth year students did not rate PBL as highly, given the lack of data on student opinion of PBL in other fourth year courses.

Conclusions

Problem-based learning is an important learning method currently in use in medical schools across North America. Among the many factors associated with the perceived success or failure of PBL, the opinions of students and faculty who are experiencing it may be among the most important. Medical educators continue to advocate PBL as an innovative and effective approach, yet much remains unsettled about how to evaluate it, what constitutes a successful PBL curriculum, and whether PBL compares favorably with traditional clinical or didactic approaches to educating future physicians.

We view the major findings of this study to be two-fold: one, that PBL was less effective with a selected class of fourth year medical students at our institution; two, that student perception of PBL may be impacted by factors associated with the theoretical construct of “educational context” (e.g., cultural influence, type of educational experience, assessment methods used). The successful use of PBL may be affected by the curricular level and other contextual factors. Further research is needed to understand these factors and how they impact PBL.

The authors remain committed to problem-based learning as a viable educational method. However, the results of this study of our initial implementation of PBL have led us to re-examine how PBL will be used in our course during future rotations. As educators, we must be willing to consider adjustments to our use of PBL, while at the same time strengthening our evaluation efforts (33). Our experiences to date have taught us that the judicious use of PBL in medical education requires a careful examination of the educational context in which it is to be employed.