Student feedback on problem-based learning processes

Abstract

Background

Problem-based learning (PBL) sessions consist of a brainstorming phase, search phase, individual study, and reporting phase. At the Xavier University School of Medicine, Aruba, PBL is a new learning modality first introduced in May 2013.

Aims

PBL processes have not been studied previously at the Xavier University School of Medicine, hence the present study was carried out to obtain information about the PBL processes and note differences, if any, among different groups of students.

Method

The study was conducted among first- to fourth-semester undergraduate medical students during July 2014 using a previously validated PBL processes instrument developed by van den Hurk et al. Information about gender, semester, weekly hours of study, and learning resources used was obtained. Respondents’ agreement with a set of 23 statements was noted using a Likert-type scale, which was scored as 1=totally disagree with the statement, 2=disagree, 3=neutral, 4=agree, and 5=totally agree with the statement. Mean scores were compared among different groups of respondents.

Results

Fifty-one of the 58 students (87.9 per cent) participated. The weekly average study time was 29.9 hours. Lecture handouts and textbooks were commonly used information sources. The mean scores (scale 1 to 5) for learning-issue-driven searching, and extensiveness of searching were 3.49 and 3.45, respectively. The score for explanation-oriented preparation was 3.94, while those for breadth and depth of discussion were 3.75 and 3.62, respectively. Most scores were higher among second-semester students, but the difference was not significant.

Conclusion

The self-reported scores were comparable to those reported in previous studies done using the same PBL processes instrument in other medical schools. At Xavier University School of Medicine interactive lectures are the major teaching method and topics covered during PBL are also likely to be covered during lectures, which could influence the scores. The findings of our study providing information about how students function during PBL brainstorming and presentation sessions and how they use different learning resources would be of interest to other medical schools worldwide following a hybrid curriculum. Further studies are required.

What this study adds:

1. What is known about this subject?

   Problem-based learning (PBL) is a widely used learning modality in medical schools. Studies on group processes and the influence of tutors on PBL discussion have been carried out.

2. What new information is offered in this study?

   The present study offers information on PBL processes and use of information resources in an offshore Caribbean medical school delivering a hybrid undergraduate medical curriculum. Problem-based learning is not a common learning methodology in Caribbean medical schools and PBL processes have not been previously studied in the region.

3. What are the implications for research, policy, or practice?

   This study’s results can help medical schools fine tune problem-based learning to maximise student learning. Other schools following or considering a hybrid curriculum can apply these results to use PBL effectively. Furthermore, increased use of Internet resources during the search and self-study phase of PBL will have important implications for medical education.

Background

Problem-based learning (PBL) is becoming increasingly common in medical schools worldwide.^1,2 PBL promotes problem solving and communication skills, and students find it a useful and effective learning methodology.³ The three objectives of PBL have been described as acquisition of essential knowledge, use of knowledge in clinical contexts, and self-directed learning.⁴ In a PBL session students are actively involved and are not just passive recipients of information. A PBL session is typically conducted in phases. During phase 1, students are stimulated to discuss a clinical problem in their tutorial group.⁵ Students try to explain what is occurring in the problem, but during the discussion process certain questions may remain unanswered, which serve as learning issues in phase 2. During phase 3, students actively search for literature relevant to the study objectives. In phase 4 students prepare the literature to explain their findings to other students in their tutorial group. During phase 5, the group meets again during the reporting phase and students check whether the results of their individual study help them understand and explain the problem.⁶

During the search phase students are expected to orient themselves with as many learning resources as possible, and the process of searching acquaints them with the resources required for the next phase of learning. In a problem-based curriculum students must develop the skills to manage differences and contradictions in the literature. Having team members consult a variety of resources results in diversity and richness of discussion in the tutorial group. Studies have shown that students in a PBL curricula made greater use of the library and self-selected learning resources and felt more confident in their independent information-seeking skills.^7,8

Xavier University School of Medicine (XUSOM) started problem-based learning sessions for undergraduate medical students beginning in May 2013.⁹ Beginning in January 2014 the school shifted to a fully integrated curriculum. Interactive lectures continue to be the predominant learning modality at XUSOM; however, PBL sessions are conducted once a week. Student perception about the effectiveness of small groups during PBL sessions was recently studied in the School of Medicine using the previously validated tutorial group effectiveness instrument.¹⁰ Student perception about the effectiveness of the small groups was positive. Student groups have a two-hour brainstorming session during a particular week and at the same time a week later they have the discussion and presentation session. The sessions are facilitated by a faculty member who has been trained in PBL facilitation skills through workshops conducted by experts from within XUSOM and outside. During most sessions the group has the same facilitator, and the authors agree that the facilitator may influence the perception of the group regarding the PBL process.

During the brainstorming session students in groups identify learning issues. Previous studies have shown that student-generated learning issues play an important role in guiding the students’ independent study.^11,12 During the discussion and presentation session students try to explain aspects of the problem to others using different literature. Research has shown that students learn best when they explain to others.¹³ Students who learn using an explanation-oriented approach learn better and can better integrate the new information into their existing knowledge base.¹⁴

Since problem-based learning is a new learning methodology at XUSOM, PBL processes have not been previously studied. This study was carried out to obtain information about the PBL processes at the School of Medicine and to note differences, if any, among different groups of students.

Method

The study was conducted during July 2014 among first- to fourth-semester medical students at XUSOM Aruba. The school admits students three times a year in January, May, and September. During each semester students learn the different basic science subjects in an integrated organ system-based manner with early clinical exposure. The fourth semester has greater academic exposure compared to others as they have completed most organ systems.

The objectives of the study and the process involved were explained to the students and they were invited to participate. It was stressed that participation was voluntary; students were free not to participate, free to withdraw from the study at any time, and free not answer certain questions if they felt uncomfortable providing the information.

Written informed consent was obtained from all respondents. The study was approved by the Institutional Review Board vide notification no. XUSOM/IRB/2014/04.

The authors used a 23-item instrument developed by van den Hurk et al. and previously tested among students at Maastricht University in the Netherlands with their permission.⁵ Information about the participants’ gender and semester of study was noted. Like the authors of a previous study conducted in Turkey,¹⁵ we also collected information about the students’ weekly average study time and about their use of various resources.

Five areas with regard to self-directed learning, preparing for the PBL group session, and the reporting and presentation phase of PBL were considered. A total of 23 statements were grouped into five main categories: learning-issue-driven searching; extensiveness of searching, which represented the search phase; explanation-oriented preparation, which represented the preparation phase; and breadth of discussion and depth of discussion, which corresponds to the reporting and presentation phase. The various statements grouped together into categories are shown in Table 1. Respondents were asked to indicate their degree of agreement with each of these statements using a Likert-type scale according to: 1=totally disagree, 2=disagree, 3=neutral, 4=agree, and 5=totally agree. There was an option for respondents to provide free text comments about the PBL process if they were interested.

Table 1. Statements used in the study instrument grouped into categories⁵.

Please indicate your level of agreement with the following statements about self-study process, by rating them between 1 and 5. Use whole numbers only.
(1 = totally disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = totally agree)
Search Phase
Learning-Issue-Driven Searching
1. When I start studying, I use the learning issues as a starting point to determine what literature I will search.
2. During studying I always check the learning issues to determine whether I study deep enough.
3. I’m particularly guided by the learning issues.
4. I use learning issues to see if the literature I have studied covers the content.
5. I use learning issues as a guideline to study the literature step-by-step.
Extensiveness of Searching
6. When searching the literature, I try to judge different textbooks on their relevance for the subject to be studied.
7. When searching the literature, I compare different literature about the same subject.
8. I spent a lot of time and effort on searching the literature before I start studying.
9. When searching the literature, I compare different literature about the same subject.
Preparing Phase
Explanation Oriented
I prepare myself
10. Such that I can explain the literature without the textbooks.
I prepare myself
11. Such that I can clarify my point of view about theories.
12. Such that I can explain concepts in the literature in my own words.
13. Such that I know what needs to be discussed in each learning issue.
14. By making summaries of the literature.
15. By making notes.
Reporting and Presentation Phase
Breadth of Discussion
16. Many different findings are discussed.
17. When someone finds something that is not directly related to the learning issues, it is explained to others.
18. The members of the group frequently question different aspects of the literature.
19. Contrasting literature is explained.
Depth of Discussion
20. During the discussion during the presentation, the new facts are explained and elaborated.
21. The discussion in the presentation contains much depth.
22. In using the newly learned knowledge, we question and clarify the phenomena that underlie the problem.
23. The discussion and the presentation are very useful in addition to what I study.

The mean ± SD score for each of these five sections was calculated. The mean scores were compared among various groups of respondents. The free text comments were tabulated. Statistical Package for Social Sciences (SPSS) version 20 for Windows was used for statistical analysis. A p value less than 0.05 was taken as statistically significant. The overall Cronbach’s alpha value for each of the five subsections was calculated.

Results

Fifty-one of a total of 58 students participated in the study providing a response rate of 87.90 per cent. Table 2 shows the respondents’ gender and semester of study. Thirteen respondents did not fill in information about their gender, while three respondents did not mention their semester of study. Due to the problem of non-response we did not analyse the scores according to the respondents’ gender.

Table 2: Demographic characteristics of the respondents.

Characteristic	Number (%)
Gender
Male	22 (43.1)
Female	16 (31.4)
Semester of study
First	8 (15.7)
Second	11 (21.6)
Third	18 (35.3)
Fourth	11 (21.6)

* The numbers may not add up to 51 as all respondents did not fill in all required characteristics

The mean weekly study time of all respondents was 29.94 hours. Forty-three respondents (84.3 per cent) used lecture handouts during their self-study and preparation for PBL discussion, while 41 respondents (80.4 per cent) used textbooks as a source of information. Eighteen respondents (35.3 per cent) used the Internet, while eight respondents (15.7 per cent) used medical journals. Four respondents used United States Medical Licensing Examination (USMLE) preparation materials.

The mean score for learning-issue-driven searching was 3.49 (maximum score being 5). The mean ± SD scores for extensiveness of searching was 3.45 ± 0.90. The score for explanation-oriented preparation was 3.94 ± 0.57, while those for breadth of discussion and depth of discussion were 3.75 ± 2.12 and 3.62 ± 0.65, respectively. There were no significant differences in scores among various subgroups of respondents. Table 3 shows the mean score among students of different semesters. Though the differences were not statistically significant, scores for many parameters were higher among the second-semester students. The number of male respondents was 22, while the number of female respondents was 16; eight students were from the first semester, while 11 were from the second, 18 from the third, and 11 were from the fourth semester (Table 2). Certain respondents did not provide information about their demographic characteristics.

Table 3 : Mean scores among different semesters of respondents.

Characteristic	Mean score	P value
Semester of study
Learning-issue-driven searching
First	3.37	0.823
Second	3.62
Third	3.29
Fourth	3.51
Extensiveness searching
First	3.19	0.055
Second	4.10
Third	3.26
Fourth	3.36
Explanation oriented
First	3.87	0.143
Second	4.22
Third	3.68
Fourth	4.07
Breadth of discussion
First	3.31	0.657
Second	3.67
Third	4.29
Fourth	3.32
Depth of discussion
First	3.28	0.224
Second	3.77
Third	3.67
Fourth	3.50

The Cronbach’s alpha value for learning-issue-driven searching was 0.89, while for extensiveness of searching it was 0.79. For explanation-oriented preparation the alpha value was 0.78, while the values for breadth of discussion and depth of discussion were 0.71 and 0.72, respectively.

Free text comments included: change the PBL small groups each semester; a request for peer evaluations during PBL; and not enough study time for the presentations. Only one respondent made each free text comment, and there was a different respondent for each statement. One respondent stated, “I enjoy PBL because it allows me to show what I have understood and to extrapolate my knowledge to clinical situations.” Another responded, “I like PBL as it provides a much-needed clinical view and makes us think like doctors.” Two respondents mentioned that PBL seems unhelpful, while one mentioned that it is not beneficial now but may become so in the future. A respondent mentioned that certain students need to be more serious about PBL, while another respondent mentioned that assessment of each student by his/her peers should be carried out.

Discussion

Student feedback shows that problem-based learning is a valid teaching-learning methodology to enhance student understanding of difficult concepts in the context of a Caribbean medical school environment. The weekly average study time was around 30 hours. Respondents predominantly used lecture handouts and textbooks to prepare for PBL discussions. The mean scores for various categories were good. There were no significant differences among respondents though students in the second semester had higher values for many parameters. The internal consistency of the instrument was good as the Cronbach’s alpha values were between 0.7 to 0.9, suggesting good internal consistency of the instrument used.¹⁶

The weekly self-reported study time in the present study was higher than that reported in Turkey.¹⁵ One possible reason for this is the more condensed and compressed nature of the basic sciences curriculum in Caribbean medical schools. A semester at XUSOM is 15 weeks long,⁹ hence students have to put in more hours of study. Another reason could be that the predominant teaching-learning methodology continues to be lectures and there is only one PBL session each week. Hence students have to revise and review what they learned during the day’s lectures and also prepare for the next day’s sessions.

Our scores were broadly comparable to that reported in the study conducted at Maastricht University⁵ and at the Turkish medical school.¹⁵ Our score for extensiveness of searching was comparable to that reported in Turkey, but higher than that reported in the Netherlands. At the same time, information collected about the sources consulted revealed students mainly used lecture notes and textbooks. Our scores for other categories were also higher than reported in previous studies.^5,15

Lecture handouts were also widely used by students in a Turkish medical school. In Turkey the lecturers only provided outline handouts, which included topic titles, schemata, algorithms, and tables. At XUSOM, lecture slides and handouts are deposited in an online server titled “Class notes,” which students can access. Due to the compressed nature of the course, students tend to rely more heavily on lecture slides and handouts.

In the Turkish study conducted, third-year students used textbooks more frequently and had higher scores for learning-issue-driven searching and extensiveness of searching.¹⁵ In our study, there was no significant difference between different semesters. A possible reason is that the present study was conducted only among basic science students, and since students are admitted three times a year, the chronological difference in age and academic seniority between them may not be significant. In Caribbean offshore medical schools students do their clinical rotations in the United States, and clinical students were not included in this study.

As we follow a hybrid model where PBL is used to supplement interactive lectures, learning issues derived from the PBL group discussion and brainstorming sessions may have been influenced by the knowledge gained during the lecture sessions. We are trying to implement a system in which diseases and conditions addressed during the PBL will only be briefly addressed during lectures.

Previous studies have shown that students in the tutorial group need to have a well-structured discussion and presentation to make sense of the findings and collate information from various literature sources. The individual (i.e., a student) who leads the group and structures the discussion has a difficult role. The group leader role at XUSOM is usually rotated among different students and a faculty member facilitates. A recent study had shown that tutor- (facilitator-) related behaviours influence student learning during PBL.¹⁷ The authors concluded that academically stronger students are not as reliant on the tutor, while weaker students depend on the tutor to guide and motivate them, and help them achieve the learning goals. At XUSOM, most students are average, which may put greater responsibility on the facilitator/faculty member to motivate and guide learning.

A recent study conducted in two colleges of medicine in Saudi Arabia reported that students perceived that PBL had a positive impact in terms of their cognitive, personal, and teamwork skills.¹⁸ The author concluded that the tutor should have both content and process expertise to obtain the best outcomes from the PBL process. A recent review suggests that PBL works best when both students and faculty understand the various factors which influence learning and are aware of their roles.¹⁹ As student perceptions regarding different statements mentioned in the questionnaire may have been influenced by the dynamics of the tutorial, group factors influencing group dynamics have been mentioned in this section.

In the present study the score for learning-issue-driven searching and extensiveness of searching was highest among second-semester students. The number of students in each semester in the school is low, usually less than 20. Certain groups have reported problems with group dynamics, which could have affected the scores. The scores among the first-semester students were low for many groups of statements, but no definite trend was seen. The study conducted in the Netherlands was done only among first-year students,⁵ while in the Turkish study¹⁵ first-year students’ scores were low possibly because new students were familiarising themselves with the PBL process and with learning the various basic science subjects in an integrated manner.

Van den Hurk et al.⁵ note that the manner in which students search and prepare the literature has a significant influence on the quality of the reporting phase. The authors state that in addition to students learning how to explain things during the reporting phase, they must be taught how to develop the skills of searching literature sources effectively, understanding the literature from the viewpoint of explaining it to others, and producing summaries of the material they have read during their individual study phase. In a paper published in 2006, Visschers-Pleijers et al. suggest that students’ perceived effectiveness of the PBL reporting phase is dependent on the amount and nature of the explanations provided by the students, application and integration of knowledge, discussion of different opinions based on content, and on the guidance of the group discussion.²⁰ The discussion and reporting process had a significant influence on the generation and retention of knowledge.

As mentioned previously, problem-based learning is only one of the learning modalities at XUSOM that can influence various aspects of the PBL process: knowledge gained during lectures can influence PBL. Students also have sessions on critical appraisal of scientific literature and learn to use independent objective sources of information during sessions on rational use of medicines, which could have also influenced their searching skills.

At XUSOM Aruba, students are assessed by the facilitator during sessions using a modified version of the instrument develop by Elizondo-Montemayor.²¹ Each individual student is assessed based on a range of criteria. We have not yet started peer assessment, but are in the process of discussion regarding starting the same from the Spring 2015 semester. We stress the importance of PBL to the students during the orientation program and also introduce them to self-directed learning and various learning resources available in the school and online. In a study conducted in Pakistan, about 40 per cent of students agreed that awarding an individual mark to a student would be a good indication of the student’s performance.²² About 31 per cent of respondents were of the opinion that these marks should not be included in the semester exam marks, while about one-half the respondents were not in favour of peer evaluation. At XUSOM facilitator assessment of student performance in PBL accounts for a percentage of marks in the end-of-system student assessment.

Online resources are increasingly being used by students for self-study and for preparing for the reporting phase of PBL.²³ At XUSOM Aruba students use online resources to search for information and to support their findings (as references) during the presentation phase. At the David Geffen School of Medicine at UCLA in the United States an educational intervention to improve literature sourcing by first-year medical students was implemented.²⁴ The intervention improved the quality of resources cited by students during their first curricular block. At Michigan State University in the US, facilitators’ influence on the use of online information resources by students during PBL sessions was studied.²⁵ The facilitator’s verbal behaviour encouraging or discouraging the use of online resources had a significant impact on the use of information technology by students.

The sample size in this study was low, but the response rate was excellent. The instrument developed by van den Hurk et al. to study PBL processes mainly assessed student perceptions regarding different PBL processes. Assessment by observers of the reporting phase of the PBL to obtain objective evidence was not carried out. As many respondents did not provide information about their gender, the scores were not analysed among males and females. The number of students in different semesters was low. As the student body is small, certain students may have had apprehensions about being identified, which could have affected the responses. The PBL processes instrument was previously used among medical students in the Netherlands and Turkey, but has not been used previously in an offshore Caribbean medical school.

Conclusion

The self-reported scores of various processes involved during the search, preparation, and reporting/presentation phase of problem-based learning were good compared to previous studies. The number of students in the school is low, however, and PBL is a new educational methodology. A hybrid curriculum is followed with lectures being the dominant teaching methodology. The findings of our study would be of interest to other medical schools following a hybrid curriculum. Further studies with a larger number of students and as PBL becomes an established learning method at XUSOM are required.