Problem based learning: tutors' views 5 years after implementation at a sub-Saharan University

Abstract

Background

Curriculum evaluation is key to continuous assurance of quality of education.

Objectives

To assess the teachers' perceptions on how well student teaching and learning activities were conducted at an institution that had practiced PBL for five years.

Methods

A cross sectional descriptive study. 150 teachers from 23 departments at the College of Health Sciences participated in the study. A 25 item self-administered questionnaire was used to collect data. Activities evaluated included; students assessment, self-directed learning, feedback and clinical exposure. Data were entered in epi data and analysed using SPSS. Ethical approval was obtained.

Results

The participants' average teaching experience was eight years. The PBL/COBES approach mostly achieved the aim of producing self-directed and lifelong learners. Half of the teaching staff actively provided regular feedback about the learning and teaching activities they were involved in. Early clinical exposure was widely accepted as a highly rewarding appropriate teaching and learning strategy. COBES activities were well organized although involvement of College staff was low.

Conclusion

PBL/COBES program was successfully executed and had high acceptance among Faculty. The biggest challenge was getting all staff to provide regular feedback. Self institutional curriculum review can be done cheaply and quickly to provide feedback for continual curriculum improvement.

Introduction

This curriculum evaluation exercise surveyed the teachers' opinions and perceptions of how a new curriculum implementation at the five year point was understood and implemented.

Five years of implementing new innovative methods of teaching and learning at Health sciences at Makerere University (an over 90 year old institution) was a milestone in the institution's educational history. A milestone because change for the better was embraced. The premise for the change was anchored in an extensive stakeholders' feedback exercise that had been con- ducted seven years earlier1. The extensive stakeholders' survey had been prompted by not only an awareness of global trends but also a need to respond to calls for change from alumni, employees and others stakeholders. These stakeholders in one way or another were involved with the graduates of this College. Stakeholders' demands can be achieved by continuous selfreflection and reforms.

Community Based Education and Service (COBES) is a system where students on an annual rotation basis are placed in a preselected rural site under the supervision of a preceptor for a period of 4–6 weeks with a predetermined set of educational and health service delivery targets. This process is supported by teachers at the College. Support is mostly in form of traveling to supervise student and preceptors on site. These reforms are part of the edu cation evolution process at Makerere University.

At Makerere University, curriculum reforms occurred turning a predominately traditional teacher centred lecture based approach to a principally student centred, self-directed and small group learning described as a combination of Problem Based Learning and Community Based Education and Services (PBL/COBES) methodologies. The COBES component involved students spending part of the in-scheduled teaching time at a pre-selected prepared rural community site where the (students) engaged in Health Service Delivery activities². This exercise accorded them the opportunity to practice and learn in the context of a rural community life where nearly 90% of the country's population lives³.

These curriculum reforms were not fully accepted by some stakeholders (including students, alumni, and the teachers). Some students and teachers preferred the lecture methods and some alumni thought reforms meant less teacher supervision.

The purpose of this survey therefore was to assess the strengths and weaknesses of the curriculum design and its implementation. This would inform strategies for improving the teaching and learning environment and the overall quality of education for undergraduate education at Makerere College of Health Sciences.

Study area

Makerere University College is a 90 year old institution. It runs both undergraduate and graduate and 250 faculty (full, part- time to adjunct)⁴. The current educational reforms that started in 2003 have included over 250 sensitization/training workshop days and eight members completing courses in Masters of Health Profession Education. A complete transformation from teacher centred, lecture based to PBL/COBES curriculum was achieved after three years of preparation and implementation.

A quality assurance unit incorporated during the implementation period, formulated a manual stipulating minimum standards of education. This was informed by the World Federation of Medical Education standards (WFME) frame work⁵.

Methods

Design

A cross sectional descriptive study.

Study population

All teaching staff from Makerere University College of Health Sciences (without the School of Public Health) and the National Referral hospital, Mulago and in 23 departments. The teachers involved in this survey had a mean of eight years of teaching experience. The majority were specialists in the Clinical and Basic Sciences areas.

Student learning

The eight questions in this domain were about PBL as an appropriate strategy for achieving skills in problem identification, and problem- solving through team work and appropriate use of resources. The other questions were about tutor involvement in providing administrative feed- back (to student learning). In summary there were five questions about student effects, three about faculty involvement.

Data tool

A pre-tested 25 item self-administered hard copy hand delivered questionnaire was used. It had both the 5-point Likert and binary scales. Items in four domains were Stu dent Learning (7), Student Assessment (6), COBES (5) and Governance (6 items).

The choice of the four domains was generated by consensus in a pre-planning meeting attended by the investigators and key representatives of the College educational committee drawing on WFME and other literature^{1,5,6,7,8,9,10}. These domains were organisational culture, learning contexts and organizational values. Pretesting was done using 10 questionnaires given to 10 individuals. The appropriate adjustments were made to improve clarity of the items and questions

Data collection

The questionnaires were sent to the 23 departments after prior communication inviting members with at least 5 years teaching experience to participate voluntarily. Trained research assistants followed up the questionnaires. 150 questionnaires out of the 250 were returned and 107 were completely filled.

Data analysis

Epi data program was used for the data entry. SPSS 12 version was used for analysis, standard descriptive statistics were performed for the different domains.

Ethical approval

Ethical approval was obtained from Makerere College of Health Sciences, School of Medicine Research and Ethics Committee.

Results

Twenty one departments were represented as shown in table I, though 16 respondents did not indicate the departments they represented.

Table 1.

Departmental representation of the returned questionnaires from respondents at College of Health Sciences, Makerere 2010

Department	Surveyed Tutors Number (%)
Medical	30 (28)
Surgical	29 (27)
Pre-Clinical	24 (22)
Radiology	6 (6)
Nursing	7 (7)
Dentistry	4 (4)
Pharmacy	7 (7)
Total	107 (100)

In table 2, the views of the staff regarding PBL are shown: the majority 75% agreed that PBL curriculum empowered students to identify health problems and use available resources to solve them.

Table 2.

Indicating the views of staff regarding PBL and student learning aspects at College of Health Sciences, Makerere 2010

Student learning
		Number (%)
		Agree	Disagree	Neutral	Total
1.	The PBL curriculum has empowered students to identify health problems and use available resources and information to address them.	75 (70)	11 (10)	21 (20)	107
2.	Problem Based Learning has helped our students to appreciate the complexity of diseases and health problems, diagnostic technologies as well as treatment regimens which require continued learning.	67 (63)	10 (9)	30 (28)	107
3.	Our new PBL approach to student learning has reinforced “team work” study in small groups focused on problems with teachers facilitating the learning process.	93 (87)	5 (5)	9 (8)	107
4.	Communication skills are more likely to be developed under the PBL curriculum through varied learning opportunities.	89 (83)	7 (7)	11 (10)	107
6.	I am involved in the planning teaching activities.	50 (47)	19 (18)	38 (36)	107
7.	When I encounter challenges in facilitating the learning process (teaching), I am listened to and helped.	64 (60)	17 (16)	26 (24)	107
8.	I regularly provide administrative feedback.	64 (60)	15 (14)	28 (26)	107

COBES performance was rated highly in all the five questions, 64–94% were in agreement of high scores (see table 3) that COBES gave the desirable rural community health exposure experience to students and, it was implemented effectively. It contributes to continuous professional development for the site preceptors. The challenges faced during COBES were often well articulated.

Table 3.

COBES performance rating by staff, College of Health Sciences, Makerere 2010

Community Based Education and Services (COBES)
	Yes	No
1. Are you involved in COBES?	47 (44)	60(56)		107
	Number of those involved (%)
	Agree	Disagree	Neutral	Total
2. COBES has given students an exposure to rural health care.	44 (94)	3 (6.4)	-	47
3. COBES program is implemented effectively.	33 (70)	8 (17)	6 (13)	47
4. COBES contributes to Continuous Professional Development for site tutors.	41 (87)	2 (4)	4 (9)	47
5. COBES challenges are well articulated	30 (64)	9 (19)	8 (17)	47

The four questions about COBES included the justification for it, running the COBES program and feedback about how it was run.

The teachers evaluated the assessment methods, integration of teaching and early clinical exposure (see table 4). The majority 61% agreed that assessment methods employed under PBL were aligned with the teaching methods. Early student contact with real patient stimulated learning for 73% of students. Integration was perceived to be attained by 76% of the participants. Examination guidelines were not clear to at least 16% of the respondents.

Table 4.

Student assessment ratings by; assessment methods, integration and early clinical exposure (the assessment was by staff), College of Health Sciences, Makerere 2010

Assessment
	Number (%)
	Agree	Disagree	Neutral	Total
1. Assessment methods employed under the PBL curriculum are in alignment with teaching methods. (What is assessed is what is learned	65 (61)	15 (14)	27 (25)	107
2. Early student contact with real patients stimulates their learning	78 (73)	11 (10)	18 (17)	107
3. Horizontal integration of biomedical sciences is important for student learning.	76 (71)	6 (6)	25 (23)	107
4. Vertical integration of clinical and biomedical sciences facilitates better learning.	81 (76)	5 (4.7)	21 (20)	107
5. The guidelines regarding students attendance at examinations are clearly stated.	69 (65)	17 (16)	21 (20)	107
6. Students receive regular and timely assessment feedback under the PBL curriculum.	56 (52)	19 (18)	32 (30)	107

Only half of the staff that participated in this survey viewed the introduction of the PBL/COBES curriculum as positively transformative of the College teaching and learning landscape (see Table 5).

Table 5.

Key Governance aspects ratings as assessed by staff

Governance
	Number (%)
	Agree	Disagree	Neutral	Total
1. The transformation of the to a College of Health Sciences	115	7 (5)	12 (9)	100
potentially provides for meaningful synergy between the	(86)
various streams of medicine, public health, nursing,
pharmacy and dentistry to better meet the educational
challenges of the 21st century
2. The introduction of the PBL/COBES curriculum facilitated	73 (51)	20 (14)	50 (35)	100
the transformation of the College and this new curriculum
had a positive effect at the University.
3. The establishment of a CHS is better placed to meet the	103	16 (11)	22 (16)	100
shortage of health care providers at all levels of the system	(73)
than the Faculty of Medicine, through the establishment of
four Schools (Public Health, Medicine, Health Sciences and
Bio-medical Sciences).
4. The resources available have been optimally utilized and	55 (38)	30 (21)	59 (41)	100
managed.
5. The CHS has the potential for greater administrative,	106	9 (6)	27 (19)	100
financial and management efficiency	(75)
6. I understand the PBL/COBES management structures.	79 (56)	26 (19)	35 (25)	100

More than a third (38%) viewed the use of available resources as optimally managed, however 75% viewed the College of Health Sciences as having the potential for greater administration, financial and management efficiency.

Discussion

In this review exercise we found that PBL/COBES program was successfully executed and had high acceptance by the teachers (tutors).

PBL success

Concerning student learning, seventy to eighty percent of the teachers judged that PBL had achieved the general intended aims of students attaining problem solving skills and working as a team . The teachers responded to a set of questions in order to judge whether PBL had achieved its intended purpose. These teachers had been involved since the inception of PBL and COBES, and had previously undergone over 250 workshop days of training in various aspects of teaching and learning. These included but not limited to exploiting knowledge of how people learn to promote effective learning and dealing with the challenges of running successful tutorials^11,12,13. We therefore contend that collectively the respondents were all well grounded in the basic knowledge of the principles that underpin the practice of PBL/COBES and therefore qualify to pass judgment on its execution.

Problem solving is a basic tenet of the philosophy of teaching and learning in this regard. Faculty should be engaged in identifying better ways of implementing the problem solving aspect in a PBL curriculum. It is therefore important that there exists a mechanism that ensures Faculty frequently and regularly give feedback to the curriculum implementation committee or its equivalent. For five years, feedback was gathered through regular weekly two hour long meetings at the College of Health Sciences.

Feedback and assessment

Overall, fifty percent of the teachers in this study regularly provided feedback. Feedback concerning implementation challenges including both academic and administrative. This proportion was slightly different from those that were involved in planning learning activities; it is therefore possible though not necessarily so that those involved in planning are also involved in giving regular feedback.

Whereas fifty percent of the staff provided feedback and this is to be judged as good involvement, it is debateable as to what the benchmark involvement in giving feedback should be. Involvement would be defined as in numbers of teachers giving feedback the scope and detail of feedback.

Regular and timely feedback to students after assessment is crucial for learning and was an area for definite improvement in this evaluation. It may involve training staff on how to provide feedback and for students what feedback to expect. Students' polls to assess what percentage of them feel they get feedback and their level of satisfaction could be part of improving feedback management¹⁴.

Feedback should be immediate, inexpensive, it should require little preparation (but requires skill in giving it) and should be highly desirable to the learner. Frequency coupled with a user friendly mechanism¹⁵ ensures a steady flow of feedback, but it is also important that the administrative team responds to the feedback given¹⁶ in an appropriate and timely manner.

Alignment of teaching methods and assessment refers to using assessment methods appropriate to the style of teaching. It is also assessing students in a way similar to how teaching and learning occurred. Alignment was judged as well done by nearly two thirds of the respondents. Assessment drives learning, students learn in the way they are would be assessed¹⁷. Alignment of these two is critical and they feed into each other in order to achieve learning. Student assessment should therefore consider not only learning the objectives and content of the subject in question but also the method of learning, for example if group work is employed in a significant way, group assignments should be a significant part of the assessment portfolio.

Examinations are a central part of curriculum implementation and therefore examiners and students should be absolutely clear about examination regulations. Exams are part of assessment, a lens through which an institution assesses itself through its students work. When assessment results do not match institutional or programmatic expectations, that is the time for the staff to collectively determine how to improve student performance. So it is critical that the rules by which these assessments run are clear and well understood¹⁸. In the evaluation a third of the teachers were not clear about exam regulations. All staff should be in the know of all the regulations and rules.

An assessment blue print including examination regulations and in house trainings program may be the way forward in resolving this situation.

Community Based Education and Service (COBES)

Early clinical exposure as an essential strategy to teaching and learning was agreed on by the majority of the respondents. There was a significant buy-in of the concept. Early clinical exposure aids integration of basic and clinical studies. This integration aids better retention of knowledge and acquisition of skills. Better retention translates knowledge and acquisition of skills for competence development.

The teachers involved in COBES activities were 39% of the teaching staff. Where, there is no set target of how many teachers should be actively involved. It is desirable that as many as can do participate, perhaps a rotation policy would get more staff involved but it would sacrifice the advantage of experience and continuity for this smaller group (44%) that has already demonstrated exceptional interest and expertise. However COBES activity supervision is also taken on by site tutors or pre-receptors and staff from the school of Public Health who were not included in the 44%.

COBES was effectively implemented as judged by 70% of the respondents. One of the purposes of COBES was to expose students to rural health, whether this exposure translates into students going back to work in these areas after training awaits to be seen^19,20.

In a previous study by Mubuuke et al^20,21; that interviewed radiology COBES students at Makerere revealed that the students understood the importance and purposes of participating in learning activities in the community.

Governance

Issues of governance were explored through questions of education reform, resource utilization, professional development and program evaluation. The majority of teachers understood the justification of the transformation to the College status however, resource usage in general was judged as poor 20% and (50% were non-committal) and the reasons need to be ascertained. The word resource implied time, equipment, staff, infrastructure as well as money, though we urge that whenever a question regarding resources availability and usage is raised, a negative response is expected given the realities of general resource constraints in most Higher Education institutions in resource poor environment.

The creation of a College of Health Sciences (a higher functional status that carries with it semi autonomy) was a governance matter, that was conceived as reframing and questioning the previous vision and its interpretation. In addition it focused on both the internal and external governance models with the aim of drawing strategic direction.

It was anticipated that the College will realize more funding, more collaborations, a higher research output, more student enrolment and more staff recruitment and retention. More than it would have had it not transformed. Continuous professional development by way of regular refresher courses in select priority areas should be on going to help old and new faculty understand in depth how the new innovative methods of learning and teaching work as well as how they would make it work.

Periodic self-institutional evaluations to track improvements and areas of noncompliance^22,23, as indeed this evaluation exercise was done, and should continue and be regularized.

Among the strengths of this study; there were no significant differences in responses between those who agreed, disagreed and or had neutral responses. This implies general consensus in the opinions shared.

Eligibility to participate was based in part on longevity of service. A five year teaching experience was deemed the minimum appropriate for one to be able to pass judgement on the various aspects outlined in the study. The average teaching experience for the participants was eight years.

Study limitations

This study was not without limitations, the unknown and therefore uncharacterized non-responders who could be the critiques of the changes implemented. The responses were self -reported and therefore may be subjective. However, over the implementation period and before numerous capacity building activities occurred. We believe this provided firm grounding in the understanding of PBL/COBES. This study meant to collect teachers' views only, so the students' views were missed.

Conclusion

Regular curriculum self evaluation is possible and may be adopted as part of a continuous curriculum improvement. PBL/COBES was over all well executed in a resource limited environment.

Competing Interest

The authors declare no competing interests.

Sources of funding for the work

The College of Health Sciences, Makerere University.

Author contribution

SN, GM originated concept. GM, KA participated in data collection and analysis. GM wrote the first draft of the manuscript. All authors participated in the critical review for intellectual content. All authors approved the final manuscript.