Abstract
Objective
To determine the successful implementation and effectiveness of program-level learning outcomes for a 4-year bachelor of science pharmacy degree program.
Methods
A comprehensive and iterative program evaluation framework was implemented and quantitative and qualitative data were gathered.
Results
The critical factors in the successful development and implementation of program-level learning outcomes in this context were program accreditation, the leadership qualities of the curriculum chair, a strong and adequately resourced curriculum team that was able to engage and mobilize the faculty learning community, and scholarly approaches to curriculum reform.
Conclusion
An integrated range of institutional and programmatic strategies enhance the implementation of program-level learning outcomes in a 4-year undergraduate curriculum.
Keywords: learning outcomes, pedagogy, assessment, program evaluation, undergraduate curriculum reform
INTRODUCTION
Although not an entirely new concept in pharmacy curricula, program-level learning outcomes are part of the broader context of pedagogical reform. Program-level learning outcomes focus on higher order and integrated learning abilities (eg, demonstrate critical thinking, responsible use of ethical principles, effective research, communication, and problem-solving skills) in the context of pharmaceutical education. They are designed to be assessable, transferable, and relevant to students' lives as workers and citizens in a diverse world.1-3 Program-level learning outcomes: inform students what they can expect to achieve from a program of study, so they can organize their time and efforts; communicate curriculum/program goals in a meaningful way to a broader community; help to determine the extent to which learning has been accomplished; and guide faculty members and administrators (within resource constraints), in part, to determine program(s) of study, course objectives, appropriate learning experiences, assessment, and program evaluation strategies. In order to meet the needs and circumstances of undergraduate curricula, program-level learning outcomes should be developed by representative members of the whole learning community in order to embrace a wide range of interpretations and adaptations.4 Not surprisingly, therefore, the localized development and declaration of contemporary program-level learning outcomes can be a major undertaking for most institutions and academic units. Furthermore, addressing critical issues such as “how do we actually know that students are able to demonstrate these outcomes on completion of our degree program” present significant challenges for many faculty members and administrators.5,6 Insufficient research has investigated whether and how program-level learning outcomes are actually implemented in practice. This paper examines whether and how learning outcomes are implemented in a redesigned learning-centred 4-year undergraduate curriculum in the Faculty of Pharmaceutical Sciences at the University of British Columbia (UBC), Canada.
The UBC Faculty of Pharmaceutical Sciences includes 30 full-time equivalent faculty members and 550 undergraduate students in a 4-year bachelor of science in pharmacy (BScPharm) degree program. Within the pharmaceutical sciences program, there are 5 subdisciplinary areas: pharmaceutics, pharmacology, pharmaceutical chemistry and drug metabolism, clinical pharmacy, and pharmacy practice. Students are required to complete at least 1 year of general sciences prior to admission to the BSc program.
Following extensive consultation and curriculum redesign efforts from 1998 to 2001, the Faculty developed and launched a learning-centred curriculum which focused around critical graduation expectations and a core set of 9 learning outcomes. Furthermore, through various faculty development workshops, these learning outcomes were connected to a wide range of potential assessment methods and learning strategies, in large part, to enhance authentic implementation.7-12 Table 1 provides examples of how the ability-based learning outcomes were intended to connect to authentic methods of assessment and diverse learning modules within the redesigned 4-year learning-centred undergraduate curriculum.
Table 1.
Ability-based Outcomes, Assessment Strategies, and Learning Modules in a Four-Year Pharmacy Curriculum
To ensure a well-designed and cohesive UBC Pharmaceutical Sciences program, specific attention was paid to vertical and horizontal curriculum integration. For example, disciplinary-based working groups were established to develop course modules over the 4 years of the program and integrate (vertical integration) the learning outcomes with appropriate learning experiences and assessment strategies within the subdisciplinary field. Horizontal integration of knowledge and skills across the disciplines was coordinated by the curriculum committee and chair by designing case-based learning modules entitled Cases in Pharmaceutical Sciences (CAPS), which students take continuously throughout the 4-year program.3 This redesigned learning-centered 4-year undergraduate curriculum is now in its fourth year of implementation. A program evaluation framework was developed to examine the implementation of program-level learning outcomes in this undergraduate curriculum. Action research is central to this framework. Essentially, action research internalizes theory and practice through a systematic and cyclical process of inquiry that involves hypothesis testing, planning, observing, analysis, and action.13 Action research invites curriculum leaders to consider which research questions around program evaluation are important, what data to gather, when and how to collect and analyse these data, how to initiate positive changes to practice, how to engage curriculum stakeholders in the process, and finally, how this research might be of interest to the broader scholarly community. Data collection strategies may be in the form of quantitative (eg, numeric performance and graduation records, rating and rank-order preference scales), and/or qualitative sources (eg, internet or documentation searches, open-ended feedback forms and/or interviews, interpretation of teaching performances from video footage, teaching plans, students' assignments, workbook journals, etc). Appropriate combinations of qualitative and quantitative data can yield critical information to enhance program evaluation.14 In this study, action research design was employed to investigate four critical research objectives that focused on learning context, process, impact and follow-up evaluations.
METHODS
The program evaluation framework used has been employed by educational developers and university teachers in various higher education settings.8,3,15,16 This flexible and iterative evaluation framework was applied to examine whether and how learning outcomes were implemented in the 4-year undergraduate program in the UBC Faculty of Pharmaceutical Sciences. Figures 1 and 2 provide 2 representations of the program evaluation framework. Figure 1 illustrates a heuristic model, which captures the critical role of action research and the contextual, iterative, and interdependent nature of program evaluation components. Figure 2 illustrates a pragmatic model for the learning context, process, impact, and follow-up evaluation phases of implementation analysis.
Figure 1.
An heuristic model conceptualising the interrelationship of critical components in the program evaluation framework.
Figure 2.
A pragmatic program evaluation model conceptualising key phases of implementation analysis.
This framework took into account the institutional learning context and integrated a wide range of program evaluation strategies. Action research was used in this study to provide data on which to reflect upon the effectiveness of program processes and outcomes (eg, examine input from faculty members, practitioners, students, quality of student's work, course instructors' experiences). The following section describes how each of the 4 phases of the program evaluation framework was employed to examine whether and how learning outcomes were implemented in the 4-year pharmaceutical sciences undergraduate curriculum.
Table 2 indicates a series of action research questions that were initiated to examine whether and how learning outcomes were implemented in the pharmaceutical sciences undergraduate curriculum. Data collection strategies were tailored to each specific research question under investigation. Researchers analyzed qualitative data for common and isolated experiences and for major themes.17 Quantitative data were analyzed using descriptive statistics.
Table 2.
Implementation Analysis: Program Evaluation Questions
Learning context evaluations.
These address key issues such as the intended audience for the evaluation, the objectives of the evaluation, and available resources for conducting specified evaluation projects. For example, learning context evaluations might include: researching relevant literature, assessing perceived needs of various stakeholder groups about program processes and outcomes, and evaluating program feasibility issues. What needs to be improved, why, and how? In this study, a learning context evaluation (Q1) was conducted prior to the program, whereby data were gathered over a 1-year period as part of a comprehensive needs assessment. For example, the entire faculty was engaged collectively, and through disciplinary areas, in open dialogue to develop learning outcomes and integrated learning experiences. This was particularly effective when implemented through “town hall” meetings (ie, discussion about curriculum issues for faculty, administrators, students, and professionals in the field), notice-board information about ongoing issues and progress with curriculum reform process, individual and focus group interviews with faculty members, and e-mail surveys and consultation with student and professional groups.
Process evaluations.
These focus on periodic assessments of issues of importance that arise throughout the program (formative). For example, to what extent are learning outcomes made explicitly clear to students through, for example, course syllabi? How do students best achieve learning outcomes? To what extent do learning experiences integrate learning outcomes? What are the strengths and weaknesses of program learning experiences? What needs to be improved, why, and how? In this study, various data sources were compiled to examine the extent to which learning outcomes were reflected in individual course offerings (process evaluation, Q2). These included a review of a representative sample of course syllabi, and the reflection notes from the curriculum Chair in order to assess progress, strengths, and weaknesses of the implementation of learning outcomes in program learning experiences.
Impact evaluations.
These focus on issues of importance that occur as a result of a program (summative) evaluation. For example, how do students demonstrate learning outcomes? To what extent does the program meet, surpass, or fall short of learning outcomes, why and how? What needs to be improved, why, and how? In this study, to assess whether and how learning outcomes were demonstrated by students (impact evaluation, Q 3), researchers analysed data from summative program evaluation questionnaires, focus group interviews with curriculum stream leaders, and a representative sample of student surveys, and examples of students' work.
Follow-up evaluations.
These focus on issues of importance which arose as a result of the longer term (eg, months, year) impact of a program. For example, as a student reflects upon the program and learning outcomes, what does he/she remember and value most? Generally speaking, to whom and to what extent, if at all, did the learning outcomes make any difference? If at all, how did the program contribute to the student's development as a pharmacist and citizen in a diverse world? In this study, researchers analyzed 1-year follow-up data (follow-up evaluation, Q 4) to assess the long-term impact and applications of implementing learning outcomes in the 4-year pharmacy undergraduate program. Data were gathered from a sample of students' responses on survey instruments, pharmacist practitioner feedback comments, analysis of examples of student's final project assignments, and a review of faculty research activities in the 2005-2006 academic year. In addition, follow-up interviews were conducted with members of the Faculty's curriculum evaluation team and Chair to highlight progress, critical long-term aspects of implementing learning outcomes, and suggestions for further program-level improvements.
RESULTS
Learning context evaluation (Question 1).
Interview data with the Curriculum Chair and disciplinary “stream” leaders suggested that external program accreditation was the single biggest factor to influence the development and implementation of program-level learning outcomes. Furthermore, external consultant support, strong curriculum leadership and the ability to engage and mobilize the whole learning community (ie, representative students, faculty members, and stakeholders in the broader program context) through open dialogue and various other forms of communications were critical factors. The lack of academic release time, especially for the Curriculum Chair, hindered progress and momentum to fully develop program-level learning outcomes in a smooth and timely fashion.18 Too many ability-based outcomes (9) have likely contributed to the additional challenges experienced with implementing these outcomes across the whole curriculum and within the 5 program areas and individual courses. Several of the listed outcomes could be integrated and simplified (eg, 5 and 9, 2 and 4).
Process evaluation (Question 2).
To assist with implementation of program-level learning outcomes, curriculum planning documents revealed that specific courses at key stages throughout the curriculum (eg, CAPS courses and experiential-based courses) were strategically targeted to incorporate all 9 learning outcomes (Table 1). CAPS modules were intended to integrate learning across the subdisciplines and cases increased in complexity from year 1 to year 4. Experiential clerkships allowed students to build upon the ability-based outcomes acquired during the first, second, and third years of the undergraduate program in order to provide pharmaceutical care to patients in a variety of environments. Table 3 indicates the nature of integrated learning strategies undertaken by students in the experiential-based course (Phar 489) related to the 9 learning outcomes. To assess completion of the learning outcomes, the following data were collected: responses on end-of-year feedback surveys submitted by students and preceptors; periodic stakeholder focus group sessions; review of students' work submitted via portfolio; WebCT discussions with students; informal (telephone, site visits, face-to-face) discussions with students and preceptors; and, student/preceptor competency surveys and discussions during clinical instructor education programs. For example, a review of the Phar 489 course syllabus revealed that a student paper and oral presentation with the preceptor was employed to assess a student's ability to critically appraise primary literature sources on key aspects of pharmaceutical care in this context.
Table 3.
The Nature of Integrated Learning Strategies Undertaken by Students in the Field-Based Course Related to the Nine Learning Outcomes
All teaching faculty members were also responsible for ensuring that all individual courses within their stream included at least 3 outcomes in course syllabi. While this implementation strategy is useful for faculty members in the beginning stages of curriculum reform, it does not really address the balanced implementation of all 9 learning outcomes. Furthermore, following a review of a representative sample (40%) of course syllabi that were posted on the faculty's undergraduate curriculum WebCT site, most course syllabi had only identified 1 or 2 of the listed learning outcomes.
Impact evaluation (Question 3).
Table 4 summarizes 1176 student responses (from the teaching evaluation questionnaire) across 15 second-year courses in the BSc (Pharm) curriculum (2005) pertaining to their perceptions about the alignment of course learning outcomes with the instructional methods and assessment practices.
Table 4.
Second-professional Year Pharmacy Students' Survey Responses About the Alignment of Course Learning Outcomes With Instructional Methods and Assessment Practices (N = 1176)
Approximately 70% of students felt that instructional methods facilitated the achievement of learning outcomes. Traditional lecturing techniques and an examination assessment method remain effective strategies for ability-based outcomes such as mathematical skills. Much follow-up work remains to be done to fully integrate and monitor all learning outcomes across the curriculum. Second, approximately three fourths of students felt that assessment practices were generally related to learning outcomes. In addition to the CAPS courses, experiential-based course work (Table 3) required students to demonstrate the program-level learning outcomes in a wide range of assessment methods. The documentation/evidence was evaluated according to predetermined criteria: quality of evidence related to learning outcomes; explanatory (integration and application of learning); evaluative (quality of analysis and effectiveness of application); developmental (reflect on further goals/learning required); and, quality of presentation. However, a review of a representative sample (40%) of course syllabi indicated that overall individual course assessment methods tended to be heavily weighted toward traditional midterm and final examinations, which have significant limitations for the assessment of the outcome-based abilities, beyond the knowledge domain. Student survey data also suggested that very few individual courses offered formative assessments of learning outcomes.
Follow-up evaluation (Question 4).
Although ongoing refinements and adjustments to the curriculum are inevitable, survey feedback data and focus group interviews suggest that the program-level learning outcomes had generally been met favorably by students, practicing pharmacists, and faculty members by the fourth year of implementation. Several ongoing implementation challenges surfaced from these data. Notably, the implementation of learning outcomes (in these relatively early stages) has tended to be inconsistent across the 5 subdisciplinary areas. In some cases, only 1-2 outcomes are being addressed within courses and traditional lecturing and examinations are the format for course delivery. While this strategy can be useful in the early stages of implementing learning outcomes, a wider repertoire of teaching and learning strategies is required in order to address the broad range of learning outcomes (and student's learning styles). The sheer quantity of program-level learning outcomes (n = 9) presented overwhelming challenges for some faculty members. Most faculty members also commented on the impact of the learning outcomes on their workload:
…my initial teaching workload increased significantly [with the new curriculum] but my course is now better aligned with the whole curriculum and I now tend to invest my preparation time more toward active learning processes rather than focus exclusively on course content. I'm still working on this!
Consistent with the ethos of action research, it was clear from the survey data that there is a “culture of evaluation” within the Pharmacy program for faculty members, students, and curriculum leaders. There have been several highly funded projects around the scholarship of teaching and learning (eg, program evaluations, CAPS and experiential-based project development, technology and curriculum learning outcomes) that have been awarded by the University's Teaching and Learning Enhancement Fund to members of the Pharmacy faculty. Furthermore, several peer-reviewed publications and conference presentations have resulted from action research projects in this curriculum context. In recognition of the importance for the scholarship of curriculum practice in this setting, this Faculty has developed a part-time position for a coordinator to oversee undergraduate program evaluation projects. Ongoing action research projects will examine various aspects of program processes and outcomes (eg, How do graduates apply learning outcomes in pharmacy practice? What is the impact of learning outcomes on students' grades? What is the impact of learning outcomes on teaching evaluation scores?). More details about specific curriculum research projects can be found on the Faculty's program evaluation website: http://www.pharmacy.ubc.ca/undergraduate_programs/program_evaluation.html
DISCUSSION
As institutions and academic units globally “grapple” with the challenges of developing and adopting program-level learning outcomes, this study provides a useful framework for implementation analysis and how program-level learning outcomes are implemented in the context of a 4-year undergraduate curriculum. By taking into account the learning context and providing a long and broad perspective for investigating the implementation of program-level learning outcomes in higher education, this framework is adaptable to a wide range of institutional and programmatic settings. Action research methodology is at the very heart of this framework, and was employed to identify critical research questions for investigation, to gather and analyse authentic data on which to (cyclically) reflect on the implementation of program-level learning outcomes, as well as to respond with refinements to practice and further questions for continuous investigation.
Previous studies have provided a more detailed analysis for the above learning context evaluation question.8,19,3 In this study, there was widespread agreement among faculty members and pharmacy practitioners that student confidence and competencies have increased in experiential placement modules; the use of effective learning technologies (eg, mixed-mode blended WebCT courses) within the curriculum has greatly increased in order to monitor and integrate learning outcomes within and across subdisciplinary course areas; and a number of faculty members commented that their teaching evaluations had now become more useful and detailed with both quantitative and qualitative data pertaining to the use of learning outcomes in the context of a given course.
Students demonstrated program-level learning outcomes through various learning experiences and assessment methods (eg, integrated case-based courses, capstone projects, examinations, student presentations and electronic [E-]portfolios). In particular, the use of [E-] portfolio assessment in the context of these courses proved to be a particularly effective method for students to demonstrate and integrate all 9 learning outcomes in diverse ways. Essentially, the portfolios provided a framework, in electronic or paper format, for students to reflect on and record their professional and practice development activities (eg, problem-solving examples, patient care situations, preceptor assessments, self-assessments, and other documentation relating to their progress throughout the duration of their program).
Institutional and Faculty-level support for curriculum reform, especially in the form of adequate time for professional development is key to helping faculty members implement learning outcomes within individual course and program learning experiences. Several faculty members in this curriculum context were enrolled in an 8-month certificate program on the scholarship of teaching and learning in higher education.15 This comprehensive professional development opportunity provided a forum in which to study relevant literature and to debate, analyse and develop strategies for enhancing curriculum reform in the pharmaceutical sciences setting. Based on data in this study, specific strategies would further enhance the implementation of learning outcomes in this context including: professional development strategies for faculty members (eg, continued enrolments in the UBC Faculty Teaching Certificate Programme, workshops and 1-to-1 assistance on learning strategies, and authentic assessment of learning outcomes) and students (eg, learning outcomes orientation workshops, portfolio tutorials); and, programmatic strategies (eg, learning-centred course design, program-level student [e-] portfolios required at the end of years 2, 3, and 4 of the undergraduate curriculum).
Implementing program-level learning outcomes in an 4-year undergraduate curriculum is a contextually bound, complex and multi-faceted process. It is shaped by many factors (individual, social, political, economic, organizational, and cultural) and involves many people (administrators, facilitators, faculty, instructors, and students) at various stages and subdisciplinary areas throughout the program. Not surprisingly, despite coordinated and strategic attempts to adopt well-developed program learning outcomes, they were not fully implemented, as intended, in the complex world of curriculum practice. Furthermore, while valuable lessons can be learned about the development and implementation of program-level learning outcomes in this undergraduate context, these experiences are unlikely to transfer exactly to other undergraduate program settings.
CONCLUSION
This article provides a flexible program evaluation framework for examining whether and how program-level learning outcomes are implemented in an undergraduate curriculum. Data from this study suggest that integrating a comprehensive range of institutional and programmatic strategies will enhance the implementation of program-level learning outcomes. For example, program accreditation, the leadership qualities of the curriculum Chair, a strong and adequately resourced curriculum team that was able to engage and mobilize the Faculty learning community, and scholarly approaches to curriculum reform were critical factors in the successful development and implementation of program-level learning outcomes in this context.
ACKNOWLEDGEMENTS
The authors would like to express their thanks to the following faculty members for their assistance with data for this manuscript: Dr. David Fielding, Dr. Rosemin Kassam, and Kristin Wright.
REFERENCES
- 1.Baird LL. Documenting student outcomes in graduate and professional programs. In: Bilder AE, Conrad CF, editors. Challenges in Assessing Outcomes in Graduate and Professional Education. Vol. 92. New Directions for Institutional Research; 1996. pp. 77–87. [Google Scholar]
- 2.Clanchy J, Ballard B. Generic skills in the context of higher education. Higher Educ Res Dev. 1995;14:155–66. [Google Scholar]
- 3.Hubball HT, Burt HD. An Integrated Approach to Developing and Implementing Learning-Centred Curricula. Int J Acad Dev. 2004;9:51–65. [Google Scholar]
- 4.Cox M, Richlin L. Introduction to faculty learning communities. Building Faculty Learning Communities New Directions for Teaching and Learning, Number 97. In: Cox M, Richlin L, editors. San Francisco, Calif: Jossey-Bass; 2004. pp. 5–24. [Google Scholar]
- 5.Drummond I, Nixon I, Wiltshire J. Personal transferable skills in higher education: the problems of implementing good practice. Quality Assurance Educ. 1998;6:44–58. [Google Scholar]
- 6.Shavelson R, Huang L. Responding responsibly to the frenzy to assess learning in higher education. Change. 2003:11–8. [Google Scholar]
- 7.Adamcik B, Hurley S, Erramouspe J. Assessment of pharmacy students' critical thinking and problem-solving abilities. Am J Pharm Educ. 1996;60:256–65. [Google Scholar]
- 8.Albon S, Hubball HT. Course Design in Pharmaceutical Sciences: A Learning-Centred Approach. Am J Pharm Educ. 2004;68(5) Article 114. [Google Scholar]
- 9.Albon S, Cancilla D, Hubball HT. Using remote access to scientific instrumentation to create authentic learning activities in pharmaceutical analysis. Am J Pharm Educ. 2006;70(5) doi: 10.5688/aj7005121. Article 121. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Brandt BF, Clements M, Piascik P. Problem-solving activities for first-year pharmacy students. Am J Pharm Educ. 1998;62:450–7. [Google Scholar]
- 11.Lockhart M, Borland KW. Critical thinking goals, outcomes, and pedagogy in senior capstone courses. J Faculty Dev. 2001;18:19–25. [Google Scholar]
- 12.Raman-Wilms L. Innovative enabling strategies in self-directed, problem-based therapeutics: enhancing student preparedness for pharmaceutical care practice. Am J Pharm Educ. 2001;65:56–64. [Google Scholar]
- 13.Mills GE. Action Research: A Guide for the Teacher Researcher. Upper Saddle River, New Jersey: Merrill Prentice Hall; 2000. [Google Scholar]
- 14.Wolfe P, Hill A, Evers F. A Handbook for Curriculum Assessment. Ontario: University of Guelph Publications; 2006. [Google Scholar]
- 15.Hubball HT, Burt HD. Scholarship of teaching and learning: theory practice integration in faculty certificate programs. Innovative Higher Educ. 2006;30:327–44. [Google Scholar]
- 16.Hubball HT, Levy A. Graduate course design in health care and epidemiology a learning-centred approach. J Faculty Dev. 2004;20:11–9. [Google Scholar]
- 17.Strauss A, Corbin J. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. 2nd Edition. Thousand Oaks, Calif: Sage; 1998. [Google Scholar]
- 18.Schneider CG, Shoenberg R. Habits hard to break: how persistent features of campus life frustrate curricular reform. Change. 1999:30–5. [Google Scholar]
- 19. Burt H. Learning Outcomes Report. Faculty of Pharmaceutical Sciences Curriculum Document. 2000. Available at: http://www.pharmacy.ubc.ca/undergraduate_programs/program_evaluation.html.