A Progressive, Collaborative Process to Improve a Curriculum and Define an Assessment Program

Abstract

Objective. To revise the University of Oklahoma College of Pharmacy’s professional program outcomes and create an assessment map using results from previous peer review and mapping of all professional courses and curricular streams of knowledge, skills, and attitudes (KSAs).

Design. After consolidating 15 original program outcomes into 11 more precise outcome statements, defining KSAs for each, and getting faculty approval of them, the committee detailed measurable program expectations upon graduation for each outcome and created an assessment map identifying where KSAs were taught, how they were to be assessed, and the expected ability level (novice, competent, proficient) for each across the curriculum.

Assessment. The committee’s work identified deficits, inconsistencies, and disproportionalities in professional program assessment. It recommended assessments to capture student achievement of each outcome, identified performance levels and criteria to measure outcomes progressively in each professional year, and outlined a process to provide students periodic reports on their progress in achieving each outcome.

Conclusion. This work establishes a firm foundation for ongoing efforts to measure effectiveness of the professional program, especially in light of Accreditation Council for Pharmacy Education’s (ACPE) revised accreditation standards.

INTRODUCTION

As colleges and schools of pharmacy continue to review, map, and modify their curricula, a structured yet flexible process is needed to support the effort.¹ Faculty members at the University of Oklahoma College of Pharmacy have faced this challenge for more than 11 years as they engage continuous quality improvement while remaining compliant with the Accreditation Council for Pharmacy Education’s (ACPE) accreditation standards.²

The initial process of curricular mapping, review, and assessment of each required course in the professional sequence took place from 2003 to 2005 after the first iteration of the new entry-level doctor of pharmacy (PharmD) curriculum was completed. The purpose was a peer-review evaluation of each course’s content, delivery, and administration, academic load, expected student performance level, integration into the curriculum, and link to programmatic outcomes.³ Based on faculty members’ evaluation and recommendations, a second review in 2006-2007 focused on the effect of curriculum structure on course content, skill advancement, and expected student performance. The review resulted in a resequencing of some core courses to improve advancement of knowledge and skills.³ Based on the first two reviews, 8 longitudinal “streams” of KSAs developed across the four years of the curriculum were identified.⁴ In 2008-2009, a third curricular review evaluated how the courses in each of these “streams” provided a foundation for knowledge and skill advancement, how each was integrated into the respective stream or streams, and how they helped attain programmatic outcomes.⁴ The review identified the need to update and define the programmatic outcomes, to define the proficiency level that students needed to demonstrate knowledge and skills for each outcome and at what point in the curriculum this achievement was measured, and to develop and uniformly apply the assessment of student performance.

These three reviews improved faculty awareness of each course’s content, place in the curriculum, link to programmatic outcomes, interconnectedness to other courses, and expected student performance. The experience and insight gained through these reviews led to the next step of revising and further defining programmatic outcomes and proposing measurable performance standards for each outcome. To achieve the current review of the curriculum, an ad hoc Outcomes and Assessment Committee of faculty members and practitioners in the field were charged with reviewing and, if needed, redefining the professional program’s terminal outcomes, proposing measurable performance standards for each outcome, and outlining an assessment plan to evaluate student achievement of these standards. The OAC committee accomplished its charges using a 3-step process over 2 years.

The objective of this revision was to engage a majority of the faculty members in creating revised educational outcomes and defining KSA statements for the new outcomes. A second objective was to involve a majority of faculty members in creating an assessment map that categorized assessment types used in each professional year for every new outcome, noted assessment gaps for each outcome, outlined assessments for each year, and proposed timelines for assessments. The third objective was to create a standardized process that other schools of pharmacy could use when revising program outcomes and creating assessment maps.

DESIGN

Starting in the fall semester 2009, the dean of the College of Pharmacy at the University of Oklahoma appointed an ad hoc assessment committee to define the professional program outcomes and assessment strategies (designated “the OAC committee”). The committee consisted of 23 members including a chair, representative faculty members from the departments of pharmaceutical sciences and of pharmacy, and adjunct faculty preceptors representing diverse areas of expertise. The committee was charged with proposing: (1) definitions of KSAs expected for each professional program outcome, (2) measurable performance standards for each professional program outcome, and (3) a map of formative and summative assessments of these performance standards across the 4-year curriculum. The charges were addressed in 2 phases (designated OAC 1 and OAC 2) over a 3-year period with a 1-year intermission to complete the college’s accreditation self-study and site visit. The timeline is described in Table 1.

Table 1.

Timeline of the Outcomes and Assessment Committee (OAC)

In the first phase, the OAC committee addressed the first charge (proposing definitions of the KSAs expected for each professional program outcome) by reviewing the 15 Center for the Advancement of Pharmacy Education (CAPE) Outcomes from 1998, which were adopted when the college transitioned to an entry-level PharmD program.⁵ After updating and consolidating the 15 outcomes into 11 (Table 2), the committee created KSA statements (Table 3). These statements reflected the expectations for generalist providers capable of beginning pharmacy practice in any health care environment or progressing to postgraduate training. After finalizing the 11 outcomes and associated KSA statements, the committee presented the work at a faculty retreat, which was approved unanimously in a faculty-wide vote.

Table 2.

The 11 New Professional Program Outcomes

Table 3.

Example of Calculations Outcome Knowledge, Skills, and Attitudes Statement

The OAC 2 committee was led by the OAC 1 faculty chair and 10 faculty committee members (with some overlap of OAC 1 members). They worked to address charges 2 and 3 by proposing measurable performance standards for each outcome and by outlining an assessment plan for evaluating students’ achievement of the standards across the 4-year curriculum. This aspect of the work was important because assessment of educational outcomes is a vital step in producing qualified graduates.⁶ Faculty members should be able to articulate what graduates need to have accomplished by stating outcomes the latter are held accountable to. They should also be able to document that graduates have achieved the outcomes by using evidence collected from assessments.

The OAC 2 committee developed an assessment map of each professional outcome by identifying where KSAs were taught, how they were assessed (multiple choice, open-ended, or performance-based), and what ability level was expected (novice, competent, or proficient). Data gathered from every required course in the first through third years and from the core fourth-year advanced pharmacy practice experiences (APPEs: community, hospital, ambulatory care, and acute care) were used to create the map. Aligning this data, content, assessment method, and expected student performance level allowed the committee to track the advance of assessment throughout the curriculum and to identify where deficits existed. The data was compiled in a heat map to visually depict where outcome KSAs had high or low coverage in each course. The heat map was a color-coded table with cells highlighted in varying shades of green with bright green showing high coverage and lighter shades of green showing lower coverage. In contrast, areas of no content coverage were shaded in bright red.

EVALUATION AND ASSESSMENT

Based on a review of the heat maps, the OAC 2 committee evaluated the state of assessment at the time for each professional outcome by determining if diverse assessment methods were used, if the expected performance level was appropriate for the professional year and if the level advanced across the curriculum at an appropriate rate, if there were gaps or deficits in assessment, and if the timing and venue of the assessment was appropriate.

First, the committee evaluated assessment types used in each year’s courses and categorized them along the 11 outcomes. For each outcome, the percentage of courses using multiple-choice, open-ended, and performance-based assessments was calculated. The results revealed that courses in the didactic curriculum (first year through third year) all used multiple-choice, open-ended, and performance-based questions, although 9 of the 11 outcomes use multiple-choice questions for the majority of assessments. Courses in the experiential curriculum (fourth year) primarily utilized performance-based assessments, although multiple-choice and open-ended questions were also used. Based on these findings, the committee recommended that all faculty members employ a variety of assessment methods to capture and evaluate student performance and achievement of professional outcomes, including performance-based examinations (eg, objective structured clinical examinations, or OSCEs), portfolios, reflective papers, inventories, and menu-driven check lists. The committee also recommended continuing the end-of-semester integrated examinations each year.⁷ Since the integrated examination used at the time included multiple-choice questions, the committee recommended adding performance-based assessment of core KSA at the end of the third year.

The committee also evaluated the expected ability level (novice, competent, proficient) for each outcome for each year (Table 4). These results revealed that outcome expectation reasonably progressed from novice to proficient over the course of the professional program. The only outcome expected at the novice level in the third year was the public health outcome and 3 outcomes were expected at the proficient level (drug information systems and literature evaluation, pharmacokinetics, and public health).

Table 4.

The Proposed, Expected Ability Level for Each Outcome Each Year

Next, the OAC 2 committee reviewed gaps in assessments of specific content (KSAs). For example, in some outcomes little assessment occurred of skills and attitudes related to public health in the first year. The committee proposed at least one new assessment for each outcome, with some proposals related to assessing specific course information. However, the committee recommended that assessments not rely solely on standard measures (examination scores) but instead on new measures such as portfolios.⁶ Recommendations for when (year and/or semester) and where (in which courses) the specific assessments should occur were proposed, as well as what the performance criteria should be for the performance-based evaluations.

In addition to specific recommendations for assessing of achievement of every professional outcome in each professional year, the OAC 2 committee made global recommendations regarding the curricular assessment. These recommendations included: (1) employing a variety of assessment methods to capture and evaluate student performance and achievement of professional outcomes; (2) incorporating professional outcome KSAs into relevant course objectives; (3) tagging each assessment throughout the curriculum with professional outcome KSAs, performance level, and assessment type; (4) providing students periodic outcome KSA progress reports and offering students remediation when performance is not at expected level; (5) restructuring introductory pharmacy practice experiences (IPPEs) to develop and track outcome KSAs; (6) elevating acceptable level of performance of all core APPEs; (7) ensuring uniform assessment of all core APPEs; (8) restructuring faculty-student or student-student mentorships to assess outcomes that require nontraditional methods or that are conducted outside didactic curriculum.

These data and 8 recommendations were presented to faculty members at a faculty retreat and were unanimously accepted. One year after the OAC 1 and OAC 2 committees concluded their work, CAPE submitted revised educational outcomes, and the ACPE proposed new accreditation standards for PharmD programs.^8,9 The work of OAC 1 and 2 reveals how these 2 national revisions fit within a dynamic and ongoing nature of the curricular evaluation process. The college will need to focus its next curricular evaluation efforts on aligning the 11 program outcomes within CAPE’s 4 broad domains and 15 specific subdomains,⁸ as well as the 2016ACPE Standards.⁹ Using the structured and systematic process described in this paper will facilitate this alignment.

An ongoing curriculum evaluation, review, mapping, and assessment process is a key component in developing and sustaining an effective professional program. This cycle allows for dynamic updates to educational outcomes and assessment plans. While it may seem counterintuitive to revise program outcomes several years after starting a curriculum review and mapping project, the process allowed the college to gain a better understanding of curriculum content and that content’s integration across the curriculum, which better informed the revision of program outcomes. Moreover, understanding the existing assessments and gaps in assessments better informed the creation of a new assessment plan for capturing the new program outcomes. Essentially, the OAC committee was an internal peer review with broad faculty involvement that occurred over 2 academic years and resulted in outcomes that led to a refined assessment program.

There were 2 perceived strengths associated with the work of OAC 1 and 2. The first strength was the broad faculty involvement in the process. All college faculty members were involved in the work, either through committee assignment or through the provision of information about courses, lectures, or program outcomes. Broad faculty participation, buy-in, and engagement is important for creating and sustaining curriculum and assessment change. This process helped increase faculty awareness of the relationship among courses, program outcomes, and assessment. Without broad faculty involvement, the outcomes and assessments would only exist as part of the written curriculum. By embedding assessments into delivered curriculum, tracking results, and using the feedback in a continual quality improvement process, the college hopes to facilitate student achievement of program outcomes at expected levels of performance upon graduation.

Current efforts focus on prioritizing the 9 committee recommendations and using technology (such as Desire 2 Learn (D2L Corp., Kitchener, Ontario, Canada) or ExamSoft, (ExamSoft Worldwide Inc., Boca Raton, FL) to assist with the tagging and tracking of outcome KSAs throughout the curriculum. Doing so will provide timely feedback to faculty members and appropriate committees and inform students of their progress towards achieving the professional outcomes. The new outcome statements will also need to be tagged to to ensure they are aligned with CAPE Outcomes and ACPE Standards.

SUMMARY

The work of the OAC committee over 2 academic years delineated 11 new program outcomes and associated KSA statements for each outcome. The committee identified deficits, inconsistencies, and disproportionalities in professional program assessment and recommended assessments to capture student achievement of each outcome. The committee also identified performance levels and criteria to measure outcomes progressively in each professional year and recommended a process to provide students periodic reports on their progress in achieving each outcome. Overall , the work of the committee established a firm foundation for ongoing efforts to measure effectiveness of the professional program.