A Course Assessment Process for Curricular Quality Improvement

Abstract

Objective. To describe a systematic assessment process that provides continuous improvement in the curriculum, supports faculty development, and enhances student learning outcomes.

Design. Teams of faculty members, students, and course instructors conducted course assessments, which consisted of monitoring the delivered instruction for agreement with planned content and course integration within the curriculum, and providing ongoing feedback for improving course content, course management, faculty teaching, and student learning experiences.

Assessment. Areas needing improvement were identified and appropriate changes were made. Improvements were achieved in course policy standardization, course integration within the curriculum, faculty teaching behaviors, and student experiences.

Conclusion. The curriculum assessment plan provides a structured method of monitoring and delivering continuous quality improvement.

INTRODUCTION

The 2007 Accreditation Council for Pharmacy Education (ACPE) standards include new and specific requirements for the development, implementation, and management of curriculum. These standards require an orderly and systematic review of curricular structure, content, processes, and learning outcomes, and they charge faculty members with the development, organization, delivery, and improvement of the curriculum.¹ The University of New Mexico College of Pharmacy Curriculum Committee is responsible for the development of outcomes and competencies, curricular revisions, and continuous monitoring of the curriculum to ensure that course content, learning activities, instructional methods, and appropriate program assessments are coordinated to generate desired student learning outcomes.

Assessment models and methods used in other higher education programs often are not applicable to pharmacy education for several reasons.² First, pharmacy course structure is different from the large-lecture, single-instructor format used elsewhere in higher education, eg, a professional course may have 1 course coordinator but be taught by multiple instructors. A second difference is that the pharmacy curriculum is standardized and carefully structured with limited opportunities for students to select their courses or instructors. Students move through the curriculum in cohorts, each exposed to the same learning experiences and opportunities, with the exception of elective courses. A third difference between pharmacy and other higher education programs is the integrated structure of core courses within the pharmacy curriculum. Courses are integrated both horizontally, with reinforcement and collaboration between courses across each year of the professional program, and vertically, requiring students to successfully complete each level of the curriculum in order to progress. These differences require a comprehensive curricular design and assessment program focusing on the entire integrated pharmacy curriculum in addition to individual courses.

Several reports in the literature describe assessment efforts at colleges and schools of pharmacy, such as developing an assessment office,³ creating a culture of assessment,⁴ and conducting curricular evaluation and refinement.⁵ There also are published comprehensive reviews that outline the rationale and basic principles for assessment in pharmacy education.^6,7 However, no method for comparing the planned pharmacy curriculum to the delivered instruction was found in the literature.

Our goal was to develop a systematic assessment plan that would provide continuous improvement in the curriculum while supporting faculty development and student learning outcomes. We define curriculum to include content, student learning outcomes, instructional methods and approaches, teaching and learning processes, and student, instructor, and program assessment.^6,8 Ongoing program assessment also should assist faculty members in making informed changes in course structures and instructional methods. Information collected should provide evidence of successes as well as suggest specific steps, resources, and support needed for improvements. Table 1 provides a summary of effective program assessment principles.

Table 1.

Effective Principles and Characteristics of a Pharmacy Curriculum Assessment Program⁶

The curriculum assessment and development plan was initially created for 3 reasons: to comply with ACPE Standards 2007¹ to ensure course integration and sequencing; and to respond to major revisions in both the PharmD curriculum and the college's professional competencies and identify content gaps and duplication within and between courses. The 3 Professional Competencies and Outcome Expectations listed in Standard 12¹ were used to synthesize the 2004 Center for the Advancement of Pharmaceutical Education Educational Outcomes⁹ into 30 professional competencies. Although the Curriculum Committee worked with individual course coordinators to develop learning outcomes supporting the new professional competencies, confirmation that the delivered course content corresponded to the intended content was needed. Our main objective for this paper was to provide an example of a comparison of the planned curriculum to the delivered curriculum.

DESIGN

A curriculum committee was formed consisting of representative members of the educational community: students, faculty members, member of the college administration (assistant dean for curriculum), and members of the pharmacy community. Faculty representation on the committee included an equal number of members from the pharmaceutical science and pharmacy practice departments. Course coordinators of multi-instructor courses were encouraged to serve on the committee to facilitate the sequencing and integration of the curriculum and help disseminate knowledge of course management policies and procedures where it would be most effective.

The 3 course coordinators for the Pharmaceutical Care Laboratory (PCL) courses were included as these courses are key to reinforcement and integration of key concepts and skills across the entire curriculum. Serving on the curriculum committee gave the course coordinators insight into the content of concurrent courses as well as immediate access to other course coordinators, which greatly facilitated integration of curriculum content throughout the program.

Two students from each of the first 3 years of the curriculum were selected from a list of volunteers based on their maturity and motivation to participate in the time-intensive activity of curriculum committee work. Two students were chosen from each class to ensure class representation at all committee meetings. Students in the fourth year were not included because of conflicts with their experiential schedules.

The student committee members were encouraged to share responsibilities and discuss curricular issues with each other. As the courses progressed, students provided ongoing feedback to their peers and functioned as a conduit of feedback from their peers to the curriculum committee. Students were encouraged to provide critical but objective reviews of courses and instructors. Any negative feedback from students was presented to faculty members by the committee leadership as constructive criticism with suggestions for improvement to protect the confidentiality of the students.

Practicing pharmacists representing both community and hospital practices were invited to participate. These members provided perspective on the knowledge, skills, and values most essential for the daily practice of pharmacy.¹⁰ Members who also served as student preceptors provided an external view of common student shortcomings and offered practical solutions to identify and resolve deficiencies in the curriculum that may have led to these shortcomings.

All committee members were required to attend an orientation during which they were instructed in the policies, procedures, and confidentiality expectations of the committee. A reference manual containing all critical materials and explanations of all policies, procedures, and activities was provided as a guide to the orientation session. The training and the manual were intended to promote consistency among members and improve the reliability and validity of course and instructor evaluations.

Assessment Process

The process of assessment was multidimensional, including reviews by students and teaching peers and self-reviews by course coordinators (Figure 1). Each assessment provided documentation of effective activities and recommendations for course improvement as well as verification that the delivered course content matched the course learning outcomes in a manner that supported the relevant professional competencies.¹¹ Appropriate resources were provided or developed to make necessary changes to improve the course.

Figure 1.

Curricular assessment process designed and implemented by the University of New Mexico College of Pharmacy Curriculum Committee.

Each core course in the doctor of pharmacy curriculum was assigned to an assessment team consisting of a faculty team leader and 2 students from the curriculum committee. While most teams had only 3 members, additional faculty members could be assigned to teams conducting comprehensive course reviews. Whenever possible, pharmacy practice faculty members are assigned to pharmaceutical science courses and vice versa to facilitate faculty-wide comprehension of the entire curriculum and to enhance content integration and sequencing.

The team leader was expected to attend the initial meeting of the course to assess presentation of the course content, syllabus, schedule, and course management explanations. New courses (those with a new instructor or coordinator) or those with recent managerial or instructional difficulties underwent a comprehensive course review. The team leader or another faculty member attended 1 or more class meetings each week. An abbreviated course review was conducted for well-established courses with few managerial or instructional concerns. These abbreviated review courses were audited by the team leader every 2 to 3 weeks.

The 2 student committee members enrolled in the course were assigned to the course team. Students were a vital link in the assessment process, as they actually experienced the entire course and could provide the most informed opinions and insights. They were instructed to report their own observations and opinions and to comment on the general opinions of the class about the course. Students in each course collaborated to submit a single evaluation for that course.

Course coordinators prepared self-assessments of their role in the course including an evaluation of their effectiveness in course management; reviews of the teaching contributions of other course instructors; a summary of student successes and failures; and recommendations to improve the course.

Mid-semester feedback about the course progress was solicited with an open-ended, extra-credit question(s) on a quiz or examination. A question or 2 was presented on a separate page of the examination and no student identifiers were included on that page to ensure anonymity. All students received a bonus of 1 or 2 points on their quiz or examination score even though there was no requirement to respond. Prior to analysis, the course coordinator tabulated the responses for changes in the course presentation that needed to be made immediately or in the future. A report on the mid-semester feedback activities was included in the course coordinator's self-assessment.

At the end of the semester, the course coordinator and student members of the team forwarded their completed assessments to the team leader, who then prepared a single report synthesizing all comments and observations and including the University-required, general course/instructor evaluations. A summary of recommendations for course improvement was made before the full report was submitted to the curriculum committee 4 weeks after the end of the semester.

The curriculum committee held a comprehensive, semester-based review of assessments for all courses. Committee members were provided a copy of the synthesized course evaluation, and each team leader was given 10 to 15 minutes to present the course overview. Course coordinators were encouraged to attend and participate in the general discussion, during which issues were addressed, explained, or clarified, and final recommendations for course improvement were made by the committee. The chair of the curriculum committee prepared the final course evaluation incorporating committee discussion and recommendations. The final course evaluation was forwarded to the course coordinator and archived on Share Point (Microsoft Corporation, Redmond, WA). The committee chair prepared a brief summary of each course for presentation to the college's executive administration.

The final stage of the course review cycle was to use the archived course review to develop materials for the next course offering (Figure 1). After preparing a draft syllabus and course schedule, the course coordinator met with a member of the curriculum committee to examine the recommendations made during the previous course review and to discuss changes for course improvement. The recommendations pertained to course content, structure, management, or instructional strategies. Assistance in implementing changes to the course was provided as needed. The final course syllabus and schedule were reviewed and approved by the chair of the curriculum committee to ensure that all relevant recommendations had been implemented. The assessment program was integrated into core course preparation and is recognized as an ongoing quality-improvement effort.¹¹

EVALUATION AND ASSESSMENT

A standardized course evaluation form was completed by each course coordinator and all team members for all core courses beginning in fall 2007 through spring 2010. All participants were instructed on how to use the form to promote consistency among evaluators and across semesters. The form was designed to evaluate courses from a variety of perspectives over the semester. The open-ended items initially included:

• (1) Course policies and procedures: completeness of course syllabi, use of standardized syllabi format, and compliance with policies and procedures;

• (2) Course content and relationship to learning outcomes: content and competencies match, learning objectives addressed;

• (3) Integration within the curriculum: appropriate placement within vertical integration, appropriate sequencing and horizontal integration with concurrent courses when appropriate;

• (4) Skills: identifies that knowledge and skills are developed, practiced, and assessed;

• (5) Student assessment: types and number of assessments linked to learning objectives, student performance, and advancement;

• (6) Course coordinator performance review: course management skills;

• (7) Summary of individual faculty teaching reviews: summarized from teaching evaluation forms and student instructor evaluations;

• (8) Recommendations: specific recommendations and suggested changes for course improvement.

After 2 complete review cycles (academic years 2007-2008 and 2008-2009) of all courses, we determined that there were no longer outstanding issues with the first 3 items on the course evaluation form. These items were removed in fall 2009 and replaced with 2 new items of interest to the college:

• (1) Active learning: describe active-learning techniques observed

• (2) Key assessments and key artifacts: specific examinations or learning activities that serve as demonstration of a competency

Each instructor was observed and a review of teaching skills and ability was prepared. The teaching evaluation was included as part of the course review but also could be used as a standalone teaching review for purposes of annual performance, promotion, and tenure. Instructors were rated on verbal and visual presentations; handouts and assignments; student interactions; content; teaching techniques, and examination items. Instructor performance on each item was rated using a 5-point Likert-type scale for quantitative data with open-ended comments for a more qualitative assessment. All feedback was offered in a positive and constructive manner using a plus/delta approach¹² (method for identifying items that are being done well and items that could be improved) aimed at coaching- and skill-development opportunities for teaching faculty members.

At the time of the assessment, the college had 28 core classroom courses, 23 of which were considered “mature” courses as they had been offered consistently for more than 3 years. The remaining 5 courses were more recent developments added to the core curriculum to meet Standards 2007.¹ The content of the courses was mapped and included specific competencies of the revised curriculum.

The assessment-feedback loop brought about subtle changes to many areas of the curriculum. As areas that needed improvement were identified, solutions were developed, implemented and tracked through the assessment process. Effective solutions required a greater level of coordination among many individuals than had occurred previously, including the course coordinator, department chair, administration, other faculty and instructors, and students. The case study seen in Table 2 illustrates the extent of these individuals’ involvement needed to make effective changes.

Table 2.

Application of a Course Assessment Process in a Self-Care Therapeutics Course in the Doctor of Pharmacy Curriculum

Faculty input and involvement was solicited and concerns were addressed during the development process. While the majority of faculty members recognized the benefits of the plan, a few were skeptical, their main concerns being loss of autonomy, fear of centralized control, and erosion of academic freedom. The plan was presented as one of continuous quality improvement with faculty participation and feedback. Recalcitrant faculty members were asked to allow an audit of their course with the initial feedback provided for their review and discussion before dissemination. The annual course review then provided faculty members an opportunity to discuss and consider any revisions or changes proposed for implementation. Once they realized that the structured course design and coordinated curricular content presentation facilitated their instruction, their resistance abated.

As a result of the assessment process, faculty involvement with the curriculum increased. Attending a lecture in pharmacology, a pharmacy practice instructor identified and corrected outdated information presented on a drug class. Duplications of instruction were noticed, leading to collaboration among instructors in course preparations. Solutions were attempted, successes or failures realized, and needed corrections made. Examples of corrections included relocating calculations from the first-year pharmaceutical care laboratories to a single dedicated course and integrating additional review of the Top 200 drugs across 2 years of care laboratory with reinforcement in the Mechanism of Drug Action courses.

Faculty participation in curriculum committee meetings continued to be encouraged. Non-members were welcomed to present petitions for new courses, suggest changes, or simply to sit in on committee meetings. Course coordinators were encouraged to attend the final course evaluation at the semester review of the curriculum, and their participation in the final review served as a summative evaluation of the course.

Students became more positively involved in the curriculum. Students were full participating members of the curriculum committee, and their input was actively sought and their suggestions implemented as appropriate. For example, student comments regarding the amount of work for a single semester credit prompted a more in-depth review of course demands, leading to an increase in course credits.

The development of standardized syllabi provided students clarity on what was expected, allowing them to take more responsibility for their own learning. As a result, the number of student issues related to course management, while not tracked, seemed to decline. Immediate and objective student feedback was often given directly to members of the curriculum committee with confidence that it would be adequately addressed. One example involved a guest instructor who was editing his PowerPoint slides as he was presenting. After being notified by a student about this practice, a committee member subsequently observed and documented the inappropriate activity. Details of this event were then addressed in the course review and recommendations to discontinue this practice were made.

After implementation of the assessment process, many preceptors who had supervised and instructed students on introductory and advanced pharmacy practice experiences (IPPEs and APPEs) for several years reported that students were better prepared than in previous years. While comments were received from all areas of professional practice experiences, the advanced group of students generated the most feedback. Clinical preceptors commented that these students were better prepared and more ready to contribute during their APPEs. Community practice preceptors reported a decrease in the need to coach students through basic concepts and tasks.

DISCUSSION

The assessment process proved to be useful in that it supported student learning by focusing on the educational program. Progress toward course integration, scaffolding, and sequencing was satisfactory and ongoing. Faculty instruction also seemed to improve. While part of the improvement may be attributable to the increase in faculty observation (Hawthorne effect), some is a result of the ready availability of resources to do a better job. Faculty members self-identified areas needing improvement and independently sought assistance in making changes. Communication between the Pharmacy Practice and Pharmaceutical Sciences improved, and instructors from both groups attended courses sequenced with their own and then worked with each other to improve the overall educational experience.

Changes made by course instructors in response to the feedback supported the appropriateness of curriculum changes. Students reported they felt well-prepared for the professional practice experiences and that the classroom and laboratory exercises are appropriate for their experiential learning needs. Students attain a deeper understanding of the material encountered in historically difficult classes when prerequisite material is presented to them in the appropriate learning sequence.

While improvement in student performance may be a result of many factors, no changes were made to the admissions process, core course requirements, or instructional faculty members. Because the only intentional variance in student instruction was the implementation of the curricular assessment plan, the plan was credited with having a significant impact on student performance.

Course syllabi were standardized to provide consistent information to students about university and college policies and procedures, and course goals and objectives incorporate and support the appropriate professional competencies of the college. The assessments confirmed that curricular course placement supported integration between courses, and sequencing appeared to be appropriate. The delivered content for mature courses matched the intended content, while newer courses are making progress in matching intended and delivered content

The assessment and feedback process implemented as part of the quality improvement plan for the curriculum has been successful. All participants — students, faculty and preceptors — reported improvement in and progress toward successfully integrating the classroom curriculum while optimizing student learning outcomes. Involvement of the curriculum committee in providing proactive solutions to identified issues is recognized and becoming established within the college.

The success of this curriculum development and assessment plan depended largely on 4 strategies. Courses are reviewed by an integrated team that includes faculty members, students, and the course coordinator; the review included examination of both horizontal and vertical contributions to the PharmD curriculum; information from annual course reviews was constructively used in subsequent course planning, management and instruction; and the willingness of faculty members and administration to embrace change was respectfully nurtured.

CONCLUSIONS

This paper presents a comprehensive curriculum development plan in which instructional content was monitored. The curricular assessment model met its established goals. By design, the course assessment program was systematic, ongoing, and comprehensive. Implemented annually in each course by a team of faculty members and students, it incorporated multiple perspectives and specific feedback for maintaining and improving quality within and across courses. A key component of this program was the annual course review, which completes the assessment cycle and ensures implementation of recommendations. As carried out in our setting, assessment resulted in meaningful improvements not only in teaching and learning but also in preparing student pharmacists for pharmacy practice experiences and for professional practice after graduation.