Using the Higher Learning Commission’s Assessment Culture Matrix to Support Continuous Quality Improvement of a Simulated Patient Program

Abstract

The purpose of this commentary is to advocate for the use of the Higher Learning Commission’s Assessment Culture Matrix to support continuous quality improvement (CQI) of simulated patient (SP) programs. We will share examples from our program demonstrating our maturation as it relates to leadership, shared mission and vision, faculty, and resources. While we are at the beginning stages of engaging students, we continue to make progress accessing and systematically using assessment data. We anticipate that sharing our process for utilizing this matrix may help other institutions as they conduct CQI with their SP programs and in other areas of their assessment portfolio.

Introduction

Simulated patient (SP) programs provide pharmacy students the opportunity to practice skills, such as taking a medication history, conducting focused physical assessments, making clinical decisions using critical thinking, adopting patient-centered communication, and recommending changes in medication therapy to a primary care provider in a safe environment.^1-5 Our SP program, within a 4-year pharmacy program at a public institution, started approximately 25 years ago. We have found that continuous quality improvement (CQI) is needed to ensure that the activities continue to be high quality and align with evolving pharmacy practice expectations for new graduates.

Informed by our experiences, we advocate for SP programs to adopt the Higher Learning Commission’s Assessment Culture Matrix to guide CQI efforts.^6,7 This tool includes domains in the areas of leadership, shared mission and vision, faculty involvement, allocating resources, access to and systematic use of assessment data, and engaging students, which aligns closely with the needs of SP programs.^6,7 Within each domain, there are a series of statements and a rating system that the program can use to determine whether they are at the beginning (1 to 3), making progress (4 to 6), or maturation stages (7 to 9) as it relates to creating a culture of assessment.^6,7 For example, a program would be at the beginning stage as it relates to resources if they do not have designated funds to support assessment; making progress if there are sufficient financial investment, technological support, physical facilities, and space to sustain assessment; and at the maturation stage if additional resources were provided for professional development of faculty and staff, such as consultations and workshops.^6,7

While this tool was originally designed to be used at an institutional level, we found that we were able to adapt the statements to align with our program assessment needs. The statements were straightforward, which supports use of the tool without significant training. Our team consisted of a small, established workgroup focused on the SP program, therefore we used the tool to guide discussion during meetings that occurred on approximately a bimonthly basis for six months. If the team or scope of the work to assess was larger, we recommend that individual stakeholders score each domain independently and provide evidence or their rationale for their assessment prior to engaging in team discussions.

Below we share our experience using the Higher Learning Commission’s Assessment Culture Matrix to guide our SP program CQI efforts while encouraging other pharmacy programs to adopt this tool for SP programs or in other areas of their assessment program.

Utilization of the Assessment Culture Matrix

Leadership

Administration’s support is critical, in order to effectively operate, assess, and improve educational programs. Programs with mature assessment practices related to administration have leadership who understand, value, and put resources towards assessment efforts.⁷

As our program evolved, we identified the need for an SP program director to oversee the day-to-day operations and assessment efforts. This stemmed from a small workgroup of five clinical faculty with experience working with SPs. This group used the Plan-Do-Study- Act (PDSA) model to conduct an initial evaluation of the program and subsequently identified that the clinical faculty alone did not have the capacity to carry forth comprehensive CQI efforts vital to further develop the SP program.⁵

With administrative leadership support, the College created a staff position focused on SP program development, administration, and evaluation. While creating and sustaining a culture of assessment is a team effort and requires buy-in from all stakeholders, finding a person dedicated to upholding the principles of CQI has helped our SP program maintain its commitment to assessment.

Shared mission and vision

It is important for programs to have clear statements about their mission and vision, including as it relates to assessment practices, in order to ensure that stakeholders are working towards a common goal.⁷ These documents should include faculty input and align with Institutional values and priorities.⁷

One of the first tasks that the new SP director undertook was to document a formal mission and vision statement for the SP program. This process took several months and was led by our SP program director with iterative feedback from the faculty who include SP activities in their courses. Through this process, we identified that our mission is to prepare students for clinical practice by providing safe and realistic training opportunities for students to practice patient care skills, such as history-taking, physical assessment, patient counseling, motivational interviewing, clinical decision-making, critical thinking, and verbal and non-verbal communication. The SP program helps prepare students for patient care interactions in their Introductory Pharmacy Practice Experiences (IPPEs) and Advanced Pharmacy Practice Experiences (APPEs). Our desire to provide quality learning experiences for our students drives our commitment to CQI and informs our assessments practices.

Faculty Engagement

Faculty have a wide range of responsibilities related to effective assessment practices, such as taking responsibility for ensuring that measures of student learning align with the mission and vision of the program, developing and maintaining the skills necessary to assess programs, and serving as advocates of assessment efforts.⁷ While some individual faculty may have a limited role in the assessment of programs, it is important that all faculty have an understanding of why assessments are conducted and the importance of using data to inform CQI efforts.

The primary strategy we use to engage faculty in SP program CQI efforts is via committees. We maintain an ad hoc committee that includes faculty representatives from each didactic year of the program, as well as the SP program director, working together to identify and support CQI efforts. This committee engages with other college committees, as needed, to ensure the SP activities align with the goals of the pharmacy program. The faculty are highly engaged, likely because this is a volunteer assignment, the work improves the SP activities embedded in their courses, and they have opportunities to engage in the scholarship of teaching and learning related to the CQI efforts.

Allocating Resources

Ensuring the appropriate resources have been allocated is imperative in developing a culture of assessment. Examples of resources include appropriate funds dedicated to assessment, technological support, knowledgeable personnel, and sufficient time.⁷

Within our pharmacy program, the SP program has an operating budget that includes a line item for focus groups, retreats, and key informant interviews. Allocating dedicated time ensures the SP program director is able to carry out assessment efforts on an ongoing basis to advance the SP program. Our program chose to hire a program director with experience and training in evaluation. This helped save the college time and money that would have been otherwise spent providing additional training to ensure the program director was equipped with the skills and knowledge conducive for defining, designing, and implementing assessment activities.

Access and Systematic Use of Assessment Data

Educational programs with mature assessment practices have faculty and administration who are committed to excellence in teaching and learning.⁷ They measure and celebrate student learning and routinely use evidence from assessments to conduct CQI efforts.⁷

Embracing CQI practices and embedding assessment efforts into everyday practices allow data to be collected, analyzed, and disseminated on a routine basis. Our first step in doing so included creating our SP program assessment plan. This assessment plan outlines critical evaluation domains (e.g., program management, training of SPs, SP classroom activities, and student learning outcomes), data collection strategies focused on streamlining program processes, enhancing student experience, and advancing student outcomes. Furthermore, our SP program prides itself on the transparency and open communication we have established with program stakeholders, including SPs, faculty, students, and administration. Our SP program uses its reporting structure and the colleges’ committee structure to not only disseminate data, but also to subsequently discuss the findings, in order to identify and prioritize areas for improvement. This in turn allows our program to make data driven decisions, set our priorities and goals, and allocate efforts appropriately.

Our program is also working to elevate our systematic use of assessment data and expand our data sources. Initial brainstorming sessions included incorporating outside data from our preceptors and the APPE rotations to understand students’ preparedness for ‘real world practice.’ Using this data will further inform our SP programming and allow us to adjust SP activities and areas of emphasis to better prepare students for their IPPEs, APPEs, and other patient experiences. We are also in the process of streamlining our CQI and data reporting efforts to establish regular reporting deadlines. Our next step is for the SP program director to create an annual report that summarizes CQI efforts, data yielded, and goal for the upcoming year that will be shared with the Curriculum and Assessment Committee and Department of Clinical Pharmacy.

Student Involvement

Student involvement is crucial in constructing a culture of assessment. Students should be knowledgeable about the assessments that occur within the program and have the opportunity to provide feedback about the program to faculty.⁷ Student leaders should have the opportunity to educate their peers about assessment practices and serve as members on assessment committees.⁷

Our SP program actively acquires ongoing student feedback, such as optional electronic surveys after interactions, to improve program practices. We use these data to improve the quality of our SP activities, including providing individualized feedback to each SP to aid their own professional development. Additional feedback is gathered from students during standing meetings, office hours, class meetings, and course evaluations. The SP program director documents the information in Excel spreadsheets that are organized by the role of the individual providing feedback (e.g., SP, student, faculty) and cohort during each academic year, in order to track the information over time. The SP program director also maintains an open-door policy and students are encouraged to schedule a meeting, if they have questions or concerns regarding the SP program.

Students are also active members of standing curriculum and assessment committees at the College of Pharmacy. Students bring an important perspective regarding the SP program that complements faculty, staff, and SP perspectives. As voting members of these committees, students help to make data-driven decisions to continuously improve the SP program and pharmacy program more broadly. We are seeking to further engage students in our SP program CQI efforts, such as by providing feedback about the realism of the cases and skills that are included in the activities. As students progress through the program and gain more pharmacy practice experience, they may be able to identify areas for expansion, such as additional therapeutic topics or clinical skills that could be included.

Next Steps

By using the Higher Learning Commission’s Assessment Culture Matrix, we have worked towards maturation in the domains of shared vision and mission, leadership, faculty engagement, and allocating resources. We have now shifted our focus towards strengthening our assessment efforts by continuing to collect data from students, SPs, staff, and faculty about our SP program, in order to make efficient and timely decisions at the level of the individual activity, course, semester, academic year, and pharmacy program. We are at the beginning stages of further engaging students in CQI through the Clinical Skills Committee.

Summary

The Higher Learning Commission’s Assessment Culture Matrix is an effective tool for guiding CQI efforts for a SP program. We encourage other pharmacy programs to utilize this matrix as they determine the strengths and areas of improvement for their SP program.