Using the Plan-Do-Study-Act (PDSA) Model for Continuous Quality Improvement of an Established Simulated Patient Program

Abstract

Objective

To describe the use of a continuous quality improvement process for strengthening our simulated patient (SP) program and the initial steps that have been implemented.

Innovation

A workgroup that included five clinical faculty with significant experience working with SPs and a strong interest in improving the SP program was developed. The Plan-Do-Study-Act model was used as it allowed for incremental quality improvement changes, in order to ensure a high-quality SP program designed to optimize student learning. Data were gathered from students, SPs, and faculty. Opportunities for improvement were prioritized based on anticipated benefits and available resources. Changes related to planning, implementation, and evaluation and feedback have been executed.

Critical Analysis

Changes related to planning that were implemented included developing handbooks for SPs, faculty, and graduate student instructors, as well as material for students in order to better describe the program. SPs are now referred to as “simulated” as opposed to “standardized” as part of a broader effort to clarify the purpose of SP interactions to students. Streamlined rubrics have been piloted, including electronic rubrics for first year students. SPs are being trained on fewer cases, in order to improve the training program. When possible, activities now take place in one large classroom instead of many small classrooms to improve oversight. Finally, additional feedback has been obtained from SPs via a retreat. These changes have been well received by students, SPs, and faculty.

Next Steps

The collection of this data and initial quality improvement changes provided a basis for hiring a full-time employee who will: dedicate 50% of their time to programmatic assessment of the SP program, support faculty with logistics and training, and be the face of our program to the students and SPs. Further, formal quantitative and qualitative assessment of the SP program has begun.

Introduction

To ensure achievement of desired educational outcomes, the Accreditation Council for Pharmacy Education (ACPE) explicitly requires formative and summative performance-based assessments of students.¹ One approach is through the incorporation of simulated or standardized patients (SPs). These are individuals who have received training to portray a real patient in order to simulate a set of symptoms or problems.² SPs are effective for teaching and assessing (formative and summative) knowledge and skills, and are used in pharmacy curricula for history-taking, physical assessment, clinical decision making, critical thinking, and communication.^{3, 4}

SP activities were incorporated over 20 years ago in this four-year, campus-based, public program with approximately 85 pharmacy students per class. Initially involved in only one course, SPs are now utilized in eight courses across all three years of the didactic curriculum. Currently, each student completes 25 SP interactions prior to beginning their Advanced Pharmacy Practice Experiences (Table 1). Activities involve one student interacting with an SP, either alone or with a small group of peers observing and offering feedback during the debrief.

Table 1. SP Activities by Semester During 2016-2017 Academic Year.

Year and approximate # students each year	Semester	Course	Total # interactions	Skill or topic emphasized (# interactions)
P1 n=85	Fall	Patient Communications	81	Empathic listening (n=1) Medication history (n=2) Motivational interviewing (n=2) Patient counseling (n=2)
	Winter	Pharmacy Practice Skills II	2	Health care professional interaction (n=2)
		Self-Care	6	Motivational interviewing (n=2) Self-treating patient triage and counseling (n=4)
P2 n=85	Fall	Therapeutic Problem Solving I	3	Anticoagulation assessment (n=1) ASCVD risk factors counseling (n=1) Healthcare professional interaction (n-1)
	Winter	Therapeutic Problem Solving II	1	Substance abuse assessment (n=1)
P3 n=85	Fall	Therapeutic Problem Solving III	1	Infectious disease medication counseling (n=1)
		Pharmacy Practice Skills III	3	Asthma/COPD counseling (n=1) Cardiovascular disease counseling (n=1) Diabetes counseling (n=1)
	Winter	Therapeutic Problem Solving IV	1	Solid organ transplant medication counseling (n=1)
		Total	25

SP: simulated patient; P1: first professional year; P2: second professional year; P3: third professional year

¹One interaction is in preparation for the end-of-semester exam and students role play the pharmacist while either obtaining a medication history, motivational interviewing, or patient counseling and observe their peers perform the other two skills

The first year of the curriculum primarily focuses on dispensing of medications, self-care medications, and an introduction to patient and health care professional communication. The second and third years focus on chronic and acute disease management, respectively. The SP interactions align with the content taught in skills and therapeutics courses.

Historically, a shared model existed on campus that allowed for centralized recruitment, vetting, training, and quality improvement processes with SPs. Several schools and colleges collaborated to manage the SP program, with no centralized University oversight. In 2015, one of the key schools restructured their curriculum and withdrew involvement, which led the College of Pharmacy to take ownership of the pharmacy SP program.

Statement of Innovation

While the advantages of using SPs are many, utilizing them for teaching and assessment can be challenging. A review of health professions education literature found that areas of focus for quality assurance of SP programs include SP training programs, structured feedback to students, inter-rater reliability, and observation and evaluation of SPs to improve their performance.⁵ In order to strengthen our SP program, we utilized the iterative four-step Plan-Do-Study-Act (PDSA) method to implement and assess small and incremental quality improvement changes.^{6, 7} While this approach has been used in a variety of contexts, we believe it is an innovative approach to improving the quality of our SP program.

The Innovation

The first step we took was to develop a faculty workgroup in fall 2016 that included five clinical faculty with a strong interest in improving the SP program. While course faculty had been attempting to make continuous improvements through their individual courses, a retreat with SPs in the summer of 2016 led to the associate department chair who oversees teaching to encourage the formation of this formal workgroup. The faculty included clinical assistant, associate, and full professors to ensure that the history of the program and current practices were represented. All participants had direct experience working with SPs in their respective courses. Recognizing that the role of SPs may vary across professional year, care was taken to include faculty with experience integrating SP activities into P1, P2, and P3 years. The Institutional Review Board determined that this project was not regulated.

Workgroup faculty met for four one-hour sessions in fall 2016 in order to identify the scope of the workgroup, discuss where existing data might be found, and to review data regarding the SP program. The workgroup brainstormed strengths and opportunities for improvement, while incorporating student, SP, and faculty feedback from the past academic year. Student perspectives were drawn from course evaluations (course evaluation response rate: 43-98%); SP feedback was drawn from a retreat, faculty-led debriefings, and training encounters; and faculty perspectives were drawn from the faculty workgroup focused on the SP program. Approximately equal weight was given to each perspective.

A variety of potential opportunities for improvement were identified during fall 2016, ranging from overarching programmatic concerns to potentially smaller planning, implementation, evaluation, and feedback issues. The opportunities for improvement were then prioritized based on anticipated benefits and available resources. The work was divided between smaller groups of faculty with the entire workgroup having the opportunity to provide feedback on the products that were created (e.g., faculty and SP handbook). Most work was completed via email in winter and summer 2017; however, there were two face-to-face workgroup meetings during that time. A summary of the issues identified and actions taken was created (Tables 2 and 3).

Table 2. Plan-Do-Study-Act Cycles for Faculty Issues.

Issue (Plan)	Do	Study	Act
Informally identified that there were opportunities for improvement in the SP program New faculty or established faculty with a new teaching assignment were ill-prepared to work with SPs Lack of unified approach to designing rubrics and preparing cases	Organized SP faculty workgroup Collaboration of five faculty with experience working with SPs Developed faculty and graduate student instructor handbook Created checklist of key steps, including requesting SPs, logistics, content of cases, and debriefing with students and SPs Created and piloted streamlined rubrics Collected all rubrics across the curriculum and categorized them by type of interaction Reviewed rubrics for variations in number of items (range 6 to 50), type of grading (e.g., scales, total points), and specific language used to indicate that an item was achieved Consolidated rubrics and standardized the number of items on rubrics Intentionally weighted communication and content components of the rubric for each professional year	Faculty successfully identified opportunities for improvement, tested and refined ideas Handbook developed and need identified for easily accessible and living document Increased consistency in rubrics and weighting of items on rubrics	Continue to have faculty workgroup but composition of members will change based on teaching responsibilities; composed manuscript about process Exploring page on internal College of Pharmacy website to house information. Continuing to develop plans for maintaining content New SP staff member will help to further streamline rubrics and ensure that individual criterion can be evaluated across the curriculum for the program and individual students; further work needed to standardize case preparation
Identified that student and program assessment data was not being collected	Transitioned to electronic rubrics Moved rubrics for P1 years into electronic software Collaborated with IT staff to train SPs on the use of electronic rubrics Used tool to track P1 performance Began gathering data in order to provide individualized quality improvement feedback to SPs	Electronic rubrics were implemented for P1 students without barriers	New SP staff member will help to roll out electronic rubrics for P2 and P3 courses
Limited time to train SPs and variability of training	Trained SPs on fewer cases Decreased from approximately 10 cases per activity to 1 or 3 cases per activity resulting in several SPs playing one case A positive consequence of this change is that frequently, each student will be able to play the pharmacist in one case and observe peers play all other cases	Number of cases were decreased and this allowed for more time to be spent training SPs on each individual case	Continue to monitor changes in the number of cases to ensure that there are no unintended consequences
Inter-faculty variability related to logistics created confusion among students and SPs	Placed SPs in one large classroom when possible Moved activities from individual smalls rooms on 5 levels of building to one large classroom in order to increase the amount of IT support	Change was generally well received by faculty, students, and SPs; however, noise can be a challenge	The long-term plan is to have small rooms dedicated to simulation activities in a new College of Pharmacy building

Table 3. Plan-Do-Study-Act Cycles for Student and Simulated Patient Issues.

Issue (Plan)	Do	Study	Act
Student Unrealistic student expectations and poor understanding of structure of SP program	Developed and presented materials for students to describe SP program Overarching goals of the SP program and specific interactions Set the stage for students regarding the SP role	Anecdotally, students appear to have a better understanding of the SP program and expectations	Thematic analysis of course evaluations may show if there have been improvements related to the SP program from the student perspective.
Dissatisfaction among students about SP portrayal of patients	Transitioned to term ‘simulated patient’ Utilized the term “simulated patient” to remind students that these activities are designed to replicate clinical practice and SPs will react to students based communication skills	No significant differences have been identified	New SP staff member provide additional training to SPs to further improve role portrayal
Inconsistency in what information as available to students prior to the interaction such as rubrics or basic scenario descriptions	Expectations by professional year were identified Faculty deliberatively communicated what information would be provided to students and the rationale prior to interactions (e.g., P1 students are generally provided the grading rubric and basic scenario descriptions whereas P3 students may not be provided with the grading rubric in order to more closely simulate clinical practice	Anecdotally, students appear to have a better understanding of expectations in individual courses	Thematic analysis of course evaluations may show if there have been improvements related to the SP program from the student perspective.
Simulated patient Newly hired SPs relied on established SPs to gain critical information about the program	Developed SP handbook University and SP program policies and procedures including related to substance use Logistics, such as parking, where to report, dress code, and compensation Expectations related to attendance and punctuality, professionalism, realism and staying in character, confidentiality, and how to report complaints or concerns	No significant differences have been identified	New SP staff member will help to further formalize existing policies and identify the need for new policies. We aim to add the handbook to a website for SPs
Multiple changes were implemented and feedback from SPs was critical	Conducted retreat with SPs 4 hour meeting with 1-2 faculty and approximately 20 SPs to discuss the SP program and obtain constructive feedback Topics were areas of concern, such as how to provide constructive feedback and the use of electronic rubrics	SPs have responded positively to providing feedback	New SP staff member will continue to periodically meet with SPs to provide education and gather feedback

SP: simulated patient; P1: first professional year; P2: second professional year; P3: third professional year; IT: information technology

There were several programmatic areas of concern that faculty believed needed additional resources to begin to address. For example, it was identified that there was a growing demand for administrative support to manage the logistics of running the SP program. Most faculty employing SPs had less than 50 percent effort towards teaching. Administrative support was needed to support SP recruitment, scheduling, room reservations, parking, and accommodations (e.g., due to limited mobility). The one support staff assigned to the SP program was not able to meet its demands due to the variety of other responsibilities in their position. There was an interest in hiring a staff member to help with the program, but data needed to be collected in order to justify the position.

Additionally, because our pool of SPs were predominantly older white women, designing roles outside this limited demographic resulted in: 1) having to adapt the role if an appropriate SP could not be retained for the activity or 2) requiring the SP to play an unrealistic role. For example, students identified that it did not feel like an authentic interaction when a 70-year-old woman played a 24-year-old pregnant woman. There were also concerns that students were not being exposed to diverse patient perspectives. While there was strong interest in recruiting a more diverse pool of SPs, faculty did not have the expertise or time available to recruit and train SPs.

Critical Analysis

Eleven major areas for improvement were identified by the faculty in the planning phase of this quality improvement, all of which went through at least one PDSA cycle during the 2016-2017 academic year. The changes that made the biggest immediate positive impact for faculty, students, and SPs were the creation of streamlined rubrics and implementation of electronic rubrics using ExamSoft (ExamSoft Worldwide, Inc., Dallas, TX and Delray Beach, FL). The areas of planning that continue to require the most refinement are recruitment of diverse SPs and data collection and analysis, in order to further evaluate the impact of the changes.

Several workgroup faculty had experience with quality improvement projects, including the principles of the PDSA method. This approach was useful as it allowed for identification and discussion of areas of improvement followed by small changes to be made and then informally evaluated in order to determine next steps. Some changes were made once (e.g., development of SP handbook) with the expectation that additional PDSA cycles would be needed in the future. There were some changes, such as those related to the implementation of electronic rubrics, which required multiple PDSA cycles in short succession. With each cycle, changes were tested with the goal of reaching a stable resolution, while recognizing that continuous quality improvement is needed, in order to ensure a strong SP program. The QI process has helped to justify additional staff support, create new resources, and plan for future improvements.

Discussion

Our experience suggests that the PDSA method is an effective continuous quality improvement process for strengthening an established SP program. This is important as a wide variety of health science program, including medicine, nursing, pharmacy, and dentistry have incorporated SPs to promote student learning and assessment.⁸ While considerable resources are required to start any type of new program, it is important to recognize that ongoing measures are needed to ensure that the program remains high quality.

Standards of best practice for SPs have recently been published around the areas of safety, case development, training, program management, and professional development.⁹ The principles and objectives provide a list of areas for potential quality improvement for existing SP programs. Given there are nearly 100 best practices, it is critical that programs select a realistic number and combination for which to focus on during the quality improvement process. Furthermore, while the PDSA cycle was used at this institution, alternative methods that may be considered depending on the goals of the quality improvement process include Six Sigma, Lean, Root Cause Analysis, and Failure Modes and Effects Analysis.¹⁰

Next Steps

While we have started to address many areas related to planning, implementation, and evaluation and feedback, we are beginning the process of addressing larger programmatic issues. This list outlines specific actions we plan to incorporate at our institution, and we feel other institutions would also benefit from careful consideration of the following: 1) recruiting a diverse SP pool, especially with regard to gender and race; 2) standardization of training and evaluation tools to ensure consistency across courses, which will allow for enhanced assessment and tracking of student performance across the curriculum; 3) providing additional support for faculty who are managing SP activities (e.g., creating student/SP schedules, room reservations, video recording setup, and other logistics); and 4) quality assurance of SPs to enable individual feedback to SPs and tracking of performance across courses.

Most importantly, in addition to the items summarized above, we have recently hired 0.5 FTE staff position to focus on programmatic assessment of our SP program. In addition to programmatic assessment, this individual will support faculty with logistics and training as well as be the face of our program for students and SPs. This individual is in the process of initiating focus groups with key stakeholders to ensure a clear understanding of the opportunities to optimize our program. Future quality assurance processes will more reliably track student and SP performance across courses to ensure use of this costly and important resource is utilized effectively.

Funding

None

Other Disclosures

None