Swimming With Sharks: Teaching Residents Value-Based Medicine and Quality Improvement Through Resident-Pitched Projects

Abstract

Background

To create meaningful quality improvement (QI) curricula for graduate medical education (GME) trainees, institutions strive to improve coordination of QI curricula with hospital improvement infrastructure.

Objective

We created a curriculum to teach residents about QI and value-based medicine (VBM) and assessed curricular effectiveness.

Methods

We designed a 2-week required curriculum for internal medicine residents at a large academic program. After participating in basic skills workshops, trainees developed QI/VBM project ideas with faculty and nonclinical support and pitched them to hospital leaders at the end of the rotation. Pre-post and 1-year follow-up surveys were conducted for residents to self-assess knowledge, attitudes, and skills, participation in QI/VBM projects, and career intentions. We tracked QI/VBM project implementation.

Results

In the first 2 years (2017–2018), 92 trainees participated, and 71 of 76 (93%) recommended the curriculum. Surveys (76 of 92, 83%) show improvement in our learning objectives (12%–60% pre to 62%–97% post; P < .001 for all; Cohen's d effect size 0.7–1.2), which are sustained at 1-year follow-up (57%–95%; P < .01). Four of 19 projects have been implemented. At 1 year, 95% of residents had presented a quality/value poster presentation, 44% were involved in QI/VBM beyond required rotations, and 26% plan to pursue careers focused on improving quality, safety, or value.

Conclusions

Our project-based curriculum culminating in a project pitch to hospital leadership was acceptable to GME trainees, improved self-assessed skills sustained at 1 year, and resulted in successfully implemented QI/VBM projects.

What was known and gap

Curricular innovation is needed to bridge the gap between hospital investment in value-based medicine and graduate medical education quality improvement training.

What is new

A project-based curriculum focused on quality and value that culminates in a Shark Tank-style project pitch to hospital leadership.

Limitations

Evaluation surveys were not tested for validity and measured self-assessed knowledge and skills.

Bottom line

This 2-week curriculum resulted in improvements in self-assessed knowledge and skills and comfort with the topics that were sustained at 1 year.

Introduction

As the cost of health care continues to skyrocket, academic medical centers are increasingly investing in value-based medicine (VBM) as a means of increasing quality while decreasing costs.^1,2 Meanwhile, the Accreditation Council for Graduate Medical Education mandates that residents receive training in systems-based practice and practice-based learning and improvement.^3,4 The Clinical Learning Environment Review expects experiential learning in health care quality and patient safety (PS) and gives feedback to institutions to improve graduate medical education (GME) training.^5,6 Although national efforts, most notably the Veterans Affairs (VA) Chief Residents in Quality and Patient Safety program and the High Value Practice Academic Alliance, have made strides to bridge the gap between hospital investment in VBM and GME quality improvement (QI) training, curricular innovation is still needed.^7,8

While didactic-based rotations are a common setting to teach health care value and cost-effectiveness, a particularly engaging and lasting way to teach QI and VBM may be through creating and implementing projects to reinforce and cultivate proficiency.^9,10 Inspired by the television show Shark Tank, several institutions have undertaken hosting venture capital–style pitch competitions to generate QI projects and educational innovation.^11–13 The format of the show lends itself to problem identification and value proposition articulation, which translates naturally into teaching about health care quality and value. None of these efforts at adapting a Shark Tank–style competition to hospital QI or GME education, to our knowledge, address the gap between GME training and resident involvement in project idea generation as front-line workers.^7,14

We hypothesized that creating a new project-based curriculum focused on quality and value that culminates in a project pitch to hospital leadership would (1) increase resident confidence using essential QI tools and institutional data to solve systems-based value challenges, and (2) increase the likelihood of resident participation in future quality and value projects.

Methods

Setting and Participants

New York University School of Medicine's internal medicine residency program is a large university-based academic program that participates in the VA Chief Residents in Quality and Patient Safety program. Trainees rotate at 3 sites: a university hospital, a municipal referral hospital, and a VA hospital. Our program uses a “4 + 2” schedule (4 weeks inpatient + 2 weeks outpatient clinic). During postgraduate year (PGY) 2, residents have a mandatory 2-week quality improvement/patient safety (QI/PS) rotation during one of their outpatient blocks with 40 hours of dedicated nonclinical time over 2 weeks along with required ambulatory clinic. We repeated the curriculum in 3 consecutive blocks (over 6 weeks) each year with one-third of the residents participating in each block. During the first year (2017), only categorical residents participated in this curriculum due to scheduling limitations. During the second year (2018), all categorical and research/fast-track residents participated, as well as half of our primary care residents who chose to use elective time to participate. Faculty advisors were associate program directors or site directors within our residency program and did not receive any compensation for participation.

Intervention

After identifying learning objectives and performing a gap analysis to determine educational needs, we developed the final curriculum, which included 3 hours of interactive introductory Lean training, with a focus on charter construction, process mapping, problem identification, and solution design; and 6 hours of in-person lectures delivered by quality, safety, and value leaders from our institution, emphasizing institutional priorities and case studies to illustrate concepts (Table 1). In addition, residents applied these principles in a transitions-of-care exercise that included reviewing 3 of their own readmissions to determine contributing systems causes. The curriculum aligns trainee educational goals with existing institutional VBM and QI/PS initiatives as identified by residency program leadership.

Table 1.

Components of the Curriculum

Topic	Time	Format	Faculty Expertise
Introductory Lean Training	3 hours	Interactive with mix of short didactic lessons and practical small group exercises	Lean “Black Belt” (nonphysician)
Introduction to the Rotation/Choosing a Problem	1 hour	Case-based/local example-based lecture	Chief resident in quality and safety
What Is Value-Based Medicine?	1 hour	Case-based/local example-based lecture	VBM/hospital medicine
Metrics, Dashboards, and Health Informatics	2 hours	Case-based/local example-based lecture	Health informatics
Case Study in Systems Thinking: Antibiotic Stewardship	1 hour	Case-based/local example-based lecture	Antibiotic stewardship
History of Patient Safety and High Reliability Organization Principles	1 hour	Case-based/local example-based lecture	Patient safety
Transitions of Care	3 hours	Lecture (1 hour); guided self-reflection/chart review activity (1 hour); small group discussion and debriefing (1 hour)	Hospital medicine
Project Time	∼25 hours (15–20 protected)	Small group	Chief resident; program/site directors
Shark Tank Pitches	3 hours	Learner presentations with feedback	Hospital leadership

Abbreviation: VBM, value-based medicine.

For the remainder of the time (15 to 20 hours), residents designed a value-based QI project pitch (sample schedule provided as online supplemental material). Residents were divided into teams of 4 to 5 residents per group with 3 or 4 teams per 2-week block. Supported by the chief resident and a faculty advisor, each team was tasked with picking a process they identified as inefficient, unsafe, or low value. Each resident team submitted initial ideas and received feedback (idea form provided as online supplemental material).

Residents then worked with hospital VBM project managers (nonclinical project management staff) to use local data to analyze the opportunity for improvement, design potential solutions, and pitch a compelling argument for their idea. In the first year, VBM managers helped residents identify and request specific data, including financial data, to be extracted by data analysts. In the second year, residents had access to Slicer Dicer, a proprietary Epic module (Epic Systems, Verona, WI), which allowed rich exploration of our own data. We also shifted the focus away from specific monetary savings to generalizations about the value proposition. Residents still had the option of requesting additional data from the VBM managers, and several teams did. In both years, we had a “VBM team office hours” session (nicknamed the “Nurse Shark Tank”), in which VBM team members helped each team frame their problem, identify areas where they needed to understand the process better, or collect/request data to support their identification of the problem.

Teams were encouraged to observe the process, interview stakeholders (eg, nurses, laboratory staff, physical therapists, pharmacists), and create a process map and Ishikawa diagram of the current state.¹⁵ They identified potential solutions primarily from published literature and then worked with multidisciplinary stakeholders to choose the ones most likely to be successful. Teams were given specific deadlines for deliverables and guidance about how much time to spend on each activity during protected team time, with flexibility allowed for each team to use the time as needed to advance their project under the guidance of the chief resident and their faculty advisor.

The curriculum culminated in a 2-hour formal “Shark Tank” event on the final day of the 2-week rotation. Each resident team had 15 minutes to present their pitch followed by 15 minutes of questions and comments from the “Sharks.” The “Sharks” were senior clinical and administrative leadership, including the chief medical officer, chief quality officer, chief of medicine, and clinical lead of value-based management. They scored pitches on projected value, impact, feasibility, creativity, and alignment with departmental and institutional goals (Table 2). Each team received focused oral feedback on their ideas, methods, and presentations. At the end of the 2-week curriculum, all teams were given the opportunity to work with a faculty advisor to implement their project during the subsequent academic year. The highest-priority projects as identified by senior leadership received additional administrative and project management support. Some residents worked on implementation of their project ideas during protected time in their mandatory PGY-3 QI/PS rotation; others received coverage for clinical activities to participate in project design meetings and executive-level presentations. For successfully implemented projects, residents handed off project management to clinical and operational staff.

Table 2.

Project Scoring

Category	Description	Points (0–5, 5 = best)
Originality/creativity	The idea is innovative and exciting.
Ease of implementation	The idea will not be too difficult to study or implement.
Cost of implementation	The idea will not cost a tremendous amount to implement. The return on investment is worthwhile.
Timeline	The idea has an acceptable timeline for implementation and will not take multiple years to complete.
Projected impact	The idea will provide significant qualitative or quantitative benefit and the submission demonstrates the measurement of this impact.
Vertically coordinated	This idea is in line with value-based medicine goals of the department of medicine.
Feasibility	The idea can be accomplished by trainees within the course of their training and will not require resources and skills not available to medicine trainees.

Outcomes

We conducted retrospective, pre-post, perception-based paper surveys at the end of the 2-week curriculum and follow-up surveys among participants 1 year after completion of the curriculum. The 15-item survey of 4-point Likert-type scales to self-assess knowledge, attitudes, and skills was developed by the authors (clinician educators with QI/PS expertise) without further testing. Along with that survey, we asked graduating residents about their involvement in quality, safety, and value projects during residency and their career intentions. We also tracked resident presentations of QI/VBM projects and how many of the project proposals were actually completed.

The New York University School of Medicine deemed this project exempt from Institutional Review Board review.

Analysis

Likert scales were compressed to positive (very likely/likely) and negative (very unlikely/unlikely) for pre-post comparison (provided as online supplemental material). Pre-post results were compared using the related-sample sign test as the surveys were paired by participant. Precurriculum survey results were compared with 1-year follow-up survey results using Pearson's chi-square test (unpaired analysis). Effect sizes were estimated using Cohen's d for the precurriculum to postcurriculum comparison. Correction for multiple testing was not performed. There were no significant differences when data were analyzed separately by year (not shown), so results from the 2 years were combined. All analyses were conducted using SPSS Statistics 24 (IBM Corp, Armonk, NY).

Results

In the first 2 years of the curriculum (2017–2018), 92 residents participated in 19 teams, resulting in 19 project pitches (Table 3). Of all second-year residents, 81% (43 of 53) participated in 2017 and 92% (49 of 53) participated in 2018. Among the participants, 83% (76 of 92) consented and responded to the survey. Overall, 94% (60 of 64) recommended the course. There were significant improvements in resident self-assessed knowledge, confidence levels, and comfort with QI and value learning objectives after the curriculum (provided as online supplemental material). For example, after the curriculum, participants reported an increase in their ability to identify unsafe or inefficient processes in the hospital (56% to 96%, P < .001, Cohen's d = 0.85). They felt more comfortable in their abilities to use process mapping (18% to 86%; P < .001; Cohen's d = 1.20) and principles of Lean management to propose solutions (16% to 64%; P < .001; Cohen's d = 1.14). More participants reported being likely or highly likely to participate in quality, safety, and value projects (25% to 70%, P < .001, Cohen's d = 1.08) and to suggest quality, safety, and value proposals to hospital leadership (12% to 65%; P < .001; Cohen's d = 1.25).

Table 3.

“Shark Tank” Project Ideas

Season 1: 2017
XPERT-TB Testinga  Rule out tuberculosis in 12 hours instead of 72 hours using GeneXpert machine
L-TAPP Initiativea  Primary medicine team does inpatient bedside procedures instead of referring to interventional radiology
Team Panca  Reduce inappropriate early computed tomography use in acute pancreatitis
Glycemic Controlb  Shorten time between glucose check and insulin administration
BLUD  Reduce inappropriate repeated inpatient blood cultures
Cardiac Rehab  Increase referral to cardiac rehab after myocardial infarction
Procalcitonin  Differentiate viral/bacterial infections and shorten empiric antibiotics
Project RELISH  Residents follow subspecialty patients in discharge follow-up clinic
DBN-DP Initiative  Hire discharge planner to increase safety and efficiency of discharges and protect resident learning during rounds
Season 2: 2018
WON'T Drinka Wasted Opportunities for Naltrexone Treatment of Drinking
Lightening the Loada  Opioid-induced constipation
Where is the Love(nox)?  Safer, cheaper DVT prophylaxis
ADVOCATEb  Advanced Directives at Veterans Outpatient Clinic and Trainee Education
ICU Acquired Weakness Prevention  Supine cycle ergometer
S.E.E. the Lyte  Smart, Evidence-based Electrolytes
Drop the Drip  Treat low-risk pulmonary embolism without heparin drips
FAST Initiative  Furosemide Appropriate Starting Therapy
SiTS Down!  Selective Testing in Syncope Diagnosis
Delirium Prevention Project  Early identification and intervention for patients at risk for in-hospital delirium

Abbreviations: DVT, deep vein thrombosis; ICU, intensive care unit.

^aSelected for extra support for implementation.

^bImplemented without extra support.

Of the 43 participants from the initial year of the curriculum, 39 (91%) completed a 1-year follow-up survey. Improvements were sustained at 1-year follow-up for all survey questions (provided as online supplemental material). For example, at 1-year follow-up, the percentage of residents who felt comfortable or very comfortable creating a solution to a process in the hospital or clinic that was unsafe, inefficient, or costly was sustained (25% pre; 78% post; 78% follow-up; P < .001). Similarly, the percentage of residents who reported they were likely or very likely to collaborate with nonclinical personnel to analyze costs and opportunities of potential interventions was also sustained (17% pre; 79% post; 78% follow-up; P < .001). The same percentage of residents saw quality, safety, and value projects as part of their scholarly activity after the curriculum and at 1-year follow-up (42% pre; 84% post; 84% follow-up; P < .001 for pre/post; P = .001 for 1-year follow-up).

Of the participants, 95% (41 of 43) had at least 1 poster presentation accepted to our internal Quality and Safety Day. Seventeen of 39 (44%) survey respondents reported that they were involved in QI/VBM projects beyond their required rotations, of which 10 had resulted in presentations at national meetings prior to completion of residency training. After this curriculum, several residents applied and were selected for the High Value Practice Academic Alliance Future Leaders Program for advanced VBM training. Ten of 39 residents (26%) responded that they were very likely to consider a career that focused on improving quality, safety, and value. This curriculum has helped the institution identify leaders within the residency class, several of whom have been hired to stay on as faculty.

To facilitate dissemination, chief residents leading the curriculum identified several key elements (derived from participant feedback) that are necessary to successfully implement this type of curriculum. These include putting residents in charge, providing mentorship/advising, encouraging vertical alignment within the institution, supporting opportunity identification with local data and published best practices, and providing the tools for success through skills-based workshops (Figure).

Figure.

Keys for Successful Resident Pitches

Note: Chief residents and program directors identified elements necessary for successful resident pitches: (1) putting residents in charge of choosing a topic and executing the pitch; (2) providing mentorship/advising from clinical and nonclinical leaders; (3) encouraging vertical alignment with department priorities and ongoing projects; (4) supporting data-driven decisions by providing residents with tools to access local data and experts to help them identify where additional data are needed; and (5) teaching residents the key concepts and tools they need through workshops and training (eg, Lean).

Four of the 9 pitches from the first iteration in 2017 received institutional support, resulting in hospital improvements under the guidance of faculty advisors (Table 3). In 2018, following the second iteration, 2 of 10 projects were categorized as high priority and are in various stages of implementation.

Discussion

In this QI/VBM curriculum, in which internal medicine residents developed specific project pitches delivered to institutional leaders, self-assessment surveys showed that experiences were highly acceptable and increased perceived knowledge, skills, and comfort in these topics. These perceptions were sustained at 1-year follow-up. Successful pitches resulted in actual institutional changes.

Others have shown that experiential participation in QI projects during residency improves knowledge and engagement.^16,17 One institution published their efforts to incorporate games into teaching QI with success improving resident QI knowledge and confidence.¹⁰ Our curriculum, which culminates in a gameshow-style event, builds on this by having residents develop real project pitches with the potential to improve value for patients and the hospital. Our sustained results at 1 year suggest that it may not be essential for residents themselves to complete projects for the curriculum to change their attitudes and perceived skills in these areas.

The excitement generated by this endeavor resulted in an overwhelming interest among our hospital leadership. We had more leaders who wanted to be “Sharks” than we could accommodate, and the institution began exploring how to expand this program to other residency programs and beyond GME to include advanced practice providers. We have tested the feasibility at other sites by piloting an expansion at an affiliated community hospital with support from a chief resident and hospitalist advisor with promising initial results.

Our findings may not generalize to other settings, particularly those without institutional support and faculty leaders in quality and value education. As our survey was developed without additional validity evidence, respondents may not have interpreted questions as intended. In addition, the surveys measured self-assessed knowledge and skills, and may not reflect actual competencies. Although the survey response rates were generally high, some residents may find the curriculum less useful or unattractive.

Further research might examine whether this curriculum can be replicated in other settings, perhaps through the VA Chief Residents in Quality and Patient Safety program, with other specialties, or with nonphysicians such as advanced care practitioners, nurses, and pharmacists. Other considerations include whether the curriculum is more effective in a 2-week intensive format or spread out over half-day sessions throughout the academic year.¹⁶

Conclusions

This 2-week curriculum to teach QI/VBM to internal medicine residents through didactics, exercise, and project pitches to institutional leaders resulted in improvements in self-assessed knowledge and skills and comfort with the topics. These improvements were sustained at 1 year. Successful resident-pitched projects were implemented and produced institutional improvements as well as academic presentations. The curriculum appears to improve resident attitudes toward future participation in quality, safety, and value projects.