Developing a Practice-Based Learning and Improvement Curriculum for an Academic General Surgery Residency

Abstract

Background

Program Directors in Surgery are now facing the challenge of incorporating the ACGME's practice-based learning and improvement (PBLI) competency into residency curriculum. We introduced a comprehensive PBLI experience for PG2 residents designed to integrate specific competency goals (quality improvement, clinical thinking, and self-directed learning) within the context of residents’ clinical practice.

Study Design

Fourteen PG2 residents participated in a three-week PBLI curriculum consisting of three components: Complex Clinical Decision Making (CCDM), Individual Learning Plan, and Quality Improvement (QI). To assess how effectively the curriculum addressed these three competencies, residents rated their understanding of PBLI by answering a 12-question written survey given pre- and post-rotation. Resident satisfaction was assessed through standard post-rotation evaluations.

Results

Analysis of the pre and post rotation surveys from the fourteen participants showed an increase in all measured elements, including knowledge of PBLI (p<0.001), ability to assess learning needs (p<0.001) and set learning goals (p<0.001), understanding of QI concepts (p=0.001), and experience with QI projects (p<0.001). Fourteen QI projects were developed. Although many residents found the creation of measurable learning goals to be challenging, the process of identifying strengths and weaknesses enhanced the resident's self-understanding, and contributed to overall satisfaction with the rotation.

Conclusions

The initial implementation of our PBLI curriculum demonstrated that residents report personal progress in their clinical decision making, self-directed learning, and familiarity with quality improvement. This comprehensive PBLI curriculum was accepted by surgical residents as a valuable part of their training. We are encouraged to continue a clinically-grounded PBLI experience for PG2 residents.

INTRODUCTION

Surgical residents experience a vastly different education than surgeons trained only 10 years ago. Prior eras initially emphasized learning from master surgeons’ expertise with increasing emphasis on textbooks and experiences in the operating room, hospital wards, and clinic as surgical training itself matured. In this environment, the adequacy of a resident's education was judged primarily by the training process: completing the requisite number of training years, performing a specific number of procedures, and ultimately demonstrating a mastery of medical knowledge by passing standardized board exams. Today, however, residency training is in a state of transition. The emphasis is now on measuring performance outcomes, specifically in the six core competencies as described by the Accreditation Council for Graduate Medical Education (ACGME): patient care, medical knowledge, professionalism, interpersonal and communication skills, practice-based learning and improvement (PBLI), and systems-based practice. It is no longer sufficient for trainees to have merely participated in their residency's training activities revolving around diagnosis and management of disease. Rather, the current generation of surgeons will need to objectively demonstrate their ability to perform comprehensive care while optimizing communication, defining work objectives in relationship to the goals of the medical care organization, and maintaining high standards of professional conduct.

This paradigm shift is challenging residency programs, both medical and surgical, to reframe, redesign, and, in many instances, recreate their curricula to facilitate this new competency- and outcome-driven agenda. PBLI, in particular, has often required crafting new and targeted activities for trainees, as education specific to self-assessment and self-directed learning has typically not been an explicit part of traditional residency training. The goals of PBLI, as described by the ACGME, are action-oriented: residents should “identify strengths, deficiencies, and limits in one's knowledge and expertise”, “set learning and improvement goals”, and “systematically analyze practice using quality improvement methods, and implement changes with the goal of practice improvement”, among others.¹ However, operationalizing these objectives has proven quite challenging, as residency programs have attempted to create both meaningful educational opportunities and reliable methods for measuring individual performance outcomes in PBLI.

Published reports from a variety of university and community residency settings have often focused on quality improvement (QI) as the primary tool for achieving competency in PBLI.^2-6 The benefits gained by both organizations and residents through resident participation in QI initiatives can be substantial: improved patient care outcomes, resident learning and professional development, and resident/faculty/other team member engagement and satisfaction.⁷ However, PBLI also encompasses other important concepts, such as self-assessment and personal improvement through achievement of learning goals, which are rarely addressed through these reported curricula. In addition, many programs have chosen to implement their PBLI curriculum during research rotations due to time constraints. This may limit the curriculum's relevance by artificially abstracting the critical thinking, self-directed learning and reflection purposes of PBLI from the clinical setting within which physicians are to develop and apply these skills.

We present our experience with the introduction of a comprehensive PBLI curriculum – which incorporates clinical decision making, self-assessment and learning goals, and quality improvement – within the context of a clinical rotation.

METHODS

The General Surgery Residency Program at the University of Wisconsin graduates six chief residents per year, with clinical and operative experiences provided at four hospitals: the University of Wisconsin Hospital and Clinics, the American Family Children's Hospital, the William S. Middleton Memorial Veterans Hospital, and Meriter Hospital. We introduced a new three to four week problem-based learning and improvement (PBLI) curriculum in Fall 2007 for PGY2 residents. During this time, residents were engaged in a low-intensity clinical rotation and participated in three PBLI components: Complex Clinical Decision Making, Individual Learning Plan, and Quality Improvement. Descriptions of the goals and expectations for each component (Table 1), as well as pertinent forms, readings, and other resources, were made available on the department intranet. The clinical rotation was different for the first cohort of PGY-2 residents (2007-2008, Nutrition) and the second cohort of PGY-2 residents (2008-2009, Endoscopy) due to changing service and curriculum needs. However, residents maintained the same level of PBLI-relevant activity for a comparable amount of time within the context of both clinical rotations. Between November 2007 and June 2009, fourteen PG2 residents completed this curriculum (12 general surgery and 2 integrated plastic surgery residents). The Department of Surgery has been granted an exemption from the University of Wisconsin Institutional Review Board to conduct research in established or commonly accepted education setting, involving normal educational practices such as this PBLI curriculum.

Table 1.

Goals & Objectives for PBLI Curriculum

1. Demonstrate an ability to critically assess clinical thinking and decision making. This is measured by faculty review of resident's written analysis of two clinical decisions.

2. Demonstrate an ability to assess learning needs and develop an individual learning plan (ILP). This is measured by faculty meeting with resident to ensure that ILP is specific, measurable, achievable, relevant and time-based. Faculty also meets with resident to assess progress and again to assess whether resident achieved ILP.

3. Demonstrate knowledge of the principal components of quality improvement (QI) and the skill to conceive and implement a QI project. This is measured by faculty evaluation of resident's QI project.

Complex Clinical Decision Making

Residents were asked to identify two clinical scenarios from their own experience, and to write an analysis of the factors (clinical, situational, system, psychological) which shaped their decisions. Residents were asked to use the heuristic developed by de Cossart and Fish to frame their analysis of clinical decision making: clinical reasoning, clinical solutions/options, deliberation (personal professional judgment and practical wisdom leading to a professional judgment), and wise action.⁸ These analyses were submitted to the residency program director for review and discussion of the decision-making process.

Individual Learning Plan

The University of Wisconsin Department of Surgery developed a web-based individual learning plan (ILP) in which residents self-identified strengths and weaknesses within their clinical practice and developed a six-month plan to address specific learning needs. To begin, residents described their long-term career goals and identified training they might need beyond their general residency education to attain these goals. They next ranked 12 personal attributes from strongest to weakest, and self-rated their skill level from “novice” to “competent” on 22 criteria drawn from the six ACGME competencies (Table 2). Based on resident responses, the web-based ILP program generated a list of resident's strengths and weaknesses in both the personal attributes and competency criteria. Using this list, residents developed two learning objectives and created a strategy to meet these objectives over the next six months. Residents were instructed in the SMART (Specific, Measurable, Achievable, Relevant, Time-Based) rubric for setting goals. The residency program manager and residency program director met with residents to refine learning objectives and strategies in order to make them pertinent, achievable and measurable. Further meetings were held three months after creation of the ILP to assess progress, and after six months to assess whether the resident had met the learning objectives.

Table 2.

Individual Learning Plan (example elements)

Self-Assessment of Personal and Professional Attributes	General Surgery Resident Self-Assessment (examples)
Think about your personal and professional character. Rank the following attributes in order (1 = your strongest attribute; 12 = your weakest attribute)	Please rate your performance in the following competencies along the spectrum from Novice to Competent (adapted from H.L. Dreyfus's model of skills acquisition).
ability to assess yourself	Patient Care
ability to recognize limitations to knowledge and skills	patient management
ability to work with others	skill in surgical techniques
attention to detail	Medical Knowledge
confidence	knowledge of surgical diseases, their processes and indications
equanimity/composure under pressure	knowledge of relevant literature
honesty/accountability/response to error	Practice-Based Medicine
perseverance	ability to critique your patient care
response to feedback/criticism	ability to convert needed information into answerable & pertinent questions
responsibility/sense of duty	Systems-Based Practice
sensitivity to diverse patients and populations	cost-conscious use of diagnostic & therapeutic technologies
time management	utilization of hospital system and services to get things done
	Interpersonal & Communication Skills
	leadership skills
	patient & family counseling
	Professionalism
	compassion, integrity and respect for others
	high standards of ethical behavior
	commitment to continuity of patient care

Quality Improvement

The University of Wisconsin Hospital and Clinics has an active Quality Resources department, which provided an introductory lecture to each resident, describing the principles and methods behind quality improvement (QI) implementation. Residents were then asked to choose a QI project, either by selecting from a prepared list, or by identifying an issue from their own clinical experience. Eligible topics included those which focused on improving surgical outcomes or an aspect of the residency program (e.g., patient hand offs, specific training need, etc). The resident then met with two hospital QI staff and the residency program manager to discuss the project, assess its feasibility, troubleshoot potential obstacles, and identify methods to measure improvement. To support their efforts in designing and implementing the project using the Plan-Do-Study-Act (PDSA) model, residents completed several readings and attended meetings with the Surgical Quality Improvement Committee and the Institutional Review Board. Residents also met with the faculty National Surgical Quality Improvement Project (NSQIP) director to discuss the purpose and use of NSQIP data in surgery-specific quality improvement efforts. As the rotation progressed, residents presented their project goals and progress report or results to the surgery department faculty and residents at a department conference.

Assessment

Residents completed a 12-question written survey both at the beginning and at the end of the rotation. The survey asked participants to rate their understanding of PBLI, ability to assess their complex clinical decision making (CCDM) and learning needs, and their experience with QI (Table 3). Paired t-tests were used to test for differences in pre- and post-rotation survey scores. Analyses were performed using SAS 8.02 software (SAS Institute, Cary, North Carolina). All tests of significance used 2-sided p-values at the 0.05 level. Resident satisfaction was assessed through standard post-rotation evaluations.

Table 3.

Pre and Post Rotation Assessment Survey Questions

1. How much do you know about practice-based learning & improvement (PBLI)?

■ Absolutely nothing

■ I have heard of it, but nothing more.

■ I have heard of it and discussed it before.

■ I have heard of it and understand the basic terminology and concepts, although I could not teach it to others.

■ I have heard of it, understand the terminology and concepts, could give a lecture to residents and faculty about PBLI.

(7-pt Likert scale from “Poor” to “Excellent”)

2. How would you rate your ability to assess your clinical decision-making?

3. How would you rate your ability to assess your learning needs (your strengths and deficiencies)?

4. How would you rate your ability to set learning and improvement goals with clear objectives, defined timeframe and a specific learning plan?

5. How would you rate your ability to achieve learning and improvement goals you set for yourself?

6. How much do you know about quality improvement (QI)?

■ Absolutely nothing

■ I have heard of it, but nothing more.

■ I have heard of it and discussed it before.

■ I have heard of it and understand the basic terminology and concepts, although I could not teach it to others.

■ I have heard of it, understand the terminology and concepts, could give a lecture to residents and faculty about QI.

7. What kinds of experiences have you had with QI?

■ Absolutely none

■ I have attended a meeting (not a lecture) discussing quality improvement.

■ I have been a passive part of a QI team (i.e., not actively involved in the planning or decision making.)

■ I have been an active part of a QI team (i.e., involved in the planning and decision making).

(5-point Likert scale from “Strongly Agree” to “Strongly Disagree”

8. I believe I am able to develop and implement a QI project.

9. I would like to participate in a project if it helped improve patient care or improve the residency program.

(free response)

10. List the six ACGME competencies.

11. Who are the customers of a residency program?

12. Give a clinical example of QI in health care.

RESULTS

Pre- and post-rotation assessment survey mean scores from the fourteen participating residents are presented in Table 4. Residents reported a significant improvement in measured elements spanning domains of general PBLI concepts, self-assessed learning needs, and quality improvement. Lack of significant change was observed in questions 9 (interest in participating in QI projects) and 10 (list of the six ACGME competencies).

Table 4.

Pre and Post Rotation Assessment Survey Results (n=14)^*

Question #	Domain	Score (Mean)	Post-Test Score (Mean)	Difference in Means	95% CI	p-value
1	PBLI	2.64	4.14	1.50	0.83, 2.17	<0.001
2	CCDM	4.21	5.29	1.07	0.24, 1.90	0.015
3	Learning Needs	4.57	5.57	1.00	0.55, 1.45	<0.001
4	Learning Needs	3.79	5.50	1.71	1.10, 2.33	<0.001
5	Learning Needs	4.14	4.86	0.71	0.06, 1.37	0.035
6	QI	3.00	4.14	1.14	0.55, 1.74	0.001
7	QI	1.64	3.93	2.29	1.76, 2.81	<0.001
8	QI	4.07	4.57	0.50	0.12, 0.88	0.013
9	QI	4.57	4.79	0.22	-0.03, 0.46	0.082
10	ACGME Competencies	2.71	3.57	0.86	-0.04, 1.76	0.061

^*Scoring scales are as follows: Questions 1, 6, 8 & 9, 1-5 points; Questions 2-5, 1-7 points; Question 7, 1-4 points; Question 10, 0-6 points. All questions were coded such that high point values represented best responses.

As reported in the standard post-rotation evaluation tool, residents thought highly of the rotation, giving it an overall score of 3.23 on a 0-4 scale. The most favorable elements of the rotation were the teaching quality (3.38 on 0-4 scale) and level of attending contact (3.46 on 0-4 scale).

Fourteen QI projects have been initiated to date. Topics, addressing a range of patient care and residency program issues, are listed in Table 5.

Table 5.

Quality Improvement Project Topics

Investigate factors related to increased risk for post-operative superficial incisional surgical site infection (as reported in NSQIP data), and develop measures to reduce rates among general surgery patients.

Study ways to reduce nurse calls to residents on home call.

Create a data-collection system to log and track all major and minor surgical complications to increase rate of reporting, enhance the educational value of Morbidity & Mortality conference, and support analysis of trends, particularly in comparison with local NSQIP complication rates.

Develop an anticoagulation protocol for trauma patients.

Study approaches to minimize leaks following laparoscopic gastric bypass.

Determine appropriate use of CT scans for facial injuries.

Assess whether stress ulcer prophylaxis is prescribed in accordance with hospital guidelines and accepted treatment practices, and if not, propose/promote an appropriate treatment algorithm

Investigate ways to improve communication between referring physicians and vascular surgeons regarding post-operative management of patient.

Reduce postoperative pain, nausea and vomiting in the outpatient surgery patients by assessing the use of electroacustimulation.

Develop a chest tube protocol for traumatic hemo-pneumothorax.

Determine if intra-operative pathological diagnosis changes surgical outcome.

Decrease turnaround time of ORs.

Improve patient hand-offs between residents.

Create a catalogue of teaching topics and tips for residents to improve teaching of medical students.

DISCUSSION

We present our experience with implementing a multi-faceted PBLI curriculum at an academic surgical residency program, with the goals of creating an opportunity for resident reflection on their own learning needs and goals, their individual clinical decision-making and practice, and improvement needs of the systems surrounding their education and practice. As a result, our residents were able to engage not only in quality improvement activities, but to address other crucial domains of the PBLI competency as well. In particular, residents developed skills in self-assessment and creation of learning goals through the Individual Learning Plan, and participated in critical evaluation of their own practice through the Complex Clinical Decision Making activities. Residents reported improvement in these skill areas, as measured by the pre and post-rotation surveys. These personal practice components of PBLI are a critical part of this competency, and are not directly addressed by other educational efforts in our department. The emphasis on self-assessment, with the opportunity for honest feedback on one's performance, was welcomed by the participating residents. The residents reported that direct contact with faculty, particularly the residency program director, was a highlight of the rotation, and were grateful for designated time for reflection and feedback in the midst of an otherwise-hectic PG2 year.

In addition to these personal growth elements, residents were highly engaged in both didactic and experiential training in quality improvement. Although the curricula included interactions with hospital and university resources, project topics themselves were typically generated from the residents’ own clinical experiences. This “bottom-up” approach, as described in the ACGME 90-Day Project publication, may give residents greater ownership over problems which they identify as important, as well as encourage continued engagement in the process by focusing on projects of unique interest to the participating resident.⁷ However, this QI approach also brings accompanying challenges. Institutional buy-in is often essential to the success of QI projects. Although the institution generously provided two staff from Quality Resources to assist residents on their QI projects, some residents found themselves struggling to develop appropriate and needed alliances within the hospital and/or university to obtain resources or other needed support. For example, a project to create a more efficient triage system of paging residents on home call required coordination with and commitment from nurse managers and other administrators to even begin collecting preliminary data on the frequency and type of home calls. Being able to build a team with a shared perception of the problem and commitment to improvement is essential to achieve process change, yet this crucial skill set was not always obvious to residents as they selected their projects. In addition, a lack of understanding of their proposal's compatibility with the larger goals and interests of the institution sometimes impeded residents’ implementation of particularly daring or challenging QI projects, thus leading to a lack of successful progress. An alternate approach is, therefore, to engage residents in QI through “top-down” efforts, i.e. inviting their participation on initiatives generated at the level of the institution.⁷ While this may eliminate some of the immediate clinical relevance seen in “bottom-up” efforts, it allows residents to be involved in projects with pre-existing teams, broader significance, and more secure financial backing, while also providing important educational opportunities in the administrative aspects of hospital-level management of clinical needs. Overall, the frustrations met by our residents when attempting to engage administrators in a project of significant clinical relevance may reflect a trend in the quality operations at many academic medical centers, which may focus on data accuracy rather actual measures of quality. This experience may, in fact, challenge residents and other clinicians to rethink the methodology and approaches to quality improvement in their institutions.

Time constraints are a frequent challenge when attempting to involve residents in QI efforts.^{2-5, 7} Clinical and educational responsibilities, coupled with work hour restrictions, make a surgical resident a difficult player to capture. Residents tended to choose QI projects that required long-term planning, implementation and assessment. Although residents were told that they could either choose to continue the QI project after the rotation or hand it off to another resident doing the rotation, all residents chose to continue working on the QI project after their time on the rotation. As a result, while our residents report increased knowledge about QI and even increased interest in participation in such projects, the QI projects created through this curriculum remain in various states of implementation. However, we have identified some key features for enhanced success in the future. Faculty mentorship, as mentioned by other authors, is essential to these projects, both through didactic teaching of QI methodology and concepts, and through development of clinically relevant and realistic targets for improvement.^{2, 7} As faculty involvement in QI efforts becomes important for their own maintenance of certification, we hope to attract a dedicated core of faculty mentors for these projects. We are also considering ways to sustain projects beyond the three to six week rotation block, including involving multiple residents on various phases of project design and implementation throughout the year. Other authors have described team-based approaches, in which residents work together on topics selected through discussion with faculty and other interested parties.^{2, 9} These shared responsibilities would lend yet another educational opportunity to this curriculum, and might help address some of the time constraints that our residents face.

Finally, assessment of educational endeavors is simultaneously essential and difficult. We have utilized both our standard post-rotation evaluation survey and a targeted set of questions answered both pre and post rotation, in order to quantify the impact of this rotation on meeting the goals of the PBLI competency. We were pleased to note self-reported improvement across all domains of the curriculum, and that residents have generally enjoyed their experiences with this curriculum. We plan to track survey scores across groups of residents over the next few years, allowing a richer longitudinal assessment of this curriculum over time. Current objective assessment of the residents’ QI performance, i.e. through graded evaluation of the project development and implementation as described at other institutions, is limited in our curriculum.² However, we are investigating ways to provide more objective feedback on project quality, in order to make the QI experience more long-lasting and beneficial for the residents. Some medical schools have utilized interactive web-based programs to provide training in clinical decision-making, with the potential for quantitative assessment.¹⁰ However, there remain few applications of tools for quantitative evaluation of problem-based learning, through such methods as OSCE or simulation lab.

Despite the challenges encountered, we have found our curriculum to successfully enhance knowledge, skills, and interest in key elements of PBLI. Our curriculum has two unique and important features: (1) it is a multi-faceted approach to PBLI (involving experiential and didactic training in self-directed learning and assessment, critical reflection and analysis of clinical decision making, and quality improvement) and (2) it integrates the resident's learning of essential PBLI principles into the resident's clinical practice, thereby literally making the learning practice-based.¹¹ This provides residents with the immediate opportunity to apply their didactic education to their clinical practice. In addition, it allows all residents to participate in the curriculum as required by the ACGME, in contrast to other programs who present their PBLI curricula only during research years.² Although perhaps more subject to some of the time constraint issues mentioned above, we feel that the benefits of the clinical context are substantial, and have received resident feedback indicating that the competency training is well-timed within the residency program.

CONCLUSION

PBLI elements, including quality improvement, clinical decision making, and self-assessment of learning needs, can be presented to PG2 residents in a comprehensive, clinically-based curriculum. The incorporation of these activities into a clinical rotation provides important conceptual and educational context, and importantly ensures that all residents will participate. Our curriculum produced an increase in knowledge, skills, and interest in the objectives of PBLI, with high levels of resident satisfaction. We are encouraged to continue providing this curriculum to our residents, and look forward to further opportunities for enhancement and improvement.