A Novel Ambulatory Curriculum for Pulmonary and Critical Care Fellowship Training

Abstract

Background: Dedicated ambulatory training during pulmonary and critical care medicine (PCCM) fellowships is often limited. A novel 2-year longitudinal outpatient pulmonary fellowship curriculum was previously developed, piloted, and studied. The exportability and potential impact of this ambulatory curriculum on PCCM fellowship training nationally is not known.

Objective: We aim to understand the current state of ambulatory training in PCCM fellowships and the impact of a standardized outpatient curriculum on fellows’ ambulatory knowledge and competency.

Methods: Nineteen programs participated in the study from 2017 to 2019. Six programs received the first year of content, seven programs received the entire 2-year curriculum, and seven programs served as a control. Fellows, faculty, and program directors (PDs) completed a series of surveys assessing satisfaction with ambulatory education and the curriculum. Fellows completed a series of medical knowledge inventories, and programs submitted in-training exam scores.

Results: A total of 221 fellows (39%) and 17 PDs (89%) completed the precurriculum surveys, and 38 (12%) fellows and 10 (53%) PDs completed postcurriculum surveys. Before curriculum implementation, only 34.4% of fellows rated the quality of their ambulatory education as good or outstanding compared with 57.9% at the end of the study. Eighty-five percent of faculty and 89% of PDs rated the curriculum as good or excellent. Faculty believed that the teaching scripts were easy to use (78.4%), were factually accurate (86.3%), and provided high-yield information (82.1%). The majority of PDs indicated that the curriculum positively impacted patient care (78%) and fulfilled an unmet educational need (100%), and most planned to continue the curriculum after the study (78%). Feedback surrounded the need for updated content based on recently published guidelines and studies.

Conclusion: The curriculum is a standardized and feasible way to address a previously unmet need in PCCM fellowship education. PDs rated the curriculum highly and most plan to continue it in the future. Our limited data set suggests that the curriculum was well received by fellows and faculty and positively impacted perceptions of ambulatory education and preparedness for independent practice. Future study with a larger sample of fellows is needed to better understand the generalizability of these findings.

Upon graduation, pulmonary and critical care medicine (PCCM) fellowship trainees are expected to demonstrate the knowledge and skills necessary for independent ambulatory pulmonary practice. Accordingly, the Accreditation Council for Graduate Medical Education (ACGME) milestones and learning objectives for trainees in pulmonary medicine require fellows to practice evidence-based pulmonary medicine “across multiple health care settings” (1–4). Many of the required medical knowledge and patient care competencies are encountered only in outpatient care settings. However, the current PCCM fellowship training paradigm remains inpatient focused, as the ACGME mandates that only 7% of training be spent in the outpatient setting (1 half-day of clinic weekly for 30 months during a 36-month fellowship) (5). Thus, relying entirely on direct engagement with patients in a continuity clinic experience for all ambulatory education is likely insufficient preparation for independent practice after fellowship.

Internal medicine (IM) residency training programs have increasingly underscored the importance of ambulatory training. Accordingly, many supplement experiential learning in the ambulatory care setting with didactic conferences and case-based teaching (6). To meet these IM ambulatory needs, one training program developed a literature-based syllabus that covers more than 144 primary care topics (7, 8). Similarly, another IM training program developed an ambulatory curriculum consisting of 41 teaching modules. In total, more than 300 of the 500 IM residencies nationally subscribe to one of these curricula (9). Other asynchronous web-based outpatient curricula have also been used to improve the exportability of curricula to trainees who may be scattered across multiple geographic sites (10, 11). Similarly, outpatient curricula have also been developed, implemented, and evaluated across pediatric residency training programs (12). However, very few ambulatory curricula have been specifically developed for IM subspecialty fellowships. One program targeted outpatient learning for cardiology fellows focusing on a noon conference lecture series (13). Although outpatient pulmonary curricula have been published as a 14 topic “iBook” (14) and as four self-guided online ambulatory learning modules (15), there is currently no comprehensive standardized outpatient curriculum nationally available for PCCM fellowship trainees.

To meet the unfulfilled need, a structured case- and evidence-based outpatient fellowship curriculum was piloted at the Perelman School of Medicine at the University of Pennsylvania in 2014. After implementation, the fellows reported an improved sense of competency in outpatient pulmonary medicine and improved preparation for independent ambulatory postgraduate practice (16). However, the generalizability, exportability, and feasibility of such a curriculum are unknown.

In this study, we aimed to investigate the impact of an outpatient curriculum on PCCM fellows’ ambulatory knowledge and competency via a multiinstitution study. We also aimed to evaluate perceptions of PCCM fellows and program directors (PDs) regarding the current state of ambulatory training during PCCM fellowship. Finally, we assessed the PDs’ perspectives on the effectiveness and ease of delivery of the curriculum.

These data were previously presented at the Association of Pulmonary and Critical Care Medicine Program Directors (APCCMPD) meeting, which was held virtually on March 13, 2020. Two abstracts were accepted to the American Thoracic Society but were not ultimately presented because of the cancellation of the conference.

Methods

Subject Selection and Enrollment

To reach a broad audience, an open call to participate in the study was announced at the APCCMPD annual meeting in March 2017. Twenty-five programs initially volunteered to participate, and 19 programs were ultimately enrolled in the study. Enrolled programs were selected to represent a diverse group of fellowships with varied geographic locations, hospital affiliations (academic and community practices), and program sizes. All programs were accredited by the ACGME. Programs with pulmonary only or critical care only fellowships were excluded. Six programs were not enrolled, as they were not combined PCCM programs, ultimately decided they could not commit to the requirements of curriculum implementation, or were unable to complete a local institutional review board application in the requested time frame. Once enrolled, fellowship programs were randomly assigned to one of the following three groups: 1) control group (CG; did not receive the curriculum), 2) 1-year curricular group (1Y received only partial curriculum with Year 1 of curricular content beginning in July 2018), or 3) 2-year curricular group (2Y; received full 2 years of content beginning in July 2017).

Fellows at participating programs were invited, but not required, to participate in the study. Each program was asked to provide the study coordinators (S.K. and J.L.) with a list of current trainees at their institutions. Fellows were emailed and asked to notify the study team if they wished to opt out of the study. Regardless of their participation, all fellows were allowed to attend the new conference series. Participating fellows’ e-mail addresses were used as a point of contact, but their responses to all questionnaires were deidentified. To preserve trainee anonymity, subjects’ longitudinal survey responses and performance on knowledge questionnaires were not linked to one another. The study was reviewed by the University of Pennsylvania institutional review board and was deemed exempt from review. The study was also reviewed and approved by local participating institutions institutional review board according to local review board requirements.

Curriculum Design and Implementation

A novel, standardized, 12-month outpatient pulmonary fellowship curriculum was previously developed, piloted, and studied at the University of Pennsylvania in 2014 (8). The curriculum targeted disease states and clinical management areas that are not often seen in the inpatient setting and/or are focused on outpatient care. Subsequently, a more comprehensive 24-month curriculum, consisting of 40 core topics (Table 1), was developed. Conferences were designed for delivery in a small-group setting using a literature-based teaching script. These teaching scripts followed a standardized case-based format inspired by the previously published “Yale Office-Based Medicine Curriculum” (7, 8).

Table 1.

Summary of ambulatory curriculum topics

Year 1 Content Topics		Year 2 Content Topics*
COPD 1: inhaler overview	Idiopathic pulmonary fibrosis	Approach to pneumonia and pulmonary infections	Introduction to sarcoidosis
COPD 2: advanced pharmacologic therapies	Pulmonary hypertension	Cystic fibrosis	Post–lung transplant complications
Palliative care/management of dyspnea at end of life	Venous thromboembolic disease	Bronchiectasis	Tuberculosis and nontuberculosis mycobacterial infections
Pleural effusion	Asthma overview	Occupational lung disease	Pulmonary rehabilitation
Solitary pulmonary nodule	Lung transplantation: overview and when to refer	Post–solid and liquid transplant complications	Obstructive sleep apnea and obesity hypoventilation syndrome
Interstitial lung disease: initial evaluation	HIV-associated lung disease	Asbestos-related lung disease	Clinical applications of cardiopulmonary exercise testing
Tobacco cessation strategies	Billing and chart documentation overview	Vaccines and primary prevention in the clinic	Chronic respiratory failure/hypoventilation
Asthma during pregnancy	Neuromuscular respiratory disease	Lung cancer screening and overview of staging	Interstitial lung disease therapeutics
Approach to challenging patient interactions in the clinic	Chronic cough	Work-related asthma and reactive airways dysfunction syndrome	COPD 3: surgical approaches
Asthma mimics	Preoperative risk assessment	Oxygen delivery devices and inhaler technique overview	Cystic lung disease

Definition of abbreviations: COPD = chronic obstructive pulmonary disease; HIV = human immunodeficiency virus.

^*A total of 40 topics (20 per year) are including in the 2-year curriculum.

Each teaching script served as the basis for a 50–60-minute interactive conference facilitated by a faculty member. The goal of each session was to inspire open discussion and questions regarding practical outpatient management in a more casual, non–lecture-based setting. Each script included the following: learning objectives, clinical case vignettes, illustrative clinical questions with evidence-based answers, and references to key articles.

Institutions in the intervention arms were asked to cover the designated “Year 1” or “Year 2” content (20 topics each) over the course of 1 academic year. Programs in the control arm received their usual ambulatory content. All participating intervention programs were electronically provided with access to the conference teaching scripts. Scheduling of conferences and assignment of topics to individual faculty were at the discretion of participating programs. Thus, the order of conferences covered and the expertise of faculty on a given topic varied by institution. To standardize the methods of content delivery, participating programs were provided with a guide on best practices regarding curriculum implementation and conference structure. To understand the impact of the curriculum as written, faculty were discouraged from using Microsoft PowerPoint or other supplemental materials.

Baseline Needs Assessment and Curriculum Evaluation

PDs and fellows were surveyed in July 2017 (before the curriculum’s initiation at any site) and subsequently in June 2018 and June 2019. The PDs’ and fellows’ baseline 2017 needs assessments gauged impressions of the existing outpatient pulmonary fellowship experience and fellows’ self-perceived competency in a variety of clinical domains before curriculum implementation. PDs at institutions receiving the curriculum were asked to provide their overall impressions of the new curriculum and its impact on the ambulatory education of their trainees. Faculty who facilitated conference sessions were surveyed regarding curriculum content quality, estimated preparation time, and effort needed to deliver conference content.

Medical knowledge was assessed in a variety of ways. Before the initiation of the new curriculum, all fellows were asked to complete an electronic deidentified summative 40-question knowledge assessment. One question in the assessment was dedicated to each of the 40 topics covered in the curriculum. Questions were borrowed with permission from the Assessment in Critical Care and Pulmonology Self-Education and Evaluation of Knowledge Pulmonary Medicine question banks (17). The same summative knowledge assessment was administered at the completion of each academic year (June 2018 and June 2019).

In addition, a three-question session-specific pretest/posttest inventory was electronically administered immediately before and after each individual conference. The pretests/posttests contained case-based questions requiring application of the key learning objectives covered in a given session. The questions were developed by faculty experts who authored each individual teaching script. In addition, we also collected deidentified summative in-training exam (ITE) scores from all participating institutions from 2015 (2 years before the study) through 2019. ITE scores collected included average overall raw score, average pulmonary disease score, average general care score, and average practice skills score for each fellowship class.

All knowledge tests, pretest/posttest questionnaires, and surveys were administered electronically via the Qualtrics online electronic data management system hosted by the University of Pennsylvania.

Data Analysis

Nonparametric analysis of data collected from trainees, PDs, and faculty was performed. Questions with discrete answer choices were analyzed for the percentage of respondents indicating a particular response. For questions using a Likert scale for agreement with a particular statement, we calculated the percentage of respondents strongly agreeing or agreeing with a given statement. For questions that appeared on pretest and posttest assessments, comparative analysis was performed using χ² test, Fishers exact test, and related-samples Wilcoxon signed rank test depending on the number of responses to a particular question. P values ≤ 0.05 were considered to be statistically significant. Statistical analysis was performed using SPSS version 26. (IBM). Qualitative analysis of free-text comments was also performed.

Results

A total of 221 fellows (39% overall response rate; 38.5% first-years, 27.2% second-years, and 33.3% third-years) completed the precurriculum surveys. Characteristics of fellow respondents are outlined in Table E1 in the data supplement. PDs from 17 of the 19 (89%) programs completed the initial ambulatory curriculum needs assessment, and 140 fellows completed the precurriculum medical knowledge inventory (see Table 2 for participating fellowship program characteristics). Postcurriculum surveys were completed by 56 fellows (18.1%) in 2018 and 38 (12.2%) fellows at the end of the study in 2019. Nine of the 12 (75%) PDs in the 1Y and 2Y intervention cohorts completed postcurriculum surveys. Fifty-two(14.4%) faculty members who delivered the ambulatory conferences completed assessments of the sessions’ quality.

Table 2.

Participating fellowship training program characteristics based on program director survey responses

		n	Percentage
Total number of fellows in training at institution	1–5	1	5.9
	6–10	2	11.8
	11–15	7	41.2
	16–20	5	29.4
	21–25	2	11.8
Fellow future career aspirations	Academic medicine (research focused)	2	11.8
	Academic medicine (clinical focused)	10	58.8
	Private practice	5	29.4
	Military	0	0.0
	Industry/pharmaceuticals	0	0.0
	Other	0	0.0
Primary ambulatory clinic site	Academic affiliated community-based hospital	2	11.8
	Community practice	0	0.0
	County hospital	1	5.9
	Large tertiary care academic medical center	13	76.5
	Private practice	0	0.0
	VA medical center	0	0.0
	Other	1	5.9
Geographic location	Northeast	5	29.4
	Southeast	4	23.5
	Midwest	4	23.5
	Southwest	1	5.9
	Pacific/West Coast	3	17.6

Definition of abbreviation: VA = Veterans Affairs.

Responses represent 17 of the 19 programs. Two program directors did not respond to the initial precurriculum survey.

Precurriculum Needs Assessment

Although most fellows (77.4%) anticipated that ambulatory pulmonology would be a substantial part of their future practice, only 34.4% rated the quality of their ambulatory education as good or outstanding on the precurriculum assessment. A similar number (34.3%) rated their ambulatory education as fair or poor. Ambulatory teaching preintervention was characterized by faculty precepting in clinic (93.9%), self-directed learning (75.3%), didactics (18.6%), and small-group teaching (8.8%). Of the 47.1% of programs that had ambulatory teaching beyond clinic precepting, 50% noted that these additional sessions occurred only a few times per year.

The majority of fellows (73.9%) believed that they would benefit from additional ambulatory teaching. Accordingly, only 47% of all fellows and 55% of third-years believed that their training was adequately preparing them for independent practice. Self-reported confidence in managing patients in the outpatient setting was only 60% compared with 72.6% in inpatient consultation and 79% in critical care. Among third-year fellows, these numbers remained disparate, with confidence reported by 77.2% in outpatient management, 89.1% in inpatient consultation, and 94.5% in critical care. Detailed findings of the baseline needs assessment are outlined in Tables E2–E4.

Impressions were similar among PDs, with only 47.1% noting satisfaction with their program’s current ambulatory education and 82.4% noting too little focus on ambulatory teaching overall. Free-text comments focused on the desire for a dedicated ambulatory curriculum, including didactic-based content.

Fellow Perceptions of Curricular Impact

Before curriculum implementation, 34.4% of all fellows rated their ambulatory education as good or outstanding compared with 68.8% in the 2Y intervention (P = 0.001), 57.1% in the 1Y intervention, and 42.9% in the CG at the end of the study (P = 0.001). The reported confidence in managing outpatients improved the most in the 2Y intervention after curriculum (60% to >93.8%). Confidence in managing patients in the intensive care unit was more consistent across all groups (from 78.8% to 87.5% in the 2Y group) (see Figure 1).

Figure 1.

Fellow comfort with care provision by care setting and study cohort represented as the percentage of fellows agreeing or strongly agreeing that they feel comfortable. Data represented show that of the precurriculum baseline (2017). Data from the control group, 1-year curriculum group (received first year of curricular content beginning in 2018), and 2-year group (received full 2 years of curricular content beginning in 2017) are from after the curriculum in 2019. ICU = intensive care unit.

Fellow impressions of their ambulatory education were most favorable in the 2Y group compared with the precurriculum group and CG (Figure 2). Statistical significance could not be calculated because of the small sample size. This was true for perceptions of strength of ambulatory knowledge, confidence in managing an outpatient panel, preparation for independent ambulatory practice, and perception that faculty were actively engaged in ambulatory teaching. In contrast, there was no marked difference in the desire for more ambulatory teaching or patient care exposure between the subgroups.

Figure 2.

Fellow impressions regarding their amb education by study cohort. The bars represent the percentage of fellows who strongly agree or agree with the listed statement on the x axis. Data represented show that of the precurriculum baseline (2017). Data from control group, 1-year curriculum group (received first year of curricular content beginning in 2018), and 2-year group (received full 2 years of curricular content beginning in 2017) are from after the curriculum in 2019. amb = ambulatory.

Fellows who completed 2 years of the curriculum in the 2Y group demonstrated more comfort in managing the disease states assessed compared with those in the precurriculum group and CG (Figure E1). Notably, there was marked improvement in interstitial lung disease, idiopathic pulmonary fibrosis, human immunodeficiency virus–associated lung disease, and lung transplant referral domains, which may be disease entities with limited exposure during fellowships, depending on the training program. The topics in which the 1Y group did not show improvement compared with the 2Y group were those not included in Year 1 curricular content.

Faculty Perceptions of Curricular Impact

Of the 52 faculty facilitators who completed the curriculum surveys, the majority believed that the teaching scripts were easy to use (78.4%), high yield (82.1%), and factually accurate (86.3%). Importantly, 92% noted that fellows were actively engaged, and 88.2% were willing to lead another conference in the future. Individual teaching scripts were rated as good or excellent by 84.9% of faculty facilitators (see Table E5).

PDs had similarly positive impressions, with 78% noting that the conferences positively impacted patient care and 100% noting that the curriculum fulfilled an unmet educational need. Eighty-nine percent of PDs agreed that it was easy to recruit faculty to teach the sessions, and 78% plan to continue the curriculum after the study (see Table 3). Overall, the curriculum was rated as good or excellent by 88.1% of PDs. PDs pointed out the need for regularly updating content on the basis of recently published guidelines and studies.

Table 3.

Program director impressions of the curriculum in 2019

	Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree
Teaching scripts were factually accurate, %	0.0	0.0	11.1	22.2	66.7
Teaching scripts were well written, %	0.0	0.0	11.1	44.4	44.4
Provided high-yield content, %	0.0	0.0	11.1	22.2	66.7
Faculty impressions of curriculum were positive, %	0.0	0.0	0.0	55.6	44.4
Fellow impressions of curriculum were positive, %	0.0	0.0	0.0	44.4	55.6
Upper year fellows regularly attended, %	0.0	11.1	0.0	44.4	44.4
Curriculum met an unmet need in the program, %	0.0	0.0	0.0	55.6	44.4
Easy to recruit faculty, %	0.0	11.1	0.0	44.4	44.4
Likely to continue conferences after study, %	0.0	11.1	11.1	0.0	77.8
Conferences positively impacted ambulatory patient care, %	0.0	0.0	11.1	33.3	44.4

N = 9 (of the eligible 12 programs who were in the intervention arm).

Impact on Fellow Medical Knowledge

The average precurriculum summative knowledge assessment score was 42.3% correct. A total of 38 fellows (11 in the 2Y group) completed the knowledge assessment in 2018. The average score of the 11 2Y group fellows was 39.1% compared with 40.0% for the 27 fellows who had no curriculum at that time. Unfortunately, only one fellow completed the summative knowledge assessment in 2019, so we could not make any significant conclusions using these data.

We also assessed medical knowledge using pretests and posttests for each session. An average of 28 fellows completed each of the 16 pretests, and 17 completed each of the posttests. The average pooled score for 36 postsession knowledge assessments improved by 15.2% compared with session pretests. Regardless of the study cohort, the average ITE scores decreased slightly for all years (see Table E6).

Discussion

This study is the first multiinstitution evaluation of ambulatory education across PCCM fellowship programs. The precurriculum needs assessment demonstrates a desire for more robust ambulatory training by both fellows and PDs. Although the majority of fellows surveyed were likely to have a future ambulatory clinical practice, less than half of fellows felt prepared to practice ambulatory pulmonology independently. The relatively small percentage of fellows with a favorable impression of their outpatient education suggests that current ACGME-mandated outpatient clinical time is likely insufficient to meet ambulatory education needs. The desire for more robust ambulatory training was shared by the PDs, the overwhelming majority of whom agreed that there is currently too little time spent on ambulatory education in PCCM fellowship.

Through the dissemination of a standardized ambulatory curriculum, we provided programs with accessible, high-yield content that was implemented across institutions with relative ease. The feasibility and sustainability of implementation at other institutions on a broader scale is supported by the ease of facilitator recruitment, plans to continue the curriculum after the study at most institutions, and high ratings of the overall curriculum. One major challenge we faced was the low response rates to our surveys, which limits our ability to make conclusions about faculty and fellow impressions of the curriculum. Within the small number of faculty facilitator respondents, there was an overall favorable impression of the curriculum. In addition, the limited data from fellows in Years 2 and 3 suggest that this curriculum might be an effective tool for improving fellow confidence and perceived preparedness for independent practice. Fellows’ perceptions of their ambulatory education, core ambulatory knowledge, and confidence in managing ambulatory care issues were all better in the 2Y group compared with the control and precurriculum cohorts. However, the generalizability and reproducibility of these findings across training programs remains uncertain given our small sample size and should be the focus of further study.

Unfortunately, we were unable to make any definitive conclusions regarding curricular impact on trainee medical knowledge because of the low response rate to both the summative and presession/postsession medical knowledge inventories. Observed improvement on pretest/posttest scores in certain subspecialty areas (e.g., lung transplant and interstitial lung disease) compared with other more general pulmonary knowledge topics suggests that the curriculum may have utility in addressing education gaps that may result from lack of local expert availability. Although there was noted improvement on pooled postsession test scores, we were not able to track changes in any individual score to protect trainee identity. Although we cannot fully explain the observed trend in slightly decrease ITE scores across all cohorts, we suspect that there may be external factors in training unrelated to this study. Future study will focus on detailed evaluation of the curriculum on trainee knowledge. Although this is the first longitudinal, multiinstitutional study to explore the impact of a novel ambulatory curriculum on PCCM fellowship training, we acknowledge that the low survey response rate, particularly on the postcurriculum inventories, limits the generalizability of the study and the conclusions we can reach regarding its impact on fellow knowledge and confidence in ambulatory care. Although we were able to recruit diverse PCCM training programs to participate, these programs may represent programs that were more in need of ambulatory education than the average training program nationally, and the majority of the participants were at large academic medical centers. To protect trainee identity, data were pooled rather than tracked on an individual basis, making it difficult to identify the absolute impact of the curriculum at the individual level. Finally, although the study was conducted over 2 years, we cannot make conclusions regarding the long-term impact of the curriculum on ambulatory knowledge or changes in direct patient care.

Based on our findings, there is a need for increased ambulatory training in PCCM fellowship. Through the dissemination of a standardized ambulatory curriculum, we provided programs with a feasible, sustainable, and efficient means to incorporate more robust ambulatory education. Our limited data set suggests that the curriculum was well received by fellows and faculty and positively impacted perceptions of ambulatory education and preparedness for independent practice. Future study is needed to better understand the generalizability of these findings, particularly with respect to curricular impact on fellow confidence, medical knowledge, and patient care.