Decay in Physiologic Knowledge since Medical School among Critical Care Fellows (DIP Study)

Abstract

Background

Rapid accumulation of knowledge and skills by trainees in the intensive care unit assumes prior mastery of clinically relevant core physiology concepts. However, for many fellows, their foundational physiology knowledge was acquired years earlier during their preclinical medical curricula and variably reinforced during the remainder of their undergraduate and graduate medical training.

Objective

We sought to assess the retention of clinically relevant pulmonary physiology knowledge among pulmonary and critical care medicine (PCCM) and critical care medicine (CCM) fellows.

Methods

A composite examination was developed from an initial set of questions used in preclinical pulmonary physiology courses at four separate medical schools. These questions passed through multiple rounds of review by various educators to arrive at a set of 15 multiple-choice questions. The test was administered to incoming first-year PCCM and CCM fellows at seven institutions during their 2021 fellowship orientation.

Results

Forty-one first-year PCCM (n = 24) and CCM (n = 17) fellows completed the examination, and the proportion correct among the fellows was compared with that of medical students for each item. Although all questions were deemed to be clinically relevant, preclinical medical students significantly outperformed the incoming fellows.

Conclusion

These findings suggest considerable decay of clinically relevant pulmonary physiologic knowledge during residency training and point to a need for longitudinal retrieval practice to reinforce these concepts during the course of medical school clerkship years and postgraduate clinical training as well as consideration of dedicated pulmonary physiology curricula for PCCM and CCM fellowship programs.

A considerable portion of preclinical medical student education is centered on rapidly acquiring and consolidating a large volume of knowledge related to physiology. Educators frequently assume that students and trainees learn and, more importantly, retain foundational concepts on which they subsequently build a more complex understanding of pathophysiology to enhance their approach to medicine (1, 2). However, even for learners training in specialties in which an understanding of physiology is thought to be essential, there are limited opportunities during residency to review the core physiologic concepts. By the time trainees reach critical care fellowship, they are expected to readily access the foundational knowledge, gain a deeper understanding, and apply physiologic concepts to complex clinical scenarios.

The literature is mixed with respect to the strength of long-term retention of foundational knowledge (3). In the 1930s, Leslie Barrett Cole coined the term “disuse atrophy” to describe the loss of basic science knowledge when students have moved from the classroom to the clinical realm (4). In subsequent years, many critics of medical pedagogy lamented this seemingly inevitable decay in knowledge (5). Hermann Ebbinghaus, a psychologist and philosopher in the late 1800s, described an exponential decay in memory when there is no additional effort made to retain information (6). His findings suggest that new memory is halved in a matter of days to weeks unless the material is actively reviewed to combat the loss. Focusing on retention of Spanish language vocabulary learned in high school, Bahrick retested his learners over a 50-year period and ultimately proposed a triphasic retention curve, beginning with a rapid period of forgetfulness in the first 6 years, followed by a prolonged period between 25 and 30 years in which the retention curve is flat. The third phase is characterized by another period of further gradual knowledge decay, probably related to the age of the subjects. Bahrick referred to the knowledge retained over the prolonged flat period as the “permastore,” which is durable and stable over the long term (7). Initially deeper learning is more likely to enter the permastore and potentially endure for decades. Therefore, the task at hand is for educators to ensure that core physiological principles persist beyond the initial period of rapid forgetfulness and become encoded in the learner’s permastore.

Within the domain of medical education, Custers and ten Cate analyzed the performance of medical students and physicians on a basic science examination. They found that the participants demonstrated only moderate knowledge attrition, as physicians who had been in practice for several years achieved approximately 75% of the scores of students still in medical school (8). When they looked at “unrehearsed” knowledge between the groups, they found minimal knowledge decay in the first 24-month period after the most recent exposure to the material; however, following that interval, the exponential decay described by Ebbinghaus was again demonstrated.

Regarding physiology-specific knowledge, a study among medical students did not demonstrate greater knowledge decay with an increasing retention interval (9). The authors suggest that this was likely attributable to the high applicability of physiology to clinical practice (relative to other basic sciences learned in medical school), as well as frequent physiology-based questioning in the early clinical training period. In another study of medical students by D’Eon, knowledge attrition was demonstrated for physiology, although it was significantly less than the rate of attrition for other topics such as neuroanatomy (10).

Although these studies provide useful insights into the issues of retention, the medical education literature has not evaluated the attrition of foundational physiology knowledge among trainees practicing in their intended fields (11). For critical care fellows, an understanding of pulmonary physiology and pathophysiology is invaluable, as they are expected to possess a level of understanding of core pulmonary physiology on which they can base their approach to complex bedside scenarios and synthesize clinical data to guide decision-making. However, we hypothesize that clinically relevant physiology knowledge gained in medical school wanes over time spent in the clinical undergraduate and graduate training arenas. Therefore, we sought to better characterize the knowledge decay of pulmonary physiology among a group of critical care trainees by developing and administering a curated set of clinically relevant questions from preclinical medical school examinations to pulmonary and critical care medicine (PCCM) and CCM fellows.

Some of the results of this study have been previously reported in the form of an abstract (12).

Methods

Assessment Development

There is no gold standard for the assessment of pulmonary physiology knowledge. As such, the investigators developed a practical evaluation tool during the 2020–2021 academic year and administered it to incoming first-year fellows in July 2021.

Three primary investigators (S.I.M., M.A., and B.W.L.) solicited actual multiple-choice question (MCQ) examinations from four U.S. allopathic medical schools’ foundational courses administered in 2020 that covered the preclinical pulmonary physiology and pathophysiology content. A modified Delphi method was used to establish consensus on a streamlined assessment that combined items from these four sites (13, 14). For each MCQ item, the investigators considered the question, “How relevant is this question to the day-to-day clinical practice of an incoming PCCM or CCM fellow?”

In the first round, each item from these examinations was reviewed independently by the three primary investigators (S.I.M., M.A., and B.W.L.) and organized thematically until all items were categorized. Topics not applicable to critical care were excluded (e.g., therapeutics of outpatient pulmonary disease). We also excluded nonpulmonary critical care physiology domains (e.g., pathophysiology of shock) because we focused exclusively on pulmonary physiology. The investigators convened to discuss the themes they each identified and agreed on a set list of four primary categories with several subcategories (Table 1). The investigators then individually reviewed their categorization of the questions and made adjustments to align all the items with the four primary agreed-upon categories. The questions were then ranked as follows: top tier, a high-quality question that effectively tests the understanding of a concept thought to be highly relevant to an incoming PCCM or CCM fellow; middle tier, a medium-quality question or one of moderate relevance; and lower tier, a lower-quality question or one of less relevance. Questions in the top tier were required to test material that not only could be memorized but rather synthesized in the courses.

Table 1.

Domains and subdomains of assessment items

Oxygenation

A-a O2 difference

Diffusion

Hemoglobin–O2 dissociation curve

Respiratory failure

VA/Q mismatch/shunt

Ventilation and carbon dioxide

CO2 content

CO2 equation

Control of respiration

Dead space

Hemodynamics and gas transport

O2 delivery, O2 transport, O2 content

Pulmonary vascular physiology

Mixed venous O2 saturation (SVO2)

Pulmonary mechanics

Compliance

Pulmonary function testing

Resistance

Reynolds number

Surfactant, LaPlace’s equation

Tau, autoPEEP

Definition of abbreviations: A-a = Alveolar-arterial; autoPEEP = auto-positive end-expiratory pressure; S_VO₂ = mixed venous oxygen saturation; VA/Q =  Ventilation-perfusion.

In the second round, the investigators met to adjudicate their rankings and identify their top 20 questions while balancing the variety of questions and representation of different institutions. Questions identified as top tier by two or more of the three investigators were retained, whereas questions that none of the investigators viewed as top tier were discarded. Questions that only one investigator ranked as top tier were reviewed by the investigators during a subsequent round. Criteria for selection included clarity of language, clinical relevance, quality of the answer choices, and whether the content overlapped with other questions. The investigators reflected on the quality of the questions not only from their own institutions but of the entire compilation of questions to select questions that demonstrated applied physiology concepts with clinical relevance.

In the third round, the investigators individually reviewed the draft examination to assess its cohesiveness and the breadth of topics covered. Through this round, two questions were identified that had excessive overlap, prompting one final review to identify replacement questions. Ultimately, the investigators identified 16 items that reflected the desired content without duplication.

To lessen the outsized influence and bias of the three primary authors who performed the first round, in the fourth round, the draft examination was sent to three pulmonary student course directors (A.M.L., J.P., and N.G.S.) who had provided the original questions to review the newly created examination with attention to the quality of the questions and clinical relevance for PCCM and CCM fellows. They were asked to score each question on a Likert scale from 1 to 5 for combined quality and relevance. If the course directors scored the question as a 3 or less, they were asked to comment on why it warranted a lower score (based on intrinsic quality of the item’s language vs. its clinical relevance). The student course directors were also invited to suggest alternative questions (from the original question bank) for consideration by the group if they believed a new question highlighted a topic in a clearer way or tested an important topic that was not previously emphasized. The entire group of investigators then met to further adjudicate the questions and ultimately agreed on the final set of 30 questions that satisfied the quality and clinical relevance requirements. All 30 questions received a consensus score of 4 or 5.

For the final round, four PCCM or CCM fellowship program directors (J.A., K.H., D.B.J., C.J.W.) were invited to review the 30 items and rank them into tertiles based on clinical relevance to PCCM and CCM fellows in the intensive care unit (ICU). The 15 highest ranked questions were used as the final assessment tool. The 15-item examination was piloted on a group of graduating fellows at two programs in June 2021. This pilot testing did not result in any changes to the examination’s content or structure.

Subjects

The final 15-item assessment was administered to 41 incoming first-year PCCM and CCM fellows at seven institutions during their July 2021 orientation. Fellows had completed residency training in a variety of specialties, including internal medicine (n = 31), emergency medicine (n = 4), combined internal medicine/emergency medicine (n = 2), neurology (n = 3), and anesthesia (n = 1). There was no explicit pulmonary physiology educational intervention before testing. This educational study was reviewed and deemed exempt by the institutional review boards at each of the institutions where fellows participated.

Statistical Analysis

Because the questions were answered by different groups of medical students from different schools, we chose first to compare student versus fellow performance for each individual question, rather than in aggregate for the entire test. Therefore, the primary outcome of interest was the performance of incoming fellows on each of the 15 questions assessing knowledge of core concepts in pulmonary physiology relative to the performance of medical students. The comparator student examination scores are those of the 665 medical students who took the original physiology courses and examinations in 2020 at the four institutions of the coinvestigators who contributed questions to the composite assessment. Subgroup analyses were then performed to determine the performance of fellows versus students on content related to specific domains within pulmonary physiology: 1) oxygenation, 2) ventilation and carbon dioxide, 3) hemodynamics and gas transport, and 4) pulmonary mechanics. χ² tests or Fisher exact tests were used to compare the performance of fellows and students on individual questions. Logistic regression was used to compare the performance of fellows and students within specific content domains. Stata/SE 16.0 (StataCorp) and SAS version 9.4 (IBM Corp.) were used for the statistical analyses. A two-sided P value of less than 0.05 was considered significant.

Results

Forty-one incoming first-year fellows, 24 training in PCCM and 17 in CCM, completed the 15-question pulmonary physiology examination and were eligible for inclusion in the analysis. Despite the prioritization of clinically relevant pulmonary topics, fellows did not outperform the preclinical medical students on any of the 15 questions (Figure 1 and Table E2 in the data supplement). In fact, student performance was statistically significantly higher than that of the fellows on 11 of the 15 questions. When data from all 15 questions were pooled together, the students also performed significantly better than the fellows (P < 0.0001).

Figure 1.

The difference in fellow and medical student scores for each of the 15 questions are shown (scores of first-year fellows minus those of medical students). A negative number indicates that the fellows performed worse than the medical students. The medical students outperformed the fellows in 11 of the 15 questions, and the students performed similarly as the fellows in the other 4 questions.

We next compared the performances of medical students versus fellows within specific domains of pulmonary physiology. Questions had previously been categorized as pertaining to oxygenation, ventilation and carbon dioxide, hemodynamics and gas transport, and pulmonary mechanics by three of the authors (M.A., S.I.M., B.W.L.). A by-domain analysis demonstrated that students again outperformed the fellows in all four domains (P < 0.0001 for each comparison; Table 2).

Table 2.

Fellow versus medical student performance by question domain using logistic regression model

Educational Domain	Question Nos.	Log(OR) ± SE	P Value
Ventilation and CO2	3, 13	−1.29 ± 0.20	<0.0001
Hemodynamics and gas transport	2, 5, 7, 8	−2.74 ± 0.24	<0.0001
Pulmonary mechanics	1, 12, 15	−1.03 ± 0.20	<0.0001
Oxygen	4, 6, 9, 10, 11, 14	−1.28 ± 0.15	<0.0001

Definition of abbreviations: OR = odds ratio; SE = standard error.

The negative log(OR)s indicate that the students outperformed the fellows in all four domains.

Discussion

We found that preclinical medical students outperformed incoming first-year PCCM and CCM fellows on an assessment of basic clinically relevant pulmonary physiologic knowledge, irrespective of the specific domain within pulmonary physiology (oxygenation, ventilation, hemodynamics, or pulmonary mechanics). These findings suggest that trainees experience a decay in knowledge of pulmonary physiology over the course of their medical training, challenging the conventional view that medical education relies on core knowledge obtained during medical school that is subsequently reinforced and expanded upon at each additional level of medical training. To effectively transfer the short-term physiology learning from medical school preclinical courses to permastore or long-term memory, learners appear to need further intentional opportunities for retrieval practice to effectively relearn and reinforce those complex concepts by connecting them to immediately relevant clinical applications.

Decay of knowledge is not surprising, and is arguably expected, for trainees who ultimately choose specialties that do not require retention of pulmonary physiologic knowledge. However, it is concerning that such knowledge decay exists among those specifically training in critical care, in which an understanding of basic pulmonary physiology is essential. For these individuals, it would be reasonable to assume that such concepts were reinforced or even built upon during their ICU or emergency room rotations during residency training.

These findings raise important concerns when considered in the context of new curricular goals promoted by many medical schools. Many schools have shifted from the traditional 4-year curriculum that comprises 2 years of preclinical work and 2 years of clinical work to models with shorter preclinical phases with earlier entry into the clinical phase in an effort to lengthen students’ clinical exposure (15). Furthermore, there is an increased focus on active, small-group, and peer-mediated learning with less time devoted to traditional large-group lectures in which many of these physiologic concepts were previously taught. Although this shift has benefits, students spend less time in the classroom learning basic science and foundational physiology. Viewed through the lens of our findings, this shift toward reduced preclinical time raises the question of whether students have sufficient time to consolidate core physiologic principles and retain this information as they move forward. More significantly, the integration of relevant physiology learning is difficult to standardize and reinforce during student clerkships and residencies. Thus, we suspect there is significant variability in the emphasis on physiology during clinical training.

Significant work has been done to elucidate strategies to make memory and knowledge more “durable.” Ebbinghaus described memory retention as a function of not only the number of repetitions, but also as a function of time. In more modern educational terminology, these concepts can be translated to three key teaching principles: spaced learning, retrieval practice, and interleaving. Spaced learning refers to learning that takes place over an extended period of time, with opportunities for revisiting material and engaging with what are known as “retrieval practices.” Retrieval practices, such as the use of flash cards, ask learners to revisit old material. When this material is “interleaved” or intermittently punctuated with other, perhaps new, material, learners are challenged to check their memory of previously learned material while simultaneously actively engaging with new content. Several studies have demonstrated the positive impact of spaced learning and retrieval practices in the medical field, suggesting that medical educators should consider the incorporation of these practices into their teaching methodologies across the entire training timeline (16–18). It is possible that spacing, retrieval practice, and interleaving the basic physiologic concepts into the undergraduate and graduate clinical teaching arenas may mitigate this decay in physiologic knowledge.

Although we recognize that trainees and clinicians may remain able to practice safely because of pattern recognition and clinical judgment (i.e., fast thinking) without deep physiologic understanding, we strive as educators and practitioners of medicine to ensure the integrity and depth of medical knowledge we impart to our learners and graduates of our programs (19). Solid comprehension of these complex physiologic concepts facilitates sound clinical reasoning for more challenging cases (i.e., slow thinking) that may not so closely hew to the patterns those with more superficial physiology knowledge more easily navigate. It is imperative that thoughtful intensivists invest in reinforcing their physiology foundations to guard against cognitive error.

This study has several strengths. Our data set includes first-year fellows from a variety of academic institutions and clinical backgrounds to improve the external validity of our findings. We used actual examination questions from four well-respected academic institutions from different geographic areas (Pacific Northwest, East Coast, and Midwest) with independent curricula, providing a good representation of questions asked on medical student assessments around the country. Finally, our own examination was developed through a modified Delphi method, with iterative feedback from experts, including student course directors as well as fellowship program directors, to create an assessment that not only adequately captures core pulmonary physiologic principles, but also reflects the material that expert educators agree to be most clinically relevant.

Our study also has several limitations. A validated pulmonary physiology assessment by which to assess learners, to our knowledge, does not exist; we attempted to overcome this by using a rigorous evaluation process to develop our own examination. Although we are unable to share these secure test items that remain in use at the participating medical schools, we undertook a dedicated round of expert evaluation by critical care fellowship program directors in which they were specifically asked to rank the questions with regard to bedside clinical applicability. We believe that by assembling a diverse group of intensivists focused on ensuring the adequacy of clinical training in their programs to review a set of questions from several medical school courses, the resultant assessment material is not esoteric but rather integral to ensuring a sound understanding of common physiologic phenomena.

We acknowledge that education in medical school is not strictly standardized, which is a further limitation to our methods. However, this group of investigators representing different levels of educators along the continuum, as well as several institutions, came to agree upon shared expectations around core physiologic concepts that are directly applicable to common clinical scenarios in the ICU in which fellows should demonstrate understanding.

Additionally, we intentionally limited the examination to 15 questions so it could be administered within a 20-minute time frame to minimize the burden on the fellows during a busy orientation period. We recognize, however, that a 15-item multiple-choice examination may not be sufficient to fully assess expertise in pulmonary physiology, even though our assessment did span several domains and encompass a wide variety of key clinical topics. We recognize that using an MCQ examination has its own limitations in deeply assessing the understanding of complex physiologic concepts or clinical decision-making. Yet, as a community of medical educators, we largely continue to rely on knowledge-based tests through which we extrapolate much more complex abilities to synthesize the data and make sound decisions in dynamic clinical and social environments. Further, regarding question characteristics, we have item analysis for some but not all questions and institutions, which resulted in our inability to report analytics for the composite assessment.

From a feasibility standpoint, we were unable to test learners over several years to demonstrate these particular fellows’ medical school performance, but accept the face validity that the large and diverse group of students from multiple medical schools are likely reflective of the fellows from several fellowship programs. By extension, then, we also do not know exactly when during the timeline of training that the knowledge decay occurred or whether it decayed rapidly to a nadir then recovered somewhat. Indeed, it would be ideal to administer the composite assessment to a group of medical students and periodically reassess their physiologic knowledge throughout their training until completion of their critical care fellowships. However, such a longitudinal cohort study would be impractical, and we are therefore compelled to make certain reasonable assumptions that have face validity.

Furthermore, these data suggest a decay in knowledge pertaining to one specific facet of physiology; this work does not elucidate whether learners experience a decay in knowledge of physiology related to other relevant organ systems. Finally, we assessed only incoming fellows, which means that our findings reflect the impact of residency training rather than the impact of their fellowship training. Further work should be done to retest this cohort of fellows after their first and subsequent years of fellowship to determine whether their knowledge of pulmonary physiology has rebounded or further decayed during fellowship training.

In conclusion, first-year PCCM and CCM fellows performed significantly worse than medical students on an assessment of clinically relevant pulmonary physiologic concepts, suggesting that there is decay in this knowledge during residency training. Although it is not clear where during the training arc the physiology knowledge is lost, critical care fellows, more than undifferentiated medical students and residents, have a recognized need to relearn these concepts. Thus, targeting critical care fellows may have the best likelihood of accomplishing the deeper learning of these physiology concepts and more easily demonstrating their relevance daily at the bedside. Certainly, it is more crucial that a graduating critical care fellow demonstrates proficiency compared with a new fellow. Further studies are needed to assess the impact of fellowship training on the acquisition and retention of physiologic knowledge and how best to support fellows’ knowledge retrieval. As the process of forgetting and relearning is integral to deep and long-term learning, a return to this material in the context of highly relevant clinical applications during terminal fellowship training may have more staying power and lend more substantial and sophisticated physiologic understanding. Thus, further investigation into how graduating fellows or practicing intensivists perform on the same assessment will provide more opportunities to delve into these theories.