Impact of a respiratory ICU rotation on resident knowledge and confidence in managing mechanical ventilation

ABSTRACT

Objective:

To develop and apply a competency-based test to assess learning among internal medicine residents during a respiratory ICU rotation at a university hospital.

Methods:

We developed a test comprising 19 multiple-choice questions regarding knowledge of mechanical ventilation (MV) and 4 self-assessment questions regarding the degree of confidence in the management of MV. The test was applied on the first and last day of a 30-day respiratory ICU rotation (pre-rotation and post-rotation, respectively). During the rotation, the residents had lectures, underwent simulator training, and shadowed physicians on daily bedside rounds focused on teaching MV management.

Results:

Fifty residents completed the test at both time points. The mean score increased from 6.9 ± 1.2 (pre-rotation) to 8.6 ± 0.8 (post-rotation; p < 0.001). On questions regarding the approach to hypoxemia, the recognition of patient-ventilator asynchrony, and the recognition of risk factors for extubation failure, the post-rotation scores were significantly higher than the pre-rotation scores. Confidence in airway management increased from 6% before the rotation to 22% after the rotation (p = 0.02), whereas confidence in making the initial MV settings increased from 31% to 96% (p < 0.001) and confidence in adjusting the ventilator modes increased from 23% to 77% (p < 0.001).

Conclusions:

We developed a competency-based test to assess knowledge of MV among residents before and after an rotation in a respiratory ICU. Resident performance increased significantly after the rotation, as did their confidence in caring for patients on MV.

INTRODUCTION

Since the polio epidemic in Copenhagen in 1952, ^{1 - 3} mechanical ventilation (MV) has been the key element in the care of patients with respiratory failure ⁴ and continues to be the most widely used technique for providing life support . ⁵

The use of MV is associated with significantly higher daily costs for patients receiving treatment in the ICU. ⁶ In addition to the costs, the need for MV continues to be associated with high mortality, in Brazil and in other countries, despite scientific advances in the area. ^{4 , 7 , 8}

Strategies that result in shorter ICU stays or less time on MV could lead to substantial reductions in the total cost per inpatient. ⁹ In addition, the appropriate adjustment of MV settings could reduce mortality in various clinical scenarios. ^{4 , 10 - 13} Therefore, instruction regarding the indications for and management of MV is essential to ensure the appropriate training of residents who will treat critically ill patients. Given that most ICU patients in Brazil and elsewhere are treated by clinicians who are not intensivists, ^{14 - 16} an MV curriculum for residents is important not only residency programs in intensive care but also in those in internal medicine.

There are few data in the literature on whether and how residency programs provide appropriate training in the management of MV. ^{17 , 18} In a study conducted by Willcox et al., ¹⁹ emergency medicine residents reported that, although they frequently treated patients on MV, they felt that they had not received sufficient instruction on the management of MV. In a study involving medical students, residents, and emergency medicine physicians in Brazil, 85% of the participants believed that they had not received enough information about MV during their medical training and considered themselves underqualified to care for patients on MV. ²⁰ In addition, there is no consensus on how to assess resident knowledge of MV. Although the development of validated assessment tools to measure learning is the most highly recommended method, ^{21 , 22} the effective application of such tools still presents a challenge in all areas of medicine. ²³

For an assessment to be considered well-planned, competencies and learning objectives in MV must first be established. ^{18 , 24}

In the present study, we developed and applied a test to assess the knowledge and learning of internal medicine residents who completed a supervised rotation at a university hospital respiratory ICU. Our hypothesis was that their knowledge of MV and confidence in managing patients on MV would increase after a 30-day respiratory ICU rotation focused on MV management.

METHODS

Location and participants

This study was carried out in the Respiratory ICU of the Hospital das Clínicas da Faculdade de Medicina da Universidade de São Paulo (HCFMUSP, University of São Paulo School of Medicine Hospital das Clínicas), located in the city of São Paulo, Brazil, between September of 2014 and August of 2015. The study was approved by the local research ethics committee. Given that it was a project designed to assess resident performance, an activity that is within the scope of the medical residency program, the requirement for informed consent was waived.

The study participants were second-year internal medicine residents at the HCFMUSP. The study inclusion criteria were being an internal medicine resident at the HCFMUSP and being an intern in the HCFMUSP respiratory ICU during the study period. The exclusion criterion was not completing the assessments during the rotation.

Test development

We developed a multiple-choice question test based on clinical cases, initially comprising 22 questions. The questions were developed on the basis of lists of competencies and learning objectives in MV in adult individuals, described in the literature, ^{18 , 24} as well as on the basis of our respiratory ICU residency program. Most questions were based on clinical cases, and a given case could be used for multiple questions.

First, the questions were discussed with our senior team, consisting of five MV specialists with more than 10 years of experience in ICUs, all of whom were involved in researching and teaching MV. On the basis of those discussions, 3 questions were excluded because they were not considered highly relevant for assessing knowledge of MV. Therefore, the final version of the test comprised 19 questions. To improve clarity, the group of specialists modified some of those questions, and the consensus opinion was that the final version of the test (Chart S1, supplementary material) was appropriate.

The key knowledge topics addressed on the test were cardiopulmonary physiology, interpretation of waveforms, adjustment of ventilator settings, and ventilator mode recognition (Chart 1). In addition to the questions regarding adjustment of MV, we included 4 self-assessment questions about the degree of confidence in airway management, in applying noninvasive ventilation (NIV), in making the initial MV settings, and in adjusting basic ventilator modes. Those four questions were scored on a Likert scale with 5 possible answers, ranging from 0 (I am not at all confident) to 4 (I am completely confident). The scores were later regrouped into two categories: low confidence (0, 1, and 2); and high confidence (3 and 4).

Chart 1. Competencies associated with the knowledge questions on the test.

Recognize patterns of respiratory system resistance/compliance and their relationships with prevalent diseases

Apply the basic mechanical ventilation modes in patients with acute respiratory failure

Interpret ventilator waveforms

List risk factors for a difficult airway

Recognize acute respiratory failure due to ARDS and pathophysiological mechanisms of hypoxemia

Select ventilator settings consistent with protective ventilation in patients with ARDS

List strategies for securing the airway in a patient who is difficult to intubate

Describe the advantages and risks of rescue measures for refractory hypoxemia and their indications

Appropriately indicate the need for ventilatory support in patients with acute respiratory failure due to neuromuscular disease

Recognize risk factors for extubation failure

Adjust ventilator settings according to the clinical status and laboratory test results of the patient

Identify particularities in the ventilatory management of patients with certain lung diseases

List indications for noninvasive mechanical ventilation

ARDS: acute respiratory distress syndrome.

Data collection

The test was administered to residents twice during their rotation in the HCFMUSP respiratory ICU. The rotation lasted 30 days, and the test was administered on the first and last days of that period (pre-rotation and post-rotation, respectively).

During the between-test interval, the residents were responsible for the care of patients on MV, making changes to the ventilator settings with the help of pulmonology residents, pulmonologists, and intensivists, who were responsible for supervising the residents during their ICU rotation. Rotation activities included daily multidisciplinary rounds to discuss cases and plan treatment, with a focus on MV, as well as to provide overall care to the patients. Rounds lasted an average of two hours, with the participation of a supervising physician, all residents, and the ICU respiratory therapist. In addition, the residents participated in bedside discussions that focused on the care of patients on MV, involving ventilator waveform interpretation and ventilator setting adjustments. Those discussions were based on the needs of the resident. Whenever the MV settings were going to be changed in the morning, the supervising physician and the residents gathered at the bedside to discuss the ventilator settings. Throughout the day, depending on the complexity of each case, the settings were reviewed, and the residents had the opportunity to adjust the settings under the supervision of the pulmonology residents or supervising physicians. The residents had four one-hour lectures addressing the following topics: ventilator modes; MV in acute respiratory distress syndrome (ARDS) and severe hypoxemia; MV in patients with obstructive lung disease; and weaning from MV. The lectures combined a traditional expository model, in which ventilator waveforms were drawn on a whiteboard, and a participatory model, in which the residents posed questions and made suggestions. In addition, MV simulations were carried out using online software (xlung Excellence in teaching mechanical ventilation management simulator; Pulmocenter, Fortaleza, Brazil). The practical training with the virtual tool lasted approximately 90 min, and, using standardized clinical cases, the residents were encouraged to suggest adjustments to the virtual ventilator settings and evaluate the result of their decisions. The same supervising physicians were responsible for teaching the residents throughout the study period.

The primary outcome variable was the test score, normalized to vary from 0 to 10. The secondary outcome variables were the degrees of confidence (high or low) for the four key competencies addressed in the self-assessment questions.

Statistical analysis

We tested the primary outcome variable for normality with the Shapiro-Wilk test. We used paired t-tests to compare the pre- and post-rotation total scores, whereas we used McNemar’s test to compare the pre- and post-rotation rates of correct answers to each knowledge question and to compare the pre- and post-rotation scores on the questions regarding confidence. The level of significance was set at p < 0.05. We used the R statistical program, version 3.5.1 (The R Foundation for Statistical Computing, Vienna, Austria).

RESULTS

During the study period, 59 residents completed an rotation at the HCFMUSP respiratory ICU, and 50 of those residents completed the test at both time points. The mean score increased from 6.9 ± 1.2 (pre-rotation) to 8.6 ± 0.8 (post-rotation; p < 0.001), with a mean pre-rotation to post-rotation difference of 1.67 (95% CI: 1.3-2.0; Figure 1).

Figure 1. The blue bars represent the means and the error bars represent the standard deviation of the pre- and post-rotation scores (on a scale from 0 to 10).

Table 1 shows the proportion of individuals who answered each question correctly at each of the two time points, and Table S1 (supplementary material) describes the knowledge topic assessed by each question and its correspondence with a list of competencies described in the literature. ^{18 , 24} The questions with the highest pre-rotation rates of correct answers were those assessing knowledge of how to calculate respiratory system compliance and resistance, of the characteristics of the basic MV modes, of protective ventilation in ARDS, and of permissive hypercapnia. In contrast, the questions regarding the recognition of asynchrony and of risk factors for extubation failure had the lowest pre-rotation rates of correct answers.

Table 1. Knowledge topics assessed and participant performance on each question before and after the respiratory ICU rotation (N = 50).^a .

Knowledge topic assessed	Question	Pre-rotationa	Post-rotationa	p
Recognize patterns of resistance and compliance of the respiratory system and their relationships with prevalent diseases	4	48 (96)	49 (98)	1
Apply the basic mechanical ventilation modes in patients with acute respiratory failure	5	45 (90)	49 (98)	0.22
Interpret ventilator waveforms	3	42 (84)	48 (96)	0.08
	6	47 (94)	49 (98)	0.62
	19	16 (32)	37 (74)	< 0.001
List risk factors for a difficult airway	7	39 (78)	43 (86)	0.39
Recognize acute respiratory failure due to ARDS and pathophysiological mechanisms of hypoxemia	8	40 (80)	43 (86)	0.51
	9	30 (60)	45 (90)	< 0.001
Select ventilator settings consistent with protective ventilation in patients with ARDS	10	47 (94)	50 (100)	0.24
List strategies for securing the airway in a patient who is difficult to intubate	11	22 (44)	35 (70)	0.002
Describe the advantages and risks of rescue measures for refractory hypoxemia and their indications	12	23 (46)	32 (64)	0.07
Appropriately indicate the need for ventilatory support in patients with acute respiratory failure due to neuromuscular disease	13	31 (62)	42 (84)	0.003
Recognize risk factors for extubation failure	14	18 (36)	35 (70)	< 0.001
Adjust ventilator settings according to the clinical status and laboratory test results of the patient	15	42 (84)	45 (90)	0.51
	16	33 (66)	43 (86)	0.02
Identify particularities in the ventilatory management of patients with certain lung diseases	1	26 (52)	31 (62)	0.30
	2	47 (94)	49 (98)	0.48
	19	16 (32)	37 (74)	< 0.001
List indications for noninvasive mechanical ventilation	17	38 (76)	49 (98)	0.003
	18	27 (54)	43 (86)	0.001

Pre-rotation: test administered on the first day of the rotation; post-rotation: same test administered on the last (30th) day of the rotation; and ARDS: acute respiratory distress syndrome. ^aValues expressed as number of participants (%).

We found that the post-rotation rates of correct answers to questions regarding the treatment of hypoxemia, the recognition of patient-ventilator asynchrony, the risks of high inspiratory pressures, and the recognition of risk factors for extubation failure were significantly higher than the pre-rotation ones. The pre-rotation and post-rotation rates of correct answers were both low for only one question, which addressed particularities in the ventilatory management of patients with COPD (Table 1).

The questions regarding the degree of confidence in managing MV after the rotation were answered by 48 residents. Comparing the pre- and post-rotation scores on those questions (Figure 2), we found that the degree of confidence increased from 6% to 22% for airway management (p = 0.02), from 96% to 100% for applying NIV (p = 0.47), from 31% to 96% for making the initial MV settings (p < 0.001), and from 23% to 77% for adjusting the ventilator modes increased (p < 0.001).

Figure 2. Proportion of participants (n = 48) by degree of confidence as measured by the self-assessment questions before the rotation (light blue) and after the rotation (dark blue).

DISCUSSION

In the present study, we found that the overall performance of second-year internal medicine residents on a clinical case-based test assessing knowledge of MV was moderately good; however, their performance on questions assessing knowledge related to core competencies, such as the recognition of asynchrony and of risk factors for extubation failure, was poor. After a 30-day rotation in a respiratory ICU, their overall performance was significantly higher on most questions. Resident confidence in caring for patients on MV, which was low at the beginning of the rotation, increased significantly as a result of the training.

Although MV management is considered an important skill for residents of various specialties, ¹⁷ studies have shown that physicians and other health professionals who work in ICUs have difficulty in interpreting ventilator waveforms, ²⁵ show poor adherence to protective MV protocols in patients with ARDS ^{12 , 26} and feel poorly prepared to care for patients on MV. ²⁰ This is probably due, in part, to the fact that instruction in MV is not well standardized, because there is no widely used list of competencies in MV, nor is there consensus on how to develop a specific curriculum in MV and to quantify learning. ^{18 , 24 , 27 - 29}

In the present study, the pre-rotation total score was moderately high, reflecting the knowledge of MV that was acquired during the ICU rotations in the first year of the residency program. However, the increase in the mean total score after the rotation shows a significant gain of knowledge after the training focused on MV management, a gain that was also observed in a study involving residents who underwent intensive training in MV. ²⁷

We found that the rate of correct answers to the question assessing protective ventilation in patients with ARDS increased from 94% (pre-rotation) to 100% (post-rotation), a rate that is higher than the 52% reported by Cox et al. ¹⁷ in another study involving medical residents. That difference might be due to the fact that protective ventilation has come to be more widely used in recent years and that the study by Cox et al. ¹⁷ was published more than a decade ago. With regard to managing NIV, we found rates of correct answers of 76% (pre-rotation) and 98% (post-rotation), compared with 73% in the Cox et al. study. ¹⁷ With regard to adjusting ventilator settings to correct auto-positive end-expiratory pressure, the rate of correct answers in the Cox et al. study ¹⁷ was 65%, whereas we found rates of 84% (pre-rotation) and 96% (post-rotation).

The rate of correct answers to some questions was low before the rotation but improved significantly after the rotation. That could be attributed to the training and discussions that took place during the rotation, corroborating findings from similar studies. ²⁷ The high rate of correct answers to some questions before the rotation may be related to knowledge of MV gained during previous rotations or to the varying degree of difficulty of the questions.

We found that resident confidence in managing MV before the rotation varied depending on the question. The residents reported a high degree of confidence only in applying NIV, and that remained unaltered after the rotation. That finding differs from one reported by Tallo et al., ²⁰ who found that only 23% of the participants in their study felt confident enough to initiate NIV. However, in comparison with ours, the sample evaluated in that study ²⁰ was more heterogeneous, including final-year medical students and emergency physicians, which may have contributed to the difference relative to our finding.

The participants in our study had low confidence in airway management, in making the initial MV settings, and in adjusting ventilator modes before the rotation, corroborating previous findings. ²⁰ Their confidence increased significantly for those three skills but remained relatively low for airway management. Although airway management in an emergency setting was not a learning objective of the rotation, the low degree of resident confidence even after the rotation indicates the need for additional training in this area.

Our study has some limitations that should be taken into account when interpreting our results. First, this was a single-center study. Therefore, performance on the test may be associated with the prior knowledge level of the residents selected for admission to our residency program; with specific training, provided in our residency program, in rotations in emergency rooms and ICUs; and with the particularities of the rotation in our respiratory ICU. Second, the test was developed only by professionals in our department, which again may represent the specific view of our residency program. Third, education experts have conflicting opinions on multiple choice tests to measure learning and many now consider the use of simulations to be a more appropriate means of assessment. ^{29 - 31} Finally, there is no list of validated and widely accepted MV competencies. Nevertheless, our study has some strengths. We conducted an objective assessment based on clinical cases with scenarios similar to those that residents are exposed to in ICUs in real life. In addition, when we developed the questions, we took into consideration the pre-defined learning objectives in our residency program and in the literature. ²⁴ Furthermore, this was the first study to assess knowledge of MV in a low- or middle-income country.

In terms of future directions, it is essential for residency programs that train physicians who will work in ICUs to have a well-structured MV curriculum based on well-defined learning objectives and competencies, including diversified teaching strategies that combine theoretical classes, discussions, simulations, and bedside teaching. ^{27 , 28 , 31 , 32} The validation of a list of competencies in MV that can be adopted by various medical residency programs would also be quite useful to ensure that the training provided to residents is appropriate. ²⁴ Finally, the development of tools that are more effective in assessing competency in MV, not only in terms of knowledge, but also in terms of skills and attitudes, such as tools based on simulations or structured objective assessments, would further contribute to improving residency programs so that the training given to physicians working in ICUs in Brazil is appropriate for the care of critically ill patients on MV, which could result in reduced mortality and reduced ICU costs.

In conclusion, we have developed a test, based on lists of competencies in MV, that made it possible to assess the knowledge and learning of second-year internal medicine residents before and after a 30-day rotation in a respiratory ICU, as well as to determine their degree of confidence in the management of MV. At the beginning of the rotation, resident performance on some core topics was insufficient and their confidence in their skills related to the application of MV was low. After the rotation, the performance of the residents improved significantly, as did their confidence in caring for patients on MV.