Supplemental Digital Content is available in the text.
Keywords: intensive care unit liberation bundle, novel coronavirus, pandemic; postintensive care syndrome; quality improvement; supportive care
Abstract
Objectives:
To investigate implementation of evidence-based and supportive cares in ICUs, such as the ABCDEF, nutrition therapy, and ICU diary, for patients with coronavirus disease 2019 infection in ICUs and their association with ICU clinical practice and setting.
Design:
A worldwide, 2-day point prevalence study.
Setting:
The study was carried out on June 3, 2020, and July 1, 2020. A total of 212 ICUs in 38 countries participated. Clinicians in each participating ICU completed web-based online surveys.
Patients:
The ICU patients with coronavirus disease 2019.
Interventions:
None.
Measurements and Main results:
The implementation rate for the elements of the ABCDEF bundle, other supportive ICU care measures, and implementation-associated structures were investigated. Data were collected for 262 patients, of whom 47.3% underwent mechanical ventilation and 4.6% were treated with extracorporeal membrane oxygenation. Each element was implemented for the following percentages of patients: elements A (regular pain assessment), 45%; B (both spontaneous awakening and breathing trials), 28%; C (regular sedation assessment), 52%; D (regular delirium assessment), 35%; E (early mobility and exercise), 47%; and F (family engagement and empowerment), 16%. The implementation of element E was 4% for patients on mechanical ventilation and 8% for patients on extracorporeal membrane oxygenation. Supportive care, such as protein provision throughout the ICU stay (under 1.2 g/kg for more than 50% of the patients) and introduction of ICU diary (25%), was infrequent. Implementation rates of elements A and D were higher in ICUs with specific protocols and fewer ICU beds exclusively for patients with coronavirus disease 2019 infection. Element E was implemented at a higher rate in ICUs that had more ICU beds assigned for them.
Conclusions:
This point prevalence study showed low implementation of the ABCDEF bundle. Specific protocols and the number of ICU beds reserved for patients with coronavirus disease 2019 infection might be key factors for delivering appropriate supportive care.
It is clear that evidence-based and supportive ICU care synchronized with treatment of the underlying disease should be the standard of care to prevent weakness and disabilities in patients after resolution of their critical illness (1–3). The ABCDEF bundle is a collection of six elements which represents an evidence-based approach for clinicians to optimize patients recovery and outcomes in ICU (1), the guidelines for the prevention and management of pain, agitation/sedation, delirium, immobility, and sleep disruption in the ICU (2), and nutrition guidelines (3, 4), are key elements that have been shown not only to decrease mortality and morbidity (5), but also to rehabilitate functional abilities and health-related quality of life (6). A variety of guidelines recommend incorporation of these evidence-based approaches into clinical ICU practice (1–3), according to the local situation and available resources (7, 8).
However, the novel coronavirus pandemic (severe acute respiratory syndrome coronavirus 2) rapidly changed ICU practice internationally (9). Challenges include an inadequate number of beds to meet the staggering increase in the number of patients with coronavirus disease 2019 (COVID-19) infection (10) and insufficient interprofessional staff resources to meet the demand (11). These challenges may reduce the quality of clinical care in ICUs, thereby worsening outcomes and increasing mortality (12). In addition, recent evidence has highlighted the severe physical disabilities and prolonged symptoms during recovery from COVID-19 infection, which limit patients’ daily activities and quality of life after discharge from the hospital (13–16). Thus, an enhanced recovery program that optimizes evidence-based ICU care and patient recovery is essential (15–18). Despite specific recommendations for ICU care, such as using the ABCDEF bundle (19–22) and providing adequate nutrition (23, 24) that could enhance the effect of the bundle if synchronized (1, 3, 4), data demonstrating implementation of these aspects of care in the ICU are insufficient. There are serious concerns regarding second waves in many countries, and these data are essential to improve future patient outcomes and minimize disabilities.
Therefore, we conducted an international, Internet-based, 2-day point prevalence survey to investigate the implementation of the ABCDEF bundle and other supportive care provided to patients with COVID-19 infection in ICUs, with consideration of the specific ICU structure, such as presence of protocols, ICU staffing, and multidisciplinary-rounds, which have been suggested as stimulants for ICU care in the literature (2, 6–8).
MATERIALS AND METHODS
Design and Setting
This worldwide 2-day point prevalence study of evidence-based ICU care for critically ill patients with COVID-19 infection was approved by the ethics committee of the Saiseikai Utsunomiya Hospital (2020-07) as the central institution, was registered in the registration system named University Medical Information Network (ID: 000046103), and followed the STROBE cross-sectional guidelines which is a statement to strengthen the reporting of observational, especially for cross-sectional, studies (Supplemental Table 1, http://links.lww.com/CCX/A515). Surveys were performed on June 3, 2020, and July 1, 2020. This project was led by the Japanese Society of Intensive Care Medicine (JSICM) in collaboration with the World Federation of Intensive and Critical Care (WFICC), European Society of Intensive Care Medicine (ESICM), Indian Society of Critical Care Medicine (ISCCM), and other networks (Appendix 1, http://links.lww.com/CCX/A512).
Surveys were anonymous and the information collected did not include specific data that could identify the facility or individual; therefore, ethical approval at each participating facility was omitted according to the Ethical Guidelines for Medical and Health Research Involving Human Subjects in Japan (25). This ethical policy was explained to all clinicians, and ICUs participated only if they agreed after referring to the ethics policies in their regions and countries. If participation presented difficulty on the ethical policy, clinicians could decide to participate after obtaining ethical approval. Under this policy, registration for each participating site and survey completion were fully managed online with Google Forms (Google Inc.).
The Study Process
We recruited ICUs to participate by distributing information about the study and an invitation letter to members of JSICM, WFICC, ESICM, and ISCCM via their own networks between May 16, 2020, and July 1, 2020. Social networking services and a web-based advertisement were also used (Appendix 1, http://links.lww.com/CCX/A512). All ICUs in the world were eligible, regardless of whether they had patients with COVID-19 infection at the time. There were no exclusion criteria based on the structure of the facility or ICU.
After registration of the name of the country, hospital, ICU, and representatives, all confirmed representatives were requested to provide basic information about the hospital/ICU demographics and backgrounds (35 items, 5–10 min) via Google Form. Only those who provided the information were enrolled as participating sites by issuing a facility registration number to each ICU. All ICUs on the first survey date were automatically enrolled for the second survey and requested to complete the survey with the same facility registration number. One day before each of the survey dates, participating representatives received the URL for the ICU care survey (39 items, 5–10 min), which asked for baseline characteristics of patients and daily ICU care provided to patients with COVID-19 infection for each patient in the participating ICUs on the survey date. Patients receiving end-of-life care or receiving palliative care were excluded.
Data Collection
The data collected for the hospital/ICU demographics are shown in Appendix 2 (http://links.lww.com/CCX/A513). The questionnaires and response sheets for the ICU care survey are shown in Appendix 3 (http://links.lww.com/CCX/A514). These questionnaires and response sheets had been pilot-tested and checked by five of the coauthors (K.N., H.K., P.N., E.W.E., and S.K.) before the survey. The facility registration number was used to link the basic information of the participating hospital/ICU with the data for baseline characteristics of the patients and daily ICU care. The collected data were managed by the collaborators (Appendix 1, http://links.lww.com/CCX/A512). The operational definitions of the ABCDEF bundle, composed of elements A, B, C, D, E, and F, were cited from a previous study (Supplemental Table 2, http://links.lww.com/CCX/A515) (5). Other ICU cares, such as nutrition, an ICU diary, and physical restraints, are defined in Supplemental Table 2 (http://links.lww.com/CCX/A515).
Study Outcomes
The primary outcome was the implementation of each element of the ABCDEF bundle. The following elements associated with the bundle were examined: the presence of a target goal for pain and sedation management; the prevalence of delirium; the highest mobility level according to the ICU Mobility Scale (26); assessment tools and agents used for pain, sedation, and delirium; the reason why element B was not implemented; nonpharmacologic interventions to control delirium; classifications of professionals delivering mobilization/rehabilitation; devices used during rehabilitation; barriers that prevented sitting on the edge of the bed or more; and the visiting habits for family members. The implementation of nutrition therapy, sleep assessment, an ICU diary, the use of physical restraints, and the route and amount of energy and protein in nutrition were also investigated.
The association between implementation of ICU care and ICU structure, including the presence of specific protocols, the frequency of multidisciplinary and/or multiprofessional rounds, the nurse-to-patient ratio, the number of ICU beds exclusively for patients with COVID-19 infections, the presence of dedicated physiotherapists, the ability of physiotherapists to enter the room of patients with COVID-19 infections, and visiting hours were also evaluated.
Data Analysis
Nonnormally distributed continuous data are presented as medians with interquartile range, and categorical data, without missing data, are presented as numbers or percentages. The chi-square test and Fisher exact test were used for categorical data appropriately. For analysis of ICU care implementation, patients were classified into three groups: no mechanical ventilation (MV) or extracorporeal membrane oxygenation (ECMO), undergoing MV, and undergoing ECMO. The highest mobility level and the energy and protein provided per day were evaluated according to three phases of critical illness: the early period in the acute phase (ICU days 1–3), the late period in the acute phase (ICU days 4–7), and after the acute phase (ICU days 8–14 and from 15) (3, 4).
All statistical analyses were carried out with EZR (Saitama Medical Center, Jichi Medical University, Saitama, Japan) (27), which is a graphical user interface for R (Vienna, Austria). Statistical tests were two-sided, and statistical significance was set at p < 0.05.
RESULTS
Characteristics of Participating Hospitals and ICUs
For the first survey date, 166 ICUs were enrolled, and for the second survey date, 212 were enrolled (Fig. 1; and Supplemental Table 3, http://links.lww.com/CCX/A515). The respondents were intensivists (76%), followed by other physicians (12%), physiotherapists (6%), and nurses (5%). The basic structure of the hospitals/ICUs at each site did not differ between the two dates (Table 1; and Supplemental Table 4, http://links.lww.com/CCX/A515). Most ICUs were medical-surgical mixed ICUs with a 1:2 nurse-to-patient ratio and had fewer than five beds assigned for patients with COVID-19 infection. Intensivists, physicians, and nurses were generally allowed to enter the rooms of patients with COVID-19 infection. Approximately 40% of ICUs used written protocols for ICU care and 60% performed multidisciplinary/professional rounds daily. Visiting hours were reduced or banned during the COVID-19 pandemic (Supplemental Table 5, http://links.lww.com/CCX/A515).
TABLE 1.
Parameter | First Survey (n = 166) | Second Survey (n = 212) |
---|---|---|
Basic information for participating ICUs | ||
Type of ICU, n (%)a | ||
Medical-surgical mixed ICU | 140 (84) | 169 (80) |
Medical ICU | 11 (7) | 26 (12) |
Surgical ICU including cardiac surgery | 7 (4) | 9 (4) |
Other types of ICUb | 8 (5) | 8 (4) |
Nurse-to-patient ratio, n (%) | ||
1:1 | 22 (13) | 35 (17) |
1:2 | 133 (80) | 151 (71) |
1:3 or higher | 11 (7) | 26 (12) |
Number of ICU beds, n (%) | ||
< 10 | 43 (26) | 59 (28) |
10–19 | 75 (45) | 90 (42) |
≥ 20 | 48 (29) | 63 (30) |
ICU beds exclusively for patients with COVID-19, n (%) | ||
< 5 | 101 (61) | 113 (53) |
5–19 | 43 (26) | 66 (31) |
≥ 20 | 22 (13) | 33 (16) |
Professionals who may enter rooms of patients with COVID-19 under regional infection control regulations of the participating hospital, n (%) | ||
Intensivists | 157 (95) | 202 (95) |
Physicians other than intensivists | 134 (81) | 169 (80) |
Nurses | 166 (100) | 212 (100) |
Physiotherapists | 93 (56) | 122 (58) |
Respiratory therapists | 48 (29) | 68 (32) |
ICUs with dedicated intensivists, n (%) | 159 (96) | 205 (97) |
ICUs with dedicated physiotherapists, n (%) | 78 (47) | 108 (51) |
ICUs with dedicated respiratory therapists, n (%) | 44 (27) | 65 (31) |
ICU structure associated with evidence-based and supportive ICU care, n (%) | ||
ICUs with a written pain management protocol | 73 (44) | 98 (46) |
ICUs with a written spontaneous awakening trials management protocol | 60 (36) | 83 (39) |
ICUs with a written spontaneous breathing trials management protocol | 70 (42) | 91 (43) |
ICUs with a written sedation management protocol | 88 (53) | 115 (54) |
ICUs with a written delirium management protocol | 66 (40) | 82 (39) |
ICUs with a written mobilization/rehabilitation management protocol | 73 (44) | 93 (44) |
ICUs with a written sleep management protocol | 21 (13) | 30 (14) |
Frequency of multidisciplinary rounds for patients with COVID-19 in the ICU, n (%) | ||
Daily | 100 (60) | 135 (64) |
At least once a week | 9 (5) | 10 (5) |
Not applicable | 57 (34) | 67 (32) |
Visiting hr/d for patients with COVID-19 in the ICU (hr), n (%) | ||
No visiting hours | 142 (86) | 182 (86) |
0–5 | 21 (13) | 26 (12) |
6–23 | 1 (1) | 2 (1) |
No limitation on visiting hours | 2 (1) | 2 (1) |
COVID-19 = coronavirus disease 2019.
aOne ICU was managed as a tele-ICU by another hospital or ICU.
bAmong the other types of ICUs on both days, five were PICUs.
Baseline Patient Characteristics
The number of ICU patients was 135 on the first survey date and 127 on the second survey date (Table 2), with equal distributions of age groups. Most patients were male (68%). The median ICU length of stay was 9 days (5–35 d). Of the 262 patients, 124 (47.3%) underwent MV and 12 (4.6%) were treated with ECMO. There was no statistically significant difference between the first and second survey days for these characteristics, except for the use of MV (p < 0.001), the number of patients who underwent prone positioning (p = 0.002), and duration of prone positioning (p < 0.05) (Table 2).
TABLE 2.
Characteristics | Total Patients (n = 262) | Patients in the First Survey (n = 135) | Patients in the Second Survey (n = 127) |
---|---|---|---|
Demographics | |||
Age, yr, n (%) | |||
< 50a | 69 (26) | 27 (20) | 42 (33) |
50–59 | 69 (26) | 24 (18) | 45 (35) |
60–69 | 62 (24) | 37 (27) | 25 (20) |
70–79 | 46 (18) | 35 (26) | 11 (9) |
≥ 80 | 16 (6) | 12 (9) | 4 (3) |
Male gender, n (%) | 177 (68) | 96 (71) | 81 (64) |
Race, n (%) | |||
Asian | 118 (45) | 63 (47) | 55 (43) |
White | 103 (39) | 54 (40) | 49 (39) |
Hispanic | 10 (4) | 4 (3) | 6 (5) |
Black | 7 (3) | 5 (4) | 2 (2) |
Other | 24 (9) | 9 (7) | 15 (12) |
Body mass index, kg/m2, n (%) | |||
< 18.5 | 19 (7) | 8 (6) | 11 (9) |
18.5–24.9 | 118 (45) | 58 (43) | 60 (47) |
25.0–29.9 | 79 (30) | 41 (30) | 38 (30) |
30.0–34.9 | 27 (10) | 14 (10) | 14 (11) |
≥ 35.0 | 19 (7) | 14 (10) | 4 (3) |
ICU length of stay, d, median (interquartile range) | 9 (5–35) | 24 (7–50) | 6 (4–10) |
Ambulatory independence prior to ICU admission, n (%) | 222 (85) | 118 (87) | 104 (82) |
ICU treatment provided on the survey date | |||
Respiratory assistance, n (%) | |||
No respiratory device | 20 (8) | 12 (9) | 8 (6) |
Oxygen, such as nasal cannula, face mask, reserved face mask, and others | 89 (34) | 37 (27) | 52 (41) |
Nasal high-flow cannula | 22 (8) | 10 (7) | 12 (9) |
Noninvasive ventilation | 33 (13) | 9 (7) | 24 (19) |
Mechanical ventilation | 124 (47) | 86 (64) | 38 (30) |
ECMO | 12 (5) | 11 (8) | 1 (1) |
Patients receiving continuous/intermittent renal replacement therapy, n (%) | 38 (15) | 22 (16) | 16 (13) |
Patients receiving continuous neuromuscular blocking agents, n (%) | 38 (15) | 25 (19) | 13 (10) |
Patients receiving continuous vasoactive drugs, n (%) | 65 (25) | 42 (31) | 23 (18) |
Patients receiving continuous analgesics, n (%)b | 118 (45) | 45 (33) | 73 (57) |
Patients receiving continuous use of sedatives, n (%)c | 102 (39) | 59 (44) | 43 (34) |
Patients receiving prone positioning, n (%) | 57 (22) | 19 (14) | 38 (30) |
Prone positioning without mechanical ventilation or ECMOd | 35 (26) | 6 (13) | 29(33) |
Prone positioning with mechanical ventilatione | 22 (18) | 13 (15) | 9 (24) |
Prone positioning with ECMOd | 2 (17) | 2(18) | 0 (0) |
Scheduled total hours of prone positioning, hr, n (%) | (n = 57) | (n = 19) | (n = 38) |
0–5 | 26 (46) | 5 (26) | 21 (55) |
6–11 | 9 (16) | 4 (21) | 5 (13) |
12–17 | 17 (30) | 8 (42) | 9 (24) |
18–23 | 3 (5) | 1 (5) | 2 (5) |
24 (all day) | 2 (4) | 1 (5) | 1 (3) |
ECMO = extracorporeal membrane oxygenation.
aOf 69 patients, two were under 20 yr old.
bOf 118 patients, 85 patients (70%) were on mechanical ventilation. Among 33 patients without mechanical ventilations, 12 patients received prone positioning.
cOf 102 patents, 82 patients (80%) were on mechanical ventilation. Among 20 patients without mechanical ventilations, 10 patients received prone positioning.
dPercentages were calculated by dividing by the number of patients without mechanical ventilation or ECMO. The number of patients without mechanical ventilation or ECMO was 48 on the first survey, 89 on the second survey, and 137 in total.
ePercentages were calculated by dividing by the number of patients on mechanical ventilation. The number of patients on mechanical was 86 on the first survey, 38 on the second survey, and 124 in total.
dPercentages were calculated by dividing by the number of patients on ECMO. The number of patients on ECMO was 11 on the first survey, 1 on the second survey, and 12 in total.
Implementation of the ABCDEF Bundle
Among the different patient groups, significant differences were present in implementation of elements A, C, D, and E (Fig. 2). The implementation rates included: element A, “regular pain assessment,” (45%); element B, “both spontaneous awakening and breathing trials,” (28%); element C, “regular sedation assessment,” (52%); element D, “regular delirium assessment,” (38%); element E, “early mobility and exercise,” (35%); and element F, “family engagement and empowerment,” (16%) (Table 3). The details associated with implementation of each element are shown in Supplemental Table 6 (http://links.lww.com/CCX/A515). Most patients undergoing MV or ECMO were treated with fentanyl and benzodiazepines for pain control and sedation. The Numerical Rating Scale, Critical-care Pain Observation Tool, and Behavioral Pain Scale were used as pain-assessment tools, the Richmond Agitation-Sedation Scale was used for sedation assessment, and the Confusion Assessment Method for the ICU was used to monitor delirium. The most common reason for spontaneous awakening trials (element B) not being performed was respiratory instability, followed by the absence of a protocol. Overall, 11% of patients were diagnosed with delirium. Nonpharmacologic interventions, such as orientation, optimizing sleep conditions, and mobilization, were used more frequently than pharmacologic interventions to control delirium. Although 64% of patients without MV or ECMO could mobilize to the level of standing, the implementation of element E for the patients on MV or ECMO was less than 10%. The mobility level increased gradually during the ICU stay, but the median level in patients undergoing MV remained at an ICU Mobility Scale score of 0, the level of passive exercise in bed (Fig. 3). The barriers preventing increased mobility to sitting at the edge of the bed or higher were primarily respiratory factors, such as desaturation or an excessive respiratory rate, followed by consciousness factors. Intensivists were less involved in mobilization (20%) compared with nurses (60%) or physiotherapists (56%). The cycle-ergometer and electrical neuromuscular stimulation were rarely used. Electronic devices with a monitor were used to facilitate meetings with 40% of the patients. All elements of the ABCDEF bundle, even if element F is excluded, were fully implemented for only 1% of the patients.
TABLE 3.
Variables | Total Patients (n = 262) | Patients Without Mechanical Ventilation and ECMO (n = 137) | Patients on Mechanical Ventilation (n = 124) | Patients on ECMO (n = 12) |
---|---|---|---|---|
Implementation of each element of the ABCDEF bundle | ||||
Patients receiving element A, n (%)a | 118 (45) | 42 (31) | 75 (60) | 12 (100) |
The presence of a target goal to control patient’s pain, n (%) | 136 (52) | 49 (36) | 86 (69) | 10 (83) |
Patients receiving element Ba | ||||
Spontaneous awakening trial during continuous sedation, n (%)a,b | 29 (28) | 4 (19) | 25 (32) | |
Spontaneous breathing trial on mechanical ventilation, n (%)a,c | 35 (28) | 35 (28) | ||
Patients receiving element C, n (%)a | 136 (52) | 45 (33) | 90 (73) | 12 (100) |
The presence of a target goal to control patient’s sedation, n (%) | 157 (60) | 49 (36) | 107 (86) | 12 (100) |
Patients receiving element D, n (%)a | 100 (38) | 37 (27) | 62 (50) | 7 (58) |
Patients diagnosed with delirium, n (%) | 30 (11) | 9 (7) | 21 (17) | 2 (17) |
Patients receiving element E, n (%)a | 93 (35) | 88 (64) | 5 (4) | 1 (8) |
The highest mobility level according to the ICU Scale, median (interquartile range)d | 2 (0–6) | 5 (3–8) | 0 (0–1) | 0 [0-2] |
Patients receiving element F, n (%)a | 42 (16) | 23 (17) | 19 (15) | 1 (8) |
The implementation of other essential ICU care | ||||
Patients receiving standardized nutrition support, n (%)e | 259 (99) | 136 (99) | 122 (98) | 12 (100) |
Feeding tube, n (%) | 149 (57) | 39 (28) | 110 (89) | 10 (83) |
Oral, n (%) | 113 (43) | 105 (77) | 0 (0) | 1 (8) |
Total parenteral nutrition, n (%) | 16 (6) | 5 (4) | 12 (10) | 2 (17) |
Estimated energy in nutrition provided within last 24 hr (kcal/d), n (%) | ||||
< 1,000 | 24 (9) | 12 (9) | 12 (10) | 2 (17) |
1,000 to < 1,500 | 118 (45) | 76 (55) | 42 (34) | 3 (25) |
1,500 to < 2,000 | 105 (40) | 45 (33) | 60 (48) | 6 (50) |
≥ 2,000 | 15 (6) | 4 (3) | 10 (8) | 1 (8) |
Estimated protein in nutrition provided within last 24 hr (g/kg/d), n (%) | ||||
< 1.2 g/kg/d | 157 (60) | 82 (60) | 75 (60) | 7 (58) |
≥ 1.2 g/kg/d | 105 (40) | 55 (40) | 49 (40) | 5 (42) |
Patients receiving a regular standardized sleep assessment, n (%) | 86 (33) | 34 (25) | 51 (41) | 6 (50) |
Patients receiving ICU diary, n (%) | 65 (25) | 50 (36) | 15 (12) | 1 (8) |
Patients placed in physical restraints at any time on the survey date, n (%) | 58 (22) | 23 (17) | 35 (28) | 5 (42) |
ECMO = extracorporeal membrane oxygenation.
aOperational definitions of each element of the ABCDEF bundle are shown in Supplemental Table 3 (http://links.lww.com/CCX/A515).
bPercentages were calculated by dividing by the numbers of sedated patients. The total number of sedated patients was 102. Twenty-one patients who were sedated were not receiving mechanical ventilation or extracorporeal membrane oxygenation. Seventy-nine sedated patients were receiving mechanical ventilation and 10 were receiving extracorporeal membrane oxygenation.
cPercentage was calculated by dividing by the number of the patients on mechanical ventilation or 124.
dThe highest exercise/rehabilitation level performed at the survey date was assessed based on the ICU Mobility Scale (26). IMS 0: nothing (lying in bed, passive exercise); 1: sitting in bed, exercises in bed; 2: passively moved to chair (no standing); 3: sitting over edge of bed; 4: standing; 5: transferring bed to chair; 6: marching in place (at bedside); 7: walking with assistance of two or more people; 8: walking with assistance of one person; 9: walking independently with a gait aid; 10: walking independently without a gait aid.
eNineteen patients received nutrition via multiple routes. Of those, 15 received enteral and oral nutrition, two received enteral and total parenteral nutrition, and two received oral and parenteral nutrition.
The ABCDEF bundle is a collection of six elements (A, B, C, D, E, and F) which represents an evidence-based approach for clinicians to optimize patients recovery and outcomes in ICU.
Implementation of Other Supportive ICU Care
Most patients received nutrition therapy with 1,000–2,000 kcal/d as their total energy intake via feeding tube or oral intake (Table 3). Protein greater than 1.2 g/kg/d was supplied to fewer than 50% of patients at any time during the ICU stay (Fig. 3). Similar trends for energy and protein provision were seen for patients undergoing MV or ECMO. Sleep was assessed in 33% of patients, ICU diaries were used for 25% of patients, and physical restraints were used in 20% of patients. Approximately half of the patients received either nonpharmacologic (47%: arrange monitor lighting, and 47%: sound limitation) or pharmacologic interventions to promote sleep (20%: benzodiazepines, 24%: nonbenzodiazepines).
Association Between ICU Structure and Implementation of the ABCDEF Bundle and Nutrition Support
In ICUs with protocols for pain and sedation management, more patients received elements A and C, whereas fewer patients received element E in ICUs with the protocols for mobilization/rehabilitation (Supplemental Table 7, http://links.lww.com/CCX/A515). Daily multidisciplinary rounds (Supplemental Table 8, http://links.lww.com/CCX/A515) and a 1:1 nurse-to-patient ratio (Supplemental Table 9, http://links.lww.com/CCX/A515) did not improve the rate of bundle implementation or nutrition therapy. Elements D and E and nutrition therapy were more frequently implemented in an ICU with a 1:2 nurse-to-patient ratio.
In ICUs with more ICU beds assigned exclusively to patients with COVID-19 infections, elements A and D were less frequently implemented, whereas element E and nutrition therapy were more frequently implemented (Supplemental Table 10, http://links.lww.com/CCX/A515). No ICU structure led to the implementation of elements B and F. The presence of dedicated physiotherapists and the ability of physiotherapists to enter the rooms of patients with COVID-19 infections did not improve the implementation of element E. Visiting hours were limited for both patients who received element F and those who did not (Supplemental Table 11, http://links.lww.com/CCX/A515).
DISCUSSION
This worldwide 2-day point prevalence study demonstrated an overall low rate of ABCDEF bundle implementation, inadequate protein intake, and rare use of ICU diaries for patients with COVID-19 infection. However, implementation of evidence-based and supportive ICU care was associated with ICUs that used specific protocols and had a defined number of ICU beds exclusively assigned to patients with COVID-19 infection. Structural elements, such as daily multidisciplinary/multiprofessional rounds and a 1:1 nurse-to-patient ratio, were not associated with implementation.
The delivery of ABCDEF bundle to patients with COVID-19 infection was lower than that shown in nationwide and international prevalence surveys conducted before the pandemic (Supplemental Table 12, http://links.lww.com/CCX/A515) (5, 28). The ABCDEF bundle is a strong evidence-based approach that prevents ICU patients from developing the physical, cognitive, and mental disabilities of postintensive care syndrome (PICS) (29), which has long-lasting effects even after intensive care and hospital discharge. Many studies have demonstrated benefits of single elements of the bundle (1, 5, 6, 8), benefits of synchronistic bundle implementation (30), and the positive effects of complete bundle compliance (100% of elements) (31). Despite ongoing research about the specific outcomes of patients who survive COVID-19 infection (NCT04360538, NCT04508712), it is likely that these patients have problems similar to survivors of other critical illnesses. The findings of this study call for urgent efforts to incorporate the ABCDEF bundle into routine clinical practice, especially as many countries are dealing with a second wave of COVID-19 infections. The serious disabilities and symptoms that persist after COVID-19 infection (13–16) raise concerns about the long-term outcomes associated with PICS induced by COVID-19 infection (17, 18).
Relatively high rates of implementation for elements A, C, and D for patients undergoing MV might reflect the need for intense management of pain, sedation, agitation, and delirium to stabilize symptoms, such as strong spontaneous breathing and coughing (32, 33), and to prevent exacerbation of pulmonary injury by self-inflicted lung injury (34–37). The relatively low prevalence of delirium in this population could be secondary to missed delirium due to low implementation of element D and noneligibility for delirium assessment because of deep sedation with relative frequent use of benzodiazepines as a recent paper reported (38). Deep sedation could also affect the implementation and intensity of mobilization.
Compared with previous reports (39, 40), implementation of element E for patients with COVID-19 infections undergoing MV was quite low. Given the detected barriers to mobility, the complicated pathophysiology of this pulmonary illness with two different phases that require different ventilation strategies (34–37) and the variety of neurologic complications (41) could limit aggressive mobilization of patients. In addition, no increase in mobility level during MV might indicate that strict infectious regulations limited interprofessional involvement and/or led to inadequate supplies of personnel protective equipment or staff time constraints limiting provision of mobility. Additional research is needed to develop the most efficient approach to early rehabilitation of patients with COVID-19 infections.
We found that enteral nutrition was provided to most patients, possibly owing to recent evidence (3, 4, 23, 24), but that protein intake did not reach the target level at any time during the ICU stay. Although a nutrition strategy with adequate energy and protein is recommended to preserve skeletal muscle and function (3, 4) and enhance the benefits of mobility/rehabilitation, protein intake did not often reach the target of 1.2 g/kg/d. The absence of nutritionists under strict infection regulations might hinder provision of adequate protein. Protocol-driven nutrition strategies that focus on providing enriched protein, high-protein enteral formulas (> 20%) (42) and sometimes amino-acid parenteral nutrients if there are concerns on digestive functions associated with COVID-19 (43) must be considered.
The ICU diary is used to supplement the patient’s memory in the ICU and helps mitigate anxiety, depression, and posttraumatic stress disorder (44). Just 20% of ICUs provided diaries, a low rate compared with ICUs in Scandinavia (45). To introduce ICU diaries while considering limitations imposed by serious infections, clinicians might consider a novel strategy, such as electronic ICU diaries shared online or video-based ICU diaries (46).
The introduction of protocols, especially for pain and sedation management, could provide an ICU with a systematic and resource-conserving approach that would facilitate delivery of evidence-based ICU care (5, 8, 11). However, our results showed that presence of a protocol for mobilization did not facilitate implementation of element E, possibly because COVID-19 infection has several complicated mechanisms of lung injury and requires different ventilation strategies in various phases of the illness (34–37). A mobilization protocol for other patient populations might not apply to patients with COVID-19 infection. In this setting, the aggressive involvement of intensivists, which was low in this study, and a specialized mobilization program for patients with COVID-19 infection might facilitate the delivery of safe and efficient rehabilitation with appropriate considerations of risk (47).
Controlling the number of ICU beds might allow staff workload to be adjusted appropriately (48, 49). Having more ICU beds for patients with COVID-19 infection may increase the burden and responsibility of medical staff, making it difficult to implement evidence-based and supportive ICU care, as seen with the poor implementation levels of elements A and D. However, it could also lead to greater implementation of element E and nutrition. Patients might benefit from admission to ICUs with higher capacity by gaining greater access to interprofessional and structured interventions from ICU staff who have more experience treating patients with COVID-19 than those who have treated few patients with COVID-19 (50, 51). The effectiveness of centralization according to the local resources and staffing capacity under a standardized or specialized protocol should be investigated.
Daily multidisciplinary/multiprofessional rounds and a 1:1 nurse-to-patient ratio, which were regarded as important aspects of care (19), might consume excessive time and resources in an ICU. Optimizing distribution of resources according to the clinical needs might be critical to the implementation of evidence-based and supportive ICU care.
This study has several strengths and limitations. Although data were collected from many countries around the world, including locations considered to be COVID-19 infection “hotspots,” the relatively high proportion of data from Japan could introduce a potentially large bias and limit the generalizability of the results to ICUs in other countries. The limited number of patients without a comparison group also hampers application of the results to other ICUs. Instead, this survey used previously defined operational definitions, which could strengthen comparability with prior studies. Second, surveys were conducted at two time points, 1 month apart, to include more data. However, as evidence and recommendations for the care of patients with COVID-19 infection changed rapidly during that time, the policies for ICU care might have changed. For example, an increase in the number of patients who received prone positioning on the second survey and a decrease in its duration may be due in part to a study that showed the positive effect of short-term prone positioning on patients without MV (52). Third, the nature of a point prevalence study prevents us from definitively establishing a causal relationship for factors that facilitate or limit ICU care. Fourth, some of our data were collected based on estimations or recollections by ICU clinicians. Fifth, no clear definition of COVID-19 could potentially include data from suspected patients. Finally, we did not investigate potentially confounding factors that might affect the implementation of evidence-based ICU care, such as extubation rate related to spontaneous breathing trials, consciousness level of patients, frailty, and complications related to COVID-19. The actual extent of overwhelming against hospital capacity was not also investigated. Additional investigation, which includes both patients with and without COVID-19, will be needed to validate these results and determine the influence of the COVID-19 pandemic on the implementation of evidence-based ICU care.
CONCLUSIONS
This worldwide, 2-day point prevalence study revealed a low rate of ABCDEF bundle element implementation, inadequate protein supply, and infrequent use of ICU diaries in patients with COVID-19 infection. Introducing specific protocols and controlling the number of beds exclusively for patients with COVID-19 infection in an ICU may facilitate the delivery of evidence-based and supportive ICU care during the pandemic.
ACKNOWLEDGMENT
We thank the Japanese Society of Intensive Care Medicine, which took on the role as an executive office for this study, and all the collaborating societies: the European Society of Intensive Care Medicine, the World Federation of Intensive and Critical Care, the Indian Society of Critical Care Medicine, and the Covid-19 Critical Care Consortium. We also thank all the investigators from Japan and overseas listed in the Appendix. Finally, we thank Claire Levine, MS, for her editorial assistance with this article.
Supplementary Material
Footnotes
The Saiseikai Utsunomiya Hospital, as the central institution, performed the study in collaboration with overseas intensive care units listed in Appendix 1 (http://links.lww.com/CCX/A512).
Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s website (http://journals.lww.com/ccxjournal).
Some authors report potential conflicts of interest outside of this submitted study. Dr. Nakamura reports personal fees from Abbott Laboratory, Nestle, Terumo, Getinge, Asahi Kasei Pharma, Ono Pharmaceutical, Japan Blood Products Organization, Nihon Pharmaceutical, Otsuka Pharmaceutical, Pfizer, Toray, and Baxter, and grants from Asahi Kasei Pharma outside the submitted work. Dr. Katsukawa receives a salary from the Japanese Society for Early Mobilization (nonprofit society) as a chair (full time) outside the submitted work. Dr. Ely reports grants from the Department of Veterans Affairs and the National Institutes of Health; personal fees from Pfizer, Orion, and Lilly; personal fees from Masimo; and grants from Kohler outside the submitted work. Dr. Inoue reports personal fees from Abbott Laboratory, Teijin Pharma, Nestle, and Nihon Pharmaceutical. Dr. Nishida reports grants from Asahi Kasei Pharma, Ono Pharmaceutical, Baxter, Maruishi Pharmaceutical, Torii Pharmaceutical, Teijin Pharma, Shionogi Pharmaceutical, and Fuso Pharmaceutical outside the submitted work. The remaining authors have disclosed that they do not have any potential conflicts of interest.
REFERENCES
- 1.Marra A, Ely EW, Pandharipande PP, et al. The ABCDEF bundle in critical care. Crit Care Clin. 2017; 33:225–243 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Devlin JW, Skrobik Y, Gélinas C, et al. Clinical practice guidelines for the prevention and management of pain, agitation/sedation, delirium, immobility, and sleep disruption in adult patients in the ICU. Crit Care Med. 2018; 46:e825–e873 [DOI] [PubMed] [Google Scholar]
- 3.Singer P, Blaser AR, Berger MM, et al. ESPEN guideline on clinical nutrition in the intensive care unit. Clin Nutr. 2019; 38:48–79 [DOI] [PubMed] [Google Scholar]
- 4.Sheean P, Gonzalez MC, Prado CM, et al. American Society for Parenteral and Enteral Nutrition clinical guidelines: The validity of body composition assessment in clinical populations. JPEN J Parenter Enteral Nutr. 2020; 44:12–43 [DOI] [PubMed] [Google Scholar]
- 5.Pun BT, Balas MC, Barnes-Daly MA, et al. Caring for critically ill patients with the ABCDEF bundle: Results of the ICU liberation collaborative in over 15,000 adults. Crit Care Med. 2019; 47:3–14 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Ely EW. The ABCDEF bundle: Science and philosophy of how ICU liberation serves patients and families. Crit Care Med. 2017; 45:321–330 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Moraes FDS, Marengo LL, Silva MT, et al. ABCDE and ABCDEF care bundles: A systematic review protocol of the implementation process in intensive care units. Medicine (Baltimore). 2019; 98:e14792. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Stollings JL, Devlin JW, Lin JC, et al. Best practices for conducting interprofessional team rounds to facilitate performance of the ICU liberation (ABCDEF) bundle. Crit Care Med. 2020; 48:562–570 [DOI] [PubMed] [Google Scholar]
- 9.Thornton J. COVID-19: How coronavirus will change the face of general practice forever. BMJ. 2020; 368:m1279. [DOI] [PubMed] [Google Scholar]
- 10.Mareiniss DP. The impending storm: COVID-19, pandemics and our overwhelmed emergency departments. Am J Emerg Med. 2020; 38:1293–1294 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Aziz S, Arabi YM, Alhazzani W, et al. Managing ICU surge during the COVID-19 crisis: Rapid guidelines. Intensive Care Med. 2020; 46:1303–1325 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Woolf SH, Chapman DA, Sabo RT, et al. Excess deaths from COVID-19 and other causes, March-April 2020. JAMA. 2020; 324:510–513 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Halpin SJ, McIvor C, Whyatt G, et al. Postdischarge symptoms and rehabilitation needs in survivors of COVID-19 infection: A cross-sectional evaluation. J Med Virol. 2021; 93:1013–1022 [DOI] [PubMed] [Google Scholar]
- 14.Carfì A, Bernabei R, Landi F; Gemelli Against COVID-19 Post-Acute Care Study Group. Persistent symptoms in patients after acute COVID-19. JAMA. 2020; 324:603–605 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Prescott HC, Girard TD. Recovery from severe COVID-19: Leveraging the lessons of survival from sepsis. JAMA. 2020; 324:739–740 [DOI] [PubMed] [Google Scholar]
- 16.Curci C, Pisano F, Bonacci E, et al. Early rehabilitation in post-acute COVID-19 patients: Data from an Italian COVID-19 rehabilitation unit and proposal of a treatment protocol. A cross-sectional study. Eur J Phys Rehabil Med. 2020; 56:633–641 [DOI] [PubMed] [Google Scholar]
- 17.Biehl M, Sese D. Post-intensive care syndrome and COVID-19—implications post pandemic. Cleve Clin J Med. 2020. August 5. [online ahead of print] [DOI] [PubMed] [Google Scholar]
- 18.Hosey MM, Needham DM. Survivorship after COVID-19 ICU stay. Nat Rev Dis Primers. 2020; 6:60. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Devlin JW, O’Neal HR, Jr, Thomas C, et al. Strategies to optimize ICU liberation (A to F) bundle performance in critically ill adults with coronavirus disease 2019. Crit Care Explor. 2020; 2:e0139. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Kotfis K, Williams Roberson S, Wilson JE, et al. COVID-19: ICU delirium management during SARS-CoV-2 pandemic. Crit Care. 2020; 24:176. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Thomas P, Baldwin C, Bissett B, et al. Physiotherapy management for COVID-19 in the acute hospital setting: Clinical practice recommendations. J Physiother. 2020; 66:73–82 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Salawu A, Green A, Crooks MG, et al. A proposal for multidisciplinary tele-rehabilitation in the assessment and rehabilitation of COVID-19 survivors. Int J Environ Res Public Health. 2020; 17:4890. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Barazzoni R, Bischoff SC, Breda J, et al. ; Endorsed by the ESPEN Council. ESPEN expert statements and practical guidance for nutritional management of individuals with SARS-CoV-2 infection. Clin Nutr. 2020; 39:1631–1638 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Martindale R, Patel JJ, Taylor B, et al. Nutrition therapy in critically ill patients with coronavirus disease 2019. JPEN J Parenter Enteral Nutr. 2020; 44:1174–1184 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Ministry of Health, Labour and Welfare, Japan. Ethical Guidelines for Medical and Health Research Involving Human Subjects. 2015. Available at: https://www.mhlw.go.jp/file/06-Seisakujouhou-10600000-Daijinkanboukouseikagakuka/0000080278.pdf. Accessed May 17, 2020
- 26.Hodgson C, Needham D, Haines K, et al. Feasibility and inter-rater reliability of the ICU mobility scale. Heart Lung. 2014; 43:19–24 [DOI] [PubMed] [Google Scholar]
- 27.Kanda Y. Investigation of the freely available easy-to-use software ‘EZR’ for medical statistics. Bone Marrow Transplant. 2013; 48:452–458 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Morandi A, Piva S, Ely WE, et al. Worldwide ABCDEF (assessing pain both spontaneous awakening and breathing trials, choice of drugs, delirium monitoring/management, early exercise/mobility, and family empowerment). Crit Care Med. 2017; 45:e1111–1122 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Balas MC, Pun BT, Pasero C, et al. Common challenges to effective ABCDEF bundle implementation: The ICU liberation campaign experience. Crit Care Nurse. 2019; 39:46–60 [DOI] [PubMed] [Google Scholar]
- 30.Hsieh SJ, Otusanya O, Gershengorn HB, et al. Staged implementation of awakening and breathing, coordination, delirium monitoring and management, and early mobilization bundle improves patient outcomes and reduces hospital costs. Crit Care Med. 2019; 47:885–893 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Barnes-Daly MA, Phillips G, Ely EW. Improving hospital survival and reducing brain dysfunction at seven California community hospitals: Implementing PAD guidelines via the ABCDEF bundle in 6,064 patients. Crit Care Med. 2017; 45:171–178 [DOI] [PubMed] [Google Scholar]
- 32.Fan E, Beitler JR, Brochard L, et al. COVID-19-associated acute respiratory distress syndrome: Is a different approach to management warranted? Lancet Respir Med. 2020; 8:816–821 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Wiersinga WJ, Rhodes A, Cheng AC, et al. Pathophysiology, transmission, diagnosis, and treatment of coronavirus disease 2019 (COVID-19): A review. JAMA. 2020; 324:782–793 [DOI] [PubMed] [Google Scholar]
- 34.Gattinoni L, Chiumello D, Caironi P, et al. COVID-19 pneumonia: Different respiratory treatments for different phenotypes? Intensive Care Med. 2020; 46:1099–1102 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Yoshida T, Fujino Y, Amato MB, et al. Fifty years of research in ARDS. Spontaneous breathing during mechanical ventilation. Risks, mechanisms, and management. Am J Respir Crit Care Med. 2017; 195:985–992 [DOI] [PubMed] [Google Scholar]
- 36.Cruces P, Retamal J, Hurtado DE, et al. A physiological approach to understand the role of respiratory effort in the progression of lung injury in SARS-CoV-2 infection. Crit Care. 2020; 24:494. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Marini JJ, Gattinoni L. Management of COVID-19 respiratory distress. JAMA. 2020; 323:2329–2330 [DOI] [PubMed] [Google Scholar]
- 38.Pun BT, Badenes R, La Calle GH, et al. Prevalence and risk factors for delirium in critically ill patients with COVID-19 (COVID-D): A multicentre cohort study. Lancet Respir Med. 2021. January 8. [online ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Lee H, Ko YJ, Suh GY, et al. Safety profile and feasibility of early physical therapy and mobility for critically ill patients in the medical intensive care unit: Beginning experiences in Korea. J Crit Care. 2015; 30:673–677 [DOI] [PubMed] [Google Scholar]
- 40.Jolley SE, Moss M, Needham DM, et al. ; Acute Respiratory Distress Syndrome Network Investigators. Point prevalence study of mobilization practices for acute respiratory failure patients in the United States. Crit Care Med. 2017; 45:205–215 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Varatharaj A, Thomas N, Ellul MA, et al. ; CoroNerve Study Group. Neurological and neuropsychiatric complications of COVID-19 in 153 patients: A UK-wide surveillance study. Lancet Psychiatry. 2020; 7:875–882 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Deutz NE, Bauer JM, Barazzoni R, et al. Protein intake and exercise for optimal muscle function with aging: Recommendations from the ESPEN Expert Group. Clin Nutr. 2014; 33:929–936 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Gupta A, Madhavan MV, Sehgal K, et al. Extrapulmonary manifestations of COVID-19. Nat Med. 2020; 26:1017–1032 [DOI] [PubMed] [Google Scholar]
- 44.Barreto BB, Luz M, Rios MNO, et al. The impact of intensive care unit diaries on patients’ and relatives’ outcomes: A systematic review and meta-analysis. Crit Care. 2019; 23:411. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Egerod I, Storli SL, Åkerman E. Intensive care patient diaries in Scandinavia: A comparative study of emergence and evolution. Nurs Inq. 2011; 18:235–246 [DOI] [PubMed] [Google Scholar]
- 46.Jones C, Bäckman C, Egerod I, et al. Report on third international intensive aftercare conference in Norrköping, Sweden. Nurs Crit Care. 2015; 20:271–273 [DOI] [PubMed] [Google Scholar]
- 47.Liu K, Ogura T, Takahashi K, et al. The safety of a novel early mobilization protocol conducted by ICU physicians: A prospective observational study. J Intensive Care. 2018; 6:10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Litton E, Bucci T, Chavan S, et al. Surge capacity of intensive care units in case of acute increase in demand caused by COVID-19 in Australia. Med J Aust. 2020; 212:463–467 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Phua J, Weng L, Ling L, et al. Intensive care management of coronavirus disease 2019 (COVID-19): Challenges and recommendations. Lancet Respir Med. 2020; 8:506–517 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Ostermann M, Vincent JL. How much centralization of critical care services in the era of telemedicine? Crit Care. 2019; 23:423. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Peek GJ, Mugford M, Tiruvoipati R, et al. ; CESAR Trial Collaboration. Efficacy and economic assessment of conventional ventilatory support versus extracorporeal membrane oxygenation for severe adult respiratory failure (CESAR): A multicentre randomised controlled trial. Lancet. 2009; 374:1351–1363 [DOI] [PubMed] [Google Scholar]
- 52.Ding L, Wang L, Ma W, et al. Efficacy and safety of early prone positioning combined with HFNC or NIV in moderate to severe ARDS: A multi-center prospective cohort study. Crit Care. 2020; 24:28. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.