Abstract
Objective
Surveys are often used to assess intensive care unit (ICU) organizational characteristics for quality improvement. Typically these surveys target ICU nurse managers and/or physician directors. However, it is unclear whether these providers’ assessments differ. We sought to determine whether differences existed in nurse- and physician-assessed ICU characteristics using a standardized survey.
Design
We administered a previously developed survey to nurse managers and medical directors in adult ICUs within a single healthcare system in 2013. The survey asked about interprofessional staffing and evidence-based protocols. We examined differences between nurse managers’ and medical directors’ responses using McNemar's test and assessed concordance using the kappa statistic.
Setting
Twenty-three ICUs in 10 hospitals in Southwestern Pennsylvania.
Results
Sixteen (69%) were specialty ICUs. The median number of ICU beds was 34. Concordance was moderate for high- vs. low-intensity physician staffing (κ = 0.60) and almost perfect on questions related to interprofessional staffing (κ = 0.83 nurse practitioners/physician assistants; 1.0 respiratory therapists; 0.83 physical therapists). However, concordance was slight to fair with regard to the presence of these providers on rounds (κ = 0.20–0.21) and poor to slight for protocols for liberation from mechanical ventilation (κ = 0.19), sedation (κ = −0.03) and central line insertion (κ = −0.03).
Conclusions
Despite a standardized survey, we found substantial disagreement on ICU characteristics when assessed by the nurse manager or physician director. This study raises questions about the use of surveys to examine ICU organizational characteristics and suggests that differences in nurse managers’ and medical directors’ assessments could be helpful in guiding future ICU quality improvement projects.
Keywords: quality improvement; surveys; general methodology, intensive care; quality measurement
Introduction
One strategy for improving quality of care for critically ill patients is to target the organization of the intensive care unit (ICU). Many ICU organizational characteristics are associated with improved outcomes, including high-intensity physician staffing [1], daily multidisciplinary rounds [2], protocolized care [3] and checklists [4]. Understanding the patterns of use for these practices is important for quality improvement and research into the organizational determinants of the ICU outcome.
A typical way to assess organizational characteristics is through surveys, asking nurses or physicians about current organization [2, 5, 6]. Surveys can provide detailed assessments of practice across large numbers of hospitals but can also be subjective and influenced by an individual's perspective or motivation [7]. In turn, surveys about an individual's organization and not their own health behaviors may lead to systematically different responses dependent upon the individual [7]. In the ICU, it is unknown whether the most often surveyed providers—nurse managers and medical directors—perceive organizational characteristics differently and, consequently, respond differently. Thus, we sought to determine whether differences existed in nurse manager- and medical director-assessed responses using a standardized ICU survey.
Methods
Study design and sample
We surveyed 28 adult ICUs in a healthcare system in southwestern Pennsylvania during 2013 about their organizational characteristics. The survey was part of a quality improvement initiative designed to understand and standardize ICU practice across the system. The survey sample was all ICU nurse managers and ICU physician directors within the system. We used a previously developed survey derived from a series of clinician interviews [5, 6]. We developed the survey from a literature review and qualitative content analysis of 64 interviews with multidisciplinary clinicians at non-study ICUs. [8] We drafted the survey to identify key objective ICU organizational characteristics and then piloted the survey with 12 ICU clinicians at three non-study hospitals. A system-wide ICU quality improvement committee identified a single nurse manager and physician director in each ICU and supplied their email addresses. We emailed the surveys using a commercially available electronic survey tool (Survey Monkey Inc., Seattle, WA, USA) over 6 weeks. We sent four rounds of email requests, beginning on 10 October 2013 and ending on 26 November 2013.
ICU and hospital characteristics
We obtained ICU and hospital characteristics from the survey and from the Centers for Medicare and Medicaid Healthcare Cost Report Information System (HCRIS) dataset, a national repository of hospital and ICU bed data. From the survey, we obtained information on ICU type, asking respondents to identify the patient population of their respective unit. Using these responses, we categorized ICUs into mixed medical/surgical ICUs vs. specialty ICUs (only medical, only surgical, only cardiac, only transplant or only patients with neurological diagnoses). From HCRIS, we obtained objective data regarding the number of hospital beds, ICU beds and teaching status. We categorized hospital teaching status based on resident to bed ratios: major (≥0.25), minor (<0.25) and none.
Survey domains
The survey asked about clinician staffing, rounding patterns and the use of care protocols and checklists. A complete copy of the survey is available in the online appendix.
Clinician staffing
We asked about intensivist physician staffing patterns (high- vs. low-intensity model) [1], type of clinician staffing during the day [physician trainees, nurse practitioner/physician assistants (NP/PA)] [6] and type of night-time staffing (intensivists, physician trainees, NPs/PAs) [5]. We also asked whether respiratory therapists, clinical pharmacists, nutritionists, social workers or physical therapists were involved in care.
Rounding patterns
We asked about daily multidisciplinary rounds [2] and what providers participated in rounds in addition to the physician and bedside nurse (i.e. respiratory therapists, clinical pharmacists, nutritionists, social workers and physical therapists).
Protocols and checklists
We asked about the presence of four clinical protocols: a protocol for liberation from mechanical ventilation [9], a protocol for lung-protective ventilation [10], a sedation protocol and a daily interruption of sedation protocol [3]. We also asked about delirium screening [11], checklists for central line insertions [12] and checklist during morning rounds [4].
Analysis
We included only ICUs that had complete data, i.e. both the nurse manager and medical director for the ICU responded to the survey. We used summary statistics to describe the sample. We examined survey responses by provider type, assessing the prevalence of each ICU organizational characteristic in the sample according to each respondent type. We used McNemar's test, a significance test for paired nominal data, to evaluate whether there were significant differences in the overall prevalence of each characteristic as assessed by nurse managers and medical directors.
We used the kappa statistic to examine agreement between nurse managers and medical directors at the ICU level, taking into consideration what would be observed by chance. We used Landis & Koch's standard [13] for the strength of agreement for the kappa coefficient: ≤0 = poor; 0.01–0.20 = slight; 0.21–0.40 = fair; 0.41–0.60 = moderate; 0.61–0.80 = substantial and 0.81–1 = almost perfect.
Statistical analyses were performed with Stata 13.0 (College Station, TX, USA). This study was a quality improvement project, as its primary goal was to measure current ICU practices in one healthcare system.
Results
We surveyed 28 adult ICUs in 11 hospitals in the healthcare system. We received responses from 25 ICUs in 11 hospitals, yielding a response rate of 89%. We excluded 2 ICUs from 1 hospital with responses from only one respondent, leaving 23 ICUs in 10 hospitals in the final sample. ICU and hospital characteristics are shown in Table 1.
Table 1.
Hospital and ICU characteristics
| Hospital beds, median (IQR) (n = 9)a | 303 (266–341) |
| ICU beds, median (IQR) | 34 (29–38) |
| Hospital teaching status | |
| Major | 6 (60%) |
| Minor | 1 (10%) |
| None | 2 (25%) |
| ICU unit type (n = 23) | |
| Mixed | 7 (30%) |
| Specialty | 16 (69%) |
aOnly nine hospitals are represented in HCRIS data. One hospital was built in 2012 and thus not identified in 2011 HCRIS data.
Table 2 shows the survey responses by the provider types at the group level, which should be interpreted as the prevalence of each of these factors in the sample according to each respondent group. Medical directors were significantly more likely to report high-intensity physician staffing (P = 0.05), night-time physician trainees (P = 0.05) and a lung-protective ventilation protocol (P = 0.03) than nurse managers. Similar proportions were reported for clinician staffing (i.e. respiratory therapy involvement in care, pharmacist involvement in care) and protocols and screening (i.e. sedation protocols, central line insertion checklist and delirium screening).
Table 2.
Group-level survey responses by provider type: nurse manager and medical directora
| Nurse manager response | Medical director response | P-value | |
|---|---|---|---|
| Intensivist physician staffing (high vs. low) | 14 (60.9%) | 18 (78.3%) | 0.05 |
| Weekday providers | |||
| Physician trainees | 19 (82.6%) | 17 (73.9%) | 0.32 |
| Nurse practitioner/Physician assistants | 13 (56.5%) | 11 (47.8%) | 0.15 |
| Night-time providers | |||
| Intensivists | 20 (86.9%) | 19 (82.61%) | 0.57 |
| Physician trainees | 19 (82.6%) | 15 (65.2%) | 0.05 |
| Nurse practitioner/Physician assistants | 6 (26.1%) | 5 (21.7%) | 0.32 |
| Clinicians involved in patient care | |||
| Respiratory therapistsb | 23 (100%) | 23 (100%) | – |
| Clinical pharmacists | 21 (91.3%) | 23 (100%) | 0.16 |
| Nutritionists | 18 (78.3%) | 21 (91.3%) | 0.18 |
| Social worker | 18 (78.3%) | 18 (78.3%) | 1.00 |
| Physical therapist | 22 (95.7%) | 19 (82.6%) | 0.08 |
| Daily multidisciplinary rounds | 20 (86.9%) | 19 (82.6%) | 0.32 |
| Daily rounds with | |||
| Respiratory therapy | 14 (60.9%) | 16 (69.6%) | 0.48 |
| Clinical pharmacy | 17 (73.9%) | 17 (73.9%) | 1.00 |
| Nutrition | 6 (26.1%) | 8 (34.8%) | 0.32 |
| Social work | 9 (39.1%) | 11 (47.8%) | 0.48 |
| Physical therapy | 3 (13.0%) | 3 (13.0%) | 1.00 |
| Protocols | |||
| Liberation from mechanical ventilation | 19 (82.6%) | 21 (91.3%) | 0.41 |
| Lung-protective ventilation strategy | 11 (47.8%) | 17 (73.9%) | 0.03 |
| Sedation | 14 (60.9%) | 20 (86.9%) | 0.06 |
| Daily interruption of sedation | 21 (91.3%) | 22 (95.7%) | 0.32 |
| Checklists | |||
| Central line insertions | 16 (69.6%) | 16 (69.6%) | 1.00 |
| Checklist for morning rounds | 12 (52.2%) | 14 (60.9%) | 0.32 |
| Delirium screening | 5 (21.7%) | 3 (13.0%) | 0.32 |
aResponses above describe group-level summary statistics for each organizational characteristic and do not describe percent agreement. This should be interpreted as the prevalence of the organizational characteristics in the sample according to each respondent group.
bMcNemar's test was not applicable for respiratory therapists involved in care as all indicated that these clinicians were involved and thus no testable difference.
Concordance varied substantially between nurse managers and medical directors (Table 3). Concordance was moderate for physician staffing [high vs. low intensivist staffing (κ = 0.60), the presence of physician trainees during the week (κ = 0.49) or at night (κ = 0.57) and night-time intensivist staffing (κ = 0.49)]. We found almost perfect concordance on questions related to interprofessional staffing [(NP/PA staffing during the week (κ = 0.83) and at night (κ = 0.88)], respiratory therapists involvement in clinical care (κ = 1.00), pharmacist involvement in care (κ = 0.83) and daily multidisciplinary rounds (κ = 0.83)]. However, concordance was slight to fair for the presence of providers on rounds (κ = 0.20–0.21) and poor to slight for the presence of protocols for liberation from mechanical ventilation (κ = 0.19), sedation (κ = −0.03) and a central line insertion checklist (κ = −0.03).
Table 3.
Agreement and reliability (kappa statistic) between nurse and physician responses on key ICU organizational characteristics
| % Agreement | Kappa | |
|---|---|---|
| Intensivist physician staffing (high vs. low) | 82.60 | 0.60 |
| Weekday providers | ||
| Physician trainees | 82.60 | 0.49 |
| Nurse practitioner/Physician assistants | 91.30 | 0.83 |
| Night-time providers | ||
| Intensivists | 86.96 | 0.49 |
| Physician trainees | 82.60 | 0.57 |
| Nurse practitioner/Physician assistants | 96.65 | 0.88 |
| Clinicians involved in patient care | ||
| Respiratory therapists | 100.00 | 1.00 |
| Clinical pharmacists | 91.30 | 0.00 |
| Nutritionists | 78.26 | 0.18 |
| Social worker | 73.91 | 0.21 |
| Physical therapist | 86.96 | 0.36 |
| Daily multidisciplinary rounds | 95.65 | 0.83 |
| Daily rounds with | ||
| Respiratory therapy | 65.22 | 0.20 |
| Clinical pharmacy | 73.91 | 0.21 |
| Nutrition | 82.61 | 0.20 |
| Social work | 65.22 | 0.21 |
| Physical therapy | 82.61 | 0.21 |
| Protocols | ||
| Liberation from mechanical ventilation | 73.91 | 0.19 |
| Lung-protective ventilation strategy | 65.22 | 0.18 |
| Sedation | 56.52 | −0.03 |
| Daily interruption of sedation | 95.65 | 0.65 |
| Checklists | ||
| Central line insertions | 56.52 | −0.03 |
| Checklist for morning rounds | 82.61 | 0.65 |
| Delirium screening | 82.61 | 0.4 |
Discussion
Using a standardized survey in one healthcare system, we found considerable variation between nurse manager's and medical director's responses. Medical directors were more likely to report high-intensity physician staffing, night-time physician trainees and availability of a lung-protective ventilation protocol than nurse managers. Concordance among providers was highest for questions about human resources but was much lower for questions about available care practices and protocols. These data raise questions about the use of surveys to examine ICU organizational characteristics.
Surveys are a common method to assess healthcare organizational practices in both quality improvement and research. Among the benefits of surveys are that they can be administered to large samples. However, they can be influenced by non-response bias and recall bias [14]. Our results suggest that concordance between provider types may depend on the type of organizational characteristic as well—structure vs. process [15]. We found that nurse managers and medical directors were in agreement most when responding to questions about staffing, typically structural measures [15]. In contrast, we found poor concordance for questions pertaining to process, i.e. what practices and protocols were present [15].
Most notably, we find that survey results differed based on the role and discipline of respondent, even for seemingly objective factors like ICU organizational practices. This lack of concordance between nurse managers and medical directors on certain ICU characteristics may be consistent with potential differences in training, background and occupational responsibilities. This discordance could form the basis for future quality improvement efforts on the ICU. For example, to the degree that nurse and physician respondents differ on their assessment, it could represent a communication gap surrounding the use of key protocols and pathways, identifying areas for improvement.
Interestingly, medical directors were more likely to report high-intensity staffing, the presence of trainees at night and the use of lung-protective ventilation protocols than nurse managers. Medical directors by definition are involved in the staffing and supervision of physicians and trainees in academic hospitals. They are also involved in incorporating evidence-based practice in their ICUs and thus may be more up to date on current practices. However, nurse managers may be more up to date on factors directly related to nursing practice, such as delirium screening.
Nonetheless, the differences in assessments of ICU organizational characteristics based on occupational responsibility and training could be leveraged for more efficient quality improvement (QI) projects. A range of factors facilitate effective implementation of quality improvement projects [16, 17], yet it is infeasible to address all facilitators in every quality improvement project. Among the most important ingredients for QI success though are human resources and leadership [16]. Given the wide variation in responses to care processes between providers, partnering with both the medical director and nurse manager to design and facilitate a quality improvement project to improve ventilator management or another care process would be ideal; each clinician in that scenario brings a unique and distinct perspective. This approach could make design and implementation of QI projects more efficient and robust, ultimately improving care and outcomes.
We acknowledge several limitations of our study. This study was conducted in one healthcare system and only adult ICUs. Findings may not be generalizable to other hospitals or pediatric ICUs. We were also unable to address nesting of nurse managers and medical directors within ICUs and then hospitals, or test for systematic differences in concordance by ICU characteristics, due to our small sample size. We surveyed only nurse managers and medical directors, not bedside staff. We made no attempt to determine the ‘right’ answer and instead focused on differences between provider assessments. We are also unable to adjust for demographic differences between nurse managers and medical directors that could influence their assessments.
Conclusion
We found varying agreement between nurse managers and medical directors on ICU organizational characteristics in one healthcare system using a standardized survey. Nurse managers and medical directors were in agreement most when responding to question about ICU staffing. We found less concordance in nurse managers’ and medical directors’ responses to questions about care protocols and processes. This finding calls into question the use of surveys as an isolated modality to assess ICU organizational characteristics for quality improvement and research. This finding also provides evidence for the need to incorporate both nurse managers and medical directors in quality improvement projects, as they appear to offer unique perspectives on care and practice in the ICU setting. Incorporating these results to better inform future quality improvement efforts, to leverage the knowledge and skills of each respective clinician, would serve future efforts to improve care and outcomes of critically ill adults.
Funding
This work was supported by the National Institutes of Health (T32HL007820 and K23HL082650). D.K.C. acknowledges funding from National Institutes of Health, National Heart, Lung and Blood Institute (T32HL007820). J.M.K. acknowledges funding from National Institutes of Health, National Heart, Lung and Blood Institute (K23HL082650), The Gordon & Betty Moore Foundation and in-kind research support from the Cerner Corporation.
References
- 1.Pronovost PJ, Angus DC, Dorman T, et al. Physician staffing patterns and clinical outcomes in critically ill patients: a systematic review. JAMA 2002;288:2151–62. [DOI] [PubMed] [Google Scholar]
- 2.Kim MM, Barnato AE, Angus DC, et al. The effect of multidisciplinary care teams on intensive care unit mortality. Arch Intern Med 2010;170:369–76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Mehta S, Burry L, Cook D, et al. Daily sedation interruption in mechanically ventilated critically Ill patients cared for with a sedation protocol: a randomized controlled trial. JAMA 2012;308:1985. [DOI] [PubMed] [Google Scholar]
- 4.Weiss CH, Moazed F, McEvoy CA, et al. Prompting physicians to address a daily checklist and process of care and clinical outcomes: a single-site study. Am J Respir Crit Care Med 2011;184:680–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Wallace DJ, Angus DC, Barnato AE, et al. Nighttime intensivist staffing and mortality among critically ill patients. N Engl J Med 2012;366:2093–101. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Costa DK, Wallace DJ, Barnato AE, et al. Nurse practitioner/physician assistant staffing and critical care mortality. Chest 2014;146:1566–73. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Biemer PP, Groves RM, Lyberg LE, et al. Measurement Error in Surveys. Hoboken, NJ: Wiley & Sons, 1991. [Google Scholar]
- 8.Costa DK, Barg FK, Asch DA, et al. Facilitators of an interprofessional approach to care in medical and mixed medical/surgical ICUs: a multicenter qualitative study. Res Nurs Health 2014;37:326–35. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Ely EW, Baker AM, Dunagan DP, et al. Effect on the duration of mechanical ventilation of identifying patients capable of breathing spontaneously. N Engl J Med 1996;335:1864–9. [DOI] [PubMed] [Google Scholar]
- 10.The Acute Respiratory Distress Syndrome Network. Ventilation with lower tidal volumes as compared with traditional tidal volumes for acute lung injury and acute respiratory distress syndrome. N Engl J Med 2000;342:1301–8. [DOI] [PubMed] [Google Scholar]
- 11.Devlin JW, Fong JJ, Schumaker G, et al. Use of a validated delirium assessment tool improves the ability of physicians to identify delirium in medical intensive care unit patients. Crit Care Med 2007;35:2721–4. [DOI] [PubMed] [Google Scholar]
- 12.Pronovost PJ, Needham DM, Berenholtz SM, et al. An intervention to decrease catheter-related bloodstream infections in the ICU. N Engl J Med 2006;355:2725–32. [DOI] [PubMed] [Google Scholar]
- 13.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–74. [PubMed] [Google Scholar]
- 14.Villar A. Response bias. In: Lavrakas PJ. (ed). Encyclopedia of Survey Research Methods. Thousand Oaks, CA: Sage Publications, Inc., 2008,752–4. [Google Scholar]
- 15.Donabedian A. The quality of medical care. Science 1978;200:856–64. [DOI] [PubMed] [Google Scholar]
- 16.Braithwaite J, Marks D, Taylor N. Harnessing implementation science to improve care quality and patient safety: a systematic review of targeted literature. Int J Qual Heal Care 2014;26:321–9. [DOI] [PubMed] [Google Scholar]
- 17.Kowitlawakul Y, Leong BSH, Lua A, et al. Observation of handover process in an intensive care unit (ICU): barriers and quality improvement strategy. Int J Qual Health Care 2015;27:99–104. [DOI] [PubMed] [Google Scholar]