Climate of Respect Evaluation in Intensive Care Units: Development of an Instrument (CORE-ICU)

Abstract

Objective

To develop a valid, reliable measure that reflected the environment of respectfulness within the Intensive Care Unit (ICU) setting.

Design

We developed a preliminary survey instrument based on conceptual domains of respect identified through prior qualitative analyses of ICU patient, family member and clinician perspectives. The initial instrument consisted of 21 items. After 5 cognitive interviews and 16 pilot surveys, we revised the instrument to include 23 items. We used standard psychometric methods to analyze the instrument.

Setting

8 ICUs serving adult patients affiliated with a large university health system.

Subjects

ICU clinicians

Interventions

N/A

Measurements and Main Results

Based on 249 responses we identified three factors and created subscales: general respect, respectful behaviors, and disrespectful behaviors. The general respect subscale had 7 items (α=0.932) and reflected how often patients in the ICU are treated with respect, in a dignified manner, as an individual, equally to all other patients, on the ‘same level’ as the ICU team, as a person, and as you yourself would want to be treated. The respectful behaviors subscale had 10 items (α=0.926) and reflected how often the ICU team responds to patient and/or family anxiety, makes an effort to get to know the patient and family as people, listens carefully, explains things thoroughly, gives the opportunity to provide input into care, protects patient modesty, greets when entering room, and talks to sedated patients. The subscale measuring disrespect has 4 items (α=0.702) and reflects how often the ICU team dismisses family concerns, talks down to patients and families, speaks disrespectfully behind their backs, and gets frustrated with patients and families.

Conclusions

We created a reliable set of scales to measure the climate of respectfulness in intensive care settings. These measures can be used for ongoing quality improvement that aim to enhance the experience of ICU patients and their families.

INTRODUCTION

Over the past several decades, the medical community has gained an appreciation for the role organizational culture contributes to individual clinician behaviors and the quality of patient care. Most notably, the Institute of Medicine recommended in 2000 that hospitals improve their ‘culture of safety.’¹ Subsequently reliable and valid instruments have been developed to measure and track changes in the safety climate over time and in response to quality improvement interventions.^2-4 More recently, Martinson et al. developed a measure of organizational climate focused on research integrity.^{5, 6} Measuring organizational culture in any domain (e.g., safety, research integrity) allows institutions to assess perceptions of local attitudes and practices to help meet the important goal of improving areas that are in need of attention.

Respect for persons is a core principle of ethical clinical practice⁷, which is of special relevance in the intensive care unit (ICU) setting.^8-11 Patients, families, and ICU clinicians are all affected by the environmental intensity, albeit in different ways. Patients are (by definition) severely ill, injured, or have undergone life-threatening surgeries; they are often intubated, bedridden, dressed in a hospital gown, and unable to speak or present themselves as their own individual selves. Families may be frightened and/or grieving (potential) loss, and unaccustomed to seeing their loved ones in such a manner. ICU clinicians experience a high level of burnout including depersonalization,¹² and the cynicism that results may be contagious to others. Because ICUs are relatively closed units with many of the same staff members working in close proximity to one another, each unit can easily develop its own culture.

For all these reasons, evaluating the climate of respect within an ICU is an important step in assessing and if necessary improving the quality of care and patient, family, and staff experiences. We have previously conducted qualitative and quantitative research with patients and families, and with ICU staff, to develop an understanding of important domains of respect within the ICU setting. We have also developed a systematic approach to directly observing the care in ICUs as well as an ICU patient (or family) reported measure of respect.^{9, 13} However, like all patient experience measures, patient and family reports are highly skewed and there are dimensions of a respectful ICU environment identified in our clinician focus groups that may not be observed by patients/families (e.g., talking about patients behind their back in a demeaning manner). Therefore, the aim of this study was to develop a clinician-reported measure of the ICU’s climate of respect.

METHODS

This study was reviewed and approved by a Johns Hopkins Medicine Institutional Review Board. The development of the ICU culture of respect evaluation (ICU-CORE) measure was accomplished in 3 phases.

Phase 1: Development of a draft instrument

We developed a preliminary survey instrument based on conceptual domains of respect identified through interviews with patients and family members and focus groups with ICU clinicians, both described in detail elsewhere.^{8, 10} The study team reviewed the instrument internally and made revisions based on experience studying respect and dignity in ICU settings using a patient/family survey and direct observation.^{9, 13, 14} The initial instrument consisted of 21 items.

The research team conducted five cognitive interviews with clinicians, including two nurses, a physician, a physical therapist, and a service coordinator. Based on the cognitive interviews, the research team revised the survey instrument to reduce the use of clinical terminology (e.g., changing “alleviating pain” to “controlling pain”) and restructuring questions to avoid confusion or bias (e.g., alternate questions assessing positive and negative actions instead of grouping them together). In addition, two questions were considered double-barreled and rewritten as 4 questions. The revised survey instrument included 23 items. Response options for the ICU climate items consisted of a six point scale rating the frequency of events (never, rarely, some of the time, most of the time, nearly all of the time with rare exceptions, and all of the time without exception).

Sixteen pilot surveys were collected in a single ICU in August and September 2015. After reviewing those preliminary responses, we found variability across all response options for all respect items and did not make any further changes. However the survey was slightly revised to capture additional demographic data (described below).

Phase 2: Data collection with draft instrument

Eight ICUs from two hospitals (Johns Hopkins Hospital and Johns Hopkins Bayview Medical Center) within the same urban health system that all delivered care to adults were recruited to participate in our research on treatment with respect and dignity. These included two surgical ICUs, two medical ICUs, and four specialty (cancer, neurologic, burn, and cardiac) ICUs. The research team contacted leaders at each ICU (e.g., medical directors or nurse coordinators) to obtain lists of clinicians who had patient contact on the particular ICU within the previous year. All surveys were administered using RedCap Version 6.8.2.¹⁵ Data were collected via an anonymous link to the survey sent by email to 792 clinicians in 8 units between November 2015 and February 2016. Completion of the survey served as consent to participate.

Phase 3: Statistical analysis and psychometric testing of the instrument

We exported data from RedCap to Stata 14 for analysis.¹⁶ We first examined frequency distributions and found that all items had reasonable variability in terms of clinician responses. Following standard psychometric methods,^17-19 we verified that items hypothesized to assess the same domain demonstrated at least a moderate and positive correlation (r> 0.30) to the underlying scale (item-test correlation) and have factor loadings > 0.4 based on exploratory factor analysis. Further, if Cronbach’s alpha, a measure of internal consistency reliability that ranges from 0 to 1.0, for a scale substantially increases when an item is removed, that is an indication that the item should be deleted. Items that did not meet these standards were not retained. We calculated Cronbach’s alpha for each of the final scales; scales with alphas > 0.70 are generally accepted as having adequate reliability.

Additional Data and Analyses

In addition to the respect items, the instrument asked respondents to report their gender, race/ethnicity, professional background/discipline, and to indicate in which of the eight participating ICUs across the two hospitals they primarily worked. We used descriptive methods to characterize our sample and to examine possible differences in respect scores by these characteristics using linear regression. Because there were 3 subscales each compared across 4 different respondent-level variables, we used the Bonferroni method²⁰ to create a cut-off for statistical significance of 0.004.

RESULTS

Study Sample

We received 249 responses to our survey, representing 31.4% of the total clinician population that works in all the adult intensive care units at Johns Hopkins Hospital and Johns Hopkins Bayview Medical Center. Most (n=144, 58%) of our clinician respondents were nurses, with 45 physicians (18%) and 60 additional clinicians of various disciplines such as physical and occupational therapy, respiratory therapy and social work. Most clinicians identified as female (80%) and as white (77%). Clinicians represented medical (n=53, 22%), surgical (n=87, 35%), and specialty ICU settings (n=106, 43%).

Respect Scales

Psychometric analyses produced three factors: overall respect (7 items), respectful behaviors (10 items), and disrespectful behaviors (4 items). The items making up each of these scales are shown in Table 1 with the full range of responses. The factor loadings, item-test correlations, and alpha coefficient if the item was removed from the scale are included as supplemental digital content. The overall respect and respectful behaviors scales each had excellent internal consistency (α=0.9315 for the general respect scale and α=0.9256 for the respectful behaviors scale) and the disrespectful behaviors scale had adequate internal consistency (α=0.7016).

Table 1.

Sub-scales, items, factor loadings, Item-to-test correlations, and alpha values if removed for items in ICU-CORE (N=249)

Scale	Item	All of the time without exception	Nearly all the time with rare exception	Most of the time	Some of the	Rarely	Never
Overall Respect	With respect	67 (27%)	127 (51%)	52 (21%)	3 (1%)	0 (0%)	0 (0%)
α=0.9315	In a dignified manner	57 (23%)	121 (49%)	61 (24%)	9 (4%)	1 (0%)	0 (0%)
7 items	The way you yourself would want to be treated as a patient	55 (22%)	105 (42%)	69 (28%)	17 (7%)	2 (1%)	1 (0%)
How often would you say patients in the ICU are treated:	As a person (and not just a patient with a disease or number)	59 (24%)	86 (35%)	78 (32%)	23 (9%)	1 (0%)	0 (0%)
	Like they are ‘on the same level’ with or equal to the ICU team	36 (15%)	72 (29%)	65 (26%)	61 (25%)	13 (5%)	1 (0%)
	Equally to all other patients	86 (35%)	86 (35%)	57 (23%)	14 (6%)	4 (1%)	0 (0%)
	As an individual	83 (34%)	83 (34%)	60 (24%)	16 (6%)	5 (2%)	0 (0%)
Respectful Behaviors	Respond to patient and/or family anxiety	33 (13%)	79 (32%)	96 (39%)	36 (14%)	4 (2%)	0 (0%)
α=0.9256	Make an effort to get to know who the patient and family members are as people and what is important to them	27 (11%)	75 (30%)	78 (31%)	62 (25%)	7 (3%)	0 (0%)
10 items	Listen carefully and consider the opinion of each member of the ICU team in formulating treatment plans	34 (14%)	63 (25%)	93 (38%)	50 (20%)	7 (3%)	0 (0%)
How often does the ICU team:	Listen carefully to the patient and/or family	31 (13%)	93 (37%)	103 (41%)	18 (7%)	4 (2%)	0 (0%)
	Explain things thoroughly to patients and/or families	43 (17%)	67 (27%)	97 (39%)	37 (15%)	5 (2%)	0 (0%)
	Give patients and/or families the opportunity to provide input into the care plan	32 (13%)	58 (23%)	86 (35%)	60 (24%)	12 (5%)	0 (0%)
	Protect patient modesty	42 (17%)	84 (34%)	85 (34%)	33 (13%)	3 (1%)	1 (1%)
	Greet patients and/or families when entering their room	50 (20%)	108 (44%)	68 (27%)	19 (8%)	2 (1%)	1 (0%)
	Talk to sedated patients and tell them what is happening	19 (8%)	52 (21%)	80 (32%)	71 (29%)	20 (8%)	6 (2%)
	Focus on patient comfort	48 (20%)	91 (37%)	73 (30%)	26 (11%)	5 (2%)	1 (0%)
Disrespectful Behaviors	Dismiss patient and/or family concerns	6 (3%)	8 (3%)	5 (2%)	55 (22%)	142 (57%)	32 (13%)
α=0.7016	Speak disrespectfully about patients and/or families behind their backs	5 (2%)	7 (3%)	9 (4%)	59 (24%)	126 (51%)	40 (16%)
4 items	Talk down to patients and/or families	4 (2%)	10 (4%)	1 (0%)	20 (8%)	138 (55%)	76 (31%)
How often does the ICU team:	Get frustrated with demands of patients and/or families	1 (0%)	8 (3%)	13 (5%)	128 (52%)	86 (35%)	12 (5%)

The Overall Respect Scale had a mean score of 33.5 (SD 5.56) with a range between 16 and 42 (possible range 7-42). In terms of overall respect, the most-endorsed item was how often the ICU team treated patients equally to all other patients (35% of respondents reported ‘all of the time without exception’) and the least-endorsed item was how often patients are treated like they are ‘on the same level’ with or equal to the ICU team (15% of respondents reported ‘all of the time without exception’).

The Respectful Behaviors Scale had a mean score of 43.5 (SD 7.64) with a range between 18 and 60 (possible range 10-60). In terms of specific respectful behaviors, the most-endorsed item was greeting patients and families when entering the room (20% of respondents reported ‘all of the time without exception’) and the least-endorsed item was talking to sedated patients to tell them what is happening (8% of respondents reported ‘all of the time without exception’).

The Disrespectful Behaviors Scale had a mean of 9.3 (SD 2.80) with a range between 4 and 24 (possible range 4-24). Higher scores on this scale represent more disrespectful behaviors. In terms of specific items, the least-endorsed negative behavior was talking down to patients and families (31% reported ‘never’) and the most-endorsed negative behavior was getting frustrated with demands of patients and families (5% reported ‘never’).

Association of Respect Scales with Respondent Demographic Characteristics

Table 2 identifies the mean scores for all three respect scales across respondent gender, race/ethnicity, occupation and ICU type. Although there are a few associations that meet the traditional p<0.05 threshold for statistical significance, there are none that met our more stringent corrected significance level of p<0.004.

Table 2.

Variation in CORE-ICU Scores by Respondent Characteristics and Setting

Respondent Characteristic	N=249	Overall Respect		Respectful Behaviors		Disrespectful Behaviors
	N (%)	Mean Score (SD)	p-value	Mean Score (SD)	p-value	Mean Score (SD)	p-value
Gender
Female	190 (80%)	33.3	-ref-	43.2	-ref-	9.5	-ref-
Male	49 (20%)	33.8	0.535	44.0	0.507	8.5	0.023
Race
White	176 (77%)	32.8	-ref-	42.8	-ref-	9.3	-ref-
African-American/Black	22 (10%)	32.7	0.90	42.6	0.931	10.2	0.145
Asian	19 (8%)	36.6	0.005	45.6	0.129	9.5	0.715
Other	12 (5%)	36.3	0.031	48.6	0.018	8.8	0.539
Occupation
Nurse	144 (58%)	33.4	-ref-	42.7	-ref-	9.5	-ref-
Physician	45 (18%)	33.9	0.608	43.8	0.392	8.8	0.154
Other	60 (24%)	33.4	0.969	45.1	0.050	9.2	0.497
ICU Type
Specialty	106 (43%)	34.0	-ref-	44.0	-ref-	9.1	-ref-
Surgical	53 (22%)	34.1	0.978	44.0	0.988	9.6	0.271
Medical	87 (35%)	32.5	0.067	42.5	0.174	9.3	0.712

DISCUSSION

The CORE-ICU is the first tool designed to assess ICU clinicians’ perspectives about the level and consistency of respectful treatment within the ICU environment. The CORE-ICU has undergone a rigorous content development process from multiple perspectives, and we have demonstrated that the general respect and respectful behaviors subscales of the CORE-ICU have excellent internal consistency and the disrespectful behaviors subscale has adequate internal consistency.

The CORE-ICU is an efficient measure that can provide data on local ICU climates that are respectful of patients and families, monitor progress in developing more respectful climates, and raise awareness among respondents about the importance of different domains of respectfulness. The respectful and disrespectful behaviors that are asked about in the questionnaire could be targets for specific behavior changes, whereas the overall respect items are more reflective of attitudes that might require different types of quality improvement interventions. This measure can be used in conjunction with patient/family experience surveys because it provides a perspective that is only available to clinicians and focuses on the overall environment that affects all patients, families, and staff as opposed to an individual patient’s experience. Although we suspect that clinicians have a perspective on some behaviors that patients might not have (e.g. how patients and families are spoken about when they aren’t around), it would be interesting to compare clinician and patient/family perceptions of respect.

The frequency with which respondents reported the most favorable rating varied considerably between the items, which suggests that respondents were paying attention to the items, answering honestly, and were not simply reporting the most socially-desirable response. For example, within the overall respect subscale, respondents were far more likely to report that patients are ‘all of the time without exception’ treated equally to all other patients than that they are treated like they are ‘on the same level’ or equal to the ICU team. Further, the percent of respondents who reported the most favorable rating for each of the individual respectful behaviors items was considerably lower than the percent who responded most favorably to the overall respect items. Variability in responses to different items also lends support to the instrument’s validity: respondents would consider treatment with respect generally to include an array of attitudes and behaviors, whereas the particular behaviors (such as greeting a patient/family when entering a room) might not occur every time, despite a general sense of respect as present.

In theory, if a questionnaire about organizational climate was truly only about the climate, rather than about the person answering the questionnaire, we should not expect to see significant differences in respondent reports based on demographic characteristics. In reality, a questionnaire that measures nearly any type of organizational climate will always be subject to interpretation by the individual answering the questionnaire. In this study, because people have diverse personal experiences of respect in society and in healthcare,²¹ they will be varyingly attuned to how patients and families are treated. Although we did not see significant differences in respondent reports based on the type of unit or their own personal characteristics, there were a few trends worth noting. First, women tended to rate respect levels lower across all three subscales, and particularly reported higher levels of disrespect. Although there was very little racial/ethnic diversity in our sample, there was a trend towards higher levels of disrespect being reported among African Americans, and greater overall respect reported among Asian respondents.

Our study has some limitations that are important to consider. Our response rate raises some concern about non-response bias, as it is possible that respondents were people who were more interested in the concept of respect than non-respondents. There was also limited diversity in terms of race/ethnicity in our respondent sample, and we did not collect data on years of ICU experience. In addition, data collection across a single academic urban health system (although we included 8 ICUs in two distinct physical facilities) raises some concern about generalizability of findings. Finally, we sampled ICUs that treated adults, so it is conceivable that there may be important differences in ICUs focused on the treatment of children. Because of these issues, it will be important for this instrument to be validated in other samples and settings.

Treatment with respect and dignity is a primary concern for patients and their families in intensive care environments, and the lapses in treatment with respect and dignity may contribute to moral distress and burnout among health professionals. Thus, finding ways to measure it in action has great value. We have rigorously developed the CORE-ICU to measure the climate of respect from the perspective of health professionals, and suggest that routine measurement with intervention to address areas in need of improvement would draw attention to it and its related behaviors, thereby improving patient care and reducing staff burnout.