Adaptation of a Communication Interaction Behavior Instrument for use in Mechanically Ventilated, Nonvocal Older Adults

Abstract

Background

Valid and reliable instruments are needed to measure communication interaction behaviors between nurses and mechanically ventilated (MV) intensive care unit (ICU) patients who are without oral speech.

Objectives

To refine and evaluate preliminary validity and reliability of a Communication Interaction Behavior Instrument (CIBI) adapted for use with nonvocal, MV ICU patients.

Methods

Raters observed nurse-patient communication interactions using a checklist of nurse and patient behaviors, categorized as positive and negative behaviors. We used 3-minute video-recorded observations of 5 MV ICU adults (<60 years) and their nurses to establish preliminary inter-rater reliability and confirm appropriateness of definitions (4 observations per dyad, N=20). Based on expert input and reliability results, the behaviors and item definitions on the CIBI were revised. The revised tool was then tested in a larger sample of 38 MV ICU patients (≥60 years) and their nurses (4 observations per dyad, N=152) to determine inter-rater reliability.

Results

For preliminary testing, percent agreement for individual items ranged from 60–100% for nurse behaviors and 20–100% for patient behaviors across the 5 pilot cases. Based on these results, 11 definitions were modified and 4 items were dropped. Using the revised 29-item instrument, percent agreement improved for nurse behaviors (73–100%) and patient behaviors (68–100%). Kappa coefficients ranged from 0.13–1.00, with lower coefficients for patient behaviors.

Conclusion

Preliminary results suggest that the revised CIBI has good face validity and demonstrates good inter-rater reliability for many of the behaviors but further refinement is needed. The use of dual raters with adjudication of discrepancies is the recommended method of administration for the revised CIBI.

Introduction

Mechanical ventilation poses a significant barrier to communication for critically ill patients. Mechanically ventilated patients, especially those who require oral intubation, find themselves unable to use the most natural method of communication, oral speech. Critical care nurses are the primary initiators of communication in the ICU and therefore, tend to control the topic of the interaction (Ashworth, 1980; Hall, 1996; Happ et al., 2011). Mechanically ventilated patients must rely on alternative methods, such as mouthing or gesturing to communicate messages to their care providers (Thomas & Rodriguez, 2011). These methods of communication can be unreliable (Leathart, 1994; Menzel, 1998); consequently, messages communicated from the patient to the care provider can be easily misinterpreted. Difficulty communicating with providers can lead to unmet needs and unrelieved symptoms for patients (Magnus & Turkington, 2006; Nelson et al., 2004; Puntillo et al., 2010; Samuelson, 2011).

Provider-patient communication, especially with mechanically ventilated patients, is complex and cannot be represented by a single molecular behavior (Suen & Ary, 1989). Previous studies have identified positive and negative talk and behaviors as primary components of provider-patient communication in a variety of patient populations and settings (de los Ríos Castillo & Sánchez-Sosa, 2002; Hall, 1996; Meystre, Bourquin, Despland, Stiefel, & de Roten, 2013; Morse et al., 1992; Roter, Geller, Bernhardt, Larson, & Doksum, 1999; Salyer & Stuart, 1985; Shapiro, 1990). Positive talk, which includes interaction behaviors such as laughing, and approval, has been associated with patient satisfaction and clinician-patient alliance in the outpatient setting (Meystre et al., 2013). The behavior that nurses utilize during communication interactions with critically ill, mechanically ventilated patients can influence the quality of the encounter and patient outcomes, including satisfaction (de los Ríos Castillo & Sánchez-Sosa, 2002; Hall, 1996; Riggio, Singer, Hartman, & Sneider, 1982; Salyer & Stuart, 1985). For mechanically ventilated, nonvocal patients, these responses are primarily measured by nonverbal behaviors instead of spoken communication (Hall, 1996; Salyer & Stuart, 1985).

Attempting to measure nonverbal behaviors is challenging; therefore, a valid and reliable instrument is essential in order to evaluate the role that interaction behaviors, individually and collectively, have on safety and quality outcomes. However, few instruments have been developed for the purpose of identifying interaction behaviors between providers and mechanically ventilated, critically ill patients. In a review of the literature, three previous studies were identified that used similar but not identical behavioral observation tools to measure nurse-patient interaction behaviors in an intensive care unit (ICU) (de los Ríos Castillo & Sánchez-Sosa, 2002; Hall, 1996; Salyer & Stuart, 1985). Behaviors in the Categories of Nurse-Patient Interaction tool developed by Salyer and Stuart (1985) lacked definitions, making it difficult to understand positive and negative behavior categorizations. In addition, the psychometric assessment performed on this instrument was limited and minimal information was provided regarding patient characteristics in study samples. There were only two patient actions identified in the Categories of Nurse-Patient Interaction tool. These categories include “initiation of nonverbal communication” and hostility (Salyer & Stuart, 1985). These categories are broad and do not address the intent or message that the nonverbal communication act represents.

In contrast, item definitions provided for the de los Ríos Castillo and Sánchez-Sosa’s instrument were detailed enough to justify category assignment but many of the behavior definitions focused predominantly on verbalization by the patients. Although patients in the de los Rios Castillo and Sanchez-Sosa study were described requiring “assistance breathing”, it is not clear what type of support patients required or the number who were unable to speak due to mechanical ventilation. While percent agreement for the subscales of behaviors was good (93–99% for nurse behaviors and 95–98% for patient behaviors) (de los Ríos Castillo & Sánchez-Sosa, 2002), little attention was given to alternative methods of communication beyond head nods, facial expressions, and gesturing. Therefore, the de los Ríos Castillo and Sánchez-Sosa’s instrument was selected for adaptation for use with nonvocal, critically ill patients (see table 1 and 2). The purpose of this study was to: 1) refine a previously developed instrument designed to measure communication interaction behaviors between critically ill patients and their nurse providers for use with mechanically ventilated, nonvocal patients and their nurse providers in the ICU and 2) evaluate the revised instrument’s preliminary reliability and face validity.

Table 1.

Nurse Interaction Behaviors and Definitions

Positive Behaviors (14)
Sharing	Facing the patient, the nurse offers him/her such items as a blanket, pillow, ice chips, prescribed food (ex: ice cream), the patient’s CD, TV (ex: on/off or closer), or MP3 player or some other objects* used to support the patient’s wellbeing or treatment. *objects should not include medication or treatments (Ex: offering pain medication or suction)”
Praising	Verbal comments involving approval, recognition or praise to the patient, such as “good”, “that was very well done”, “you look much better today”, and “you are recovering real fast”. All comments must involve clear, audible and a kind tone of voice, and may or may not involve such physical contact as patting the patient’s feet, arms, hands or shoulders.
Visual Contact	The nurse looks the patient in the eyes for as long as the nurse is at the patient’s bedside (unless engaged in incompatible technical procedures, assessment of monitor, or interruptions by family, clinical or other healthcare professionals), regardless of whether the patient is looking at her/him. *This is limited to when the nurse is within the camera frame.
Brief Contact	The nurse stands at a distance no longer than arm’s length from the patient, for a period no shorter than five seconds
Brief Contact with speech	The nurse stands at a distance no longer than arm’s length from the patient, for a period no shorter than five seconds AND includes spoken information from the nurse
Proximity	The nurse stands at a distance no longer than arm’s length from the patient, for a period longer than five seconds
Proximity with speech	The nurse stands at a distance no longer than an arm’s length from the patient’s upper body (from waist up), for a period of > five seconds AND includes spoken information from the nurse. The nurse may be performing medical interventions or procedures.
Physical contact	The nurse touches, pats or hugs the patient. Physical contact includes attention getting touch and touch for comfort. Does NOT include technical/procedural touch.
Social Politeness	The nurse uses terms including “please”, “thank you”, and greets the patient by name. Social politeness also includes asking the patient permission before an action. Ex: “Can I turn off the light”
Preparatory Information	Information given before a procedure. Examples include “you are going to feel a mild sting but it will hurt very little”, “we’re going to give you a sponge bath”. *To achieve “present’, the procedure needs to be started within 10 seconds of preparatory information being given. This does NOT apply if preparatory information occurs simultaneously to the procedure or in reaction to a patient response.
Expanded Preparatory information (education)	Information given before a procedure that includes expanded explanation/education prior to starting the procedure. Examples include: “I am going to give you your medication now. I have your Pepcid. It is used to help prevent stomach ulcers and treat reflux”. To achieve “present’, the procedure needs to be started within 10 seconds of preparatory information being given. This does NOT apply if preparatory information occurs simultaneously to the procedure or in reaction to a patient response.
Preparatory Information with Brief Delay	Same definition as preparatory information but the start of the procedure is > 10 seconds after the information is given as long as the nurse stays on task (ex: collecting supplies, drawing up medications)
Expanded Preparatory information with Brief Delay	Same definition as expanded preparatory information but the start of the procedure is >10 seconds after the information is given as long as the nurse stays on task (ex: collecting supplies, drawing up medications)
Smiling	Lifting lips corners while looking the patient in the eyes
Modeling	Body changes or movements accompanied by the corresponding descriptive verbalization, reproduced by the patient within the following ten seconds (“Please cough like this”, “lift your tongue like this”). Requires patient to model nurses’ gestures, such as thumbs up.
Laughing	Lifting the lips corners or congruently opening the mouth while emitting the characteristic voiced laughter sound, with or without an appropriate comment such as “that was funny Mrs/Mr.(patient’s name)”.
Augmenting	Augments patient’s auditory comprehension be writing, gesturing, showing object, etc.
Negative Behaviors (3)
Disapproving	Verbalizations implicating disagreement, negation, disgust or criticism of the patient. Ex: “No, not like that”, “I’ve already told you how to turn around”.
Yelling	Loud verbalizations or utterances containing comments, threats, criticism or disapproval of the patient. Ex: “Hey, that was really bad!”, “Don’t get out of bed!”, “Don’t remove that bandage!”
Ignoring the Patient	After a request or summons by the patient, the nurse does not answer verbally within five seconds in a congruent manner, or does not perform the requested action or does not give an explanation of why it cannot be done, or simply nods (yes or not), without establishing distinct visual contact with the patient

Note. These definitions have been revised from previous by de los Ríos Castillo, J.L. and Sánchez-Sosa, J. (2002). Changes to the original definitions are identified with italics.

Table 2.

Patient Interaction Behaviors and Definitions

Positive Behaviors (9)
Acceptance	After the nurse offers/performs a health related or comfort providing function, the patient says “yes”, “mmhm”, thank the nurse, nods affirmatively with the head, eyes or hand, expressing agreement, acceptance or satisfaction. Patients can utilize AAC strategies (ex: writing, alternative yes/no signals, communication boards, or speech generating devices) to demonstrate agreement, acceptance, or satisfaction *May receive a “present” if the nurse provides a plan of care for the rest of the shift (ex: Nurse offers to change patient’s bed/linens after the physicians comes back and finishes a procedure. The patient nods yes to accept the plan)
Following Instructions	Engaging a behavior (within the patient’s actual capabilities) in response to an appropriate request or instruction by the nurse, within ten seconds of the request. Ex: posture changes, answering questions. *If patient’s response is ambiguous and not interpretable to the nurse, then this behavior is absent.
Visual Contact	Looks the nurse in the eye when the nurse asks a question or addresses the patient while the nurse is at the bedside (unless the nurse is engaged in incompatible technical procedures, assessment of monitor, or interrupted by family, clinical or other healthcare professionals), regardless of whether the nurse is looking at her/him. *Patient may receive a “present” if the nurse is out of the frame but it is clear that the patient is focused on and responding to the nurse.
Physical Contact	Touches, pats or hugs the nurse. Includes attention-getting touch and touch for comfort. Ex: reaching out to nurse to pat their arm or shake their hand. Does not include grabbing for support during position changes.
Request	Verbal, digital or manual indications initiated by the patient to express a need or request, followed by the corresponding nurse appropriate behavior. Ex: requesting a glass of water, pain medication, etc.
Smiling	Lifting lips corners while looking the patient in the eyes *Exceptions may be made if the patient has a neurological deficit (facial droop) or other impediment that doesn’t allow one corner of the mouth to raise
Maintaining Attention	Keeps sustained eye contact while the nurse provides an explanation, information, instruction or appropriate comment. *may receive a “present” if the nurse is out of the frame but the patient is still focused on and responding to the nurse.
Laughing	Lifting the lips corners or congruently opening the mouth. Patients may appear to take an extra breath while their chest and shoulders rise. It may appear to be more of a chuckle.
Praise	A clearly distinguishable gesture or message expressing gratefulness or approval of an action by the nurse.
Negative Behaviors (3)
Disagreement/Negativity	Verbalization or actions expressing opposition to nurse’s, includes threatening gestures, striking or refusal.
Disgust	Gestures or facial expressions indicating disgust, exasperation, annoyance, or frustration. Examples: head shaking, turning away, upward eye movement or eye roll *In order to determine if the eye roll/movement is an expression of disgust, the rater must take into account verbal context, and other nonverbal indicators.
Ignoring the Nurse	Same definition as the category for nurses in absence of a justifying situation such as being asleep or unconscious.

Note. These definitions have been revised from previous by de los Ríos Castillo, J.L. and Sánchez-Sosa, J. (2002). Changes to the original definitions are identified with italics.

Methods

Study Background

The psychometric evaluation of the Communication Interaction Behavior Instrument (CIBI) was performed as part of an expanded secondary analysis to evaluate the association between interaction behaviors and nursing care quality indicators in mechanically ventilated, critically ill older adults. The expanded secondary analysis utilized a subset of patients enrolled in the Study of Patient-Nurse Effectiveness with Communication Strategies (SPEACS). All patient and nurse participants agreed to allow their video-recorded observations to be utilized in further analysis and IRB approval was received. Video-recordings permit repeated viewing of an entire interaction or segments of observations. The ability to pause, rewind, and review can contribute to improved reproducibility (Haidet, Tate, Divirgilio-Thomas, Kolanowski, & Happ, 2009) which makes it an appropriate choice for evaluating nurse-patient interaction behavior, especially nonverbal communication behaviors that can be fleeting or occur simultaneously with other behaviors.

Sample and Setting

The sample consisted of mechanically ventilated patients who were awake, following commands, and attempting to communicate and their ICU nurses. The SPEACS study was conducted in the medical and cardiothoracic ICUs of a large academic medical center in southwestern Pennsylvania. Eligibility and recruitment procedures for the SPEACS study have been previously reported (Happ, Sereika, Garrett, & Tate, 2008; Nilsen et al., 2013)

Procedures

The four steps performed to adapt the instrument and conduct preliminary psychometric assessment included: 1) definition refinement and expansion, 2) rater training and preliminary testing, 3) instrument revision and 4) application and testing in a larger sample.

Step 1- Definition Refinement

We first refined the tool to make it applicable for use in a nonvocal population. In addition to modifying the behavior definitions, defining criteria and rating decision rules were developed for each item to help establish parameters for raters. An example of definition refinement was the interaction behavior titled, “Sharing”. The original definition of “Sharing” was “facing the patient, the nurse offers him/her such items as a glass of water, prescribed food, special urinals, the patient’s audio cassette player or transistor radio or some other object used to support the patient’s well-being or treatment” (de los Ríos Castillo & Sánchez-Sosa, 2002). The revised definition of “Sharing” included more culturally appropriate technologies and objects such as CD players and MP3 players. In addition, the following rating decision rule “objects should not include medication or treatments (Example: offering pain medication or suction)” was incorporated into the definition. The rating decision rule for “Sharing” provided criteria to clarify what items should and should not be included as support for the patient’s well-being. Because “providing medication” is a required task for the nurse, it was not included in this behavior definition.

Patient interaction behaviors were also revised. Many of the original definitions included vocalization that mechanically ventilated patients typically cannot produce. For example, the original definition of “Acceptance” stated “after the nurse offers or performs a health-related or comfort providing function, the patients says ‘yes’; ‘mmhm’; thanks the nurse; or nods affirmatively with the head, eyes, or hand, expressing agreement, acceptance, or satisfaction”. The revised definition of acceptance added that patients could also utilize augmentative and alternative communication (AAC) strategies including writing, alternative yes/no signals, communication boards, or speech generating devices to demonstrate agreement, acceptance, or satisfaction. The CIBI was expanded to include 33 interaction behaviors. The behaviors were divided into the following 4 categories, i.e., positive nurse (n=17), negative nurse (n=3), positive patient (n=10) and negative patient (n=3) (see Table 1).

Step 2- Rater Training and Preliminary Testing

Rater training was performed using video-recorded observations of interactions between 5 randomly selected nurse-patient dyads from the SPEACS study. This sample included patients less than 60 years of age to permit future selection of older adults from the available pool of videos for the primary study. Each dyad had four 3-minute observations for a total of 20 observations for the sample (Happ et al., 2011; Happ et al., 2008). During training, the raters reviewed the behaviors on the observation tool, discussed the definitions and resolved questions. Raters then reviewed two cases together and identified behaviors according to the tool. Finally to attain rating competency, raters proceeded to review and rate training cases independently until 90% agreement was achieved. Rater competency was achieved after review of 4 additional cases (N=16 observations), which corresponded to approximately 18 hours of training.

Five pilot cases were randomly selected to determine preliminary inter-rater reliability. For these cases, two raters independently rated each video-recorded observation of nurse-patient communication. Behaviors were recorded on scannable form developed using TeleForm^™ (version 6.0, Cardiff Software, San Jose, CA) configured with the behaviors divided into the 4 subscales including: positive nurse, negative nurse, positive patient, and negative patients. If a behavior occurred at least once, it was rated as present. Cumulative counts of individual behaviors were not preformed. Raters were required to watch the video at least 4 times: 1) without rating, 2) to rate patient behaviors 3) to rate nurse behaviors, 4) to review for any missed behaviors. Raters met to review and adjudicate differences by watching the video observation together, discussing behavior definitions, and coming to consensus on whether a behavior was present or absent. If the two raters could not come to a consensus, a third experienced rater reviewed the observation in question and provided feedback and arbitration. The cases were adjudicated to resolve inconsistencies in raters’ interpretation of definitions and to modify definitions or rating guidelines before rating the main study sample. Individual item percent agreement and individual item percent agreement averaged over the four observations were investigated.

Step 3- Definition Revision

Once preliminary testing was completed, behaviors and definitions were revised. If a behavior had low percent agreement (<60%) for multiple observations, the description was revised. In addition to percent agreement, expert feedback and persistent issues identified during rating were used to guide behavior and definition revisions. Revisions included removing behaviors from the instrument, merging behaviors, modifying definitions, and providing additional criteria for raters. The revised CIBI consisted of 29 behaviors, i.e., 17 nurse (14 positive and 3 negative) and 12 patient behaviors (9 positive and 3 negative) (see Tables 2 and 3). Dr. Mary Beth Happ and Dr. Ann Kolanowski, who have expertise in observational research with nonvocal care recipients, reviewed the behaviors and definitions for face validity.

Table 3.

Counts and percentages of interaction behaviors by observation and for the total sample

	Observation				Total
	1	2	3	4
Positive Nurse Behaviors- n (%)
Sharing	9 (23)	8 (21)	7 (18)	11 (29)	35 (23)
Praising	12 (32)	15 (40)	19 (50)	13 (34)	59 (39)
Visual Contact	30 (79)	33 (87)	34 (90)	34 (90)	131 (86)
Proximity with Speech	35 (92)	35 (92)	37 (97)	36 (95)	143 (94)
Physical Contact	17 (45)	17 (45)	18 (47)	20 (53)	72 (47)
Social Politeness	29 (76)	29 (76)	25 (66)	29 (76)	112 (74)
Preparatory Information	12 (32)	14 (37)	10 (26)	14 (37)	50 (33)
Expanded Preparatory Information	8 (21)	6 (16)	1 (3)	2 (5)	17 (11)
Preparatory Information with Delayed Action	3 (8)	4 (11)	4 (11)	5 (13)	16 (11)
Expanded Preparatory Information with Delayed Action	1 (3)	5 (13)	1 (3)	1 (3)	8 (5)
Smiling	6 (16)	8 (21)	10 (26)	11 (29)	35 (23)
Modeling	0 (0)	2 (5)	0 (0)	2 (5)	4 (3)
Laughing	5 (13)	6 (16)	9 (24)	8 (21)	28 (18)
Augments	22 (58)	26 (68)	29 (76)	33 (87)	110 (72)
Negative Nurse Behaviors- n (%)
Disapproving	0 (0)	0 (0)	1 (3)	0 (0)	1 (1)
Yelling	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)
Ignoring	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)
Positive Patient Behaviors- n (%)
Acceptance	27 (71)	23 (61)	25 (66)	22 (58)	97 (64)
Instruction Following	33 (87)	33 (87)	34 (90)	30 (79)	130 (86)
Visual Contact	19 (50)	16 (42)	17 (45)	19 (50)	71 (47)
Physical Contact	3 (8)	2 (5)	3 (8)	1 (3)	9 (6)
Request	11 (29)	15 (40)	12 (32)	12 (32)	50 (33)
Smiling	8 (21)	6 (16)	8 (21)	8 (21)	30 (20)
Maintaining Attention	19 (50)	17 (45)	19 (50)	19 (50)	74 (49)
Laughing	0 (0)	0 (0)	2 (5)	3 (8)	5 (3)
Praising	0 (0)	3 (8)	1 (3)	2 (5)	6 (4)
Negative Patient Behaviors- n (%)
Disagreement	2 (5)	0 (0)	0 (0)	0 (0)	2 (1)
Disgust	2 (5)	0 (0)	1 (2.6)	0 (0)	3 (2)
Ignoring	3 (8)	0 (0)	2 (5)	1 (3)	6 (4)

Step 4- Application and Testing

After completion of preliminary reliability and validity evaluation, the revised CIBI was tested using 3-minute video-recorded observations of 38 nurse-patient dyads (four observations per dyad, total=152) in mechanically ventilated, nonvocal older adults (≥60 years of age) drawn from the SPEACS study. We repeated the rating strategies used in the preliminary testing. Two raters independently evaluated all 152 observations to assess inter-rater reliability on 100% of the sample.

Data Analysis

Data were analyzed using IBM^® SPSS^® Statistics (version 20.0, IBM Corp., Armonk, NY). The most common method of assessing reliability of behavioral observational data is through the use of statistical methods to evaluate interobserver agreement (Foster, Bell-Dolan, & Burge, 1988). During preliminary testing, interobserver agreement for individual behaviors between the two raters was determined through the computation of percent agreement. Percent agreement was calculated by taking the total number of behaviors that the 2 raters agreed upon and then dividing the numerator by the total number of possible behaviors per observation (Watkins & Pacheco, 2000). A cut-point of 80% was identified for percent agreement and is supposed by previous research as an appropriate cut-point to measure interobserver agreement in observational coding from video (de los Ríos Castillo & Sánchez-Sosa, 2002; Morse, Beres, Spiers, Mayan, & Olson, 2003b).

Cohen’s kappa coefficient and intra-class correlations (ICC) were used to assess interobserver agreement on the larger sample only. Kappa coefficients provide an assessment of agreement, which corrects for chance agreement (Landis & Koch, 1977; Viera & Garrett, 2005a). Since there were only two raters, kappa coefficient assessed interobserver agreement for the binary individual nurse and patient behaviors for each observation. A kappa coefficient of 0.60 has been recommended as the minimally acceptable kappa value for interobserver agreement (Landis & Koch, 1977; Wynd, Schmidt, & Schaefer, 2003).

Intraclass correlations are a measure of reliability that assesses consistency between raters (Shrout & Fleiss, 1979). Two-way mixed effects intraclass correlations coefficients were performed in order to remove the between-raters variance, while accounting for the fixed numbers of raters (n=2) in this study (Bartko, 1966). Intraclass correlations were performed for the sum of nurse positive and patient positive behaviors at each observation. We were unable to perform ICC for the sum of nurse negative and patient negative behaviors because of the minimal occurrence of these behaviors Moderate agreement is considered to be an ICC of 0.61 to 0.80 and 0.81 to 1.00 is considered excellent agreement (Bartko, 1966).

Results

Rater Training and Preliminary Reliability (Step 1 and 2)

The 5 patients included in the rater training pilot cases were on average (±SD) 48.4 ± 15.1 years of age (range= 29–67) and were all male (100%). They were predominantly white (89%) with an average of 14.4 years of education (range= 12–18, SD=2.8). The study nurses (n=5) were on average 37.0± 8.3 years old (range=24–45, SD= 8.3) and over half were female (60.5%). They were baccalaureate prepared (80%) with a mean of 8.0 years in nursing practice (range=2–14, SD=5.8) and 5.4 years in critical care (range=2–13, SD= 4.7).

Individual item percent agreement ranged from 60% to 100% for nurse behaviors and 20–100% for patient behaviors. Individual item percent agreement averaged across the four observations for the all behaviors ranged from 65% to 100% with mean item percent agreement for individual nurse behaviors ranging from 75% to 100% and 65% to100% for patient behaviors (See Table 4 for the percent agreements for each behavior for all observation periods).

Table 4.

Percent Agreement by Observation and Mean Item Agreement

Behaviors	Percent Agreement				Mean Item Agreement
Positive Nurse	Observation
	1	2	3	4
Sharing	100	100	100	100	100
Praising	60	80	80	100	80
Visual Contact	60	100	100	80	85
Brief Contact	100	80	80	80	85
Brief Contact with Speech	100	80	80	80	85
Proximity	100	100	80	100	95
Proximity with Speech	80	100	100	100	95
Physical Contact	100	60	100	100	90
Social Politeness	80	80	100	80	85
Preparatory Information	80	80	100	80	85
Expanded Preparatory Information	100	80	60	80	80
Preparatory Information with Delay Expanded Preparatory Information	100	80	80	100	90
Expanded Preparatory information with Delay	100	60	100	80	85
Smiling	80	100	100	100	95
Modeling	100	100	100	100	100
Laughing	60	100	100	100	90
Augmenting	80	100	60	60	75
Negative Nurse
Disapproving	100	100	100	100	100
Yelling	100	100	100	100	100
Ignoring	100	100	100	100	100
Positive Patient
Acceptance	100	60	100	100	90
Following Instructions	100	80	80	80	85
Full Visual Contact	80	100	60	80	80
Partial Visual Contact	100	100	40	20	65
Physical Contact	100	100	100	100	100
Requests	60	100	100	100	90
Smiling	80	100	100	100	95
Maintaining Attention	80	100	60	100	85
Laughing	60	100	100	100	90
Praise	100	100	100	100	100
Negative Patient
Disagreement	100	100	100	80	95
Disgust	80	100	100	80	90
Ignoring	100	80	100	100	95

Definition Revisions and Validity (Step 3)

Based on the above results and expert feedback, 11 behavior definitions, including 7 nurse behaviors and 4 patient behaviors, were modified. ‘Laughing’ is a prime example of a definition that was modified in order to be appropriate for use in mechanically ventilated patients. Initially the definition for the patient positive behavior of ‘Laughing’ was the same as the definition for the nurse. “Laughing” by the nurse was characterized as “lifting the corners of the lip or congruently opening the mouth while emitting the characteristic voiced laughter sound, with or without an appropriate comment.” While patients who are mechanically ventilated can laugh, the characteristics of laughter while on the ventilator are quite different. The laugh may not be audible and may resemble more of a chuckle where the patient’s shoulders and chest raise briefly. Patients who are mechanically ventilated via oral endotracheal tube may not be able to “open the mouth” or “lift lip corners” due to the presence of the endotracheal tube and/or the devices to secure the tube to the face and mouth. The definition was modified to highlight the physical features that may be present when the patient laughs while on the ventilator (See Table 2).

Four behaviors, including 3 positive nurse behaviors and 1 positive patient behavior, were removed from the instrument. The three positive nurse behaviors removed were: brief contact, brief contact with speech, and proximity (Table 1). In the majority of these interactions, the nurses were performing brief technical procedures or tidying the bed/bedside area. Although the nurses were in close proximity to the patients, there was little real social interaction in these very brief technical encounters or contacts. We felt that these one-sided behaviors did not meet the criteria for a communication interaction.

The original category “Partial visual contact” that only required the patient to look at the nurse if asked a question or when a comment was directed towards the patient was renamed “Visual contact” and replaced two separate visual contact behaviors. “Full visual contact,” where the patient was required to look the nurse in the eye for as long as the nurse was at the patient’s bedside, regardless of whether the nurse was looking at the patient was the only patient behavior removed from the instrument. It was viewed as extremely uncommon for a patient to have visual contact with the nurse the entire time they were at the bedside. In addition, this behavior did not appear to be a realistic behavior for the majority of critically ill patients given difficulties with energy, focus and attention (Ely et al., 2004; Li & Puntillo, 2006; Nelson et al., 2004; Pandharipande, Jackson, & Ely, 2005).

Inter-rater Reliability (Step 4)

The patients in the sample utilized to assess inter-rater reliability (n=38) were on average 70.3 ± 8.5 years of age (range= 60– 87), predominantly white (90%), with an average of 12.9 years of education (range= 8–21, SD=2.8). The study nurses (n=24) were on average 35.1 ± 10.4 years of age (range=22–55) and were largely female (79%). They were baccalaureate prepared (83%) with a mean of 10.0 years in nursing practice (range=1–33, SD=10.7) and 7.2 years in critical care (range= 1–33, SD=9.3).

Tables 1 and 2 provide an outline of the revised instrument with definitions and decision rules. When using the revised instrument, the following positive nurse behaviors were observed to occur in over 50% of the observations: “Proximity with speech”, “Visual contact”, “Social politeness”, and “Augmenting”. “Disapproval” was the only negative nurse behavior that did occur. “Following instructions” and “Acceptance” were the only patient behaviors that occurred in over half of the observations. Negative interaction behaviors were relatively rare occurrences: “Disagreement”, “Disgust”, and “Ignoring the nurse” occurred in less than 5% of the observations (see table 3 for the adjudicated counts and percentage for the interaction behaviors by observation).

For the positive nurse behaviors, percent agreements for individual items ranged from 73.6% to 100% with estimates of inter-rater reliability based on Cohen’s kappa ranging from 0.13 to 1.00. “Social politeness”, “Preparatory information”, and “Augmenting” all had at least one observation with agreement less than 80%. “Modeling” had the highest agreement with 3 observations with 100% agreement. “Social politeness”, “Augmenting” and all 4 categories related to preparatory information had at least 2 observations with a kappa coefficient of less than 0.60 (see Table 5).

Table 5.

Percent Agreement and Kappa Coefficients

Behaviors	Percent Agreement				Kappa Coefficients
Positive Nurse	Observation				Observation
	1	2	3	4	1	2	3	4
Sharing	84.2	92.1	92.1	89.5	0.42	0.77	0.73	0.71
Praising	94.7	86.8	84.2	97.4	0.87	0.72	0.69	0.94
Visual Contact	94.7	89.5	97.4	94.7	0.84	0.61	0.87	0.72
Proximity with Speech	97.4	100.0	94.7	97.4	0.84	1.00	0.48	0.66
Physical Contact	84.2	92.1	92.1	92.1	0.68	0.84	0.84	0.84
Social Politeness	81.6	76.3	81.6	84.2	0.56	0.43	0.60	0.63
Preparatory Information	86.8	73.7	81.6	81.6	0.70	0.43	0.51	0.58
Expanded Preparatory Information	89.5	86.8	86.8	94.7	0.54a	0.37a	-	0.47a
Preparatory Information with Brief Delay	92.1	92.1	92.1	92.1	0.36a	-	0.36a	0.62a
Expanded Preparatory Information with Brief Delay	89.5	81.6	100	97.4	0.30a	0.13a	1.00	-
Smiling	92.1	89.5	89.5	86.7	0.69a	0.61a	0.68	0.63
Modeling	97.4	100	100	100	-	1.00	-	1.00
Laughing	94.7	92.1	94.7	97.4	0.72a	0.68a	0.84	0.92
Augmenting	92.1	73.7	92.1	84.2	0.84	0.47	0.80	0.53
Negative Nurse
Disapproving	100	100	97.4	100	-	-	-	-
Yelling	97.4	100	100	100	-	-	-	-
Ignoring	100	100	100	100	-	-	-	-
Positive Patient
Acceptance	68.4	84.2	78.9	81.5	0.39	0.68	0.56	0.63
Following Instructions	92.1	94.7	86.8	89.5	0.73	0.80	0.54	0.71
Visual Contact	73.7	81.6	84.2	86.7	0.49	0.61	0.68	0.74
Physical Contact	97.4	100.0	97.4	100	0.79	1.00	0.79	1.00
Requests	86.8	78.9	76.3	81.6	0.69	0.51	0.44	0.51
Smiling	89.5	97.4	81.6	89.5	0.66a	0.89	0.37a	0.65a
Maintaining Attention	68.4	81.6	89.5	89.5	0.35	0.60	0.79	0.79
Laughing	97.4	100	94.7	97.4	-	-	0.47a	0.79
Praise	100	97.4	100	100	-	0.79	1.00	1.00
Negative Patient
Disagreement	94.7	100	100	100	-	-	-	-
Disgust	92.1	100	94.7	97.4	-	-	-	-
Ignoring	89.5	94.7	86.8	94.7	-	-	-	0.48a

Note: The symbol – denotes behaviors that kappa coefficients were unable to be calculated due to limited variability in the response.

^adenotes behaviors with limited variability in response, which may contribute to the lower kappa.

For positive patient behaviors, the individual item agreements ranged from 68.4% to 100% and the kappa coefficients ranged from 0.35 to 1.00 for positive patient behaviors. “Physical contact” and “Praise” were the only two patient interaction behaviors that had kappa coefficients of greater than 0.60 for all the observations. Percent agreement for identification of the three negative patient behaviors ranged from 91.4 to 98.7% but there was not enough variability to calculate kappa coefficients for majority of observations (see Table 5).

The two-way mixed effects intraclass correlations (ICC) for the count of different positive nurse behaviors ranged from 0.817 to 0.921 (observation 1=0.918, observation 2=0.817, observation 3=0.862, observation 4=0.921). For the count of different positive patient behaviors, intraclass correlation ranged from 0.871 to 0.910 (observation 1= 0.871, observation 2= 0.910, observation 3=0.877, observation 4=0.893). There appears to be no proportional bias between raters for the count of different nurse positive behaviors (all sessions p >.100) or for the count of different patient positive behaviors (all sessions p >.100).

Discussion

Our study 1) adapted interaction behaviors and definitions to address the needs of mechanically ventilated patients, and 2) demonstrated that good reliability is possible when using the instrumentation to document many of the interaction behaviors. Individual percent agreements for the revised CIBI ranged from 73.6% to 100% with kappa coefficients ranging from 0.13 to 1.00. Overall, the majority of the nurse behaviors had a percent agreement of 80% or greater. This level of agreement has been identified in previous research to be an appropriate cut-point for inter-rater reliability of observational rating from video (de los Ríos Castillo & Sánchez-Sosa, 2002; Morse, Beres, Spiers, Mayan, & Olson, 2003a). There were only three behaviors, including “Social politeness”, “Preparatory information”, and “Augmenting” that fell below this cut-point on one or more observations.

The nurse behaviors of social politeness and the 4 behaviors related to preparatory information had multiple observations with kappa coefficients below 0.60, which indicates that further definition refinement or category collapse is needed. It should be noted however that a lack of occurrence, which was experienced with several of the interaction behaviors, can contribute to smaller, unreliable kappa coefficients (Viera & Garrett, 2005b) (see Table 5). The percent agreement and kappa coefficients tended to be lower for patient behaviors; however, similar inter-rate reliability estimates were seen in previous work (de los Ríos Castillo & Sánchez-Sosa, 2002).

Nonverbal behaviors were the primary method for patients to demonstrate interaction behaviors. These behaviors can be very brief and may be difficult to identify when lighting and position of the camera are not optimal. Many instances of disagreement between raters occurred because they had difficulty hearing the nurse due to poor sound quality or were not able to clearly see the patient’s face because of low light or movement of the video camera. Finally, the two nurse raters had different levels of clinical experience, which may have played a role in how they viewed and interpreted some of the behaviors. The adjudication process, especially during the rater training, was essential in identifying and resolving these issues.

Limitations

The sample size and lack of variability of behaviors limited our ability to perform more comprehensive psychometric evaluation. In addition, the larger sample utilized for primary application testing was comprised solely of older adults, thus may limit the generalizability of the results. There were behaviors that we intended to observe that had limited occurrences across all the observations. Previous literature has shown that components of patient-provider communication can differ between age groups (Siminoff, Graham, & Gordon, 2006) and that communication with older adults can be less effective due to physiological changes (Ebert & Heckerling, 1998; Yorkston, Bourgeois, & Baylor, 2010), and environmental conditions (e.g. noise) (Ebert & Heckerling, 1998; Park & Song, 2005; Pope, Gallun, & Kampel, 2013). Younger patients and the nurses caring for them may rely on a different array of interaction behaviors during communication exchanges. In order to complete a more extensive psychometric evaluation, such as exploratory or confirmatory factor analysis, a larger more diverse sample would be needed.

While analysis of video-recorded observations provided the advantage of replaying interactions, it also limited observations. There were times when the nurse was out of frame or faced away from the camera and therefore behaviors could not be rated. It is important to note that this is a secondary analysis and, therefore, the primary purpose of the video observations were targeted to measure SPEACS study outcomes and not necessarily interaction behaviors. Because of the complex, dyadic nature of nurse-patient communication, further research focusing on interaction behaviors would benefit from multiple methods of observation, including direct and video-recorded analysis. Finally, behaviors were recorded in regard to whether or not they occurred at least once over a three-minute observation. A count of individual behaviors during an entire observation period may also provide more robust data for psychometric analysis.

Conclusion

This study provides preliminary psychometric evaluation for the newly adapted CIBI for use in mechanically ventilated, nonvocal older adults. These findings demonstrate respectable inter-rater reliability on several of the behaviors but further work is needed in order to perform more comprehensive evaluation of psychometric properties. Currently, the most appropriate use of this tool would be to have dual raters observe interaction and adjudicate when discrepancies arise because of the fleeting nature of patient interaction behaviors.