The Safety Assessment Measure for persons with traumatic brain injury: Item pool development and content validity

Abstract

BACKGROUND:

Persons with moderate to severe TBI are at increased risk for unintentional injury or harm in the home and community; however, there is currently no standard measure of safety risk they face now and in the future.

OBJECTIVE:

To develop comprehensive and content valid scales and item pools for assessing safety and risk for persons with moderate to severe traumatic brain injuries.

METHOD:

Qualitative psychometric methods for developing scales and items were used including literature review, item development and revision, focus groups with interdisciplinary rehabilitation staff (n = 26) for rating content validity, and cognitive interviewing of TBI family members (n = 9) for assuring item clarity.

RESULTS:

The Safety Assessment Measure is comprised of 6 primary scales – Cognitive Capacity, Visuomotor Capacity, Wheelchair Use, Risk Perception, Self-Regulation, and Compliance Failures with Safety Recommendations – in which family caregivers or clinicians rate the risk for unintentional injury or harm in adults who have sustained moderate or severe TBI. The scale item pools encompass a broad spectrum of everyday activities that pose risk in the home and community and were rated as having excellent levels of content validity.

CONCLUSIONS:

The Safety Assessment Measure scales and items cover a broad range of instrumental activities of daily living that can increase the risk of unintentional injuries or harm. Empirical evidence suggests that the Safety Assessment Measure items have excellent content validity. Future research should use modern psychometric methods to examine each scale unidimensionality, model fit, and precision.

1. Introduction

Persons with moderate to severe TBI are at increased risk for unintentional injury or harm (UIH) when transitioning from medical inpatient and outpatient rehabilitation to home and community settings (Seel, et al., 2007). Four TBI registry studies show that unintentional injury is a leading cause of death (18–20%) in the year following discharge, and the standardized mortality rate from UIH is 3 to 36 times greater for people with TBI compared to the general population (Harrison-Felix, et al., 2006; McMillan & Teasdale, 2007; Selassie, et al., 2005; Ventura, et al., 2010). Unintentional injuries also cause high rates of emergency room (ER) visits and hospitalizations. Carlson et al. (2012) found that 32% of TBI participants (n = 504) had unintentional injuries that resulted in 228 ER visits or hospitalizations from 3-months to several years following discharge for a TBI. These unintentional injuries were similar to those found in the general population, such as falls; motor vehicle crashes; pedestrian, bicycle or motorcycle crashes; improper use of electrical equipment; fire and burns; being struck by or against an object; poisoning; and firearm-related incidents (Carlson, et al., 2012). Additional risk of harm may occur due to victimization, loss of money or valuables, property damage, medication errors, and inappropriately responding to emergency situations or managing medical conditions (J. S. Kreutzer, et al., 2009; Reichard, et al., 2007; Tyson, Pham, Brown, & Mayer, 2012). Caregivers and payers often manage these risks by providing part-or full-time supervision, which have high economic and social costs.

Self-managing activities that increase UIH risk is challenging for persons who have sustained TBI. Impaired attention, memory, visuomotor skills, awareness and self-regulation affect individuals’ abilities to perform everyday activities, recognize risk, take preventive actions, learn safety strategies, and apply compensatory skills (Riley, Brennan, & Powell, 2004; Tyson, et al., 2012). Persons with TBI who report difficulties performing activities often experience anxiety, frustration, loss of confidence, and depression that result in activity avoidance and diminished life quality (Riley et al., 2004; Seel, Macciocchi, & Kreutzer, 2010; Selassie et al., 2008). Conversely, persons with TBI who highly value their independence may dismiss family and provider concerns about safety, which can lead to confrontation, conflict, power struggles, and strained relations (Durgin, 2000).

For family members, balancing supervisory needs, independent return to activities, and safety is complicated. Families often rely on rehabilitation providers’ determinations of supervision needs (Cooney, Kennedy, Hawkins, & Hurme, 2004; Galski, Ehle, McDonald, & Mackevich, 2000; Macciocchi & Stringer, 2001; Tyson et al., 2012) but short rehabilitation stays often do not allow providers to assess patient safety on instrumental activities of daily living (IADLs) in the home and community. Further, little time is available to train families how to manage high risk situations. Providers often recommend high levels of supervision and activity restrictions, which may unnecessarily limit clients’ autonomy (Banja, 1994; Macciocchi, 2009; Macciocchi & Stringer, 2001; Ruchinskas, Macchiocchi, Howe, & Newton, 2001). While many persons with TBI-related impairments show improvement in physical and cognitive functioning over time, some families continue to follow providers’ initial supervision recommendations or may not know how or when to help survivors transition to independent and safe engagement in IADLs. Other families may have limited support or may need to return to work, which may force them to leave TBI survivors on their own and in environments that present hazards and increased risk (Pai, Zadov, & Hickman, 2012). Consequently, many people with TBI who are at high risk for UIH are under-supervised while others who have low risk are overly restricted (Durgin, 2000).

Currently, there is no established measure to guide family members, rehabilitation providers, and payers in assessing safety risk in the home and community for persons who have sustained TBI. An empirically validated safety assessment measure would improve rehabilitation practice by providing observable, quantified evidence that can improve caregiver knowledge of activities and factors associated with unsafe events in the home and community following TBI. Such a measure would provide an evidence basis to guide the level of family involvement required to keep persons that have TBI-related impairments safe within the least restrictive environment. A safety assessment measure may also provide an evidence basis to justify additional healthcare service provision including outpatient rehabilitation, home health/community supports, or life skills coaching to facilitate selfmanagement of safety-related activities in high risk TBI survivors.

Using a contemporary, state-of-the-science, mixed qualitative-quantitative methods approach within the framework of modern measurement theory would facilitate the development of a safety assessment measure that could enhance clinical practice and research. Qualitative methods that incorporate stakeholder input and feedback to develop an ecologic and content valid item pool have become a standard initial phase in the outcome measurement development process (Administration, 2009; Cella et al., 2007; Lasch et al., 2010; Magasi et al., 2012; Velozo, et al., 2012). A safety assessment measure based on item response theory (IRT) methods would possess a hierarchical, progressive set of items that could serve as a behavioral diagnostic for treatment planning, modifying home environments, and implementing technological supports. An empirically valid, interval level, safety assessment measure would also provide clinicians and researchers with a reliable instrument that could evaluate longitudinal changes in safety risk, including outcomes for clinical trials that evaluate interventions to improve safety.

This study is the first phase of a three phase scale development project to develop and evaluate the Safety Assessment Measure (SAM) for persons who have sustained moderate or severe TBI. The SAM is a set of scales that captures a wide range of safety issues that persons with TBI face in their everyday living situations in the home and the community. The SAM is intended for use following inpatient rehabilitation discharge with the primary raters of safety and risk being family and non-family caregivers, as well as clinicians and paraprofessionals in outpatient rehabilitation, day program, brain injury clubhouse, home health, and neurobehavioral treatment settings. The primary objectives of this study are to define the conceptual framework and constructs that ground the SAM scales and items, describe the item pool for each proposed scale, and evaluate each scale’s content validity.

2. Method

2.1. Qualitative approach to develop the safety assessment measure

Based on state-of-the-science recommendations on developing health outcome measures, we undertook three qualitative steps (see Fig. 1) to develop the SAM item pool: step 1-conceptualize a theoretical model and scale constructs; step-2 develop an item pool; and step 3-evaluate and revise the item pool (Cella et al., 2007; Velozo et al., 2012). Participants included the research team, the scientific and clinical advisory boards, five clinicians from our TBI day rehabilitation program, additional TBI rehabilitation health providers who participated in focus groups, and family members of persons with TBI who provided feedback in cognitive interviews. Key methods within each step are denoted using capital letters. The resulting primary products (denoted within framed text boxes at the bottom of each step) were then used to inform proceeding steps (indicated by arrows).

Fig. 1.

Qualitative methodology implemented for theoretical model and item pool development.

2.2. Conceptualize model and constructs

2.2.1. Methodological framework for measurement

Item response theory (IRT) and use of Rasch methods are the state-of-the-science framework for developing and evaluating outcome measures (Administration, 2009; Cella et al., 2007; Reeve & Fayers, 2005; Velozo et al., 2012). A core principle of IRT is that the probability of rating an item with a response that indicates a higher level of functioning is dependent on “person ability,” “item difficulty,” and the item’s success in discriminating person ability on an outcome of interest (Velozo et al., 2012). Thus, a difficult item on a measurement scale should have a higher probability of being successfully performed by persons with high ability than persons with medium or low ability. Another core principle of IRT is that scale items must be unidimensional (i.e., measure a single latent construct), and be capable of detecting differences in person ability related to the construct (Reeve & Fayers, 2005; Velozo et al., 2012). Differentiation within a scale requires that items are developed in an a priori hierarchical difficulty order within each of the scales (Stone, 1997; Velozo et al., 2012).

2.2.2. Definition of safety risk and contexts for scale use

We defined safety as “the ability to participate in home activities and community living free from harm.” Safety risk is a state of uncertainty in which unintentional harm may arise from a future event, activity, or behavior. Temporally, the threat of a safety risk is likely or imminent. Safety risk is observable and can lead directly to unintentional harm including death, physical injury, financial loss, property damage, or incarceration. The scope of safety risk is not limited to the person with TBI-safety risk may affect other persons, animals, and property. Exposure to violent and aggressive acts is also a safety risk. Conversely, pre-meditated, intentional harmful acts to one’s self (e.g., self-mutilation, suicide), another person, or property secondary to sociopathic and psychiatric disorders are outside the scope of our assessment.

The target of evaluation is persons aged 16 and older who sustained moderate or severe TBI, required acute hospitalization for their injury, and are now returning to everyday activities of daily living in the home and community. Self-assessment by persons with attention, memory, judgment, and self-control deficits are affected by impaired awareness and metacognition (Cavallo, Kay, & Ezrachi, 1992; Hart, Sherer, Whyte, Polansky, & Novack, 2004; Malec, 2004; Seel, Kreutzer, & Sander, 1997). Therefore, we focused on collecting safety ratings from family members and health providers in this initial phase of measure development.

The primary use for the safety measure is clinical with an emphasis on using the measure to: (a) make empirically-based prognoses on risk of unsafe events;(b) assess current safe functioning level and identify primary person factors that drive risk; (c) inform individualized treatment planning related to prevention, supervision needs, and specific functional areas and activities to be targeted; and (d) assess change in safety risk over time, including evaluating individual treatment as well as program effectiveness. A second important use is research and includes longitudinal, prognosis and treatment efficacy and effectiveness studies. Long-term, payers and policy-makers may find epidemiological and treatment effectiveness data useful for reimbursing home health care/supervision and/or prevention education and treatment.

2.2.3. Model and construct development

The research team used a simple concept mapping approach to develop models and scale constructs. First, we reviewed the literature and evaluated models related to unintentional injury and prevention, and self-management of chronic medical conditions, to identify potential person-centered behavioral determinants (Sleet & Gielen, 2007; Sleet, et al., 2006). Next, we considered how TBI-related impairments–cognitive, physical, and behavioral-might play a role in safety risk. Social Cognitive Theory models provided input on person-environment-behavior factors and interactions (Bandura, 1986; Simons-Morton, 2006; Sleet et al., 2006). Then, we constructed a model based on an individual level behavior change framework with four person factor constructs and operational definitions that would serve as the basis for creating scales. Two conference calls each were conducted with the clinical and scientific advisory boards to review the proposed constructs, obtain agreement on their importance, and evaluate the operational definitions of constructs for completeness and unidimensionality.

2.3. Develop item pool

2.3.1. Item identification, binning and winnowing

Item pool content was developed with consideration given to the primary scales and their operational definitions, safety concerns in the general population, safety concerns specific to the TBI population, and feedback from TBI therapists and our clinical advisory board. We reviewed the research literature to identify measures and items that were relevant to the core scale constructs that would comprise the Safety Assessment Measure (SAM) (DeWalt, et al., 2007). We reviewed 11 self-report or clinician-rated measures of neurobehavioral function, decision-making, risk-taking, awareness of impairment, and safety judgment and function in rehabilitation or psychiatric populations (Blais & Weber, 2006; Borgaro & Prigatano, 2003; Chiu & Oliver, 2006; Hart, 2000; Hart et al., 2006; J. Kreutzer & Marwitz, 2000;J. S. Kreutzer et al., 2009; J. Kreutzer, Seel, & Marwitz, 1999; Morrongiello & Corbett, 2006; Oliver, Blathwayt, Brackley, & Tamaki, 1993; Patton, Stanford, & Barratt, 1995; Sherer et al., 1995; Velozo & Peterson, 2001; Whyte, Hart, Bode, & Malec, 2003; Yudofsky, et al., 1986). Only 2 measures specifically addressed safety issues in the home and community, both of which were organized predominately by activity type (Chiu & Oliver, 2006; J. Kreutzer & Marwitz, 2000). Conversely, the SAM scale constructs are predominately organized by functional abilities that would identify the underlying causes of safety risk across everyday activities and participation. From these 11 measures, we binned and winnowed items that were potentially associated with our 6 primary scale constructs (DeWalt et al., 2007) Forty-six of these extracted items were modified and included in the field test version of the SAM scale item pools.

2.3.2. Item revision and development

To assure full coverage of real life experiences within each scale construct, the research and clinical team developed additional items. In addition to personal experience with the TBI population, we used the Centers for Disease Control and Prevention-National Center for Injury Prevention and Control (Centers for Disease Control and Prevention, (n.d.)) injury reporting structure for the general population to assist in identifying activities in which injuries may be likely to occur (see Table 1) (Centers for Disease Control and Prevention, (n.d.)). We supplemented the CDC list, based on feedback from the clinical team, clinical advisory board and family members, with other activities and aspects of harm that are common in persons who sustain moderate to severe TBI (see Table 1, column 1). We also considered the settings in which UIH occurs including inside and outside of the home; on streets, highways, sidewalks, and parking lots; in sports facilities, recreation centers, lakes, rivers and pools; and in the community such as in public buildings. Because the focus of our scale was adults who were transitioning from basic self-care to instrumental activities of daily living, we did not create items for hospital-based, school, or workplace settings.

Table 1.

Activities associated with unintentional injury or harm and their representation in the safety assessment measure item pool

Activities/Causes of Unintentional Injury or Harm	# Items on SAM	% of Items on SAM
CDC-NCIPC Related Activities or Causes
► Transportation (including motor vehicle, bicycle, motorcycle, pedestrian, train, boat, and airplane).	30	24%
► Falls, e.g., walking, standing, transferring	18	14%
► Aggressive or violent behavior	12	10%
► Identification, avoidance or response to persons or animals who intend harm	11	9%
► Medication (and medical condition) management	9	7%
► Recreation, e.g., sports, water activities	8	6%
► Fires, burns, scalds, e.g., use of fire, hot water, or chemicals	8	6%
► Poisoning (swallowing or breathing harmful substances such as bleach, carbon monoxide, or too many pills or drugs)	4	3%
► Use of electric devices or equipment	4	3%
► Use of firearms	2	2%
► Use of sharp objects or devices	2	2%
► Eating (food poisoning, allergic reaction)	2	2%
Additional Risk Activities in Persons with TBI
► Wheelchair Use	6	5%
► Money, Valuables Management	6	5%
► Sexual	4	3%
► Emergency Identification and Response	3	2%

Notes. CDC-NCIPC = Centers for Disease Control and Prevention, National Center for Injury Prevention and Control; Activities based on CDC-NCPIC: A Report to Congress, 1999. Total number of items reflected in column 2 (130) exceeds the total number of items on scale (126) due to a few items covering more than one activity.

Several sets of administration instructions and item rating schemes were considered that emphasized family and clinician raters’ direct observation of person behavior on specific activities, controlled for the confounds of assistance or supervision during activities, and allowed for judgment in determining levels of risk (DeVellis, 2006). The SAM rating instructions read as follows: “The following Items in this questionnaire ask about behaviors that could put a person with a brain injury at increased risk for harm. Based on your best judgment of the person’s current ability and behavior, please rate the likelihood of each behavior occurring if the person with TBI did not have any help or supervision – for example, if no one was there to step in, how likely is it that the person will not be able to exit a room on their own in an emergency?”

The item rating scheme has the following four responses: 4-highly likely to occur; 3-somewhat likely to occur; 2-somewhat unlikely to occur; and 1-highly unlikely to occur. A not applicable response is offered for a subset of items with a specific reason for each not applicable response noted, e.g., for the item, “Is not responsible with firearms,” “Not applicable, person with TBI does not use firearms”.

The research team edited and standardized item wording to: (a) fit the selected rating scheme; (b) be unidirectional, e.g., not reverse scored; (c) be easy to understand with examples provided if deemed helpful; and (d) maintain no higher than a 9th grade reading level. To meet the key assumptions of IRT, the research and clinical teams re-reviewed items and the proposed scale constructs for unidimensionality (Reeve & Fayers, 2005). Each item was then rated as being of high, medium, or low difficulty to assess coverage and identify “gaps” in the scale’s hierarchical difficulty continuum including floor and ceiling effects (Stone, 1997; Velozo et al., 2012). The clinical and scientific advisory boards provided feedback on the rating scheme, item selection and editing, and the final model and scales.

2.4. Evaluate and refine the item pool

2.4.1. Health provider focus groups and rating of content validity

Health providers participated in 6 focus groups to discuss item content, clarity, and comprehensiveness. Participants were recruited from Shepherd Pathways Day Rehabilitation Program, Marcus Community Bridge Program, Side-by-Side Brain Injury Clubhouse, and Restore Neurobehavioral Center in Atlanta, Georgia. The 5 clinicians involved in item development did not participate in the focus groups. Focus group sizes ranged from 2–6 participants and lasted 60–70 minutes. The research coordinator led semi-structured group interviews. Following group discussion, simple concept mapping was used in which healthcare providers independently quantified the extent that each scale item represented the safety experiences of persons with TBI using a 4-point rating scale (4-critically important, 3-very important, 2-somewhat important, and 1-not important). All focus group proceedings were audio taped and transcribed with identifiers removed.

Following the completion of all focus groups, the research coordinator reviewed the transcripts, created a spreadsheet summarizing respondent comments, and entered health provider item ratings into an SPSS data file. The mean importance score for each item was calculated. An Item Level-Content Validity Indicator (I-CVI) ratio was determined by computing the percentage of critically important plus very important ratings for each item (Polit & Beck, 2006). Excellent item validity is indicated by an I-CVI ratio of ≥78% (Polit & Beck, 2006). The research team reviewed the focus group summaries and content validity ratings, and modified, expanded, deleted, or added items accordingly.

2.4.2. Cognitive interviewing of family members and final item revisions

The lead research coordinator conducted cognitive interviews with family members of persons with TBI using a semi-structured script to obtain feedback on item comprehension, meaningfulness, and comprehensiveness (DeWalt et al., 2007; Willis, 2005). Interviews were audio-taped and transcribed. Five family members were interviewed and items revised based on their feedback. Following these revisions, four more family members were interviewed. With minor changes following their feedback, the research team was satisfied that the item pool was comprehensive and understandable.

3. Results

3.1. Social cognitive theory framework of unsafe events following TBI

Social cognitive theory was selected as the most suitable framework for understanding person, environment, and behavior factors that represent safety risk in the home and community. We conceptualized unsafe behavior and events as the result of interactions between the person with TBI, his or her everyday environment, and behavior (see Fig. 2). Personal factors including the abilities, beliefs and expectancies of a person who has sustained moderate to severe TBI are considered the primary determinants of safety risk and behaviors that result in UIH. The 6 primary SAM scales measure person abilities (indicated by bold lettering) and are hypothesized to be the most critical factors associated with UIH. Environmental factors such as family members’ abilities, beliefs, and expectancies; the home and community settings; social relationships; and community supports play mediating roles to varying degrees in shaping personal factors and behavior. In particular, the amount and proximity of family involvement (e.g., supervision, care coordination) is a primary environmental factor that interacts with person factors to influence UIH. The behavior factors of persons with TBI include their activity levels, performance of activities, experiential learning, social interactions, emotional responses, and working alliance with family members. These behavior factors provide an observable gauge of safe and unsafe functioning and feedback that shapes person and environmental responses to behavior.

Fig. 2.

Social Cognitive Theory model of safety risk in the home and community following TBI.

3.2. Scale constructs (Personal Ability Factors) and definitions

The proposed constructs, a summary of stakeholder and expert feedback on the constructs, and the final construct/scale names and their operational definitions are presented in Table 2. Primary decisions reflected in Table 2 include: (a) removing primarily upper extremity based activities from the Physical Capacity Scale and focusing on walking mobility as a primary construct, (b) adding Wheelchair Use as a primary construct/scale; and (c) separating activities and behaviors related to complying with medical recommendations from perceiving risk and creating separate scales. The six scales are: Cognitive Capacity, Visuomotor Capacity, Wheelchair Use, Risk Perception, Compliance Failures with Safety Recommendations, and Self-Regulation.

Table 2.

Person factor constructs (scales) and operational definitions for the safety assessment measure

Proposed Construct	Stakeholder / Expert Feedback and Research Team Decisions	Field Testing Scale Name	Operational Definition of Scale Construct
Cognitive Capacity	Agreement on core construct and that operational definition has unidimensionality	Cognitive Capacity	Cognitive impairments lead to a “lack of behavior” that creates specific safety risks Forgetfulness, inattention, slowed processing speed, lack of initiation, and difficulty understanding information drives risk Difficulty responding in an emergency situation
Physical Capacity	Determined that physical capacity related to upper and lower extremity functioning was not unidimensional Agreed on visuomotor capacity as a unidimensional core construct Agreed to measure presence of upper extremity impairments (e.g. spasticity, hemiparesis) in a physical and history questionnaire	Visuomotor Capacity	Visuomotor impairments lead to “unsuccessful behavior” that creates specific safety risks Visuomotor safety risks are secondary to vision, strength, coordination, balance, and dexterity
	Determined that a subset of persons with TBI used a wheelchair and that this posed risk Agreed that a range of behavior related to wheelchair use would be a unidimensional core construct	Wheelchair Use	Impaired attention, processing speed, judgment, balance, and/or impulsivity lead to wheelchair use behaviors that place self and/or others at risk Does not follow wheelchair safety precautions
Perceived Susceptibility to Harm	Agreed on core construct of perceiving risk; changed the construct name due to concerns that patients and families would not understand “perceived susceptibility” Created separate construct, “Compliance Failures with Safety Recommendations” to capture not following medical recommendations or activity restrictions	Risk Perception	Impaired judgment of risk due to naiveté or limited awareness of impairments Difficulty anticipating potentially harmful situations or consequences Does not follow common safety precautions
		Compliance Failures with Safety Recommendations	Does not seek medical treatment when needed Does not comply with recommended treatments, activity restrictions, or situational supervision Does not use needed assistive equipment
Self-Regulation	Agreement on core construct and that operational definition has unidimensionality	Self-Regulation	Impulsivity-acts directly on drives/emotions Disinhibition-inability to control impulsive responses to situations Aggression, sexual disinhibition, and wandering

Note. Following field testing, patients and families in focus groups to develop a translational report recommended renaming the scales; the final scale names reflect their feedback and research team decisions.

Risk due to substance use was developed as a stand-alone SAM scale and 19 items were generated. Following clinical and research advisory board review, there was consensus that alcohol and other drug use is not a unidimensional construct and that items would not conform to a hierarchical continuum. We decided to use existing measures to assess the presence and severity of alcohol and other drug use, and evaluate their association with safety risk. Motor vehicle operation and childcare items were initially developed and distributed across the 6 primary SAM scales. Given these activity sets are applicable to only a relatively small subset to persons with moderate or severe TBI, the clinical and research advisory boards and research team agreed that these activities would best be assessed on supplemental scales named Motor Vehicle Operation and Child Caretaking.

3.3. Safety Assessment Measure item pool and content validity ratings

Focus groups evaluated the Safety Assessment Measure item pool (n = 145 items) for the 6 primary and 2 supplemental scales. Participants were 26 TBI healthcare professionals with backgrounds in case management, nursing, occupational therapy, physical therapy, speech language pathology, therapeutic recreation, and psychology. On average, healthcare professionals were aged 40.7 (SD = 12.2), predominately female (85%), predominately white (69%) or black (19%), and were well-educated (42% with master’s and 35% with bachelor’s degrees). Health providers had a mean of 9.2 (SD = 7.9) years of TBI rehabilitationexperienceandreportedonaverage36.6 (SD = 9.5) hours weekly treating clients with TBI.

Based on focus group feedback and core research team re-examination of items, 32 items were removed from the pool primarily due to confounds (e.g., multiple activities or behaviors assessed within item content), similarity with retained items, or the activity lacked an imminent safety threat. The retained items (n = 115 items) and their content validity ratings, as well as 11 new items recommended in either the focus groups or the family member cognitive interviews are presented in Table 3, columns 1–3. Overall, the SAM item pool was rated as having high levels of content validity. Of the 115 retained SAM scale items, 106 items (92%) had Item-Content Validity Index ratios ≥78%, indicating excellent content validity. Sixty-nine (60%) of the retained items had I-CVI ratios ≥90%.

Table 3.

Safety assessment measure: Scales, items, and content validity

Scale and Associated Item Pool (n = 126 total items)	I-CVI	Mean	Activity/Cause	Primary Harm to:
Cognitive Capacity (n = 21 items)
Crosses street inappropriately (e.g., does not look, misinterprets walk signs, misjudges traffic)	100%	3.96	Transportation	Self
Mistakenly takes wrong dose of medication (e.g. too little, takes twice)	100%	3.96	Medical Management	Self
Forgets to take prescribed medication (e.g. seizure, diabetes, high blood pressure)	100%	3.92	Medical Management	Self
Uses other people’s prescribed medications (e.g. pain, sleep)	100%	3.92	Medical Management	Self
Does not know when an injury or illness requires medical care (e.g. deep cut, chest pain)	100%	3.77	Medical Management	Self
Leaves cooking food unattended	100%	3.67	Fire/Burns	Self. Property
Forgets to use mitt or towel when picking up hot items	100%	3.63	Fire/Burns	Self
Leaves flammable items too close to fire (e.g. paper bag near burner)	96%	3.74	Fire/Burns	Self, Property
Forgets to turn off appliances (e.g. stove, iron, space heater)	96%	3.74	Fire/Burns	Self, Property
Does not respond to emergency alarms (e.g. smoke, carbon monoxide, weather)	93%	3.81	Emergency Response	Self
Sets stove burner too high	93%	3.30	Fire/Burns	Self, Property
Does not follow emergency procedures	89%	3.76	Emergency Response	Self
Forgets to avoid foods that cause allergic reaction	85%	3.30	Eating	Self
Unable to find way if lost in the community	82%	3.44	Avoid Harmful Others	Self
Forgets to lock doors and windows when leaving home	74%	2.85	Money, Possessions	Property
Misuses microwave (e.g. puts metal objects inside, burns food)	74%	2.81	Electric, Equipment	Property
Loses wallet or purse	67%	2.81	Money, Possessions	Property
Forgets to put medications away	62%	2.76	Poisoning	Others
Forgets to put sharp objects away (e.g., scissors, knives)	62%	2.71	Sharp Objects	Others
* Does not react appropriately to warning signs (e.g. wet floor, construction, etc.)	NR	NR	Avoid Harmful Others	Self
* Does not check surroundings before walking through a parking lot	NR	NR	Transportation	Self
Visuomotor Capacity (n = 14 items)
Loses balance when walking up or down stairs	100%	3.85	Falls	Self
Does not move to an exit in an emergency	100%	3.81	Emergency Response	Self
Loses balance getting in or out of tub	100%	3.74	Falls	Self
Loses balance getting in and out of car	96%	3.67	Falls	Self
Loses balance while getting on or off toilet	96%	3.63	Falls	Self
Loses balance when attempting to stand from a sitting position	96%	3.56	Falls	Self
Loses balance when reaching forward	96%	3.41	Falls	Self
Trips on things (e.g., curbs, objects, uneven surfaces)	93%	3.38	Falls	Self
Loses balance when walking on fiat surfaces	93%	3.59	Falls	Self
Loses balance when carrying objects	93%	3.52	Falls	Self
Trips when walking in low lit areas	93%	3.41	Falls	Self
Loses balance when dressing	89%	3.44	Falls	Self
* Bumps into walls, objects, or people	NR	NR	Transportation	Self, Property
* Trips when walking in well lit areas	NR	NR	Falls	Self
Wheelchair Use (n = 6 items)
Uses wheelchair in potentially dangerous environments (e.g. street, rough terrain)	93%	3.78	Wheelchair Use	Self
Does not follow wheelchair precautions (e.g. locking brakes, wearing seatbelt, keeping fingers away from spokes, etc.)	89%	3.59	Wheelchair Use	Self
Uses wheelchair without required assistance	85%	3.58	Wheelchair Use	Self
Moves too fast in wheelchair	81%	3.27	Wheelchair Use	Self, Others
Loses control of wheelchair (e.g. tips wheelchair)	81%	3.12	Wheelchair Use	Self
* Backs up without checking for people or objects	NR	NR	Wheelchair Use	Self, Others
Risk Perception (n = 17 items)
Stands on unstable objects (e.g. chair with wheels, top step of ladder, cardboard box)	100%	3.85	Falls	Self
Rides in a motor vehicle driven by someone who has been drinking or doing drugs	100%	3.77	Transportation	Self
Inappropriately reveals personal information (e.g. bank acct number, social security number)	100%	3.73	Money, Possessions	Property
Easily influenced to do risky behaviors	96%	3.63	Avoid Harmful Others	Self, Property
Falls asleep while smoking	93%	3.85	Fire, Burns	Self, Property
Invites strangers into home	93%	3.67	Avoid Harmful Others	Self, Property
Operates equipment while drowsy (e.g., power tools, lawnmower)	93%	3.63	Electric, Equipment	Self
Meets in person with strangers found on the internet	93%	3.58	Avoid Harmful Others	Self, Property
Goes alone in dangerous areas (e.g. high crime, poorly lit areas)	89%	3.65	Avoid Harmful Others	Self, Property
Jumps or dives into a body of water without knowing depth	85%	3.62	Recreation	Self
Goes with stranger (e.g., gets in car, goes to home)	85%	3.60	Avoid Harmful Others	Self, Property
Swings objects without looking for others in path (e.g., clubs, yard equipment)	85%	3.31	Recreation and Equip.	Others, Property
Goes on slippery surfaces	82%	3.37	Falls	Self
Easily tricked into giving strangers money	81%	3.19	Avoid Harmful Others	Property
Readily gives money to family or friends without considering consequences	78%	3.04	Money, Possessions	Property
Approaches unknown animals	74%	3.00	Avoid Harmful Others	Self
Does not prepare for weather (e.g. dress warm on a cold day, wear sunscreen on sunny day)	67%	2.73	Recreation	Self
Compliance Failures with Safety Recommendations (n = 20 items)
Refuses to take prescribed medications (e.g., seizure, diabetes, high blood pressure)	100%	3.96	Medical Management	Self
Drives motor vehicle against recommendations	100%	3.89	Transportation	Self, Others
Does not wear a helmet when recommended (e.g. bicycle, scooter, motorcycle)	100%	3.77	Transportation	Self
Does not follow precautions when using electrical equipment (e.g. power tools, kitchen appliances)	100%	3.73	Electric, Equipment	Self, Property
Does not follow precautions when using fire	100%	3.69	Fire, Burns	Self, Property
Does not follow precautions when using sharp devices	100%	3.48	Sharp Objects	Self
Uses firearms against recommendations	96%	3.78	Firearms	Self, Others
Operates recreational vehicle (e.g. golf cart, boat, jet ski) against recommendations	96%	3.63	Recreation	Self, Others
Climbs against recommendations (e.g., stool, ladder, stairs)	93%	3.62	Falls	Self
Ignores advice to seek medical care	93%	3.48	Medical Management	Self
Participates in sports or recreational activities against recommendations	93%	3.46	Recreation	Self
Does not follow precautions on warning labels	93%	3.33	Medical/Fire, Burns	Self
Is not responsible with firearms	91%	3.61	Firearms	Self, Others
Does not wear a seatbelt	85%	3.44	Transportation	Self
Does not follow exercise precautions (e.g. runs in severe heat, lifts too much weight)	85%	3.37	Recreation	Self
Goes on amusement park rides (e.g. roller coaster) against recommendations	85%	3.22	Recreation	Self
Ignores important diet restrictions for medical condition (e.g. diabetes, swallowing disorder)	82%	3.42	Medical/Eating	Self
Goes in water without recommended supervision (e.g., bath, pool, natural body of water)	82%	3.35	Medical/Recreation	Self
Does not follow precautions when using chemical solutions (e.g. mixing cleaning products, insecticide in closed space)	78%	3.40	Poisoning	Self
Does not use recommended safety equipment (e.g., walker, safety rails, shower chair)	74%	3.20	Falls	Self
Self Regulation (n = 20 items)
Physically attacks strangers	100%	3.89	Aggressive, Violent	Self, Others
Easily provoked into physical fight	100%	3.78	Aggressive, Violent	Self, Others
Makes unwelcome physical contact with others	100%	3.73	Sexual	Self, Others
Keeps doing activities after being told to stop	100%	3.62	Falls/Transportation	Self
Physically abuses family members or caregivers	96%	3.81	Aggressive, Violent	Self, Others
Exchanges in sex for goods (e.g., money, drugs, cigarettes; pays for sex)	96%	3.69	Sexual	Self, Others
Engages in unsafe sex (e.g., unprotected sex, one night stands, multiple partners)	96%	3.69	Sexual	Self, Others
Shoves or pushes others	93%	3.48	Aggressive, Violent	Self, Others
Physically abuses animals	92%	3.58	Aggressive, Violent	Self, Animals
Wanders	89%	3.48	Avoid Harmful Others	Self
Makes unwelcome sexual remarks	89%	3.33	Sexual	Others
Throws, hits, or kicks objects in anger	89%	3.26	Aggressive, Violent	Self, Others
Gambles excessively	88%	3.20	Money, Possessions	Self, Property
Verbally abuses others	85%	3.52	Aggressive, Violent	Others
Sends abusive or obscene messages (e.g. email, text, voicemail)	85%	3.31	Aggressive, Violent	Self, Others
Tries to eat non-food items	85%	3.11	Poisoning	Self
Cannot stop spending money	84%	3.12	Money, Possessions	Self, Property
Verbally abuses family and friends	81%	3.36	Aggressive, Violent	Self, Others
Takes other people’s things	81%	3.28	Aggressive, Violent	Self, Others
Provokes animals	74%	2.93	Avoid Harmful Others	Self, Animals
Motor Vehicle Operation (n = 19 items)
Does not stop at red lights or stop signs	100%	3.89	Transportation	Self, Others
Texts messages while driving	100%	3.89	Transportation	Self, Others
Does not yield when required (e.g. for pedestrians, before merging onto road)	100%	3.85	Transportation	Self, Others
Does not look out for pedestrians or bicyclists	100%	3.85	Transportation	Self, Others
Changes lanes without looking	100%	3.78	Transportation	Self, Others
Operates motor vehicle when on sedating medications	100%	3.78	Transportation	Self, Others
Backs up without looking	93%	3.78	Transportation	Self, Others
Easily distracted when driving	93%	3.77	Transportation	Self, Others
Talks on cell phone while driving	93%	3.70	Transportation	Self, Others
Operates motor vehicle when drowsy	93%	3.63	Transportation	Self, Others
Misjudges turns (e.g. hits curbs, goes too wide)	89%	3.58	Transportation	Self, Others
Reacts slowly to changing traffic (e.g., missing traffic signs, not braking soon enough)	89%	3.54	Transportation	Self, Others
Drives too closely behind other vehicles	89%	3.52	Transportation	Self, Others
Makes illegal turns	87%	3.63	Transportation	Self, Others
Drives too fast (15 + MPH over limit)	83%	3.33	Transportation	Self, Others
Gets lost when driving	81%	3.33	Transportation	Self
* Drifts out of lane	NR	NR	Transportation	Self, Others
* Impulsively exits or makes turns (e.g. crosses multiple lanes, swerves at last minute)	NR	NR	Transportation	Self, Others
* Forgets where he or she is going	NR	NR	Transportation	Self
Child Caretaking (n = 9 items)
Leaves children unattended (e.g., left alone in home, in bath, in park)	100%	4.00	Multiple	Self, Others
Drives with child in the car after using alcohol or drugs	100%	3.96	Transportation	Self, Others
Physically harmful to children	100%	3.93	Aggressive, Violent	Self, Others
Does not buckle child in seatbelt, car seat, or booster seat	100%	3.85	Transportation	Self, Others
Allows small child to ride in the front passenger seat	96%	3.65	Transportation	Self, Others
Verbally harmful to children	93%	3.74	Aggressive, Violent	Self, Others
* Leaves child in car while running errands	NR	NR	Poisoning	Self, Others
* Does not seek medical help when child is injured or ill	NR	NR	Medical Management	Self, Others
* Carries child despite physical limitations	NR	NR	Falls	Self, Others

Each SAM item was rated for its relevance to preventing unintentional injury or harm as follows: 1 = not at all important; 2 = somewhat important; 3 = very important; 4 = critically important. I-CVI = Item Level-Content Validity Index (the percentage of stakeholders who rated an item as very or critically important; Mean = Average of Item Level Content Validity Ratings; ‘*’ = items added as a result of feedback from the health provider focus groups or family interviews. NR = not rated because they were newly generated from feedback during the focus group and consumer rating sessions

Overall, the SAM item pool covered the full range of activities, behaviors, and causes of injury in the general population and specific to persons with TBI (see Table 1, column 2.). Items related to transportation activities (24%), falls (14%), aggressive or violent behaviors (12%), and identifying, avoiding, or responding to potentially harmful others (9%) comprised 57% of the item pool content. For each of the 6 primary scales and 2 supplemental scales, item content reflected a wide range of activities, behaviors, and causes of injury (see Table 3, columns 4) as well as the people and property that could be potentially affected by harmful activities(see Table 3, columns5). The Cognitive Capacity and Compliance Failures with Safety Recommendations scales covered the largest number of activities/behaviors within their scale content with 10 each. The Risk Perception and Self-Regulation scales had 7 and 6 different activities or behaviors, respectively, covered in their item content. The Visuomotor Capacity, Wheelchair Use, and Motor Vehicle Operation scales had more focused content.

4. Discussion

4.1. Summary of study findings

Family, provider, and payer understanding of the specific activities in the home and community that pose threats and the cognitive, physical, and behavioral impairments that underlie risk is essential to managing the safety of persons who have sustained TBI. Our study presents a conceptual framework for understanding safety risk in the home and community following moderate to severe TBI. Person factors were identified as the key determinants of unsafe behavior and six primary constructs-Cognitive Capacity, Visuomotor Capacity, Wheelchair Use, Risk Perception, Compliance Failures with Safety Recommendations, and Self-Regulation-represented the most critical person factors that underlie safety risk. This conceptual framework and the operational definitions for the primary person constructs served as the basis for developing the SAM scales and item pool.

The SAM assesses risk on a large number of IADLs as well as the underlying abilities/impairments that underlie safe and unsafe performance of these activities. Our findings provide evidence that the SAM scales and item pool (n = 126 items) cover a wide range of real life experiences and everyday activities that place persons with TBI at risk for UIH. The SAM items cover 16 broad sets of activities, behaviors, and causes of UIH, most of which map to how the CDC monitors unintentional injury causes and activities in the general population. The SAM items also reflect the primary settings such as the home, roads, recreational centers, and the wider community in which harm occurs and the casualties of harm including persons with TBI, other people, animals, and property. The SAM items were generally viewed as being comprehensive, rated as having excellent content validity, and seen as being easily understandable based on feedback from rehabilitation providers participating in focus groups and family members who engaged in cognitive interviews.

4.2. Safety assessment measure methodological considerations and limitations

The structured qualitative methods that we used to develop the SAM scales and item pools are consistent with recommendations to develop “patient-centered” outcome measures that encompass patients’ needs, values and everyday experiences when assessing their health, function, and participation (Administration, 2009; Frank & Polkinghorne, 2010; Lasch et al., 2010; Ohman, 2005; Onwuegbuzie, Bustamante, & Nelson, 2010). Fifteen months was invested in developing a conceptual framework and item pool that reflects a wide range of activities that pose potential harm to the general population as well as persons who have sustained TBI. This time investment including feedback from clinical and family stakeholders was critical to achieving an item pool with high levels of content validity.

Our emphasis on using health provider feedback in developing SAM content reflected our decision to leverage a highly experienced group of clinicians who could reflect on and report their observations of thousands of persons who have sustained TBI in inpatient facilities, outpatient clinics, and in the home and community. We sought family member feedback when evaluating item clarity and understandability so that the SAM would be most usable to a wide range of family members making observations in the home and community. Based on our experience with developing item pools for individuals with disabilities, we used an interview format for the family caregivers, which allowed the instruction that family caregivers need to understand the review process and the individualized attention and time needed to provide comprehensive feedback. Field testing of the SAM with a large group of family raters, including 30-day follow-up interviews on unsafe events, will provide further opportunities to obtain detailed feedback on item content comprehensiveness and clarity.

Our decision to develop items that rate unsuccessful behaviors in the home and community reflects the nature of assessing risk behavior. We also received feedback from families that they found it easier to rate unsuccessful behavior when assessing levels of safety risk. A number of healthcare providers recommended increasing the specificity of causal assessment data as part of the SAM items, e.g., differentiating visual and physical components of losing balance in the Walking Mobility scale item content. Given the importance of obtaining reliable, family rated observations of behavior in the home and community, we decided that these distinctions would be difficult for family raters to make. While pre-injury abilities can play significant roles in safety risk (Pietrapiana et al., 2005), the SAM measures current ability and does not distinguish between pre-injury and injury-related abilities.

The SAM item pool addresses instrumental activities of daily living that may lead to unintentional injuries or harm in the home and community. The SAM does not assess unsafe activities in inpatient medical, school or work settings. The SAM item pool does not describe pre-meditated, intentional harmful acts to one’s self (e.g. self-mutilation, suicide), another person, or property, which are best assessed by existing psychiatric measures. The SAM also does not address the risk of persons with severe TBI being victims of violence by a parent, spouse, child, or other family caregiver in their home. This issue is not well-suited for family and caregiver self-report assessment. The SAM may not be applicable to persons who are dependent in most basic self-care activities and thus would not attempt more complex instrumental activities of daily living.

The SAM was developed for persons at least aged 16 and older who are likely to have been independent in many everyday activities prior to sustaining TBI. Young children, adolescents, or adults with significant pre-injury developmental disability would likely have different presentations, interests, and developmental expectations. A pediatric measure of safety would need to address the expectations and environments in which children participate (e.g., school), as well as consider developmental changes that reflect recovery, ongoing physical and cognitive development, and the acquisition of new versus previously learned functional skills.

4.3. Contributions of this study to the literature and future research

Our development of the SAM item pool is among the first efforts to use state-of-the-science methods to identify a broad range of specific activities and behaviors that are linked to risk of UIH and categorized by common TBI-related impairments. The SAM is unique compared to the few existing safety instruments in that it’s scales and item content are: (a) more highly connected to a theoretical framework for understanding safety risk following TBI; (b), more comprehensive and categorized by person functional ability rather than a limited number of activities; and (c) linked to how the U.S. CDC monitors activities and causes of unintentional injury. The comprehensiveness of the activities covered by the SAM scale items and their categorization by person functional ability will help clinicians and family members identify individual activities and their underlying causes that drive safety risk and prioritize those areas for treatment planning, intervention, progress monitoring, and prognosis.

The SAM is also unique compared to existing safety instruments in that our construct and scale development procedure included formally evaluating the item pool by using semi-structured focus groups with rehabilitation providers and cognitive interviewing of family members. This formal evaluation process with multiple stakeholders produced empirical, quantitative evidence that the content validity of the SAM item pool is generally perceived to be excellent, and increases the likelihood that the SAM scales and items can ultimately standardize observations and improve communication between families, clinicians and payers. Lastly, the SAM is unique in that it was developed using an item response theory framework that emphasized developing items within scales that were unidimensional and covered a range of low, medium and high activity difficulty. This will allow each SAM scale to be further evaluated using Rasch analysis.

4.4. Future research

The SAM scales and items will be field tested on a TBI development sample rated by family members and rehabilitation clinicians. Scale validation using IRT methods and Rasch analysis would evaluate the evidence of scale unidimensionality, person and item fit, and precision. Longitudinal tracking of persons with TBI unsafe behavior and events following a baseline SAM assessment would provide important data to evaluate the diagnostic and prognostic validity of the SAM scale scores. Evidence of diagnostic and prognostic validity would improve interpretation of SAM scale scores so that they could inform optimal levels of family involvement in person with TBI activities and the least restrictive environment in which persons who have sustained TBI can safely participate in activities. Health payers may find value in using SAM scale scores to identify high risk individuals who would benefit from community supports or home healthcare. The relationship between SAM scale scores and formal tests of cognitive and executive function in persons with TBI would also be of interest.

5. Conclusions

The Safety Assessment Measure scales and associated item pools assess a broad range of instrumental activities of daily living that can lead to unintentional injuries or harm in the home and community for adults who have sustained moderate or severe TBI. The Safety Assessment Measure scales assess risk related to diminished cognitive capacity, visuomotor capacity and/or wheelchair use, risk perception, compliance with safety recommendations, and selfregulation. Interdisciplinary rehabilitation clinicians rated most SAM items as having excellent content validity. Future research should use modern psychometric methods to examine scale unidimensionality, model fit, and precision.