Development and Content Validity of the Behavioral Assessment Screening Tool (BAST)

Abstract

Purpose:

Develop and establish the content validity of the Behavioral Assessment Screening Tool (BAST_β), a self-reported measure of behavioral and emotional symptoms after traumatic brain injury.

Methods:

This was an assessment development study, including two focus groups of individuals with traumatic brain injury (n=11) and their family members (n=10) and an expert panel evaluation of content validity by experts in traumatic brain injury rehabilitation (n=7). We developed and assessed the Content Validity Index of the BAST_β.

Results:

The BAST_β initial items (n=77) corresponded with an established conceptual model of behavioral dysregulation after traumatic brain injury. After expert panel evaluation and focus group feedback, the final BAST_β included 66 items (60 primary, 6 branching logic) rated on a three-level ordinal scale (Never, Sometimes, Always) with reference to the past two weeks, and an Environmental Context checklist including recent major life events (n=23) and 4 open-ended questions about environmental factors. The BAST_β had a high Content Validity Index of 89.3%.

Conclusion:

The BAST_β is a theoretically grounded, multidimensional self-reported assessment of behavioral dysregulation after traumatic brain injury, with good content validity. Future translation into mobile health modalities could improve effectiveness and efficiency of long-term symptom monitoring post-traumatic brain injury. Future work will establish and validate the factor structure, internal consistency reliabilities, and other validities of the BAST.

Introduction

Over 5 million United States residents are currently living with traumatic brain injury (TBI)-related disabilities.[1] The long-term consequences of TBI include physical, cognitive, emotional, and behavioral symptoms that can persist for decades and negatively affect community participation, health, and quality of life.[2–5] Of particular significance is the increased risk for suicidal ideation and attempt after TBI.[6–10] Suicidality is highly associated with behavioral symptoms after TBI, such as impulsivity, that link directly to injury-induced neuroanatomical changes.[11–16] Identifying individuals with these behavioral symptoms through community-based symptom monitoring could have substantial impact on TBI-related mental health care and rehabilitation.

Behavior refers to the way in which a person acts, specifically in response to a particular situation, person, or other internal or external stimulus. In other words, it is the outward, observable manifestation of the complex interaction between a person’s internal world – thoughts, feelings, personality, biological processes – and external world (e.g. environment). Behavioral problems, including aggression, disinhibition, poor motivation or initiation, and difficulty with planning/executing actions,[17,18] often occur as part of a larger syndrome that includes emotional and cognitive changes that can persist for decades after moderate to severe TBI.[4,5,19] These behavioral problems are most likely to occur in the context of a chronic emotional stressor, such as depression,[20–25] which occurs in ~50% of individuals in the first year post-TBI.[26] Effectively assessing behavioral problems is critical for the provision of appropriate and effective interventions, as behavior is known to be one of the greatest contributing factors to poor outcomes – both acutely and chronically – after injury.[15,19,27]

While the long-term negative consequences of behavioral problems are well documented, behavioral symptoms remain particularly challenging to measure given their broad range, individual variability, and scope. Behavioral problems often manifest in response to environmental stimuli.[28] Therefore, effectively monitoring behavioral/emotional symptoms as a method to identify behavioral problems requires a self-reported measurement tool that assesses the complexity of behavioral/emotional symptoms and environmental/personal contexts repeatedly over time and within an individual’s natural environment. Frequent monitoring of symptoms in real-world environments is critical for preventing symptom escalation and providing interventions that target coping deficits.

While scientifically rigorous studies have validated behavioral and emotional assessments for use after TBI, specific weaknesses in these existing assessments still result in challenges that impede their usefulness in community-based research and clinical practice. These challenges include caregiver/clinician report versus self-report, length/breadth of measures, complexity of administration, and ecological validity. Many behavioral surveys rely on the report of a knowledgeable observer, which is rarely feasible to obtain for individuals living in the community and not, itself, without bias.[29] Additionally, many of these symptoms reflect internal states (e.g. feeling agitated or depressed), rather than external and observable characteristics (e.g. getting into fights, sleeping all day), which cannot be reliably reported by an external observer.

Existing self-reported assessments are often domain-specific (e.g. an aggression scale), which may not best represent the profile of behavioral and emotional symptoms experienced by individuals with TBI. While there are numerous tools available to measure a wide variety of constructs, ranging from broad scales measuring neurobehavioral symptoms to scales focusing on specific symptoms or behaviors (e.g. impulsivity, aggression, apathy),[29,35] there are few tools that measure multiple components of complex behaviors. For example, when examining the literature related to impulsivity, the challenges in defining and measuring these types of complex behaviors become apparent. Impulsivity can have motor and/or verbal components and has been linked with decision-making (impulse/inhibitory control).[50] It also shares many features with aggression, supporting the need to assess impulsivity and aggression simultaneously.[17] Additionally, coping style may influence the manifestation of behavioral changes after TBI. Positive coping involves multiple dimensions, including Active Coping (directly addressing an event and its effects), Planning, Inhibition of other activities, and Restraint (waiting for an appropriate time to address a problem).[28] Dysfunctional coping may include behavioral/mental disengagement (avoidant coping).[28] Coping style, though highly overlapping with behavioral symptoms, is not currently assessed as part of any behavioral assessment for TBI.

Assessments are often long, involve shifting and confusing scales, and are designed for administration by a trained professional who can explain technical language or confusing wording as needed. Common cognitive impairments after TBI may make most questionnaires challenging to read, comprehend, and respond to in a valid manner. Therefore, they do not effectively translate to community-based monitoring, which relies on independent self-reporting over time. Though self-reported assessments may be prone to reporting errors as a result of impaired self-awareness or recall bias commonly associated with TBI-related cognitive impairment,[30,31] they still represent the most efficient and practical method for monitoring and measuring long-term symptoms.

Increasing the frequency of assessments and focusing on individuals’ immediate experiences may reduce limitations of self-reporting. This type of assessment could be done through ecological momentary assessment (EMA), which involves repeated measures of, for example, behavioral symptoms, in real time (i.e. focused on a specific, short time frame versus a global assessment over a longer time frame) and in an individual’s natural environment (i.e. community settings versus the clinic).[31] An easy to complete and relatively brief assessment would maximize the likelihood that individuals could complete the high frequency repeated assessments necessary for EMA. However, measurement tools must be developed and validated for use in EMA prior to implementation. It is also crucial for self-reported assessments to be accessible, comprehensive, and as brief as possible. Our pilot work revealed that tracking emotional symptoms after TBI via mobile health (mHealth) technology (e.g. smartphone app) was feasible,[32] but the well-validated emotional assessments we used, including the Patient Health Questionnaire 2,[33] General Anxiety Disorders 2,[33] and Positive and Negative Affect Schedule,[34] were not specifically validated for use in EMA. Participants with TBI in the same study stated that assessments should be brief, be less redundant, employ a single response scale, use accessible language (simple, minimal jargon), and be presented in an easy to follow format. No available assessments of behavioral and emotional symptoms currently meet all of these criteria specified by individuals with TBI.

Therefore, there is a significant need for an ecologically valid measure of behavioral problems that incorporates cognitive, emotional, and coping components and can be completed independently via self-report by individuals with moderate to severe TBI who are living in the community. We previously developed a conceptual model of behavior to capture the multidimensional nature of behavioral problems after TBI and to serve as a foundation for the development of an ecologically valid behavioral assessment that could leverage mHealth technology.[24] Our conceptual model frames behavior as an overarching concept that incorporates multiple interacting domains, including emotions, cognitive function, and personal factors, all in the context of environmental supports and stressors.[24] Assessing these multiple domains simultaneously is necessary to adequately identify behavioral problems. While there are numerous tools available to measure a wide variety of behavior-related constructs, ranging from broad scales measuring neurobehavioral symptoms to scales focusing on specific symptoms or behaviors (e.g. impulsivity, aggression, apathy),[29,35] no current behavioral or emotional assessment comprehensively covers all of the domains represented in this conceptual model.

The purpose of the present study was to initially develop a self-reported assessment tool that measures behavioral problems after TBI, the Behavioral Assessment Screening Tool (BAST), to provide a long-term symptom monitoring measure that addresses these current limitations. The aims of the study were to develop the beta version of the BAST (BAST_β), based on our conceptual model, and establish its content validity through an iterative process of consumer input and expert panel review.

Materials and Methods

Development and Content Validity of the BAST

Development and establishment of the content validity for the BAST_β occurred through an iterative process of consumer and expert feedback and revision, including: 1) development of an initial item set; 2) a consumer focus group on the initial item set; 3) expert panel evaluation of the initial item set; 4) revision of the initial item set; 5) a consumer focus group on the revised item set; and 6) final expert evaluation of the revised items. All participants provided verbal informed consent, per approval from the University of [REDACTED] Institutional Review Board.

Development of the initial item set

Initial proposed items for the BAST_β were generated by the principal investigator on the project (SJ) based on our previously published conceptual model[24] (see Figure 1) and general content from validated assessments covering multiple components of behavioral dysregulation – including the Frontal Systems Behavior Scale,[36] Aggression Questionnaire,[37] and Coping Inventory in Stressful Situations[38] – and validated measures of emotional symptoms – including the Patient Health Questionnaire-9,[39] General Anxiety Disorders-7,[40] and Positive and Negative Affect Schedule.[34] Items were also added to assess self-awareness, based generally on content from the Self-Awareness of Deficits Interview and the Self-Regulation of Skills Interview.[41–43]

Figure 1. Mapping previously validated assessments to the conceptual model of behavioral dysregulation to guide content generation for the Behavioral Assessment Screening Tool.

This figure demonstrates how previously validated assessments and expert knowledge were selected for content review, based on our conceptual model of behavioral dysregulation after TBI. The original and complete conceptual model was previously published in the Journal of Clinical and Experimental Neuropsychology.[24] FrSBe=Frontal Systems Behavior Scale; PHQ9=Patient Health Questionnaire 9; GAD7=Generalized Anxiety Disorders 7; SADI=Self-Awareness of Deficits Interview; SRSI=Self-Regulation Skills Interview; PANAS=Positive and Negative Affect Schedule; CISS=Coping Inventory in Stressful Situations.

The principal investigator removed redundant items and rewrote remaining items into more accessible language, including: 1) shorter sentences communicating a single concept rather than multiple concepts in a single sentence; 2) language targeted to the 8^th grade reading level; 3) questions formulated to be answered using a single scaling rubric (ordinal scale) over a common time frame; and 4) no double-barreled questions (e.g. offering multiple options, such as “x or y”).

Expert Panel

The principal investigator, a rehabilitation counselor, oversaw the expert panel, which included one mother of a 30-years post-injury survivor of TBI and six professionals with expertise and lengthy experience in TBI rehabilitation, including a physiatrist, two neuropsychologists, a physical therapist, an occupational therapist, and a speech-language pathologist. Expert panel members provided feedback by: 1) rating the clarity and the relevance of individual items on a 1–5 point scale; 2) voting to include or exclude each item; and 3) providing qualitative feedback regarding content and wording. The threshold for inclusion of an item was 83% agreement to include the item, based on proposed guidelines.[44] A final content validity index was calculated by averaging the individual item percent agreement for inclusion, with a threshold of .80 (80% agreement) set as an indicator of good content validity.[45]

Focus Groups

We obtained consumer input on the instrument, based on established methods,[46,47] in two focus groups including a mix of adults with TBI (n=11 total) and their family members (n=10 total) attending two local hospital-based TBI support groups. Though no specific data were collected about injury severity, all participants with TBI had been hospitalized for their injuries. For the first focus group, individuals with TBI (n=4) ranged in age from 50–60 years and all were male. They sustained their injuries between 1976 and 1987. Their family members (n=5) ranged in age from 45–76 and two (40%) were male. For the second focus group, individuals with TBI (n=7) ranged in age from 25–68 years old and were injured between 1975 and 2013. Four (57%) were male. Their family members (n=5) ranged in age from 57–72 and two (40%) were male. We solicited feedback through structured and open-ended questions about the appearance, instructions, item wording and content, and response format of the BAST_β. The first focus group provided feedback on the initial item set, concurrently with the expert panel. The second focus group provided feedback on the item set revised based on the first focus group and expert panel feedback.

Final Expert Evaluation

A physiatrist and a neuropsychologist with extensive expertise in TBI rehabilitation who did not participate on the expert panel, and who were collaborators in the development of the conceptual model upon which the BAST_β was based, reviewed the final item set for content, clarity, and consistency with the conceptual model of behavior.

Process

Figure 2 depicts the overall process through which the development and establishment of the content validity occurred.

Figure 2. BAST Item Development Process.

Instrument design and item selection occurred through an iterative process involving clinicians and researchers with expertise in traumatic brain injury rehabilitation, individuals with a history of traumatic brain injuries, and care partners of individuals with traumatic brain injuries.

Figure 2 depicts the overall process through which the development and establishment of the content validity occurred.

Results

Development and Content Validity

The expert panel reviewed and rated 77 initial items on the BAST_β. We used the follow criteria to revise the BAST_β items after the expert panel review: 1) 83% threshold for agreement to include; 2) expert panel scoring of clarity and relevance for each item on a scale of 1 (not clear/relevant) to 5 (very clear/relevant). Scores below 3 out of 5 prompted revision or removal of the item; and 3) expert qualitative feedback on wording and content. Based on expert panel feedback, we made the following revisions to the BAST_β items: 1) 34 items were kept; 2) 22 items were removed; 3) 21 items were revised (2 of these revised items were combined into 1 item, leaving 20 final revised items); and 4) 3 new items were created. We calculated a Content Validity Index (CVI) of 89.3% for the 54 kept and revised items, indicating good content validity.[45] Since revised items changed only wording rather than specific content, the initial expert panel agreement to include was used for the CVI. On a scale of 1–5 (“not at all” to “extremely”), the expert panel rated the average relevance and clarity of these 54 items as 4.30 and 4.47, respectively, indicating high relevance and clarity.

Based on feedback obtained from the two focus groups, we added 10 new items, modified the instructions to be clearer, reordered items, and changed all language to past tense. The 3-point response scale of “Never, Sometimes, Frequently” was selected based on consensus across both focus groups.

The final BAST_β (see Supplementary Table s1) included 67 items (54 from the original, 3 new items from the expert panel, and 10 new items from the focus groups); six of these were branching logic items related to coping strategies only asked if an individual endorsed the item, “I felt stressed”. The BAST_β also included a checklist of major life events (n=23) in the past 6-months, two yes/no questions asking about other factors or supports that could have affected the individual, followed by open-ended explanation if an individual selected yes, and an open-ended question about the individual’s greatest problem or need in the past 2 weeks. We used Microsoft Word’s readability statistics, specifically the Flesch-Kincaid Grade level test, to assess the reading level of the BAST_β. The Flesch-Kincaid score was 4.2, indicating the language fell at a 4^th-5^th grade reading level.

Discussion

We developed the BAST_β, a screening tool for measuring and monitoring behavioral and emotional symptoms among community-dwelling adults with TBI, and established its content validity. The BAST addresses many of the challenges that impede the usefulness of currently available behavioral measures for community-based research and clinical practice, including issues related to caregiver or clinician report versus self-report; length, breadth, and complexity of existing measures; and ecological validity. Based on a conceptual model that proposes a complex, multidimensional structure to behavioral problems post-TBI, the BAST_β is not intended as a comprehensive evaluation of behavior that will replace clinical interviews and observational assessments; rather, it is designed for long-term monitoring and community-based screening of a significant problem post-TBI that often goes undetected, and therefore, untreated.

There are multiple limitations of self-reported assessments for individuals with TBI. However, for long-term, community-based tracking of behavioral and emotional symptoms, self-report is the most practical and, in the case of internal processes (e.g. emotions), the only viable assessment method. Self-reported assessments are vulnerable to impaired self-awareness and poor recall, which are particularly common in the context of associated cognitive impairment after TBI. Increasing the frequency of assessment and focusing more on individuals’ immediate experiences, as done when using EMA, may reduce the limitations associated with self-reporting after TBI.[31] EMA is an effective and increasingly popular means of assessing a variety of constructs in diverse, community-based populations. However, even with validated assessment tools, conducting EMA after TBI presents several challenges, including, but not limited to, burden on healthcare systems to complete telephone calls, send reminders to complete assessments, and collect data over multiple time points, as well as burden on individuals to complete assessments. With the ever increasing availability of accessible technology (e.g. smartphones), EMA continues to evolve in being less burdensome and more efficient.[51] We developed the BAST with the intention of later translation into an electronic medium to increase efficiency of long-term monitoring that incorporates EMA and to improve incorporation into clinical practice through mobile health (mHealth) systems.

Understanding the environmental factors present at the time of behavioral assessment is critical to clinical interpretation. For example, certain symptoms that could be indicative of behavioral problems in the absence of a significant stressor may be appropriate in the context of, say, the recent death of a close friend or relative. Assessing environmental factors, such as recent changes in substance use, may also reveal antecedents to specific behavioral symptoms that could serve as potential treatment targets to prevent behavioral problems. To this end, the BAST_β also includes an environmental context checklist to identify recently experienced stressors and supports.

Strengths and Limitations

As noted earlier, there are limitations with self-report after TBI to which the BAST_β is vulnerable, including impaired self-awareness. Unlike other measures that ask for an assessment of behavior itself, the BAST_β items focus on concrete examples of behavior and experience of emotions that are less prone to errors in insight. For example, indicating how often over the past two weeks “I got mad easily” is less prone to errors of insight than asking a person to rate the extent to which he or she has anger problems. The former is a concrete example of an experience that did or did not occur, whereas the latter requires more metacognitive awareness and overall judgment of one’s own emotional regulation ability. As noted, poor recall may also affect self-reporting, but the two-week time frame of the BAST is consistent with other symptom measures validated for use after TBI, like the PHQ9.[39,52] Further, a study of 374 adults at 6-months after moderate to severe TBI concluded that a proxy report is generally not necessary with regard to functional limitations or health-related quality of life post-TBI,[53] supporting the validity of self-reported assessments in a chronic TBI population. Another potential limitation of the BAST_β is the lack of validity test items – that is, items that one would expect everyone to endorse at the same level that would therefore be sensitive to exaggeration (e.g. “I ate something during the day”). Though less common in more severe TBI where under-reporting due to impaired self-awareness is a greater concern, over-reporting of symptoms (e.g. symptom exaggeration) is an issue in mild TBI.[54–56] Future work to validate the BAST, among those with mild TBI in particular, should include validity test items addressing these issues.

Future work

will include further development and item reduction using a parallel analysis approach with factor analysis and Rasch analysis among community-dwelling adults with complicated mild to severe TBI. Subsequent pilot testing will confirm the factor structure of the BAST in adults with complicated mild to severe TBI, evaluate the factor structure among adults and adolescents with mild TBI, and establish the convergent and discriminant validity of the BAST’s subscales. Additionally, cut-off scores will be established for identifying potentially problematic symptoms. To address the goal of developing a mobile app-based platform for EMA, future work will also identify items within each subscale that will make up the shorter, mobile health version of the BAST (BAST_mHealth) and will pilot test and evaluate the psychometric properties of the BAST_mHealth for EMA after TBI. The BAST could then serve as a long-term screening tool for behavioral and emotional symptoms, with the BAST_mHealth used for EMA when potentially concerning symptoms were identified, to further inform the need for additional clinical evaluation or intervention.

Conclusions

The BAST is a consumer-informed assessment tool designed to measure a common consequence of neurological injury (problematic behaviors) with direct links to biology (bench science) and long-term health outcomes (community). The BAST is not intended to replace comprehensive assessment, but instead to track individuals long-term and screen for likely behavioral/emotional problems, identify those in need of more comprehensive assessment, and obtain an ecologically valid overview of behavioral and emotional symptoms through community-based monitoring and EMA. The BAST will be a valuable measurement tool for incorporation into many clinical settings to inform clinical triage and as an evidence-based method for assessing intervention efficacy and effectiveness in both clinical and pragmatic trials. The flexibility of this tool may make it applicable to other clinical populations (e.g. stroke, post-traumatic stress disorder) as well. Leveraging the utility inherent in mobile technology can improve access to and delivery of healthcare, but success of mHealth is dependent on rigorous research to develop and validate every aspect of a mHealth system, from the validity of assessments and interventions delivered via mHealth modalities to the usability and consumer preferences to ensure optimal and ongoing use and satisfaction.