TBI-CareQOL Military Health Care Frustration in Caregivers of Service Members/Veterans with Traumatic Brain Injury

Abstract

Purpose:

Caregivers of service members/veterans (SMVs) encounter a number of barriers when navigating the military health care system. The purpose of this study was to develop a new measure to assess potential caregiver frustration with the systems of care and benefits in the United States Departments of Defense and Veterans Affairs.

Method:

The TBI-CareQOL Military Health Care Frustration measure was developed using data from 317 caregivers of SMVs with TBI who completed an item pool comprised of 64 questions pertaining to their anger or frustration with accessing military health care services.

Results:

Exploratory and confirmatory factor analyses supported the retention of 58 items. Constrained graded response model (GRM) overall fit and item fit analyses and differential item functioning investigations of age and education factors supported the retention of 43 items in the final measure. Expert review and GRM item calibration products were used to inform the selection of two 6-item static short forms (TBI-CareQOL Military Health Care Frustration-Self; TBI-CareQOL Military Health Care Frustration-Person with TBI) and to program the TBI-CareQOL Military Health Care Frustration computer adaptive test (CAT). Preliminary data supported the reliability (i.e., internal consistency and test-retest reliability), as well as validity (i.e., convergent, discriminant, and known-groups) of the new measure.

Conclusions:

The new TBI-CareQOL Military Health Care Frustration measure can be used to examine caregiver perceptions of and experience with the military health care system, to target improvements.

Traumatic brain injury (TBI) is common among service members injured in combat. Comorbid mental health problems, such as post-traumatic stress disorder (PTSD) and depression (Belanger, Uomoto, & Vanderploeg, 2009; Brenner, Vanderploeg, & Terrio, 2009; Lew et al., 2009) and/or bodily injuries can complicate recovery from a TBI (Belanger, Kretzmer, Yoash-Gantz, Pickett, & Tupler, 2009; French, Lange, & Brickell, 2014; Lange, Brickell, Ivins, Vanderploeg, & French, 2013; Lange et al., 2014; Storzbach et al., 2015; Vanderploeg et al., 2012). Service members and veterans (SMVs) with more severe TBI and/or bodily injuries often require extensive hospital treatment followed by long-term rehabilitation and support following discharge (Griffin et al., 2012). Family and friends frequently play an active role in the recovery and reintegration of injured SMVs, starting at bedside soon after injury and continuing after discharge or return from deployment for long-term care. In doing so, they take on the role of caregiver and assume responsibility for assisting the SMV. Caregivers play an important role in the treatment continuum and are often responsible for helping with at-home rehabilitation, performing day-to-day care and maintenance activities, providing emotional support, aiding with family and community reintegration, conducting medical and legal advocacy, navigating health care and benefits systems, and managing the family household (Friedemann-Sanchez, Sayer, & Pickett, 2008; Griffin, Friedemann-Sanchez, Hall, Phelan, & van Ryn, 2009). Caring for an SMV can be time consuming and burdensome thus negatively impacting the caregiver’s own physical and mental health, employment, finances, social participation, relationships, and family functioning (Brickell, French, Lippa, & Lange, 2018; Griffin et al., 2017; Ramchand et al., 2014; Saban et al., 2016; Van Houtven et al., 2012).

There are a range of U.S. Department of Defense (DoD), U.S. Department of Veterans Affairs (VA), and community programs available to support military caregivers (see warriorcare.dodlive.mil/; www.caregiver.va.gov/). Some of the most common services include general health care, mental health care, structured social support, religious or spiritual guidance, education and training, legal guidance, financial support, patient advocacy or case management, respite, and wellness activities (Ramchand et al., 2014). Despite the many services available to military caregivers, they report challenges in accessing these services, such as not being aware of available services, difficulty finding services, stringent eligibility requirements, difficulty scheduling appointments, a lack in consistency and quality of care provided, lack of proximity to services, challenges navigating the DoD/VA health care and benefits systems, a lack of insurance coverage, out-of-pocket expenses, difficulty getting time off work or child care, respite care not suited to the SMV, and stigma associated with getting help (Carlozzi, Lange, et al., 2018; Eaton et al., 2008; Friedemann-Sánchez, Griffin, Rettmann, Rittman, & Partin, 2008; National Alliance for Caregiving, 2010; Ramchand et al., 2014; Tanielian, Ramchand, Fisher, Sims, & Harris, 2013; Van Houtven, Sperber, & Smith, 2016).

Unmet needs among caregivers of SMVs following TBI/polytrauma can adversely impact a caregiver’s health. Stevens and colleagues (2015) examined the relationship of unmet training needs in (a) navigating the VA/DoD benefit or medical systems and (b) supporting the SMV’s emotions or feelings. About half of the studied sample reported that they did not have their training needs met in each of these areas. Caregivers with unmet training needs endorsed higher depression, anxiety, burden, and lower self-esteem compared to those who did receive training. Brickell and colleagues (2019) examined the impact of eight unmet health care and social service needs on caregiver’s health, including help with (a) caregiving duties, (b) improving care provision, (c) providing emotional support, (d) finding services, (e) navigating the VA/DoD, (f) interacting with other caregivers, and managing their own (g) medical and (h) emotional health. Six of the eight needs were individually endorsed by a majority of caregivers, and, for each need, from nearly one-third to one-half of caregivers reported unmet needs. As the number of unmet needs increased, physical health, mental health and caregiving stress all worsened.

As part of a recent effort to validate existing state-of-the-science patient-reported outcome (PRO) assessments among caregivers assisting SMVs and civilians following TBI through the Patient-Reported Outcomes Measurement Information System (PROMIS), focus group discussions were conducted with both military and civilian caregivers (Carlozzi et al., 2016; Carlozzi et al., 2015; Carlozzi, Lange, et al., 2018). Frustration with the DoD and VA health care and benefits systems was identified by caregivers as one of the most important and relevant health-related quality of life (HRQOL) domains for caregivers assisting SMVs during focus group discussions (Carlozzi et al., 2016). Military caregivers directed their frustration specifically at the perceived lack of availability, accessibility, ease of navigating, and/or lack of care coordination with these DoD/VA systems (Carlozzi, Lange, et al., 2018). The purpose of this study was to develop a military-specific PRO item bank measures of self-reported HRQOL related to caregiver frustration with the DoD and VA health care and benefits systems, to be named TBI-CareQOL Military Health Care Frustration using established PROMIS methodology (PROMIS Standards, 2019). Once the measure was developed, computer adaptive test (CAT) and static short form versions of the measure would be established as part of a larger, comprehensive measurement system (the TBI-CareQOL), which has been developed to capture important aspects of HRQOL for caregivers of persons with TBI (Carlozzi et al., Provisional Acceptance; Carlozzi, Hanks, et al., 2018; Carlozzi, Ianni, Lange, et al., 2018; Carlozzi, Ianni, Tulsky, et al., 2018; Carlozzi, Kallen, et al., 2018; Carlozzi, Kallen, Hanks, et al., 2018; Carlozzi, Kallen, Ianni, Hahn, et al., 2018; Carlozzi, Kallen, Ianni, Sander, et al., 2018; Carlozzi, Kallen, Sander, et al., 2018; Carlozzi, Lange, French, Sander, Ianni, et al., 2018; Carlozzi et al., Under Review).

Methods

Study Participants

A total of 317 caregivers participated in this study. Hospital and community-based recruitment and site-specific research registries were used to recruit caregivers of SMVs (this included 22% of the sample primarily recruited using hospital-based recruitment by the U.S. Department of Veterans Affairs in Tampa [Tampa, FL], 47% of the sample primarily recruited using community-based outreach at Walter Reed National Military Medical Center [Bethesda, MD], and 31% of the sample primarily recruited using community- and registry-based outreach at the University of Michigan [Ann Arbor, MI]). Community-based outreach efforts included social media (e.g., military, caregiver, and brain injury social media sites), participation in local and regional community-based brain injury events (e.g., military, caregiver, and brain injury events), as well as publically placed flyers for military-specific community groups (e.g., VFW, etc). We targeted all branches of the military, active duty, national guard, reservists, special forces. Caregivers were required to be caring for an individual with a medically-documented TBI diagnosis (documentation must be from a DoD or VA treatment facility), caring for an individual who was ≥ 18 years of age at the time of injury and was at least one-year post-injury, able to read and understand English, and providing physical assistance, financial assistance, or emotional support to an individual with TBI. Documentation of TBI severity was not required for eligibility. Professional (i.e., paid) caregivers were not eligible for this study. All study activities were conducted in accordance with local institutional review boards. Caregivers provided informed consent prior to their participation in the study.

Measures

TBI-CareQOL Military Health Care Frustration.

The TBI-CareQOL Military Health Care Frustration item bank was developed to assess caregivers’ possible concerns with the health services offered by the DoD/VA. All participants were provided general instructions, “This survey contains questions about your feelings related to the services you and the person you care for receive from the military, including your experience with the United States Department of Defense (DoD) and the United States Department of Veterans Affairs (VA) health care systems.

” These instructions were followed by several questions about frustration with health care services. An iterative process was used to refine the original item pool; this included feedback from expert review, item reading level assessment, translatability review, cognitive interviews, and a final consensus meeting attended by study team investigators. The iterative process for the selection of the Military Health Care Frustration item pool is described below (under Item Bank Development). The final item bank is scored on a T-score metric (mean = 50; SD = 10), with higher scores indicating more frustration with health care services. For reliability and validity analyses, we examined T-scores derived from the two 6-item short form measure versions and from the simulated CAT (scores were simulated using Firestar Version 1.3.2; Choi, 2009).

Patient-Reported Outcomes Measurement Information System (PROMIS).

The PROMIS Anger item bank was administered to assess negative mood, irritability, and frustration. Items are rated on a 5-point Likert-type scale ranging from “never” (1) to “always” (5). Scores are on a T-score metric (mean = 50; SD = 10); higher scores indicate more anger. PROMIS Anger was administered to examine the convergent validity of the new Military Health Care Frustration measure.

Caregiver Appraisal Scale (CAS).

The CAS assesses positive and negative aspects of the caregiving role (Lawton, Kleban, Moss, Rovine, & Glicksman, 1989). For the purposes of this study, we administered the 35-item version of this measure. Four separate subdomain scores (perceived burden, caregiver relationship satisfaction, caregiving ideology, and caregiving mastery; Struchen, Atchison, Roebuck, Caroselli, & Sander, 2002) were used to examine convergent and discriminant validity of the new Military Health Care Frustration measure.

Global Ratings of Frustration.

Caregivers were asked to provide global ratings of their experiences with the military health care system. Specifically, they were asked the following four questions: 1) “Please rate your anger with the services that the person under your care receives from the United States Department of Defense (DoD) health care system;” 2) “Please rate your anger with the services that the person under your care receives from the United States Department of Veterans Affairs (VA) health care system;” 3) “Please rate your anger with the services that you [the caregiver] receive from the United States Department of Defense (DoD) health care system;” and 4) “Please rate your anger with the services that you [the caregiver] receive from the United States Department of Veterans Affairs (VA) health care system.” Response options were on a 5-point Likert-type scale, with the options of “not at all” (1), “a little bit” (2), “somewhat” (3), “quite a bit” (4), and “very much” (5); participants that indicated that they either did not receive these benefits or were unsure if they received these benefits were excluded from these analyses. These ratings were used in analyses examining the convergent validity of the Military Health Care Frustration measure; the distribution of responses on these items is provided in the Supplemental Figure. In additional supportive convergent validity analyses, responses to these same four items were used to create dichotomous groups. Specifically, the two questions that referenced the caregivers’ own experiences with the military health care system (i.e., question 3 and 4) were used to create a group that reported “Frustration” with the military health care system and one that reported “No Frustration.”

Caregivers had to report “not at all” for both questions in order to be included in the “No Frustration” group, otherwise, they were included in the “Frustration” group. A similar dichotomy was created for the two questions that referenced health care for the SMV with TBI (i.e., “the person under your care”).

Data Collection

All self-report data were collected using assessmentcenter.net. Participants completed assessments on either a personal or publically-available computer with internet access, or using a study-specific research computer. A subset of participants (n=42), recruited by staff at the University of Michigan, completed a retest survey within three weeks of the initial survey. Statistical Analyses

Item Bank Development.

Item bank development was conducted according to published measurement development standards (PROMIS Standards, 2019). Literature reviews and a qualitative focus group study provided the basis for the TBI-CareQOL Military Health care Frustration item pool (Carlozzi et al., 2016; Carlozzi, Lange, et al., 2018). Specifically, nine, 90-minute focus groups (N=45 caregivers) were held to examine HRQOL of these caregivers (including the discussion of both barriers and supports that caregivers encountered when navigating the military health care system which provided the basis for the development of this new measure). Groups were moderated by one or two PhD-level clinical psychologists. Additional detail for this qualitative study are reported in Carlozzi and colleagues (2016), and Carlozzi and colleagues (2018). In constructing new items for the item pool, we relied considerably on the actual language used by caregivers which included the terms “bothered,” “frustrated,” and “angry” as used by caregivers. In addition, the observed relationship between the item stem (i.e., “bother,” “anger,” or “frustration”) was maintained; for example, a caregiver being “bothered” and what that caregiver was “bothered” about. That is, if a caregiver was “bothered” about something, the associated item would not indicate that the caregiver was instead, “angry” about that particular thing. Please note that a post-hoc examination an examination of the hierarchy of item difficulties indicated that there was no pattern of endorsement likelihood related to the stem. The item pool went through several iterations of expert review (with rehabilitation professionals with expertise in caregiving for persons with TBI and/or in psychometrics), cognitive interviews with caregivers of SMVs with TBI, reading level assessment, and translatability review (i.e., to facilitate future translations into Spanish); see Figure 1.

Figure 1.

Iterative Process for the TBI-CareQOL Military Health Care Frustration Item Pool

Next, exploratory and confirmatory factor analyses (EFA, CFA), in conjunction with clinical input, were used to identify a unidimensional set of items (Cook, Kallen, & Amtmann, 2009; McDonald, 1999; Reise, Morizot, & Hays, 2007). With regard to EFA, essential unidimensionality was supported if: 1) the ratio of eigenvalue 1 to eigenvalue 2 was > 4; and 2) the proportion of variance accounted for by eigenvalue 1 was > 0.40. Items with sparse cells (response categories with n < 10 respondents), low correlations for item-adjusted total scores (< 0.40), or that were non-monotonic were excluded from the item pool. Non-parametric IRT models examining item-rest plots and expected score by latent trait plots were used to examine monotonicity (Testgraf Software; Ramsay, Aug 1, 2000). With regard to CFA, essential unidimensionality was supported by the following fit criteria: comparative fit index (CFI) ≥ 0.90, Tucker-Lewis index (TLI) ≥ 0.90, and root mean square error of approximation (RMSEA) < 0.15 (Bentler, 1990; Hatcher, 1994; Hu & Bentler, 1999; Kline, 2005). Items with low factor loadings (lx < 0.50) and items demonstrating local dependence (residual correlation > 0.20; correlated error modification index ≥ 100) were candidates for exclusion (Cook et al., 2009; McDonald, 1999; Reise et al., 2007). EFA and CFA were conducted using Mplus (version 7.4; Muthén & Muthén, 2011).

Then, a constrained graded response model (GRM) was used to establish item parameters (Samejima, 1969). This common-slope version of the IRT model is appropriate when standard sample size requirements for GRM modeling (i.e., N = 500) are not feasible (Ruo, Choi, Baker, Grady, & Cella, 2010). Items were removed if they displayed significant misfit (S-X2 / df effect size > 3; Stark, Chernyshenko, Drasgow, & Williams, 2006). Following these overall model and item fit analyses, differential item functioning (DIF) was investigated to identify potential item bias (e.g., to identify items that might unfairly discriminate for or against individuals in a given group). Given that, in our analytic framework (i.e., constrained GRM estimation), DIF analyses can be performed provided that there are ~100 participants within each DIF factor condition (Clauser & Hambleton, 1994a), we were able to examine DIF for the factors age (≤ 40 vs. > 40 years) and education (college degree or more vs. less than college degree). Items were excluded if they exhibited impactful DIF: (a) a statistically significant (p < 0.01) group-specific item parameter difference for any DIF candidate item tested, plus (b) > 2% of DIF-corrected vs. uncorrected score differences exceeding individual case uncorrected score standard errors. DIF was examined using iterative Wald-2 testing, which involved establishing a DIF-free set of anchor items to test candidate items for DIF (Wang & Woods, 2017; Woods, Cai, & Wang, 2013). These IRT-based analyses were conducted in IRTPRO (version 3.1.2; Cai, Thissen, & du Toit, 2015). After the IRT analyses were completed, CFA was again employed to ensure essential unidimensionality of the remaining item set; using the same fit criteria outlined above. Calibration parameters from the GRM analyses were used to program the item bank to be administered as a computer adaptive test. Two 6-item short forms (SFs) were also constructed, one specific to services intended for caregivers (TBI-CareQOL Military Health Care Frustration – Self) and another specific to services provided for the person with the injury (TBI-CareQOL Military Health Care Frustration – Person with TBI). SF items were selected by a consensus process among TBI, caregiver, and measurement development experts. Items were purposefully included in the short forms to represent the full range of concept coverage while simultaneously referencing item calibration and calibration-related statistics (e.g., item slope, thresholds, average item difficulty, and item information).

Reliability and Validity Analyses.

Assessment of data skewness and kurtosis indicated that the item response data were normally distributed and appropriate for parametric analyses. Cronbach’s alpha was calculated for the Military Health Care Frustration SF scores and IRT-based internal consistency, based on mean score-level standard error (SE), was calculated for the Military Health Care Frustration simulated CAT scores to establish estimates of internal consistency reliability. Intraclass correlations were obtained to estimate test-retest or “stability” reliability; minimal acceptable internal consistency and test-retest stability specified as ≥ 0.70 (Cohen, 1988; DeVellis, 2017). We also report the percentage of participants who had the highest possible scores (ceiling effect – most frustration) and the percentage of participants who had the lowest possible scores (floor effect – least frustration) for the newly developed SFs. For simulated CAT scores, we divided the raw CAT scores by the number of items administered in order to examine floor and ceiling effects (a score of 1 was considered a “ceiling effect” and a score of “5” a “floor effect” for CATs). Acceptable floor and ceiling effects were specified as ≤ 20% (Andresen, 2000; Cramer & Howitt, 2004). In addition, we report administration times for both CAT and SF versions as evidence of measurement administration feasibility (start and stop times for each item were recorded electronically).

Convergent and discriminant validity of the Military Health Care Frustration CAT and SFs were examined using Pearson correlations. Strong correlations (r > 0.6) between Military Health Care Frustration and the Global Ratings of Frustration items and moderate correlations (r between 0.4 and 0.6) between Military Health Care Frustration and negative aspects of caregiving (i.e., CAS caregiver burden) and Military Health Care Frustration and PROMIS Anger were interpreted as being good supporting evidence for convergent validity. Weak correlations (r < 0.3) between Military Health Care Frustration and positive aspects of caregiving (i.e., CAS caregiver relationship satisfaction, caregiving ideology, and caregiving mastery) were interpreted as supporting evidence for discriminant validity (Campbell & Fiske, 1959).

Additional supportive convergent validity analyses were also conducted for Military Health Care Frustration. For these group-based analyses, independent sample t-tests were used to compare the “No Frustration” and “Frustration” groups, as defined by their responses to the Global Ratings of Frustration (the caregiver-specific group dichotomy was used for comparisons with the Military Health Care Frustration – Self 6-item SF, whereas the SMV with TBI-specific group dichotomy was used for comparisons with the Military Health Care Frustration Item Bank and Military Health Care Frustration – Person with Injury 6-item SF). Convergent validity would be supported if we saw group differences for those expressing “No Frustration” vs. “Frustration” on the Global Rating of Frustration questions, with the “Frustration” group reporting higher levels of Military Health Care Frustration. Finally, rates for elevated scores (i.e., percentage of participants whose scores were > 1 SD worse than the sample mean of 50) were examined to determine if caregivers of persons who expressed global health care frustration were at greater risk than those who expressed no global health care frustration. Finally, rates that exceeded 16% were expected for those individuals in the “Frustration” group; this would also be supportive of convergent validity (Heaton, Miller, Taylor, & Grant, 2004).

Sample Size Requirements.

Sample size for this study was determined based on recommendations for the constrained GRM analyses and Wald-2 DIF analyses utilized during the item bank development process. Specifically, existing recommendations for GRM-based analyses indicate that 200 to 1000 participants are needed to establish stable item parameters (Orlando, 2004). For a slope or threshold constrained GRM, sample sizes of a minimum N = 200 have been suggested to provide stable parameter estimation (Muraki, 1990; Ruo et al., 2010). Additionally, DIF analyses can be performed provided there are ~100 participants within each subgroup or condition (Clauser & Hambleton, 1994b).

Results

Study Participants

Three hundred and seventeen caregivers of SMVs with TBI participated in this study. Descriptive information is provided in Table 1.

Table 1.

Descriptive data for study participants

Variable	Caregivers of Military-TBI
	(N = 317)
Age (Years)
M (SD)	42.5 (11.6)
Sex (%)
Female	95.0
Male	5.0
Ethnicity (%)
Not Hispanic or Latino	90.5
Hispanic or Latino	9.5
Race (%)
White	87.7
Black/African American	3.5
Other	8.8
Education (%)
Less than High School	2.8
High School Graduate or Equivalent	5.7
Some College	35.3
College Degree	40.7
Master Degree or More	15.5
Marital Status (%)
Single, Never Married	1.3
Married/Cohabitating	90.9
Separated/Divorced	4.7
Widowed	3.2
Years in Caregiver Role
M (SD)	7.1 (3.2)
Relationship to Person with TBI (%)
Spouse	79.8
Parent	12.9
Child/Other Family Member	5.1
Other (e.g. Friend)	2.2
Age of person with TBI
M (SD)	39.7 (8.9)
Sex of Person with TBI (%)
Male	97.42
Female	2.58
Time since injury (Years)
M(SD)	9.1 (3.4)*
TBI Severity** (%)
Mild	21.5
Complicated Mild	4.4
Equivocal Mild	3.8
Moderate	19.9
Severe	14.5
Penetrating	6.9
Unknown	29.0***
Blast or Non-blast Related Injury (%)
Blast	56.2
Non-Blast	39.4
Unknown	4.4
Mechanism of Injury (%)
MVA	42.5
Falls	5.7
Struck by an object or thrown against an object	21.3
Gunshot or Assault	5.4
Other Accidents (e.g., Bicycle accident, pedestrian struck by motor vehicle)	2.5
Sports-related TBI	1.6
Other/Unknown	21.0

Note. Entries in the table represent percentage of participants unless otherwise specified;

^*Documentation of time since injury was unavailable for n=87 participants sampled through community outreach at the University of Michigan:

^**TBI severity was determined according to Department of Defense criteria

^***Although all participants had documentation of a TBI diagnosis, documentation of TBI severity was unavailable for the majority of the military sample collected through community outreach by the University of Michigan.

Item Bank Development.

Table 2 outlines the primary findings for the item bank development process. Briefly, EFA analyses supported essential unidimensionality of the item pool: the ratio of eigenvalue 1 to eigenvalue 2 was 13.4; eigenvalue 1 accounted for 62.3% of modeled variance, while eigenvalue 2 accounted for only 4.7%. Of the 64 items in the Health Care Frustration item pool, one item was eliminated due to sparse response option cells (i.e., n < 10), four items were eliminated due to high residual correlations (criterion > 0.20), and one item was eliminated due to a high correlated error modification index value (criterion ≥ 100). Subsequent IRT modeling of the 58 remaining items revealed 15 items with misfit, resulting in a total of 43 items available in the item bank. A final CFA model of these 43 items indicated good overall model fit (Table 3); fit statistics met or exceeded recommended cutoffs. Items did not exhibit DIF for any of the variables of interest investigated (i.e., age and education).

Table 2.

Unidimensional Modeling and Analyses for TBI-CareQOL Military Health Care Frustration Item Pool

	Unidimensional Modeling								Initial Item Performance		IRT Modeling
Domain	Item pool	EFA E1/E2 ratio (criterion >4)	Percent of variance for E1 (criterion >40)	1-factor CFA loading (criterion <.50)	1-factor CFA residual correlation (criterion >.20)	1-factor CFA modification index (criterion >100)	Item-adjusted total score correlations (Criterion <.40)	Sparse cells (criterion<10)	Problems with monotonicity	IRT item misfit	DIF	Interim/Final item bank
Military Health Care Frustration	64 items	13.4	62.3	0 items	4 items	1 item	0 items	1 item	0 items	15 items	0 items	43 items

Note. CFA = Confirmatory Factor Analysis; EFA = Exploratory Factor Analysis; IRT = Item Response Theory

Table 3.

Final Item Parameters for TBI-CareQOL Military Health Care Frustration Item Bank

Domain	Item Bank	CFI (criterion >.90)	TLI (criterion >.90)	CFA-based RMSEA (criterion < .15)	Alpha Reliability (criterion > .80)	IRT-based RMSEA (criterion < .15)	Response Pattern/Person Reliability (criterion > .80)
Military Health Care Frustration	43 items	.98	.98	.08	.99	.07	.97

Note. CFI = Comparative Fit Index, TLI = Tucker-Lewis Index, RMSEA: Root Mean Square Error of Approximation.

The final item bank parameters are presented in Table 4. Given that this analysis employed a constrained GRM model, a common slope value was estimated. Thresholds ranged from −0.91 to +1.13 for the Military Health Care Frustration full item set. Information was good (i.e., ≥ 10, with corresponding reliability ≥ 0.90) for scaled scores between approximately 30 and 70 (i.e., between +/− 2 SDs; see Figure 2 for the scale information function); marginal reliability was 0.97. The maximum number of items (i.e., 12) was administered by CAT at roughly −2.0 SD units and at +2.0 SD units; CAT administration tended to use the minimum number of items (i.e., 4) from the item bank from approximately −0.9 SD units to +1.1 SD units (See Figure 3). Two 6-item SFs (TBI-CareQOL Military Health Care Frustration – Self; TBI-CareQOL Military Health Care Frustration – Person with TBI) were constructed using items from the final item bank, employing calibration and calibration-based statistics (e.g., slope, item characteristic curves, item information, and average item difficulty) in conjunction with content-related clinical characteristics (e.g., items were selected that represent caregivers’ perceptions of their SMV’s care within the military health care system as well as their own caregiving experiences within this system) by a panel that included experts in TBI, caregivers of persons with TBI and measurement experts. Look-up tables to convert raw (sum) scores to T-scores are available in Appendices A (TBI-CareQOL Military Health Care Frustration – Self) and B (TBI-CareQOL Military Health Care Frustration – Person with TBI). The reliability for the Military Health Care Frustration – Self SF was examined on a measurement continuum from theta = −1.7 (T-score = 33) to +1.6 (T-score = 66). Expected score-level reliability was excellent (≥ 0.90) for thetas between −1.2 and +0.9, very good or excellent (i.e., ≥ 0.80) for thetas between −1.2 and +1.1, and good, very good, or excellent (i.e., ≥ 0.70) for thetas between −1.7 and +1.6. The reliability for the Military Health Care Frustration – Person with TBI SF was examined on a measurement continuum from theta = −1.6 (T-score = 34) to +1.7 (T-score = 67). Expected score-level reliability was excellent (≥ 0.90) for thetas between −0.9 and +1.2, very good or excellent (i.e., ≥ 0.80) for thetas between −1.1 and +1.2, and good, very good, or excellent (i.e., ≥ 0.70) for thetas between −1.6 and +1.7.

Table 4.

Item Parameters for the TBI-CareQOL Military Health Care Frustration Item Bank

Item	Slope	Threshold 1	Threshold 2	Threshold 3	Threshold 4
I am bothered by the inaccurate information my service member receives from his/her treatment team.	3.33	−0.43	0.09	0.49	1.03
I am angry that the military does not tell me about all the services that I am eligible for.	3.33	−0.75	−0.31	0.11	0.62
I am angry with how getting services for my service member has disrupted my family life. I am bothered with how long it takes for the	3.33	−0.51	−0.08	0.38	0.78
military to provide services for my service member.	3.33	−0.79	−0.24	0.23	0.78
I am bothered by how difficult it is to request medical documents for my service member.	3.33	−0.47	−0.13	0.30	0.67
I am bothered by the fact that my service member is treated by different medical teams.	3.33	−0.29	0.05	0.55	0.95
I am frustrated with the lack of communication with my service member's health care team.	3.33	−0.53	0.00	0.34	0.84
I am angry about the lack of social services for service members provided by the military.	3.33	−0.56	−0.07	0.31	0.84
I am bothered by how much time I have to spend getting services for my service member.	3.33	−0.66	−0.15	0.26	0.80
I am frustrated by the lack of communication among my service member's treatment team.	3.33	−0.63	−0.10	0.34	0.86
I am frustrated with how long it takes for the military to provide services for caregivers.	3.33	−0.82	−0.34	0.12	0.55
I am frustrated by the lack of concern the military has for my service member's needs.	3.33	−0.76	−0.29	0.12	0.63
I am bothered by the way the staff at the military hospitals treat my service member.	3.33	−0.29	0.12	0.58	1.10
I am angry about the lack of social services for caregivers provided by the military.	3.33	−0.82	−0.26	0.10	0.67
I am angry that I have to find information about services for my service member on my own.	3.33	−0.90	−0.35	0.11	0.54
I am bothered by the lack of comprehensive care my service member receives.	3.33	−0.72	−0.15	0.27	0.69
I am bothered that there are frequent changes to my service member's treatment team.	3.33	−0.65	−0.27	0.18	0.57
I am bothered with how long my service member has to wait to obtain services.	3.33	−0.91	−0.31	0.08	0.52
I am angry with the way my service member is treated by his/her treatment team.	3.33	−0.24	0.22	0.73	1.08
I am frustrated by how long it took my service member to receive a diagnosis.	3.33	−0.51	−0.08	0.15	0.55
I am frustrated with barriers to accessing services for my service member.	3.33	−0.72	−0.22	0.26	0.63
I am angry that my service member's medical team ignores what I have to say.	3.33	−0.21	0.21	0.61	0.96
I am frustrated with the barriers my service member has to access services.	3.33	−0.66	−0.17	0.32	0.82
I am bothered by the amount of time I have to wait to obtain services for myself.	3.33	−0.37	0.03	0.55	0.86
I am frustrated with barriers to accessing services for caregivers.	3.33	−0.65	−0.15	0.27	0.63
I am frustrated with how difficult it is to access care for my service member.	3.33	−0.60	−0.11	0.29	0.86
I am frustrated about the amount of caregiver services provided by the military.	3.33	−0.88	−0.32	0.20	0.63
I am bothered by the lack of individualized care my service member receives.	3.33	−0.61	−0.02	0.43	0.91
I am frustrated when I have to fight to get services I need as a caregiver.	3.33	−0.79	−0.23	0.05	0.34
I am angry that I have to find information about caregiver services on my own.	3.33	−0.63	−0.21	0.23	0.59
I am angry that the military does not address my needs as a caregiver.	3.33	−0.72	−0.22	0.21	0.63
I am frustrated with the lack of services available for my service member.	3.33	−0.62	−0.13	0.46	0.91
I am frustrated with obstacles to getting services for my service member.	3.33	−0.73	−0.11	0.40	0.86
I am angry with the military due to the lack of support for caregivers.	3.33	−0.71	−0.19	0.20	0.68
I am angry about the lack of care coordination for my service member.	3.33	−0.64	−0.07	0.38	0.76
I am bothered by how long it takes to receive information from the military.	3.33	−0.76	−0.28	0.23	0.62
I am angry that I have to fight to get services for my service member.	3.33	−0.65	−0.12	0.26	0.66
I am bothered by the way I am treated by employees of the military.	3.33	−0.09	0.35	0.79	1.13
I am angry with the medical care my service member receives.	3.33	−0.32	0.14	0.73	1.12
I am bothered by the lack of services for caregivers.	3.33	−0.90	−0.25	0.19	0.56
I am frustrated by the inaccurate information I receive.	3.33	−0.57	−0.09	0.34	0.70
I am bothered by the inadequacy of the military.	3.33	−0.54	0.11	0.53	0.90
I am frustrated with the military.	3.33	−0.29	0.19	0.70	1.00

Note. Items in boldfacewere selected for the Military Health Care Frustration – Self 6-item short form

Underlined items were selected for the Military Health Care Frustration – Person with TBI 6-item short form. Slopes are the discrimination parameters (which are held constant in a Rasch model). Thresholds are the location (or difficulty parameters); they indicate the locations on the measurement continuum where an item can provide its most precise measurement. Thus, items with lower value thresholds measure most precisely at those lower score values (the item can therefore be thought of as an easier item). Items with higher threshold values measure most precisely at those higher score values (the item can therefore be thought of as a harder item).

Figure 2.

TBI-CareQOL Military Health Care Frustration Test Information Plot

Caption: In general, we want total information per score-level to be ≥ 10.0 and the resultant standard error to be ≤ 0.32 (which provides a score-level reliability of ≥ 0.9). This figure shows excellent total information and standard errors for Military Health Care Frustration scale scores between approximately 30 and 70.

Figure 3.

TBI-CareQOL Military Health Care Frustration Number of CAT Items by CAT Theta

Caption: This figure shows the number of CAT items used for different scale score levels in standard deviation units: at approximately ≤ −2.0 SD units and ≥ +2.0 SD units the maximum of 12 items from the item bank were used by the CAT; from approximately −0.9 to +1.1 SD units the CAT tended to use the minimum of four items from the item bank.

Reliability and Validity Analyses.

Internal consistency reliability was excellent for both the CAT and two SF administrations (Table 5). In addition, 3-week test-retest “stability” reliability ranged from very good to excellent across the CAT and SF administrations (Table 5). All administration formats were free of floor and ceiling effects, and administration time was brief (all forms took less than one minute on average for administration; Table 5).

Table 5.

Descriptive Data for the different TBI-CareQOL Military Health Care Frustration administration formats

	N	Internal consistency reliability	Test-retest stability	Mean (SD)	% at Floor	% at Ceiling	Administration Time (sec)	Average administration time per item (sec)
Military Health Care Frustration – CAT	317	0.92 *	0.88	50.5 (10.4)	10.2	3.4	40.9	7.7
Military Health Care Frustration – Caregiver Short Form	294	0.96 **	0.89	50.5 (10.4)	11.2	17.4	40.0	6.7
Military Health Care Frustration – Person with TBI Short Form	307	0.95 **	0.94	50.1 (9.9)	12.4	8.1	46.2	7.7

Note: CAT = Computer Adaptive Test; TBI – Traumatic Brain Injury

^*Mean standard error-based reliability

^**Cronbach’s alpha reliability

Correlations supported convergent and discriminant validity (Table 6). As hypothesized, there were strong correlations between Military Health Care Frustration and global ratings of anger with health care services. There were moderate correlations between Military Health Care Frustration and measures of burden and anger; and there were weak correlations between Military Health Care Frustration and positive aspects of caregiving.

Table 6.

Convergent and Discriminant Validity for the TBI-CareQOL Military Health Care Frustration Item Bank

	Convergent Validity (strong relationships)				Convergent Validity (moderate relationships)		Discriminant Validity
TBI-CareQOL Military Health Care Frustration Administration Format	Please rate your anger with the services that the person you care for receives from the United States Department of Defense (DoD) health care system.	Please rate your anger with the services that the person you care for receives from the United States Department of Veterans Affairs (VA) health care system.	Please rate your anger with the services that you receive from the United States Department of Veterans Affairs (VA) health care system.	Please rate your anger with the services that you receive from the United States Department of Defense (DoD) health care system.	CAS Burden	PROMIS Anger	CAS Satisfaction	CAS Ideology	CAS Mastery
CAT	0.67	0.6	0.57	0.66	−0.47	0.49	−0.22	0.03	−0.32
6-item SF - Self	0.74	0.77	0.68	0.74	−0.6	0.53	−0.26	0.1	−0.4
6-item SF - Person w/ TBI	0.78	0.73	0.65	0.84	−0.56	0.54	−0.28	0.02	−0.36
Mean (SD)	2.4 (1.4)	2.8 (1.6)	2.6 (1.3)	3.0 (1.5)	39.5 (12.5)	53.9 (10.0)	43.8 (6.0)	14.7 (3.6)	13.6 (2.8)

Convergent validity was also supported by our group-based global frustration analyses (Table 7). Caregivers who endorsed global frustration with the military health care system had higher Military Health Care Frustration scores than those who reported no global frustration. Additionally, participants who reported global frustrations with health care were at greater risk of having elevated (+1 SD) Military Health Care Frustration scores relative to those who did not.

Table 7.

Additional Convergent Validity Support for TBI-CareQOL Military Health Care Frustration Item Bank

Military Health Care Frustration Sample and Administration Format	Global Rating of No Frustration			Global Rating of Frustration
	n	Mean (SD)	% Impaireda	n	Mean (SD)	% Impaireda	t	p
Military Health Care Frustration CAT	67	42.4 (8.0)	0.0**	193	52.8 (8.3)	21.8**	8.95	<.0001
Military Health Care Frustration - Self Short Form	45	39.9 (6.3)	0.0**	239	52.8 (9.7)	25.5**	11.49	<.0001
Military Health Care Frustration - Person with TBI Short Form	67	39.8 (7.1)	1.5**	190	53.7 (8.1)	23.2**	12.46	<.0001

Note.

^a = Military Health Care Frustration score ≥ 60

^**p<.0001

Discussion

This report describes the development of a new PRO designed to capture possible caregiver-reported concerns with the military health care system. To the authors’ knowledge, this is the first measure of its kind. This new measure, the TBI-CareQOL Military Health Care Frustration item bank, includes 43 items that can be administered to caregivers as a long form (i.e., all 43 items), as a CAT, or as a SF. There are two static SFs: a 6-item SF that is focused on frustration with services for the caregiver and a 6-item SF focused on frustration with services for the SMV. The final items included in the bank are unidimensional and devoid of bias for age or education. Preliminary examinations of measurement reliability across the CAT and SF administrations all indicated excellent score-level reliability, excellent internal consistency reliability, and very good to excellent test-retest stability. In addition, both CAT and SF administrations were devoid of floor and ceiling effects; administration time was also brief (CAT and SF administrations took less than one minute each for completion).

Our preliminary analyses also indicated support for both convergent and discriminant validity; specifically, all formats of the new measure had the strongest correlations with caregiver global ratings for health care frustration. In addition, there were moderate relationships with self-reported anger and caregiver burden, and little to no relationship with measures of positive aspects of caregiving. Known-groups validity was also supported. In all cases, persons with global endorsements of frustration with the health care system had significantly higher scores on TBI-CareQOL Military Health Care Frustration (regardless of administration format). In addition, those reporting frustration also demonstrated increased risk for frustration with the military health care system, providing further evidence for validity.

High scores on TBI-CareQOL Military Frustration can be used to identify caregivers that are in distress (as evident by high levels of frustration) and may be at high risk for negative outcomes. While scores between 40 and 60 indicate a level of frustration with the military health care system that is similar to other caregivers, scores ≥60 indicate a level of frustration that may warrant additional consideration (this score is worse than 84% of the broader caregiver population), whereas scores ≥70 indicate a level of frustration that warrants additional follow-up (this score is worse than 95.45% of the broader caregiver population). While this measure does not provide an objective assessment of health care services, high scores may reflect unmet service needs, barriers to accessing available services, or a level of caregiver-specific distress, a lack of confidence in caregiving responsibilities, and /or inability to cope with negative feelings. As such, clinicians and researchers can use scores on this measure in conjunction with other measures and/or clinical follow-up to identify potential unmet needs and/or relevant resources/supports for both the caregivers and the SMV. The successful identification of caregivers that are in distress, is the first step in the process of improving outcomes for these individuals. This information, in conjunction with a more detailed needs assessment, can help link caregivers and their families to available services and/or support, which is especially important given that caregiver well-being is directly related to the well-being and successful rehabilitation of the person with TBI (Anderson et al., 2001; Holland & Schmidt, 2015; Kreutzer, Marwitz, Godwin, & Arango-Lasprilla, 2010; Ramkumar & Elliott, 2010; Sander et al., 2002; Sander, Maestas, Sherer, Malec, & Nakase-Richardson, 2012; Schonberger, Ponsford, Olver, & Ponsford, 2010; Smith & Schwirian, 1998; Temple, Struchen, & Pappadis, 2016; Vangel, Rapport, & Hanks, 2011; Verhaeghe, Defloor, & Grypdonck, 2005).

While these findings highlight a rigorous development approach that capitalizes on both classical test theory and item response theory (Cella, Gershon, Lai, & Choi, 2007; Cella et al., 2010) and provide preliminary support for this measure displaying strong psychometric properties, there are also several study limitations that should be acknowledged. First, caregivers in this study were primarily women and spouses of the person with TBI. Therefore, this precluded our ability to examine item bias (i.e., DIF) for these important demographic variables. Future work is needed to determine if these findings are generalizable to male caregivers and other family members. Furthermore, data for TBI severity was unavailable for ~30% of the sample, preventing the ability to examine findings by injury severity, as well as an examination of DIF for this important variable. As SVM with different types and severity of injuries and illness have different pathways of care in the military healthcare system, future work will need to examine how findings differ based on types and quantity of services utilized. Finally, we did not collect detailed information about the location(s) where SMVs and their families received services. While the majority of our recruitment occurred using community-based outreach, two of our data collection sites (Department of VA in Tampa and Walter Reed National Military Medical Center) are better resourced than many other VA and DoD facilities. Thus, it is possible that the reported levels of frustration might underestimate generic feelings of frustration at treatment centers that have fewer resources. Future work is needed to better understand the impact that a specific facility (or facility type) might have on health care frustrations in this population.

Regardless, this is the first PRO designed to focus solely on caregiver perceptions of the military health care system (DoD and VA). The military and VA health care systems, through surveys and other mechanisms, actively solicit feedback from consumers of their services. Indeed, the National Defense Authorization Act of 1992 requires an annual survey to assess satisfaction. While surveys of that type can detect broad trends, there is no formal way of assessing perceptions of shortcomings in service on an individual level. This type of measure is well positioned for use in identifying caregivers and SMVs that may warrant additional services. Ultimately, it is our hope that this measure can be used to aid in the evaluation of important efforts within the military heath care system focused on targeting unmet family and spousal needs among caregivers (Eaton et al., 2008; Gorman, Blow, Ames, & Reed, 2011; Lewy, Oliver, & McFarland, 2014; Misra-Hebert et al., 2015; Verdeli et al., 2011) and efforts focused on providing education about the military health care service system to SMVs and their families (Eaton et al., 2008; Gorman et al., 2011; Lewy et al., 2014). Finally, it is important to note that caregivers’ frustration with services is not necessarily indicative of the quality of services offered by the DOD and/or VA. Caregivers may be frustrated for a variety of reasons, including lack of awareness of existing services, lack of understanding of how to access services, or their coping styles and/or predispositions to perceive things negatively. In spite of this, understanding caregivers’ perceptions may help military health care professionals in targeting services and increasing their accessibility- real and perceived.

Impact and Implications.

• There is a need for a measure of caregiver-reported concerns with navigating the military health care system. Given that caregivers are an important part of the recovery process for persons with TBI, it is important to evaluate their perception of services that their loved ones may or may not receive.

• A new measure, the TBI-CareQOL Military Health Care Frustration item bank, was developed for this purpose. It includes a long form (43 items), two 6-item short forms (a version focused on services for the caregiver and a version focused on services for the person with TBI in the military), and a computer-adaptive test (CAT).

• This measure can assist the military health care system to focus on targeting unmet needs of the caregivers of those with TBI.

Appendix A.

TBI-CareQOL Military Health Care Frustration - Self 6-item Short Form Conversion Table

Raw Score	T-score	SE *
6	33.00	4.90
7	37.91	3.15
8	39.91	2.86
9	41.42	2.66
10	42.64	2.55
11	43.76	2.42
12	44.76	2.34
13	45.68	2.29
14	46.55	2.26
15	47.38	2.24
16	48.19	2.22
17	48.98	2.21
18	49.75	2.20
19	50.52	2.20
20	51.30	2.21
21	52.08	2.22
22	52.88	2.24
23	53.72	2.28
24	54.60	2.33
25	55.55	2.41
26	56.63	2.58
27	57.83	2.76
28	59.23	2.97
29	60.90	3.17
30	65.95	5.03

^*SE = Standard error

Appendix B.

TBI-CareQOL Military Health Care Frustration - Person with TBI 6-item Short Form Conversion Table

Raw Score	T-score	SE *
6	33.55	4.98
7	38.58	3.18
8	40.64	2.87
9	42.16	2.69
10	43.43	2.55
11	44.57	2.42
12	45.59	2.34
13	46.54	2.29
14	47.43	2.26
15	48.28	2.24
16	49.12	2.22
17	49.93	2.22
18	50.74	2.21
19	51.55	2.22
20	52.36	2.22
21	53.19	2.24
22	54.03	2.26
23	54.91	2.29
24	55.84	2.34
25	56.84	2.41
26	57.95	2.54
27	59.20	2.69
28	60.66	2.88
29	62.56	3.14
30	67.38	4.85

^*SE = Standard error