Understanding Health-related Quality of Life in Caregivers of Civilians and Service Members/Veterans with Traumatic Brain Injury: Establishing the Reliability and Validity of PROMIS Mental Health Measures

Abstract

Objective:

To provide important reliability and validity data to support the use of the PROMIS Mental Health measures in caregivers of civilians or service members/veterans with traumatic brain injury (TBI).

Design:

Patient-reported outcomes surveys administered through an electronic data collection platform.

Setting:

Three TBI Model Systems rehabilitation hospitals, an academic medical center, and a military medical treatment facility.

Participants:

560 caregivers of individuals with a documented TBI (344 civilians and 216 military)

Intervention:

Not Applicable

Main Outcome Measures:

PROMIS Anxiety, Depression, and Anger Item Banks

Results:

Internal consistency for all of the PROMIS Mental Health item banks was very good (all α > .86) and three-week test retest reliability was good to adequate (ranged from .65 to .85). Convergent validity and discriminant validity of the PROMIS measures was also supported. Caregivers of individuals that were low functioning had worse emotional HRQOL (as measured by the three PROMIS measures) than caregivers of high functioning individuals, supporting known groups validity. Finally, levels of distress, as measured by the PROMIS measures, were elevated for those caring for low-functioning individuals in both samples (rates ranged from 26.2% to 43.6% for caregivers of low-functioning individuals).

Conclusions:

Results support the reliability and validity of the PROMIS Anxiety, Depression, and Anger item banks in caregivers of civilians and service members/veterans with TBI. Ultimately, these measures can be used to provide a standardized assessment of HRQOL as it relates to mental health in these caregivers.

Moderate to severe traumatic brain injury (TBI) is often associated with functional declines^1,2 that may require family members to assume a caregiver role.^3–7 Caring for an individual with a TBI can profoundly impact well-being and is often associated with increased stress and declines in caregiver health-related quality of life (HRQOL),^8–15 a multidimensional construct encompassing mental, physical and social well-being.¹⁶ With regard to mental well-being, caregivers of individuals with TBI often report elevated levels of anxiety and depression,^12,17–26 as well as anger.^27–31 Depression, anxiety, and anger rates among these caregivers are higher than the general population.^32–40 The caregiver’s emotional distress can also have a negative effect on the individual with the TBI including poorer post-rehabilitation outcomes^8–15 and social integration.^41–45

Research suggests that caregiver anxiety tends to be high immediately post-injury,¹¹ but appears to lessen within the first 3 months¹⁷ and remains relatively stable thereafter for the first year post-injury.^20,21,26 Evidence also indicates that anxiety is associated with symptoms related to the person with the TBI (personality changes, behavioral outbursts, and cognitive problems),^12,21,25 as well as the caregiver’s perceptions of discrimination and stigma.⁴⁶

While caregiver depression may decrease during the first 6-months post-TBI,¹⁷ significant levels of depression persist long-term.^18–21 As with caregiver anxiety, evidence indicates that caregiver depression is related to symptoms following TBI^{12,18,21–25} at both 6- and 12-months post-TBI, with a stronger relationship at 12-months.^18,26 Caregiver depression is also associated with perceived caregiver discrimination and stigma.⁴⁶ Caregivers with a history of depression are at greater risk for experiencing depressive symptoms post TBI.¹⁹ However, caregiver depression is not related to TBI severity,^18,19 longer-term functional ability of the individual with the TBI¹¹ (although it is related to more immediate functional ability post-TBI^11,12), time since injury,¹⁹ employment status of the individual with the TBI,¹⁸ nor relationship (e.g., parent or spouse).^18–20 Finally, depressive symptoms are associated with anger in caregivers of individuals with TBI³¹ and some caregivers may be more likely to report distress as anger rather than depression or anxiety.³⁰

Because mental health complaints are common in caregivers of individuals with TBI, there is a need for clinical interventions to address these concerns,^47,48 and evidence suggests that clinical interventions may improve these symptoms (especially depression and anxiety).^10,49 Unfortunately, the lack of comprehensive and well-validated assessment tools limits our ability to measure these factors. The existing literature typically focuses on broad concepts of caregiver burden^50,51 or distress,^52,53 rather than on anxiety, depression, or anger specifically. For studies that focus on anxiety, depression or anger, there is no consensus assessment, and the measures that have been used⁵⁴ are typically borrowed from other clinical populations without data to support their psychometric properties among caregivers. Thus, there is a need for well-validated measures that have the ability to capture changes in caregivers’ mental health status.

The Patient Reported Outcomes Measurement Information System (PROMIS)^55,56 was designed to assess symptoms and associated HRQOL. PROMIS includes item banks that measure key health symptoms/concepts for the general population and several chronic conditions. While PROMIS item banks are available to evaluate emotional HRQOL (depression, anxiety, and anger), these measures do not currently have psychometric data to support their reliability and validity in caregivers of individuals with any illness or injury, much less in caregivers of individuals with TBI. Thus, the purpose of this study was to establish the reliability and validity of the PROMIS Anxiety, Depression and Anger Item banks in caregivers of civilians and service member/veterans (SMVs) with TBI.

Methods

Participants

A total of 560 caregivers of individuals with a documented TBI (344 civilians and 216 SMVs) participated in this study. Data collection for the civilian sample included Kessler Foundation, Rehabilitation Institute of Michigan, TIRR Memorial Hermann, and University of Michigan (UM); data collection for the SMV sample included Walter Reed National Military Medical Center (WRNMMC) and UM. Civilian-based recruitment leveraged TBI Model System centers, TBI caregiver databases, medical record data capture systems⁵⁷ and community outreach efforts. SMV-based recruitment involved hospital-based and community outreach efforts at WRNMMC and community-based outreach at UM. A total of 145 caregivers (56 civilians and 89 SMVs) completed a retest approximately three weeks after the initial study visit; the retest only involved participants from UM (72.5% of UM participants completed the retest).

Caregivers of individuals with TBI were ≥18 years old, able to read and understand English, and caring for an individual with a medically documented TBI sustained at ≥16 years old. The person with the TBI also had to be ≥1-year post-injury. For caregivers of civilians with TBI, medical record confirmation specified that the individual with TBI meet TBI Model System inclusion criteria for a complicated mild, moderate or severe TBI.⁵⁸ For caregivers of SMVs with TBI, medical record confirmation specified that the individual with the TBI had to have a Department of Defense or Veteran Affairs diagnosis of TBI. For both caregivers of civilians and SMVs, a caregiver was defined as someone that provides physical assistance, financial assistance, or emotional support.

The study was approved by each site’s institutional review board, and participants provided consent prior to participation.

Measures

Participants completed several patient-reported outcome measures including generic measures of emotional HRQOL (PROMIS Anxiety, Depression and Anger), a measure of overall HRQOL (RAND-12), measures of caregiver burden (Caregiver Appraisal Scale [CAS] and Zarit Burden Inventory [ZBI]), and a measure of the functional status of the person with TBI (Mayo-Portland Adaptability Inventory-Fourth Edition [MPAI-4]).

The PROMIS Anxiety, Depression, and Anger⁵⁹ measures were administered as a computer adaptive test (CAT) and a short form (SF). A CAT is a “smart” test, wherein each item selected is based on the participant’s previous response. A SF is a static set of items designed to be comparable to legacy measures of similar constructs. These measures are scored on a T-metric (i.e., M= 50, SD=10) where higher scores represent worse mental health.

The RAND-12 Health Status Inventory⁶⁰ is a 12-item measure that provides a component score for physical (Physical Health Composite, -PHC) and mental health (Mental Health Composite, -MHC). PHC and MHC scores range from 0 to 100 with higher scores indicating better health.

The CAS⁶¹ is a 47-item measure designed to assess perceived burden, caregiving relationship satisfaction, caregiving ideology, and caregiving mastery. Scores were calculated according to Struchen and colleagues,⁵¹ with higher scores on each subscale indicating better functioning. The ZBI⁶² is a 22-item measure designed to assess perceived burden. Higher scores indicate greater levels of burden.

The MPAI-4⁶³ is a 35-item measure designed to assess caregiver perceptions of post-brain injury functional status of the individual with the TBI. This measure is scored on a T-metric (i.e., M= 50, SD=10) where higher scores indicate lower functioning.

Data Analysis

Consideration for sample size requirements were based on the sampling needs of the broader study; see Carlozzi and colleagues, this issue.⁶⁴ The data were assessed for normality (i.e., skewness and kurtosis).⁶⁵ Results suggested that the data were normally distributed.

Reliability.

Cronbach’s alphas were used to evaluate internal consistency reliability for PROMIS SFs, whereas IRT-based reliability (i.e., marginal reliability) was used to evaluate the internal consistency reliability for PROMIS CATs. Three-week test-retest reliability was examined to evaluate temporal reliability. Minimal acceptable reliability was specified as ≥0.70.^66,67

Floor and Ceiling Effects.

Floor and ceiling effects were defined as follows: floor effects = the percentage of participants who had the lowest possible score on a given scale/subscale as measured by negative functioning (i.e., high levels of anger, depression, or anxiety); ceiling effects = the percentage of participants who had the highest possible score on a given scale/subscale as measured by positive functioning (i.e., low levels of anger, depression, or anxiety).

Acceptable floor and ceiling rates were specified as ≤ 20%.^68,69

Administration time.

The online data capture system recorded start and finish times for each item.

Convergent and Discriminant Validity.

Convergent and discriminant validity was established by examining Pearson correlations between similar and dissimilar constructs.⁷⁰ Convergent validity was supported by strong correlations among mental health measures constructs (r>0.6),⁷¹ whereas discriminant validity was supported by moderate correlations between mental health measures and measures of burden (r’s between 0.4 and 0.6), and weak relationships between mental health measures and positive aspects of caregiving (r <0.3).⁷¹

Known-groups validity.

Known-groups validity was examined by comparing caregivers of individuals that were considered “high functioning” (MPAI-4 scores <60) and caregivers of individuals that were considered “low functioning” (MPAI-4 scores ≥60) on the three PROMIS measures using independent sample t tests; cutoffs for the different groups were based on functional categories provided within the MPAI-4 manual.⁷² Caregivers of high-functioning individuals should report better emotional HRQOL (i.e., less depression, anxiety and anger) relative to those caring for low-functioning individuals.

Impairment Rates.

Clinical impairment rates for caregivers were established relative to published normative data (i.e., participants whose scores were > 1 SD worse than the PROMIS normative sample mean [n = 2208; M = 50, SD = 10]⁷³). Impairment rates that exceed 16% (the proportion expected based on the normal curve⁷⁴) were taken to suggest increased risk of problems with emotional HRQOL.

Missing Data.

We imputed scores using expectation maximization⁷⁵ for participants that were missing less than 10% of items on any measure. This resulted in data imputation for 39 participants on the MPAI-4 and 15 participants on the CAS. Two participants were excluded due to missing >10% of data.

Results

Demographic and injury-related characteristics for the civilian and military samples are reported elsewhere by Carlozzi and colleagues in this issue.⁶⁴ On average, caregivers of SMVs with TBI were younger than caregivers of civilians with TBI (M=37.2 vs. M=51.6 years of age; t(555.94)=15.04, p<.01). Caregivers of SMVs were also more likely to be women compared to caregivers of civilians (98.1% vs. 78%; χ2(1)=44.74, p<.001). Caregivers of civilians were more likely than caregivers of SMVs to be African American (20.3% vs. 3.3%; χ2(1)=31.22, p<.001). Caregivers of SMVs were also more educated (χ2(2)=11.10, p<.01), more likely to be married (χ2(1)=37.74, p<.001), and less likely to be parents or other family members (χ2(4) = 174.8, p < .001) than caregivers of civilians. Finally, caregivers of SMVs were caring for individuals that were younger than the civilian care recipients (M=37.1 vs M=42.3 years of age; t(535.41)=5.56, p<.01). The groups did not differ in time providing care (t(513.244)=.82, p=.41).

With regard to injury severity, 56.1%, 20.9%, and 17.7% of caregivers of civilians were caring for a complicated mild injury, moderate, or severe TBI, respectively. For caregivers of SMVs with TBI, 17.3% were caring for someone with an uncomplicated mild, 1.9% complicated mild, 1.4% moderate, 1.4% severe, 14.9% ‘equivocal’ mild, 1.4% unknown, 1.4% penetrating TBI, and 60.3% unknown. TBI severity data were unavailable for the SMV participants recruited by UM (n=129); since these individuals were recruited from the community, it can be reasonably assumed that most of this sample (i.e., >80%) falls in the mild TBI classification (according to existing prevalence rates of TBI severity in the military⁷⁶). For SMVs, an overwhelming majority of TBIs were sustained during military involvement (<2% were sustained outside of military duty).

The retest sample differed from the baseline sample in many ways. Retest participants were significantly younger (M=42.2 vs M=46.1 years of age; t(704)=3.07, p=.002) and had higher educational attainment (51.9% vs 39.5% college graduate; χ2(2)=15.65, p<.001). Also, the average age of the person with TBI was younger for retest participants than for the entire sample ((M=37.4 vs. M=40.3; t(700)=2.52, p=.01). Finally, retest participants cared for persons who were more impaired (as per MPAI-4 ratings) than those cared for by the entire sample (MPAI-4 M=54.14 vs 51.30; t(690)=2.36, p=.02).

Internal Consistency Reliability.

For caregivers of civilians and SMVs with TBI, all of the PROMIS emotion measures exceeded a priori standards for internal consistency (≥ 0.70; Table 1). In addition, most of the PROMIS emotion measures met the a priori criterion for test-retest reliability (≥ 0.70) for the civilian, but not the SMV caregiver groups, though the military test-retest reliabilities for the PROMIS measures were very close to 0.70 (0.65 – 0.70).

Table 1.

Descriptive Information and Reliability Data for Self-Report Measures

PROMIS Mental Health Measures	N	Internal Consistency	3 Week Test-Retest Reliability	% of the sample with floor effects	% of the sample with ceiling effects	Measure Mean (SD)	Administration Time (sec)	Mean # of Items per Measure	Average Administration Time per Item
Civilian sample
Anger CAT	335	.91	.76*	0.0	4.2	51.24 (9.80)	44.9	7.1	6.3
Anger SF	336	.86	.77*	0.0	7.1	51.30(9.01)	34.8	5	7.0
Anxiety CAT	335	.92	.68*	0.0	5.1	52.43(9.12)	22.2	5.3	4.2
Anxiety SF	337	.86	.76*	0.0	24.6	52.46 (8.78)	18.3	4	4.6
Depression CAT	335	.92	.79*	0.0	12.7	50.21 (9.57)	21.7	6.1	3.6
Depression SF	337	.89	.85*	0.6	35.6	50.83 (8.67)	14.0	4	3.5
Military sample
Anger CAT	208	.92	.68*	0.8	0.0	56.71 (9.16)	24.8	6.2	4.0
Anger SF	208	.89	.69*	0.5	0.5	57.06 (8.68)	20.8	5	4.2
Anxiety CAT	208	.93	.66*	1.6	0.0	58.61 (8.93)	14.0	4.4	3.2
Anxiety SF	208	.89	.65*	1.4	10.6	57.34 (9.43)	15.1	4	3.8
Depression CAT	208	.94	.69*	0.8	0.0	54.96 (9.46)	14.2	4.8	3.0
Depression SF	208	.94	.70*	3.4	24.5	54.46 (10.20)	10.9	4	2.7

Note.

^*= higher scores = better functioning; Cronbach’ s alphas are reported for all measures except the computer adaptive tests which are reported as marginal (i.e., item response theory based) reliabilities. PROMIS = Patient Reported Outcomes Measurement Information System; CAT = Computer Adaptive Test; SF = Short-Form; CAS = Caregiver Appraisal Scale; MPAI = Mayo-Portland Adaptability Inventory; ZBI = Zarit Burden Inventory.

Floor and Ceiling Effects (Table 1).

For the SF administration, all PROMIS measures were free of floor effects in both samples. However, the Anxiety SF had elevated ceiling effects in the civilian sample (24.6%), and the Depression SF had elevated ceiling effects in both samples (civilian=35.6%, SMV=24.5%). The CAT administration consistently performed better than the SF for both samples, with much smaller ceiling effects.

Administration Times.

Average administration times were all less than one minute for each PROMIS measure (Table 1).

Convergent and Discriminant Validity (Table 2).

Table 2.

Convergent and discriminant validity of the PROMIS Mental Health CATs

	Convergent validity				Discriminant validity
PROMIS MENTAL HEALTH MEASURES (CATs)	Anger	Anxiety	Depression	RAND-12 Mental Health	RAND-12 Physica Health	Caregiver Appraisal Scale				Zarit Burden Interview
						Burden	Satisfaction	Ideology	Mastery
Civilian sample
Anger	--	--	--	−.67**	−.17**	−.54**	−.29**	−.08	−.26**	.58**
Anxiety	.78**	--	--	−.68**	−.19**	−.56**	−.23**	−.03	−.32**	.58**
Depression	.76**	.81**	--	−.73**	−.13*	−.55**	−.28**	−.08	−.29**	.59**
Military sample
Anger	--	--	--	−.64**	−0.04	−.46**	−. 42**	.06	−.38**	.62**
Anxiety	.74**	--	--	−.66**	−0.10	−.56**	−.36**	.12	−.37**	.73**
Depression	.79**	.83**	--	−.73**	−0.07	−.55**	−.44**	.07	−.41**	.74**

Note. PROMIS = Patient Reported Outcome Measurement Information System; CAT = Computer Adaptive Test

^*p < .05,

^**p < .01

With regard to convergent validity, large correlations among the three PROMIS Mental Health measures were found. Convergent validity was also supported by large correlations between the three PROMIS measures and Rand-12 MHC. Discriminant validity was supported by negligible to small correlations among the three PROMIS measures and Rand-12 PHC. Correlations between the three PROMIS measures and the two measures of caregiver burden (CAS and ZBI) were moderate to high, which was slightly larger than anticipated. Finally, correlations between the three PROMIS measures and positive aspects of caregiving from the CAS (i.e. satisfaction, ideology, and mastery) were generally small in both samples, providing additional support for discriminant validity.

Known Groups Validity.

Caregivers of high-functioning individuals had significantly higher scores on all PROMIS measures relative to caregivers of low-functioning individuals with the exception of Anger in the SMV sample (Table 3).

Table 3.

Known Groups Validity for PROMIS Mental Health CATs

	Caregiver of a High Functioning Individual		Caregiver of a Low Functioning Individual
PROMIS CATs	(MPAI −4<60)		(MPAI−4≥60)
	Mean (SD)	% Impaired*	Mean (SD)	% Impaired*	t	p
Civilian sample	N = 293		N = 42
Anger	50.46 (9.82)	19.8	56.68 (7.84)	40.5	3.93	<.001
Anxiety	51.78(9.13)	19.1	56.96 (7.73)	40.5	3.50	.001
Depression	49.53 (9.48)	13.0	54.95 (8.91)	26.2	3.49	.001
Military sample	N = 104		N = 101
Anger	55.48(9.21)	27.9	57.87 (8.92)	30.7	1.88	.061
Anxiety	56.60 (8.62)	32.7	60.50 (8.74)	43.6	3.22	.002
Depression	52.84 (8.75)	16.3	56.93 (9.60)	33.7	3.19	.002

Note.

^*PROMIS = Patient Reported Outcome Measurement Information System; CATs = Computer Adaptive Tests; MPAI = Mayo-Portland Adaptability Inventory – Fourth Edition

Impairment rates.

Overall impairment rates in emotional health for caregivers of high-functioning individuals were comparable to the U.S. general population (Table 3). However, impairment rates in mental health were elevated for those caring for low-functioning individuals in both samples.

Discussion

These findings provide support for the reliability and validity of three PROMIS Mental Health measures, Anxiety, Depression and Anger, for use in caregivers of civilians or SMVs with TBI. Internal consistency reliability for the PROMIS Mental Health measures was good to excellent for both samples (.86 to .94), with the CAT generally performing better than its associated SF. While the PROMIS measures were free of floor effects in both samples, there was a small ceiling effect for the Anxiety and Depression SFs in the civilian sample, and for the Depression SF in the SMV sample. The short forms do not appear to include enough items to discriminate between those low rates of anxiety; thus, CAT administrations would be preferable for these constructs. While most of the PROMIS Mental Health measures exceeded the a priori criterion for 3-week temporal stability (i.e., test-retest-reliability ≥ .70) for the civilian sample (all measures except the Anxiety CAT were >.75), test-retest reliability was generally below a priori standards for the SMV sample (.65 to .69 across the measures with the exception of the Depression SF which = .70). While this was below what we hypothesized, it may reflect the long duration between administrations. Test-retest reliability for measures where day-to-day fluctuations are expected tend to use short time frames; administrations are typically within hours or days of the initial assessment, rather than weeks as in this study. It is also possible that these low values are related to group differences between the retest and initial sample; namely, the retest sample was younger, had higher educational attainment, and were caring for someone that was younger and more impaired than the initial sample. Furthermore, the fact that reliability was typically higher for the civilian sample, relative to the military sample, may indicate that these measures are more reliable in caregivers of civilian-related TBI. It is plausible that the military caregivers experience more variability in these emotions during a 3-week time frame. This possibility is supported by the fact that average scores on the PROMIS Mental Health measures were consistently at least one SD higher for the SMV caregivers relative to the civilian caregivers. These findings indicate that these are reliable measures of emotional health for caregivers of both civilians and SMVs with TBI.

Convergent and discriminant validity of the PROMIS Mental Health measures were also supported by our findings. Large correlations among the three PROMIS Mental Health measures and between the three Mental Health measures and RAND-12 MHC supported convergent validity. Negligible to small correlations among the three PROMIS Mental Health measures and RAND-12 PHC provided support for discriminant validity. Discriminant validity was also supported by moderate relationships among the three PROMIS measures and the two measures of caregiver burden, and negligible to small correlations between the three PROMIS measures and positive aspects of caregiving (mastery and satisfaction). Finally, known groups validity of the PROMIS measures was also supported in that those that were caring for high functioning individuals reported less mental health problems than those that were caring for low functioning individuals for both caregiver groups. The single exception was for caregivers of SMVs with TBI, where there was only a trend for group differences for anger. Regardless, this finding is consistent with qualitative findings that have highlighted the high levels of anger in SMV caregivers.¹⁴

Study Limitations

While this study provides psychometric support for PROMIS Anxiety, Depression, and Anger in caregivers of civilian- and military-related TBI, several study limitations must be acknowledged. First, caregivers were primarily Caucasian, women, and spouses of the person with the TBI, limiting the generalizability to other racial/ethnic minorities, male caregivers and non-spousal caregivers. Second, medical record documentation for injury severity was not available for the majority of the SMV caregivers. Third, the eligibility criteria differed between the civilian and SMV samples, which may account for some of the differences (i.e., the SMV sample included caregivers of individuals with uncomplicated mild and equivocal TBI, whereas the civilian sample did not). Finally, our test-retest reliability timeframe of 3 weeks was longer than typical test-retest reliability analyses.

Conclusions

Despite these limitations, this study supports the reliability and validity of the PROMIS Anxiety, Depression, and Anger item banks in caregivers of both civilians and SMVs with TBI. Ultimately, these measures can be used to provide a standardized assessment of HRQOL as it relates to caregivers’ mental health. Such work is needed so that future interventions designed at improving either mental health and/or HRQOL of caregivers of individuals with TBI have reliable and valid measures for use in quantifying the effectiveness of these interventions.

Highlights:

• Findings support using the PROMIS Mental Health measures in caregivers

• Caring for a person with a brain injury can negatively impact mental health

• Smart tests had some advantages over static self-report measures

List of abbreviations:

CAS

Caregiver Appraisal Scale

CAT

Computer Adaptive Test

HRQOL

Health-Related Quality of Life

MHC

Mental Health Composite of RAND-12

MPAI-4

Mayo-Portland Adaptability Inventory – Fourth Edition

PHC

Physical Health Composite of RAND-12

PROMIS

Patient-Reported Outcomes Measurement Information System

SF

Short Form

SMV

Service member/veteran

UM

University of Michigan

TBI

Traumatic Brain Injury

WRNMMC

Walter Reed National Military Medical Center

ZBI

Zarit Burden Inventory