Development and Preliminary Performance of a Risk Factor Screen to Predict Posttraumatic Psychological Disorder After Trauma Exposure

Abstract

Objective

We examined data from a prospective study of risk factors that increase vulnerability or resilience, exacerbate distress, or foster recovery to determine whether risk factors accurately predict which individuals will later have high posttraumatic (PT) symptom levels and whether brief measures of risk factors also accurately predict later symptom elevations.

Method

Using data from 129 adults exposed to traumatic injury of self or a loved one, we conducted receiver operating characteristic (ROC) analyses of 14 risk factors assessed by full-length measures, determined optimal cutoff scores and calculated predictive performance for the nine that were most predictive. For five risk factors, we identified sets of items that accounted for 90% of variance in total scores and calculated predictive performance for sets of brief risk measures.

Results

A set of nine risk factors assessed by full measures identified 89% of those who later had elevated PT symptoms (sensitivity) and 78% of those who did not (specificity). A set of four brief risk factor measures assessed soon after injury identified 86% of those who later had elevated PT symptoms and 72% of those who did not.

Conclusions

Use of sets of brief risk factor measures shows promise of accurate prediction of PT psychological disorder and probable PTSD or depression. Replication of predictive accuracy is needed in a new and larger sample.

1. Introduction

Each year, as many as one million Americans may develop posttraumatic stress disorder (PTSD) or depression after sudden illness or injury. According to the most recent available national hospital survey data from 2011, there were over 5 million hospitalizations for sudden illnesses and injuries requiring very urgent treatment (< 15 minutes) [1]. In one large-scale study that assessed hospitalized trauma patients 12 months after the injury, over 20% were found to have PTSD and/or depression [2]. PTSD can be debilitating for individuals [2, 3] and place a burden on families and society due to impaired occupational and daily functioning [2, 4–6], lost productivity, chronic medical problems [7, 8], disability [3, 9, 10], and increases in suicidality [11], violence [12], and criminal behavior [13]. Predictive screening after traumatic illness or injury for risk of later PTSD or depression could facilitate early intervention to prevent or reduce the severity of disorder, but accurate prediction has proved elusive.

Currently, hospitals in the U.S. do not routinely screen patients treated in trauma or emergency departments to identify those at risk for later PTSD or other mental health (MH) problems, although early screening for PTSD and depression following injury has been recommended by the American College of Surgeons’ Committee on Trauma [14]. In addition, the American Red Cross has begun collecting data on a set of pre-trauma and time-of-trauma risk factors in populations exposed to disasters, but the predictive performance of the triage tool used has not been studied. [15] Screening to detect current symptoms (as opposed to predicting future PTSD symptoms), is not as helpful, because early responses to trauma do not predict who will develop MH problems to a degree of accuracy that is clinically useful [16]. Most people exposed to traumatic events do have some symptoms in the days following the event and recover fairly quickly, but some do not. And some of those who are initially asymptomatic develop MH problems over time [17]. Dozens of studies have measured risk factors for PTSD [18, 19], but, typically, the risk factors are not assessed prospectively and do not capture enough variance in PTSD to make accurate predictions about individuals.

A recent review examined seven PTSD predictive screening tools and noted that for all but one, analyses to select items and analyses to test “performance” were conducted on the same data [20]. Selecting items that are highly related to the outcome yields a set of items that has the strongest possible relation to the outcome in the sample studied, but that set may not be as strongly related to the outcome in a new sample. A few other risk screening tools have been published, but they cannot be used for early predictive screening because they include variables that are typically not available until weeks or months after the event [21–25]. None of these screening tools have been given within days of an event and accurately differentiated between those who developed disorder and those who did not [21–25]. Screening tools have also been developed to identify those who currently have PTSD, but those are not designed to prospectively predict PTSD at a future time point [26, 27].

The only predictive screening tool that has had its performance examined in different data than was used to develop it is the Posttraumatic Adjustment Scale (PAS), which was developed and tested in Australia [28]. Performance of the PAS in a U.S. sample has not been reported, and differences between the populations seem likely to affect the screening tool’s accuracy for predicting PTSD in the U.S. In the Australian patients, the relative importance of risk factors to predict PTSD was different than observed in U.S. patients [28, 29]. In the Australian data, 52% of the variance in PTSD was accounted for by 5 factors: 1. acute stress symptoms, 2. prior social support, 3. pre-injury emotional and trauma history, 4. expectations about ability to cope with post-trauma life stress, and 5. trauma appraisals and post-trauma pain [28]. In contrast, in a U.S. sample that we have reported on previously, the risk factors that were most strongly related to PTSD were post-trauma life stress and negative thinking [29]. It is possible that universal coverage of medical care costs in Australia and the availability of financial support to Australians disabled by injuries explain some of these differences.

Two studies have examined the capacity of data mining methods or machine learning to predict long-term PTSD symptom levels. A large study of World Health Organization survey data from 24 countries (N = 126,096) used data mining methods to categorize the PTSD risk level of trauma exposures in a subsample of individuals who reported exposure to one or more traumas (N = 42,634). Exposure events were categorized into PTSD risk levels based on retrospective data on trauma exposure type, socio-demographics, and reported prior history of trauma exposure and mental disorder. While the study models performed well in classifying traumatic events that were associated with the vast majority of cases of PTSD, performance to predict PTSD in individuals was not reported [30]. Two reports on analyses of data from a study of Israeli patients analyzed the capacity of algorithms based on a wide range of predictors assessed by telephone 6 to 14 days post-trauma to predict a non-remitting PTSD symptom trajectory over 15 months. The algorithms provided fair overall predictive accuracy (AUC = .75) but did not allow accurate prediction for individuals [31, 32]. While these data mining studies show promise for predicting long-term PTSD symptoms, none demonstrated the capacity to predict short-term symptom elevation at the level of the individual, which could permit preventive intervention. Use of data from electronic medical records (EMRs) to screen for risk would also require overcoming major obstacles to implementation in the U.S., including lack of EMR systems in many settings and the high degree of effort and cost involved in introducing new elements into existing EMR systems.

To accurately predict mental health problems at the level of the individual after exposure to traumatic events, it is necessary to assess variables that account for a great deal of variance in outcomes. In a longitudinal study of risk and resilience to traumatic events, we studied fourteen variables that are hypothesized to increase vulnerability or resilience to traumatic stress or to impede or foster recovery. A model including these 14 risk factors accounted for 72% of the variance in posttraumatic (PT) symptoms (including symptoms of PTSD, depression, negative thinking, and dissociation) two months after trauma [29]. We studied these four categories of symptoms, because they are consistent with the DSM-5 diagnostic criteria for PTSD, allow dimensional measurement of symptoms, and are all strongly associated with exposure to traumatic stress [33–36].

The purpose of the current study was to determine whether the risk factors we assessed could accurately identify those who would later have high levels of PT symptoms and those who would not. Given the impracticality and high burden to patients of screening using full-length measures, we also investigated the performance of sets of briefer risk factor measures. We first conducted analyses to examine whether some of the risk factors we studied could be accurately assessed using fewer items and whether a smaller number of risk factors assessed by fewer items could accurately identify individuals who would later have high PT symptoms and those who would not. We also calculated the performance of risk factors that were measured soon after the traumatic injury. Analyses to determine the performance of Acute Stress Disorder symptoms to predict PT symptoms was also conducted to provide information about the relative performance of early symptoms alone to predict later symptoms. Since prior research has shown that prevalence of PTSD is elevated in some racial/ethnic minority groups [37, 38], it is important that any risk screening tool accurately predicts risk for these groups. Accordingly, we conducted a separate analysis of the performance of risk factors to accurately predict PT symptoms in the subset of participants who self-identified as members of racial or ethnic minority groups. Lastly, in order to provide information about performance of brief risk factors to predict whether participants would meet familiar diagnostic criteria, we also conducted a subset of the analyses to predict the outcome of probable diagnosis of PTSD or depression.

2. Materials and methods

2.1 Setting and participants

Participants were 147 trauma-exposed adults who were recruited to participate in a study of factors that influence responses to traumatic stress [29] and a study of an intensive ambulatory data collection method [39]. Participants had been treated in a Level I trauma center and hospitalized for a severe injury (54%) or had loved ones treated and hospitalized for life-threatening injuries (46%). Patients’ injuries were due to motor vehicle crashes, including motorcycle crashes, biking accidents, and pedestrians hit by cars (36%); accidental injuries at home or work (15%); and physical assault (3%). Women comprised 58% of the sample and 31% reported minority racial/ethnic identities, including 15% who self-identified as Hispanic, 8% who self-identified as Asian or Pacific Islander, 7% who reported two or more racial/ethnic identities, and 4% who self-identified as African-American. In this sample, 24% of the injured patients and 22% of the family members appeared to meet DSM-IV diagnostic criteria for PTSD two months post-trauma.

2.2 Measures

Detailed descriptions of the measures are provided in an earlier report [29] and in the online Supplementary File for this article. The risk factor variables assessed, measures used, and the number of items in each measure are listed in Table 1. Childhood home life was assessed with a single item inquiring “How was your home life growing up?” (very happy, pleasant, OK, mixed, unhappy) Pre-trauma home life was assessed with a single item inquiring “How is your home life now?” (very happy, pleasant, OK, mixed, unhappy). Trauma Severity was the sum of ratings of how “terrible” and “out of control” the event seemed (not at all, a little, some, much, very much). To assess mental health two months after the injury, we used measures of PTSD symptoms (Screen for Posttraumatic Stress Symptoms [40]), depression (short form of the Beck Depression Inventory – Short Form [BDI-SF] [41]), and dissociation (Dissociative Symptoms Scale [42]). Given that negative thoughts about self, others, and the world are now included in the DSM-5 diagnostic criteria for PTSD [43], we also assessed negative thinking two months post-trauma with 18 items from the Posttraumatic Cognitions Inventory (PTCI) [44].

Table 1.

Fourteen Risk Factors and Measures to Assess Them

Variable	Measure	Number of Items	Area Under Curve for Full Measure
Childhood Home Life	Childhood Home Life	1	.72
Past Traumatic Events	Trauma History Screen [34]	16	.73
Age at First Trauma	Age at First Trauma	1	.75
Prior Psychopathology	Millon Clinical Multiaxial Inventory III [51]	175	.79
Acute Stress Symptoms	Screen for Posttraumatic Stress Scale [40]	17	.82
	Dissociative Symptoms Scale [42]	20
Negative Thoughts	Posttraumatic Cognitions Inventory [44]	18	.95
Post-trauma Life Stress	Perceived Stress Scale [52]	10	.89
Early Social Support	Social Support Survey [53]	19	.71
Early Social Constraints	Social Constraints Scale [54]	15	.72
Pre-trauma Home Life	Pre-trauma Home Life Question	1	.69
Past High Magnitude Stressors	Trauma History Screen [34]	16	.68
Trauma Severity	Trauma Severity Score	2	.66
Parental Dysfunction	Parental Dysfunction Score	4	.63
Years of Education	Years of Education	1	.59

As in our prior publication [29], the primary outcome in this study was an index of posttraumatic psychological disorder derived from scores on measures of symptoms of PTSD (SPTSS), depression (BDI-SF), negative thinking (PTCI), and dissociation (DSS) collected two months post-trauma. This index was thought to best represent the DSM-5 diagnostic criteria for PTSD and allow dimensional measurement of symptoms. Scores on the measures of PTSD, depression, negative thinking, and dissociation were highly correlated (r = .69 to r = .83), and principal components analysis of these four symptom categories identified one posttraumatic symptoms factor (PT symptoms) that accounted for 82% of variance in scores [29]. The primary outcome for analysis was the PT symptoms factor score.

In order to conduct analyses of the performance of risk factors to predict probable to diagnoses, SPTSS scores and BDI-SF scores were used to categorize participants as having probable PTSD and/or probable depression when assessed two months after the traumatic injury. SPTSS scores of 20 or higher were used to categorize subjects as positive for PTSD based on strong concurrent validity with the diagnosis of PTSD from structured interviews [40, 45] and on analysis of data from a subsample of 40 participants in the current study who were administered the Clinician-Administered PTSD Scale (CAPS) [46]. BDI-SF scores of 10 or higher were used to categorize subjects as positive for depression based on a study of hospitalized medical patients showing high sensitivity (1.0) and high specificity (.83) of scores of 10 or higher in predicting a diagnosis of depression determined by a structured interview. [47]

2.3 Procedures

Admitted patients were identified via hospital electronic records and approached in their hospital rooms between one and fourteen days after injury. Family members were approached in the patient’s room or in waiting areas. For each traumatic injury event, only a single patient or family member was enrolled. After obtaining informed consent, participants completed all measures listed in Table 1 with the following exceptions. The measures of social support and social constraints were completed one week after enrollment and again two months after the traumatic injury. Only data from the first administration are included in the analyses reported here. Two months post-trauma, participants completed the measures of post-trauma life stress, PTSD symptoms, depression, dissociation, and negative thinking for a second time.

2.3 Data analysis methods

The measures of social support, social constraints, and post-trauma life stress were added to the study mid-way through, which resulted in missing data for social support and social constraints from 53% of participants and missing data for post-trauma life stress for 45% of participants. Data were also missing for 13% of participants for psychopathology, 11% for acute stress, and 2 to 5% for all other variables. Consideration of the reasons for most of the missing data (measures not included in the study at that time) and analysis of patterns of missingness led us to consider the data to be missing Completely at Random. Multiple imputation was conducted to generate five complete datasets. Predictive analyses were conducted on pooled statistical values from the five imputed datasets in the subsample of 129 for whom follow-up assessments were available.

Receiver Operating Characteristic (ROC) analyses were conducted on the 14 risk factors shown in Table 1 to select the most predictive risk factors to include in further analyses. The value for classifying PT symptoms status as elevated was the PT symptom factor score value associated with a score of 45 or higher on the CAPS [46, 48], which was administered to a subset of 40 participants. Selection of risk factors to include in further analyses was based on Area Under the Curve (AUC) values of .70 or higher to predict PT symptom status. ROC analyses for the full measures were also used to determine optimal cut points to classify participants as positive on each risk factor.

To create brief risk measures for Negative Thoughts, Post-trauma Life Stress, Post-trauma Social Support, Post-trauma Social Constraints, and Acute Stress Symptoms, forward regression analyses were conducted on all available data for risk factors (N = 147) to identify items that accounted for 90% of the variance in total measure scores. ROC analyses were used to determine cut-points for presence of each risk. Brief risk factors were not created for Past Traumatic Events, Age at First Traumatic Event, and Psychopathology, because measurement of these risk factors requires a large number of items. There was no need to create a brief measure of Childhood Home Life, because only one item was used to assess this risk factor.

3. Results

Of 147 enrolled participants for whom baseline data were available, 129 (88%) completed follow-up assessments. Table 1 shows all 14 risk factors and their AUC values. Nine risk factors met our criterion of an AUC of .70 or higher to predict PT symptom status and were included in further analyses. Figure 1a shows the per participant number of positive risks of the nine risk factors assessed by full measures. The mean number of positive risk factors per participant was 4.0 (median = 4). Table 2 shows the performance of a risk count of 5 or more risks assessed by full measures to classify elevation in PT symptoms assessed two months post-trauma. Also shown in Table 2 is the performance of Acute Stress Disorder symptoms (assessed using full measures of PTSD and dissociation) to predict later PT elevation status.

Figure 1.

a. Frequency distribution of positive risk factors per participant assessed by nine full risk factor measures.

b. Frequency distribution of positive risk factors per participant assessed by six brief risk factor measures.

Table 2.

Classification Performance of Full Risk Factor Measures

Risks Assessed (Cut point for Positive Classification)	Positive Classification Rule	AUC	% Correctly classified	SE	SP
Childhood Home Life (score of 2 or higher)	5 or more risks	.92	81%	.89	.78
Past Traumatic Events (3 or more)
Age at First Trauma (21 or older)
Prior Psychopathology (score of −.40 or higher)
Acute Stress Symptoms (score of 23 or higher)
Negative Thoughts (score of 13 or higher)
Post-trauma Life Stress (score of 20 or higher)
Post-Trauma Social Support (score of 3.65 or higher)
Post-Trauma Social Constraints (score of 28 or higher)
Acute Stress (score of 23 or higher)	Score of 23 or higher	.81	70%	.79	.67

Note: SE = sensitivity, SP = specificity

Correlations between brief and full risk measures ranged from .91 to .97. Figure 1b shows the per participant number of “positive” risk factors assessed by six brief measures. The mean number positive per participant was 2.3 (median = 2). Table 3 shows the performance of six, five, four, three, and two risk factors to correctly classify participants for PT elevation. Also shown in Table 3 is the performance of four variables that were assessed prospectively soon after the event and the performance of Acute Stress Disorder symptoms to predict PT elevation at two months post-trauma. Lastly, Table 3 shows the performance of the four risk factors assessed soon after trauma to predict later PT elevation in the 37 participants who self-identified as members of one or more racial or ethnic minority group. Figure 2 shows the ROC curves for analyses presented in Table 3. Figure 3 shows the percent of participants who later had high and low PTPD by risk score with a cutoff of 2 or more risks marked.

Table 3.

Classification Performance of Brief Risk Factor Measure Sets to Predict PT Symptoms

Risks Assessed (items per risk)	Number of Items	Positive Classification Rule	AUC	% Correctly classified	SE	SP
Negative Thoughts (6)	21	3 or more risks	.90	80%	.91	.75
Post-trauma Life Stress (2)
Post-Trauma Social Constraints (3)
Post-Trauma Social Support (3)
Acute Stress Symptoms (6)
Childhood Home Life (1)
Negative Thoughts	20	3 or more risks	.91	84%	.85	.83
Post-trauma Life Stress
Post-Trauma Social Constraints
Post-Trauma Social Support
Acute Stress Symptoms
Negative Thoughts	14	2 or more risks	.90	76%	.97	.68
Post-trauma Life Stress
Post-Trauma Social Constraints
Post-Trauma Social Support
Negative Thoughts	11	2 or more risks	.89	77%	.97	.70
Post-trauma Life Stress
Post-Trauma Social Constraints
Negative Thoughts	8	one risk	.87	82%	.88	.80
Post-trauma Life Stress
Acute Stress (PTSD and dissociation items that account for 90% of variance in total Acute Stress score)	6	5 or more risks	.78	70%	.72	.69
Four Prospective Risks:	18	2 or more risks	.91	76%	.86	.72
Negative Thoughts
Post-Trauma Social Constraints
Post-Trauma Social Support
Acute Stress Symptoms
Four Prospective Risks for Racial/Ethnic Minority Subjects:	18	2 or more risks	.82	78%	.78	.78
Negative Thoughts
Post-Trauma Social Constraints
Post-Trauma Social Support
Acute Stress Symptoms

Note: Brief risk measures contain the same specific items for all analyses reported.

Figure 2.

Receiver Operating Characteristic Curves for Brief Risk Measure Sets

Figure 3.

Percent of Participants with Elevated PTPD by Risk Score for Four Prospectively Assessed Risk Factors

Table 4 shows results of analyses to examine performance of risk factors to predict probable diagnoses of PTSD and/or depression.

Table 4.

Classification Performance of Brief Risk Factor Measure Sets to Predict Probable PTSD or Depression

Risks Assessed (items per risk)	Number of Items	Positive Classification Rule	AUC	% Correctly classified	SE	SP
Negative Thoughts (6)	21	3 or more risks	.89	80%	.90	.77
Post-trauma Life Stress (2)
Post-Trauma Social Constraints (3)
Post-Trauma Social Support (3)
Acute Stress Symptoms (6)
Childhood Home Life (1)
Four Measures Collected Early:	18	2 or more risks	.86	76%	.85	.73
Negative Thoughts
Post-Trauma Social Constraints
Post-Trauma Social Support
Acute Stress Symptoms
Four Measures Collected Early for Racial/Ethnic Minority Subjects:	18	2 or more risks	.80	75%	.76	.75
Negative Thoughts
Post-Trauma Social Constraints
Post-Trauma Social Support
Acute Stress Symptoms

Note: Brief risk measures contain the same specific items for all analyses reported.

Discussion

The analyses conducted indicate that the set of risk factors we studied shows promise of accurate prediction of elevation in post-traumatic symptoms at the level of the individual. Accurate prediction was also possible with small sets of brief measures and for individuals who self-identify as ethnic minorities. Sets of full and brief risk factor measures yielded higher sensitivities and specificities than acute stress symptoms alone.

The findings of sensitivities to predict PT symptom elevation and probable PTSD and/or depression ranging from .85 to .97 and specificities ranging from .68 and .83 indicate strong predictive performances for risk factor counts that compare favorably to the performance of the PAS to predict PTSD (SE = .82, SP = .84) and a major depressive episode (SE = .72, SP = .75) in an Australian sample [28]. The two studies assessed many of the same risk factors, but the only variable included in our sets of brief risk factor measures and the PAS was acute stress symptoms. This may be the result of different methods for selecting the most predictive risks. PAS items were selected based on results of exploratory and confirmatory factor analysis of a large pool of items that measured the risk factors. The PAS excluded items loading onto multiple risk factors and included two items to measure each of five risk factors. In contrast, we selected risk factors based on our theoretical model, then sought to reduce the number of items required to assess each risk factor, and used two to six items to measure most risk factors.

Strengths of this study include the theoretical basis for selection of risk factors, the prospective measurement of most risk factors, and selection of items based on total measure scores rather than relationships to the outcome to be predicted. An important limitation to the study was that one highly predictive risk factor (post-trauma life stress) was measured retrospectively at the same time as the outcome. While expected life stress could be measured soon after trauma, that variable may not be as highly predictive as the variable measured retrospectively. It is important to note, however, that the predictive performance was also strong of a set of four brief measures that were all collected soon after the trauma (SE = .86; SP = .72).

Additional limitations were that some risk factors were measured with as few as one item, and some variables that are thought to increase resilient responses to traumatic stress, such as emotion regulation capacity [49] and attachment security [50], were not included in the study. Further research is needed to replicate (i.e., cross-validate) these results in a larger sample and to validate the screening tool in a variety of racial and ethnic minority groups. Such a study should include only risks that can be measured soon after a traumatic event and should include diagnostic interviews of PTSD and depression to assess outcomes. An accurate and cross-validated predictive screening tool would be a valuable addition to available clinical tools to care for those exposed to traumatic events.