A brief screening tool for assessing psychological trauma in clinical practice: development and validation of the New York PTSD Risk Score

Abstract

Objective

The objective was to develop a brief posttraumatic stress disorder (PTSD) screening instrument that is useful in clinical practice, similar to the Framingham Risk Score used in cardiovascular medicine.

Methods

We used data collected in New York City after the World Trade Center disaster (WTCD) and other trauma data to develop a new PTSD prediction tool — the New York PTSD Risk Score. We used diagnostic test methods to examine different clinical domains, including PTSD symptoms, trauma exposures, sleep disturbances, suicidal thoughts, depression symptoms, demographic factors and other measures to assess different PTSD prediction models.

Results

Using receiver operating curve (ROC) and bootstrap methods, five prediction domains, including core PTSD symptoms, sleep disturbance, access to care status, depression symptoms and trauma history, and five demographic variables, including gender, age, education, race and ethnicity, were identified. For the best prediction model, the area under the ROC curve (AUC) was 0.880 for the Primary Care PTSD Screen alone (specificity=82.2%, sensitivity=93.7%). Adding care status, sleep disturbance, depression and trauma exposure increased the AUC to 0.943 (specificity=85.7%, sensitivity=93.1%), a significant ROC improvement (P < .0001). Adding demographic variables increased the AUC to 0.945, which was not significant (P=.250). To externally validate these models, we applied the WTCD results to 705 pain patients treated at a multispecialty group practice and to 225 trauma patients treated at a Level I Trauma Center. These results validated those from the original WTCD development and validation samples.

Conclusion

The New York PTSD Risk Score is a multifactor prediction tool that includes the Primary Care PTSD Screen, depression symptoms, access to care, sleep disturbance, trauma history and demographic variables and appears to be effective in predicting PTSD among patients seen in healthcare settings. This prediction tool is simple to administer and appears to outperform other screening measures.

1. Introduction

The goal of this study is to identify risk assessment instruments for posttraumatic stress disorder (PTSD) that are useful and effective in clinical practice. To develop screening interventions that can become part of routine care, there is a need for decision tools to be brief and useful for both clinicians and patients [1]. Strategies for disease screening are dictated by evidence that the testing is practical and effective. Most screening tests in medicine are readily available, easy to administer and inexpensive (e.g., tests for cholesterol, blood pressure assessments for hypertension, etc.) [1]. New screening tests need to be subjected to evaluation of sensitivity, specificity, impact on disease and cost-effectiveness. Clinicians are continuously introduced to new screening tests, oftentimes before evaluation is complete. For example, the use of genetic testing has been advocated as a screener for many disorders. However, due to the costs and risk of false-positive results, there is little evidence to justify this approach in many clinical areas [2].

To meet our study objective, we used a prospective study of adults exposed to the World Trade Center disaster (WTCD) in New York City (NYC) [3–14]. This study collected extensive clinical data, including mental health, physical health, trauma exposure, and psychosocial and demographic data spanning both the pre- and postdisaster periods [15]. The WTCD data set included a large sample of community-dwelling adults randomly selected from the five boroughs of NYC. In addition, as we note below, PTSD was assessed in this study based on the full Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition criteria [16]. The WTCD data were used together with other trauma data from other studies, including a study of chronic pain patients and a study of trauma patients discharged from a Level I Trauma Center.

To date, a number of brief PTSD screening tools are available. These screeners include the Primary Care PTSD Screen (PCPS), the Short Screening Scale for PTSD (SSSP), the abbreviated PTSD Checklist, the Short PTSD Rating Interview and the Screening Tool for Early Prediction of PTSD (STEPP), among others [17–25]. With the exception of the STEPP scale, which was designed for children and parents, these screening scales were developed for adult populations. They are all relatively short, appear to have acceptable specificity and sensitivity, and are generally focused on only PTSD symptoms. Presently, the PCPS is being used not only among Department of Defense and Department of Veterans Affairs personnel but also among civilian populations [18,20,26]. This usage would make the PCPS the most common PTSD screener in current use in the United States.

The goal of the current research was to develop a new screening instrument that was multidimensional, was effective and could be used in clinical settings. To guide our approach, we used the Framingham Risk Score (FRS) as a clinical model [27,28]. The FRS uses clinical data to estimate a patient’s risk for future coronary heart disease. Although there are different Framingham scores in use [29,30], the risk factors used in the FRS model typically include age, total cholesterol, high-density lipoprotein cholesterol, systolic blood pressure, treatment for hypertension and cigarette smoking. The objective of the FRS is to identify the common factors in different clinical domains that contribute to disease onset. Consistent with the FRS, we chose to examine a range of risk factors that would extend beyond PTSD screeners in current use [17,22]. Our plan was to evaluate a range of core symptoms, psychosocial risk measures and demographic factors to develop robust prediction models that use the minimum number of data elements.

Past research regarding the health consequences of traumatic events guided our current study [31–34]. Although some investigators have contended that persons recover quickly from traumatic experiences [35], reviews of studies have concluded that traumatic event exposures can result in significant health consequences for some persons [36–38]. In terms of PTSD, it has been suggested that this condition was associated with impairment comparable to other serious mental disorders [39]. Similarly, national studies of veterans have reported that many with PTSD had significant impairments [40,41]. Recent research also suggests that PTSD is associated not only with neuroendocrine and immune system alterations [42–45] but also with the onset of chronic medical conditions [46–48]. Furthermore, research on community disasters has suggested that survivors of these events have increased psychological problems, health concerns and chronic problems in living and typically experience significant psychosocial resource losses [12,38,49]. In summary, past research related to traumatic stress exposures and PTSD suggests that psychological, social and physical health consequences are associated with these experiences [50]. Based on this body of research, the focus of the current study was to develop valid and reliable PTSD diagnostic tools that were multifactorial, but that could still be used in routine clinical practice.

2. Methods

2.1. Conceptual approach

Although level of exposure and trauma-related loss are often associated with the impact of traumatic events [36,38], there are other mediating variables. Research suggests that increased PTSD vulnerability often occurs among those with a history of mental health disorders, child adversity and previous traumas [42,51,52]. Socioeconomic and racial/ethnic factors are also reported to influence these experiences [10,53]. In addition, research has identified the role of social support among those exposed to traumatic events [12,49]. Thus, the degree of exposure, social/cultural variables and other factors are often involved in determining the impact of traumatic stressors [3]. In addition, the psychobiological bases of these stress syndromes have also become apparent [54,55]. Consequently, it was expected that a number of behavioral health problems would emerge among traumatized persons, including sleep disturbances, substance misuse, alterations in functional health and mental health status, and the onset of other health problems and concerns [47,56].

Currently, PTSD is thought to be associated with outcomes along several causal pathways that encompass psychological, behavioral and biological domains [57,58]. For example, PTSD can directly result in changes in health status through alterations in physiologic functions associated with stress exposure or, indirectly, through altered health behaviors (e.g., alcohol abuse, increased cigarette smoking, etc.) resulting from altered psychological states (e.g., increased anxiety) brought on by PTSD psychopathology (e.g., intrusive thoughts). The onset of PTSD symptoms can also result in treatment or social support seeking, which may affect future mental health status. Personality and genetic factors are also thought to mediate these symptoms [59]. Consequently, we used a multifactorial conceptual approach to guide model building combined with agnostic (i.e., atheoretical) examination of statistical results. It is also noted that earlier studies assessing the relationship between trauma exposure and psychosocial outcomes have been often confined to smaller, institutional or nonrepresentative community samples [33,34,60]. As described below, the WTCD cohort represents one of the major longitudinal studies that examined mental health outcomes following a major traumatic event among a random population sample of adults [3–14]. These data, combined with the other validation data described below, enabled the testing of specific models that were both conceptually and empirically grounded and included sample sizes adequate for data analysis. Table 1 presents the main measures used in this study related to the WTCD development sample.

Table 1.

Main measurements in WTCD study and measurement timeframes

Measurement domain	Measure 1	Measure 2	Measure 3	Measure 4
1. Mental health	DSM-IV PTSD (lifetime, 12 months after WTCD, 24 months after WTCD)	DSM-IV major depression (lifetime, 12 months after WTCD, 24 months after WTCD)	PTSD and depression screeners (12 months after WTCD, 24 months after WTCD)	BSI-18 symptom scale 12 months after WTCD, 24 months after WTCD)
2. Other mental health outcomes	DSM-IV panic attack (lifetime, 12 months after WTCD, 24 months after WTCD)	Peri-event panic attack during or immediately after WTCD	Suicidal thoughts (post-WTCD only)	Fear of death (post-WTCD only)
3. Substance use/misuse	Tobacco use (lifetime, 12 months after WTCD, 24 months after WTCD)	Alcohol consumption (12 months before WTCD, 12 months after, 24 months after WTCD)	Binge drinking (12 months before WTCD, 12 months after, 24 months after WTCD)	Cage alcohol scale (12 months before WTCD, 12 months after, 24 months after WTCD)
4. Healthcare visits, treatments and interventions	Outpatient visits & hospitalizations (lifetime, 12 months after WTCD, 24 months after WTCD)	Outpatient mental health visits & hospitalizations (lifetime, 12 months after WTCD, 24 months after WTCD)	Psychotropic medication use (lifetime, 12 months after WTCD, 24 months after WTCD)	Mental health interventions & access to care (12 months after WTCD, 24 months after WTCD)
5. Stress exposures	Level of exposure to WTC disaster events (12 months after WTCD)	Traumatic exposures (lifetime, 12 months after WTCD, 24 months after WTCD)	Stressful life events (12 months before WTCD, 24 months after WTCD)
6. Social/community resources	Social support (12 months after WTCD, 24 months after WTCD)	Assistance from friends & neighbors (12 months after WTCD, 24 months after WTCD)	Social capital scale (lifetime)	Zip-code-level census & NYC health data for year 2000
7. Psychological status and personality	Rosenberg self-esteem scale (12 months after WTCD, 24 months after WTCD)	Anomie hostility scale (lifetime)	Antisocial personality screen (lifetime)	History of attention deficient disorder (lifetime)
8. Functional health status	SF-12: mental & physical functioning (12 months after WTCD, 24 months after WTCD)	Reported work productivity (12 months after WTCD, 24 months after WTCD)	Sleep disturbance and pain status (12 months after WTCD, 24 months after WTCD)
9. Demographic measures	Age, gender, income, education, ethnicity, race, immigration status, language spoken (current)	Physician status, insurance coverage, employment status, work productivity (current)	Religion, church attendance (current)	Household composition (current)
10. Other measures	Use of alternative services (12 months after WTCD, 24 months after WTCD)	Disaster rescue & recovery involvement (12 months after WTCD	Handedness scale (lifetime)	Reported medical conditions (lifetime)

BSI-18=Brief Symptom Inventory-18; SF-12=Short-Form-12.

2.2. Statistical approach

We used a process of moving variables in and out of the prediction models to allow for the manipulation of specificity and sensitivity for different combination of variables [61]. This calibration permitted establishment of risk score cut-points that would be the most useful for clinicians. This approach was sensitive to a balance between statistical and clinical significance in developing models that could be useful in clinical decision-making. We used methods developed for diagnostic test assessments, including sensitivity, specificity, receiver operating characteristic (ROC) curves and bootstrap techniques [61]. First, an initial model (i.e., the base model) was developed using traditional variables thought to be related to PTSD. This model was then extended to include the unique collection of measurements of interest available from the WTCD cohort study. These variables included, but were not limited to, mental health status, substance misuse, stress exposures, social/community resources and functional status measures, among others (Table 1). The goal of this model building was to estimate the area under the ROC curve (AUC), while using the fewest number of parameters and the briefest measurement scales. The AUC was estimated at each step to quantify the prediction accuracy of the diagnostic models [62]. That is, it provided a quantitative measure of the discrimination ability of a model to correctly classify patients with and without PTSD. The sequential addition of variables to the base model was evaluated in terms of increasing the AUC irrespective of the change in the likelihood function in logistic regressions [61].

A nonparametric method was used to hierarchically compare the added effects of other variables above the contribution of the base model [63]. This was achieved by comparing the areas under two correlated ROC curves using a nonparametric approach described by DeLong et al. and implemented in Stata 11.2 and SAS 9.2 statistical packages [63–65]. Specifically, this comparison is achieved by generating ROC curves using each possible outcome of the diagnostic test as a classification cut-point and computing the corresponding sensitivity and 1–specificity. These points are then connected by straight lines, and the resulting AUC is compared using a trapezoidal rule [63]. The results of the final models identified were then used to construct risk scores for PTSD onset following trauma exposure by subgroups of key predictor variables. The properties of these risk scores were examined in terms of sensitivity, specificity and AUC and by use of a nomogram, which is a graphical tool used to pictorially represent the model [66]. In addition, a common problem in estimating measures of diagnostic ability is that of overestimation. This is because the same data set is used for both model identification and prediction [67]. This problem was overcome by estimating a bias-corrected version using a bootstrap procedure that pulled 1000 random WTCD samples to provide an estimate of the AUC [68]. This procedure has been shown to be superior to the method of cross-validation using a training and validation data set [68]. Additionally, bootstrap 95% confidence intervals (CIs) for the bias-corrected AUC were also calculated for the ROC curves reported for the WTCD development sample. In addition to estimating the AUC, we also used Youden’s Index [61,69]. Youden’s Index is a summary measure of the ROC curve, as it provides a criterion for choosing a specific cutoff value for which both sensitivity and specificity are maximized [63,70].

It is expected that the NY PTSD model may not perform well for all participants. Therefore, we used these techniques to model sensitivity and specificity as functions of participant demographics including age, gender and ethnicity [71]. A single model was developed to create a risk score from resultant logistic regression analyses for each final prediction model identified. Using Youden’s Index, we obtained an optimal cutoff value of the risk score to classify the participants as PTSD cases or noncases. Leisenring et al. described how logistic regression may be used to model sensitivity and specificity as a function of covariates [71]. We utilized this approach to identify subgroups of interest in which the risk assessment model performs better than other subgroups. For our analyses, we used SAS version 9.2 and Stata version 11.2 [63–65]. In addition, Pepi software (version 4.0) was also used to estimate Youden’s Index and positive predictive and negative predictive values for each final model identified [69].

2.3. WTCD development and validation study

The WTCD was a mass-casualty event and represented the largest trauma-related loss of life and destruction in the United States since the American Civil War [3,72]. To study the impact of this event, baseline diagnostic interviews, using random-digit dialing, were conducted among NYC adults (18 years and older) by telephone in English and Spanish 1 year after the attack. A follow-up interview was conducted 1 year later. For the baseline survey, 2368 residents completed the interview. For this study, PTSD was diagnosed based on the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition criteria (DSM-IV) [16]. This PTSD measure was developed for telephone administration and used in previous mental health surveys [72–74]. To meet criteria for PTSD in this study, the person had to meet the full diagnostic criteria (A–F) for PTSD [15]. Cronbach’s alpha for the symptoms used in this scale was 0.90 [3], and the concordance of this PTSD scale compared to a clinical diagnostic interview was reported to be good [75]. Versions of this scale have been used in mental health surveys involving over 15,000 telephone interviews, including several WTCD surveys [3,72–74,76]. An important aspect of the WTCD survey is that it used the full PTSD diagnostic criteria, which is relatively rare in trauma studies [15]. Additional information on the WTCD study has been published elsewhere [3–14].

2.4. Chronic pain validation study

For this validation study, adult subjects (18+ years old) were selected from among noncancer chronic pain patients, defined as those prescribed four or more opioid pain prescriptions over a 12-month period. These individuals were selected from primary and specialty care patients seen in the Geisinger Clinic, a large integrated healthcare system that serves 2.5 million residents within 40 central and northeastern Pennsylvania counties. The majority of these pain patients had a history of physical injury, a condition often associated with PTSD [77,78]. Patients were eligible for this study if they received care from one of nine community practice clinics or from the three specialty clinics, including a pain management and an orthopedics clinic, and were prescribed opioid medications for non-malignant pain. Altogether, 12 clinics participated in the study, providing a sample pool of 2459 chronic pain patients. Diagnostic telephone interviews were completed from August 2007 through November 2008. Following a patient notification letter, telephone recruitment was initiated. Interviewers administered structured diagnostic interviews by telephone that included assessment of PTSD, depression, general anxiety and psychological trauma using the diagnostic interview adopted from the WTCD Study discussed above. This survey was administered using a computer-assisted telephone interviewing (CATI) system (WinCati, version 4.2; Sawtooth Technologies, Northbrook, IL, USA). Altogether, 705 patients were interviewed in this study. The survey cooperation rate (i.e., percentage interviewed after patient contact) was 51% (705/1390) [79]. This study was approved by the Geisinger Institutional Review Board (IRB). Additional information related to this study has been published elsewhere [80].

2.5. Trauma center validation study

For this validation study, adult subjects (18+ years old) were selected from trauma patients discharged from Geisinger Clinic’s Level I Trauma Center. As noted above, Geisinger serves 2.5 million residents located within 40 central and northeastern Pennsylvania counties. Patients were eligible for this trauma study if they were discharged alive from the Trauma Center and were not institutionalized at the time of the survey follow-up. Study interviews took place from June 2008 through December 2008. Patients for this study were contacted approximately 6 months after discharge from the Trauma Center. Following a patient notification letter, telephone recruitment was initiated. Following consent, interviewers also administered structured diagnostic interviews adopted from the WTCD Study that included assessment of PTSD, depression, anxiety and psychological trauma using a diagnostic interview designed for this purpose [15]. As in the pain study described, the survey was administered using a CATI system (WinCati, version 4.2; Sawtooth Technologies, Northbrook, IL, USA). Altogether, 225 patients were interviewed in the trauma study. Our survey cooperation rate for this study was 50% (225/451) [79]. This trauma study was approved by the Geisinger IRB.

2.6. Inclusion of existing PTSD screener instruments

As part of our study effort, we reviewed existing PTSD screening instruments currently in clinical use [17,21,22]. In our study, we included two instruments that are in common use, including the SSSP and the PCPS [19,26,81,82]. This inclusion was made possible because, although there were some minor variations in question wording, the PTSD symptom items used in these screeners were also included in the PTSD scales deployed in all three trauma studies.

3. Results

Table 1 presents the core predictor variables identified in the WTCD developmental data set. As seen, these variables span the pre- and the postdisaster periods and bridge different clinical domains, including the social, environmental and psychological factors. The general population profiles for three studies used in our analyses are shown in Table 2. Clearly, the WTCD developmental sample appears to differ from the pain and trauma studies in terms of race/ethnicity and level of education. These study samples, however, appear more comparable with respect to lifetime trauma exposure and the prevalence of PTSD (Table 2). Nevertheless, χ² tests indicated that differences between study samples shown were all statistically significant (P < .05).

Table 2.

Study profiles of populations used in New York PTSD risk score study

Study variablesa	WTCD study (N=2368) % (n)	Pain study (N=705) % (n)	Trauma study (N=225) % (n)
Mean age (S.D.)	43.1 (15.5)	54.5 (13.7)	48.4 (16.9)
Gender
Male	42.9 (1016)	32.9 (232)	55.1 (124)
Female	57.1 (1352)	67.1 (473)	44.9 (101)
Race
White	42.9 (1015)	98.4 (694)	99.1 (223)
African American	25.6 (606)	0.9 (6)	0.4 (1)
Hispanic/Latino	23.6 (559)	0.4 (3)	0.0 (0)
Other	7.9 (188)	0.3 (2)	0.4 (1)
Lifetime trauma exposure
None	28.0 (664)	21.4 (151)	37.3 (84)
Low exposure	23.6 (558)	24.8 (175)	25.3 (57)
Moderate exposure	28.2 (667)	30.9 (218)	22.7 (51)
High exposure	20.2 (479)	22.8 (161)	14.7 (33)
Trouble sleeping	32.6 (772)	15.2 (107)	32.0 (72)
Regular doctor/source of care	88.0 (2084)	98.4 (694)	95.1 (214)
College graduate	44.5 (1053)	19.7 (139)	26.7 (60)
Lifetime depression	26.2 (621)	35.3 (249)	14.2 (32)
Lifetime PTSD	10.0 (237)	13.8 (97)	12.4 (28)
Current PTSD	7.3 (174)	9.9 (70)	11.8 (26)

^aThe percentages shown for the WTCD are weighted for sample design, but the numbers shown are unweighted; χ² tests indicated that differences between study samples shown were all statistically significant (P < .05).

Table 3 summarizes the results for each study by different groups of predictor variables. As suggested, the SSSP and the PCPS were selected because they were in general use and also included in the PTSD symptoms scales used in the studies evaluated. As can be seen, these screeners alone are good predictors of PTSD. For example, in the WTCD bootstrapped evaluation sample, the SSSP had a specificity of 88.8% and a sensitivity of 92.5% (AUC=0.907); the PCPS had a specificity of 82.2% and a sensitivity of 93.7% (AUC=0.880). Among the pain patients, the results for the SSSP were a specificity of 95.6% and a sensitivity of 94.3% (AUC=0.949); the PCPS also had a specificity of 95.6% and a sensitivity of 94.3% (AUC=0.949). When we used these screeners in the trauma study, the results for the SSSP was a specificity of 85.9% and a sensitivity of 65.4% (AUC=0.757). However, the PCPS performed better among trauma patients, with a specificity of 80.9% and a sensitivity of 96.2% (AUC=0.885).

Table 3.

Prediction results: WTCD, pain and trauma studies using different prediction models

Study and prediction model	Cutoff score	% Specificity	% Sensitivity	PV+	PV−	AUC	AUC 95% CI	P value
PTSD screens only
WTCD (N=2368)
SSSP	–	88.8	92.5	39.4	99.3	0.907	0.883–0.930	–
PCPS	–	82.2	93.7	29.3	99.4	0.880	0.856–0.903	–
Pain study (N=705)
SSSP	–	95.6	94.3	70.1	99.4	0.949	0.917–0.982	–
PCPS	–	95.6	94.3	70.1	99.4	0.949	0.917–0.982	–
Trauma study (N=225)
SSSP	–	85.9	65.4	38.3	94.9	0.757	0.645–0.869	–
PCPS	–	80.9	96.2	40.3	99.4	0.885	0.829–0.942	–
PTSD screens+risk factors
WTCD (N=2368)
SSSP+risk factors	124	89.4	92.5	40.7	99.4	0.945	0.930–0.959	<.0001
PCPS+risk factors	158	85.7	93.1	33.9	99.4	0.943	0.932–0.956	<.0001
Pain study (N=705)
SSSP+risk factors	124	95.6	92.9	69.8	99.2	0.965	0.944–0.985	.0725
PCPS+risk factors	158	95.6	92.9	69.8	99.2	0.970	0.956–0.985	.0516
Trauma study (N=225)
SSSP+risk factors	124	87.4	65.4	40.1	95.0	0.874	0.813–0.935	<.0001
PCPS+risk factors	158	86.9	92.3	48.6	98.8	0.929	0.892–0.965	.0063
PTSD screens+risk factors+demos
WTCD (N=2368)
SSSP+risk factors+demos	178	91.2	90.2	42.7	99.2	0.949	0.937–0.961	.0862
PCPS+risk factors+demos	194	88.1	91.9	38.0	99.2	0.945	0.934–0.955	.2497
Pain study (N=705)
SSSP+risk factors+demos	178	95.9	88.6	70.4	98.7	0.966	0.947–0.984	.7779
PCPS+risk factors+demos	194	95.9	87.1	70.1	98.6	0.972	0.958–0.985	.3673
Trauma study (N=225)
SSSP+risk factors+demos	178	90.5	61.5	46.3	94.6	0.883	0.829–0.938	.2337
PCPS+risk factors+demos	194	88.4	80.8	48.3	97.2	0.928	0.892–0.965	.8929

Demos=demographics.

In the WTCD development sample, adding the psychosocial predictors to the model with the SSSP resulted in a significant improvement (AUC=945, P < .0001). This is also true for the PCPS (AUC=0.943, P < .0001). This level of improvement was not observed in the pain study, either for the SSSP (AUC=0.965, P=.0725) or for the PCPS (AUC=0.970, P < .0516), after psychosocial variables were added to the model. By comparison, in the trauma study, the results were significant when psychosocial variables were added both for the SSSP (AUC=0.874, P < .0001) and the PCPS (AUC=0.929, P=.0063). However, adding demographic variables did not improve the predictions in the WTCD study (AUC=0.949, P=.0862 for the SSSP model; AUC=0.945, P=.2497 for the PCPS model). The results were similar for the pain study (AUC=0.966, P=.7779 for the SSSP model; AUC=0.972, P=.3673 for the PCPS model) and for the trauma study (AUC=0.883, P=.2337 for the SSSP model; AUC=0.928, P=.8929 for the PCPS model).

Because the prevalence of PTSD in our study samples was relatively low (7.3%–11.6%), the predictive value of a positive test (PV+) was generally less than 50%, while the predictive value of a negative test (PV−) was typically 99% (Table 3). However, we note that given our prediction models, if our study populations had a PTSD prevalence of ~20%, statistical simulations using Pepi software suggested that the positive predictive value of a positive test would be about 80% to 90%, a substantial improvement.

Following the assessments using a combination of predictors, we also examined the predictive value of the psychosocial risk factors alone (Table 4). As can be seen, the use of the psychosocial predictors alone (i.e., sleep disturbance, depression symptoms, trauma history, not having a regular source of care) also worked well in predicting PTSD. In Table 4, the AUC results for the WTCD, pain and trauma studies range from 0.873 to 0.946, suggesting good predictive results using only these variables. However, adding demographic factors to the latter prediction models only improved the prediction results for the WTCD study, with AUC=0.906 and P=.004, but not for the pain or the trauma studies (Table 4).

Table 4.

Prediction results: WTCD, pain and trauma studies using psychosocial risk factor and demographic models only

Study and prediction model	Total cutoff score	% Specificity	% Sensitivity	PV+	PV−	AUC	AUC 95% CI	P value
Risk factors only
WTCD	152	81.2	85.1	26.3	98.6	0.898	0.879–0.916	–
Pain study	152	93.9	84.3	60.1	98.2	0.946	0.923–0.968	–
Trauma study	152	82.9	69.2	35.2	95.3	0.873	0.820–0.926	–
Risk factors+demos
WTCD	159	72.0	96.5	21.6	99.6	0.906	0.890–0.923	.0040
Pain study	159	77.6	92.9	31.3	99.0	0.946	0.924–0.969	.8113
Trauma study	159	77.9	80.8	32.8	96.8	0.878	0.827–0.929	.3749

Table 5 presents PTSD scoring results (i.e., the final derived weights) for the PCPS used to generate the classification results shown in Table 3 for these PCPS models. We present the PCPS results because these appeared to produce the best overall predictions and the PCPS has fewer scale items than the SSSP (four items vs. seven items). As seen, a positive score on the PCPS (i.e., three or more positive items) is given a base score of 100, and the psychosocial and demographic items are given weights (or scores) relative to this score, This scoring is based on the logistic regression analyses, whereby the b coefficients in these logistic regression models are converted to standard weights with the aid of a nomogram, as discussed (Fig. 1). The last row of Table 5 shows the cutoff score for a PTSD classification based on these weights: 100 for the PCPS used alone, 158 for the PCPS+psychosocial predictors and 194 for PCPS+psychosocial+demographic factors. Cutoff scores are also shown for psychosocial predictors alone (score=152) and for psychosocial+demographic factors (score=159). Thus, to be classified as a PTSD case using the full NY PTSD Risk Score model simply requires a positive score on the PCPS (score=100), exposure to two lifetime traumatic events (score=26), current sleep disturbance (score=46) and having no regular source of medical care (score=23), which sum to a total score of 195.

Table 5.

New York PTSD risk scores for PCPS, psychosocial risk factors and demographic factors

Predictor variables	(A) PC-PTSDa screen only	(B) PC-PTSD screen+ psychosocial factors	(C) PC-PTSD screen+psychosocial factors+demographics	(D) Psychosocial factors only	(E) Psychosocial factors+ demographics
Positive PCPS resultsa	100	100	100	–	–
PHQ2=1	–	22	21	46	44
PHQ2=2	–	60	62	100	100
Trauma events=1	–	−5	−4	0	2
Trauma events=2–3	–	25	26	31	33
Trauma events=4+		26	31	40	45
Sleep disturbance	–	46	46	86	81
No regular care or regular physician	–	21	23	26	24
Not college graduate	–	–	4	–	9
Female	–	–	12	–	13
Nonwhite	–	–	7	–	18
Latino	–	–	3	–	0
Age (years)b	–	–	0.10×(100−age)	–	0.25×(100−age)
PTSD cutoff score	100	158	194	152	159

^aPCPS with three positive items (out of four) equals an NY PTSD Risk Score=100.

^bAge score is calculated as age−100×0.10 or 0.25, respectively, depending on the model used.

Fig. 1.

Nomogram for PCPS with psychosocial predictors included. Fig. 1 shows scale metrics used in score development with the aid of a nomogram, whereby final regression coefficients are converted to additive risk scores relative the PCPS using a linear predictor associated with the probability of the outcome. Note that the variables “regdoc” and “traumcat1” are reversed (i.e., negatively weighted) and the values for each predictor variable in the nomogram correspond to those shown in Table 5 for column B. pcps=Primary Care PTSD Screen; phq2_1=one positive depression symptom on PHQ2 scale; phq2_2=two positive depression symptoms on PHQ2 scale; traumcat1=history of low trauma exposure (one event); traumcat2=history of moderate trauma exposure (two to three events); traumcat3=history of high trauma exposure (four+ events); prbslp1y=difficulty sleeping in past year; regdoc=has regular source of care/doctor. For an example of the use of a nomogram in clinical practice, see Kattan M.W., Wheeler T.M., Scardino P.T. Postoperative nomogram for disease recurrence after radical prostatectomy for prostate cancer. Journal of Clinical Oncology, 1999; 17: 1499–1507.

Finally, we assessed these results for men and women separately and found that while the overall results were generally the same, the PTSD screeners tended to contribute more to the prediction models for the women, while the depression symptoms contributed more to the prediction model for the men. We plan to analyze these findings further by gender to determine if separate prediction models are justified in a future study.

4. Discussion

We examined different clinical domains, including stressor exposures, psychosocial measures, sleep disturbances, suicidal thoughts, PTSD symptoms, depression symptoms and demographic factors, to evaluate different PTSD prediction models. As stated, our goal was to develop a prediction tool that was brief and consistent with the FRS concept. As shown, five prediction domains, including two PTSD symptom screeners (i.e., the Brief and PC-PTSD screeners), sleep disturbance, access to care, depression symptoms and past trauma exposure, and five demographic variables, including gender, age, education, race and Hispanic ethnicity, were identified. For the best overall model, the AUC was 0.880 for the PCPS alone (specificity=82.2%, sensitivity=93.7%). Adding physician status, sleep disturbance, depression symptoms and trauma exposure to this model increased the AUC to 0.943 (specificity=85.7%, sensitivity=93.1%), a significant ROC improvement (P < .0001). Adding demographic variables increased the AUC to 0.945, which was not significant (P=.2497). As discussed, to validate these findings, we also applied the prediction results to 705 pain patients treated at a large, multispecialty group practice and 225 patients discharged from a Level I Trauma Center.

In summary, these results were validated within the WTCD cohort development sample, as well as for the pain and trauma samples, for both the Brief PTSD and the PC-PTSD screeners. The results for both screeners were generally similar, with and without the other predictor variables. The one exception was that, in the trauma study, the Brief PTSD screener tended to have lower sensitivity (65.4%), while the specificity for this was reasonably good among these trauma patients (85.9%). In addition, using the psychosocial risk factors alone tended to also predict PTSD onset relatively well in the WTCD study. This was also true in the pain and trauma data sets (Table 4), which was not expected.

The use of PTSD screeners has increased recently with the growing interest in the impact of traumatic stressors in primary care. As indicated, the Department of Veterans Affairs and the Department of Defense are routinely using the PC-PTSD screener in clinical practice to assess veterans and active duty personnel [18]. As seen in the current study, the PC-PTSD screener appears to work well with non-veterans and nonmilitary personnel, as has been reported in other studies [26]. The addition of psychosocial predictors increases the predictive ability of the PC-PTSD screener, but adding demographic factors did not. However, even when used alone, the predictive utility of the PC-PTSD screener is impressive, with AUCs=0.880–0.949. The PC-PTSD screener consists of four PTSD symptom questions, which would require only a few minutes to administer in most cases. If the psychosocial questions are added, which include two depression questions, a trauma question, a sleep question and a healthcare status question, this would likely require less than 5 min to administer. Thus, the objective of the current study to develop a brief PTSD screener with high specificity and sensitivity appears to have been achieved. Our estimate is that the core instrument (i.e., PCPS+risk factors, without the demographics) would consist of nine questions and would require less than 5–6 min to administer in most cases.

The current study has several strengths and limitations. A major strength was that our study involved a large-scale random survey among a multiethnic urban population and three validation studies. The latter included the WTCD bootstrapped validation, the pain study validation and the trauma study validation. Thus, this study included a total combined sample size of 3298 adults, which included 270 PTSD cases diagnosed using the full DSM-IV criteria. We also assessed a broad range of psychological and interpersonal risk factors using standardized instruments and medical test development methods. Potential study limitations include the omission of individuals without a telephone and those who were institutionalized or homeless. Also, the pain and trauma studies were conducted by telephone and excluded those too ill to be interviewed or who were institutionalized. In addition, nonresponse bias also could have affected all our survey result — our survey response rates could have been higher in our surveys. The lower response rates could have biased our results. Additionally, the item wording for the PTSD symptoms included in the two screeners we used (i.e., the SSSP and PCPS) was slightly different than what is actually used in these screeners. This may have affected our results. Finally, we did not use the predictor identified to predict PTSD beyond the first year after trauma exposure. As part of this investigation, we are planning to study additional predictor variables to forecast PTSD 2 years after exposure. As noted elsewhere [7], longer-term PTSD onset, especially what has been termed “delayed” PTSD, is more difficult to predict.

Despite these limitations, our study suggests that simple, multifactor screening tools, such as the NY PTSD Risk Score, can be highly effective in PTSD screening. Using the Framingham Study as a conceptual model, we developed a multidimensional PTSD risk score based on brief PTSD screeners, depression symptoms, sleep disturbance, trauma history and health care/physician status that was highly effective in discriminating PTSD cases from noncases in three different study populations. This classification system can be used based on the available patient or provider time, including the PCPS alone or in combination with psychosocial risk factor predictors and demographic factors.

As suggested, the goal of this effort was to develop risk assessment tools that are sensitive to statistical and clinical significance in order to develop data useful for clinical decision-making. Our plan was to develop PTSD prediction models to facilitate early intervention by making it possible to identify high-risk groups from among all persons exposed to trauma. Judging by the results of our study, our study goal appears to have been met. Use of the New York PTSD Risk Score, and use of other measures, would make it possible to target resources for those at highest risk of developing PTSD while allowing lower-risk patients to be managed more conservatively [8]. For example, it has been recently reported that brief mental health interventions made available shortly after a traumatic exposure may be more effective than traditional psychotherapy sessions delivered at a later point in time [8]. If this intervention finding can be confirmed, then it may make sense clinically to offer this type of treatment after traumatic exposures among those who screen positive following such events. Further research is recommended to further verify our findings and to make contextual modifications to improve our results.

Appendix

Primary Care PTSD Screen (three positive symptoms out of four, past 12 months)

1. You had repeated bad dreams or nightmares or had disturbing or unpleasant memories, thoughts, or images that kept coming into your mind whether you wanted to think of them or not.

2. You deliberately tried hard not to think about something that happened to you or went out of your way to avoid certain places or activities that might remind you of something that happened in the past.

3. You felt you had to stay on guard much of the time or unexpected noises startled you more than usual.

4. You felt cut off from other people, found it difficult to feel close to other people, or you could not feel things anymore or you had much less emotion than you used to have.

Depression Symptoms (lifetime)

1. Have you ever had a period of 2 weeks or longer when you were feeling depressed or down most of the day or nearly everyday?

2. Have you ever had a period of 2 weeks or longer when you were uninterested in most things or unable to enjoy things you used to do?

Trauma Exposure (lifetime)

1. How many traumatic events do you think you have ever experienced? These are events outside of everyday experiences and include being in combat or a war zone, being assaulted or sexually attacked, being in a major disaster, fire, or accident, experiencing the sudden and unexpected death of a loved one, and things like these.

2. Would you say you never experienced events like these, experienced events like these only once, experienced these events 2–3 times, or experienced these event 4 times or more in your lifetime?

Sleep Disturbance (past 12 months)

1. In the past 12 months, have you had difficulty falling asleep or staying asleep?

Regular Doctor/Source of Healthcare (current)

1. Do you have a regular doctor or a usual source of care that you can go to for medical care?

Demographics

1. What is the highest level of education or schooling you completed (record as college graduate vs. not college graduate)?

2. How old are you (record in years)?

3. Are you of Spanish or Hispanic origin?

4. How would you describe your racial background: White, Black/African American, Asian, or something else (record as White vs. or not White)?

5. Patient/person’s gender (record by observation): female or male?

Study PTSD Criteria

The PTSD measure used was based on the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition [16]. To have PTSD, a person had to meet the full PTSD criteria (A–F). In the current study, this measure was based on the past 12 months. For each of the 17 DSM-IV PTSD symptoms, the respondent was asked if there was ever been a period of 2 weeks or more during which he/she experienced that symptom. If the respondent answered “yes,” he/she was asked if that symptom was related to up to three traumatic events. Following this, the person was asked when was the last time he/she experienced this symptom, how long he/she had experienced it and if this symptom resulted in significant impairment/distress. If these A–F criteria were met, the person was defined as a PTSD case [7,8].