A Psychometric Examination of Computerized Adaptive Measures of Posttraumatic Stress Disorder Among Military Veterans

Abstract

Brenner et al. (2021) developed and initially tested two computerized adaptive measures of posttraumatic stress disorder (PTSD), one that provides a provisional diagnosis (PTSD computerized adaptive diagnostic screen [CAD-PTSD]) and one that estimates symptom severity (PTSD computerized adaptive severity test [CAT-PTSD]). We expanded on the initial psychometric findings by collecting data regarding test–retest reliability, incremental validity, and respondent burden. A sample of veterans (N = 156, 32% women, 36% Black, 13% identified as Spanish, Hispanic, Latino, Puerto Rican, or Cuban) recruited from three Veterans Affairs medical centers completed the CAT-PTSD and CAD-PTSD, the Clinician-Administered PTSD Scale for DSM-5, the PTSD Checklist for DSM-5 (PCL-5), Primary Care PTSD Screen for DSM-5 (PC-PTSD-5), and a battery of other self-rated scales. Fifty-three participants (34%) completed the measures a second time within 7 days (M_days = 5.41; SD = 1.97) of their first visit. CAT-PTSD scores revealed good convergent validity (r = .78 with Clinician-Administered PTSD Scale for DSM-5 total score), discriminant validity, and test–retest reliability (r = .80). Scores on the PCL-5 and PC-PTSD-5 had similar characteristics. The CAD-PTSD demonstrated poor diagnostic efficiency, κ(.5, 0) = .40, and test–retest reliability (κ = .25), whereas previously established cut-scores for the PCL-5 and PC-PTSD-5 showed fair to good diagnostic utility and adequate to good test–retest reliability. Results suggest that the CAT-PTSD may provide a valid indicator of PTSD symptom severity, but does not offer incremental value beyond the PCL-5 and PC-PTSD-5. The CAD-PTSD was markedly inferior to the use of PCL-5 or PC-PTSD-5 cut scores.

In contrast to traditional questionnaire measures, which administer the same questions to everyone, computerized adaptive tests“adapt” to the individual respondent, selecting only items that will be maximally informative based on the individual’s responses to previous questions. In this way, computerized adaptive testing can be more efficient than traditional approaches, often requiring fewer test items to reach the same level of measurement precision. Adaptive tests have long been used in educational settings and are increasingly used in mental health care settings to screen for mental disorders, such as posttraumatic stress disorder (PTSD; Del Vecchio et al., 2011; Eisen et al., 2016).

Brenner et al. (2021) tested the PTSD computerized adaptive diagnostic screen (CAD-PTSD) and computerized adaptive severity test (CAT-PTSD), which were developed to efficiently provide provisional Diagnostic and Statistical Manual of Mental Disorders, fifth edition (DSM-5; American Psychiatric Association, 2022) diagnostic and symptom severity information. The full item pool of the CAT/CAD-PTSD has not been made publicly available, but it includes newly created items as well as items from numerous existing measures, including the Combat Exposure Scale (Keane et al., 1989), Davidson Trauma Scale (Davidson et al., 1997), Impact of Event Scale, revised (Weiss & Marmar, 1997), Minnesota Multiphasic Personality Inventory-2 Keane PTSD (PK) scale (Lyons & Keane, 1992), Personality Assessment Inventory Traumatic Stress subscale (Morey, 1991), PTSD Checklist for DSM-5 (PCL-5; Weathers, Litz, et al., 2013), and Trauma Symptom Checklist (Briere & Runtz, 1989), among others. The CAT-PTSD and CAD-PTSD both draw from the same item pool. The two measures differ only in the underlying algorithms governing item selection (Brenner et al., 2021). The CAT-PTSD adaptively selects items that are maximally informative for determining the respondent’s severity on the underlying latent PTSD construct. The CAD-PTSD adaptively selects items maximally informative for determining diagnostic status based on an external criterion (i.e., a clinician-administered PTSD assessment). The CAD-PTSD is also limited to six items while the CAT-PTSD administers items until a minimal level of measurement precision is reached.

In their initial psychometric examination, Brenner and colleagues found that the CAD-PTSD diagnostic screen had stronger diagnostic utility (area under the curve [AUC] = .91, 95% CI [0.87–0.95]) in predicting the Clinician-Administered PTSD Scale for the DSM-5 (Clinician-Administered PTSD Scale [CAPS]-5; Weathers, Blake, et al., 2013) diagnostic status than a traditional cutoff score (≥33) for the PCL-5 (AUC = .75, 95% CI [0.68–0.82]). Further, they found the CAT-PTSD severity score, which required an average of 10 items, was strongly correlated with the PCL-5 total score (r = .88). The authors concluded that the CAD-PTSD and CAT-PTSD are valid and efficient measures of PTSD diagnosis and symptom severity.

Although these results are promising, their generalizability is unclear as the authors did not validate the models on unseen testing data. Brenner et al. (2021) also did not report metrics calibrated for chance agreement between test and diagnosis, nor did they examine the CAT/CAD-PTSD measures’ broader nomological network, test–retest reliability, or incremental validity. On this last point, Brenner et al. did not compare the diagnostic performance of the CAD-PTSD with the five-item Primary Care PTSD Screen for the DSM-5 (PC-PTSD-5; Bovin et al., 2021) or the validity of the CAT-PTSD severity score to other PTSD measures beyond the PCL-5. This question of incremental validity is particularly important given the proprietary nature of the CAT/CAD-PTSD measures and the additional resources (i.e., computer, tablet, or other mobile internet-enabled device) required for their administration. Added value beyond available public-domain measures (e.g., PCL-5, PC-PTSD-5) is needed to justify these financial and technological costs. We addressed these limitations by contrasting the nomological network of the CAT/CAD-PTSD measures with those of three nonadaptive self-report PTSD measures previously developed for the purposes of PTSD diagnostic and severity screening, the PCL-5 (Weathers, Litz, et al., 2013), PC-PTSD-5 (Bovin et al., 2021), and the International Trauma Questionnaire (ITQ; Cloitre et al., 2018). The PCL-5 served as the primary point of comparison for the CAT-PTSD as it is the most widely used self-report PTSD symptom severity measure (Armour et al., 2016). The PC-PTSD-5 (Bovin et al., 2021) was the primary reference variable for the CAD-PTSD as it is similarly brief (i.e., five items) and was developed to serve primarily as a diagnostic screener. We included the ITQ for exploratory purposes as it is based on a different conceptualization of the PTSD construct (i.e., ICD-11 PTSD; Maercker & Brewin, 2024).

We contrasted the diagnostic utility and construct validity of these measures using a range of criterion variables. Diagnostic utility was examined using the CAPS-5, a clinician-administered diagnostic interview and the current gold standard for PTSD assessment. We examined construct validity through associations with other PTSD measures as well as criterion variables selected to range from theoretically strongly related to unrelated to the PTSD construct.

Criteria directly related to PTSD included lifetime trauma exposure, depersonalization, and risky behaviors. Lifetime trauma exposure (Weathers, Litz, et al., 2013) is necessary to assess because, as it is currently defined, PTSD symptoms cannot occur in the absence of a Criterion A event. Measures of depersonalization (Sierra & Berrios, 2000) and risky behaviors (Contractor et al., 2020) were included as both constructs are established PTSD symptoms and are prevalent among those with the disorder (Sommer et al., 2020; White et al., 2022). We also included measures of psychosocial functional impairment (Marx et al., 2019), problematic alcohol use (Saunders et al., 1993), and nonspecific emotional distress (i.e., depression, anxiety, stress; Lovibond & Lovibond, 1995), each of which is independent of PTSD but strongly correlated with the disorder (Debell et al., 2014; Goldstein et al., 2016; Ramchand et al., 2015).

Last, we included measures of intelligence and antagonistic interpersonal tendencies as criteria not expected to be strongly associated with PTSD symptom severity. There is no reason to suspect that an individual’s ability to read phonemically irregular words should be associated with PTSD symptom severity. Similarly, although PTSD is associated with increased anger and aggressive behaviors (MacManus et al., 2015; Morland et al., 2020), there is no evidence to suggest that it is associated with antagonistic tendencies like deceitfulness and manipulative behavior. A significant association with these constructs would therefore imply measurement confounding.

Method

Transparency and Openness

All data were analyzed using R Version 4.4.0 (R Core Team, 2024). Assessment materials and R analytic code for this study are available at https://osf.io/6yjvq/ (Lee et al., 2025). Data can be made available pending an approved data use agreement. Data requests should be directed to the corresponding author (B.P.M.). This study’s design and its analysis were not preregistered.

Participants and Procedure

We recruited veterans from three VA Medical Centers. Inclusion criteria were military veteran status, age 18–65, and ability to speak, read, and write in English. We intentionally used broad inclusion and limited exclusion criteria to recruit a heterogeneous sample. Methods for participant recruitment included publicly posted flyers, direct outreach to VA mental health clinics, and contacting local veterans who expressed interest in participating in research.

All study procedures were conducted in person. Following informed consent procedures, all veterans completed demographic and trauma history questionnaires. After participants self-identified an index event on the Life Events Checklist for the DSM-5 (Weathers, Litz, et al., 2013), a trained interviewer asked them to provide a brief verbal description of the event. The purpose of this procedure was to ensure the identified event (a) satisfied DSM-5 PTSD Criterion A as a traumatic event, and (b) was in fact the event that bothered the participant the most. The interviewer then instructed the participant to complete all PTSD questionnaire measures in reference to this event. The remaining questionnaires were administered in a randomized order to prevent order effects. We counterbalanced randomization such that measure presentation alternated between a PTSD measure (e.g., PCL-5) and non-PTSD measure (e.g., personality measure). The CAT/CAD-PTSD was administered as one of the randomized PTSD measures. Last, following completion of all questionnaire measures, study staff administered the interview measures. The CAPS-5 was administered in reference to the same index trauma as the questionnaire measures. All study assessors were mastersand doctoral-level personnel who received extensive training in the CAPS-5. Interviews were audio-recorded so interrater reliability could be examined.

Participants who completed these procedures were invited to return within 7 days of their initial participation. In the interest of maximizing recruitment for the initial procedure, the follow-up was presented as optional and participants were recruited and enrolled for participation regardless of their availability for the follow-up assessment. Procedures on both study days were identical except that we did not re-administer the demographics questionnaire or trauma history measures on the second day. The same index event was retained for both study days and participants were again instructed to complete PTSD measures with that event in mind. We administered all questionnaires except the CAT/CAD-PTSD using Qualtrics. We administered the CAT/CAD-PTSD using the Veterans Health Administration (VHA) online medical record platform (Mental Health Assistant–Web) as this was the only VA Institutional Review Board approved method for collecting these data. Participants were compensated $125 for the initial procedure. Those who participated in the follow-up were compensated an additional $100. All study procedures were approved for each study site. Study procedures were only initiated after obtaining written informed consent.

Of the 170 participants who provided written informed consent, a small number (n = 10; 5.88%) were excluded from the analyses because either CAT/CAD-PTSD or CAPS-5 data were not available (reasons included participants choosing to discontinue the assessment, errors in the CAT/CAD-PTSD administration program, and failure to meet inclusion criteria). An additional two participants (1.17%) were excluded due to protocol administration errors and a final participant was excluded because the assessor raised concerns regarding response validity. Among the final Day 1 sample (N = 157), 53 participants (33.76%) completed the Day 2 assessment; mean test–retest interval was 5.51 days (SD = 1.87).¹

Sample demographic characteristics were as follows: M_age = 47.45 (SD = 10.63); 105 participants (67.31%) identified as a man, 50 (32.05%) identified as a woman, and the remaining 2 endorsed another category (e.g., nonbinary/genderqueer, prefer not to answer) or did not respond; 90 participants (57.32%) identified as White, 56 (35.67%) identified as Black/African American, remaining participants endorsed another category or did not respond; 124 participants (78.98%) identified as not Spanish, Hispanic, or Latino; 88 participants (56.05%) reported being currently married or living with their partner, 50 (31.85%) reported being separated or divorced, and 17 (10.83%) reported having never married; 98 participants (62.42%) reported having a 2-year college degree or more.

Measures

We used a questionnaire to gather demographic (e.g., sex, gender, age, race, ethnicity, socioeconomic status) and basic military information (e.g., branch and years of service, rank, number of deployments, discharge status).

CAT-PTSD and CAD-PTSD

The CAT/CAD-PTSD was administered using the VHA online medical record platform (Brenner et al., 2021). The CAT-PTSD and CAD-PTSD can be administered individually or as a single combined assessment tool, which Brenner et al. (2021) recommended for the purpose of both diagnostic screening and symptom severity measurement. We followed this guidance and administered the CAT/CAD-PTSD as a single combined measure. In the combined administration procedure, the CAD-PTSD is administered first, and then additional items are administered in line with the CAT-PTSD until a PTSD symptom severity score with the necessary level of precision is reached. As with all computerized adaptive measures, the exact number and content of items presented varied by participant. For the CAD-PTSD, output included provisional diagnosis (positive or negative) and a probability score indicating the estimated likelihood of a positive PTSD diagnosis. For the CAT-PTSD, output included a severity score (ranging from 0 to 100), precision estimate (representing uncertainty in the severity estimate on a 100-point scale), and categorical severity stratification: none (severity <35), mild (severity ≥ 35), moderate (severity ≥ 47), and severe (severity ≥ 60). For both measures, the output included items administered, participant responses, and administration time.

Other Measures of PTSD

The PCL-5 (Weathers, Litz, et al., 2013) is a DSM-5 congruent 20-item questionnaire measure of PTSD symptom severity. Respondents rate the degree to which they have been bothered by each symptom during the past month. We summed items to generate a total score reflecting overall PTSD symptom severity; internal consistency for this measure was very high (Time 1 α = .95; Time 2 α = .95). A total score greater than or equal to 33 was used to examine diagnostic utility (Bovin et al., 2016).

The PC-PTSD-5 (Bovin et al., 2021) is a five-item questionnaire that was designed to identify individuals with probable PTSD in primary care settings. Respondents indicate using binary response options (yes/no), if they experienced each prompt in the past month. We summed the number of items endorsed to a total score ranging from 0 to 5. Internal consistency for this measure was acceptable (Time 1 α = .70; Time 2 α = .72). A score greater than or equal to four was used to examine diagnostic utility.

The International Trauma Questionnaire (ITQ; Cloitre et al., 2018) is an 18-item questionnaire measure of ICD-11 PTSD (World Health Organization, 2019). Respondents rate the degree to which they were bothered by each of the items during the past month on a five-item scale ranging from 0 = at all to 4 = extremely. We calculated a PTSD symptom severity score by summing the first six items, which correspond to ICD-11 PTSD; internal consistency for this measure was high (Time 1 α = .88; Time 2 α = .90). We also used the ICD-11 PTSD diagnostic algorithm (i.e., at least one reexperiencing, avoidance, and sense of current threat item, and at least one of the three related distress/functioning items endorsed as 2—moderately or higher) to examine diagnostic utility. An ICD-11 Complex PTSD (C-PTSD; Time 1 α = .92; Time 2 α = .93) severity score was also assessed by summing the 12 PTSD and C-PTSD items.

The CAPS-5 (Weathers, Blake, et al., 2013) is a structured clinical interview of PTSD diagnostic status and symptom severity. The CAPS-5 has demonstrated high test–retest and interrater reliability and good convergent and discriminant validity (Weathers et al., 2018). We used the CAPS-5 to determine diagnostic status and calculated a total score by summing the severity scores of the 20 items corresponding to Criteria B-E; internal consistency was high (Time 1 α = .89; Time 2 α = .87). We selected a random 25% of interviews (n = 39) for interrater reliability.

Trauma-Related Measures

We assessed lifetime trauma exposure history using the Life Events Checklist for the DSM-5 (Weathers, Litz, et al., 2013). The Life Events Checklist for the DSM-5 is a measure of lifetime exposure to potentially traumatic events. We used the extended version of this measure, which consists of two parts. In the first portion of the measure, respondents report their degree of lifetime exposure to several categories of stressors (e.g., natural disaster, transportation accident, sexual assault). We used this information to create a count variable of the number of categories of stressors participants reported directly experiencing as a proxy indicator of collective trauma exposure level. In the second portion of the measure, respondents were asked to identify the worst event they have experienced and were asked to provide a brief written description of the event as well as some summary details (e.g., how long ago it occurred, whether anyone’s life was in danger, if anyone was seriously injured or killed). This portion of the Life Events Checklist for the DSM-5 facilitated identification of the index trauma for the CAPS-5 interview.

The Cambridge Depersonalization Scale (Sierra & Berrios, 2000) is a 29-item measure of depersonalization. For the purpose for this study, we administered the 12-item unreality/detachment subscale. Respondents rate (a) how often they experienced each feature over the past 6 months on a 5-point scale ranging from 0—never to 4—all of the time and (b) the duration of each feature on a 6-point scale ranging from 1—a few seconds to 6—more than a week. We created a total score by summing the frequency and duration scores reflecting overall unreality/detachment severity; internal consistency for this score was very high (Time 1 α = .97; Time 2 α = .98).

The Posttrauma Risky Behaviors Questionnaire (Contractor et al., 2020) is a 16-item measure of engagement in numerous reckless and self-destructive behaviors following trauma exposure. Respondents rate how often they engaged in each behavior in the past 30 days on a 5-point scale ranging from 0—never to 4—very frequently. Because responses were entered into an online survey portal, responses would not be immediately visible to interviewers. Accordingly, we excluded an item assessing suicide preparatory behaviors and attempts. Two additional items assessing functional impairment and trauma-relatedness were also excluded from the analyses. We summed the remaining 13 reckless and self-destructive behavior items to generate a total score; internal consistency for this measure was adequate (Time 1 α = .73; Time 2 α = .76).

We administered a brief user experience questionnaire immediately following the CAT/CAD-PTSD, PCL-5, PC-PTSD-5, and ITQ to evaluate the subjective experience of while completing each measure. We provide a copy of this measure in the Supplemental Appendix. Our intent in administering this measure was to assess accessibility and user satisfaction.

Other Criteria

The Brief Inventory of Psychosocial Functioning (Kleiman et al., 2020; Marx et al., 2019) is a seven-item measure of functioning across multiple domains. Respondents rate the degree to which they had trouble in their relationships (i.e., romantic partners, children, family, and friends), work, education, and activities of daily living over the past 30 days on a 7-point scale ranging from 0—not at all to 6—very much. Participants can respond N/A to items that are not relevant to them. The Brief Inventory of Psychosocial Functioning total score is generated by summing the completed items and dividing that score by the maximum possible score as determined by number of items completed; internal consistency for this measure was high (Time 1 α = .90; Time 2 α = .83).

The Alcohol Use Disorder Identification Test (Saunders et al., 1993) is a 10-item measure of alcohol use disorder symptoms. We summed item scores to a total score reflecting overall alcohol use disorder symptoms; internal consistency was high (Time 1 α = .91; Time 2 α = .92).

The Personality Disorder Inventory for the DSM-5, short-form (Maples et al., 2015) is an abbreviated questionnaire measure of personality disorder characteristics. Respondents rate each item on a 4-point scale ranging from 0—very false or often false to 3—very true or often true. Only the 12-item antagonism scale was used for this study. We calculated the mean of these items to generate a total score; internal consistency was high for this scale (Time 1 α = .85; Time 2 α = .90).

The 21-item Depression, Anxiety, and Stress Scale (Lovibond & Lovibond, 1995) is a questionnaire measure of unipolar depressive symptoms, autonomic arousal anxiety symptoms, and generalized anxiety/stress symptoms. Respondents rate the degree to which they experienced each symptom during the past week on a 4-point scale based on the following response options: (0) did not apply to me at all, (1) applied to me to some degree or some of the time, (2) applied to me a to a considerable degree or a good part of the time, and (3) applied to me very much or most of the time. In an administrative error, an additional 5th response option, a great deal, was also included. This item was also scored as (3).² Depression, anxiety, and stress scale scores are created by summing the corresponding items. Internal consistency was adequate or higher for all scales: anxiety Time 1 α = .81, Time 2 α = .83; depression Time 1 α = .88, Time 2 α = .87; stress Time 1 α = .87, Time 2 α = .86.

The Test of Premorbid Functioning (Wechsler, 2009) is a measure of reading ability that can be used to estimate an intelligence quotient (IQ) level. Respondents read a list of 70 words that have atypical grapheme to phoneme translations. We used established norm values to calculate a standard score that functions as an estimate of fullscale IQ.

Data Analysis

Diagnostic utility of the CAD-PTSD provisional diagnostic classification was examined relative to CAPS-5 diagnostic status. We used both the traditional DSM-5 CAPS-5 diagnostic algorithm (SEV2) and diagnosis plus a CAPS-5 total score of ≥26 scoring methods (SEV2/26; Weathers et al., 2018). We evaluated diagnostic classification using Kraemer’s (1992) quality indices, which are weighted κ coefficients that correct for chance levels of performance: quality index of sensitivity, κ(1, 0); proportion of correct positive diagnoses, quality index of specificity, κ(0, 0); proportion of correct negative diagnoses, and quality index of efficiency, κ(.5, 0); correct classification rate. We calculated these metrics using functions provided in DAG_STAT (Mackinnon, 2000) and evaluated them using established guidelines (Cicchetti, 1994): values <.40 are poor, values ≥.40 are fair, values ≥.60 are good, and values ≥.75 are excellent. Raw sensitivity, specificity, and efficiency values, as well as negative predictive value (probability of true negative diagnosis given negative test), and positive predictive value (probability of true positive diagnosis given positive test) were also calculated. Diagnostic utility of the CAD-PTSD was contrasted with accepted cutoff scores on existing PTSD screening tools. Although the CAD-PTSD is presented as the primary diagnostic screening tool, Brenner et al. (2021) report that the CAT-PTSD’s moderate severity threshold was selected to maximize sensitivity and specificity for the CAPS-5 diagnosis. We therefore supplemented our evaluation of the CAD-PTSD’s diagnostic utility with an analysis of the CAT-PTSD using a moderate severity (i.e., ≥47) cut score.

We examined construct validity of the CAT-PTSD severity score by examining zero-order correlations with all included PTSD measures as well as criteria ranging from theoretically related (e.g., depression, functioning) to unrelated (e.g., IQ). We completed tests of dependent (i.e., correlated) correlation differences (Meng et al., 1992) to compare criterion associations both across measures (e.g., to determine whether one measure was more strongly associated with the CAPS-5 than the others) and within measures (e.g., to determine whether the CAT-PTSD’s strongest association was with the CAPS-5).

Last, we examined test–retest reliability for the CAT-PTSD severity score by calculating both the Pearson correlation between time points and the single-rater two-way mixed effects absolute agreement intraclass correlation coefficient (ICC; McGraw & Wong, 1996). We estimated the test–retest reliability of the CAD-PTSD provisional diagnostic status using Cohen’s kappa (κ).

Results

Of the full sample, 103 participants (65.60%) met DSM-5 PTSD diagnostic criteria per the CAPS-5 SEV2 scoring rule, and 92 (58.60%) met the CAPS-5 SEV2/26 rule. The most frequently identified index trauma category was combat (n = 43; 27.39%), followed by sexual assault (n = 27; 17.20%). Of the full sample, 24 (15.29%) met diagnostic criteria for the DSM-5 PTSD dissociative subtype. Interrater reliability analyses revealed excellent reliability for both diagnosis (SEV2 κ = .87; SEV2/26 κ = .94) and PTSD severity score (ICC = .97).

Measure Characteristics

We provide a summary of CAT/CAD-PTSD characteristics in Table 1. The mean number of total items administered for both PTSD severity and provisional diagnostic status was 10.77, and the median amount of time needed to complete the measures was 131 s. This was comparable to the median administration time for the PCL-5 (138 s) and significantly slower than the median administration time of the PC-PTSD-5 (33 s). We present a table of responses to the user experience questionnaire by PTSD measure in Supplemental Table 2; user-rated acceptability did not differ by measure.

Table 1.

Descriptive Statistics of CAT/CAD-PTSD Data at Time 1

Measure	M/n	SD/%
CAD-PTSD
Diagnosis
Negative	94	59.87%
Positive	63	40.12%
Probability	.43	.26
Time (s)	82.73	37.35
Median time	75
Number of items	5.97	0.16
CAT-PTSD
Category
None	20	12.74%
Mild	23	14.65%
Moderate	46	29.30%
Severe	68	43.31%
Severity	55.87	18.53
Precision	5.61	0.58
Time (s)	58.32	60.06
Median time	49
Number of items	4.82	2.50
CAT/CAD total
Time (s)	141.06	84.24
Median time	131
Number of items	10.77	2.50
PCL-5
Time (s)	150.01	71.97
Median time	137.97
Number of items	20
PC-PTSD-5
Time (s)	41.45	36.36
Median time	33.35
Number of items	5

Note. M/n column contains M and n values, except where median time is identified. SD/% column provides SDs and % values for sample M and n values, respectively. CAD = computerized adaptive diagnostic screen; CAT = computerized adaptive severity test; PC-PTSD-5 = Primary Care PTSD Screen for DSM-5; PCL-5 = PTSD Checklist for DSM-5.

Diagnostic Utility

We present results of diagnostic classification analyses in Table 2. The CAD-PTSD’s diagnostic efficiency was poor for both the CAPS-5 SEV2, κ(.5, 0) = .30 and CAPS-5 SEV2/26, κ(.5, 0) = .40 diagnostic criteria. The CAD-PTSD had fair, SEV2 κ(0, 0) = .58 to good, SEV2/26 κ(0, 0) = .62 levels of diagnostic specificity, but was well below the other measures in its sensitivity, SEV2 κ(1, 0) = .21; SEV2/26 κ(1, 0) = .29. In contrast to the CAD-PTSD, the CAT-PTSD ≥ 47 classification had fair diagnostic efficiency for both criteria, SEV2 κ(.5, 0) = .54; SEV2/26 κ(.5, 0) = .53.

Table 2.

Diagnostic Accuracy by Measure and CAPS-5 Scoring Algorithm

Measure (cutoff)	Criterion	Sensitivity	Specificity	Efficiency	NPV	PPV	κ(1, 0)	κ(0, 0)	κ(.5, 0)
	CAPS-5 SEV2
CAD-PTSD		.52	.83	.63	.48	.86	.21	.58	.30
CAT-PTSD (≥47)		.90	.61	.80	.77	.82	.65	.46	.54
PCL-5 (≥33)		.89	.65	.81	.76	.83	.64	.50	.56
ITQ PTSD		.77	.76	.76	.63	.86	.44	.59	.50
PC-PTSD-5 (≥4)		.84	.67	.78	.69	.83	.53	.50	.52
	CAPS-5 SEV2/26
CAD-PTSD		.58	.85	.69	.59	.84	.29	.62	.40
CAT-PTSD (≥47)		.93	.57	.78	.86	.75	.76	.41	.53
PCL-5 (≥33)		.96	.65	.83	.91	.79	.85	.50	.63
ITQ PTSD		.85	.78	.82	.78	.85	.63	.63	.63
PC-PTSD-5 (≥4)		.90	.66	.80	.83	.79	.70	.49	.58
	Test-retest (κ)
CAD-PTSD	.25
CAT-PTSD (≥47)	.66
PCL-5 (≥33)	.67
ITQ PTSD	.81
PC-PTSD-5 (≥4)	.69

Note. SEV2 = traditional DSM-5 CAPS-5 scoring algorithm; SEV2/26 = modified CAPS-5 scoring algorithm requiring both traditional DSM-5 PTSD diagnostic criteria and a total CAPS-5 severity score ≥ 26; Sens. = sensitivity; Spec. = specificity; Eff. = efficiency; NPV = negative predictive value; PPV = positive predictive value; CAPS = Clinician-Administered PTSD Scale; International Trauma Questionnaire; CAD = computerized adaptive diagnostic screen; CAT = computerized adaptive severity test; PC-PTSD-5 = Primary Care PTSD Screen for DSM-5; PCL-5 = PTSD Checklist for DSM-5; PTSD = posttraumatic stress disorder; DSM-5 = Diagnostic and Statistical Manual of Mental Disorders, fifth edition; ITQ = International Trauma Questionnaire; κ(1, 0) = quality index of sensitivity; κ(0, 0) = quality index of specificity; κ(.5, 0) = quality index of efficiency; κ values <.40 are poor, values ≥0.40 are fair, values ≥.60 are good, values ≥.75 are excellent.

The PC-PTSD-5 ≥ 4 classification had fair diagnostic efficiency for both criteria, SEV2 κ(.5, 0) = .52; SEV2/26 κ(.5, 0) = .58. Both the PCL-5 ≥ 33 classification and ITQ ICD-11 classification had fair diagnostic efficiency for the CAPS-5 SEV2 criterion, PCL-5 κ(.5, 0) = .56; ITQ κ(.5, 0) = .50 and good diagnostic efficiency for the CAPS-5 SEV2/26 criterion, PCL-5 κ(.5, 0) = .63; ITQ κ(.5, 0) = .63.

Construct Validity

We present correlation coefficients among trauma-related measures in Table 3. Table 4 contains correlations with select criteria (ranging from theoretically related to unrelated) and tests of dependent correlation differences. All PTSD severity scores had theoretically consistent patterns of association. The CAT-PTSD, PCL-5, PC-PTSD-5, and ITQ-PTSD were each most strongly correlated with the CAPS-5 severity score and did not have significant correlations with measures of antagonism or intelligence as measured by the Test of Premorbid Functioning. The CAPS-5 and Brief Inventory of Psychosocial Functioning were more strongly associated with the PCL-5 than with any of the other PTSD measures. The CAT-PTSD, PCL-5, PC-PTSD-5, and ITQ-PTSD were all similarly associated with measures of trauma exposure, risky behaviors, depression, anxiety, stress, and problematic alcohol use.

Table 3.

Descriptive Statistics, Correlations, and Test–Retest ICC Among Trauma-Related Measures

Measure	M	SD	ICC	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17
Time 1
1. CAPS-5	28.77	12.71		—
2. CAT-PTSD	55.87	18.53		.78	—
3. PCL-5	41.89	19.27		.83	.84	—
4. PC-PTSD-5	3.75	1.42		.76	.75	.76	—
5. ITQ PTSD	14.18	6.30		.74	.81	.87	.76	—
6. ITQ C-PTSD	26.81	11.53		.81	.82	.90	.76	.90	—
7. PRBQ	5.47	5.45		.45	.39	.39	.31	.29	.38	—
8. B-IPF	47.90	29.70		.64	.59	.67	.55	.59	.70	.38	—
9. LEC-5	6.63	2.80		.33	.26	.27	.29	.26	.28	.22	.29	—
Time 2
10. CAPS-5	29.90	11.25	.93	.92	.69	.64	.69	.70	.71	.38	.53	.41	—
11. CAT-PTSD	53.87	17.28	.82	.81	.80	.79	.67	.82	.82	.32	.70	.44	.77	—
12. PCL-5	38.47	18.46	.87	.87	.81	.87	.69	.84	.88	.40	.68	.45	.82	.86	—
13. PC-PTSD-5	3.81	1.43	.72	.82	.62	.75	.72	.73	.71	.30	.48	.37	.75	.73	.78	—
14. ITQ PTSD	12.91	6.26	.92	.83	.84	.83	.71	.92	.89	.30	.63	.42	.80	.87	.91	.75	—
15. ITQ C-PTSD	23.92	11.66	.92	.79	.64	.75	.61	.66	.92	.34	.62	.32	.66	.68	.82	.64	.72	—
16. PRBQ	5.60	5.64	.83	.46	.32	.35	.23	.32	.41	.84	.17	.40	.41	.29	.46	.35	.38	.45	—
17. B-IPF	40.98	24.03	.78	.69	.59	.59	.53	.58	.64	.40	.75	.55	.61	.69	.65	.56	.61	.58	.44	—

Note. CAPS-5 = Clinician Administered PTSD Scale for the DSM-5; CAT-PTSD = computerized adaptive PTSD severity scores; PCL-5 = PTSD Checklist for the DSM-5; PC-PTSD-5 = Primary Care PTSD Screen for the DSM-5; ITQ PTSD = International Trauma Questionnaire for PTSD; ITQ C-PTSD = International Trauma Questionnaire for Complex PTSD; PRBQ = Posttrauma Risky Behaviors Questionnaire; B-IPF = Brief Inventory of Psychosocial Functioning; ICC = intraclass correlation coefficient; LEC-5 = Life Events Checklist for the DSM-5; PTSD = posttraumatic stress disorder; DSM-5 = Diagnostic and Statistical Manual of Mental Disorders, fifth edition; ICC column provides the measures test–retest reliability as measured by the single-rater two-way mixed effects absolute agreement ICC. All Time 1 r ≥ .16 and Time 2 r ≥ .29 are statistically significant at p < .05.

Table 4.

Correlations Between PTSD Screeners and Select Criteria

Criterion	CAT-PTSD	PCL-5	PC-PTSD-5	ITQ-PTSD
CAPS-5	.78*a	.83*b	.76*a	.74*a
LEC-5	.26*	.27*	.29*	.26*
B-IPF	.59*a	.67*b	.55*a	.59*a
CDS-U/D	.57*a	.59*a	.45*b	.53*
DASS anxiety	.64*	.65*	.61*	.61*
DASS depression	.59*	.62*a	.56*	.50*b
DASS stress	.60*	.66*	.59*	.62*
AUDIT	.28*	.29*	.19*	.29*
PID-5 antagonism	−.05	−.02	−.01	−.09
TOPF	.01	.02	.07	−.04

Note. CAT-PTSD = Computerized Adaptive PTSD severity scores; PCL-5 = PTSD Checklist for the DSM-5; PC-PTSD-5 = Primary Care PTSD Screen for the DSM-5; ITQ PTSD = International Trauma Questionnaire for PTSD; CAPS-5 = Clinician Administered PTSD Scale for the DSM-5; LEC-5 = Life Events Checklist for the DSM-5; B-IPF = Brief Inventory of Psychosocial Functioning; CDS-U/D = Cambridge Depersonalization Scale, Unreality/Detachment Subscale; DASS = Depression, Anxiety, and Stress Scale; AUDIT = Alcohol Use Disorder Identification Test; PID-5 = Personality Disorder Inventory for DSM-5, Short Form; TOPF = Test of Premorbid Functioning, Standard Score; PTSD = posttraumatic stress disorder; DSM-5 = Diagnostic and Statistical Manual of Mental Disorders, fifth edition. Within each row, correlations with different subscripts are significantly different in magnitude. Within each column, each of the PTSD screeners had a correlation with the CAPS-5 that was significantly greater than all other criteria at p < .05.

^*p < .05.

Test–Retest Reliability

Test–retest reliability was acceptable for all assessed PTSD severity screens. The PC-PTSD-5 (r = .72; ICC = .72) had adequate reliability, the CAT-PTSD (r = .80; ICC = .82) and PCL-5 (r = .87; ICC = .87) had good reliability, and the ITQ-PTSD (r = .92; ICC = .92) had excellent reliability across the test–retest interval.

For the CAD-PTSD diagnostic classification, test–retest reliability was poor (κ = .25). This contrasts with the diagnostic classifications provided by the CAT-PTSD, PCL-5, ITQ, and PC-PTSD-5, which had test–retest reliabilities (κ) ranging from good to excellent. Reviewing the CAD-PTSD classifications revealed that classification was consistent across time for 35 participants (66.04%; negative on both occasions for 26 participants, positive on both occasions for 9). CAD-PTSD diagnostic classification transitioned from positive to negative for 11 participants (20.75%) and from negative to positive for 7 participants (13.21%).

Discussion

Brenner et al. (2021) introduced and provided initial validation of the CAT/CAD-PTSD. The CAT/CAD-PTSD measures were presented as complementary PTSD screeners capable of facilitating diagnostic screening and severity estimates within routine clinical care. We completed an independent replication and extension by examining a broader segment of the CAT/CAD-PTSD measures’ nomological network, their test–retest reliability, and incremental value beyond existing PTSD screening options. Our analyses found that the CAT-PTSD symptom severity estimate is comparable to that of other self-report symptom screeners, but the CAD-PTSD diagnostic screen is far less sensitive to the detection of positive PTSD diagnoses than other available options.

The CAT-PTSD had good construct validity. Test–retest reliability was good and severity scores were strongly correlated with the PCL-5, CAPS-5 total scores, and measures of theoretically related constructs (e.g., depression, functioning); CAT-PTSD scores were not correlated with measures of theoretically unrelated constructs (e.g., IQ). However, the CAT-PTSD did not outperform any of the traditional (i.e., nonadaptive) PTSD self-report measures in its associations with other criteria. The CAT-PTSD severity score was strongly correlated with the CAPS-5 total score, but at a level comparable to that of the PC-PTSD-5 and the ITQ-PTSD measure, and less than that of the PCL-5.

The CAD-PTSD’s test–retest reliability and diagnostic efficiency was poor, particularly in relation to the performance of the other measures. Although the CAD-PTSD’s specificity was good, it did not exceed that of other available measures (i.e., ITQ), and its diagnostic sensitivity was far below acceptable levels. In this sample, the CAD-PTSD was prone to misclassifications in the form of false negatives. This is particularly problematic for a measure intended for routine screening, where high sensitivity is typically emphasized to minimize the risk of missing individuals in need of further evaluation.

The combined CAT/CAD-PTSD measures did require fewer items than the PCL-5, but the total length of time it took to complete the adaptive tests was comparable to that of the other self-report measures. It is difficult to say why the shorter length of the CAT/CAD-PTSD measures was not reflected in comparably shorter administration times. On average, CAT/CAD-PTSD measures required 13 s per item. That is in contrast to 8.3 s per item for the PC-PTSD-5 and 7.5 s per item for the PCL-5. It is possible that the use of varying item stems and response options within the measures required greater processing time for the veterans. The single-item administration necessitated by computerized adaptive measures may also be relevant. Depending on how the CAT/CAD-PTSD administration site’s internal timer worked, it is possible that internet connection speed and associated page loading times played a role. Regardless, the PCL-5 took most participants just over 2 min to complete, which stands in contrast to suggestions that existing questionnaire measures represent substantial respondent burden.

Although the CAT/CAD-PTSD measures draw from the same pool of items, their underlying models are fundamentally different, which may account for why the CAD-PTSD was outperformed by all other measures, including its CAT counterpart. The CAT-PTSD was developed using item response theory and adaptive administration involves selecting the most informative item for the respondent’s estimated level of severity (Brenner et al., 2021). This provides an efficient estimate of the participant’s standing on the latent construct captured by the item pool (i.e., PTSD). Item response theory item parameters are intended to be population invariant,³ and our results suggest that the CAT-PTSD measure provides a reliable and valid estimate of the latent PTSD construct. The CAD-PTSD model differs in its development and adaptive item selection process (Brenner et al., 2021). It was developed using a supervised machine-learning model trained to predict standing on a particular external criterion. The primary criterion used in this study (i.e., PTSD diagnostic status as determined by the CAPS-5) is consistent with that used by Brenner et al. (2021), but the CAD-PTSD was far less effective in the current sample. It is possible that sample differences in the rates of trauma exposure and positive PTSD diagnoses contributed to this change in performance. In the validation sample, only 28.4% received a current PTSD diagnosis on the CAPS-5 (Brenner et al., 2021). In our sample, 66% met PTSD diagnostic criteria based on CAPS-5 SEV2 and 59% met criteria based CAPS SEV2/26. The low base rate of the training data may have resulted in an algorithm calibrated to assume a negative PTSD diagnosis in most cases, or whenever ambiguity is present in the response pattern. This possibility is consistent with the observation that 84.5% of the CAD-PTSD’s misclassifications were in the form of false negatives (i.e., a negative CAD-PTSD screen when the CAPS-5 assigned a positive diagnosis). These results suggest that the CAD-PTSD may underestimate the frequency of PTSD diagnoses in any population with a base rate surpassing that observed by Brenner et al. (2021).

Although we did not have access to the full item pool, we were able to manually explore some of the item content as each participant’s CAT/CAD output included the administered items. There were cases in which items assessed particular types of traumatic events (e.g., seeing homes or communities destroyed, witnessing someone being injured or killed), and symptoms specific to particular forms of trauma exposure (e.g., survivor guilt, experiencing bad feelings during sex due to the trauma). Such items may have reduced measurement performance as it is probable that their relationship with the underlying PTSD construct and associated diagnostic status is dependent on the individual trauma experiences of the respondent (i.e., differential item functioning).

Collectively, results from this study indicate that the CAT-PTSD severity score is a reliable, valid indicator of PTSD symptom severity, but it does not appear to offer incremental value beyond existing, brief, public-domain, nonadaptive questionnaires. Regarding the diagnostic classification offered by the CAD-PTSD, poor diagnostic accuracy and test–retest reliability raise serious validity concerns.

To our knowledge, this is the only psychometric evaluation of the CAT/CAD-PTSD outside of the initial validation study, and the differences between our findings and those of Brenner et al. (2021) highlight the importance of replication within the measurement development process. Measurement validation is an ongoing and iterative process and additional work is needed to identify the specific factors that may account for the inconsistencies across these two studies.

Constraints on Generality

The sample consisted entirely of veterans enrolled in the VHA system. We attempted to maximize the representativeness of the sample by using minimal exclusion criteria and recruiting from three VHA medical centers across the country. However, results may not generalize to other groups. Additionally, the sample was not large enough to examine differences in psychometric performance across subgroups separated by gender, race, or ethnicity.

Conclusion

In contrast to the initial validation study, our findings indicate that the CAT/CAD-PTSD measures offer no notable advantages over other established measures. The CAD-PTSD showed inadequate diagnostic reliability and accuracy, calling into question its utility in clinical settings. We would particularly discourage the model of conditional testing (i.e., administering the CAT-PTSD only in cases of a positive CAD-PTSD screen) proposed by Brenner et al. (2021) given the CAD-PTSD’s poor sensitivity in this sample. Although the CAT-PTSD severity score demonstrated good psychometric properties, it did not surpass traditional measures such as the PCL-5 and PC-PTSD-5 in efficiency, diagnostic accuracy, and associations with relevant criteria.

Given PTSD’s significant impact on quality of life and the critical importance of timely intervention, brief and accurate measures of the construct are vital. Unfortunately, the adaptive measures evaluated in this study did not constitute a step forward in this regard. Our findings suggest that the PC-PTSD-5 represents a briefer and more accurate diagnostic screening tool, and the PCL-5 is more strongly associated with clinician-rated PTSD symptom severity. Both the PC-PTSD-5 and PCL-5 are freely available to the public.⁴ Researchers and clinicians are encouraged to continue utilizing these existing, well-validated, traditional measures while exploring innovations that may better address current gaps in PTSD assessment.

Supplementary Material

https://doi.org/10.1037/pas0001429.supp

Public Significance Statement.

We examine recently developed proprietary PTSD severity and diagnostic screeners to determine whether they are superior to freely available measures. This work suggests that neither the severity screener (PTSD computerized adaptive severity test) nor the diagnostic screener (PTSD computerized adaptive diagnostic screen) offers incremental value over measures that are available in the public domain. The PTSD computerized adaptive diagnostic screen has significant limitations as a diagnostic tool.