Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2024 Nov 1.
Published in final edited form as: Psychol Trauma. 2022 May 19;15(8):1238–1247. doi: 10.1037/tra0001235

Online Crowdsourcing to Study Trauma and Mental Health Symptoms in Military Populations: A Case for Amazon’s Mechanical Turk (MTurk) Platform

Shannon R Forkus 1, Ateka A Contractor 2, Svetlana Goncharenko 1, Silvi C Goldstein 1, Nicole H Weiss 1
PMCID: PMC10312215  NIHMSID: NIHMS1894554  PMID: 35587433

Abstract

Amazon’s Mechanical Turk (MTurk) appears to be a reliable resource for studying clinical populations and accessing hard-to-reach populations. Recent research suggests that MTurk may also be a viable option for military recruitment.

Objective:

The goal of the current study was to examine the utility of collecting clinical data on military samples recruited via MTurk.

Method:

Participants were 535 military veterans (Mage = 37.45; 71.8% men; 69.5% White) who completed measures assessing trauma and mental health.

Results:

Findings indicate that rates of military traumas and mental health diagnoses were higher than published comparisons; PTSD and depression symptoms were found to be higher than values found in a nationally representative sample, lower than a treatment-seeking sample, and comparable to a MTurk-recruited military sample. Alcohol misuse was found to be higher than both nationally representative and treatment-seeking samples. Psychometric analyses indicated support for convergent validity of measures and CFA results demonstrated that empirically-supported factor models of PTSD were replicated in the current sample; the Hybrid Model demonstrated the best fit.

Conclusions:

Our findings support the utility of MTurk for collecting clinical data on military samples. Increasing access to and recruitment of military samples is important for advancing the field of military psychology.

Keywords: MTurk; military; psychometrics, trauma; posttraumatic stress disorder; alcohol use; depression

Military personnel and veterans report high rates of physical and psychiatric morbidity (Betancourt et al., 2020; Thomas et al., 2017), greater suicide mortality (Affairs, 2019; Hoffmire et al., 2015), unique occupational stressors (Campbell & Nobel, 2009), and reintegration challenges (Sayer et al., 2014; Wilcox et al., 2015). Thus, research with this population is of particular clinical and public health relevance. This being said, military samples can be difficult to recruit (Braun et al., 2015; Bush et al., 2013). By identifying a recruitment option that is cost and time-efficient, and capable of producing reliable data, research in military samples can be expanded, enhancing the external validity of reported findings in military populations and advancing the field as a whole.

One such recruitment option is Amazon’s Mechanical Turk (MTurk), which is a crowdsourcing website that facilitates online data collection by allowing researchers to recruit a diverse sample that closely represents the general population (Buhrmester et al., 2016; Paolacci et al., 2010). Procedurally, the MTurk platform allows researchers to post a Human Intelligence Task (HIT), such as an online survey, which is available to qualified participants. Broadly, MTurk has several advantages. First, MTurk allows for the rapid recruitment of a large sample at relatively low costs, offering a recruitment method that is both time and cost-efficient and capable of generating high-quality data. Critically, participants’ quality of work does not seem to be affected by compensation rates, as equivalent results have been produced across varying incentive rates (Buhrmester et al., 2016). Second, psychometric investigations of measures administered via MTurk have indicated high internal reliability (Buhrmester et al., 2016), testretest reliability, and concurrent and convergent validity (Chandler & Shapiro, 2016; Shapiro et al., 2013), as well as measurement equivalence across samples (Feitosa et al., 2015). Lastly, patterns of attentive and honest responding have been found to be comparable to other data collection platforms (Hauser & Schwarz, 2016; Paolacci et al., 2010). Overall, MTurk is recognized as a useful recruitment option for online data collection.

There is evidence to suggest that MTurk may be an optimal platform to study clinical populations (Shapiro et al., 2013) and hard-to-reach populations such as military samples (e.g., Duffy et al., 2015; Forkus et al., 2019; Lancaster & Miller, 2019; Morgan & Desmarais, 2017). Military individuals experience unique stressors and traumatic events (e.g., combat, moral injury events, military sexual trauma; Fulton et al., 2015; Wilson, 2018; Wisco et al., 2017) and are at a disproportionate risk for related mental health consequences (Fulton et al., 2015; Seal et al., 2009), including posttraumatic stress disorder (PTSD; Fulton et al., 2015). In this regard, studies have found rates of mental health problems in MTurk-recruited samples to be comparable to the general population (Shapiro et al., 2013); others have found these rates to be elevated (Arditte et al., 2016). Relatedly, recent studies have demonstrated the utility of studying trauma sequelae among civilian samples via MTurk (Engle et al., 2019; van Stolk-Cooke et al., 2018).

Notably, there is also strong evidence for content and criterion-related validity of measures assessing mental health outcomes administered via MTurk, reflected by strong associations between psychopathology and relevant demographic/clinical correlates (Arditte et al., 2016; Shapiro et al., 2013; van Stolk-Cooke et al., 2018). Therefore, relations among data obtained from clinical measures on MTurk align with empirical/theoretical expectations. Further, the reliability of MTurk responses has been evinced through retesting procedures, as well as by incorporating measures to detect malingering, which found low rates of symptom fabrication (Shapiro et al., 2013). These findings emphasize the utility of MTurk for studying clinical populations, as well as the importance of comparing clinical characteristics across recruitment platforms to identify potential differences in presentation/prevalence of mental health disorders.

Given the aforementioned literature and the fact that MTurk appears to be a viable option for military recruitment (Forkus et al., 2019; Lancaster & Miller, 2019), it is surprising that no empirical research has examined the utility of collecting clinical data on military samples recruited via MTurk, a population that is particularly vulnerable to barriers associated with disclosing trauma experiences and related symptoms (Lorber & Garcia, 2010). Specifically, no studies have directly compared clinical data collected from a MTurk-recruited military sample to military samples recruited from other sources. Such information may inform and guide the use of MTurk to research mental health among military samples, as well as address barriers to recruitment/retention in research studies (e.g., avoiding face to face symptom disclosure; van Stolk-Cooke et al., 2018). Thus, drawing from previous studies (Engle et al., 2019; van StolkCooke et al., 2018), we examined MTurk’s utility of assessing trauma and mental health symptoms among military samples in the following ways. For Aim 1, we statistically compared prevalence rates and mean scores of mental health outcomes (i.e., PTSD, depression, and alcohol use disorder [AUD]) and prevalence rates of military traumas (i.e., combat, MST, moral injury events) obtained via a MTurk-recruited military sample vs. published studies using other recruitments sources. We expected to find that the MTurk-recruited military sample would endorse higher rates of trauma and mental health outcomes relative to comparison samples. For Aim 2, we examined psychometrics (specifically convergent validity) of measures administered to MTurk-recruited participants by testing whether assessed constructs related to one another in theoretically-meaningful and empirically-expected ways. Relationships between trauma and mental health symptoms are hypothesized to be positive and significant consistent with past research; and therefore, indicative of convergent validity. Relatedly, we examined the structural validity of the PTSD Checklist for DSM-5 (PCL-5) by evaluating whether the optimal factor structure is consistent with empirically validated models of PTSD (especially the seven-factor Hybrid Model; Wortmann et al., 2016). Specifically, there is continued debate on which model best represents PTSD, with studies showing stronger support for alternative models of PTSD. Previous research highlights six competing models1, with performance varying by specific contexts and populations. Each of these models were examined for the PCL-5 (see Supplemental Table 1 for the item mapping for these models).

Methods

Procedure and Participants

Participants were recruited via MTurk. The HIT asked participants to respond to a series of questions about their military experiences, emotions, and behaviors. The HIT was posted in August 2019. Participants could only access the HIT if they were located in the U.S and had >95% approval rating. Additional inclusion criteria were: 1) at least 18 years old, 2) working knowledge of the English language, and 3) a veteran of the U.S. military. Participants who met eligibility criteria provided informed consent and completed the survey on the Qualtrics data collection platform. To improve data quality, we incorporated four validity checks to assess attention and comprehension (Meade & Craig, 2012; Oppenheimer et al., 2009). Participants were also required to correctly respond to two validity questions about military service that are not typically common knowledge to civilians (Lynn & Morgan, 2016). Participants who failed to correctly respond to any one of the six validity checks were excluded. Participants were compensated $2.00 for their participation. All procedures were approved by the Institutional Review Board at University of Rhode Island.

In total, 2,644 individuals accessed the survey; 997 were excluded for not meeting inclusionary criteria, 899 were excluded for failing attention/comprehension validity questions, 134 were excluded for failing military validity questions, and 79 were excluded for answering the questionnaire more than once. From this effective sample of 535 individuals, we created a subsample of 465 trauma-exposed (i.e., excluded 70 participants for not endorsing a traumatic experience [see Measures]). Reducing the sample to those with trauma exposure is a required step when referencing PTSD, which is measured in relation to an index trauma (APA, 2013). See Table 1 for descriptive information for both the full and trauma-exposed samples.

Table 1.

Sample Characteristics

Full Sample (n = 535) Trauma-Exposed Sample (n = 465)
Variables M (SD) M (SD)
Age 37.45 (11.24) 38.00 (11.45)
n (%) n (%)
Gender
 Woman
 Man
 Woman to Man Transgender
148 (27.9%)
381 (71.8%)
2 (0.4%)
131 (28.2%)
332 (71.6%)
1 (0.2%)
Race
 White
 Black
 Asian
 American Indian/Alaska Native
 Native Hawaiian/Other Pacific Islander
 Not Listed
372 (69.5%)
118 (22.1%)
28 (5.2%)
22 (4.1%)
6 (1.1%)
5 (0.9%)
323 (69.5%)
108 (23.2%)
26 (5.6%)
19 (4.1%)
5 (1.1%)
4 (0.9%)
Ethnicity
 Hispanic or Latino/a
 Not Hispanic or Latino/a		  
126 (24.2%)
394 (75.8%)		  
112 (24.6%)
344 (75.4%)
Employment Status
 Employed Full-time
 Employed Part-time
 Not in Labor Force (Student, Homemaker)
 Unemployed
451 (85.7%)
48 (9.1%)
15 (2.9%)
12 (2.3%)
394 (85.7%)
40 (8.7%)
14 (3.0%)
12 (2.6%)
Highest Level of Education
 < 12 years
  12 years
 > 12 years
47 (9.1)
55 (10.5%)
421 (80.4%)
32 (7.0%)
49 (10.7%)
376 (82.3%)
Branch of Service
 Army
 Navy
 Air Force
 Marines
 Coast Guard
342 (63.9%)
51 (9.6%)
102 (19.0%)
33 (6.1%)
6 (1.1%)
301 (64.7%)
42 (9.0%)
89 (19.2%)
29 (6.2%)
6 (1.3%)
Pay Grade
 Enlisted
 Officer
 Warrant Officer
441 (87.2%)
60 (11.8%)
5 (1.0%)
397 (90.2%)
39 (8.9%)
4 (0.9%
Number of Deployments
 None
  One
 Two
 Three or more
116 (22.7%)
139 (27.1%)
116 (22.7%)
141 (27.5%)
98 (21.8%)
127 (28.2%)
105 (23.3%)
120 (26.7%)
Open in a new tab

Note. Participants were able to report more than one option for race and branch of service. All reported percentages are valid percentages to account for missing data.

Measures

Military Traumatic Events

Life Events Checklist for DSM-5 (LEC-5; Weathers et al., 2013).

The LEC-5 is a 17-item self-report measure of lifetime traumatic exposure. Participants indicated their exposure to each traumatic event on a 6-point scale: happened to me, witnessed it, learned about it, part of my job, not sure, and does not apply. Traumatic exposure, consistent with the DSM-5 Criterion A (APA, 2013), involved the endorsement of any of the first four response options.

Combat Involvement.

Combat involvement was assessed by asking participants to indicate yes or no to one question (“Were you involved in combat operations?”).

Military Sexual Trauma (MST).

MST was assessed with two items used extensively in research (Mengeling et al., 2019). These items capture military sexual harassment (“While in the military…Did you ever receive uninvited or unwanted sexual attention, such as touching, cornering, pressure for sexual favors, or verbal remarks?”) and sexual assault (“While in the military…Did someone ever use force or the threat of force to have sex against your will?”). A positive MST screen was indicated by responding “yes” to either question.

Moral Injury Events Scale (MIES; Nash et al., 2013).

The MIES is a 9-item self-report scale measuring exposure to potentially morally injurious experiences, including perceived transgressions committed by self, others, and betrayal. Participants rated each item using a 6-point Likert scale (1 = strongly disagree, 6 = strongly agree). Total scores were computed by summing items on each respective scale, with higher scores indicating greater exposure to events that lead to moral injury from self, others, and betrayal. Dichotomized scores were also calculated to capture whether each of the three MIE types were endorsed, by coding items that indicated either “moderately agree” or “strongly agree” on any of the items that comprise each scale (consistent with Wisco et al., 2017). This scale has good psychometric properties (Bryan et al., 2016; Nash et al., 2013). For the current study, reliability was excellent for transgressions committed by self (α = .93), good for betrayal (α = .89), and adequate for others (α = .78).

Mental Health

PTSD Checklist for DSM-5 (PCL-5; Weathers et al., 2013).

The PCL-5 is a 20-item self-report measure assessing DSM-5 criteria for PTSD in response to the most distressing trauma endorsed on the LEC-5. Participants indicated how bothered they were by symptoms over the past month using a scale ranging from 0 (not at all) to 4 (extremely). Total scores were computed by summing items, with higher scores representing greater PTSD symptoms severity. A cut-off score of ≥ 33 was used to indicate PTSD (Bovin et al., 2016). The PCL-5 has good psychometric properties (Blevins et al., 2015; Bovin et al., 2016; Wortmann et al., 2016). The PCL-5 in the current sample demonstrated excellent reliability (Cronbach’s α = .98).

Patient Health Questionnaire (PHQ-9; Kroenke & Spitzer, 2002).

The PHQ-9 is a 9-item self-report scale assessing depressive symptoms over the last two weeks. Respondents were asked to respond to questions on a 4-point Likert-type scale ranging from 0 (not at all) to 3 (nearly every day). A score was obtained by summing the items; higher scores indicated greater depression symptom severity. A cut-off score of ≥ 10 was used to indicate depression (Kroenke & Spitzer, 2002). The PHQ-9 has good psychometric properties (Kroenke et al., 2001). The PHQ-9 in the current sample demonstrated excellent reliability (Cronbach’s α = .93).

Alcohol Use Disorders Identification Test (AUDIT; Saunders et al., 1993).

The AUDIT is a 10-item self-report scale that assesses alcohol consumption, behaviors, and problems over the past 12 months. Responses are provided on a 5-point scale. A total score was calculated by summing all items, with higher scores indicating greater alcohol misuse. A score of 8 or higher was used to indicate alcohol use disorder (AUD). The AUDIT has good psychometric properties (Searle et al., 2015). The AUDIT in this sample had excellent reliability (Cronbach’s α = .92).

Data Analysis

Preliminary Analyses

Preliminary analyses were conducted in SPSS v.25 to assess normality and examine demographic characteristics of the full and trauma-exposed samples.

Comparison Analyses

To examine the first aim of the current study, we computed rates of military trauma and mental health diagnoses (i.e., PTSD, depression, AUD) using established cut-offs. Next, we conducted chi-square tests to examine presence/absence of statistically significant differences between computed rates of trauma and mental health diagnoses in the current study’s MTurk sample vs. comparison samples (as mentioned above). Finally, we conducted independent t tests to statistically compare the mean severity scores for mental health symptoms across the current study’s MTurk sample vs. comparison samples (as mentioned above).

Psychometric Analyses

For the second aim of the current study referencing convergent validity, we used correlations to examine associations among each of the military traumatic events and mental health symptoms across the full and trauma-exposed samples. Next, we examined the optimal factor structure of the PCL-5 using confirmatory factor analyses (CFAs). We compared the fit of six PTSD factor models: DSM-5 model (APA, 2013), Dysphoria model (Simms et al., 2002), Dysphoric Arousal Model (Elhai et al., 2011), Anhedonia Model (Liu et al., 2014), Externalizing Behaviors Model (Tsai et al., 2015), and Hybrid Model (Armour et al., 2015). Analyses were conducted using the lavaan package in R (Rosseel, 2012), with maximum likelihood estimation; all PCL-5 items were treated as continuous. Several fit indices were used to assess model fit: chi-square (p > .05), Root Mean Square Error of Approximation (RMSEA) <.10 (Browne & Cudeck, 1993), Standardized Root Mean Square Residual (SRMR) <.08 (Hu & Bentler, 1999), Tucker Lewis Index (TLI) >.95, and Comparative Fit Index (CFI) >.95 (Hu & Bentler, 1999). Relative model fit was compared using the chi-square difference test for nested models, whereby difference scores are calculated to determine whether there is a difference between models. For non-nested models, the Bayesian Information Criterion (BIC) values were used to compare models, with lower values indicating better fit (Kass & Raftery, 1995).

Comparison Values

Comparison values were obtained from meta-analyses and representative samples. Rates of combat (38.3%; Wisco et al., 2017), military sexual trauma (7.6%; Klingensmith, Tsai, Mota, Southwick, & Pietrzak, 2014), and moral injury events (10% transgressions by self, 25.5% transgressions by others, and 25.5% perceived betrayals; Wisco et al., 2017) were obtained from a study that analyzed a nationally representative sample of veterans. Rates of PTSD (23%; Fulton et al., 2015) and depression (13.1%; Gadermann et al., 2012) were derived from meta-analyses and past-year AUD (14.8%) was drawn from a nationally representative sample of military veterans (Fuehrlein et al., 2016). To compare mean scores, published studies were chosen based on whether they used the same (or similar) measures (i.e., PCL-5, PHQ-9, and AUDIT), as well as the same timeframes (i.e., 30 days, 2 weeks, and past year, respectively). Comparison studies were chosen from a treatment-seeking (Smigelsky et al., 2019), nationally representative (Erwin et al., 2017), and MTurk-recruited military (Forkus et al., 2019) samples. See Supplemental Table 3 for descriptive information of each comparison sample.

Results

All variables were normally distributed. See Table 1 for demographic information. The mean age of our sample was 37.45 and the majority were non-Hispanic/Latinx, White, men. Most individuals in the sample reported being employed full-time and having >12 years of education. Majority of the sample indicated that they served in the Army, were enlisted, and served on active duty. These demographic characteristics are comparable to military samples (Department of Defense, 2018), including veterans recruited from the VA Healthcare Systems (Bovin et al., 2016) and nationally representative veteran samples characterized by VA users and non-users (Meffert et al., 2019). Notably, our sample had significantly more women compared to the treatment-seeking veteran sample (χ2[1] = 39.12, p < .001) and the nationally representative veteran sample (χ2[1] = 129.69, p < .001); was more racially diverse sample than the nationally representative sample (χ2[1] = 143.16, p < .001) but less racially diverse than the treatment-seeking sample (χ2[1] = 8.38, p =.004); and participants were significantly younger in both the treatment-seeking (t[1001] = 20.46, p < .001) and nationally representative (t[3685] = 40.87, p < .001) samples.

Comparison Analyses

Prevalence rates of traumatic military events and mental health conditions are presented in Table 2 for full and trauma-exposed samples. Our results were compared to estimates found in literature for combat involvement (68.5% vs. 38.3%; Wisco et al., 2017); MST (21.9% vs. 7.6%; Klingensmith et al., 2014); and transgressions by self (39.8% vs. 10.8%), others (51.2% vs. 25.5%), and perceived betrayals (37.8% vs. 25.5%; Wisco et al., 2017). Results indicated that military participants in our sample (vs. other samples) were more likely to endorse combat (χ2[1] = 14.64, p < .001), MST (χ2[1] = 79.16, p < .001), perceived transgressions by self (χ2[1] = 148.56, p < .001), others (χ2[1] = 98.88, p < .001), and betrayals (χ2[1] = 20.21, p < .001).

Table 2.

Prevalence Rates Across the Full and Trauma-Exposed Sample

Full Sample (n = 535)
 Full Trauma-exposed Sample (n = 465)
n (%) n (%)
Mental Health PTSD
Depression
Alcohol		 --
220 (41.1%)
232 (43.4%)		 218 (46.9%)
204 (43.9%)
217 (46.7%)
Traumatic Exposure Combat
MST
Transgression by self
Transgressions by others
Betrayals		 254 (68.5%)
117 (21.9%)
213 (39.8%)
274 (51.2%)
202 (37.8%)		 231 (69.6%)
108 (23.2)
196 (42.2%)
255 (54.8%)
189 (40.6%)
Open in a new tab

Note. PTSD = Posttraumatic stress disorder; MST = Military sexual trauma. All reported percentages are valid percentages to account for missing data.

Using established cutoffs, 46.9% of our trauma-exposed sample reported scores consistent with PTSD. For the full-sample, 41.1% had scores consistent with depression and 43.4% reported scores consistent with AUD. Results indicated that our sample had significantly higher estimates of PTSD (χ2[1] = 149.73, p < .001), depression (χ2[1] = 428.65, p < .001), and AUD (χ2[1] = 691.63, p < .001) compared to prevalence rates found in the literature.

Next, severity scores for each mental health construct were compared to a treatment-seeking, nationally representative, and MTurk-recruited military sample. Findings indicated that the current sample (M = 29.62, SD = 23.32) had lower PTSD severity scores than the treatment-sample (M = 53.26, SD = 13.65, t[675] = 13.72, p < .001) and higher PTSD severity scores than the nationally representative sample (M = 14.74, SD = 14.64, t[1376] = 14.48, p < .001). Similarly, the current sample (M = 8.82, SD = 7.43) had lower depression severity scores than the treatment-seeking sample (M = 16.10, SD = 5.52, t[745] = 12.92, p < .001) and higher depression severity scores than the nationally representative sample (M = 1.28, SD = 2.39, t[1446] = 28.27, p < .001). The current sample alcohol misuse scores (M = 9.23, SD = 9.68) were higher than both the treatment-seeking (M = 4.36, SD = 3.66, t[745] = 7.12, p <.001) and nationally representative samples (M = 4.92, SD = 4.71, t[1446] = 11.36, p < .001). See Table 3.

Table 3.

Severity Score Comparisons

Mental Health Symptom/Measure Sample N M (SD) t Df p Cohen’s d
PTSD
 PCL-5
 PCL-5
 PCL-5
 PCL-5
MTurk-Recruited (Current) Sample
Nationally Representativea
Treatment-Seekingb
MTurk-Recruitedc
465
913
212
203
29.62 (23.32)
14.74 (14.64)
53.26 (13.65)
28.47 (22.27)

14.48
13.72
0.59

1376
675
666

<.001
<.001
.553

0.76
1.24
0.05
Depression
 PHQ-9
 PHQ-4
 PHQ-8
 PHQ-9
MTurk-Recruited (Current) Sample
Nationally Representativea
Treatment-Seekingb
MTurk-Recruitedc
535
913
212
203
8.82 (7.43)
1.28 (2.39)
16.10 (5.52)
9.18 (7.61)

28.27
12.92
0.58

1446
745
736

<.001
<.001
.560

1.37
1.11
0.05
Alcohol Misuse
  AUDIT
 AUDIT
 AUDIT-C
 AUDIT
MTurk-Recruited (Current) Sample
Nationally Representativea
Treatment-Seekingb
MTurk-Recruitedc
535
913
212
203
9.23 (9.68)
4.92 (4.71)
4.36 (3.66)
10.39 (9.14)
10.85
11.36
7.12
1.48
1278
1446
745
736

<.001
<.001
.140

0.57
0.67
0.12
Open in a new tab

Note. PCL-5 = PTSD Checklist for DSM-5; AUDIT = Alcohol Use Disorder Identification Test; AUDIT-C = Alcohol Use Disorder Identification Test – Consumption Scale; PHQ = Patient Health Questionnaire 4 item, 8 item, and 9 item. PHQ-4 measures depression and anxiety. Sensitivity analyses were conducted to modify the PHQ and AUDIT to be consistent with the comparison sample; levels of significance did not change, as a result, the original scale was retained.

c

Forkus, Breines, & Weiss, 2019. All mean scores were compared to the current sample.

Psychometric Analyses

Table 4 provides correlations between traumatic military events and mental health symptoms. As expected, significant and positive associations were found between all traumatic military events (i.e., combat, MST, transgressions by self/others/betrayals) and mental health symptoms (i.e., PTSD, depression, AUD). Findings provide evidence for convergent validity.

Table 4.

Correlations among Traumatic Military Events and Mental Health Symptoms

1 2 3 4 5 6 7 8
1. Combat Involvement -- -.06 .32* .22* .25* .27* .23* .27*
2. MST .05 -- .21* .32* .29* .24* .34* .24*
3. Transgressions by Self .31* .21* -- .64* .73* .58* .43* .60*
4. Transgressions by Others .21* .31* .63* -- .66* .45* .35* .51*
5. Betrayals .25* .29* .72* .65* -- .58* .45* .62*
6. Depression Symptoms .29* .24* .58* .44* .58* -- .55* .80*
7. Alcohol Misuse .26* .35* .43* .32* .44* .54* -- .53*
8. PTSD Symptoms -- -- -- -- -- -- -- --
Open in a new tab

Note. MST = Military sexual trauma; PTSD = Posttraumatic stress disorder. MST captures a positive screen for sexual harassment or assault. Combat involvement and MST are dichotomous variables. Transgressions by self, transgressions by others, betrayals, depression symptoms, alcohol misuse, and PTSD symptoms are all continuous variables, calculated as severity scores (see measures). Estimates for the full sample are provided above the diagonal. Estimates for the trauma-exposed sample are provided below the diagonal.

*

p <.001

Supplemental Tables 1 and 2 provide information on factor loadings and fit indices for different PTSD factor models. CFA results indicated that all models fit the data well. Comparative fit analyses indicated that the Hybrid Model’s fit was optimal. Of the remaining models, the Dysphoric Arousal Model had a better fit than the DSM–5 and Dysphoria models. The Anhedonia Model had a better fit than the DSM-5, Dysphoria, and Dysphoric Arousal models. The Externalizing model had a better fit than the Dysphoria and Dysphoric Arousal models. For non-nested models, the Dysphoria Model had a better fit than the DSM-5 Model, and the Externalizing Behaviors Model had a better fit than the Anhedonia Model based on BIC value comparisons.

Discussion

The overarching goal of the current study was to examine the utility and feasibility of using MTurk-recruited military samples for trauma and mental health symptom assessment. Specifically, this study aimed to: 1) compare military trauma and mental health symptom prevalence rates and mean scores obtained via a MTurk-recruited military sample vs. published comparison samples; and 2) examine psychometrics of mental health measures and structural validity of the PCL-5. We found higher rates of traumatic events and mental health diagnoses relative to selected comparison samples. Further, mental health mean scores from our sample were found to be significantly higher than values found in nationally representative military samples, lower than treatment-seeking samples, and comparable to another MTurk-recruited military sample. The construct of alcohol misuse was an exception to this pattern; AUDIT scores were higher than the nationally representative and treatment-seeking samples. Correlation results found positive associations among traumatic events and mental health symptoms, supporting the convergent validity of measures for the current study. Lastly, CFA results demonstrated that empirically-supported factor models of PTSD were replicated in the current sample; the Hybrid Model demonstrated the best fit. Broadly, findings support the potential utility of recruiting military samples via MTurk, as well as highlight important areas for future research.

Consistent with past MTurk studies (Arditte et al., 2016; van Stolk-Cooke et al., 2018), we found heightened rates of traumatic exposures and mental health diagnoses in the current sample. This may be due to greater comfort in disclosing trauma and mental health symptoms via MTurk due to the visual anonymity (Joinson, 2001). Indeed, Shapiro et al. (2013) found that participants indicated greater comfort in disclosing clinical information in MTurk online surveys. Heightened rates may also be a reflection of the more diverse military sample (e.g., more women, ethnoracial minorities) that we were able to recruit via MTurk, as women and ethnoracial minorities have higher rates of lifetime PTSD (Olff et al., 2007; Roberts et al., 2011). Consistent with this, we found significantly higher scores on measures of depression (t(472) =5.96, p <.001), PTSD (t(517) =5.27, p <.001), and alcohol use (t(524) =5.34, p <.001) among ethnoracial minorities versus those who identified as white; however, we found no significant differences in these scores between men and women in the sample (ps >.05). Notably, there has been significant variability in the estimates of military events and mental health diagnoses across recruitment methods, samples, and assessment methods, more broadly, with studies finding wide ranges for combat (e.g., 11.4 – 50.5%; Donoho et al., 2017; Jacobson et al., 2008), MST (1.4 – 70.3%; Wilson, 2018), moral injury event types (10.8 – 41.1%; Maguen et al., 2020; Wisco et al., 2017), PTSD (e.g., 1.4 – 60.0%; Ramchand et al., 2010), depression (e.g., 2.0 – 47%; Gadermann et al., 2012; Holliday et al., 2016), and AUD (e.g., 10.0 – 57.0%; Seal et al., 2011; Stecker et al., 2010). More work is needed to clarify the differences between military samples recruited via MTurk vs. other sources, to identify whether this sample is truly representative or whether there are meaningful differences worth considering.

Partially consistent with expectations, we found that the PCL-5 and PHQ-9 mean scores were higher than national representative samples, lower than treatment-seeking samples, and statistically equivalent to another MTurk recruited military sample. Higher mean scores relative to nationally representative samples may be a product of a greater comfort disclosing mental health symptoms on MTurk. Further, individuals with higher rates of trauma and mental health symptoms may self-select into a study that is advertised as focusing on the mental health of military samples, as they may feel like their experiences uniquely qualify them for the study. Lower scores on the PCL-5 and PHQ-9 compared to treatment-seeking samples was not surprising; treatment-seeking behavior suggests that there is a mental health concern that requires intervention; as such, symptom severity would be expected to be elevated relative to a community MTurk-recruited sample. Unexpectedly, the AUDIT mean scores were higher than both the nationally representative and treatment-seeking military samples; this may be due to unexamined characteristics that systematically differ between MTurk and non-MTurk-recruited samples.

Another important aim of the current study was to examine psychometrics of clinical measures in an MTurk military sample. Past research has indicated strong positive associations between measures of traumatic events (i.e., combat, MST, and moral injury events) and mental health outcomes of PTSD, depression, and AUD (e.g., Cesur et al., 2013; Griffin et al., 2019; Suris & Lind, 2008). As expected, we found that all measures of traumatic events and mental health symptoms had significant and positive relations, thereby providing support for convergent validity of the measured constructs. Our pattern of findings aligns with theoretical and empirical literature. Relatedly, several PCL-5 factor-analytical models validated by research (APA, 2013; Armour et al., 2015; Elhai et al., 2011; Liu et al., 2014; Tsai et al., 2015) were replicated in the current sample. Further, our finding that the seven-factor Hybrid Model was the optimal model was consistent with literature (Wortmann et al., 2016). Thus, data from the current study’s sample was behaving in ways consistent with theoretically- and empirically-driven expectations.

Our findings support the utility of MTurk for collecting clinical data on military samples. Military personnel and veterans may experience barriers to participating in research (Greene-Shortridge et al., 2007), particularly that involving disclosure of trauma experiences/symptoms. The visual anonymity offered by MTurk may overcome these barriers by increasing comfort and thus willingness to disclose sensitive information. MTurk may also increase access to diverse military samples, replicating/extending previous findings and enhancing external validity, as well as introduce diverse perspectives from varied researchers. Moreover, the affordability and ease of MTurk recruitment can facilitate increased research with military populations. Finally, conducting research outside of the VA may enable researchers to capture a broader range of military samples, including veterans who may not fall within traditional definitions of being a “veteran.” Recruiting from non-VA sources may also increase participation among certain military samples, such as individuals who experience institutional distrust. However, we did not assess for VA treatment-seeking, and thus do not have access to information about how many participants were VA vs. non-VA users.

Several important limitations should be considered when interpreting study findings. First, the study relied on self-report measures and thus responses may be influenced by an individual’s willingness and ability to respond accurately. Future studies could incorporate follow-up interviews that include structured diagnostic interviews to more rigorously assess for diagnoses, as well as compare the level of agreement between self-report measures obtained via MTurk and “gold” standard clinical assessments. Second, the process of selecting comparison studies is imperfect. To reduce a biased selection of articles, we chose studies that used either large and representative samples or obtained estimates via meta-analysis; for the mean score comparison, we conducted a literature review to identify studies that used the exact (or similar) versions of the measures used in the current study. Despite these strategies, comparisons may be affected by the methodological and sample characteristic differences between the current study and comparison studies: recruitment source/method, assessments, inclusion/exclusion criteria, sample size, targeted military sample, and geographic area. Third, the current study did not examine all psychometric considerations of used measures due to lack of relevant data (e.g., test-retest reliability, discriminant validity). Fourth, we used the version of the LEC-5 that does not include the Criterion A assessment; thus, there is no way to verify that all participants endorsed an event that would meet for a Criterion A trauma. Future research should use the LEC-5 with the Criterion A assessment to more rigorously assess for Criterion A. Lastly, although MTurk broadens recruitment, there have been concerns related to fraudulent and careless responding and its impact on data quality (Chandler et al., 2020; Dennis et al., 2020). Specifically, more sophisticated attempts (i.e., virtual private servers [VPS]) are being used to generate multiple IP addresses to bypass screening and validity questions. Further, online surveys are susceptible to dishonest or careless responses. Thus, it is necessary to implement procedures to reduce and remove poor-quality responses. Following recommendations (Chandler et al., 2020), we prevented duplicate IP addresses (which has been shown to reduce fraudulent response by about 85%; Chandler & Paolacci, 2017), used reasonable compensation rates (i.e., $2; higher pay is more susceptible to fraud), and required participants to correctly respond to four separate validated measures of attention and comprehension (Meade & Craig, 2012). Future research should include these precautions, as well as newer approaches such as using tools to block non-US and VPS IP addresses (Dennis et al., 2020; Kennedy et al., 2020).

Despite these limitations, the results of this study add to the growing body of literature on the utility of MTurk, particularly as it relates to hard-to-reach populations. Findings provide support for crowdsourcing as a useful tool for recruiting military samples and assessing clinical characteristics. Following recommended approaches to protect the quality of the data, MTurk appears to be an important resource for collecting data on military samples, as it is time and cost-efficient, is not geographically limited, and provides access to more diverse military samples. Increasing access to and recruitment of military samples is important to inform intervention as well as mechanism-driven experimental research.

Supplementary Material

Supplemental Material

Clinical Impact Statement.

The current study examined the utility of recruiting military samples for clinically-focused research via Amazon’s Mechanical Turk (MTurk). Consistent with past MTurk research, we found greater reported trauma, mental health diagnoses, and symptom severity. Investigation of the psychometrics of clinical measures indicated the data were behaving in ways consistent with theoretically- and empirically-driven expectations. Findings suggest that if recommended approaches are taken to protect the quality of the data, MTurk appears to be a useful resource for recruiting military samples.

Acknowledgments

Work on this paper by the last author (NHW) was supported by National Institutes of Health Grants K23DA039327 and P20GM125507.

Footnotes

1

The DSM-5 Model is comprised of four symptom factors: intrusions (items 1–5), avoidance (items 6–7), NACM (items 8–14), and AAR (items 15–20). The Dysphoria Model (Simms et al., 2002) is comprised of four factors: intrusions (items 1–5), avoidance (items 6–7), dysphoria (items 8–16; 19–20), and AAR (items 17–18). The Dysphoric Arousal Model (Elhai et al., 2011) is comprised of five factors: intrusions (items 1–5), avoidance (items 6–7), NACM (8–14), dysphoric arousal (items 15–16; 19–20), and anxious arousal (items 17–18). The Anhedonia Model (Liu et al., 2014) is comprised of six factors: intrusions (items 1–5), avoidance (items 6–7), negative affect (items 8–11), anhedonia (12–14), dysphoric arousal (items 15–16; 19–20), and anxious arousal (items 17–18). The Externalizing Behavior Model (Tsai et al., 2015) is comprised of six factors: intrusions (items 1–5), avoidance (items 6–7), NACM (items 8–14), externalizing behaviors (items 15–16), anxious arousal (items 17–18), and dysphoric arousal (items 19–20). Lastly, the Hybrid Model (Armour et al., 2015) is comprised of seven factors: intrusions (items 1–5), avoidance (items 6–7), negative affect (items 8–11), anhedonia (items 12–14), externalizing behaviors (items 15–16), anxious arousal (items 17–18), and dysphoric arousal (items 19–20).

References

  1. Arditte KA, Çek D, Shaw AM, & Timpano KR (2016). The importance of assessing clinical phenomena in Mechanical Turk research. Psychological Assessment, 28(6), 684–691. 10.1037/pas0000217 [DOI] [PubMed] [Google Scholar]
  2. Armour C, Tsai J, Durham TA, Charak R, Biehn TL, Elhai JD, & Pietrzak RH (2015). Dimensional structure of DSM-5 posttraumatic stress symptoms: support for a hybrid anhedonia and externalizing behaviors model. J Psychiatr Res, 61, 106–113. 10.1016/j.jpsychires.2014.10.012 [DOI] [PubMed] [Google Scholar]
  3. Betancourt JA, Stigler Granados P, Pacheco GJ, Shanmugam R, Kruse CS, & Fulton LV (2020). Obesity and Morbidity Risk in the US Veteran. Paper presented at the Healthcare. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Blevins CA, Weathers FW, Davis MT, Witte TK, & Domino JL (2015). The posttraumatic stress disorder checklist for DSM-5 (PCL-5): Development and initial psychometric evaluation. Journal of Traumatic Stress, 28(6), 489–498. 10.1002/jts.22059 [DOI] [PubMed] [Google Scholar]
  5. Bovin MJ, Marx BP, Weathers FW, Gallagher MW, Rodriguez P, Schnurr PP, & Keane TM (2016). Psychometric properties of the PTSD checklist for diagnostic and statistical manual of mental disorders–fifth edition (PCL-5) in veterans. Psychological Assessment, 28(11), 1379. 10.1037/pas0000254 [DOI] [PubMed] [Google Scholar]
  6. Braun LA, Kennedy HP, Sadler LS, & Dixon J (2015). Research on US military women: recruitment and retention challenges and strategies. Military Medicine, 180(12), 1247–1255. 10.7205/MILMED-D-14-00601 [DOI] [PubMed] [Google Scholar]
  7. Browne MW, & Cudeck R (1993). Alternative ways of assessing model fit In: Bollen KA, Long JS, editors. Testing structural equation models. Beverly Hills, CA: Sage, 111–135. [Google Scholar]
  8. Bryan CJ, Bryan AO, Anestis MD, Anestis JC, Green BA, Etienne N, . . . Ray-Sannerud B. (2016). Measuring moral injury: Psychometric properties of the moral injury events scale in two military samples. Assessment, 23(5), 557–570. 10.1177/1073191115590855 [DOI] [PubMed] [Google Scholar]
  9. Buhrmester M, Kwang T, & Gosling SD (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet highquality, data? Perspectives on Psychological Science, 6, 3–5. 10.1177/1745691610393980 [DOI] [PubMed] [Google Scholar]
  10. Bush NE, Sheppard SC, Fantelli E, Bell KR, & Reger MA (2013). Recruitment and attrition issues in military clinical trials and health research studies. Military Medicine, 178(11), 1157–1163. 10.7205/MILMED-D-13-00234 [DOI] [PubMed] [Google Scholar]
  11. Campbell DJ, & Nobel OB-Y (2009). Occupational stressors in military service: A review and framework. Military Psychology, 21(sup2), S47–S67. 10.1080/08995600903249149 [DOI] [Google Scholar]
  12. Cesur R, Sabia JJ, & Tekin E. (2013). The psychological costs of war: Military combat and mental health. Journal of Health Economics, 32(1), 51–65. 10.1016/j.jhealeco.2012.09.001 [DOI] [PubMed] [Google Scholar]
  13. Chandler J, & Shapiro D. (2016). Conducting clinical research using crowdsourced convenience samples. Annual Review of Clinical Psychology, 12. 10.1146/annurev-clinpsy-021815-093623 [DOI] [PubMed] [Google Scholar]
  14. Contractor AA, Greene T, Dolan M, & Elhai JD (2018). Relations between PTSD and depression symptom clusters in samples differentiated by PTSD diagnostic status. Journal of Anxiety Disorders, 59, 17–26. 10.1016/j.janxdis.2018.08.004 [DOI] [PubMed] [Google Scholar]
  15. Donoho CJ, Bonanno GA, Porter B, Kearney L, & Powell TM (2017). A decade of war: prospective trajectories of posttraumatic stress disorder symptoms among deployed US military personnel and the influence of combat exposure. American Journal of Epidemiology, 186(12), 1310–1318. 10.1093/aje/kwx318 [DOI] [PubMed] [Google Scholar]
  16. Duffy RD, Jadidian A, Douglass RP, & Allan BA (2015). Work volition among US veterans: Locus of control as a mediator. The Counseling Psychologist, 43(6), 853–878. 10.1177/0011000015576800 [DOI] [Google Scholar]
  17. Elhai JD, Biehn TL, Armour C, Klopper JJ, Frueh BC, & Palmieri PA (2011). Evidence for a unique PTSD construct represented by PTSD’s D1–D3 symptoms. Journal of Anxiety Disorders, 25(3), 340–345. 10.1016/j.janxdis.2010.10.007 [DOI] [PubMed] [Google Scholar]
  18. Engle K, Talbot M, & Samuelson KW (2019). Is Amazon’s Mechanical Turk (MTurk) a Comparable Recruitment Source for Trauma Studies? Psychological Trauma: Theory, Research, Practice, and Policy. Advance online publication. 10.1037/tra0000502 [DOI] [PubMed] [Google Scholar]
  19. Erwin MC, Charak R, Durham TA, Armour C, Lv X, Southwick SM, . . . Pietrzak RH (2017). The 7-factor hybrid model of DSM-5 PTSD symptoms and alcohol consumption and consequences in a national sample of trauma-exposed veterans. Journal of Anxiety Disorders, 51, 14–21. 10.1016/j.janxdis.2017.08.001 [DOI] [PubMed] [Google Scholar]
  20. Feitosa J, Joseph DL, & Newman DA (2015). Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English. Personality and Individual Differences, 75, 47–52. 10.1016/j.paid.2014.11.017 [DOI] [Google Scholar]
  21. Forkus SR, Breines JG, & Weiss NH (2019). Morally injurious experiences and mental health: The moderating role of self-compassion. Psychological Trauma: Theory, Research, Practice, and Policy, 11(6), 630–638. 10.1037/tra0000446 [DOI] [PubMed] [Google Scholar]
  22. Fuehrlein BS, Mota N, Arias AJ, Trevisan LA, Kachadourian LK, Krystal JH, . . . Pietrzak RH (2016). The burden of alcohol use disorders in US military veterans: results from the National Health and Resilience in Veterans Study. Addiction, 111(10), 1786–1794. 10.1111/add.13423 [DOI] [PubMed] [Google Scholar]
  23. Fulton JJ, Calhoun PS, Wagner HR, Schry AR, Hair LP, Feeling N, . . . Beckham JC (2015). The prevalence of posttraumatic stress disorder in Operation Enduring Freedom/Operation Iraqi Freedom (OEF/OIF) veterans: A meta-analysis. Journal of Anxiety Disorders, 31, 98–107. 10.1016/j.janxdis.2015.02.003 [DOI] [PubMed] [Google Scholar]
  24. Gadermann AM, Engel CC, Naifeh JA, Nock MK, Petukhova M, Santiago PN, . . . Kessler RC (2012). Prevalence of DSM-IV major depression among US military personnel: meta-analysis and simulation. Military Medicine, 177(suppl_8), 47–59. 10.7205/MILMED-D-12-00103 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Greene-Shortridge TM, Britt TW, & Castro CA (2007). The stigma of mental health problems in the military. Military Medicine, 172(2), 157–161. 10.7205/MILMED.172.2.157 [DOI] [PubMed] [Google Scholar]
  26. Griffin BJ, Purcell N, Burkman K, Litz BT, Bryan CJ, Schmitz M, . . . Maguen S. (2019). Moral injury: an integrative review. Journal of Traumatic Stress, 32(3), 350–362. 10.1002/jts.22362 [DOI] [PubMed] [Google Scholar]
  27. Hauser DJ, & Schwarz N. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48(1), 400–407. 10.3758/s13428-015-0578-z [DOI] [PubMed] [Google Scholar]
  28. Hoffmire CA, Kemp JE, & Bossarte RM (2015). Changes in suicide mortality for veterans and nonveterans by gender and history of VHA service use, 2000–2010. Psychiatric Services, 66(9), 959–965. 10.1176/appi.ps.201400031 [DOI] [PubMed] [Google Scholar]
  29. Holliday SB, Pedersen ER, & Leventhal AM (2016). Depression, posttraumatic stress, and alcohol misuse in young adult veterans: The transdiagnostic role of distress tolerance. Drug and Alcohol Dependence, 161, 348–355. 10.1016/j.drugalcdep.2016.02.030 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Hu L. t., & Bentler PM (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. 10.1080/10705519909540118 [DOI] [Google Scholar]
  31. Jacobson IG, Ryan MA, Hooper TI, Smith TC, Amoroso PJ, Boyko EJ, . . . Bell NS (2008). Alcohol use and alcohol-related problems before and after military combat deployment. Jama, 300(6), 663–675. 10.1001/jama.300.6.663 [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Joinson AN (2001). Self-disclosure in computer-mediated communication: The role of self-awareness and visual anonymity. European Journal Of Social Psychology, 31(2), 177–192. 10.1002/ejsp.36 [DOI] [Google Scholar]
  33. Kass RE, & Raftery AE (1995). Bayes factors. Journal Of The American Statistical Association, 90(430), 773–795. [Google Scholar]
  34. Klingensmith K, Tsai J, Mota NP, Southwick SM, & Pietrzak RH (2014). Military sexual trauma in US veterans: results from the National Health and Resilience in Veterans Study. Journal of Clinical Psychiatry, 75(10), e1133–e1139. doi: 10.4088/JCP.14m09244 [DOI] [PubMed] [Google Scholar]
  35. Kroenke K, & Spitzer RL (2002). The PHQ-9: a new depression diagnostic and severity measure. Psychiatric Annals, 32(9), 509–515. 10.3928/0048-5713-20020901-06 [DOI] [Google Scholar]
  36. Kroenke K, Spitzer RL, & Williams JB (2001). The PHQ-9: validity of a brief depression severity measure. Journal of General Internal Medicine, 16(9), 606–613. 10.1046/j.1525-1497.2001.016009606.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Lancaster SL, & Miller M. (2020). Moral decision making, religious strain, and the experience of moral injury. Psychological Trauma: Theory, Research, Practice, and Policy, 12(2), 156–164. 10.1037/tra0000518 [DOI] [PubMed] [Google Scholar]
  38. Liu P, Wang L, Cao C, Wang R, Zhang J, Zhang B, . . . Elhai JD (2014). The underlying dimensions of DSM-5 posttraumatic stress disorder symptoms in an epidemiological sample of Chinese earthquake survivors. Journal of Anxiety Disorders, 28(4), 345–351. 10.1016/j.janxdis.2014.03.008 [DOI] [PubMed] [Google Scholar]
  39. Lorber W, & Garcia HA (2010). Not supposed to feel this: Traditional masculinity in psychotherapy with male veterans returning from Afghanistan and Iraq. Psychotherapy: Theory, Research, Practice, Training, 47(3), 296–305. 10.1037/a0021161 [DOI] [PubMed] [Google Scholar]
  40. Lynn BM-D, & Morgan JK (2016). Using Amazon’s Mechanical Turk (MTurk) to recruit military veterans: Issues and suggestions. The Military Psychologist, 31(3), 8–14. [Google Scholar]
  41. Maguen S, Griffin BJ, Copeland LA, Perkins DF, Finley EP, & Vogt D. (2020). Gender Differences in Prevalence and Outcomes of Exposure to Potentially Morally Injurious Events among Post-9/11 Veterans. Journal of Psychiatric Research. 130, 97–103. 10.1016/j.jpsychires.2020.06.020 [DOI] [PubMed] [Google Scholar]
  42. Meade AW, & Craig SB (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. 10.1037/a0028085 [DOI] [PubMed] [Google Scholar]
  43. Meffert BN, Morabito DM, Sawicki DA, Hausman C, Southwick SM, Pietrzak RH, & Heinz AJ (2019). US Veterans Who Do and Do Not Utilize VA Healthcare Services: Demographic, Military, Medical, and Psychosocial Characteristics. The Primary Care Companion for CNS Disorders, 21(1). 10.4088/PCC.18m02350 [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Mengeling MA, Booth BM, Torner JC, & Sadler AG (2019). Post–sexual assault health care utilization among OEF/OIF servicewomen. Medical Care, 53(4s1), S136–S142. 10.1097/MLR.0000000000000267 [DOI] [PubMed] [Google Scholar]
  45. Morgan JK, & Desmarais SL (2017). Associations between time since event and posttraumatic growth among military veterans. Military Psychology, 29(5), 456–463. 10.1037/mil0000170 [DOI] [Google Scholar]
  46. Nash WP, Marino Carper TL, Mills MA, Au T, Goldsmith A, & Litz BT (2013). Psychometric evaluation of the moral injury events scale. Military Medicine, 178(6), 646–652. 10.7205/MILMED-D-13-00017 [DOI] [PubMed] [Google Scholar]
  47. Olff M, Langeland W, Draijer N, & Gersons BP (2007). Gender differences in posttraumatic stress disorder. Psychological Bulletin, 133(2), 183–204. 10.1037/0033-2909.133.2.183 [DOI] [PubMed] [Google Scholar]
  48. Paolacci G, Chandler J, & Ipeirotis PG (2010). Running experiments on amazon mechanical turk. Judgment and Decision Making, 5(5), 411–419. [Google Scholar]
  49. Ramchand R, Schell TL, Karney BR, Osilla KC, Burns RM, & Caldarone LB (2010). Disparate prevalence estimates of PTSD among service members who served in Iraq and Afghanistan: possible explanations. Journal of Traumatic Stress, 23(1), 59–68. 10.1002/jts.20486 [DOI] [PubMed] [Google Scholar]
  50. Roberts AL, Gilman SE, Breslau J, Breslau N, & Koenen KC (2011). Race/ethnic differences in exposure to traumatic events, development of post-traumatic stress disorder, and treatment-seeking for post-traumatic stress disorder in the United States. Psychological Medicine, 41(1), 71–83. 10.1017/S0033291710000401 [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Rosseel Y. (2012). Lavaan: An R package for structural equation modeling and more. Version 0.5–12 (BETA). Journal of Statistical Software, 48(2), 1–36. [Google Scholar]
  52. Saunders JB, Aasland OG, Babor TF, De la Fuente JR, & Grant M. (1993). Development of the alcohol use disorders identification test (AUDIT): WHO collaborative project on early detection of persons with harmful alcohol consumption-II. Addiction, 88(6), 791–804. [DOI] [PubMed] [Google Scholar]
  53. Sayer NA, Carlson KF, & Frazier PA (2014). Reintegration challenges in US service members and veterans following combat deployment. Social Issues and Policy Review, 8(1), 33–73. 10.1111/sipr.12001 [DOI] [Google Scholar]
  54. Seal KH, Cohen G, Waldrop A, Cohen BE, Maguen S, & Ren L. (2011). Substance use disorders in Iraq and Afghanistan veterans in VA healthcare, 2001–2010: Implications for screening, diagnosis and treatment. Drug and Alcohol Dependence, 116(1–3), 93–101. 10.1016/j.drugalcdep.2010.11.027 [DOI] [PubMed] [Google Scholar]
  55. Searle AK, Van Hooff M, McFarlane AC, Davies CE, Fairweather-Schmidt AK, Hodson SE, . . . Steele N. (2015). The validity of military screening for mental health problems: diagnostic accuracy of the PCL, K10 and AUDIT scales in an entire military population. International journal of methods in psychiatric research, 24(1), 32–45. 10.1002/mpr.1460 [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Shapiro DN, Chandler J, & Mueller PA (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1(2), 213–220. 10.1177/2167702612469015 [DOI] [Google Scholar]
  57. Simms LJ, Watson D, & Doebbelling BN (2002). Confirmatory factor analyses of posttraumatic stress symptoms in deployed and nondeployed veterans of the Gulf War. Journal of Abnormal Psychology, 111(4), 637. 10.1037//0021-843X.111.4.637 [DOI] [PubMed] [Google Scholar]
  58. Skinner HA (1982). The drug abuse screening test. Addictive Behaviors, 7(4), 363–371. 10.1016/0306-4603(82)90005-3 [DOI] [PubMed] [Google Scholar]
  59. Smigelsky MA, Malott JD, Veazey Morris K, Berlin KS, & Neimeyer RA (2019). Latent profile analysis exploring potential moral injury and posttraumatic stress disorder among military veterans. Journal of Clinical Psychology, 75(3), 499–519. 10.1002/jclp.22714 [DOI] [PubMed] [Google Scholar]
  60. Stecker T, Fortney J, Hamilton F, Sherbourne CD, & Ajzen I. (2010). Engagement in mental health treatment among veterans returning from Iraq. Patient Preference and Adherence, 4, 45. 10.2147/ppa.s7368 [DOI] [PMC free article] [PubMed] [Google Scholar]
  61. Suris A, & Lind L. (2008). Military sexual trauma: A review of prevalence and associated health consequences in veterans. Trauma, Violence, & Abuse, 9(4), 250–269. 10.1177/1524838008324419 [DOI] [PubMed] [Google Scholar]
  62. Tabachnick BG, & Fidell LS (2007). Using multivariate statistics, 5th. Needham Height, MA: Allyn & Bacon. [Google Scholar]
  63. Thomas MM, Harpaz-Rotem I, Tsai J, Southwick SM, & Pietrzak RH (2017). Mental and physical health conditions in US combat veterans: Results from the National Health and Resilience in Veterans Study. The Primary Care Companion for CNS Disorders, 19(3). 10.4088/PCC.17m02118 [DOI] [PubMed] [Google Scholar]
  64. van Stolk-Cooke K, Brown A, Maheux A, Parent J, Forehand R, & Price M. (2018). Crowdsourcing Trauma: Psychopathology in a Trauma-Exposed Sample Recruited via Mechanical Turk. Journal of Traumatic Stress, 31(4), 549–557. 10.1002/jts.22303 [DOI] [PMC free article] [PubMed] [Google Scholar]
  65. Weathers FW, Litz BT, Keane TM, Palmieri PA, Marx BP, & Schnurr PP (2013). The ptsd checklist for dsm-5 (pcl-5). Scale available from the National Center for PTSD at www.ptsd.va.gov, 10. [Google Scholar]
  66. Wilcox SL, Oh H, Redmond SA, Chicas J, Hassan AM, Lee P-J, & Ell K. (2015). A scope of the problem: Post-deployment reintegration challenges in a National Guard Unit. Work, 50(1), 73–83. 10.3233/WOR-141935 [DOI] [PubMed] [Google Scholar]
  67. Wilson LC (2018). The prevalence of military sexual trauma: A meta-analysis. Trauma, Violence, & Abuse, 19(5), 584–597. 10.1177/1524838016683459 [DOI] [PubMed] [Google Scholar]
  68. Wisco BE, Marx BP, May CL, Martini B, Krystal JH, Southwick SM, & Pietrzak RH (2017). Moral injury in US combat veterans: Results from the national health and resilience in veterans study. Depression and Anxiety, 34(4), 340–347. 10.1002/da.22614 [DOI] [PubMed] [Google Scholar]
  69. Wortmann JH, Jordan AH, Weathers FW, Resick PA, Dondanville KA, Hall-Clark B, . . . Hembree EA (2016). Psychometric analysis of the PTSD Checklist-5 (PCL-5) among treatment-seeking military service members. Psychological Assessment, 28(11), 1392. 10.1037/pas0000260 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental Material
NIHMS1894554-supplement-Supplemental_Material.pdf (167.5KB, pdf)

RESOURCES