Skip to main content
NCBI home page
Military Psychology logoLink to Military Psychology
. 2021 Dec 17;36(1):49–57. doi: 10.1080/08995605.2021.1997501

Assessing the dimensionality and construct validity of the military stigma scale across current service members

Carlos A Vidales a,✉, Derek J Smolenski b, Nancy A Skopp b, David Vogel a, Nathaniel Wade a, Sean Sheppard c, Katrina Speed d, Kristina Hood d, Patricia Cartwright d
PMCID: PMC10790807  PMID: 38193877

ABSTRACT

US service members are at elevated risk for distress and suicidal behavior, compared to the general US population. However, despite the availability of evidence-based treatments, only 40% of Service members in need of mental health care seek help. One potential reason for the lower use of services is that service members experience stigma or concerns that the act of seeking mental health care from a mental health provider carries a mark of disgrace. The Military Stigma Scale (MSS) was designed to assess two theoretical dimensions of help-seeking stigma (public and self), specifically among service members. The goal of the current study was to further examine the validity of the MSS among 347 active duty service members. Examination of unidimensional, two-factor, and bifactor models revealed that a bifactor model, with a general (overall stigma), two specific factors (public and self-stigma), and one method factor (accounting for negatively worded items) provided the best fit to the data. Ancillary reliability analyses also supported the MSS measuring a broad stigma factor associated with seeking mental health care in the military. Subsequent model analyses showed that the MSS was associated with other stigma-related constructs. Overall, findings suggest that the MSS is a reliable and validated scale that can be used to assess military help-seeking stigma and to evaluate results of programs designed to reduce stigma.

KEYWORDS: Mental health stigma, military, seeking care, self-stigma, public stigma

What is the public significance of this article?—This article provides additional validity evidence in support of the Military Stigma Scale. This scale measures stigma for seeking mental health care amongst active duty members. Based on the findings of this article, the Military Stigma Scale can be used to measure the impact of stigma reduction interventions amongst active duty members.

US service members are at elevated risk for mental health (MH) distress and suicidal behavior as compared to the civilian population (see Hom et al., 2017 for a review). Despite the availability of evidence-based treatments (Hom et al., 2017), roughly 60% of Service members in need of MH services do not seek care because of concerns related to stigma (Sharp et al., 2015). Untreated MH problems are associated with exacerbation of symptoms, poor occupational functioning, family disruption, and related costs in health care and human suffering (Boyd et al., 2016; Fox et al., 2016; Hoge et al., 2004). Many service members experience stigma or concern that the act of seeking care from an MH provider carries a mark of dishonor and renders one vulnerable to prejudice and discrimination (Clement et al., 2014). A recent study showed that military personnel were particularly concerned about the impact of seeking MH care on their careers, for instance, fearing that they would be passed over for promotion if they sought help (Brown & Bruce, 2016). MH stigma is also associated with premature treatment dropout among service members who seek MH care because of concerns about differential treatment and negative career impact (e.g., Jennings et al., 2016). In brief, understanding and assessing stigma associated with seeking MH care is vital to the health and well-being of military personnel and overall force readiness (VanSickle et al., 2016).

To assess the extent to which stigma associated with seeking MH care in the military influences care-seeking, as well as the efficacy of interventions designed to ameliorate it, a comprehensive, reliable and valid measure is essential. Such a measure is needed to shed light on targets for prevention and enable clinicians to evaluate the factors driving patients’ stigma-related reluctance to seek MH care prior to and during treatment to help mitigate dropout (Britt et al., 2015; Hoge et al., 2014; Jennings et al., 2016) and maximize treatment effects (Kulesza et al., 2017; Ociskova et al., 2018). However, there are relatively few validated measures of stigma for seeking MH care, and a majority of the existing measures were developed and tested in civilian populations. One notable study conducted by Vogt et al. (2014) developed a measure of endorsed stigma (i.e., the internalization process of self-stigma) and anticipated stigma (i.e., aspects of expected public stigma such as devaluation) intended for use among military personnel and Veterans. While this was an important contribution to the field, the scale items do not appear to reflect stigmata specifically associated with the military; rather these items seem to generally refer to beliefs and concerns about mental illness and treatment that may not differentiate from civilian perceptions (for example, “People with mental health problems can’t take care of themselves”). Moreover, it is important to understand the nuances within active duty personnel that are unique from Veterans and other military personnel. For instance, expectations for how one’s military peers while in duty may differ from one’s expectations from one’s peers during veterancy. For example, active duty Service members may expect to be devalued and viewed as weak by their chain of command if they sought help, while Veterans may expect to be viewed as unstable or dangerous by civilians if they sought help (Mittal et al., 2013). Further, stigma associated with seeking MH care in the military may differ from civilian care-seeking by virtue of cultural factors such as warrior ethos (e.g., self-sufficiency, duty to the mission) as well as the public nature of seeking MH care in the military health systems. For example, service members are required to notify their command and to be excused from work to access MH care; thus, they are not afforded the same level of privacy associated with seeking such care in civilian sectors. Consistent with this, concerns that military commanders and peers will view MH care-seeking negatively and respond to such behavior with differential treatment are among the most cited reasons for service member reluctance to seek MH care (Hoerster et al., 2012; Iverson et al., 2011; Sharp et al., 2015). However, measures of help-seeking stigma tend to ask generic questions (e.g., “Seeing a psychologist for emotional or interpersonal problems carries social stigma”; Komiya et al., 2000) that may not capture the most salient concerns of military personnel (e.g., “I would worry about my personal problems being part of my military records”).

To fill this need for a comprehensive, reliable, and valid measure associated with help-seeking stigma in the military, Skopp et al. (2012) developed the Military Stigma Scale (MSS; Skopp et al., 2012). The MSS is a 26-item measure that assessed military-specific stigma (e.g., “I would be given less responsibility, if chain of command knew I was seeing a mental health provider”). Consistent with prior research suggesting the importance of stigma (e.g., Greene-Shortridge et al., 2007), service members who had had prior experience with an MH provider scored lower in stigma than those who had not (Skopp et al., 2012). Male service members also reported significantly higher self-stigma than female service members. However, despite the importance of the MSS several limitations were noted including only assessing one aspect of the construct validity of the MSS (i.e., levels of stigma in individuals who had received MH services in the past; Skopp et al., 2012). Specifically, the researchers (Skopp et al., 2012) suggested that future research should address this shortcoming by examining the links between the MSS and other MH stigma measures and other factors theoretically related to receiving mental healthcare in the military, such as organizational barriers (see Greene-Shortridge et al., 2007). Military-specific organizational barriers, for example, may include the role of military unit leadership, which has been associated with stigma and perceived barriers to care (Jones et al., 2018; Wright et al., 2009).

Another limitation that needs to be further addressed is the factor structure of the MSS. Consistent with the larger research on help-seeking and previous military research (e.g., Greene-Shortridge et al., 2007), the developers of the MSS conceptualized stigma for seeking MH care in terms of both public (i.e., perceived reactions of the general public toward individuals who seek MH care) and self-stigma (i.e., internalization of negative perceptions of the public toward individuals who seek MH care; Coleman et al., 2017; Corrigan & Watson, 2002; Vogt, 2011). However, while public and self-stigma have been implicated in the help-seeking behavior of military personnel (Hantzi et al., 2019; Jennings et al., 2015; Wade et al., 2015), it is not fully clear if the MSS actually assesses two conceptually distinct constructs, as the authors did not account for potential item wording (i.e., method factor) or assess a bi-factor model. The correlation between the public and self-stigma subscales of the MSS was .58, which suggests that the MSS may be defined by a strong general factor among active duty service members (Reise et al., 2010). A bifactor model approach is to scale validation is designed to examine the extent to which a general factor can account for the commonality between the specific (i.e. public- and self-stigma) factors (Cheung & Lau, 2012; Reise et al., 2013). More explicitly, a bifactor model specifies that all items simultaneously load onto: 1) a general factor reflecting the common variance shared by the items and 2) their respective subscale or specific factor (Holzinger & Swineford, 1937; Reise et al., 2010). An additional advantage is that a bifactor approach assesses model-based reliability and the dimensionality of general and specific factors (Reise et al., 2013; Rodriguez et al., 2016); as well as the incremental validity of the general latent factors through their relations with relevant outcomes (e.g., other stigma measures and barriers to accessing MH care).

The purpose of the current study, therefore, was to provide a comprehensive assessment of the dimensionality, internal-consistency reliability, and validity of the MSS among active duty service members. First, we examined the underlying factor structure of the MSS by testing a series of models to assess dimensionality (Reise et al., 2010). We evaluated fit for first-order correlated traits and bifactor models (Reise et al., 2010). Second, we examined the construct validity of the MSS. We expected the MSS to correlate positively with other MH stigma measures. Additionally, as military units characterized by strong leadership and unit cohesion moderate perceived barriers to stigma of care such that to lower, but not higher, levels of stigma, predict MH care seeking (Wright et al., 2009), we expected unit leadership and unit cohesion characteristics to correlate negatively with the MSS.

Method

Participants and procedures

The study consisted of 347 active duty service members recruited from online websites that cater to military populations (e.g., afterdeployment.org; see Table 1 for demographic information). Afterdeployment.org staff attended pre-and post-deployment “Yellow Ribbon” ceremonies (www.yellowribbon.mil) for National Guard and Reserve Service members to disseminate information about the website and to provide an opportunity for Service members to volunteer for participation in the survey via the website. Additionally, there was a banner on the afterdeployment.org advertising the opportunity, for those visiting the website, to participate with a link to the survey. The data were collected anonymously, and participants were provided with a study information sheet prior to volunteering. All study procedures were approved by the institutional review board approval prior to participant enrollment. Data collection was anonymous and completed via online surveys using secure encrypted servers, and the data were stored on secure, password protected, encrypted servers with limited access. In addition to completing the MSS (Skopp et al., 2012) and demographic information including age, sex, race ethnicity, marital status, rank/grade, and education, participants also completed additional measures used to assess construct validity, including the Stigma Scale for Receiving Psychological Help (SSRPH; Komiya et al., 2000), Unit Cohesion – Peers; (Podsakoff & MacKenzie, 1994; Wright et al., 2009), and Unit Cohesion – Leader Behavior (Wright et al., 2009) measures.

Table 1.

Demographic information and means (SD) of the MSS by group.

  M (SD)
Age 32.7 (9.3)%
Sex  
 Female 17.3
 Male 82.7
Race/ethnicity  
 White, not Hispanic 72.9
 Black, not Hispanic 6.6
 Hispanic, any race 9.5
 Other 10.7
Marital status  
 Never married 18.7
 Married 67.7
 Separated 2.6
 Divorced 10.4
 Widowed 0.6
Rank/grade  
 E1-E4 26.8
 E5-E9 53.3
 Officer 19.0
 No Report 0.9
Education  
 High school 21.6
 Some college 43.5
 Four-year degree 21.6
 Postgraduate 13.3
Open in a new tab

MSS = Military Stigma Scale; Pub = MSS Public-Stigma Subscale; Self = MSS Self-Stigma Subscale.

Measures

Military stigma scale (MSS; Skopp et al., 2012)

The MSS is a 26-item measure that contains 16-items originally developed to represent anticipated public stigma (e.g., “I would be given less responsibility if chain of command knew I was seeing a mental health provider”) and 10-items representing anticipated self-stigma (e.g., “I would feel inadequate if I went to a therapist for psychological help”) constructs. The items are scored on a 4-point scale ranging from 1 (Definitely Disagree) to 4 (Definitely Agree). In the initial development study, Skopp et al. (2012) reported fit for a two-factor model, with reliability for the public and self-stigmas being .94 and .89, respectively.

Stigma scale for receiving psychological help (SSRPH; Komiya et al., 2000)

The SSRPH is a five-item measure designed to assess perceptions of stigma associated with seeking MH care (e.g., “It is advisable for a person to hide from people that he/she has seen a psychologist”) scored on a four-point scale from 0 (Strongly Disagree) to 3 (Strongly Agree). Prior research has reported that this measure displays good internal consistency and construct validity (Komiya et al., 2000; Topkaya, 2014; Tucker et al., 2013). Internal consistency of the scores in the current study was .89 (95% CI = 0.87, 0.92).

Unit cohesion – peers

Service member’s perceptions of unit cohesion were assessed using a 3-item scale that asks respondents are asked to rate the extent to which they agree or disagree with the following statements: 1) The members of my unit are cooperative with each other; 2) The members of my unit know they can depend on each other; 3) The members of my unit stand up for each other (Podsakoff & MacKenzie, 1994; Wright et al., 2009). This measure is commonly used in military research (e.g., Britt & Dawson, 2005; Britt et al., 2007). Internal consistency of the scores in the current sample was 0.96 (95% CI = 0.95, 0.97).

Unit cohesion – leader behavior

This measure contains four items that assess military leader behavior on a 5-point scale ranging from 1 (Strongly Disagree) to 5 (Strongly Agree) (Wright et al., 2009). Respondents rated each item in response to the stem, “Thinking about your unit, rate how often the following occur. In your unit, Officers ___.” Example: Tell soldiers when they have done a good job. These items have been used in population-based military research (Hoge et al., 2004) and positive ratings relate to adherence to battlefield ethics and MH symptoms (Office of the Surgeon Multinational Force-Iraq and Office of the Surgeon General United States Army Medical Command, 2006). Internal consistency of the scores in the current sample was 0.82 (95% CI = 0.78, 0.86).

Analytic plan

We used confirmatory factor analysis to test three factor structures: a unidimensional model with a single latent variable, a first-order model with two latent variables for public and self-stigma, and a bifactor model (Murray & Johnson, 2013) with one general stigma factor and the two specific-factors (public and self-stigma). We estimated each of these models with and without an additional method factor to account for wording valence of the four reverse-coded items (DiStefano & Motl, 2006). In the bifactor models, both the general and the specific factors (i.e., public and self-stigma) were orthogonal to all other factors in these models. All models were estimated using mean- and variance-adjusted weighted least squares (WLSMV) in Mplus 8.2 (Muthén & Muthén, 1998–2017).

We assessed all models for goodness of fit using the χ2 test statistic, the comparative fit index (CFI), the Tucker-Lewis index (TLI), and the root mean square error of approximation (RMSEA) with its associated 90% confidence interval, and the standardized root mean square residual (SRMR). Values of 0.95 or greater on the CFI and TLI and an upper bound of the 90% confidence interval for the RMSEA and the SRMR at or below 0.08 were used as indicators of good model fit (Hu & Bentler, 1999). The three different configural models were not nested; consequently, we could not conduct formal difference testing to compare model fit. For the bifactor models, we evaluated the dimensionality of the scale using measures described in Rodriguez et al. (2016) as well as the incremental validity of the factors by examining their relationships with other outcomes (i.e., stigma and unit cohesion). The dimensionality analyses included the explained common variance (ECV), internal-consistency reliability (coefficients ω and ωh), the H index, factor determinacy (FD), Percentage of Reliable Variance (PUC), Percentage of Reliable Variance (PRV), and Average Relative Parameter Bias (ARPB). These values were calculated using the Microsoft Excel calculator developed by Dueber (2017).

Results

Factor structure

The model fit estimates for the unidimensional, correlated traits, and bifactor models, with and without the method factor are presented in Table 2. Model fit for both the first-order and the bifactor models were good, and the inclusion of the method factor substantially improved model fit. The model fit indices favored the bifactor model, with method factor, over the other models. Thus, this factor structure was selected as the optimal model for further analysis (see Table 3 for the standardized factor loadings). Overall, the factor loadings for the general factor were large. Notable exceptions were the loadings for the four items affected by the wording method factor. Based on the ECV, the general factor accounted for the largest percentage of variance. Internal-consistency reliability, when moving from ω to ωh indicated that the majority of the reliability in the public and self-stigma specific factors was the result of the general factor (i.e., 84% of the reliable variance in total scores is attributable to the general factor). In terms of construct reliability, measured by the H index, the general factor and the public stigma-specific factor met the criterion of ≥0.70 (Hancock & Mueller, 2001) to indicate a reasonably defined latent variable based on the indicators. Based on FD, only the general factor met the criterion of ≥0.90 (Gorsuch, 1974).

Table 2.

Model fit statistics for unidimensional, first-order, and bifactor models with and without wording method factor.

Model χ2, df RMSEA [90% CI] CFI TLI SRMR
Unidimensional 2027.10, 299 0.13 [0.12, 0.13] 0.93 0.92 0.08
 Method factor 1850.67, 295 0.12 [0.12, 0.13] 0.93 0.93 0.08
Correlated traits 975.86, 298 0.08 [0.08, 0.09] 0.97 0.97 0.05
 Method factor 801.10, 294 0.07 [0.07, 0.08] 0.98 0.98 0.04
Bifactor 717.31, 273 0.07 [0.06, 0.08] 0.98 0.98 0.04
 Method factor 693.19, 269 0.07 [0.06, 0.07] 0.98 0.98 0.03
Open in a new tab

df = degrees of freedom; RMSEA = root mean squared error of approximation; CI = confidence interval; CFI = comparative fit index; TLI = Tucker-Lewis index; AIC = Akaike information criterion; BIC = Bayes information criterion.

Table 3.

Standardized factor loadings for the general and specific factors.

Item g Pub Self Rev
2 0.65 0.50    
3 0.26 0.32   0.36
4 0.63 0.46    
5 0.65 0.46    
6 0.75 0.46    
8 0.66 0.53    
10 0.49 0.52    
11 0.64 0.42    
12 0.65 0.36    
14 0.78 0.36    
17 0.79 0.51    
19 0.74 0.50    
20 0.74 0.28    
21 0.79 0.18    
23 0.69 0.43    
24 0.80 0.47    
1 0.76   0.32  
7 0.76   0.41  
9 0.28   0.53 0.52
13 0.77   0.16  
15 0.85   0.38  
16 0.81   0.37  
18 0.86   0.36  
22 0.88   0.25  
25 0.43   0.31 0.29
26 0.35   0.43 0.45
ECV 0.71 0.17 0.08 0.04
ω/ωs 0.98 0.97 0.95 0.73
ωh/ωhs 0.82 0.28 0.20 0.32
H 0.97 0.79 0.62 0.46
FD 0.96 0.86 0.74 0.72
Open in a new tab

g = general stigma factor; pub = public stigma subscale; self = self-stigma subscale; rev = reverse-coded item method factor; ω = model-based internal consistency reliability estimates (ω = general factor; ωs = specific factor; ωh = general factor hierarchical; ωhs = specific factor hierarchical); H = H index; FD = factor determinacy.

In addition to examining explained common variance, it is important to also examine the percentage of uncontaminated correlations (PUC) to evaluate the bias in potentially forcing a unidimensional model (Rodriguez et al., 2016). PUC values above .80 implicate less importance on ECV values. Conversely, when PUC values are observed to be lower than .80, and also given ECV values greater than .60 and ωH value greater than .70, there is evidence to suggest that potential multidimensionality does not prohibit the implication of a unidimensional measure (Reise et al., 2013). The observed PUC for the current measure is .49, calculated by achieving the ratio between the number of correlations between items on both of the specific factor (i.e., public and self-stigma) and the total number of possible correlations. As the observed ECV value is .71, and the ωH is .82, it suggests that the unidimensional structure of the data is not a product of bias.

Percentage of Reliable Variance (PRV) was also calculated. The PRV is an indicator of the reliable variance the specific factors may entail separate from the general factor. The PRV values for the general factor and specific public and self-stigma factors were .83, .29, and .21, respectively, indicating that the amount of reliable variance found in the specific factors independent of the general factor is substantially less than the percent of reliable variance due to the general factor. Finally, the Average Relative Parameter Bias (ARPD) was calculated to assess the fit of the final bifactor solution against a forced unidimensional solution. A confirmatory factor analysis with a principal axis factoring extraction and direct oblimin rotation was used to force a unidimensional solution. Following Hammer’s (2016) calculation, we found the ARPB to be 13.4%, within the range of acceptable parameter biases (Rodriguez et al., 2016).

Construct validity

Next, we tested the extent to which the general stigma factor and the public and self-stigma specific factors are uniquely associated with the three outcome variables (i.e., stigma, peer cohesion, and leader cohesion). Specifically, a bifactor model containing the three outcomes (SSRPH, UC-Peer, and UC-Leader) was predicted by the three latent factors of interest (i.e., general, self, and public stigma) while controlling for the method factor accounting for negatively worded items. The overall model fit was good (CFI = .98, TLI = .98, SRMR = .03, RMSEA = .06 [.05 .07]). The general factor showed the strongest link with SSRPH scores (β = .78, p < .001), though the public stigma-specific factor was still significant (β = .20, p < .001). For UC-Peers (β = −.12, p = .05) and UC-Leader (β = −.34, p < .001) the general factor was the only factor that was significantly associated.

Discussion

Despite the availability of evidence-based treatments, only 40% of Service members in need of MH care seek help (Sharp et al., 2015). The most cited reason being the stigma associated with seeking such care. The Military Stigma Scale (MSS; Skopp et al., 2012) was designed to provide comprehensive assessment of help-seeking stigma, specifically among service members. Overall, this study provides additional psychometric evidence for the MSS. Examination of unidimensional, two-factor, and bifactor models revealed that a bifactor model with a general (overall stigma) and two specific factors (public and self-stigma) provided the best fit to the data. Ancillary reliability analyses also supported the MSS measuring a broad stigma factor associated with seeking MH care in the military. Consistent with this, our subsequent bifactor model analyses comparing the associations between the three factors of the MSS (i.e., general, self, and public) and stigma-related outcomes showed that the general factor showed largest associations with other measures. Specifically, the MSS had a strong positive relationship with the Social Stigma of Receiving Professional Help scale (Komiya et al., 2000). Additionally, the MSS general factor also showed small to moderate relationships with military identity (i.e., unit cohesion). In total, the current findings support the validity of the scale as a measurement of stigma associated with seeking psychological help in military samples and can be used to assess military help-seeking stigma and to evaluate results of programs designed to reduce stigma.

The findings that the MSS assesses a broad general factor are particularly interesting, given previous researchers having emphasized the importance of conceptualizing stigma for seeking MH care in terms of both public and self-stigma, separately (e.g., Coleman et al., 2017; Vogt, 2011), with some suggesting that self-stigma may be even more important with respect to MH care-seeking behavior than public stigma (Hantzi et al., 2019; Jennings et al., 2015; Wade et al., 2015). For example, in civilian populations, self-stigma and not public stigma of seeking help, was found to predict MH care seeking behavior (Jennings et al., 2015; Ludwikowski et al., 2009; Vogel et al., 2010, 2006). In addition, help-seeking self-stigma, on its own, has been found to decrease use of MH care seeking behavior for service members over a two-year period (Seidman et al., 2019). However, as noted above, previous help-seeking stigma scales (validated on civilian samples), largely assess generic questions (i.e., “Seeing a psychologist for emotional or interpersonal problems carries social stigma”), while the MSS directly assesses military-specific public stigma concerns (e.g., “I would be given less responsibility, if chain of command knew I was seeing a mental health provider”). The decreased salience of self-stigma, separate from public stigma, may reflect this increased attention to having public stigma items which reflect military specific contexts. This finding suggests that general stigma factors specific to military concerns may need to be clinically addressed. Interventions, therefore, should focus not only on individuals but also their perceptions of messages coming from their chain of command and from peers. These implications may be best suited for active duty personnel compared to Veterans because such concerns reflect nuances in active military culture (e.g., perceiving that peers or commanders will think less of them for seeking MH care). Non-active duty personnel may experience different forms of public and self-stigma from those in active-duty, and thus have concerns that reflect the social environment shared with civilians. One such example may be the expectation that civilians perceive Veterans to have psychological disorders such as post-traumatic stress disorder and thus be violent or aggressive (Mittal et al., 2013).

Limitations and future directions

Although this research study has several strengths, some limitations should be noted. One of the limitations of this research is the lack of a longitudinal component. Understanding the degree to which the MSS can predict expected relationships over time (i.e., use of services) are integral to assessing the sensitivity of the MSS to changes, which will be important to consider in monitoring the effects of military-specific stigma reduction interventions. Monitoring the effects of military-specific stigma reduction interventions and understanding the longitudinal role of stigma in help seeking are ultimately integral to service member access to MH care. Additional research is also needed to clarify the role of the general and specific factors and their relations to other relevant outcomes (i.e., intentions, MH seeking behavior, self-disclosure to peers). Whereas public and self-stigma have been differentiated in civilian samples, it is possible that these constructs are less differentiated within the military, or that the scale items do not adequately capture the constructs of interest. For example, as noted above, self- and public stigma of seeking MH care may be less differentiated among military personnel or perhaps additional military specific self-stigma items may be required.

There are many costs associated with the delayed MH treatment of Service members. A lack of focus on ameliorating stigma of seeking MH care among service members may lead to additional burden on the Veterans Affairs Healthcare system. MH Stigma is associated with lasting negative effects on well-being, even when MH symptoms remit, and may counteract positive treatment outcomes (Link et al., 1997). Conversely, reductions in MH stigma relate to lasting positive treatment effects and of quality of life (Rosenfield, 1997). As we continue to follow the lead of the US military by identifying specific member MH concerns, it is important to proceed using the well-developed and sound instruments specifically designed to measure military-related stigma. To meet this need, the MSS was developed and appears to be a valid, reliable measure of the stigma associated with seeking help in military samples. As such, the MSS should allow for the future examination of specific clinical interventions to decrease these stigmas in military settings.

Acknowledgments

The authors wish to extend our appreciation to Dr. Kevin Kipp and Dr. Edward Hickling for their support of this research project.

Disclaimer

The opinions or assertions contained herein are the private views of the authors and are not to be construed as official or reflecting the views of the Department of Army or the Department of Defense

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The authors acknowledge that they will share data if reasonable requests are made.

References

  1. Boyd, J. E., Hayward, H., Bassett, E. D., & Hoff, R. (2016). Internalized stigma of mental illness and depressive and psychotic symptoms in homeless veterans over 6 months. Psychiatry Research, 240, 253–259. 10.1016/j.psychres.2016.04.035 [DOI] [PubMed] [Google Scholar]
  2. Britt, T. W., & Dawson, C. R. (2005). Predicting work-family conflict from workload, job attitudes, group attributes, and health: A longitudinal study. Military Psychology, 17(3), 203–227. 10.1207/s15327876mp1703_5 [DOI] [Google Scholar]
  3. Britt, T. W., Dickinson, J., Moore, D., Castro, C. A., & Adler, A. B. (2007). Correlates and consequences of morale versus depression under stressful conditions. Journal Occupational Health Psychology, 12(1), 34–47. 10.1037/1076-8998.12.1.34 [DOI] [PubMed] [Google Scholar]
  4. Britt, T. W., Jennings, K. S., Cheung, J. H., Pury, S. L. C., & Zinzow, H. M. (2015). The role of different stigma perceptions in treatment seeking and dropout among active duty military personnel. Psychiatric Rehabilitation Journal, 38(2), 142–149. 10.1037/prj0000120 [DOI] [PubMed] [Google Scholar]
  5. Brown, N. B., & Bruce, S. E. (2016). Stigma, career worry, and mental illness symptomatology: Factors influencing treatment-seeking for operation enduring freedom and operation enduring freedom soldier and veterans. Psychological Trauma, 8(3), 276–283. 10.1037/tra0000082 [DOI] [PubMed] [Google Scholar]
  6. Cheung, G. W., & Lau, R. S. (2012). A direct comparison approach for testing measurement invariance. Organizational Research Methods, 15(2), 167–198. 10.1177/1094428111421987. [DOI] [Google Scholar]
  7. Clement, S., Schauman, O., Graham, T., Maggioni, F., Evans-Lacko, S., Bezborodovs, N., Morgan, C., Rüsch, N., Brown, J. S. L., & Thornicroft, G. (2014). What is the impact of mental health-related stigma on help-seeking? A systematic review of quantitative and qualitative studies. Psychological Medicine, 45(1), 11–27. 10.1017/S0033291714000129 [DOI] [PubMed] [Google Scholar]
  8. Coleman, S. J., Stevelink, A. M., Hatch, S. L., Denny, J. A., & Greenberg, N. (2017). Stigma-related barriers and facilitators to help seeking mental health issues in the armed forces: A systematic review and thematic synthesis of qualitative literature. Psychological Medicine, 47(11), 1880–1892. 10.1017/S0033291717000356 [DOI] [PubMed] [Google Scholar]
  9. Corrigan, P. W., & Watson, A. C. (2002). The paradox of self‐stigma and mental illness. Clinical Psychology: Science and Practice, 9(1), 35–53. 10.1093/clipsy.9.1.35. [DOI] [Google Scholar]
  10. DiStefano, C., & Motl, R. W. (2006). Further investigating method effects associated with negatively worded items on self-report surveys. Structural Equation Modeling, 13(3), 440–464. 10.1207/s15328007sem1303_6 [DOI] [Google Scholar]
  11. Dueber, D. M. (2017). Bifactor indices calculator: A Microsoft Excel-based tool to calculate various indices relevant to bifactor CFA models. 10.13023/edp.tool.01. [DOI]
  12. Fox, A. B., Smith, B. N., & Vogt, D. (2016). The relationship between anticipated stigma and work functioning for individuals with depression. Journal of Social and Clinical Psychology, 35(10), 883–897. 10.152/jscp.2016.35.10.883 [DOI] [Google Scholar]
  13. Gorsuch, R. L. (1974). Factor analysis. Saunders. [Google Scholar]
  14. Greene-Shortridge, T. M., Britt, T. W., & Castro, C. A. (2007). The stigma of mental health problems in the military. Military Medicine, 172(2), 157–161. 10.7205/MILMED.172.2.157 [DOI] [PubMed] [Google Scholar]
  15. Hammer, J. H. (2016, October). Average relative parameter bias (ARPB) calculator: A Microsoft Excel-based tool to calculate the ARPB statistic. http://DrJosephHammer.com/
  16. Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability within latent variable systems. In Structural equation modeling: Present and future (pp. 195–216). Lincolnwood, IL: Scientific Software International. [Google Scholar]
  17. Hantzi, A., Anagnostopoulos, F., & Alexiou, E. (2019). Attitudes towards seeking psychological help: An integrative model based on contact, essentialist beliefs about mental illness, and stigma. Journal of Clinical Psychology in Medical Settings, 26(2), 142–157. 10.1007/s10880-018-9573-8 [DOI] [PubMed] [Google Scholar]
  18. Hoerster, K. D., Malte, C. A., Imel, Z. E., Ahmed, Z., Hunt, S. C., & Jakupcak, M. (2012). Association of perceived barriers with prospective use of VA mental health care among Iraq and Afghanistan veterans. Psychiatric Services, 63(4), 380–382. 10.1176/appi.ps.201100187 [DOI] [PubMed] [Google Scholar]
  19. Hoge, C. W., Castro, C. A., Messer, S. C., McGure, D., Cotting, D. I., & Koffman, R. L. (2004). Combat duty in Iraq and Afghanistan, mental health problems, and barriers to care. The New England Journal of Medicine, 351(1), 13–22. 10.1056/NEJMoa040603 [DOI] [PubMed] [Google Scholar]
  20. Hoge, C. W., Grossman, S. H., Auchterlonie, J. L., Riviere, L. A., Milliken, C. S., & Wilk, J. E. (2014). PTSD treatment for soldiers after combat deployment: Low utilization of mental health care and reasons for dropout. Psychiatric Services, 65(8), 997–1004. 10.1176/appi.ps.201300307 [DOI] [PubMed] [Google Scholar]
  21. Holzinger, K. J., & Swineford, F. (1937). The bi-factor method. Psychometrika, 2(1), 41–54. 10.1007/BF02287965 [DOI] [Google Scholar]
  22. Hom, M. A., Stanley, I. H., Schnedider, M. E., & Joiner, T. E. (2017). A systematic review of help-seeking and mental health utilization among military service members. Clinical Psychology Review, 53, 59–78. 10.1016/j.cpr.2017.01.008 [DOI] [PubMed] [Google Scholar]
  23. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. 10.1080/10705519909540118 [DOI] [Google Scholar]
  24. Iverson, A. C., van Staden, L., Hughes, J. H., Greenberg, N., Hotopf, M., Rona, R. J., … Fear, N. T. (2011). The stigma of mental health problems and other barriers to care in UK armed forces. BMC Health Services, 11(1), 31. 10.1186/1472-6963-11-31 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Jennings, K. S., Cheung, J. H., Britt, T. W., Goguen, K. N., Jeffirs, S. M., Peasley, A. L., & Lee, A. C. (2015). How are perceived stigma, self-stigma, and self-reliance related to treatment-seeking? A three-path model. Psychiatric Rehabilitation Journal, 38(2), 109–116. 10.1037/prj0000138 [DOI] [PubMed] [Google Scholar]
  26. Jennings, K. S., Zinzow, H. M., Britt, T. W., Cheung, J. H., & Pury, C. L. S. (2016). Correlates and reasons for mental health treatment dropout among active duty soldiers. Psychological Services, 13(4), 356–363. 10.1037/ser0000082 [DOI] [PubMed] [Google Scholar]
  27. Jones, N., Campion, B., Keeling, M., & Greenberg, N. (2018). Cohesion, leadership, mental health stigmatisation and perceived barriers to care in UK military personnel. Journal of Mental Health, 27(1), 10–18. 10.3109/09638237.2016.1139063 [DOI] [PubMed] [Google Scholar]
  28. Komiya, N., Good, G. E., & Sherrod, N. B. (2000). Stigma scale for receiving psychological help (SSRPH) [Database record]. APA PsycTests. 10.1037/t23348-000 [DOI] [Google Scholar]
  29. Kulesza, M., Watkins, K. E., Ober, A. J., Osilla, K. C., & Ewing, B. (2017). Internalized stigma as an independent risk factor for substance use problems among primary care patients: Rationale and preliminary support. Drug and Alcohol Dependence, 180, 52–55. 10.1016/j.drugalcdep.2017.08.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Link, B. G., Struening, E. L., Rahav, M., Phelan, J. C., & Nuttbrock, L. (1997). On stigma and its consequences: Evidence from a longitudinal study of mean with dual diagnoses and substance abuse. Journal of Health and Socieal Behavior, 38(June), 177–190 . 10.2307/2955424 [DOI] [PubMed] [Google Scholar]
  31. Ludwikowski, W. M. A., Vogel, D., & Armstrong, P. I. (2009). Attitudes toward career counseling: The role of public and self-stigma. Journal of Counseling Psychology, 56(3), 408–416. 10.1037/a0016180 [DOI] [Google Scholar]
  32. Mittal, D., Drummond, K. L., Blevins, D., Curran, G., Corrigan, P., & Sullivan, G. (2013). Stigma associated with PTSD: Perceptions of treatment seeking combat veterans. Psychiatric Rehabilitation Journal, 36(2), 86–92. 10.1037/h0094976 [DOI] [PubMed] [Google Scholar]
  33. Murray, A. L., & Johnson, W. (2013). The limitations of model fit in comparing bi-factor versus higher-order models of human cognitive ability structure. Intelligence, 41(5), 407–422. 10.1016/j.intell.2013.06.004 [DOI] [Google Scholar]
  34. Muthén, L. K., & Muthén, B. O. (1998–2017). Mplus user’s guide (Eighth Edition). Muthén & Muthén. [Google Scholar]
  35. Ociskova, M., Prasko, J., Vrbova, K., Kasalova, P., Holubova, M., Grambal, A., & Machu, K. (2018). Self-stigma and treatment effectiveness in patient with anxiety disorders – A mediation analysis. Neuropsychiatric Disease and Treatment, 14, 383–392. 10.2147/NDT.S152208 [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Office of the Surgeon Multinational Force-Iraq and Office of the Surgeon General United States Army Medical Command . (2006, November 17). Mental health advisory team (MHAT) IV operation Iraqi freedom 05-07.
  37. Podsakoff, P. M., & MacKenzie, S. B. (1994). An examination of the psychometric properties and nomological validity of some revised and reduced substitutes for leadership scales. Journal of Applied Psychology, 79(5), 702–713. 10.1037//0021-9010.79.5.702 [DOI] [Google Scholar]
  38. Reise, S. P., Bonifay, W. E., & Haviland, M. G. (2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95(2), 129–140. 10.1080/00223891.2012.725437 [DOI] [PubMed] [Google Scholar]
  39. Reise, S. P., Moore, T. M., & Haviland, M. G. (2010). Bifactor models and rotations: Exploring the extent to which multidimensional data yield univocal scale scores. Journal of Personality Assessment, 92(6), 544–559. 10.1080/00223891.2010.496477 [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016). Applying bifactor statistical indices in the evaluation of psychological measures. Journal of Personality Assessment, 98(3), 223–237. 10.1080/00223891.2015.1089249 [DOI] [PubMed] [Google Scholar]
  41. Rosenfield, S. (1997). Labeling mental illness: The effects of received service and perceived stigma on life satisfaction. American Sociological Review, 62(4), 660–672. 10.2307/2657432 [DOI] [Google Scholar]
  42. Seidman, A. J., Wade, N. G., Vogel, D. L., & Armistead-Jehle, P. (2019). The impact of stigma on behavioral health care utilization among active duty service members. Military Psychology, 31(1), 11–17. 10.1080/08995605.2018.1522927 [DOI] [Google Scholar]
  43. Sharp, M., Fear, N. T., Rona, R. J., Wessely, S., Greenberg, N., Jones, N., & Goodwin, L. (2015). Stigma as a barrier to seeking health care among military personnel with mental health problems. Epidemiologic Reviews, 37(1), 144–162. 10.1093/epirev/mxu012 [DOI] [PubMed] [Google Scholar]
  44. Skopp, N. A., Bush, N. E., Vogel, D. L., Wade, N. G., Sirotin, A. P., McCann, R. A., & Metzger-Abamukong, M. J. (2012). Development and initial testing of measure of public and self-stigma in the military. Journal of Clinical Psychology, 68(9), 1036–1047. 10.1002/jclp.21889 [DOI] [PubMed] [Google Scholar]
  45. Topkaya, N. (2014). Gender, self-stigma, and public stigma in predicting attitudes toward help-seeking. Educational Sciences: Theory & Practice, 14(2), 480–487. 10.12738/estp.2014.2.1799 [DOI] [Google Scholar]
  46. Tucker, J. R., Hammer, J. H., Vogel, D. L., Bitman, R. L., Wade, N. G., & Maier, E. J. (2013). Disentangling self-stigma: Are mental illness and help-seeking self-stigmas the same? Journal of Counseling Psychology, 60(4), 520–531. 10.1037/a0033555 [DOI] [PubMed] [Google Scholar]
  47. VanSickle, M., Werbel, A., Perera, K., Pak, K., DeYoung, K., & Ghahramanlou-Holloway, M. (2016). Perceived barriers to seeking mental health care among United States Marine corps noncommissioned officers serving as gatekeepers for suicide prevention. Psychological Assessment, 28(8), 1020. 10.1037/pas0000212 [DOI] [PubMed] [Google Scholar]
  48. Vogel, D. L., Shechtman, Z., & Wade, N. G. (2010). The role of public and self-stigma in predicting attitudes toward group counseling. The Counseling Psychologist, 38(7), 904–922. 10.1177/0011000010368297. [DOI] [Google Scholar]
  49. Vogel, D. L., Wade, N. G., & Haake, S. (2006). Measuring the self-stigma associated with seeking psychological help. Journal of Counseling Psychology, 53(3), 325. 10.1037/0022-0167.53.3.325 [DOI] [Google Scholar]
  50. Vogt, D., Di Leone, B. A., Wang, J. M., Sayer, N. A., Pineles, S. L., & Litz, B. T. (2014). Endorsed and anticipated stigma inventory (EASI): A tool for assessing beliefs about mental illness and mental health treatment among military personnel and veterans. Psychological Services, 11(1), 105–113. 10.1037/a0032780 [DOI] [PubMed] [Google Scholar]
  51. Vogt, D. (2011). Mental health-related beliefs as a barrier to service use for military personnel and veterans. Psychiatric Services, 62(2), 135–142. 10.1176/ps.62.2.pss6202_0135 [DOI] [PubMed] [Google Scholar]
  52. Wade, N. G., Vogel, D. L., Armistead-Jehle, P., Meit, S. S., Heath, P. J., & Strass, H. A. (2015). Modeling stigma, help-seeking attitudes, and intentions to seek behavioral healthcare in a clinical military sample. Psychiatric Rehabilitation Journal, 38(2), 135–141. 10.1037/prj0000131 [DOI] [PubMed] [Google Scholar]
  53. Wright, K. M., Cabrera, O. A., Bliese, P. D., Alder, A. B., Hoge, C. W., & Castro, C. A. (2009). Stigma and barriers to care in soldiers postcombat. Psychological Services, 6(2), 108–116. 10.1037/a0012620 [DOI] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

The authors acknowledge that they will share data if reasonable requests are made.

Articles from Military Psychology are provided here courtesy of Division of Military Psychology of the American Psychological Association and Taylor & Francis

RESOURCES