Mental health screening: severity and cut-off point sensitivity of the Athlete Psychological Strain Questionnaire in male and female elite athletes

Abstract

Objectives

To examine the sensitivity/specificity of the Athlete Psychological Strain Questionnaire (APSQ) in both male and female elite athletes, and also assess internal consistency and convergent/divergent validity, and determine discriminative validity relative to current injury status.

Methods

Data were provided by 1093 elite athletes (males n=1007; females n=84). Scale validity and reliability values were benchmarked against validated measures of general psychological distress and well-being. ROC curve analysis determined a range of optimal severity cut-points.

Results

Bias-corrected area under curve (AUC) values supported three APSQ cut-points for moderate (AUC=0.901), high (AUC=0.944) and very high (AUC=0.951) categories. APSQ total score Cronbach coefficients exceeded those observed for the Kessler 10 (K-10). Gender ×injury status interactions were observed for the APSQ total score and K-10, whereby injured female athletes reported higher scores relative to males and non-injured female counterparts.

Conclusion

By providing a range of cut-off scores identifying those scoring in the marginal and elevated ranges, the APSQ may better facilitate earlier identification for male and female elite athletes vulnerable to mental health symptoms and developing syndromes. Use of the APSQ may support sports medicine practitioners and allied health professionals to detect early mental ill health manifestations and facilitate timely management and ideally, remediation of symptoms.

What are the new findings?

• The Athlete Psychological Strain Questionnaire provides cut-off scores to discriminate between ‘moderate’, ‘high’ and ‘very high’ levels of psychological distress in elite athletes.

• Female athletes suffering from an injury tend to experience greater psychological distress relative to uninjured female or male athletes.

• The Athlete Psychological Strain Questionnaire can be used to support sports medicine practitioners and allied professionals to detect early mental health symptoms in elite athletes.

There is growing interest into athlete mental health, as evidenced, in part, by the International Olympic Committee’s (IOC) Consensus Statement on mental health in elite athletes.¹ While self-report data provided by elite athletes indicates comparable prevalence rates of mental health symptoms to the general population (eg, anxiety/depression and sleep disorders),^{2 3} athletes are also exposed to various sports-related stressors not observed in the general population (such as susceptibility to serious or chronic injury, performance impairments, popular and social media scrutiny and early or unplanned career termination).¹ These stressors tend to be experienced during the peak age of onset for mental health disorders,⁴ which when combined, may increase athletes’ susceptibility to psychological distress and mental health symptoms.

Despite this risk, there is a lack of well-validated athlete-specific screening tools that can assist with early detection of elite athletes experiencing psychological distress. Such screening tools may facilitate more timely intervention in the course of developing mental health symptoms. Rice et al⁵ developed the multidimensional Athlete Psychological Strain Questionnaire (APSQ), which was modelled on the widely used and validated Kessler 10 (K-10).⁶ The APSQ is the recommended triage screening tool within the IOC’s Sports Mental Health Assessment Tool, with a positive APSQ result leading to further symptom specific screening and assessment.⁷

The APSQ’s psychometric evaluation included analyses demonstrating its reliability and validity among a large, representative sample of 1007 male elite athletes from three national Australian sports (Australian football, cricket and soccer), providing a single cut-off score. However, there is a need to provide support and coaching staff working in the elite sport environment with cut-off scores for less severe cases. Such earlier case identification may better facilitate timely referral and intervention, especially among athletes at elevated risk of impaired well-being, such as injured athletes.⁸ This short report extends earlier APSQ validation work by providing a range of less severe cut-off scores for mental health symptoms, while also including a female elite athlete sample. The aims of this study were to:

1. Explore the sensitivity/specificity of the APSQ in both male and female elite athletes with regard to their scores on the K-10.

2. Assess internal consistency and convergent/divergent validity of APSQ in male and female elite athletes.

3. Assess the discriminative validity of the APSQ to differentiate between injured and uninjured athletes in male and female athletes.

Method

Participants

Data were collected from a sample of 1091 elite athletes (males n=1007; females n=84) from three national Australian sports; Australian football (males only n=676; 84% response rate), cricket (males n=126; 82% response rate, females n=54; 53% response rate), soccer (males n=203; 78% response rate, females national squad n=30; 100% response rate) N’s do not total 100% due to missing data. All athletes were aged 18 years and over, identified as binary gender, and no exclusion criteria were applied (see table 1).

Table 1.

Sample demographics and scale reliabilities

Variable		Males	Females
		(n=1007)	(n=84)
Age	M (SD)	23.7 (4.1)	23.8 (5.2)
Highest education completed
High school	n (%)	797 (79.3)	39 (46.4)
Trade or technical education	n (%)	143 (14.2)	14 (16.7)
Completed a university degree	n (%)	65 (6.5)	29 (34.5)
Currently studying	n (%)	431 (42.8)	46 (54.8)
Marital status
Partnered/de-facto	n (%)	512 (50.9)	34 (40.4)
Single	n (%)	332 (33.0)	47 (56.0)
Married	n (%)	161 (16.0)	3 (3.6)
Separated	n (%)	2 (0.2)	0 (0)
Moved from home to play sport	n (%)	468 (69.0)	24 (28.6)
Another region within state	n (%)	134 (13.3)	5 (6.0)
Another state within Australia	n (%)	451 (44.8)	16 (19.0)
Another country	n (%)	65 (6.5)	3 (3.6)
Living situation
Renting	n (%)	464 (46.1)	21 (25.0)
My own home with mortgage	n (%)	277 (27.5)	17 (20.2)
Living with a host family	n (%)	56 (5.6)	2 (2.4)
My own home outright	n (%)	43 (4.3)	0 (0)
Family home	n (%)	152 (15.1)	38 (45.2)
Other	n (%)	14 (1.4)	6 (7.1)
Cultural background
Born in Australia	n (%)	916 (91.1)	75 (89.3)
Aboriginal or Torres Strait Islander	n (%)	69 (6.9)	5 (6.8)
Maori	n (%)	3 (0.3)	0 (0)
Time in current elite competition
<1 year	n (%)	108 (10.7)	13 (15.5)
1–5 years	n (%)	500 (49.7)	51 (60.7)
>5 years	n (%)	399 (39.6)	20 (23.8)
Current injury status
Uninjured	n (%)	717 (71.3)	61 (72.6)
Currently injured/on adapted training programme	n (%)	288 (28.7)	22 (26.2)
Scale reliability (cronbach coefficient)
APSQ self-regulation	α	0.82	0.74
APSQ performance	α	0.74	0.72
APSQ external coping	α	0.68	0.80
APSQ total score	α	0.87	0.84
K-10	α	0.86	0.80
WEMWBS	α	0.94	0.93

APSQ, Athlete Psychological Strain Questionnaire; K-10, Kessler 10; WEMWBS, Warwick-Edinburgh Mental Well-Being Scale.

Materials

Respondents completed three validated self-report scales; the 10-item APSQ⁵ assessing three domains of self-regulation difficulties, performance concerns and externalised coping, in addition to providing a total score; the 10-item Kessler Psychological Distress Scale⁶ providing a measure of general psychological distress and the 14-item Warwick-Edinburgh Mental Well-Being Scale assessing mental well-being (WEMWBS).⁹

Procedure

All contracted athletes for the participating sports were invited to participate in an an anonymous online survey, conducted in group-based sessions. Clinician researchers were present to introduce and oversee the data collection, which occurred via tablets. Data collection occurred nationally in local club-based facilities and was coordinated by the participating sports and clubs.

Statistical analyses

The present study used secondary data analysis for the n=1007 male athletes reported in Rice et al,⁵ and primary data analysis for the included n=84 female athletes. Cronbach alpha coefficients and Pearson correlations were calculated to assess internal consistency and validity (convergent and divergent validity). A series of receiver operating characteristic (ROC) curve analyses were undertaken to identify the sensitivity and specificity of the APSQ total score in discriminating levels of severity on the K-10. Three analyses were conducted, with the ROC curve state variable either ‘moderate’ distress and above (K-10=16+), ‘high’ distress and above (K-10=22+) or ‘very high’ distress (K-10=30+). For each of these analyses, Youden’s (J) index was identified, which is a coefficient maximising the sensitivity and specificity of the respective cut-point.¹⁰ Previously, Rice et al⁵ reported a single preliminary cut-off score on the APSQ for identifying athletes who may be at risk of psychological distress and warrant mental health intervention. In order to facilitate even earlier detection for less severe cases, we sought to identify a series of APSQ cut-points. To guide interpretation, we used the accepted area under curve (AUC) values of >0.90 (very good), >0.80 (good) and >0.70 (fair).¹¹ Between-group analyses examined differences by injury and gender status. Internal validation of AUC values and between-groups outcomes were undertaken with bootstrapping procedures using 10 000 resamples. Analyses were conducted using SPSS V.26.0, with bias-corrected bootstrapping conducted in STATA V.15.0.

Results

In addition to the sample of n=1007 male elite athletes reported in Rice e t al,⁵ this analysis incorporates data from 84 female elite athletes (of whom n=22 were currently injured). Internal consistency values for the APSQ were all in the acceptable range (see table 1) for both male and female athletes, with the APSQ total score values exceeding those observed for the K-10.

ROC curves and APSQ cut-offs

A series of ROC curve analyses were undertaken to identify the sensitivity and specificity of the APSQ total score in discriminating levels of severity on the K-10. The diagnostic odds ratio (ie, positive likelihood ratio ÷ negative likelihood ratio) of the three cut-off points indicted that the very high APSQ cut-off was better at discriminating cases than was the high or moderate cut-offs, both of which reported lower sensitivity and specificity, with bootstrap results validating the AUC values (see table 2).

Table 2.

APSQ ROC curve and cut-point values (including bias-corrected AUC and 95% CIs)

APSQ cut-off	APSQ positive (% distribution)	Sensitivity, %	Specificity, %	Diagnostic OR	AUC (95% CI)	BC AUC (95% CI)	Youden’s (J)Index	K-10 category	K-10 positive (% distribution)
≥15 ‘Moderate’	n=407 (upper 38.0)	84.8	79.3	21.31	0.901*** (0.880 to 0.922)	0.901*** (0.879 – 0.920)	0.641	‘Moderate’	n=289 (upper 26.7)
≥17 ‘High’	n=289 (upper 23.2)	93.4	79.2	54.06	0.944*** (0.926 to 0.962)	0.944*** (0.924 to 0.960)	0.726	‘High’	n=91 (upper 8.3)
≥20 ‘Very high’	n=170 (upper 12.8)	95.5	85.8	126.79	0.951*** (0.918 to 0.984)	0.951*** (0.910 to 0.977)	0.813	‘Very high’	n=22 (upper 2.0)

BC 95% CI with 10 000 bootstrap resamples.

***p<0.001.

APSQ, Athlete Psychological Strain Questionnaire; AUC, area under curve; BC, Bias corrected; K-10, Kessler 10.

Convergent and divergent validity

Pearson correlations were evaluated separately by gender (see table 3). Moderate-to-strong positive correlations were observed between the APSQ domains and the K-10, providing support for convergent validity. Conversely, moderate negative correlations were observed between the APSQ domains and the WEMWBS, supporting divergent validity. Bootstrapping indicated these correlations were internally valid (see online supplementary table 1).

Table 3.

APSQ, K-10 and WEMWBS correlations by gender

	APSQ self-regulation	APSQ performance	APSQ external coping	APSQ total	K-10 total	WEMWBS
APSQ self-regulation	–	0.506***	0.528***	0.821***	0.716***	−0.463***
APSQ performance	0.635***	–	0.563***	0.883***	0.501***	−0.346***
APSQ external coping	0.580***	0.499***	–	0.757***	0.355*	−0.226*
APSQ total	0.891***	0.893***	0.719***	–	0.658***	−0.437***
K-10 total	0.768***	0.643***	0.564***	0.787***	–	−0.574***
WEMWBS	−0.495***	−0.440***	−0.363***	−0.520***	−0.481***	–

Female coefficients bolded above the diagonal.

*P<0.05, **P<0.01, ***P<0.001

APSQ, Athlete Psychological Strain Questionnaire; K-10, Kessler 10; WEMWBS, Warwick-Edinburgh Mental Well-Being Scale.

Gender and current injury effects

Between-groups analysis was conducted to examine gender (male, female)×injury (injured, uninjured) effects (see table 4). Gender ×injury interactions were observed for the APSQ total score F(1, 1067)=17.82, p<0.001, η²=0.016, (bootstrap p=0.001), and also the K-10 F(1, 1082)=13.38, p<0.001, η²=0.012, (bootstrap p=0.001). There was a significant main effect for gender on the WEMWBS F(1, 1064)=18.31, p<0.001, η²=0.017, (bootstrap p<0.001) and a trend-level effect for injury F(1, 1064)=3.86, p=0.050, η²=0.014, (bootstrap p<0.001) but not a significant interaction (p=0.197; bootstrap p=0.225). For the APSQ and K-10 gender ×injury interactions, females experiencing current injury tended to report higher scores, relative to uninjured females or males.

Table 4.

Means, SDS and 95% CIs and bias-corrected 95% CIs by gender and injury status

Injured			Uninjured
M (SD)	95% CI	BC 95% CI	M (SD)	95% CI	BC 95% CI
Male
APSQ self-regulation	5.97 (2.76)	(5.64 to 6.28)	(5.65 to 6.29)	5.68 (2.34)	(5.50 to 5.84)	(5.51 to 5.88)
APSQ performance	6.81 (3.00)	(6.46 to 7.16)	(6.47 to 7.17)	6.26 (2.63)	(6.06 to 6.45)	(6.06 to 6.45)
APSQ external coping	2.58 (1.29)	(2.42 to 2.72)	(2.43 to 2.50)	2.46 (1.05)	(2.39 to 2.54)	(2.39 to 2.50)
APSQ total score	15.36 (6.10)	(14.64 to 16.10)	(14.67 to 16.10)	14.40 (5.18)	(14.02 to 14.79)	(14.04 to 14.79)
K-10	14.08 (4.74)	(13.53 to 14.63)	(13.55 to 14.64)	14.19 (4.82)	(13.88 to 14.55)	(13.84 to 14.55)
WEMWBS	50.84 (9.09)	(49.77 to 51.90)	(49.72 to 51.89)	51.61 (8.83)	(50.96 to 52.27)	(50.97 to 52.27)
Female
APSQ self-regulation	9.14 (2.44)	(8.06 to 10.21)	(8.13 to 10.13)	6.04 (2.32)	(5.45 to 6.65)	(5.49 to 6.65)
APSQ performance	9.50 (4.26)	(7.61 to 11.39)	(7.80 to 11.37)	6.74 (2.73)	(6.04 to 7.44)	(6.08 to 7.45)
APSQ external coping	3.36 (2.38)	(2.31 to 4.42)	(2.46 to 4.41)	2.30 (0.74)	(2.11 to 2.48)	(2.12 to 2.48)
APSQ total score	22.00 (7.99)	(18.46 to 25.54)	(18.91 to 25.52)	15.08 (4.41)	(13.95 to 16.21)	(14.00 to 16.23)
K-10	19.36 (5.46)	(16.94 to 21.79)	(17.17 to 21.74)	14.97 (4.06)	(13.93 to 16.01)	(14.02 to 16.02)
WEMWBS	44.41 (9.13)	(40.36 to 48.46)	(40.41 to 48.24)	48.17 (8.80)	(45.89 to 48.46)	(45.95 to 50.39)

BC 95% CI=bias corrected 95% CI with 10 000 bootstrap resamples.

APSQ, Athlete Psychological Strain Questionnaire; K-10, Kessler 10; WEMWBS, Warwick-Edinburgh Mental Well-Being Scale.

Discussion

The need for validated athlete-specific questionnaires that are acceptable and appropriate for measuring mental health symptoms in the athlete environment is increasingly manifest.¹² The APSQ was designed specifically with the athletic context in mind, relative to behaviours that may indicate underlying distress in sports settings. The results of the present analyses not only support the use of the APSQ among both male and female elite athletes, but provide a validated range of cut-off points, reflecting moderate, high and very high scores. If measured repeatedly at regular intervals, the moderate cut-off point in particular may facilitate earlier case identification and subsequent intervention for athletes vulnerable to mental health symptoms and developing syndromes. The ROC curve AUC values for the three cut-off scores (all >0.90) indicate that the APSQ is ‘very good’ at correctly discriminating moderate, high and very high distress in both male and female athletes, benchmarked against the K-10. Professional sports organisations may consider use of the APSQ as a periodic screening tool, which could be implemented at key stages throughout the training or competitive season, or coinciding with athlete injury or career transition. Future research is needed to explore the prognostic value of the APSQ including associations with clinician diagnosed mental health disorders, in addition to exploring correlations between APSQ scores and current athletic performance or achievement.

Gender and injury effects

The results also indicated that currently injured female athletes reported the highest APSQ scores, as evidenced by the gender ×injury interactions. While we acknowledge that the sample included a relatively small number of injured female athletes (n=22), this result may in part reflect the less resourced supports and well-being services available to female sports, given their relatively earlier stages of professionalisation.¹³ In addition, the present sample of female athletes was insufficient to conduct confirmatory factor analysis, where samples of >300 cases are recommended. Nonetheless, scale reliability values indicated satisfactory internal consistency of the APSQ among female athletes, with the Cronbach coefficient for the APSQ total score exceeding the value observed for the K-10 (for both females and males). Additional research using the APSQ in larger, representative female athlete samples is required, as is research exploring gender differences for the APSQ.

Conclusion

Through the use of a range of cut-off scores, the APSQ may assist sports medicine practitioners and allied health professionals to better detect the early manifestations of mental health symptoms experienced by both male and female elite athletes. This in turn can facilitate timely management and ideally, remediation of symptoms.