Utility of the Counseling Center Assessment of Psychological Symptoms Screen in a Collegiate Athlete Population

Abstract

Context

Mental health screening as a part of collegiate athletic preparticipation evaluations is becoming increasingly common, but effective and efficient screening depends on a screening tool that can accurately identify mental health symptoms and the need for mental health intervention.

Design

Case-control study.

Setting

Archival clinical records review.

Patients or Other Participants

Two cohorts of incoming National Collegiate Athletics Association Division I collegiate athletes (N = 353).

Main Outcome Measure(s)

Athletes completed the Counseling Center Assessment of Psychological Symptoms (CCAPS) Screen as a part of their preparticipation evaluation. These data were then matched with basic demographic data and mental health treatment history from clinical records, and the utility of the CCAPS Screen in determining a future or ongoing need for mental health services was analyzed.

Results

Score differences for each of the 8 CCAPS Screen scales (Depression, Generalized Anxiety, Social Anxiety, Academic Distress, Eating Concerns, Frustration, Family Distress, and Alcohol Use) were found based on several demographic variables. Logistic regression analysis demonstrated that female sex, team sport participation, and the Generalized Anxiety scale score predicted future participation in mental health treatment. Decision tree testing of the CCAPS scales showed low utility in classifying those who received mental health treatment versus those who did not.

Conclusions

The CCAPS Screen did not appear to differentiate well between those who eventually received mental health services and those who did not. This should not be taken to mean that mental health screening is not useful but rather that a 1-time, state-based screening is not sufficient for athletes who experience intermittent but recurring stressors in a dynamic environment. A proposed model for improving the current standard of practice for mental health screening is provided as a focus of future research.

Key Points

• Mental health preparticipation evaluations are becoming commonplace in collegiate athletics, given increased rates of mental health symptoms. Current standard practice is a 1-time screening. A frequently used instrument, the Counseling Center Assessment of Psychological Symptoms Screen, had limited predictive and classification ability, with Generalized Anxiety being the only scale score that was a predictor or contributor.

• The predictive utility of the Counseling Center Assessment of Psychological Symptoms Screen may be limited because measuring state-based psychological functioning is not intended to predict future mental health functioning, mental health treatment, or both.

• Suggestions for future study include evaluating state versus trait screeners for sensitivity and predictive ability and whether screening for trait anxiety alone (given that only the Generalized Anxiety scale score was significant) is sufficient to identify those athletes at higher risk for future mental health symptoms.

Preparticipation examinations (PPEs) have become standard practice in collegiate athletic departments in the past 4 decades.¹This practice evolved in sports medicine as the rates of sudden death in sport increased. The purpose of a PPE was 2-fold: (1) ensure the athlete’s immediate health status was sufficient to endure sport training and performance demands and (2) identify any factors that could put the athlete at risk of future medical complications during sport participation.

Similarly, in the past decade, student-athlete mental health conditions have become an area of growing concern for health care providers in the elite sport setting, given the rise in student-athlete mental health symptoms and diagnosis. Mental health problems in elite athletes can lead to declines in sport performance, medical leave from sport, retirement from sport, and most tragically, sudden death by suicide.^2,3These major health care incidents have prompted health care providers to prioritize the early identification of mental health symptoms and risk factors in an effort to ensure timely, competent treatment.⁴Analogous to the original rationale for a standardized PPE for physiological risk factors, sports medicine departments have also begun to include a mental health evaluation as a key component of the PPE.

To address the growing recognition of mental health concerns faced by collegiate student-athletes, the National Collegiate Athletics Association (NCAA) Sport Science Institute put forth recommendations for mental health best practices.⁵One recommendation was that all member institutions use a mental health screening as part of their PPE of student-athletes.⁵The purpose of a mental health screening is to identify current mental health symptoms, aid in diagnosis, and determine whether immediate intervention may be warranted.

Although incorporating a mental health assessment in the PPE is a step in the right direction toward a standard of practice for athletic or sport medicine departments, current instruments used for mental health screenings of collegiate student-athletes have several limitations. In 2021, Drew et al⁶surveyed NCAA Division I, II, and III member institutions and found that, of 264 institutions surveyed, 70.8% were using a formal screening procedure and 59.9% were using validated mental health screening measures (ie, Counseling Center Assessment of Psychological Symptoms [CCAPS] Screen, Generalized Anxiety Disorder-7 [GAD-7], or Patient Health Questionnaire-9 [PHQ-9]). The use of validated screeners increases the likelihood of catching emerging or existing mental health concerns,⁷but most of the measures being used have not been normed on a student-athlete population,^7,8nor can the measures predict the likelihood of future mental health problems. These screeners often measure recent psychological states rather than psychological traits. Researchers^9,10have shown that psychological traits, which are pervasive, stable patterns of personality that generalize across similar situations, are more robust predictors of future mental health outcomes, such as the onset of psychopathology, symptom chronicity, functional impairment, and treatment outcomes.

Furthermore, Drew et al⁶determined that 85.5% of institutions screened student-athletes at only 1 time point per year, in either the fall or spring. The fact that many NCAA member institutions are placing attention on mental health screening programs is promising, yet the utility of a screener can be limited if it is not used for its intended purpose or with a sample on which it has been normed.⁷The PHQ-9, GAD-7, and CCAPS were all identified as frequently used mental health screeners,⁶but these measures are only intended to assess the presence of symptoms across the most recent 2 weeks. The sensitivity, specificity, or both of the GAD-7 and the PHQ-9 have not been evaluated in a student-athlete population.

Given the unique demands and expectations placed on collegiate student-athletes,^11–15it is imperative that effective mental health screening measures be developed and used. These screeners must be sensitive to indicators of symptoms that would predicate the need for mental health treatment without overpathologizing some of the experiences and emotions unique to but normal in collegiate athletes.⁸Early identification of mental health problems can lead to appropriate referrals for mental health services or programs⁸and subsequently reduce the severity and chronicity of mental health symptoms that can interfere with sport participation and performance.⁴

The Center for Collegiate Mental Health has published 1 of the very few mental health screeners that was normed on and provides clinical cutoff scores for collegiate athletes. The CCAPS Screen is a brief web-based mental health screening instrument that is intended to provide mental health clinicians and nonclinicians with an easy-to-administer, user-friendly, and valid assessment that gives immediate feedback on the general mental health of student-athletes.¹⁶

The CCAPS Screen has demonstrated usefulness in accurately identifying common psychological problems if student-athletes have experienced them within the past 2 weeks.¹⁶However, currently, minimal research exists on the effectiveness of a 1-time mental health screening assessment to predict the student-athletes who might be at risk for future mental health decompensation and subsequently need mental health intervention or care.¹⁷Given the high-stakes, high-stress nature of being a collegiate student-athlete, a screening instrument that has strong predictive capability could assist health care providers in a number of ways. It could identify the student-athletes who may benefit from psychological care when faced with heightened sport and nonsport stressors. In turn, this could reduce the likelihood that a significant emerging mental health problem goes unnoticed or untreated, and it may also aid in the timely referral for and initiation of mental health services.^4,8Moreover, timely, competent psychological intervention and care may lower the incidence rates of medical leave from sport, sport retirement, and major medical problems due to significant mental health conditions.^4,8As the current standard of practice in mental health PPE in collegiate sport is to use a 1-time mental health screener, the purpose of our study was to assess the utility of a validated mental health screen for collegiate student-athletes, the CCAPS Screen, in predicting the risk for the development of future mental health symptoms that would warrant intervention.

METHODS

Participants

We used a convenience sample of archival clinical data of CCAPS mental health screenings completed by a sample of NCAA Division I student-athletes (N = 353). The athletes attended the same university and completed the CCAPS Screen once as a part of their preperformance or baseline evaluation before being cleared for sport participation. This method of screening is consistent with best practices and the NCAA Sport Science Institute recommendation.⁵Before completing the screen, athletes were informed that the screen was completed by all incoming athletes, that it was only viewed by the sport psychology providers, that coaches and other staff would not see it, and that the purpose of the screening was for the sport psychology staff to understand the baseline functioning for each individual in order to provide optimal care.

Data were collected over 2 years of preparticipation evaluations. The demographic data and treatment records were then retrieved from clinical records and matched with the CCAPS screens. A relatively even split was present between female (n = 148, 42%) and male (n = 205, 58%) athletes and team (n = 192, 54%) and individual (n = 161, 46%) sports. A small minority of athletes were international students (n = 33, 9%) or endorsed a diagnosis of attention-deficit/hyperactivity disorder (ADHD), learning disability (LD), or both (n = 47, 13%). A total of 187 student-athletes (53%) were subsequently seen by a sport psychologist or athletic counselor, but only 136 (38.5%) were seen for mental health care. Informed consent was waived by the institutional review board, as all data were de-identified by an honest broker before analysis and were not considered to be human participants’ data.

Materials

The CCAPS Screen was normed on 5926 varsity collegiate athletes for the purpose of developing an athletic preparticipation mental health screening tool. The 36-item instrument asks athletes to indicate how well each statement describes them over the past 2 weeks on a 5-point Likert scale from 0 (not at all like me) to 4 (extremely like me). The items assess 8 domains of clinical mental health problems: depression, generalized anxiety, social anxiety, academic distress, eating concerns, frustration, family distress, and alcohol use. A single item also evaluates suicidal ideation. Scores for each scale are calculated by averaging the athlete’s responses for the items for that scale (range = 0–4, with high scores indicating more symptoms in that particular domain). A profile report is generated with cutoff scores. Low (green) is interpreted as minimal to no distress in that area, moderate (yellow) as a moderate level of distress that warrants consideration of counseling for that concern, and elevated (red) as a level of distress similar to that of individuals who receive counseling services for the specific concern.

Procedures

The CCAPS Screen was administered as a part of incoming athletes’ baseline PPEs. Screeners were completed in person by athletes in groups of 2 to 10 in a large computer laboratory and were overseen by a licensed psychologist.

Athletes who were then seen by a sport psychologist over the next 2 years were coded by their treating provider as either a 1 (performance psychology services only) or a 2 (clinical mental health concern, such as adjustment disorder or chronic mental health condition). Performance psychology services were operationally defined as psychological skills taught for the exclusive purpose of improving sport performance in the absence of mental health symptoms (ie, goal setting, self-regulation skills for autonomic control, sleep improvement strategies, self-talk), or both.

If patients were seen for both mental health and performance services, they were coded as a 2 (clinical mental health concern). Those coded as receiving only performance psychology services were combined with those who received no services for data analysis. Athletes receiving only performance psychology services did not need to be identified in the PPE screening process, as they did not receive treatment for mental health symptoms and were not at risk for related decompensation.

Data Analysis

The CCAPS Screens were identified only by a numerical identifier that was generated when the athlete finished the screen. These identifiers were recorded by the proctoring psychologist, and then the screens were re-identified to facilitate clinical use. The CCAPS Screens were matched with the treatment code provided by the treating provider and basic demographics from clinical records. Data were de-identified by an honest broker, resorted, and returned to the researchers.

To facilitate analysis, the coding by the treating psychologist was broken into a binary variable with one group consisting of those cases with either no mental health services or only performance services and the other group consisting of those cases who received treatment for clinical mental health concerns.

Due to the large number of zero values in response to the CCAPS Screens, we treated these values as binary (zero values or nonzero values). Descriptive statistics for participants were analyzed in logistic regression models to examine the difference in zero values for each of the 8 scales based on sex; international student status; ADHD, LD diagnosis, or both; team versus individual sport; and those who received mental health treatment compared with those who did not. Chi-square analyses were run to determine if a difference existed in the numbers of those who received mental health treatment based on sex; international student status; ADHD, LD diagnosis, or both; or team versus individual sport.

Logistic regression analysis was conducted using the cut scores for each of the 8 scales, sex, and team versus individual sport as predictor variables. Mental health treatment (versus no mental health treatment) was the dependent variable. A receiver operating characteristic analysis was then performed to assess the logistic regression model. Finally, a decision tree was created to attempt to distinguish individual thresholds for the predictor variables used in the regression.

Data analysis was accomplished using SPSS (version 29.0: descriptives, χ²; IBM Corp), SAS (version 9.4: logistic regression; SAS Institute), and rpart package in R (version 4.2.1: decision tree; The R Project for Statistical Computing).

RESULTS

Mean scores for each of the 8 scales were relatively low (ranging from 0.09–0.93, SD = 0.31–0.74; for reference, the yellow cutoffs on the CCAPS Screen range from 0.60–1.82), suggesting that, as a group, in contrast to previous normative groups, athletes in this sample were not experiencing particularly high levels of distress as they entered collegiate athletics.

Logistic regression models evaluating zero values demonstrated some differences between groups (Table 1). Small but significant relationships existed between mental health treatment and several demographic factors, with female athletes (χ²₁[N = 353] = 5.92, P = .010, Cramer V = 0.13); athletes with an ADHD, LD diagnosis, or both (χ²₁[N = 353] = 6.46, P = .009, Cramer V = .14); and team sport athletes (χ²₁[N = 353] = 5.86, P = .010, Cramer V = .13) overrepresented in the group that received mental health treatment. No relationship was found between international versus US student status and mental health treatment.

Table 1.

Demographic Category (P Value, Higher Group)

Counseling Center Assessment of Psychological Symptoms Screen	Female (F) Versus Male (M) Athletes	International (I) Versus US	Learning Disability or Attention-Deficit/Hyperactivity Diagnosis (Dx) Versus None	Team (T) Versus Individual (I) Sport	Mental Health Treatment (MH)
Depression	.814, F	.140, I	.0363, Dx	.987, I	.0035
Generalized anxiety	.534, F	.401, I	.0172, Dx	.364, I	<.001
Social anxiety	.703, F	.624, US	.670, None	.11, I	.828
Academic distress	.111, M	.119, I	.065, Dx	.359, I	.025
Eating concerns	.073, F	.267, I	.041, Dx	.418, I	.353
Frustration	.006, M	.055, I	.508, Dx	.221, I	<.001
Family distress	.415, M	.213, I	.187, Dx	.318, I	.002
Alcohol use	.020, M	.986, I	.093, Dx	.811, T	.181

The logistic regression model with cutoff scores, sex, and team sports as predictor variables was constructed with 348 complete observations, as 5 cases were missing a scale score. Sex (P = .0057) and team sports (P = .0025) were significant predictors of future mental health treatment, while generalized anxiety was marginally significant (P = .0763). Female athletes had higher odds of seeking mental health care when compared with male athletes (odds ratio [OR] = 2.075), and those participating in individual sports had lower odds of seeking mental health care than those in team sports (OR = 0.457; Table 2). The receiver operating characteristic analysis of the regression indicated the area under the curve was 0.72. According to the decision tree analysis, the only variable that effectively reduced cross-validation error was the Generalized Anxiety cut score, with a cutoff value of 1.1. This resulted in a cross-validation error of 36.4%, which was larger than the cross-validation error when the given cut scores were used (33.4%). Note that, when no decisions were made, the cross-validation error would be 38.5% (Table 3).

Table 2.

Odds Ratios (ORs) and Adjusted P Values for Sex and Counseling Center Assessment of Psychological Symptoms Scale Scores

Cutoff	Factor 1a	Factor 2	Adjusted P	α	OR	Adjusted Lower OR	Adjusted Upper OR
Sex	Female	Male	.0057	.05	2.075	1.236	3.483
Depression	Green	Red	.6451	.05	2.857	0.181	44.998
	Green	Yellow	.7234	.05	0.580	0.109	3.069
	Red	Yellow	.4395	.05	0.203	0.010	4.312
Generalized Anxiety	Green	Red	.4717	.05	0.229	0.012	4.399
	Green	Yellow	.0953	.05	0.328	0.093	1.156
	Red	Yellow	.9594	.05	1.432	0.067	30.753
Social Anxiety	Green	Red	.3439	.05	0.299	0.039	2.274
	Green	Yellow	.9078	.05	0.874	0.413	1.853
	Red	Yellow	.4469	.05	2.922	0.366	23.298
Academic Distress	Green	Red	.9415	.05	0.767	0.117	5.027
	Green	Yellow	.7577	.05	0.782	0.346	1.763
	Red	Yellow	.9997	.05	1.019	0.150	6.912
Eating Concerns	Green	Red	.7576	.05	0.656	0.163	2.643
	Green	Yellow	.7237	.05	1.496	0.436	5.132
	Red	Yellow	.5172	.05	2.283	0.390	13.356
Frustration	Green	Red	.3609	.05	0.423	0.096	1.858
	Green	Yellow	.6726	.05	0.529	0.091	3.066
	Red	Yellow	.9655	.05	1.251	0.156	10.012
Family Distress	Green	Red	.8451	.05	0.669	0.122	3.676
	Green	Yellow	.9497	.05	0.760	0.092	6.250
	Red	Yellow	.9929	.05	1.135	0.084	15.366
Alcohol use	Green	Red	.5945	.05	0.312	0.019	5.178
	Green	Yellow	.9962	.05	1.067	0.171	6.658
	Red	Yellow	.6369	.05	3.424	0.141	82.864
Team vs individual sport	Individual	Team	.0025	.05	0.457	0.275	0.759

Table 3.

Cross-Validation Error (Scaled to Root Node 0.3853)

Split(s)	Using Scale Cut Scores	Using Scale Raw Scores
0	1.000	1.000
1 (Generalized Anxiety)	0.868	0.949
4	0.904	1.029

DISCUSSION

In collegiate sports, PPEs have been standard practice for nearly 4 decades,¹and the mental health PPE has emerged as a standard of care within the past decade. The predictive accuracy of mental health risk assessment is diametrically different from that of physiological risk assessment. The understanding of physiological risk factors for physical decompensation is relatively well defined and easily measured, whereas risk factors for mental health decompensation are not nearly as well defined, are typically more influenced by environmental stressors, and are not measured as objectively as physiological risk factors.¹⁸Therefore, it is reasonable to perform a single medical PPE screening but much more problematic to conduct a 1-time mental health PPE screening. Additionally, the current standard practice of the mental health PPE often includes using exclusively state-based measures, which only capture how the athlete is feeling at the time of assessment. Without periodic reassessments to track mental and behavioral health changes related to environmental or situational stressors, a state-based measure at a single time point is limited in its ability to identify who may be at greatest risk for decompensation and therefore need treatment.

In the past decade, student-athlete mental health conditions have become an area of health care concern in sports medicine.^2,3,5As the number of major medical incidents related to mental health has risen, sports medicine professionals have become increasingly focused on early identification of mental health symptoms and risk factors in an effort to ensure competent treatment.⁴Similar to the original rationale for a standardized PPE for physiological risk factors, sports medicine departments have also begun to include the mental health evaluation as a key component of the PPE. However, these mental health screening procedures often (1) do not use standardized assessments, (2) do not have clinical cutoffs for a student-athlete population, (3) are not using trait-based assessments that might aid in flagging individuals for future risk, and (4) are not periodically administered.

Using archival data, we examined the effectiveness of using a standardized mental health screener (CCAPS Screen) normed on collegiate student-athletes as a part of the student-athlete PPE. The purpose of a mental health PPE is 2-fold: (1) to identify the presence of current mental health concerns and (2) to determine the risk for future decompensation and need for mental health intervention and evaluation. The second item was the focus of our study.

This sample of matriculating student-athletes endorsed relatively low levels of mental health symptoms. The known confounders of time-limited, self-report measures may have played a role in these low scores.¹⁹Two additional plausible conclusions are that student-athletes may prioritize impression management over accurate reporting when they arrive on campus^19,20or they may generally be in less distress because they have yet to be fully immersed in their academic and athletic obligations.¹⁹Currently, no normative data exist on the CCAPS Screen that are specific to sex or team versus individual sport, both factors that have been demonstrated to play a role in mental health symptoms.²¹This creates possible limitations for both the sensitivity and specificity of the CCAPS Screen. In this study, we identified scale score differences based on sex, international student status, and self-reported diagnoses of ADHD, LD, or both, which would support the need for more specific norms.

In this research, we also demonstrated that the CCAPS Screen appears to have relatively limited utility for the second focus of PPEs, identifying risk and the future need for mental health intervention. Other than the Generalized Anxiety scale score, which was only marginally significant, no other scale scores at PPE screening predicted future psychological help-seeking for clinical mental health conditions. Although the CCAPS Screen scores were not particularly useful in predicting future mental health treatment, female sex and participation in a team sport (versus an individual sport) were predictive, suggesting that individual and environmental factors may need to be considered when determining risk. This finding is congruent with previous research showing that youth and adolescents participating in organized team sports compared with non–sport-participating peers experienced better mental health outcomes, such as less anxiety and depression, fewer social problems, and fewer attentional issues.^21,22In addition, Walton et al²³observed that elite female athletes endorsed mental health symptoms at a higher rate than elite male athletes.

It is not surprising that the CCAPS Screen had limited utility in identifying risk and predicting future mental health care use in a collegiate athlete population. Most psychological conditions in student-athletes are transient in nature and often coincide with increased demands based on timing of a sport season or academic semester. The CCAPS Screen is intended to exclusively evaluate mental health symptoms for the most recent 2 weeks, with the purpose of identifying state-based experiences that might necessitate further mental health evaluation for current diagnosis. If athletes complete the CCAPS Screen at a time of low stress, they will likely endorse minimal symptoms regardless of a propensity for future mental health difficulties. Ultimately, this type of measure was not developed for the purpose of long-term risk assessment. Evaluating current mental health symptoms may be helpful in recognizing an immediate risk of performance decompensation, but it is insufficient for identifying the long-term psychological needs in order to provide timely interventions.

Our current findings may appear to state the obvious, that a state-based measure administered once was not effective in predicting mental health deterioration or the need for future mental health intervention; however, this is currently both standard and best practice across most NCAA athletic medicine programs.⁶Our results should serve as a call to improve mental health screening processes by including measures that are psychometrically sound and do a better job of predicting future risk for emerging mental health concerns and assessing psychological vulnerabilities on starting college. This type of risk or vulnerability profiling could consider additional environmental and individual factors, such as personality traits, historical mental health data, and anxiety sensitivities. Furthermore, a combination of risk profiling and brief but ongoing mental health screenings would be the criterion standard for mental health care among elite athletes, so that those at highest risk for decompensation can be monitored closely and timely interventions applied when the risk is identified. This practice would not only improve patient care but could also improve athletic performance.

Limitations

Myriad individual factors may affect the validity of an individual’s responses to a screening tool, including social desirability, overreporting of symptoms to communicate the urgent need for services, timing of the assessment, misunderstanding the intent of the assessment, or cultural and socialization factors.^16,18In collegiate student-athletes, accurate reporting may be further affected by their perception that their responses could negatively affect their sport-related opportunities.^7,19

Given that the timing of our CCAPS Screen administration was before full sport and academic integration, it is possible that the student-athletes were experiencing overall less psychological distress because they had minimal academic and athletic demands.

This sample was limited to a specific group (NCAA Division I athletes at a Power 5 institution with a well-resourced athletic department) at a somewhat unique time point (acutely post-COVID), so it is unclear how these results would generalize to other athlete samples.

Finally, even though the overall sample was robust, some of the groups (ie, athletes with ADHD, LD, or both; those who endorsed high levels of alcohol use) were relatively small.

Future Directions

A more effective proposed mental health screening protocol may be a PPE screening to assess active mental health symptoms in conjunction with a trait-based measure of anxiety predisposition, such as the Anxiety Sensitivity Index-3, which has evidenced strong predictive validity for the onset and continuation of anxiety and mood disorders.²⁴Given that anxiety was the single scale with predictive ability from the CCAPS Screen, a trait anxiety measure for anxiety, such as the Anxiety Sensitivity Index-3, may be indicated. Data from any screening measures should then be considered in the context of individual athlete factors that may play a role in mental health symptoms. Factors in this risk profile could include psychological variables (eg, personality traits, coping skills, motivation, goals, attitudes), environmental or situational variables (eg, sport season, life events, injury, external pressures to perform), and affiliation variables (eg, sex, national origin, individual versus team sport). However, further research into which combination of trait-based characteristics, demographic variables, and adverse life events (sport and nonsport) put a student-athlete at low, medium, or high risk for impaired mental and behavioral functioning is needed. This could lead to a dimensional model of classification in which student-athlete mental health is seen on a continuum from optimal to nonoptimal,²⁵and mental health researchers and clinicians can better understand which student-athlete traits or characteristics may protect against decompensation in reaction to adverse sport and nonsport life events and which traits may exacerbate decompensation. In addition, a dimensional risk model could potentially identify which athletes may need mental health services in response to situational stressors and the level of mental or behavioral health services required to ameliorate the underlying symptoms.

Additionally, to improve the early identification, referral, and treatment of student-athletes for mental health conditions that affect welfare and performance, mental health functioning should be reassessed frequently. Wellness questionnaires have become commonplace as a component of sport science readiness determinations in elite sport and could likely be developed as a brief psychological screener similar to the GAD-7 or PHQ-9, which would make ongoing evaluation more feasible in a dynamic and complicated sport setting.