Sensitivity and Specificity of Cognitive Health Screening in a Coordinated Specialty Care Setting

Abstract

Objective:

Cognitive health should be addressed as part of routine clinical care for people with schizophrenia spectrum disorder, including early psychosis. Providers need an efficient workflow including screening tools with adequate sensitivity and specificity to facilitate appropriate cognitive symptom management. This study examined the utility of a cognitive health screening protocol by analyzing data from N=102 participants at OnTrackNY, a specialized network of programs for people experiencing early psychosis.

Methods:

Screening queried participants’ perceived value for better memory, attention, and/or critical thinking skills and presence of cognitive difficulties via self- and clinician report. The sensitivity and specificity for detecting cognitive impairment defined by a brief cognitive assessment were evaluated.

Results:

Global cognitive impairment was present in about half of the sample whereas over 90% demonstrated impairment in at least one cognitive domain. Screening items demonstrated high sensitivity for detecting global (≥0.80) and specific cognitive impairment (≥0.74) such that those with objectively measured cognitive impairment were likely to be detected, though specificity remained low (≤0.32).

Conclusions:

The screening protocol may best be considered as a means of initiating shared decision making for next steps which may include brief cognitive assessment, or more resource intensive evaluation, to inform appropriate treatment recommendations.

Multinational guidelines for the treatment of schizophrenia recognize the importance of addressing cognition (1–3), raising questions about how to best integrate cognitive screening and assessment into routine practice. Implementation research indicates practice settings may benefit from a referral workflow (4–6) that exists in the context of a culture that values the personal or practical relevance of cognitive assessment and treatment (7, 8). This culture of cognitive health makes the cognitive assessment and treatment processes, which can be costly and time consuming, more acceptable and clinically relevant. Many practitioners lack experience talking openly about cognition and may be wary of highlighting what could be perceived as potentially stigmatizing symptoms (9, 10). However, direct query and report of service participants’ experience is a valued component of measurement-based care and supports participants’ agency in identifying priority areas for treatment (11, 12). Indeed, cognition is often reported as a high priority treatment target (13). Referral and assessment practices might therefore start with screening procedures that prompt open discussion about cognitive health needs. This paper focuses on that earliest stage in the assessment process and considers how well screening questions identify the need for further services.

We have previously described a Cognitive Health Toolkit, used in a large system of care serving people with early psychosis, which provides workflows and resources to facilitate systematic assessment and integration of psychoeducation, strategy coaching, and cognitive skills training interventions like cognitive remediation to address cognitive health (8, 14). Cognitive health needs screening begins with three self-assessment questions, framed from a strengths-based perspective, and query whether improvement in memory, attention and/or critical thinking skills would be useful for doing activities participants value. These are followed with questions for the participant and clinician about perceived cognitive difficulties. To date the toolkit has been used for over 5 years in a total of 28 Coordinated Specialty Care (CSC) clinics serving over 3,200 participants, and it has identified that more than 70% of participants would like to learn how to better address their cognitive health (8).

This novel cognitive screening protocol contrasts with performance-based methods (6,15) which rely on measures previously validated for use in adult psychiatric care settings. Further, the protocol may have an advantage in recovery-oriented, person-centered systems of care, such as specialized early intervention services, by emphasizing functional recovery over the identification of deficit (16). However, more data is needed to understand how well specific screening components work to identify for whom formal cognitive assessment and treatment are appropriate next steps.

Screening questions that foster open dialogue are designed to be highly sensitive to cognitive health concerns. However, in the process of emphasizing sensitivity to cognitive health needs, it is possible that some people who do not have objective evidence of cognitive impairment will voice a wish to have even better cognition. For example, when a person says they would like to have better memory, that does not necessarily mean that they have cognitive impairment or require cognitive treatment. This raises the issue of test specificity - the ability of a measure to designate an individual who does not have a condition as negative. When screening measures do not have high specificity, clinicians may refer people for cognitive assessment and treatment that is not indicated. Rendering services not clinically indicated may add unnecessary burden to clinicians and participants and reduce the capacity of service systems to channel resources to those who might benefit the most.

This study aimed to investigate the clinical utility of the cognitive health screening procedures, a part of the referral workflow, in the Cognitive Health Toolkit. We used program evaluation of clinical practice data to examine the relationship between screening items and performance on standardized cognitive assessments. We report the findings here as they are relevant to clinicians and systems of care seeking an efficient process to identify who to refer for services to enhance cognitive health.

Methods

Setting

Data for this study was collected between 2019–2023 at 11 CSC clinics affiliated with OnTrackNY (OTNY), one of the largest networks of CSC clinics in the USA, providing early intervention services to people ages 16–30 within two years of experiencing non-affective psychosis. Multidisciplinary OTNY teams serve a broad geographic region in New York state and offer a range of evidence-based treatments which are delivered in a personalized, culturally responsive fashion to promote recovery (17). All OTNY clinics use a Cognitive Health Toolkit to address the cognitive health needs of their program participants, though only 11 clinics, which contributed data to this study, offer brief cognitive testing. OnTrack Central, located at the Center for Practice Innovations within the New York State Psychiatric Institute, serves as an intermediate purveyor organization that oversees training, implementation, and evaluation of the Cognitive Health Toolkit across New York state.

Cognitive Health Toolkit

The toolkit is a collection of resources that aims to facilitate systematic assessment, and integration of interventions to address cognitive health. Its development and implementation have been previously reported (8) and it is freely available online (https://ontrackny.org/Learn-Train/Training-Manuals). Briefly, primary clinicians are provided training and resources to facilitate screening and follow-up action steps that are coordinated between the participant and OTNY team members based on need and interest, such as psychoeducation about cognitive health, cognitive remediation, skills coaching, or medication optimization. The guideline is to screen for cognitive health need once within the first three months of an individual’s participation in OTNY. Repeat screening is encouraged every 6 months thereafter to account for how cognitive health is experienced and differently prioritized as participants’ goals and needs evolve.

Included in the toolkit are five cognitive health need screening questions which are the specific focus of this study (Table 1). The Self-Assessment asks whether a participant perceives improvement in memory, attention, and/or critical thinking skills as useful for doing things they value. These cognitive skills were the designated focus given the prevalence of deficits in these areas (18, 19), their association with poor functional outcomes in the early psychosis population (21, 22), and because they are often identified as areas of concern by people with schizophrenia (23). Cognitive health need is also captured in the clinical record by participant-report via direct query and clinician-report: Does the participant believe they have cognitive difficulties?; Do you as a clinician believe the participant has cognitive difficulties (considering all available sources of information including self-assessment, clinical observation, parent/support person report, neuropsychological evaluation)? Any positive response indicating cognitive health need leads to a discussion of specific services such as brief cognitive assessment and skills training.

Table 1.

Cognitive Health Screening in OnTrackNY

Screening Item	Method of Ascertainment
Would it be helpful to you if you remembered better what you’ve read, heard, learned? Or about appointments, events, things to do?	Self-Assessment
Would it be helpful to you if you could concentrate better or stay focused over a longer period of time?	Self-Assessment
Do you think you could accomplish more if you could stay more organized, plan ahead, or manage your time better?	Self-Assessment
Does the participant believe they have cognitive difficulties?	Direct Query
Do you as a clinician believe the participant has cognitive difficulties?	Clinician-Report

Brief Cognitive Assessment

Cognitive testing is pursued using shared decision making, taking into account participants’ needs and preferences as discussed following screening. The brief paper-and-pencil cognitive assessment battery used at the 11 clinics is administered by OTNY clinicians trained and evaluated by members of OnTrack Central for proficiency and supervised for scoring accuracy. The 30-minute evaluation includes subtests from the Brief Assessment of Cognition in Schizophrenia (BACS, 24): Verbal Memory (verbal memory), Digit Sequencing (working memory), Symbol Coding (processing speed), and Tower of London (problem solving). The WAIS-IV Digit Span Forward (25) subtest is used as a measure of immediate attention. An average T-score derived from all available subtest scores yields a cognitive composite T-score. Results are presented graphically to participants to visualize areas of personal strength and relative weakness which are discussed in the context of each participant’s experiences and goals to inform collaborative treatment planning. Cognitive remediation may be recommended to address specific cognitive goals.

Data Sources

The data for this study consisted of de-identified program evaluation data provided to OTNY, via the data infrastructure system supported by the New York State Office of Mental Health (NYS OMH). Outcome and care process data for all OTNY participants are submitted by Primary Clinicians at each OTNY program on a quarterly basis using standardized admission, follow-up and discharge forms (26). NYS OMH Office of Population Health and Evaluation facilitates aggregation of de-identified client and program data on a monthly basis, which is summarized for quality assurance. Program evaluation for cognitive health services includes analysis of cognitive health screening tool use, rates of identified cognitive health need via participant-report and/or clinician-report, specific areas of participant-perceived need (memory, attention, critical thinking skills), and cognitive testing profile, if available. Program evaluation data collected from 3/1/2019 to 10/31/2023 from 102 participants who completed the brief cognitive assessment was extracted for analyses. The New York State Psychiatric Institute IRB determined that this secondary analysis of de-identified program evaluation data was not human subjects research.

Statistical Approach

Gender, race/ethnicity, and ages at the time of screening and cognitive assessment were tabulated. Cognitive impairment was defined two ways: (a) an average T-score across all subtests of less than 37, and (b) a T-score of less than 37 on any one subtest. Consistent with common practice (19, 27) the T-score criterion of <37 operationalized cognitive impairment as scores falling >1.5 SD below the mean for age-matched samples from the general population. The sensitivity and specificity for each self-assessment item and participant/clinician report item for predicting each definition of cognitive impairment was calculated, along with 95% confidence intervals. Additionally, the performance of the three self-assessment items as a sum score was assessed by calculating the sensitivity and specificity of endorsing ≥1, ≥2, or all 3 items for each definition of cognitive impairment.

Results

Of the 102 participants who completed the brief cognitive assessment, n=82 (80.4%) had provided responses to the cognitive self-assessment screening items and all but one, n=101 (99.0%), had direct query or clinician-reported screens done. The demographics of each participant sample are summarized in Table 2. Regarding the timing of when the screening was done versus the brief cognitive assessment: for the self-assessment screen, the majority (62.2%) completed the screening and the brief cognitive assessment at the same time point and 94% completed screening and the brief cognitive assessment within one year; for the direct query and clinical-report, the majority (82.2%) completed the screening and brief cognitive assessment at the same visit and 100% completed both within a 9-month timeframe. Responses to all self-assessment, participant- and clinician-report items are summarized in the online supplement.

Table 2.

OnTrackNY Participant Sample Demographics

	OnTrackNY Samplea n = 82		OnTrackNY Sampleb n = 101
	n	%	n	%
Gender
Male	54	65.9	67	66.3
Female	26	31.7	32	31.7
Transgender/Other	2	2.4	2	2.0
Race/Ethnicity
White/Non-Hispanic	24	29.3	25	24.8
Black/Non-Hispanic	22	26.8	28	27.7
Asian/Non-Hispanic	18	22.0	19	18.8
Hispanic	17	20.7	22	21.8
Other/Unknown/Missing	1	1.2	7	6.9
Age at Cognitive Screening (years)
Mean±SD	21.8±3.5		21.9±3.3
Age at Cognitive Assessment (years)
Mean±SD	22.1±3.4		21.9 ±3.4

^aCompleted the brief cognitive assessment and self-assessment screening items

^bCompleted the brief cognitive assessment and participant/clinician report items

Of the 82 participants who responded to self-assessment screening items and were assessed using a brief standardized cognitive battery, 48.8% demonstrated global cognitive impairment (average T-score <37), whereas 91.5% demonstrated impaired performance in at least one cognitive domain. The results of the sensitivity and specificity analyses utilizing the cognitive self-assessment items as a screening method for cognitive impairment are summarized in Table 3. All three of the self-assessment items, considered separately, provided high sensitivity to detecting both definitions of cognitive impairment (.85 – .91), though specificity for cognitive impairment was low (<0.30). Sensitivity to cognitive impairment remained high at all cutoffs when the self-assessment was analyzed as a summary score (0.83 – 0.95). Specificity improved when all three items were endorsed but remained low in general (<0.30).

Table 3.

Sensitivity and Specificity of Self-Assessment Items as Screeners (n=82)

	Brief Cognitive Assessment
	Average T-Score < 37				Any T-Score < 37
	40/82 (48.8%) < 37				75/82 (91.5%) < 37
Self-Assessment	Sensitivity 1	95% CI	Specificity 2	95% CI	Sensitivity 1	95% CI	Specificity 2	95% CI
Do you wish you remembered better what you've read, heard, or learned? Or about appointments, events, things to do?	0.90	0.76–0.97	0.21	0.10–0.37	0.85	0.75–0.92	0.29	0.04–0.71
Would it be helpful to you if you could concentrate better or stay focused over a longer period of time?	0.90	0.76–0.97	0.07	0.02–0.20	0.91	0.82–0.96	0.00	0.00–0.41
Do you think you could accomplish more if you could stay more organized, plan ahead, or manage your time better?	0.88	0.73–0.96	0.12	0.04–0.26	0.87	0.77–0.93	0.00	0.00–0.41
Considered as a Sum Score
≥ 1 item endorsed	0.95	0.83–0.99	0.02	0.00–0.13	0.96	0.89–0.99	0.00	0.00–0.41
≥ 2 items endorsed	0.90	0.76–0.97	0.12	0.04–0.26	0.88	0.78–0.94	0.00	0.00–0.41
3 items endorsed	0.83	0.67–0.93	0.26	0.14–0.42	0.79	0.68–0.87	0.29	0.04–0.71

¹Sensitivity is the proportion of participants who endorsed the self-assessment item among all participants who had an Average T-Score <37 (n=40) or had any T-Score <37 (n=75)

²Specificity is the proportion of participants who did not endorse the self-assessment item among all participants who did not have an Average T-Score <37 (n=42) or did not have any T-Score <37 (n=7)

Among those with participant/clinician-reported data and cognitive assessment results (n=101), 50.5% demonstrated global cognitive impairment (average T-score <37), whereas 92.1% demonstrated impaired performance on at least one cognitive domain (any T-score <37). Table 4 summarizes the sensitivity and specificity when using the participant and clinician report items as a screening method for both definitions of cognitive impairment. Sensitivity to either definition of cognitive impairment was high using participant-reported cognitive difficulty (0.74–0.80), while specificity was low (0.22 – 0.32). Similarly, sensitivity to cognitive impairment was high using clinician-report (0.76 – 0.86); specificity remained low (0.22 – 0.34).

Table 4.

Sensitivity and Specificity of Participant or Clinician Report of Cognitive Difficulty as Screeners (n=101)

	Brief Cognitive Assessment
	Average T-Score < 37				Any T-Score < 37
	51/101 (50.5%) < 37				92/101 (92.1%) < 37
Participant and Clinician Report	Sensitivity 1	95% CI	Specificity 2	95% CI	Sensitivity 1	95% CI	Specificity 2	95% CI
Does the client believe they have cognitive difficulties?	0.80	0.67– 0.90	0.32	0.20–0.47	0.74	0.64–0.83	0.22	0.03–0.60
Do you as a clinician believe that the client has cognitive difficulties?	0.86	0.74–0.94	0.34	0.21–0.49	0.76	0.66–0.84	0.22	0.03–0.60

¹Sensitivity is the proportion of participants who endorsed the participant or clinician report item among all participants who had an Average T-Score <37 (n=51) or had Any T-Score <37 (n=92)

²Specificity is the proportion of all participants who did not endorse the participant or clinician report item among all participants who did not have an Average T-Score <37 (n=50) or did not have Any T-Score <37 (n=9)

Discussion

Screening in clinical practice is needed to maximize clinical efficiency. Screening tools are intended to be brief, low cost, simple to administer, and indicate which clients require further diagnostic evaluation ahead of making specific treatment recommendations. Optimal screening procedures have the potential to lower cost and time demands and expedite appropriate treatment decision making. This study sought to evaluate the clinical utility of the cognitive health screening procedures used in OnTrackNY, a large, diverse network of Coordinated Specialty Care programs in New York State. The screening protocol intended to fill a gap in early intervention services which aim to balance comprehensive biopsychosocial assessment with the practical constraints of routine practice. Participants are first screened for cognitive health need through a combination of self- and clinician-reported items. Screening results are used to guide shared decision making for further cognitive health services inclusive of cognitive assessment to guide treatment recommendations and planning.

We found that all screening items included in the screening protocol have high sensitivity for detecting global and specific cognitive impairment identified through brief cognitive testing. High sensitivity is clinically useful in that those with cognitive difficulties are likely to be detected and unlikely to be missed when subsequently considering cognitive health services. The specificity of screening items was, however, low. Participants who identified that better cognition, with respect to memory, attention, or critical thinking skills, would be useful for doing the things they value did not necessarily evidence impairment upon cognitive testing. We found incremental improvements in screening specificity for items that asked participants directly about perceived cognitive difficulty and clinicians for their perception of cognitive difficulty using a combination of input from participant report, clinical observation, and any available collateral information.

The clinical utility of the current screening protocol should be considered in light of the benefits weighed against the risks of using tools that evidence high sensitivity and low specificity. The risks of being over-inclusive in recommending further cognitive evaluation based solely on the self- and clinician-report items include the risk of stigma that can be associated with perceived cognitive impairment (9), and unnecessary expenditure of resources, namely clinician and participant time. However, these risks may be mitigated in multiple ways. First, we recommend using the initial set of three self-assessment questions to begin a conversation about cognition, framed in terms of cognitive health as opposed to cognitive deficit, and with respect to the potential benefit of having better cognitive health to do valued activities. In fact, a sizable majority (>85%) of participants identify that better cognition would be useful (28), a contrast to previously published studies that have identified low self-reported rates of cognitive deficit (29–31). To address perceptions of need, the Cognitive Health Toolkit includes a discussion guide and psychoeducational materials to help normalize and better understand such experiences. In addition, the impact of any perceived cognitive concerns can be subsequently, if not immediately, addressed through strategy coaching. OnTrackNY team members routinely receive training and support to integrate strategy coaching to navigate cognitive challenges in personally relevant contexts (e.g., work, school). Second, identification of cognitive difficulties via self- and clinician-report may further narrow the stream of subsequent referrals for cognitive testing, thereby reducing unwarranted resource expenditures. While the risk of being overinclusive still remains, OnTrackNY clinicians have the capacity to use a relatively brief cognitive assessment battery, rather than refer out for a more costly comprehensive evaluation; the brief assessment permits identification of areas of relative cognitive weakness but also strength. Used within the context of shared decision making, better understanding of the balance of cognitive strength and need facilitates personalized treatment planning.

The data support the clinical utility of the workflow in the Cognitive Health Toolkit, best conceptualized as a three-step process: “Ask, Assess, Address.” In asking about cognitive health in screening, study data highlight that better cognition is valued by people experiencing early psychosis. The decision to further assess may be informed by additional avenues of inquiry, for example, whether the participant thinks that any difficulties with their current memory, attention and/or critical thinking skills are a change from a previous time and what factors might contribute to that change. Current cognitive challenges that are attributable to factors such as poor sleep, suboptimal medication use, or current stressors may be addressed before recommending cognitive testing. On the other hand, when cognitive difficulties persist, impact daily life and cause distress, brief assessment may be appropriate to inform treatment. A culture orientated towards cognitive wellness and recovery aims to mitigate risk to participants when cognitive testing is conducted among those who ultimately do not evidence cognitive impairment.

These results should be interpreted in light of the limitations of this study. First, the sensitivity and specificity analyses were completed within a subset of the clinics from a larger network. Though our current sample was ethno-racially diverse, a relative strength, an expanded dataset would allow for further inquiry into whether the screening protocol performs equitably across diverse populations. Second, caution is warranted in generalizing these outcomes from people with early psychosis to the broader population of adults with psychosis and their service settings. Replication of sensitivity and specificity analyses in a different treatment setting is a logical next step.

Conclusion

The screening protocol used in OnTrackNY attends to the need for early identification of cognitive health needs within a recovery-oriented framework. The high sensitivity of the cognitive screening protocol suggests that those who stand to benefit from assessment and intervention are likely to be identified. However, the low specificity of the screening protocol supports its use, not as an indicator of cognitive impairment, but as a first step in the process of shared decision making about cognitive health services. Pursuit of further assessment or treatment thereafter is a decision made through considering multiple sources of information, not the least of which should include participants’ stated needs, preferences, and priorities.

Highlights.

• A novel protocol integrating service participant and clinician report of cognitive health needs in an early psychosis intervention setting demonstrated high sensitivity for detecting cognitive impairment as defined by performance on a brief standardized test battery.

• Participants reported valuing better cognitive health even in the absence of cognitive impairment.

• Pursuit of further cognitive assessment or treatment should align with participants’ stated needs, preferences, and priorities.