Validation of the UK English Oxford cognitive screen-plus in sub-acute and chronic stroke survivors

Abstract

Introduction:

Stroke survivors are routinely screened for cognitive impairment with tools that often fail to detect subtle impairments. The Oxford Cognitive Screen-Plus (OCS-Plus) is a brief tablet-based screen designed to detect subtle post-stroke cognitive impairments. We examined its psychometric properties in two UK English-speaking stroke cohorts (subacute: <3 months post-stroke, chronic: >6 months post-stroke) cross-sectionally.

Patients and methods:

This study included 347 stroke survivors (mean age = 73 years; mean education = 13 years; 43.06% female; 74.42% ischaemic stroke). The OCS-Plus was completed by 181 sub-acute stroke survivors and 166 chronic stroke survivors. All participants also completed the Oxford Cognitive Screen (OCS) and a subset completed the Montreal Cognitive Assessment (MoCA) and further neuropsychological tests.

Results:

First, convergent construct validity of OCS-Plus tasks to task-matched standardized neuropsychological tests was confirmed (r > 0.30). Second, we evaluated divergent construct validity of all OCS-Plus subtasks (r < 0.19). Third, we report the sensitivity and specificity of each OCS-Plus subtask compared to neuropsychological test performance. Fourth, we found that OCS-Plus detected cognitive impairments in a large proportion of those classed as unimpaired on MoCA (100%) and OCS (98.50%).

Discussion and conclusion:

The OCS-Plus provides a valid screening tool for sensitive detection of subtle cognitive impairment in stroke patients. Indeed, the OCS-Plus detected subtle cognitive impairment at a similar level to validated neuropsychological assessments and exceeded detection of cognitive impairment compared to standard clinical screening tools.

Introduction

Cognitive impairment is very common post-stroke. Whilst prevalence estimations vary depending on study protocols and patient cohorts,^1–3 almost all stroke survivors show at least one cognitive domain impairment in the early stages post-stroke,^4,5 and 8%–43% of stroke survivors experience longer term post-stroke cognitive impairment (PSCI).^1,6 PSCI negatively affects social participation, ⁷ mood ⁸ and quality of life, ⁹ over and above physical disability levels.

Multiple UK national and international guidelines identify cognitive screening as an essential part of post-stroke assessment and discharge planning.^10–12 Whilst there are several clinically used tools for post-stroke cognitive screening, these tools were primarily developed to detect dementia (e.g. Mini-Mental State Examination/MMSE ¹³ and Montreal Cognitive Assessment/MoCA ¹⁴ ) or early stroke-specific cognitive impairments (Oxford Cognitive Screen/OCS ⁴ ), and may lack sensitivity for subtler PSCI in the later stages post-stroke, where typically a full neuropsychological assessment is advised. Indeed, screening for longer-term PSCI is complex, as there is substantial heterogeneity in post-stroke cognitive recovery between patients and between cognitive domains.^15–19 Furthermore, PSCI may stem not only from stroke-specific factors, but also broader vascular factors, linking to vascular dementia and small vessel disease, ²⁰ as well as shared risk factors for stroke and dementia. ²¹

The OCS-Plus is a computer-tablet based cognitive screening tool that has been developed to screen for subtle post-stroke cognitive impairment (in particular, impairments in executive attention and memory) ²² using a reflective measurement model. ²³ The OCS-Plus is an extension of the OCS, which is routinely used in clinical practice to screen for early stroke-specific cognitive deficits. Like the OCS, the OCS-Plus was designed to minimize language demands, cultural confounds, and examiner bias. Administration is standardised via a platform independent app and takes approximately 25 min. Following administration, automated performance reports are provided, based on matched age-specific cut offs. The OCS-Plus has been standardised and normed in UK and German populations and preliminary psychometric validation has been completed in healthy ageing adults. ²² OCS-Plus performance meaningfully varies with socio-economic factors and age, demonstrating its sensitivity as a cognitive screening tool. ²⁴

Here, we present a psychometric validation of the OCS-Plus in a sub-acute (<3 months post-stroke) and chronic (⩾ 6 months post-stroke) stroke cohort. This is the first study to investigate the validity of the OCS-Plus in a clinical stroke cohort. First, we assessed construct validity by comparing OCS-Plus task performance to sub-task matched validated standardised neuropsychological tests. Second, we evaluated sensitivity of detecting cognitive impairment with OCS-Plus compared to clinically used first-line screening tools (OCS and MoCA) and compared to standardised neuropsychological tests.

Methods

Standard protocol approvals and patient consents

Participants were recruited for the OCS-Recovery study, under ethical approval of the South Central – Oxford C Research Ethics Committee (Ethics Ref: 18/SC/0550l; IRAS Ref: 248483; Protocol number PID 13803). Participants were recruited at the John Radcliffe acute stroke unit and the Oxfordshire Stroke Rehabilitation Unit. Participants were eligible for inclusion if they were ⩾ 18 years of age, able to sufficiently comprehend English and had a suspected/confirmed stroke. Participants were excluded if they could not provide informed consent, was too unwell to concentrate for approx. 30 min (as judged by the multidisciplinary team) or had severe sensory impairments which meant they could not sufficiently see the stimuli or hear the instructions. All participants provided written informed consent. The construct validation section of this study was pre-registered (pre-registration osf.io/t8zug). The COSMIN guideline for reporting measurement properties ²⁵ was followed.

Participants

This investigation included all OCS-Recovery Study participants who had completed the OCS-Plus in either the sub-acute (n = 181) or chronic (n = 166) stage post-stroke. Sub-acute testing was conducted in an inpatient clinical setting (acute stroke unit or stroke rehabilitation unit) between 2015 and 2016 and 2020 and 2022, and chronic testing was conducted in stroke survivors’ homes between 2015 and 2020. Sub-acute and chronic participants do not overlap. Table 1 reports demographic information of the sample.

Table 1.

Clinical and demographic details for sub-acute (<3 months post-stroke) and chronic (>6 months post-stroke) stroke samples.

	All		Subacute		Chronic
Characteristic	N	Sample	N	Sample	N	Sample
Age (M(SD))	346 (0)%	72.85 (13.36)	181 (0)%	71.77 (13.68)	165 (1)%	74.04 (12.93)
Education (M(SD))	255 (27)%	13 (3.46)	163 (10)%	13.17 (3.44)	92 (45)%	12.72 (3.48)
Handedness	323 (7)%	A: 0.93%; L: 9.29%; R: 89.78%	178 (2)%	A: 1.69%; L: 8.99%; R: 89.33%	145 (13)%	L: 9.66%; R: 90.34%
Sex	347 (0)%	F: 43.06%; M: 56.94%	181 (0)%	F: 39.78%; M: 60.22%	166 (0)%	F: 46.67%; M: 53.33%
Ethnicity	346 (0)%	Asian-Other: 0.29%; Black African: 0.29%; Chinese: 0.29%; Mixed White Black Caribbean: 0.29%; Senegal-Wolof: 0.29%; White-Bulgarian: 0.29%; White-Portuguese: 0.29%; Black-Caribbean: 0.58%; Other-Asian: 0.58%; White-English: 0.58%; Other: 0.86%; White-Spanish: 0.86%; White-Other: 3.46%; White-Unknown: 10.95%; White-British: 80.12%	181 (0)%	Black African: 0.55%; Chinese: 0.55%; Mixed White Black Caribbean: 0.55%; Senegal-Wolof: 0.55%; White-Bulgarian: 0.55%; White-Portuguese: 0.55%; Black-Caribbean: 1.1%; Other-Asian: 1.1%; White-English: 1.1%; White-Spanish: 1.66%; White-Unknown: 4.42%; White-Other: 5.52%; White-British: 81.77%	165 (1)%	Asian-Other: 0.6%; White-Other: 1.2%; Other: 1.81%; White-Unknown: 18.07%; White-British: 78.31%
Days Since Stroke (M(SD))	340 (2)%	102.85 (94.21)	180 (1)%	19.37 (16.37)	160 (4)%	196.76 (43.09)
Stroke type	301 (13)%	Other: 0.66%; SAH: 1%; CVA: 3.32%; TIA: 3.32%; ICH: 17.28%; Ischaemic: 74.42%	169 (7)%	Other: 1.18%; CVA: 1.78%; SAH: 1.78%; ICH: 14.2%; Ischaemic: 81.07%	132 (20)%	CVA: 5.3%; TIA: 7.58%; ICH: 21.21%; Ischaemic: 65.91%
Stroke side	304 (12)%	B: 8.55%; L: 40.13%; R: 51.32%	171 (6)%	B: 9.94%; L: 35.09%; R: 54.97%	133 (20)%	B: 6.77%; L: 46.62%; R: 46.62%
Stroke Severity	267 (23)%	7.49 (5.8)	123 (32)%	7.44 (5.13)	144 (13)%	7.54 (6.33)

For stroke type. Missing data is presented in parentheses as a percentage next to N per demographic. Stroke severity is established via the National Institute of Health Stroke Scale.

SAH: subarachnoid haemorrhage; CVA: cerebrovascular accident/stroke unspecified; TIA: transient ischaemic attack; ICH: intracerebral haemorrhage.

An a priori power analysis was conducted to examine construct validity at a correlation of ⩾.30 for convergent correlations, with 80% power, one-sided, with an alpha level of .05. ²² This indicated a minimum sample requirement of 66 participants. No a priori power analysis was conducted for sensitivity analyses and all available data was used.

Cognitive data

All participants completed the OCS-Plus. The OCS-Plus includes 10 subtasks and scores 18 impairment types based on normative data. See previously published studies for further detail. ²² Note, the OCS-Plus android and iOS application is available to clinicians and academics for research or service improvement-related activities via a free licence (see www.ocs-test.org/ocs-plus). All participants also completed the OCS ⁴ as a first-line cognitive screening tool for stroke.

Further cognitive data for validation in the subacute group was collected in several sessions. In the first session, OCS-Plus ²² was conducted, then, based on convenience and availability of both researcher and participant, up to two follow up sessions were conducted to collect further validation data with MoCA ¹⁴ and the neuropsychological test battery. Order of administration was dependent on the examiner with RR first administering MoCA and SSW first administering the test battery. Figure 1 depicts a flowchart and participant numbers completing each of the follow-up sessions. The chronic stroke survivors only completed a single session.

Figure 1.

Flow diagram of the data collection streams across subacute (<3 months post-stroke) and chronic (⩾ 6 months post-stroke) for the convergent validity portion of the current study. Figure available under CC-by 4.0 license https://osf.io/dtmke

OCS-plus and OCS

All chronic stroke survivors completed both OCS ⁴ and OCS-Plus in a single session (n = 166). The OCS-Plus research version was administered via a MATLAB ²⁶ executable file on a Windows Surface Pro tablet.

Of the 181 sub-acute participants who completed OCS-Plus, 178 had completed OCS as part of standard clinical screening practice within 30 days prior to OCS-Plus and this data was collected via the patient records.

Validation tests

MoCA was completed as a comparison to a commonly used clinical screening tool as part of a DClinPsych thesis project (author RR) in 80 participants.

Domain-specific matched neuropsychological tests were completed by 80 sub-acute participants. These included: the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) figure copy and figure recall tasks which validly measures visuospatial perception/construction and memory ²⁷ ; Behavioural Inattention Test (BIT): Star Cancellation which validly measures visuospatial neglect ²⁸ ; Brixton Spatial Anticipation (Brixton) test which validly and reliably measures executive functioning^29–31; and the Cognitive Linguistic Quick Test: Symbol Trails (CLQT) which validly measures both visuospatial scanning and executive functioning. ³² Corsi block test is a computer-tablet form of the Corsi block working memory test. ³³

Performance in all tasks was scored using established cut-offs (Supplemental Table S1).^{4,22,28,30,32,34,35} Licence requirements for use of the MoCA were met.

Data analysis

First, task-specific OCS-Plus convergent and divergent validity was examined against task-matched validated neuropsychological tests in a pre-registered Spearman’s Rho correlation analyses. We aimed for correlations >0.30 to demarcate convergence ³⁶ but also cautiously interpret correlations >0.19. ²²

Second, we investigated OCS-Plus impairment incidence and impairment sensitivity and specificity. Task-by-task OCS-Plus impairment incidence was calculated as the number of stroke survivors classified as impaired on each task (compared to age-specific normative data ²² ), divided by the number of stroke survivors who completed that task. Impairment sensitivity on OCS-Plus tasks was examined versus that of first line cognitive screening tools (MoCA and OCS) using published cut-offs (see overview in Supplemental Table S1). To further investigate the sensitivity of OCS-Plus relative to gold standard neuropsychological assessment, OCS-Plus sensitivity was compared to impairment detected on neuropsychological tests.^{4,22,28,30,32,34,35} In addition, to determine whether OCS-Plus indeed detects more subtle cognitive impairments, we calculated the proportion of stroke survivors classified as unimpaired using established cut-offs on clinically used tools (OCS and MoCA) who showed cognitive impairment on the OCS-Plus. Importantly, an assumption is made that following a recent stroke, cognition is expected to be affected in comparison to a healthy ageing normative group.

No missing data for validation analyses were imputed, each analysis was only conducted on those who had complete data for each analysis, degrees of freedom or absolute N are reported per analysis transparently.

Analysis software and scripts

Data wrangling and statistical analyses were completed MATLAB and R Studio ³⁷ (R packages: bookdown, ³⁸ yardstick, ³⁹ readxl, ⁴⁰ pROC, ⁴¹ knitr, ⁴² ggplot2, ⁴³ magick, ⁴⁴ webshot, ⁴⁵ kableExtra ⁴⁶ ). Collated data and code are available through the Open Science Framework (https://osf.io/t8zug/).

Results

Nine participants took greater than one session to complete the OCS-Plus due to fatigue or interruptions. Including only those who completed the OCS-Plus in 1 day, for time taken to complete the OCS-Plus, the chronic sample took on a median of 24 min 24 s and the subacute sample took 21 min 0 s.

Missing data on specific tasks could be due to motor, visual or perceptual impairments that could not be compensated for, or due to ward-based interruptions where rehabilitation was prioritised. We present the reasons for non-completion of OCS-Plus subtasks in Table 2.

Table 2.

Inclusion and reasons for not testing on all subtests of the OCS-Plus for chronic and some subacute stroke survivors.

Subtest	% Not complete	% Visual	% Motor	% Language	% Ran out of time	% Interruptions	% Fatigue	% Other	% Technical
Picture naming	1.01	0.51	0	0	0	0	0	0.51	0
Semantics	1.01	0.51	0	0	0	0	0	0.51	0
Orientation	3.54	0	0	1.01	0	0	1.01	1.01	0
Encoding	2.02	0	0	1.52	0	0	0	0.51	0
Trails	4.04	1.01	0.51	0.51	0	0	1.52	0.51	0
Word recall and episodic memory	5.56	0.51	0	1.01	0	0	2.53	1.01	0.51
Rule Finding	26.77	1.52	3.54	0.51	0.51	0	3.03	16.67	0
Figure Copy	20.71	1.01	0	0.51	3.54	0.51	6.57	7.07	0.51
Cancellation	7.58	1.52	0.51	0.51	2.02	0.51	1.52	0.51	0.51

Condition of testing data were only available for 138 chronic stroke survivors and a subset of 60 subacute stroke survivors, due to differences in data collection protocols.

Construct validity

First, we examined OCS-Plus task performance and construct validity. Figure 2 presents key density plots showing OCS-Plus performance distributions for healthy controls, sub-acute, and chronic stroke survivors. The figure shows that some tasks are very specific regarding identifying who does not have an impairment, while others are very sensitive to detecting impairment, this is in compliment to later sensitivity/specificity analysis. The stroke cohorts were matched for age and education. These plots show that healthy controls perform better than chronic stroke survivors on all tasks, who in turn performed better than sub-acute stroke survivors (See Supplemental Figure S1 for all density plots). Table 3 presents correlations between OCS-Plus tasks and matched neuropsychological tests. All OCS-Plus tasks showed convergent validity against neuropsychological tests. All OCS-Plus tasks showed divergent validity, except the OCS-Plus language and orientation tasks, which also correlated with visuo-spatial tests.

Figure 2.

Group performance density plots per OCS-Plus task for healthy ageing adults, as well as sub-acute (<3 months post-stroke) and chronic (⩾ 6 months post-stroke) stroke samples. Figure available under CC-by 4.0 licence https://osf.io/m3kc4.

Table 3.

Convergent correlations between OCS-Plus tests and construct- and format-matched neuropsychological validation tests.

OCS-Plus task	Convergent test	Convergent Correlation	Divergent test	Divergent Correlation
Picture Naming Accuracy	OCS Naming Accuracy	r(340) = 0.35, p < 0.001	RBANS Figure Copy Accuracy	r(73) = 0.33, p = 0.003
Picture Naming Accuracy	MoCA Naming Accuracy	r(78) = 0.34, p = 0.002
Semantics Accuracy	OCS Semantics Accuracy	r(340) = 0.17, p = 0.002	RBANS Figure Copy Accuracy	r(73) = 0.33, p = 0.004
Orientation Accuracy	OCS Orientation Accuracy	r(335) = 0.38, p < 0.001	RBANS Figure Copy Accuracy	r(73) = 0.26, p = 0.027
Orientation Accuracy	MoCA Orientation Accuracy	r(78) = 0.58, p < 0.001
Encoding 1 Accuracy	MoCA Encoding 1 Accuracy	r(77) = 0.5, p < 0.001	RBANS Figure Copy Accuracy	r(72) = 0.15, p = 0.207
Encoding 2 Accuracy	MoCA Encoding 2 Accuracy	r(77) = 0.53, p < 0.001	RBANS Figure Copy Accuracy	r(72) = 0.21, p = 0.076
Trails A Accuracy	OCS Trail A Accuracy	r(268) = 0.31, p < 0.001	MoCA Naming Accuracy	r(74) = 0.17, p = 0.154
Trails A Accuracy	CLQT A Accuracy	r(73) = 0.27, p = 0.018
Trails A Time	OCS Trail A Time	r(110) = 0.38, p < 0.001	MoCA Naming Accuracy	r(74) = 0.07, p = 0.532
Trails A Time	CLQT A Time	r(68) = 0.5, p < 0.001
Trails B Accuracy	OCS Mixed Accuracy	r(265) = 0.4, p < 0.001	MoCA Naming Accuracy	r(74) = 0.06, p = 0.597
Trails B Accuracy	CLQT B Accuracy	r(73) = 0.49, p < 0.001
Trails B Time	OCS Mixed Time	r(98) = 0.33, p < 0.001	MoCA Naming Accuracy	r(74) = 0.01, p = 0.906
Trails B Time	CLQT B Time	r(67) = 0.5, p < 0.001
Trails Exec. Score	OCS Executive Score	r(309) = 0.17, p = 0.002	MoCA Naming Accuracy	r(74) = 0.07, p = 0.558
Delayed Recall + Recognition Accuracy	MoCA Word Recall Accuracy	r(77) = 0.14, p = 0.22	RBANS Figure Copy Accuracy	r(71) = 0.05, p = 0.685
Delayed Recall + Recognition Accuracy	OCS Sentence Recall Accuracy	r(335) = 0.43, p < 0.001
Episodic Recognition Accuracy	OCS Episodic Memory Accuracy	r(338) = 0.24, p < 0.001	RBANS Figure Copy Accuracy	r(71) = 0.27, p = 0.023
Figure Copy Accuracy	RBANS Figure Copy Accuracy	r(69) = 0.66, p < 0.001	MoCA Naming Accuracy	r(70) = 0.33, p = 0.004
Figure Recall Accuracy	RBANS Figure Recall Accuracy	r(68) = 0.64, p < 0.001	MoCA Naming Accuracy	r(69) = 0.29, p = 0.014
Rule Finding Accuracy	Brixton Spatial Anticipation Test Accuracy	r(63) = 0.62, p < 0.001	MoCA Naming Accuracy	r(73) = 0.23, p = 0.045
Rule Finding Rules Learned	Brixton Spatial Anticipation Test Rules Learned	r(63) = 0.61, p < 0.001	MoCA Naming Accuracy	r(73) = 0.23, p = 0.051
Rule Finding Time	Brixton Spatial Anticipation Test Time	r(52) = 0.21, p = 0.134	MoCA Naming Accuracy	r(71) = 0.13, p = 0.256
Cancellation Accuracy	BIT Star Cancellation Accuracy	r(71) = 0.78, p < 0.001	MoCA Naming Accuracy	r(71) = 0.22, p = 0.061
Cancellation False Positives	BIT Star Cancellation False Positives	r(71) = 0.65, p < 0.001	MoCA Naming Accuracy	r(71) = 0.04, p = 0.707
Invisible Cancellation Accuracy	BIT Star Cancellation Accuracy	r(71) = 0.58, p < 0.001	MoCA Naming Accuracy	r(71) = 0.14, p = 0.224
Invisible Cancellation Correct Revisits	Corsi Block Accuracy	r(40) = 0.1, p = 0.517	MoCA Naming Accuracy	r(71) = 0.1, p = 0.4

Incidence and Sensitivity Analyses

Next, we investigated OCS-Plus task impairment incidences. Figure 3 shows the task-by-task impairment incidence for each stroke cohort. The OCS-Plus invisible cancellation task showed the highest impairment incidence for both samples (sub-acute 85.16% impaired; chronic 76.73% impaired), whereas the semantics task showed the lowest impairment incidence (subacute 10.56% impaired; chronic 5.45% impaired). A numerical decrease in impairment rate was observed in the chronic sample, versus the subacute sample, for all measures. However, the Welch Two Sample t-test found no significant difference in incidence between the samples (t(28) = 1.39, p = 0.18, d = −0.51).

Figure 3.

The proportion of stroke survivors classified as impaired on each OCS-Plus subtask are shown separately for sub-acute (<3 months post stroke) and chronic stroke (>6 months post-stroke) survivors and left hemisphere and right-hemisphere lesions. Figure available under CC-by 4.0 licence https://osf.io/3qnhx.

Next, we investigated OCS-Plus sensitivity and specificity for each subtask on matched neuropsychological tasks. Table 4 presents results of these analyses. Simpler tasks (e.g., picture naming, semantics, orientation, word encoding, delayed recall and delayed recall and recognition, and episodic recognition) on the OCS-Plus had high specificities (>0.80) at a cost of sensitivity (<0.50), however, more complex tasks in the OCS-Plus (e.g., rule finding, figure copy/recall, and cancellation) which were designed to detect subtle deficits had exceptionally high sensitivity (>0.90) and moderate to high specificity. Only the Invisible Cancellation task had a very low specificity (<0.20), though had perfect sensitivity.

Table 4.

Comparison of the OCS-Plus subtask impairment classifications to validation test impairment classifications in subacute (<3 months post-stroke) and chronic (>6 months post-stroke) stroke samples.

OCS-Plus Task	Validation Test	N	True positive	True negative	False positive	False negative	Sensitivity	Specificity
Picture Naming	OCS Picture Naming	342	11.40	64.04	9.65	14.91	43.33	86.90
Semantics	OCS Semantics	342	2.05	83.63	5.85	8.48	19.44	93.46
Orientation	OCS Orientation	337	11.28	63.50	12.46	12.76	46.91	83.59
Trails Exec. Score	CLQT Score	75	21.33	34.67	20.00	24.00	47.06	63.41
Delayed Recall	OCS Delayed Recall	295	14.92	53.22	8.81	23.05	39.29	85.79
Delayed Recall plus Recognition	OCS Delayed Recall plus Recognition	337	13.65	62.02	8.61	15.73	46.46	87.82
Episodic Recognition	OCS Episodic Recognition	340	8.24	63.82	15.59	12.35	40.00	80.37
Rule Finding	Brixton Errors	65	13.85	61.54	23.08	1.54	90.00	72.73
Figure Copy	RBANS Figure Copy	71	18.31	53.52	26.76	1.41	92.86	66.67
Figure Recall	RBANS Figure Recall	71	12.68	61.97	23.94	1.41	90.00	72.13
Cancellation	BIT Cancellation	73	28.77	58.90	9.59	2.74	91.30	86.00
Invisible Cancellation	Corsi Block	42	28.57	11.90	59.52	0.00	100.00	16.67

We use validation comparison tests as the ‘ground truth’ for impairment classifications and compare rates of true/false positive/negative impairment identifications.

Next, we investigated OCS-Plus impairment classifications versus impairment classifications on clinically used first line cognitive screening tools (OCS and MoCA). Impairment classifications were determined based on previously published cut-offs. Overall, 87.50% of the sub-acute sample scored below MoCA cut-off of 26-points, and 72.50% below the 23-point cut-off. Of those classified as ‘unimpaired’ on MoCA, 100% were impaired in at least one OCS-Plus test. On the OCS, 96.22% of stroke survivors showed a cognitive impairment. Of the remaining 13 participants without an impairment on any of the OCS domains, 12 were impaired in at least one OCS-Plus test. This gives OCS-Plus a sensitivity of 100% vs MoCA and 98.5% vs OCS.

Discussion

We conducted a psychometric validation of the OCS-Plus in a sub-acute and chronic stroke survivor cohort, and followed COSMIN criteria ²⁵ for reporting construct validity. First, we confirmed convergent construct validity for all OCS-Plus subtasks and divergent construct validity for all OCS-Plus subtasks, except the OCS-Plus language and orientation tasks, which related to visuo-spatial assessments. In addition, Figure drawing related to language assessments and episodic visuo-spatial assessments. Second, the OCS-Plus showed near perfect sensitivity for detecting subtle post-stroke cognitive impairments compared to two clinically used cognitive screening tools (MoCA and OCS). Overall, we demonstrated that the OCS-Plus is a valid and sensitive cognitive screening tool for subtle post-stroke cognitive impairments, with sensitivity comparable to detailed neuropsychological assessments. By validating the OCS-Plus in a large real-world clinical rehabilitation sample and long-term chronic survivors, these results will be generalisable to a large extent. Limitations may apply in terms of comparability of demographic factors – such as age profiles of the stroke population (here, average age at stroke was 72 years) – and clinical factors – such as stroke severity (here, average NIHSS was 7.5).

First, we demonstrated convergent and divergent validity for most OCS-Plus subtasks using subtask-matched validated standardised neuropsychological tests. The unanticipated correlation between the OCS-Plus language subtasks with the visuo-spatial assessments may be explained by the visuo-spatial components of the OCS-Plus language subtasks. For example, the OCS-Plus Picture Naming task requires visual recognition of images. Therefore, stroke survivors with visuo-spatial deficits may struggle with this task, independent of their language ability.

Second, we showed that the OCS-Plus was more sensitive in detecting subtle cognitive impairments than other widely used cognitive screening tools (OCS and MoCA). Nevertheless, task-by-task sensitivity analyses indicated that tasks with a relatively small range of available scores – such as OCS-Plus Picture Naming, Semantics, and Orientation – had higher specificity than sensitivity. This suggests there is a trade-off in terms of task complexity and specificity/sensitivity. As such, more complex OCS-Plus tasks (e.g. Rule Finding) may be ideal for detecting subtle domain-general cognitive effects from broader vascular factors, linked to cognitive hallmarks of vascular dementia and small vessel disease (e.g. executive dysfunction ²⁰ ), whereas the simpler OCS-Plus tasks may be better suited for detecting core deficits (e.g. aphasia, orientation). By combining both types of tasks in OCS-plus, the tool provides a time efficient approach to screening for both core cognitive impairments and more subtle vascular-related post-stroke cognitive impairments. This provides a representative snapshot of post-stroke cognition, where initial domain-specific impairments may be improving or stable^18,19,47 and domain-general vascular and neurodegenerative factors may impede cognitive recovery. ²⁰

Third, our data found numerically, but not statistically, lower incidence of cognitive impairments in the chronic stages post-stroke compared to the earlier subacute stage. Nevertheless, impairment prevalence remained high for the Invisible Cancellation task, a sensitive test of working memory. It may be that these chronic working memory deficits reflect not only stroke-related damage, but also more subtle vascular-related damage that accrues during ageing. As such, these deficits may be less amenable to recovery after stroke, which may explain the consistently high impairment prevalence in the chronic stage post-stroke. The higher rate of chronic impairment for the Invisible Cancellation subtask, rather than other memory subtasks, may be explained by its increased load on working memory. More specifically, the Invisible Cancellation subtask requires participants to store in working memory which targets have already been selected and their location, which loads more heavily onto working memory than simply recalling a figure or words.

The OCS-Plus offers several advantages over currently used paper-based clinical screening tools and neuropsychological test batteries. First, the OCS-Plus report gives clear information about both domain-general and domain-specific cognitive performance, in contrast to traditional paper-based screening tools such as the MMSE and MoCA, which provide a coarser evaluation of cognitive functioning overall.^13,14 Secondly, the OCS-Plus is available on a platform independent app that provides standardised administration instructions for the user and automatically scores participants against age-adjusted impairment cut-offs. This contrasts with currently used tools that require manual test scoring.^13,14 Manual scoring may increase the time burden of administration and may also increase error in the scoring process, relative to automated approaches. Thirdly, the OCS-Plus takes on average 24 min to administer ²² and thus offers substantial time advantages relative to extensive neuropsychological test batteries, which can take upwards of 1 h to administer.

These features of the OCS-Plus should be considered in the context of clinical practice. Firstly, clinicians could use fine-grained information about domain-specific and domain-general cognitive functioning – in addition to other factors – to detail prognosis and recovery, and aid conversations around adjustment to living life post-stroke. ⁴⁸ Secondly, as the OCS-Plus app provides standardised administration instructions, it could be used in clinical practice by a range of allied health professionals, without specific neuropsychology training, which is required to administer neuropsychological test batteries. Thirdly, where time is pressured, having an automatically scored tool could accelerate the assessment process and return crucial time to clinicians for other aspects of assessment, providing potential cost savings. Finally, a tool that is quick to administer and highlights subtle cognitive deficits ²² may be a valuable adjunct in discharge planning and education of patients. Overall, these factors suggest that the OCS-Plus may be a valuable cognitive screening tool for use in clinical populations, particularly in stroke survivors who may present with a mixture of domain-specific and domain-general cognitive changes.

Several limitations should be noted with regards to both the present study method and the OCS-Plus tool itself. With regards to the study method, our analyses contain different sample sizes as some participants did not complete all sessions, due to both patient-specific and environmental factors. This made it difficult to correct for multiple comparisons, and we regret that statistical power varies between analyses. Second, most participants completed tests across several brief sessions, due to factors such as fatigue. Post-stroke cognition is not constant but dynamically changing, and stroke survivors’ cognitive abilities may have fluctuated over these intervals. We attempted to mitigate this issue by ensuring that all validation tasks were completed within a maximum period of 30 days. With regards to limitations of the OCS-Plus tool itself: in some tasks, the images may have insufficient contrast for patients with pre-existing visual impairments, such as macular degeneration (e.g., see ⁴⁹ ), and this may be exacerbated on a reflective tablet surface. Practical issues associated with tablet-based testing like running out of charge, or cracks on the screen may further impact testing. Nevertheless, these issues can be considered relatively minor. Therefore, they should not – in theory – impede the use of the OCS-Plus in research studies and/or clinical practice.

Overall, the OCS-Plus is a valid and sensitive cognitive screening tool which includes detecting more subtle cognitive impairment in stroke survivors. Indeed, the OCS-Plus was found to detect cognitive impairments in a large sample of subacute and chronic stroke survivors at a similar level to selected standardized and validated neuropsychological tests, while offering substantial practical and time advantages over these tests. As such, the OCS-Plus could be considered for implementation in clinical practice. Future research could attempt to disentangle domain-specific and domain-general cognition trajectories and underlying neuroanatomical correlates using OCS-Plus. In addition, the validity of using the OCS-Plus in different clinical cohorts, which may similarly require more sensitive cognitive screening, should be investigated.