COGNITIVE PERFORMANCE SCORES FOR THE PEDIATRIC AUTOMATED NEUROPSYCHOLOGICAL ASSESSMENT METRICS IN CHILDHOOD-ONSET SYSTEMIC LUPUS ERYTHEMATOSUS

Abstract

Objective

To develop and initially validate a global cognitive performance score for the Pediatric Automated Neuropsychological Assessment Metrics (PedANAM-CPS) to serve as a screening tool of cognition in childhood lupus.

Methods

Patients (n=166) completed the nine subtests of the PedANAM battery, each of which provides three principal performance parameters (accuracy, mean reaction time for correct responses, throughput). Cognitive ability was measured by formal neurocognitive testing or estimated by the Pediatric Perceived Cognitive Function Questionnaire-43 to determine the presence or absence of neurocognitive dysfunction (NCD). A subset of the data was used to develop four candidate PedANAM-CPS indices with supervised or unsupervised statistical approaches: PedANAM-CPS_UWA i.e. unweighted averages of the accuracy scores of all PedANAM-subtests; PedANAM-CPS_PCA, i.e. accuracy scores of all PedANAM-subtests weighted through principal components analysis; PedANAM-CPS_logit i.e. algorithm derived from logistic models to estimate NCD-status based on the accuracy scores of all of the PedANAM-subtests; and PedANAM-CPS_multiscore i.e. algorithm derived from logistic models to estimate NCD-status based on select PedANAM performance parameters. Using the remaining data PedANAM-CPS candidates were validated.

Results

PedANAM-CPS indices were moderately correlated with each other (|r|>0.65). All of the PedANAM-CPS’s discriminated children by NCD-status across datasets (p<0.036). The PedANAM-CPS_multiscore had the highest area under the receiver operating characteristic curve (AUC) across all datasets for identifying NCD-status (AUC >0.74), followed by the PedANAM-CPS_logit, the PedANAM-CPS_PCA and the PedANAM-CPS_UWA respectively.

Conclusions

Based on preliminary validation and considering ease of use, the PedANAM-CPS_multiscore and the PedANAM-CPS_PCA appear to be best suited as global measures of PedANAM performance.

Systemic Lupus Erythematosus is a relatively common autoimmune disease which affects an estimated 1 in 10,000 children in the United States (1, 2). Childhood-onset SLE (cSLE) is often associated with major organ involvement, including neuropsychiatric disease (NPSLE) affecting between 43–95% of patients (1, 2). Although it is a risk factor for poor disease outcomes in all age groups (3–8), NPSLE is of special concern in pediatric populations. Children with NPSLE may experience school failure due to attention and learning difficulties (9). Additionally, NPSLE affecting the maturing brain may have significant long-term consequences, as initial evidence demonstrates that NPSLE interferes with developmentally appropriate acquisition of new cognitive skills (10).

Screening has been defined as the systematic application of a test for the early identification of individuals at risk for a specific disorder who will benefit from further investigation (11). Within a general pediatric rheumatology practice context, there is a need for an efficient screening test for monitoring cognition and for early detection of clinically relevant neurocognitive dysfunction (NCD) in cSLE patients (12). Effective screening can support early identification of individuals who require more comprehensive assessment of cognitive ability as well as help guide medical treatment decisions.

Neurocognitive function may be considered a measure of global brain health (13). Traditionally, neurocognitive function is assessed through formal neurocognitive testing (FNCT) using a battery of individually-administered standardized tests that probe various cognitive domains (14). Although FNCT generally is considered the criterion standard for quantifying cognitive ability, it has its shortcomings. FNCT is time-consuming, costly, can be difficult to access, and serial administration of FNCT can result in training effects which complicate its interpretation. Taken together, these issues suggest that FNCT is not the ideal route for surveillance of NPSLE in children.

The Pediatric Automated Neuropsychological Assessment Metrics (PedANAM) is a computerized library of tests designed to measure cognitive ability, mental processing speed, memory, and cognitive efficiency in children age 9 years and older(15). Recent studies suggest that the PedANAM can be used for the screening of NCD in cSLE (16). The PedANAM is time efficient, has minimal practice effects and only requires access to a standard computer without need for other specialized equipment, training, or a psychometric specialist.

One shortcoming of the PedANAM is related to its complexity and its lack of a validated summary or overall performance statistic. It is a data rich instrument that yields multiple scores for each subtest, including scores for response speed, accuracy, variability, and speed/accuracy trade-offs. The large amount of data generated by the PedANAM cannot be easily synthesized in a clinical setting to help determine whether a child’s overall cognitive performance has changed. This differs from the adult version of the ANAM, for which various composite scores have been proposed, including summary statistics derived from regression models or averaged ANAM performance scores (17, 18).

The current study sought to use common statistical methods, such as those proposed for the ANAM, to develop a summary cognitive performance score (PedANAM-CPS). Specifically, the objective of this study was to develop and initially validate the calculation of a single composite index (PedANAM-CPS) to serve as a summary measure of children’s cognitive performance on the PedANAM.

PATIENTS AND METHODS

This study was approved by the institutional review boards at each participating center with written consent and assent obtained as appropriate. Participants were 9 to 17 years old at the time of study enrollment and carried either a diagnosis of cSLE (n=108) (19) or juvenile idiopathic arthritis (JIA) (n=18) (20), or were healthy controls (n=40). JIA patients were used as controls as it was felt to be a population easy to access at a rheumatology clinic and not to have abnormal cognitive functioning related to the primary rheumatologic diagnosis. Healthy controls were cSLE patients’ friends within one year if their age, of the same gender, and in the same school grade. English was the primary language of participants and their caregivers. Routine sociodemographic and clinical data were collected. For participants with cSLE disease activity (SLEDAI) (3), laboratory data, and medications were recorded throughout the study course.

Definition of Datasets

FNCT and PedANAM were completed by 40 cSLE patients and 40 age and sex-matched healthy controls at baseline and 18 months later. Data collected from these 80 participants at baseline served as the development-dataset and were used to derive weightings for all of the PedANAM-CPS indices.Validation-dataset-1 was comprised of data from 61 of the 80 participants mentioned above for which 18-month follow-up data (FNCT, PedANAM) were available.

Validation-dataset-2 was collected from a separate sample, with cross-sectional assessment of cSLE patients (n=68; all different from those described above) and 18 JIA patients followed at seven pediatric rheumatology centers. These participants completed the PedANAM but were not administer FNCT. Instead, cognitive ability was estimated via caretaker proxy-report using the Pediatric Perceived Cognitive Function Questionnaire-43 (PedsPCF-43, see details below).

Measurements

Pediatric Automated Neuropsychological Assessment Metrics

The PedANAM is a battery of computer administered and scored tests of cognitive processing efficiency designed for repeated administrations to individuals ages 9 years and older. PedANAM subtests were adapted from the adult version of ANAM to display age-appropriate stimuli and instructions and to allow sufficient time for responses in a pediatric population(15).The PedANAM subtests included in this study were: Code Substitution Learning, Spatial Processing, Mathematical Processing, Matching to Sample Test, Sternberg Memory Search, Code Substitution Delayed Memory Test, Logical Relations Symbolic Test, Matching Grids Test, and Continuous Performance Test.

Each subtest produces at least three performance parameter scores: accuracy (percent correct responses), mean reaction time (for correct responses), and throughput (correct responses/minute) as a measurement of efficiency. Of all of the PedANAM performance parameters, the accuracy score has shown the highest consistency and reliability (16). Additionally, the coefficient of variation of time required for a correct response (CVc) has been proposed as a measure of consistency on the performance of a given PedANAM-subtest (21).

Measurement of Cognitive Ability

Various approaches for the assessment of cognitive functioning have been proposed in the medical literature and for cSLE (22). At present, there is no consensus about the most appropriate definition of a clinically relevant deficit or impairment in cognitive performance in cSLE (23). Therefore, we assessed cognitive ability in two ways: via FNCT and the PedsPCF-43. Given the limited sample size, we restricted analysis to a single definition of NCD for the FNCT and the PedsPCF-43, respectively.

Formal Neuropsychological Testing

FNCT was administered to the development group and validation group-1 by a trained psychometrician, using a recommended standardized neuropsychological battery for cSLE (14). The tests can be clustered into four cognitive domains as previously reported: Working Memory, Psychomotor Speed, Visuoconstructional Ability, and Attention/Executive Functioning. Using published norms, participants’ performance on each of the neuropsychological tests was expressed as a Z-score with a mean of 0 and standard deviation of 1. Following the definition used in previous studies (16, 21, 23), NCD was operationally defined as having at least one domain Z-score below −2 standard deviation or at least two domain Z-scores below −1 standard deviation on FNCT.

Pediatric perceived cognitive function questionnaire-43

The PedPCF-43 measures the caregiver’s perceptions of a child’s cognitive functioning as observed in everyday life (24). This questionnaire was completed by the caregivers of validation group-2. PedsPCF-43 items were determined via Item Response Theory (IRT) analyses with input from parents, teachers, and clinicians experienced at working with pediatric cancer survivors. The focus of the PedsPCF-43 is on fluid cognitive abilities sensitive to changes in mental status secondary to neurologic and systemic medical events. It measures aspects of cognitive functioning such as attention, memory retrieval, and working memory. PedsPCF-43 correlates with computerized neuropsychological testing and neuroimaging findings (25–27). Its reliability, validity, and clinical utility have been investigated in the U.S. general population and pediatric cancer survivors (28). However, the PedsPCF-43 has not been validated for use in cSLE. Ratings on the PedsPCF-43 are expressed in gender and age-standardized scores (T-scores), with lower scores indicating greater cognitive dysfunction. In this study, participants with a T-score <50 on PedsPCF-43 completion were classified as having NCD.

Statistical Analysis

Information collected in all groups included baseline disease characteristics, ongoing disease activity, physical examination, standard clinical and laboratory information, and PedANAM performance. Numeric variables were summarized by means and standard deviations (SD), and categorical variables by percentage values.

General Plan

As cognition is closely related to brain development, and given the influence of age on PedANAM performance (29), our analyses were statistically adjusted for age. However, analysis of unadjusted data did not differ significantly from analysis of adjusted data; therefore, only adjusted analyses are presented. It has been suggested that PedANAM accuracy scores are especially suited to capture the presence of NCD (16). Hence, accuracy scores were considered for the derivation of all candidate PedANAM-CPS indices with the exception of the PedANAM-CPS_multiscore. We had previously derived the PedANAM-CPS_multiscore by logistic regression considering various PedANAM performance parameters as predictors. Performance parameters from each PedANAM-subtest were normalized using Z-scores, i.e. Z-score= [(raw test score-mean)/test standard deviation of the mean] prior to consideration in statistical analyses. Candidate PedANAM-CPS indices were then developed by applying the different statistical methods described below with data from the development group. For validation purposes, the performances of the PedANAM-CPS indices were evaluated using data from validation datasets 1 and 2.

Composition of four candidate PedANAM-CPS indices

We used statistical modeling methods previously proposed for the development of summary scores of the instrument with details provided below.

PedANAM-CPS_multiscore

Based on earlier analyses using the development dataset, this score has a reported sensitivity and specificity to detect NCD of 100% and 86%, respectively (21). Detailed description of the methods used for derivation of the PedANAM-CPS_multiscore can be found elsewhere (21). In brief, the PedANAM-CPS_multiscore includes the accuracy score of the PedANAM Spatial Processing test, the CVc of reaction time for correct responses on the Continuous Performance and Matching-to-Sample tests, and the mean reaction time for correct responses on the Code Substitution Delayed test.

PedANAM-CPS_UWA

This PedANAM-CPS index constitutes an unweighted average of each of the nine subtests’ (normalized) accuracy scores.

PedANAM-CPS_logit

This PedANAM-CPS index is based on an algorithm derived from a logistic regression model to predict the dependent variable NCD (yes/no as based on FNCT), using all (normalized) subtests’ accuracy scores as independent (i.e. predictor) variables. The intercept and slope coefficients from this model were used to combine accuracy scores from the individual Ped-ANAM-subtests into a single predicted Logit score for each participant. Since PedANAM-CPS_logit utilizes information from both the criterion standard (FNCT result) as the dependent variable and from the PedANAM subtests as independent variables during the modeling process, it can be considered a supervised statistical method for summary score derivation.

PedANAM-CPS_PCA

In Principal Components Analysis (PCA) the variance-covariance matrix of the (normalized) accuracy scores of PedANAM subtests was decomposed into a series of eigenvectors with corresponding eigenvalues. Each eigenvalue constitutes the variance of the linear combination of all test accuracy scores weighted by values contained in the corresponding eigenvector. In order to preserve the largest portion of the total variance of the variance-covariance matrix, the first eigenvector was used in the derivation of the PedANAM-CPS_PCA. Since a criterion standard (FNCT) is not required for PCA, it is considered a non-supervised statistical method for summary score derivation.

PedANAM-CPS interpretation

Due to their methods of development, the higher the PedANAM-CPS_UWA or the PedANAM-CPS_PCA index the higher the performance accuracy on the PedANAM subtests and the lower the probability of NCD. In contrast, higher PedANAM-CPS_logit or PedANAM-CPS_multiscore values indicate higher probability of NCD.

Validation of candidate PedANAM-CPS

Following their initial development, the PedANAM-CPS indices were assessed for differences in mean values for groups of participants with different NCD-status using a Student’s t-test. Pearson’s correlation coefficients were calculated to assess relationships among the candidate PedANAM-CPS indices.

To further assess the measurement properties of the candidate PedANAM-CPS indices, a receiver operating characteristics (ROC) curve analysis was done for identifying NCD-status (yes/no). Overall precision for correctly identifying NCD-status was determined by calculating the area under the ROC curve (AUC). In addition, the sensitivity and specificity of each index were calculated under a preferred threshold approach at 80% or higher.

The ability of the candidate PedANAM-CPS indices to identify NCD (from FNCT) was first evaluated in the development group and then re-tested using validation datasets-1 and −2, defining NCD-status based on FNCT and Ped-PCF43, respectively. Values of the AUC can be interpreted as outstanding (1.0–0.91), excellent (0.81–0.90), good (0.71–0.8), fair (0.61–0.7), and poor (<0.60) performance in predicting NCD (30).

RESULTS

Study participants

Demographics of the participants and disease information are presented in Table 1. cSLE patients and controls enrolled in the development and validation cohorts did not differ on sociodemographic variables (maternal education level, family income). Over 80% of the participants were female and at least a third of those with cSLE were African American. The age (mean±SD) of participants with cSLE in the development group and the validation groups 1 and 2 was comparable, as was their disease activity. Based on FNCT, NCD was present in 22.5% of the cSLE participants of the development group.

Table 1.

Demographics of Development-group and Validation-groups at enrollment ^*

Variable	Category	Development Group			Validation Group-1			Validation Group-2
		cSLE (n=40)	Control (n=40)	p-value	cSLE (n=34)	Control (n=27)	p-value	cSLE (n=68)	Control (n=18)	p-value
Age (years)		14.8 ± 2.3	13.9 ± 3.2	0.21	16.17 ± 2.3	15.6 ± 2.3	0.34	15.3 ± 3.3	14.0 ± 2.5	0.131
Female (%)	Yes	85	85	1.0	82.4	85.2	0.77	91.2	72.2	0.032
Ethnicity (%)
White		30	32.5	0.98	29.4	33.3	0.96	30.9	94.4	<0.001
African American		45	47.5		47.1	40.7		30.9	5.6
Hispanic		17.5	15		14.7	14.8		11.8	0
Asian and other		7.5	5		8.8	11.1		26.5	0
Grade Level (%)
Elementary School (4–6)		20	20	1.0	13.3	16.7	0.91	18.6	22.2	0.321
Middle School (7–8)		17.5	17.5		6.7	8.3		15.3	33.3
High School (9–12)		62.5	62.5		80	75		55.9	38.9
College		0	0		0	0		10.2	5.6
Maternal education level (%)
No High School Diploma		7.5	10	0.7	3.1	3.9	0.72	4.5	5.6	0.523
Complete High School Diploma		30	37.5		21.9	30.1		16.7	11.1
Education Beyond High School		62.5	52.2		75	65.4		78.8	83.3
Annual Family Income
< $ 25K		20	15.8	0.81	24.2	7.4	0.24	14.3	5.6	0.057
$26K-$50K		35	34.2		27.3	33.3		31.7	5.6
$51K-$75K		20	28.9		18.2	33.3		12.7	16.7
>$75K		25	21.1		30.3	25.9		41.3	72.1
Treated with oral steroids (%)		77.5			50			76.5
Prednisone dose (mg/day)		19.8 ± 17.4			10.15 ±10.2			17.1 ± 16.1
Disease activity, SLEDAI‡		4.9 ± 4.4			4.2 ± 3.6			4.3 ± 4.7
PCF-43 (T-score)†		-	-		-	-		60.5 ± 7.9	63.2 ± 5.8	0.167
Neurocogn. dysfunction (%)¥		22.5	7.5	0.06	18.2	14.3	0.682	8.8	11.1	0.766

^*Values are means± standard deviations unless stated otherwise

cSLE = childhood-onset systemic lupus erythematosus

^‡Systemic Lupus Disease Activity Index 2k version; range 0 – 104; 0 = inactive SLE

^†PCF-43 questionnaire: Perceived Cognition Functioning −43 questionnaire

^¥Neurocognitive dysfunction categories are defined based on z-scores of the standardized tests completed for the formal neurocognitive testing (FNCT) on the research cohort, and on T-scores of the pediatric perceived cognitive function questionnaire-43 (PedsPCF-43) on the clinical cohort.

Development Dataset Analyses of PedANAM-CPS candidates

Associations between the four candidate PedANAM-CPS indices and details about the algorithms used to calculate them are presented in Table 2. The PedANAM-CPS_PCA and the PedANAM-CPS_UWA were strongly correlated with each other (r=0.99; p<0.001), while the PedANAM-CPS_logit was moderately correlated with the PedANAM-CPS_PCA, and the PedANAM-CPS_UWA (r=−0.60 and −0.57 respectively; p<0.001). The PedANAM-CPS_multiscore was moderately correlated with the other three PedANAM-CPS indices (p<0.001).

Table 2.

Pearson correlation coefficients of the candidate PedANAM-CPS in the development dataset of 80 participants

	PedANAM-CPSUWA	PedANAM-CPSPCA	PedANAM-CPSlogit
PedANAM-CPS UWA	1.000
PedANAM-CPSPCA	0.990**	1.000
PedANAM-CPSlogit	−0.579**	−0.601**	1.000
PedANAM-CPSmultiscore	−0.670**	−0.675**	0.650**

^**p-values<0.001

PedANAM-CPS_UWA is calculated by taking the average of 9 battery accuracy scores; if PedANAM-CPS_UWA < 0.25, further evaluation is recommended.

PedANAM-CPS_PCA is calculated by taking weights of 0.29, 0.13, 0.45, 0.42, 0.26, 0.35, 0.33, 0.27 and 0.39 on battery accuracy scores of CDD, CPT, CS, LRS, M2S, MG, MATH, ST6 AND SPD respectively; if PedANAM-CPS_PCA < 0.25, further evaluation is recommended.

PedANAM-CPS_logit is calculated using the following formula: PedANAM-CPS_logit = −1.99 – 0.24*CDD - 0.21*CPT - 0.07*CS - 0.35*LRS + 0.19*M2S + 0.85*MG + 0.02*Math - 0.37*ST6 - 1.00*SPD; if PedANAM-CPS_logit > −2.7, further evaluation is recommended.

PedANAM-CPS_multiscore is calculated using the following formula: PedANAM-CPS_multiscore = 15.42 – 0.19*AC.SPD + 9.75*CVc.CPT + 7.29*CVc.M2S – 0.01*MNc.CDD; if PedANAM-CPS_multiscore > 0.09, further evaluation is recommended.

Where CDD = Code Substitution Delayed accuracy score; CPT = Running Memory accuracy score; CS = Code Substitution accuracy score; LRS = Logical Relations Symbolic accuracy score; M2S = Matching to Sample accuracy score; MG = Matching Grids accuracy score; Math = Math Processing accuracy score; ST6 = Memory Search accuracy score (6); SPD = Spatial Processing (simultaneous) accuracy score; AC.SPD = Percent of Accuracy in SPD, CVc; CPT=Coefficient of Variation in CPT; CVc.M2S=Coefficient of Variance in M2S; MNc.CDD=Mean Reaction Time in CDD.

PedANAM-CPS indices differ between patients with and without NCD across all datasets

NCD status was based on FNCT (development-dataset and validation-dataset 1) or the PedsPCF-43 (validation-dataset 2). The ability of the PedANAM-CPS_UWA, PedANAM-CPS_PCA, PedANAM-CPS_logit and PedANAM-CPS_multiscore to discriminate participants as per their NCD-status (FNCT) after adjusting for age differences between groups was assessed. In contrast to individual PedANAM subtests, the PedANAM-CPS indices differ in groups of patients with and without NCD (Table 3). All candidate PedANAM-CPS indices discriminated between participants with different NCD-status (p’s< 0.036).

Table 3.

Performance of the candidate PedANAM-CPS for identifying neurocognitive dysfunction ^{^}

	Non-NCD (n=68)*	NCD (n=12)*	p-value†
PedANAM-CPSUWA	0.08 ± 0.07	−0.39 ± 0.18	0.036
PedANAM-CPSPCA	0.09 ± 0.08	−0.42 ± 0.19	0.027
PedANAM-CPSlogit	−2.26 ± 0.16	−0.60 ± 0.37	0.001
PedANAM-CPSmultiscore	4.01±0.31	7.97±0.74	<0.001

PedANAM-CPS: Pediatric Automated Neuropsychological Assessment Metrics-Cognitive Performance Score; UWA: unweighted average method; PCA: principal component analysis method; Logit: logistic regression method of all the PedANAM subtests accuracy scores; Multiscore: logistic regression method of selected PedANAM subtests accuracy, coefficient of variation and mean reaction time to correct responses scores.

^{^}NCD: Neurocognitive dysfunction as determined by Formal Neurocognitive Testing

^*Values are mean ± Standard error

^†From fixed effect models adjusted for age

Discrimination by NCD-status

The overall precision of PedANAM-CPS_UWA, PedANAM-CPS_PCA, PedANAM-CPS_logit, and PedANAM-CPS_multiscore at identifying NCD in the development group using ROC curve analysis is summarized in Table 4. The PedANAM-CPS_multiscore and the PedANAM-CPS_logit indices showed good to excellent ability to identify NCD in the development group (both AUC>0.77). For each candidate PedANAM-CPS index, specificities were determined for sensitivity values of 80% or higher as well as the statistically preferred threshold values that yielded the overall best combination of sensitivity and specificity.

Table 4.

Accuracy and point sensitivity and specificity of the candidates PedANAM-CPS in the Development-group

Candidate PedANAM-CPS	AUC ¥	Cut-off*	Sensitivity&	Specificity
PedANAM-CPSUWA	0.60	0.25	83.3 %	37.3 %
PedANAM-CPSPCA	0.60	0.25	83.3 %	41.8 %
PedANAM-CPSlogit	0.77	−2.70	91.7%	31.3 %
PedANAM-CPSmultiscore	0.85	0.09	91.7%	66.2%

^&Only sensitivities of > 80% were considered

^¥Values are measures of overall PedANAM-CPS performance to identify neurocognitive dysfunction

(NCD) as determined by formal neurocognitive testing. AUC values range from 0 – 1; values of 1.0–0.91: outstanding, 0.81–0.90: excellent, 0.71–0.8: good, 0.61–0.7: fair, and <0.6: poor accuracy to detect NCD

Overall accuracy and sensitivity for identifying NCD (as defined by FNCT) was near 90% in the validation-group-1 (Figure 1). Analyses considering the validation-dataset-2 (Table 5) yielded similar sensitivities to detect NCD (as determined by PedsPCF-43 scores) for all of the PedANAM-CPS indices, except for the PedANAM-CPS_multiscore. The latter, however, showed overall good to excellent ability to detect NCD when applied to validation groups 1 and 2 (AUC= 0.89 and 0.74, respectively).

Figure 1.

Area under (AUC) the receiver operative characteristic curve (ROC) calculated on validation group1

Sens = sensitivity; Spec = specificity

Table 5.

Accuracy and point sensitivity and specificity of the candidates PedANAM-CPS in the validation Groups summary

Candidate PedANAM-CPS	Cut-off*	Validation Group-1 (n=61)			Validation Group-2 (n=86)
		AUC¥	Sensitivity	Specificity	AUC¥	Sensitivity	Specificity
PedANAM-CPSUWA	0.25	0.78	80 %	47.9 %	0.68	80 %	39.3 %
PedANAM-CPSPCA	0.25	0.80	90 %	42.0 %	0.67	90 %	39.3 %
PedANAM-CPSlogit	−2.70	0.80	90 %	32.0 %	0.62	80 %	31.5 %
PedANAM-CPSmultiscore	0.09	0.89	89 %	55.0 %	0.74	60 %	75.3 %

^¥Interpretation of AUC values: 1.0–0.91: outstanding, 0.81–0.90: excellent, 0.71–0.8: good, 0.61–0.7: fair, and <0.6: poor

^*PedANAM-CPS_UWA or PedANAM-CPS_PCA <0.25 indicate higher likelihood of NCD. PedANAM-CPS_logit > −2.70 indicate higher

DISCUSSION

Based on recent research, the PedANAM software may offer a cost effective approach to screening for cognitive dysfunction associated with NPSLE (21). The development of a summary statistic is complementary to previous research that found the PedANAM to have construct validity and responsiveness to change in cognitive performance in cSLE (16, 21). We developed and initially validated four candidate PedANAM-CPS indices to further increase the clinical usefulness of the PedANAM using commonly accepted statistical approaches. We found that the proposed indices were able to differentiate individuals’ cognitive status. Compared to the PedANAM-CPS_logit, the PedANAM-CPS_PCA and the PedANAM-CPS_UWA showed similar or even higher accuracy (AUC) and higher specificity at the chosen sensitivity threshold of 80%. The PedANAM-CPS_multiscore featured the highest specificities among the four PedANAM-CPS indices at the chosen sensitivity threshold of 80%. According with our findings we recommend further neurocognitive assessment in subjects with a PedANAM-CPS_PCA score <0.25 or a PedANAM-CPS_multiscore >0.09.

Upon completion of the PedANAM a large number of performance variables are provided, making it difficult for clinicians to easily determine whether a patient’s cognitive function has changed or not. We, therefore, propose that the PedANAM-CPS may be used in several ways. Firstly, as the PedANAM is sensitive to cognitive dysfunction when present but not necessarily specific to a particular disease process, the PedANAM-CPS may be used to track cognitive ability over time in children with and without cSLE (31). Secondly, comparing a patient’s score to the threshold values proposed may support the decision of whether or not to pursue FNCT. This is supported by the initial threshold values for each of the PedANAM-CPS indices that are highly sensitive to the presence of NCD.

When developing the PedANAM-CPS we focused on the PedANAM accuracy performance parameter (of each subtest) as this score has demonstrated the highest consistency and reliability when compared to the other PedANAM derived scores (16). This strategy was taken given the known diversity of cognitive deficits observed in cSLE. Conversely, the PedANAM-CPS_multiscore suggests that a shorter PedANAM battery of subtests may be used for NCD-screening. However, this finding needs to be replicated in other prospective studies with larger samples before consideration of a reduced battery.

We identified threshold values for each of the four PedANAM-CPS indices that were able to recognize NCD in the participants with acceptable sensitivities and specificities. Nonetheless, these threshold levels must be considered preliminary and require confirmation in larger cohorts. However, as the performance of a screening device is related to the prevalence of the disease that it is intended to identify, a different population can produce different findings. Based upon the estimated prevalence of cSLE in the US of 9.73 per 100.000 children (32), a sample of more than 100 cSLE patients is actually quite large.

A limitation of our study may be that the distribution of PedsPCF-43 T-score in our study differed from the distribution in pediatric cancer survivors. In the current study, participants’ mean PedsPCF-43 T-score was near 60, and therefore participants with a T-score <50 on PedsPCF-43 were classified as having NCD. Although this is in line with reports from children with attention deficit disorder, epilepsy, and cerebral palsy (24), it differed from what has been suggested in the past for pediatric cancer survivors, in whom the tool was originally developed and where a score of 40 is considered an indication for referral to FNCT. Thus, the differences in performance of the PedsPCF-43 for cSLE and as compared to oncology populations will need further investigation.

Another limitation of this study perhaps will be the significant racial difference observed within cSLE patients and controls in the validation group-2. Female predominance in the cSLE was expected as JIA controls were not recruited based on sex or ethnicity. However, based on previous research it is unlikely that the results are affected by racial differences (33, 34). Rather, social advantage factors such as maternal education (34) and poverty have a stronger association with lower levels of school achievement and IQ later in childhood. Overall in this study the maternal education level was similar among groups.

To ensure the high sensitivity required for neurocognitive screening (11), the threshold levels presented provide greater than 80% sensitivity in detecting NCD. Arguably, a different sensitivity threshold could have been chosen, as we recognize that suggested PedANAM-CPS thresholds have low specificity for identifying NCD. However, we feel it better to have a screening tool that favors false positives as this may only lead to undue referrals for FNCT of children who have normal cognitive function, and will minimize failures to identify children who truly have cognitive dysfunction. Moreover, given the lack of a systematic screening options (i.e. physician perception and parent global impressions are insufficient to detect early cSLE associated cognitive changes) (16, 22) we consider the PedANAM-CPS indices to be an important tool for effective systematic screening for NCD in pediatric rheumatology clinics.

Further study is required to determine whether the PedANAM-CPS_PCA and PedANAM-CPS_multiscore indices would be complementary in the detection of NPSLE. In particular, research on the sensitivity of the PedANAM-CPS to clinically meaningful change over time is needed. Additionally, information about the minimal clinically important differences in PedANAM-CPS is needed.

Early detection and screening of cognitive decline and NCD in cSLE is a critical first step to improving prognosis and functioning of affected cSLE patients. The PedANAM-CPS provides a summary measure of cognitive performance that may be simple enough to implement as a clinically relevant NPSLE screening tool. Based on findings from this initial validation study, we recommend the PedANAM-CPS_PCA and PedANAM-CPS_multiscore as summary statistics to screen the cognitive performance of children with cSLE. However, larger studies in diverse patient populations are needed to examine the measurement properties of the PedANAM-CPS in more detail.

SIGNIFICANCE & INNOVATION.

Innovation

• This study proposes a Pediatric Automated Neuropsychological Assessments Metrics cognitive performance score (PedANAM-CPS) as an overall measure of cognitive ability.

• Initial threshold values can assist in the identification of patients with a high likelihood of clinically relevant impairment of cognitive function

Significance

• The PedANAM-CPS enhances the suitability of the PedANAM as a screening tool for cognitive dysfunction in clinical settings.