Pre-appointment online assessment of patient complexity: Towards a personalized model of neuropsychological assessment

Abstract

Recent events such as the global pandemic of COVID-19 have challenged neuropsychologists to scale up their capacity to conduct portions of their assessment remotely. While more complex patients will likely continue to require on-site, office-based interaction and assessment, the current emergency-based expansion of online and telehealth evaluation practices may ultimately lay the groundwork for more routine, online assessment of patients with less complex presentations in the future. To this end, the current study evaluated a pre-appointment, online methodology for differentiating referred pediatric patients based upon the scope and severity of their caregiver-reported adaptive, academic, attentional, behavioral, and emotional impairment. Prior to on-site assessment, parents/caregivers of 2197 children (Mean age = 10.0y, range = 4–19y, 62% male) completed an online developmental history form screening for symptoms of adaptive, attentional, learning, affective, and behavioral impairment; 71% of those children eventually underwent assessment. Using latent class analysis, the data supported a reproducible 4-class model consisting of groups of children at increased risk for: 1) severe multi-domain dysfunction; the “High Complexity” group, 30%, 2) behavioral-affective (but not academic) dysregulation; the “Behavioral Focus” group, 13%, 3) academic (but not behavioral-affective) problems; the “Academic and Inattention” group, 37%, and 4) patients with minimal clinical complexity; the “Low Complexity” group, 20%. Comparison of pre-visit classification with day-of-assessment standardized test scores supported the validity of patient subtypes. Moving forward, pre-appointment clarification of patient complexity may support efficient patient triage with regard to assessment modality (e.g., on-site or online) and length of appointment (e.g., comprehensive or targeted).

The correct “dosing” of neuropsychological assessment has become an area of interest, for the purposes of both cost management and personalization of medicine (Bauer, 2016; Gur, 2018). It is increasingly understood that “one size” does not fit all with regard to psychological and neuropsychological assessment, and smaller “doses” of assessment may be sufficient for some patients in lieu of comprehensive assessment. While day-long comprehensive assessments (i.e., ≥5 hours of testing) are common in pediatric neuropsychology (e.g., Baum et al., 2018) and may be appropriate for certain groups of children with known neurodevelopmental disorders (Mahone et al., 2017), more circumscribed models of surveillance, screening, and targeted assessments have been proposed as additional time-efficient assessment options (e.g., Baum et al., 2017). These types of screening models have become increasingly feasible with the availability of web-based platforms, electronic medical record portals, and/or online assessment tools (e.g., Brandt et al., 2014) that may support decision-making processes concerning patient triage and scope of assessment. Moreover, these types of screening and targeted assessments have taken on increased relevance in light of the global health crisis of COVID-19, as much of clinical neuropsychological practice had shifted to online and telemedicine formats at the time this manuscript was written (Winter/Spring 2020).

Planning the scope of assessment (e.g., brief screening, targeted assessment, day-long comprehensive assessment) may be supported by pre-visit identification of patient groups. To date, most group classification efforts have occurred after assessment. For instance, using data from neuropsychological assessments, researchers have employed latent class analysis to identify distinguishable clinical subtypes of individuals or groups within a diagnostic category (e.g., Libon et al., 2014; Van Hulst et al., 2015) and across a mixture of diagnostic presentations (Morin & Axelrod, 2017). Papazoglou and colleagues (Papazoglou et al., 2013) used a similar methodology (cluster analysis) to identify groups of pediatric patients distinguished by scope of deficits noted on performance-based neuropsychological measures. Papazoglou et al. reported patient clusters that ranged from broadly intact presentations, more circumscribed symptom pictures (i.e., impaired intelligence + mild executive dysfunction), and more pervasive symptom presentations (i.e., borderline-impaired intelligence + high levels of executive dysfunction + externalizing and internalizing emotional/behavioral problems).

Establishing stable and meaningful patient classes based upon the scope of patient symptoms (e.g., circumscribed or pervasive) prior to clinical appointments has many advantages, including the potential of scheduling pediatric patients to the most appropriate level of assessment time. For example, patients identified as having a more circumscribed scope of symptoms based upon pre-visit screening could be scheduled for less time-intensive-targeted assessment visits, while patients identified with a more pervasive scope of symptoms could be scheduled for comprehensive, day-long assessment. In light of recent global events, pre-visit screening of patient complexity could also identify patients with more circumscribed symptoms/concerns that could potentially be evaluated via a telehealth visit. No study, to our knowledge, has sought to identify clinical subgroups of children using parent-reported pre-appointment data to help facilitate triage-related decision making and patient scheduling. The goal of the present study was to address this gap.

The first objective of this study was to determine whether pre-visit caregiver ratings of child academic, attentional, behavioral, emotional, and adaptive functioning could be used to identify meaningfully distinct subgroups of referred pediatric patients which could potentially be used to help guide patient triage, evaluation planning, and testing resource allocation. Once identified, the second objective of this study was to assess the validity of these groups using actual day-of-assessment performance on neuropsychological tests as the criterion. Based upon earlier research involving comprehensive clinical assessment of a mixed clinical sample (Papazoglou et al., 2013), we expected to find groups distinguished by scope of symptom report. We anticipated that patient classes would include children with a circumscribed, less complex presentation (e.g., impairment in one domain) and children with more pervasive and complex profiles (i.e., patterns of impairment across multiple domains). We also anticipated the emergence of subgroups with impairments in specific domain areas that could be used to route them toward specialized assessment services within the testing array (e.g., a learning disabilities clinic) or toward a treatment program outside of the testing service (e.g., behavioral intervention clinic). Pre-visit identification of scope of symptom presentation could facilitate scheduling of patients into shorter or longer assessment appointments, evaluation in on-site or online assessment settings, and support more efficient use of assessment resources.

Methods

Participants

Children in this study (Table 1) were referred for psychological/neuropsychological assessment at an urban outpatient testing service of a regional hospital in the Mid-Atlantic region of the US between 2016 and 2017. Consistent with the scheduling practices of this clinic, all children in this convenience sample were scheduled for day-long assessments. Youth included in this study (N = 2197) ranged in age from 4 to 19 years (Mean = 10.0y, SD = 3.5y); 62% were male and 36% of families were receiving Medical Assistance/public insurance. The age range of the study was selected to reflect the full range of formal schooling (e.g., preschool through high school) for most children being referred for clinical assessment. Slightly more than half of the youth were Caucasian (54%), with the remaining mostly Black/African-American (30%) or Other races (15%). Educational status of the caregiver completing the online pre-visit caregiver ratings was fairly evenly distributed, with 26% having a high school, associates, or trade school education; 16% completing some college; 26% receiving their bachelors, and 32% having a graduate-level education. The most common referral diagnoses for the sample were: Attention-deficit/Hyperactivity Disorder (ADHD; 44%), anxiety disorders (8%), adjustment disorders (5%), oncologic diseases (3%), epilepsy (3%), unspecified encephalopathy (3%), and conduct disorders (3%). The remaining 31% were comprised of low frequent diagnoses. The only individuals that were excluded from this study were: 1) children <4 years (1%) and adults >19 years (2%), since several of the rating scales used (e.g., Revised Child Anxiety and Depression Scale [RCADS: Chorpita et al., 2000]) were not appropriate for these age ranges.

Table 1.

Demographic Characteristics of Classes.

	High Complexity	Behavioral Focus	Academic and Inattention	Low Complexity	Overall
Child Age (M, SD)	10.07 (3.44)	9.66 (3.70)	10.07 (3.30)	10.92 (3.85)	10.0 (3.50)
Child Race (%)
Caucasiana	58	67	51	46	54
Black/AA	27	21*	32*	36*	30
Other	15	12	16	17	15
Insurance (%)
Private Insurancea	54	72	66	61	62
Medical Assistance	44	27*	33*	37*	36
Other	2	2	1	2	2
Parental Education (%)
High School/AA/Tradea	31	20	23	27	26
Some College	18	15	18	13	16
Bachelors	25	24	27*	25	26
Graduatea	27	41*	32*	34	32
Race (%)
Female	40	29	39	37	38
Malea	60	71*	61	63	62

Note. AA = African American;

^*= p <.05 difference compared to the “High Complexity” Group;

^a= within variable reference category

Procedure

Online pre-visit developmental history form

Parents of referred children scheduled for comprehensive psychological or neuropsychological assessment were sent a letter providing information about their upcoming appointment. Included in that letter was a web address to an online pre-visit developmental history form that included a series of embedded parent-reported behavioral rating scales (described below), all of which were available in the public domain. The behavioral rating scales were selected for inclusion in the pre-visit questionnaire in order to gather a broad range of clinical information (e.g., attention, academic, adaptive, emotional, etc.). If a parent did not complete the form prior to the appointment (e.g., those without Internet access), parents were able to complete the form on the morning of their child’s evaluation via an on-site computer or a paper-and-pencil version of the form. Historically, parents of 65% of patients referred to this urban outpatient testing service completed the developmental history form prior to or on the day of assessment as part of clinical care. However, recent practice adjustments now require that this pre-visit questionnaire be completed prior to appointment scheduling.

Parents/caregivers of all youth included in this study (N = 2197) completed the developmental history form. Seventy-one percent (n = 1569) of the referred patients eventually underwent psychological or neuropsychological assessment within a year of their parent completing the form. Reasons for not undergoing assessment included appointment no-show/cancellation, referral to another clinic due to insurance authorization issues, or completion of a diagnostic intake appointment in lieu of full assessment.

For those children (n = 1569) who eventually underwent psychological or neuropsychological assessment, 71% of parents completed the developmental history form prior to the day of the assessment appointment (Median days between form completion and assessment was 34). Twenty-eight percent of parents completed the history form and ratings on the day of their child’s assessment, while only a few (<1%) completed it after the appointment.

All pre-visit data were maintained in a HIPAA-compliant database. Although the parent-reported rating scales included in the developmental history form were originally normed in “paper and pencil” formats, there is evidence to suggest that the psychometric properties of caregiver ratings of children’s behavioral, academic, and emotional functioning remain consistent regardless of administration method (Pritchard et al., 2017). Standardized test scores (described below) acquired via clinical assessment were maintained in the secure electronic health record. A waiver of consent to study these de-identified clinical data was granted by the local institutional review board.

Internalizing problems

The Generalized Anxiety and Major Depression subscales from the Revised Children’s Anxiety and Depression Scale-Parent Version (RCADS: Chorpita et al., 2000) were employed as a pre-visit measure of internalizing problems. Content from these two subscales was used for screening purposes; the additional anxiety and depression subscales on the RCADS were excluded (e.g., Social Phobia, Panic Disorder, Separation Anxiety). The RCADS-Anxiety (6 items) and RCADS-Depression (10 items) subscales are based on a 4-point Likert-type response. The sum of each subscale was converted to a T-score using grade- and gender-referenced normative data from the RCADS manual. T-scores of 65 and above were considered, per the manual, to represent clinically significant symptom cutoffs (Chorpita et al., 2015). Since norms were not available for youth less than 8 and older than 17 years of age, we employed the cutoff scores required for those of closest age (e.g., grades 11 and 12 for youth over 17 years of age). The RCADS is a well-established measure that has demonstrated internal consistency and convergent validity in school-based, clinical, and international samples (Chorpita et al., 2015). A total of 611 (29%) and 578 (27%) children from the sample met the RCADS-Depression and RCADS-Anxiety cutoffs, respectively.

Externalizing problems

A subset of eight items measuring oppositional defiance and conduct disorder (VAN-Conduct) from the Vanderbilt ADHD Diagnostic Parent Rating Scale (Wolraich, 2003) were used as a pre-visit measure of pediatric behavioral problems. Given the abbreviation of the scale, we were required to identify a cutoff with optimal classification properties. To accomplish this adaptation, a receiver operating characteristic curve analysis was employed based on the present sample. Using a T-score cutoff of ≥60 on the Behavior Assessment System for Children-Second Edition (BASC-2; Reynolds & Kamphaus, 2004) Conduct Problems subscale, a VAN-Conduct sum score of 16 was considered optimal (68% sensitivity, 84% specificity, 80% correctly classified, 50% positive predictive value, 87% negative predictive value). A total of 782 (37%) participants in the sample met this cutoff.

Academic problems

The Colorado Learning Difficulties Questionnaire (CLDQ; Willcutt et al., 2011) is a 5-point Likert-type parental report scale that was used as a pre-visit measure of academic difficulties. Two of the CLDQ subscales, math (CLDQ-Math; 5 items) and reading (CLDQ-Reading; 6 items), were employed in the analyses. The CLDQ-Math subscales query if the child “has” or “ever had” difficulties with calculation, mathematical concepts, and word problems, whereas the CLDQ-Reading subscale asks parents to report similarly on their child’s difficulty (or history of difficulty) with spelling, reading, and acquisition of phonics. While the original CLDQ only had three math items, the CLDQ authors described improved psychometrics when two additional items were included (Willcutt et al., 2011). As such, the current study utilized five, rather than three, CLDQ math items. Patrick et al. (2013) established clinical cutoffs for clinically referred children using this 5 (math) and 6 (reading) item model, which can be used to identify children at risk for reading and math difficulties. Accounting for the high base rate of reading and math learning disabilities within clinically referred samples, the conditional probability of identifying true negatives (96.2% reading, 85.1% math) is particularly useful for clinical screening purposes (Patrick et al., 2013). Ratings from 1138 (55%) and 1120 (55%) children from the sample exceeded the CLDQ-Reading or CLDQ-Math cutoffs, respectively.

ADHD symptoms

The Attention Deficit Hyperactivity Disorder (ADHD) Rating Scale- 5 (DuPaul et al., 2016) is a norm-referenced checklist that was used as a pre-visit measure of ADHD symptoms based upon criteria from the Diagnostic and Statistical Manual of Mental Disorders – Fifth Edition (DSM-5; American Psychiatric Association, 2013). The parent-report version of the ADHD Rating Scale-5 includes a total of 18 items addressing the hyperactivity (ADHD-HY) and inattention (ADHD-IN) symptom criteria. The ADHD Rating Scale-5 is a norm-referenced tool with demonstrated reliability (test-retest, internal consistency, inter-rater agreement) and criterion, discriminant, and predictive validity (G. DuPaul et al., 2016). The ADHD-HY and ADHD-IN subscales were dichotomized based on the DSM diagnostic criteria cutoff of the presence (i.e., ratings of “often” or “very often”) of 6 or more symptoms for each subscale. A total of 1136 (54%) and 525 (25%) children from the sample met the ADHD-IN or ADHD-HY cutoffs, respectively, based upon parent symptom report.

Impairment

The Impairment Rating Scale (IRS; Fabiano et al., 2006) was used as a pre-visit measure of impairment of patients across several domains of functioning. The IRS is based on a 7-point response format ranging from “No problem; definitely does not need treatment or special services” to “Extreme problem; definitely needs treatment or special services.” Five items from the IRS were used, covering social (IRS-Others), relationship with caregiver (IRS-Caregiver), academic progress (IRS-Academic), home life (IRS-Home), and self-esteem (IRS-Esteem) domains of functioning. Each of the five items of the IRS administered was employed in the analysis individually. The cutoff for each item was placed at a score of four (“mild problem“) on the 7-point scale. The IRS has demonstrated clinical utility and psychometric robustness (Fabiano et al., 2006). The proportion of patients who met the cutoff for impairment based upon parent report varied: IRS-Caregiver (N = 715; 33%), IRS-Others (N = 977; 45%), IRS-Home (N = 1,115; 51%), IRS-Esteem (N = 1,231; 57%), and IRS-Academic (N = 1,708; 78%).

Day-of-assessment standardized measures

As noted, 71% (n = 1569) of the referred sample with completed developmental history forms/ratings eventually underwent psychological or neuropsychological assessment. Standardized performance-based measures were administered during psychological and neuropsychological assessments at the discretion of the licensed psychologist within the context of clinical care. As such, not all assessment tests were given to all children. Assessment results were only included in the study if they were obtained within 1 year of, either prior to or following, the date that the developmental history form was filled out by the parent. Figure 1 displays data collection for the whole cohort, including the proportion of youth who received which of the performance-based assessments.

Figure 1.

Data collection flow-chart. Note. *Data were >95% complete on all demographic information, except parental education (78% complete); ** Denominator for percentages were based on those who received an evaluation (n = 1569)

Academic achievement

Educational assessment was conducted on the day of assessment using the Kaufman Test of Educational Achievement-Third Edition (KTEA-3; Kaufman & Kaufman, 2014; n=695) or the Wechsler Individual Achievement, Third Edition (WIAT-3: (Wechsler, 2009); n=176). For both measures, standard scores for word reading and math computation subtests were used in the analyses. Data from either the KTEA-3 or WIAT-3 were available for 55% of the sample who received an assessment.

Intelligence

General intelligence was measured on the day of assessment using a variety of different scales. The Wechsler Intelligence Scales for Children- Fifth Edition (WISC-5; Wechsler, 2015; n=962) was the most commonly completed measure in this domain, followed by the Differential Ability Scales-Second Edition (DAS-2; Elliott, 2007 Early Years; n = 119), and the Wechsler Adult Intelligence Scales-Fourth Edition (WAIS-4; Wechsler, 2008; n=114). A variety of other measures were also used on an infrequent basis, including the Wechsler Abbreviated Scales of Intelligence-Second Edition (WASI-2; Wechsler, 2011; n=57), and the Wechsler Preschool and Primary Scale of Intelligence – Fourth Edition (WPPSI-4; Wechsler, 2012; n=29). To harmonize the data, we utilized standardized scores from composite scales of verbally-based intelligence across all measures (e.g., WISC-5 Verbal Comprehension Index, DAS-2 Verbal Cluster; WAIS-4 Verbal Comprehension Index). Data on verbally-based IQ were available on 82% of the sample who received an assessment.

Adaptive functioning

The Adaptive Behavior Assessment System, Second Edition (ABAS-2: P. L. &. Harrison & Oakland, 2003; n=28) and Third Edition (ABAS-3: P. L. Harrison & Oakland, 2015; n=864) were used as a day of assessment parent-report measure of general adaptive behavior. The General Adaptive Composite, a standard score, was employed in the analyses. Either an ABAS-2 or ABAS-3 was completed for 57% of the sample who received an assessment.

Mental health

The Behavior Assessment System for Children, Second Edition (BASC-2; Reynolds & Kamphaus, 2004; n= 20) and Third Edition (BASC-3; Reynolds & Kamphaus, 2015; n=576) were used on the day of assessment to measure parent-reported mental health and behavioral challenges. Standardized scores from anxiety, depression, and conduct disorder subscales were employed. Data on the BASC-2 or BASC-3 were available for 38% of the sample who received an assessment.

Demographics

Most of the patient demographic information (age, race, ethnicity, and insurance type) was retrieved from the electronic health record. As such, there was very little missing data (<5%). Parental education was, however, obtained via the parent rating forms and was only available on 78% of the sample.

Data analytic strategy

Latent Class Analysis (LCA) was employed to examine different subtypes or groups of children based on parent-reported symptoms and impairment (Hagenaars & McCutcheon, 2002). LCA is a person-centered, statistical approach that assigns each individual a posterior probability of class membership based on their observed responses across a set of dichotomous indicators. The present study employed a total of 12 indicators, of which a few had missing data, that included parent-reported anxiety (RCADS-Anxiety; 3% missing), depression (RCADS-Depression; 4% missing), externalizing behavior problems (VAN-Conduct; 3% missing), math problems (CLDQ-Math; 7% missing), reading problems (CLDQ-Reading; 5% missing), ADHD Hyperactivity and Inattention (ADHD-HY and ADHD-IN; both 3% missing), and impairment ratings (IRS-Caregiver, IRS-Others, IRS-Home, IRS-Esteem, and IRS-Academic; all <1% missing). All missing data were handled via full information maximum likelihood estimation (Enders & Bandalos, 2001), which resulted in retaining all observations in the LCA.

To settle on the appropriate number of classes, ranging from 2 to 6, a series of fit statistics were employed. These included the Akaike (AIC) and Sample Size adjusted Bayesian Information Criterion (SS-BIC). Lower AIC and SS-BIC values can be interpreted as a better-fitting model (Jung & Wickrama, 2008). Bootstrapped Likelihood Ratio (BLRT) and the Vuong-Lo-Mendell-Rubin (VLMR) test were also included. These tests examined the current model compared to the K-1 model. Models are iteratively tested until p > .05, suggesting that the current model does not fit significantly better than the K-1 model (Jung & Wickrama, 2008). We also report the Pearson χ², where a significant p-value suggests that the observed responses are different from the expected pattern of results. Note that this test is highly sensitive to sample size, so when N is large α is almost always <.05 (Jung & Wickrama, 2008).

Beyond fit statistics, we applied additional metrics to arrive at class structure. These included clinical utility, attention toward small class sizes, and replicability. To address replicability, we employed cross-validation (Arlot & Celisse, 2010). The cross-validation involved randomly splitting the original sample into two independent samples. The model was developed in the initial (test) sample, and then evaluated in the second independent (validation) sample to confirm the reliability of the class structure.

After arriving at the optimal class structure, model validity was assessed by examining differences between classes across standardized parent-reported measures and performance–based metrics. Demographic differences were also examined. Both the demographic and standardized measures were examined independently using linear and multinomial logistic regressions. LCA analyses were performed in MPLUS Version 7.0 (Muthén & Muthén, 2007), and all other data management and analysis occurred in STATA 15.0 (College Station, Tx).

Results

Latent class analysis

Fit statistics

Looking at the full sample (Table 2) and the cross-validation test sample (Table 3), the χ², AIC, and SS-BIC values decreased continuously with increasing class size. However, the drop off in fit statistics values slowed after the 4-class model. The BLRT remained significant (at p < .01) through the classes; however, the VLMR suggested a 5-class model. The training sample mirrored the test sample except the VLMR suggested a 4-class model fit the data best.

Table 2.

Model Fit Statistics (N=2197).

Class Number	Pearson X2	AIC	SS-BIC	VLMR	BLRT	Smallest Class Size (%)
2	8344	29904	29967	p <.001	p <.001	48
3	7359	29398	29493	p <.001	p <.001	21
4	6390	29118	29247	p <.01	p <.001	13
5	5683	28952	29113	p <.001	p <.001	12
6	5679	28815	29009	p <.01	p <.001	11

Note. AIC=Akaike Information Criterion; SS-BIC=Sample Size adjusted Bayesian Information Criterion; VLMR=Vuong-Lo-Mendell-Rubin; BLRT=Bootstrapped Likelihood Ratio

Table 3.

Fit statistics for cross-validation training (n = 1099) and test (n = 1098) samples.

Sample	Class Number	Pearson X2	AIC	SS-BIC	VLMR	BLRT	Smallest Class Size (%)
Train	2	5820	14897	14942	p <.001	p <.001	49
Train	3	4977	14640	14830	p <.001	p <.001	19
Train	4	4564	14525	14618	p =.02	p <.001	12
Train	5	4006	14470	14791	p =.15	p <.001	10
Train	6	3951	14411	14552	p =.09	p <.001	11
Test	2	6669	15037	15083	p <.001	p <.001	49
Test	3	5556	14776	14845	p <.001	p <.001	25
Test	4	4874	14622	14715	p =.02	p <.001	12
Test	5	4334	14515	14631	p =.01	p <.001	11
Test	6	4114	14454	14594	p =.72	p <.001	10

Note. AIC=Akaike Information Criterion; SS-BIC=Sample Size adjusted Bayesian Information Criterion; VLMR=Vuong-Lo-Mendell-Rubin; BLRT=Bootstrapped Likelihood Ratio

Although the fit statistics were not perfectly clear, the 2- and 3-class models could be easily ruled out on the elevated χ², AIC, SS-BIC values. The 6-class model also found little support from the fit statistics as the VLMR was p > .05 in both the cross-validation samples. Upon visual inspection of the 3-, and 4-, and 5-class models, there was evidence of class splitting in the 5-class model. This observation occurred when the complex group, identified in the 4-class model, was simply split in two, resulting in two groups that were identical in response pattern except one group that had slightly higher probabilities of indicator values than the other. As such, this model was ruled out based on lack of utility for the fifth class.

The final decision was to choose between the 3- vs. 4-class models. The 4-class model (Figure 2) was chosen for three reasons. First, the 4-class model had substantially lower χ², AIC, and SS-BIC values. Second, the VLMR test supported the 4-class model in the training sample. Third, and most importantly, the 4-class model identified a clinically valuable population (the Behavioral Focus group) that replicated across samples (see Figures 3 and 4). Given the replication of findings in the training, compared to the test, sample, the data were combined for the remaining analyses.

Figure 2.

4-class model of caregiver-reported child clinical indicators (N = 2197). Note. IRS = Impairment Rating Scale; CLDQ = Colorado Learning Difficulties Questionnaire; ADHD = Attention Deficit Hyperactivity Disorder; IN = Inattention; HY = Hyperactivity; RCADS = Revised Child Anxiety and Depression Scale; VAN = Vanderbilt ADHD Diagnostic Parent Rating Scale

Figure 3.

Training sample of caregiver-reported child clinical indicators (n = 1099). Note. IRS = Impairment Rating Scale; CLDQ = Colorado Learning Difficulties Questionnaire; ADHD = Attention Deficit Hyperactivity Disorder; IN = Inattention; HY = Hyperactivity; RCADS = Revised Child Anxiety and Depression Scale; VAN = Vanderbilt ADHD Diagnostic Parent Rating Scale

Figure 4.

Test sample of caregiver-reported child clinical indicators (n = 1099). Note. IRS = Impairment Rating Scale; CLDQ = Colorado Learning Difficulties Questionnaire; ADHD = Attention Deficit Hyperactivity Disorder; IN = Inattention; HY = Hyperactivity; RCADS = Revised Child Anxiety and Depression Scale; VAN = Vanderbilt ADHD Diagnostic Parent Rating Scale

4-class model

Figure 2 displays the 4-group latent class model using pre-visit parent ratings. The largest group (the double line) is termed the “Academic and Inattention” class (37%). This class was demarcated by extremely high probability of having problems in academic settings (IRS-Academic) with difficulties in reading (CLDQ-Reading) and math (CLDQ-Math) as well as symptoms of inattention (ADHD-IN). On the other hand, this class had low probabilities of symptoms of depression (RCADS-Depression), anxiety (RCASDS-Anxiety), hyperactivity (ADHD-HY), and conduct problems (VAN-Conduct).

The second largest class was the “High Complexity” class (30%), shown with the solid black line. This class generally had the highest probabilities of difficulties across all of the indicators, including impairment, academic, ADHD, affect, and conduct domains. The third largest group (20%) was the “Low Complexity” class (dashed lines), who had few symptoms overall, except a slight probability (.3) of impairment in the academic (IRS-Academic) and reading (CLDQ-Reading) domains. Lastly, the smallest group (13%, dotted line), termed the “Behavioral Focus” class, had elevated probability of problems in most of the impairment domains. Similar to the “High Complexity” group, the “Behavioral Focus” class had very high elevations on externalizing behavioral indicators and somewhat elevated internalizing behavioral indicators. However, unlike the “High Complexity” class, the “Behavioral Focus” class had low probability of academic problems in general (CLDQ-Reading and CLDQ-Math).

Demographics

There were several demographic differences between the classes (see Table 1). In these analyses, the High Complexity class was chosen as the reference group since they reflect a population thought to be in greatest need of assessment. The Behavioral Focus class was distinguished by having a lower probability of being Black/African American (AA) (Relative Risk Ratio (RRR): .65, 95% CI: .45, .93) and receiving Medical Assistance (RRR: .50, 95% CI: .37, .69) when compared to the High Complexity group. On the other hand, the Behavioral Focus group had a greater probability of including boys (RRR: 1.67, 95%: 1.18, 2.21) and parents with Graduate degrees (RRR = 2.38, 95% CI: 1.54, 3.67). The Academic and Inattention class had a lower probability of receiving Medical Assistance/public insurance (RRR: .50, 95% CI: .37, .69), a higher probability of having a parent with a Bachelor’s (RRR = 1.50, 95% CI: 1.08, 2.07) or Graduate degree (RRR = 1.62, 95% CI: 1.19, 2.22), and a higher probability of being Black/AA (RRR: 1.38, 95% CI: 1.05, 1.69). The Low Complexity class was less likely to receive Medical Assistance (RRR: .79, 95% CI: .62, .99) and was more likely to be Black/AA (RRR: 1.67, 95% CI: 1.26, 2.02) (all p < .05). There was no difference in child age across the groups (F = .75, p > .05).

Validity

Table 4 presents mean scores from the standardized measures administered during the day of assessment across the four latent classes. There are a number of noteworthy findings. First, in the “High Complexity” class, mean scores on day of assessment measures of adaptive and emotional-behavioral functioning showed the highest level of impairment; academic problems were substantial as well. In many instances, the level of clinical elevation in the High Complexity group was significantly higher than the other groups, with the exception of verbally-based intelligence which was second highest (strongest; beta coefficients not shown, see Table 4 for differences across classes). For the Behavioral Focus class, mean standard scores on tests of verbally-based intelligence (β = 13.4, 95% CI: 9.56, 17.37), reading (β = 13.4, 95% CI: 8.6, 18.2), and math (β = 14.4, 95% CI: 9.7, 19.1), were higher (better) than the High Complexity group and solidly within normal limits for age, suggesting few learning-related concerns. Although second highest (worse) amongst the four classes, the Behavioral Focus group was also lower than the High Complexity class with regard to anxiety (Anxiety, β = −6.31, 95% CI: −9.81, −2.81), depression (Depression, β = −7.81, 95% CI: −11.13, −4.50), and conduct (Conduct, β = −5.34, 95% CI: −9.09, −1.53). Mean adaptive functioning of the Behavioral Focus group was significantly higher (better) than the High Complexity group (β = 6.95, 95% CI: 4.18, 9.73), but was still low compared to population norms.

Table 4.

Standardized measures used for validation; mean (SD).

	High Complexityt	Behavioral Focus	Academic and Inattention	Low Complexity	Overall
VIQ (SS)	92.60 (21.39)	106.06* (21.66)	88.32* (24.32)	89.94 (25.46)	92.25 (24.01)
Adaptive Functioning (SS)	77.59 (11.71)	84.55* (9.38)	83.68* (12.17)	88.47* (12.91)	82.85 (12.51)
Reading (SS)	88.57 (14.88)	101.95* (16.50)	87.22 (16.53)	92.14* (17.08)	89.70 (16.59)
Math (SS)	87.09 (14.35)	101.50* (15.98)	86.96 (16.34)	90.75* (14.50)	88.82 (15.82)
Anxiety (T)	61.64 (12.23)	55.32* (11.98)	54.83* (12.74)	49.54* (9.38)	55.69 (12.60)
Depression (T)	65.69 (13.73)	57.87* (10.42)	53.37* (10.46)	48.28* (8.02)	56.26 (12.82)
Conduct (T)	61.24 (14.75)	55.89* (11.66)	51.25* (11.66)	47.03* (8.42)	53.70 (12.65)

Note. VIQ = Verbal Intelligence Quotient; SS = Standard Score; T = T-score;

^*= p <.05 difference compared to the “High Complexity” Group; see Methods section for descriptions of psychometric tests employed;

^t= reference category

The Academic and Inattention group had the lowest (worst) mean verbally based IQ score amongst the groups, and it was significantly lower than the High Complexity group (β = −4.27, 95% CI: −7.33, −1.21). Mean scores on math and reading measures were no different than those of the High Complexity class, although adaptive functioning was significantly higher (better) (β = 6.09, 95% CI: 4.20, 7.98). Mean scores from standardized parent ratings of this group suggest little concern for affective dysfunction, since scores on those measures were within normal ranges. In fact, scores on scales of anxiety (β = −6.80, 95% CI: −9.3, −4.3), depression (β = −12.3, 95% CI: −14.8, −9.8), and conduct problems (β = −10.0, 95% CI: −12.6, −7.4) were all lower (better) than the High Complexity class (all p < .05).

Finally, mean test scores for the Low Complexity class were generally within normal limits across academic, and affective-behavioral domains, with a mild indication of risk for cognitive and adaptive dysfunction relative to population-based norms. Adaptive functioning (β = 10.9, 95% CI: 8.7, 13.2), reading (β = 3.56, 95% CI: .2, 6.9), math (β = 3.65, 95% CI: .56, 6.7), anxiety (β = −12.1, 95% CI: −14.6, −9.6), depression (β = −17.4, 95% CI: −19.9, −14.9), and conduct (β = −14.2, 95% CI: −16.9, −11.5) problems were all significantly different (better) than the High Complexity class.

Discussion

Historically, day-long comprehensive psychological and neuropsychological assessment has been considered an important diagnostic tool in patient care, but there are a number of limitations to this approach. First and foremost, there is a great demand for psychological and neuropsychological assessment, and exclusive use of a “one-size-fits-all” comprehensive assessment approach limits the availability of assessment slots. Sole reliance upon a day-long comprehensive assessment approach can also lead to long wait times for patients, delaying the identification and treatment of disorders and often stretching these timelines beyond important windows for intervention. Secondly, long wait times can lead to parental frustration and potential appointment cancellations and no-shows, which further delay evaluation and can decrease the efficiency and fiscal stability of the assessment clinics (Kalb et al., 2012). Third, exposing patients to unnecessarily lengthy evaluations can potentially decrease the validity of assessment and raise the likelihood of abnormal scores in the test protocol (Schretlen et al., 2008). Finally, the recent health-related crisis posed by COVID-19 and requirements for social distancing have, at least temporarily, restricted the use of day-long comprehensive testing practices.

To address the limitations associated with a one-size-fits-all comprehensive approach to psychological and neuropsychological assessment, this study sought to identify meaningful subgroups of referred patients to help facilitate efficient assessment planning, resource allocation, and personalization of psychological and neuropsychological evaluation. We proposed a model whereby patients could be scheduled for shorter or longer assessment appointments as well as online versus on-site assessment visits based upon completion of a comprehensive background/symptom questionnaire completed prior to the assessment appointment. There are numerous advantages to this approach over a more traditional phone intake triage model. These include the systematic acquisition of patient background and symptom information in advance of the appointment which can 1) help shorten clinician interview times, 2) be used for assessment planning, and 3) can facilitate clinical and quality improvement research. Moreover, during the COVID-19 pandemic, acquisition of pre-visit background/symptom information can limit the amount of time that patients are required to be “on-site” for neuropsychological evaluation.

We hypothesized that the patient classes would include one or more groups of children with more circumscribed symptom presentation (e.g., impairment in solely the academic domains) as well as groups of children with a more pervasive profile (i.e., patterns of impairment across multiple domains). This type of pre-visit identification of meaningful patient classes could provide an empirical basis for triaging and scheduling patients for specific doses of neuropsychological assessment (e.g., screening visit, targeted assessment, comprehensive assessment, telemedicine consult), thereby providing a model to address the problems of high cost, long wait times, and parental/patient frustration associated with the traditional comprehensive assessment approach. While not a substitute for clinical interview, pre-visit identification of symptom scope could potentially help triage and schedule patients to the correct length and setting of assessment to address the referral concerns.

Findings from the current study support our hypotheses, and four qualitatively different classes of patients emerged from the analyses of pre-visit parent ratings. These groups differed in meaningful ways based upon the scope and severity of parent ratings of the key variables of adaptive, academic, attentional, behavioral, and emotional impairment. In our sample, the “High Complexity” class was quite large (30% of the total sample) and was characterized by report of multi-domain dysfunction. The “Behavioral Focus” class was smallest (13%) and had heightened report of adaptive and emotional/behavioral impairment with few cognitive or academic concerns. The “Academic and Inattention” class was largest (37%) and was characterized by reports of attentional and academic impairments, but with few emotional and behavioral symptoms. Finally, the “Low Complexity” class comprised about a fifth of the sample. This group demonstrated a limited scope of sub-threshold symptoms in attention and academics. Cross-validation was employed to examine the reliability of the model across the split test and training data. The model reproduced across both samples, supporting the stability of the model within the dataset.

Once these patient classes were established, the second objective of this study was to assess the external validity of the pre-visit 4-class model using day-of-assessment performance on standardized neuropsychological measures as the criterion. Test performance differences observed between groups were supportive of the validity of the 4-class model. The High Complexity class clearly demonstrated the most impairment on the day of assessment, with the highest (or most problematic) mean scores on most of the standardized measures. The Low Complexity class had scores generally within normal limits, although there was indication of possible cognitive difficulties. The Behavioral Focus group showed little indication of problems with intelligence, reading, or math, but their ratings on standard of care measures of anxiety, depression, conduct, and adaptive functioning on the day of assessment were second highest (worst) among the groups. The Academic and Inattention group had the lowest verbally-based mean IQ score along with the lowest math and reading scores, suggestive of cognitive and academic difficulties. Taken together, the validity and stability of the 4-class model was supported by standardized clinical assessments and cross validation, respectively.

With regard to clinical application, these findings support the idea of acquiring pre-visit parent-reported symptoms to facilitate scheduling of psychological and neuropsychological assessment based upon the scope of the patient’s symptoms. For several of these groups, we recommend scheduling time-efficient screening, targeted assessment, or online/telemedicine visits as an initial step to address the referral concerns. For instance, it may be possible to assess the circumscribed assessment questions of a Low Complexity patient via a telemedicine screening or highly targeted visit, which, by its nature, would be much shorter – allowing for more patients to be seen by the individual clinician. While day-of-assessment clinical interview and testing of Low Complexity patients will likely identify some patients with diagnostic questions that cannot be answered in a short/targeted evaluation, we predict that a shorter scheduled assessment visit will be adequate to address the referral concerns of the majority of these patients.

Similarly, although full-day comprehensive assessment can be helpful in identifying and defining symptoms of patients in the Behavioral Focus group, patients in this group might be best served by brief screening or targeted assessment followed by rapid routing of the patient to behavioral intervention services. While we predict that this will be the most parsimonious and effective way of addressing the behavioral needs of most of the patients in this group, there will likely be a need for re-referral to assessment services in a minority of these patients. Using this model, when the behavioral intervention specialist determines that further assessment remains necessary for diagnostic clarification or treatment planning decision-making, return-referral of Behavioral Focus patients for comprehensive neuropsychological assessment would be possible.

In contrast, it would make sense to allocate and schedule day-long comprehensive assessment slots for those patients identified as having a more pervasive scope of symptoms based upon pre-visit parent report. For instance, we recommend scheduling day-long comprehensive assessments of High Complexity patients, as these patients’ clinical needs will likely be pervasive and involve assessment of multiple symptom domains. Similarly, the Academic and Inattention group of patients might be best served via day-long comprehensive assessment as well, which allows a broad assessment to intellectual, academic, and attention domains, but also permits a targeted “drill down” into those specific areas of cognitive functioning identified to be problematic.

It is important to highlight that there were several notable demographic differences between the groups. The High Complexity class included youth who were disproportionately receiving Medical Assistance/public insurance and had the lowest levels of parental education. Both of these indicators are associated with increased incidence of developmental disabilities (Mazza et al., 2017; Van Oort et al., 2011) and provide secondary evidence of the complexity of this class. Not surprisingly, the Behavioral Focus class was disproportionately male and was consistent with findings showing male gender being a risk factor for behavioral dysregulation (Willcutt, 2012). This group was also more likely to be Caucasian and have parents who were highly educated. The Academic and Inattention group as well as the Low Complexity groups were more likely to be Black/African American and have private insurance compared to the High Complexity group. The Academic and Inattention group had slightly higher levels of education than the High Complexity group. While these findings raise additional questions, it is our impression that they should be interpreted cautiously. As research literature in this area suggests measurement invariance in the perceptions of symptoms of ADHD and behavioral problems between African American and Caucasian parents, it is our impression that the noted racial and parental education differences between groups is likely the result of similar measurement invariance and should be interpreted very conservatively.

There were several methodological limitations to the present study, many of which involved missing or incomplete data. First, although a parent or caregiver of each of the patients in this study responded to a developmental questionnaire and rating form, portions of the questionnaire were periodically left incomplete (e.g., parental education). Second, day of assessment standardized measures were administered at the discretion of the licensed psychologist within the context of clinical care. As such, not all children were assessed in all of the domains used to validate the model, and specific measures of each domain assessed varied. Third, inter-rater reliability, or consistency, across clinicians when performing assessments is unknown. Fourth, while our sample is large, it is comprised of a large percentage of children with Attention-deficit/Hyperactivity Disorder (ADHD; 44%). As such, the generalizability of this model to assessment programs with a different clinical composition (e.g., a greater proportion of medically-involved patients) is unclear. Fifth, and finally, post-assessment diagnostic formulation was not available for review and analysis due to limitations in the electronic health record system.

Despite these limitations, the strengths of this study are thought to support the feasibility and potential utility of pre-visit data collection aimed at supporting pre-visit triage and schedule decision-making, as well as promoting personalized assessment based upon patient symptomatology. These strengths include a large sample size, use of cross-validation, and validation of class structure via standardized assessment. The study methodology is also considered a strength, as the clinical data collection approach used could be easily replicated given the declining costs of online data collection platforms and the very limited costs associated with the use of validated public domain measures.

In sum, findings identified four distinct classes of patient complexity in this large, referred, diverse sample which could potentially be used to guide patient triage, scheduling, and assessment planning. Future directions include the need for replication and prospective validation of the model in other clinical populations with complete neuropsychological assessment data.