Does Assessment Format Matter? A Comparison of In-Person Versus Teletesting Scores for Youth with ADHD

Abstract

Objective:

This study examined test score equivalency between traditional in-person assessment and teletesting among youth diagnosed with Attention-Deficit/Hyperactivity Disorder (ADHD).

Method:

In all, 896 youth with ADHD, ages 5–21 years, were administered cognitive, academic achievement, and verbal fluency measures via either teletesting (n = 448) or traditional in-person assessment (n = 448). The teletesting and in-person groups were matched on age, sex, and insurance type (as a proxy for income).

Results:

Results indicated no significant differences in test scores obtained via in-person and teletesting evaluations across all examined measures.

Conclusion:

Clinically referred youth with ADHD perform similarly on measures of cognitive functioning, academic achievement, and verbal fluency, regardless of whether these measures are administered in-person or via teletesting. While additional evidence for equivalent psychometric properties of neuropsychological instruments administered remotely is needed, this study offers support for the validity of remote administration among youth with ADHD.

Children and adolescents across the globe historically have faced barriers to accessing mental health services (Cummings et al., 2013), and mental health problems have been exacerbated in this population by the COVID-19 pandemic (Office of the Surgeon General [OSG], 2021). Data from the Centers for Disease Control and Prevention estimate that one in six children in the United States have a diagnosed mental, behavioral, or developmental disorder (Cree, 2018), and up to 80% of these youth do not have adequate access to services (Cummings et al., 2013). Given that Attention-Deficit/Hyperactivity Disorder (ADHD) is the most common neurodevelopmental disorder of childhood (Danielson et al., 2018), children with ADHD may face particular risk for access to services. A critical and often necessary first step to identifying effective services is psychological/diagnostic assessment, a service for which many youth wait an inordinate amount of time, delaying access to treatment.

Telehealth, the remote delivery of healthcare services, is a promising approach to barrier reduction when it comes to children and adolescents accessing mental health services. Telehealth has been utilized and found efficacious in the delivery of psychological treatments across a variety of patient populations (Comer et al., 2021; Ferguson et al., 2019; Ros-DeMarize et al., 2021); however, research on the utility of teletesting, the remote administration of cognitive and other performance-based psychological measures, remains limited.

Teletesting increased in prevalence as a result of the COVID-19 pandemic (e.g., Pritchard et al., 2020). There are several potential benefits to teletesting, including timely access to psychological assessment services for those living in historically underserved areas, as well as the removal of burdens often placed on families who are seeking services outside of their immediate area (e.g., travel burden, time off work, arranging childcare for siblings; Ransom et al., 2020). Additionally, research has demonstrated that teletesting is viewed favorably by pediatric patients and their families (Harder et al., 2020). While there are clearly several advantages to teletesting, the majority of assessment measures have been normed via in-person administration using paper and pencil testing procedures, which renders the validity of remote administration questionable. Research demonstrating the validity of teletesting is needed to support its continued use beyond the COVID-19 pandemic. Specifically, there is a need for research demonstrating the equivalency of teletesting to traditional in-person assessment across a range of patient populations and with a variety of commonly used psychological measures.

Existing Teletesting Equivalency Research

There is preliminary evidence suggesting equivalence of psychological test results obtained via teletesting and traditional in-person testing within mixed clinically referred samples of children and adolescents (Hamner et al., 2021; Harder et al., 2020), and with general school samples of children and adolescents (Wright, 2018, 2020). These studies offer support for equivalency of measures such as the Wechsler Intelligence Scale for Children, 5^th Edition (WISC-V; Hamner et al., 2021; Wright, 2020), the Kaufman Test of Educational Achievement, 3^rd Edition (KTEA-3; Hamner et al., 2021), the California Verbal Learning Test-Children’s version (CVLT-C; Harder et al., 2020), Delis–Kaplan Executive Function System (D-KEFS; Harder et al., 2020), and Woodcock–Johnson Cognitive and Achievement tests (Harder et al., 2020; Wright, 2018).

A meta-analysis by Marra et al. (2020), included 19 studies of teletesting across a diverse population of adult participants (i.e., participants from five different countries; various ethnic/cultural backgrounds; a variety of diagnoses). Findings indicated strong support for teletesting validity for a variety of language, attention/working memory, and memory tasks. The authors concluded that there is good support for teletesting in older adult populations.

While the evidence base demonstrating equivalence of assessment conducted in-person versus teletesting among various populations is accumulating, research demonstrating the equivalence of in-person and teletesting for specific pediatric populations is lacking. This evidence is important because specific populations may present with characteristics that could uniquely impact performance during teletesting.

Teletesting and ADHD

Children and adolescents diagnosed with ADHD may be at increased risk of underperformance when completing standardized assessment measures during psychological evaluation as core symptoms of ADHD can interfere with performance (e.g., distractibility, impulsivity). In an in-person testing environment, clinicians can employ behavioral strategies for reducing the impact of ADHD symptoms on testing performance (e.g., removal of distractions). During teletesting, clinicians have less control over the testing environment. For instance, clinicians cannot control the setting when a patient is being tested remotely in their home to eliminate distractions (caregivers or siblings may inadvertently interrupt testing) or to assist with behavioral control and environmental management of the hyperactive or impulsive patient to ensure that they remain seated and facing the stimuli. Given the difference between the in-person and teletesting environments, it is especially important to demonstrate equivalency for this population.

The goal of the present study was to expand upon the pediatric teletesting literature by examining the equivalence of subtests from cognitive, academic, and executive function measures administered via teletesting versus traditional, in-person testing within a sample of children clinically diagnosed with ADHD. These measures were chosen for comparison because they were the most frequently administered measures for patients diagnosed with ADHD in our clinical population. Based on the results of previous research examining equivalence between teletesting and in-person assessment, we hypothesized no significant differences in scores between administration type across all measures.

Method

Study Design, Inclusion Criteria, and Sample

Participants in this retrospective, cross-sectional study were referred for psychological/neuropsychological assessment at an urban, outpatient, assessment clinic at a pediatric hospital in the Mid-Atlantic region of the US. To be included in this study, participants must have (a) a clinical diagnosis of ADHD listed within the medical record and (b) received a psychological/neuropsychological assessment including any of the measures of interest between July 2019 and February 2022. Prior to the start of the COVID-19 pandemic in March 2020, this assessment clinic provided 100% of services in person. After the pandemic began, this clinic transitioned to a hybrid model of care, offering both telehealth and in-person assessments. Evaluations were conducted by clinical psychologists and neuropsychologists. Data from clinical evaluations are routinely entered into the electronic medical record, securely maintained by the hospital’s Information Systems staff, and de-identified records can be retrieved for analysis following appropriate approvals. The hospital’s Institutional Review Board approved this retrospective study under a waiver of consent.

Participants in this study were youth between age 5 and 21 years (mean age = 10.97, SD = 3.12) with a clinical diagnosis of ADHD (N = 896). Half of the participants received a psychological/neuropsychological assessment via telehealth (N = 448) between March of 2020 and February 2022, and the other half received an in-person psychological/neuropsychological assessment (N = 448) between July 2019 and February 2022. Prior to the onset of the COVID-19 pandemic in March 2020, all clinical evaluations were completed in person. Starting in late March 2020, in response to the pandemic, clinical evaluations were only offered remotely via telehealth until July 2020. Since July 2020, clinical evaluations have been offered both in person and via telehealth. Evaluations conducted in-person after the onset of the pandemic utilized any or all of the following precautions which may have interfered with standardized test administration: patient face masks, clinician face masks and face shields, and plexiglass or plastic barriers covering stimulus books. Decisions about which modality is most appropriate for a given patient are made by providers in collaboration with patients and families, taking into account a wide variety of factors. This decision-making process is detailed in Pritchard et al. (2020).

The teletesting and in-person groups were matched on age, sex, and insurance type (e.g., commercial or public insurance/Medicaid), which served as a proxy for income. Participants were identified in the medical record as White (49%), Black/African-American (32%), or Other (19%). Three age groups were calculated for use in exploratory analyses: children aged 5–10 years, children aged 11–14 years, and children 14 and older. These groupings were chosen to align with typical age groups for elementary, middle, and high school.

Dependent Variables

Academic achievement.

Selected subtests from the KTEA-3 (Kaufman & Kaufman, 2014) were used clinically for educational screening, given the comorbidity of learning disabilities with ADHD. The KTEA-3 is a psychometrically sound academic assessment designed for individuals aged 4–26 years. Scores from the Letter & Word Recognition, Nonsense Word Decoding, Math Concepts & Applications, and Reading Comprehension subtests were included in this study as they were available for both remote and in-person administration and were the most commonly used academic screening measures within the clinic.

Intelligence.

Selected subtests from the WISC-V (Wechsler, 2014) were used to assess intellectual functioning, a domain often assessed in the differential diagnosis of ADHD. The WISC-V is a validated, psychometrically sound, cognitive assessment for use in children and adolescents aged 6–16 years. The Similarities, Vocabulary, Matrix Reasoning, Visual Puzzles, and Digit Span subtests were chosen as these subtests were available for both remote and in-person administration, provide a broad assessment of intellectual functioning, and were the most commonly used measures of intellectual functioning within the clinic.

Verbal fluency.

Verbal fluency, a domain that may be impaired among individuals with ADHD (e.g., Grodzinsky & Barkley, 1999), was assessed using the D-KEFS (Delis et al., 2001) or the NEPSY-II (Korkman et al., 2007). The D-KEFS is a psychometrically sound assessment designed for individuals aged 8–89 years. The NEPSY-II is a psychometrically sound assessment designed for individuals aged 4–16 years. The D-KEFS Verbal Fluency subtest and the NEPSY-II Word Generation subtest were chosen as they were available for both in-person and remote administration, and they represent the most commonly administered measures of verbal fluency within our clinic.

Independent Variables

Mode of assessment.

Mode of assessment was classified as either in-person or telehealth. In-person assessments were completed at the clinic, under traditional, controlled, and standardized assessment conditions. Telehealth assessments were completed remotely, via the HIPAA-compliant Zoom platform. For all telehealth assessments, examiners were located in a quiet, distraction-free space (either in their office at the clinic or in a separate, unoccupied space in their home). Patients were instructed to locate a quiet, comfortable, and distraction-free environment from which to participate in their telehealth assessment. They were required to participate via a desktop, laptop, or a tablet with at least a 9.7″ screen. Appropriate devices were provided on loan to the family, if needed, via a federal telehealth grant. Given the nature of the patient population, caregivers were asked to remain nearby, but ideally not in the same room as the patient, during the telehealth appointment, so that they could be available to assist with technical difficulties or to help redirect the child if needed. Breaks were offered, as needed, throughout both in-person and telehealth appointments.

ADHD subtype.

Billing diagnoses were extracted from the electronic health record system for all participants. Billing diagnoses, based on International Classification of Diseases codes, 10^th edition, were classified as ADHD-predominantly inattentive presentation (ADHD-IA; ICD code F90.0; n = 283), ADHD-predominantly hyperactive/impulsive presentation (ADHD-HI; ICD code F90.1; n = 15), ADHD-combined presentation (ADHD-C; ICD code F90.2; n = 446), and Other Specified ADHD (ICD code F90.8; n = 93). Because very few participants were diagnosed with ADHD-HI or Other Specified ADHD, participants with these two subtypes were excluded from analyses involving the ADHD subtype variable.

Data Analysis

Independent sample t-tests and chi-square tests were used to confirm group equivalence on the demographic characteristics on which the sample was matched. Next, mean comparisons across groups on subtests administered were examined using independent-sample t-tests, presented with Cohen’s d effect size estimates. Finally, a series of multi-variate analyses of variance (MANOVAs), presented with Wilks λ effect size estimates, were used to investigate possible interactions between ADHD subtype and group in terms of subtest scores. MANOVAs were run separately for each dependent variable to maximize sample sizes for analysis, given that few participants completed measures for all dependent variables. The Bonferroni correction was applied to adjust for multiple comparisons.

Results

The in-person and telehealth groups were found to be equivalent on all of the relevant demographic variables: age at visit, race, sex, and insurance type (see Table 1), confirming that matching was effective. Table 2 presents summary scores for the in-person and telehealth groups across all examined subtests, as well as results of t-tests indexing group comparisons. The in-person and telehealth groups were equivalent across all examined subtests.

Table 1.

Group Differences on Demographic Characteristics.

	In person		Telehealth
	n	%	n	%	χ2	p-Value
Race					0.62	.735
Black	141	31.47	145	32.37
White	224	50.00	213	47.54
Other	83	18.53	90	20.09
Sex					0.00	1.00
Female	160	35.71	160	35.71
Male	288	64.29	288	64.29
Insurance type					0.00	1.00
Medicaid	173	38.62	173	38.62
Commercial	275	61.38	275	61.38
	n	Mean (SD)	n	Mean (SD)	t	p-Value
Age at visit	448	10.95 (3.14)	448	10.98 (3.11)	0.12	.902

Note. Comparisons between groups for categorical variables were conducted using chi-square tests (χ²) and for continuous variables were conducted as independent-samples t-tests. SD = standard deviation.

Table 2.

Group Differences on Performance-Based Measures.

			In person		Telehealth
	n	M	SD	n	M	SD	df	t	p-Value	Cohen’s d
WISC-V
Similarities	359	9.48	3.29	210	9.18	3.10	567	1.07	.284	.09
Matrix Rsng	360	8.88	3.33	214	8.41	3.11	572	1.70	.090	.15
Vocabulary	339	9.31	3.45	187	9.28	3.28	524	0.09	.932	.01
Digit span	368	7.89	3.03	303	8.30	3.48	669	1.63	.104	.13
Vis puzzles	320	9.35	3.39	158	9.12	3.27	476	0.73	.464	.07
KTEA-3
Letter word	262	90.00	15.40	267	89.59	16.08	527	0.30	.764	.03
Nonsense word	206	85.34	12.88	156	87.03	13.24	260	1.02	.307	.13
Reading comp	163	87.91	14.49	148	88.24	14.99	309	0.19	.847	.02
Math concepts	179	84.92	14.65	195	86.90	16.11	372	1.24	.216	.13
DKEFS
Verb flu letter	153	9.12	3.23	230	8.98	3.24	381	0.40	.682	.04
Verb flu categ	156	10.28	3.78	232	10.07	3.63	386	0.57	.568	.06
NEPSY 2
Word gen sem	64	9.05	3.72	63	9.46	3.18	125	0.67	.502	.12
Word gen lett	42	7.55	2.46	44	7.39	3.03	84	0.27	.788	.06

Note. WISC-V = Wechsler intelligence scale for children, 5th edition; Matrix Rsng = matrix reasoning; Vis Puzzles = visual puzzles; KTEA-3 = Kaufman test of educational achievement, 3rd edition; Reading Comp = reading comprehension; Math Concepts = math concepts and applications; DKEFS = Delis–Kaplan executive function system; Verb Flu Letter = verbal fluency test: letter fluency; Verb Flu Categ = verbal fluency test: category fluency; Word Gen Sem = word generation: semantic; Word Gen Lett = word generation: initial letter; M = mean; SD = standard deviation; df = degrees of freedom. Comparisons between groups were conducted using independent-samples t-tests.

A series of exploratory MANOVAs were conducted to examine the interaction between mode of assessment (on-site vs. teletesting) and (1) ADHD subtype (ADHD-IA vs. ADHD-C), (2) age group (elementary, middle, or high school aged), and (3) socioeconomic status (medical assistance/Medicaid vs. commercial insurance) on subtest performance within each measure (i.e., WISC-5, KTEA-3, D-KEFS, NEPSY-2). Across all MANOVAs no significant interaction effects emerged, suggesting that telehealth equivalence is consistent across ADHD subtype, age group, and socioeconomic status.

Discussion

This study examined the equivalence of teletesting versus in-person assessment within a sample of children and adolescents with a clinical diagnosis of ADHD. This question is of critical importance as it is imperative that the field has confidence in the results obtained via teletesting. In addition, given that teletesting will likely remain common practice for psychological assessments, evaluating its equivalency to in-person testing for specific pediatric populations, such as those with ADHD, is crucial. Results indicated no significant differences in test scores obtained via in-person and teletesting across all examined measures. These findings are consistent with previous research demonstrating equivalency between teletesting and traditional in-person testing (e.g., Hamner et al., 2021; Harder et al., 2020; Wright, 2018, 2020), and extend that literature to additional measures and to the pediatric ADHD population. Specifically, findings demonstrate that tasks assessing cognitive and academic skills, and verbal fluency can be reliably utilized in the context of teletesting for people diagnosed with ADHD.

Teletesting is poised to play a critical role in reducing long-standing barriers to accessing mental health services for children and adolescents. Benefits to teletesting can include timely access to psychological assessment services for those living in historically underserved areas, and reduction in the time between the emergence of concerns and access to mental health services (Ransom et al., 2020). Additionally, research has demonstrated that teletesting is viewed favorably by pediatric patients and their families (Harder et al., 2020).

Although teletesting is a promising approach to providing timely assessment services to children and adolescents, there are limitations to teletesting that should be considered. One potential limitation to teletesting is clinicians’ limited control over the immediate testing environment during the teletesting appointment. This lack of control could be problematic, particularly for children and adolescents who are prone to distraction and who require support to remain focused and engaged in testing. As noted by Bilder et al. (2020), this limitation could be minimized through pre-session preparation (e.g., clearly defining the need for a quiet, distraction-free space; coaching parents or caregivers on how to identify and set up an appropriate testing space) and close parental monitoring during the testing session. Notably, the clinical characteristics of ADHD that could interfere with teletesting did not appear to differentially impact the performances of the youth included in the present study, when they were appropriately prepared for the teletesting format. Furthermore, access to and familiarity with technology can be a barrier to participating in teletesting for some families.

The current study adds to the growing evidence demonstrating the equivalence of psychological tests administered during teletesting (e.g., Hamner et al., 2021; Harder et al., 2020; Wright, 2018, 2020). Of course, teletesting is not a feasible modality for all measures or to assess all domains of functioning. Measures of fine motor function, for instance, do not easily lend themselves to remote administration. Still, growing evidence suggests that many measures may be reliably administered remotely. To further establish teletesting as a reliable and valid assessment modality, and to ensure results are accepted by schools and other disability service organizations, additional research establishing the psychometric properties of commonly used psychological tests administered during teletesting is needed across varied patient populations. Similarly, additional research demonstrating how to modify specific performance-based measures to prevent further disruption to standardization of test administration is also needed (Bilder et al., 2020). Research exploring the utility of teletesting for screening to determine need for in-person assessment is also needed as this may be another option for reducing long clinic wait times, increasing access to care, and reducing the delay from onset of concerns to treatment (Ransom et al., 2020).

While the present study contributes further support for the equivalence of in-person and teletesting assessment methods, it is not without limitations. First, the sample of clinically referred children and adolescents with a diagnosis of ADHD from a single site included in this study may not be generalizable to community pediatric populations. Second, while every effort was made to control for demographic variables (age, sex, race, insurance type), there may be other variables that we did not have access to that could contribute to performance that may have been significantly different for our two groups. This seems relatively unlikely, however, given that uncontrolled between-group differences on other variables would be more likely to result in differences between groups on the measures of interest, rather than a lack of differences between groups, as observed in this study. Finally, our findings suggest that youth with a clinical diagnosis of ADHD perform similarly across teletesting and traditional in-person testing; however, it remains unclear whether the psychometric properties of the measures administered via teletesting are the same as when administered during in-person testing.

The evidence for the validity of psychological test administration via teletesting for children and adolescents is growing. The present study contributes to the existing literature by demonstrating equivalency between teletesting and traditional in-person assessment methods among children and adolescents with a clinical diagnosis of ADHD across a variety of commonly used psychological measures. The findings from this study are critically important for supporting the ongoing use of teletesting to increase access to mental health services for children and adolescents.

Biographies

Shelley M. McDermott, Ph.D., BCBA-D, is a board-certified behavior analyst and a child and adolescent clinical psychologist in the Neuropsychology Department at the Kennedy Krieger Institute. She also holds an appointment as Instructor in the Johns Hopkins School of Medicine's Department of Psychiatry & Behavioral Sciences.

Kristie Sweeney, M.S., is a registered psychology associate in and Clinical Services Manager of the Neuropsychology Department at the Kennedy Krieger Institute.

Lisa A. Jacobson, Ph.D., ABPP, is a board-certified neuropsychologist and Director of Research in the Neuropsychology Department at the Kennedy Krieger Institute. She also holds an appointment as Associate Professor in the Johns Hopkins University School of Medicine's Department of Psychiatry & Behavioral Sciences.

Rebecca W. Lieb, PhD, ABPP, is a board-certified clinical child and adolescent psychologist in the Neuropsychology Department at the Kennedy Krieger Institute. She also holds an appointment as Assistant Professor in the Johns Hopkins University School of Medicine’s Department of Psychiatry & Behavioral Sciences.

Danielle Wexler, Ph.D., is a child and adolescent clinical psychologist in the Neuropsychology Department at the Kennedy Krieger Institute.

Alison E. Pritchard, PhD, ABPP is a board-certified clinical child and adolescent psychologist and co-director of the Neuropsychology Department at the Kennedy Krieger Institute. She also holds an appointment as Assistant Professor in the Johns Hopkins University School of Medicine’s Department of Psychiatry & Behavioral Sciences.