Equivalence of In-Person and Videoconference Administration of the Physical and Neurological Examination of Subtle Signs (PANESS) Timed Motor Section

Abstract

Demand for telehealth services has increased in many settings as a means of reducing patient burden and increasing access to care. It is therefore critical to understand if clinical tools validated for in-person use are feasible to administer via telehealth and if so, how results may vary from those obtained in-person. The Revised Physical and Neurological Examination of Subtle Signs (PANESS) is a validated assessment of pediatric neuromotor functioning that is sensitive to detecting motor dysfunction in youth. The Timed Motor section of the PANESS is an ideal candidate for telehealth administration given its strong inter-rater reliability when scored via video review. Here, we preliminarily examined the feasibility of administering the Timed Motor section of PANESS via videoconferencing and its equivalence with in-person administration. We administered the Timed Motor section of the PANESS in-person and via videoconferencing in 25 typically developing youth ages 10–18 (including n=11 who had clinically recovered from concussion). All were able to complete the assessment via videoconference. We observed moderate to excellent reliability of scores obtained in-person and via videoconference (intraclass correlations ranging from .743 to .971). Results suggest that the Timed Motor Section of the PANESS can be administered remotely in typically developing youth (including those with history of concussion) and that scores obtained are stable with in-person scores. Future work is needed to examine the feasibility and equivalence of telehealth-based PANESS administration in clinical settings and patient populations.

The Revised Physical and Neurological Examination of Subtle Signs, or PANESS, is a brief behavioral assessment that characterizes subtle neuromotor functioning in pediatric populations.¹ The PANESS is administered in three sections which broadly examine 1) lateral preference, 2) gait and balance (Gaits and Stations section), and 3) motor speed (Timed Motor section). The PANESS captures information related to motor precision and speed while accounting for the presence of subtle neuromotor phenomena such as motor overflow, dysrhythmia, and choreiform movements.² The Timed Motor section of the PANESS yields four primary summary scores which are described in detail elsewhere³: Total Timed Overflow, Total Dysrhythmia, Total SFA (standard deviation from average), and Total Timed. Importantly, the total time that it takes to complete movements within the Timed Motor section can be summed to generate Raw Speed Scores (e.g., Total Timed Motor Speed), which may be particularly helpful for use within the context of longitudinal study designs.^3,4

The PANESS has been used to characterize the development of neuromotor functioning in neurotypical children,^1,2,5 as well as among children and adolescents with neurodevelopmental disorders^6–8 and moderate to severe traumatic brain injury (TBI).^9,10 More recently, the measure has also been applied to detect subtle motor changes among youth with mild traumatic brain injury (mTBI) or concussion^11–13 and to distinguish these youth from uninjured peers.¹⁴ Neuromotor functioning assessed using the PANESS has been associated with measures of brain structure and function,^15,16 highlighting its utility in both clinical and research contexts.

The psychometric properties of the PANESS have been well-described, with evidence of strong test-retest reliability,^12,17,18 internal consistency,¹⁸ construct validity,¹⁹ and inter-rater reliability.^3,18 However, the reliability of PANESS scores obtained across different testing environments (i.e., cross-context reliability²⁰) has not yet been examined for the PANESS.

With increasing demand for clinical telehealth services in the context of the COVID-19 pandemic, there has been a growing need to examine the feasibility and validity of administering neuropsychological and neuromotor measures designed for in-person administration, via telehealth.^21–25 Even as pandemic restrictions have lifted, demand for telehealth services has continued and research teams may also wish to leverage the many outlined benefits of telehealth,²⁶ including increasing access to a broader pool of research participants and reducing participant burden. However, to date, no known studies have examined the feasibility of administering the PANESS via available telehealth videoconferencing platforms nor investigated the reliability of scores obtained in person versus via videoconferencing.

We anticipated that the Timed Motor section of the PANESS would be an ideal candidate for telehealth administration given prior work demonstrating that scores obtained from this section yielded the highest inter-rater reliability using a “one-time-through” video review method for scoring in-person administrations.³ Practically, this section of the PANESS can be administered while the examinee is in a seated position and does not require multiple camera angles nor the need for a third party on location to assist with videorecording (as may be the case with the Gaits and Stations section of the PANESS). Notably, this is the same section that is used to generate Raw Speed scores which appear particularly useful in modeling trajectories of motor functioning over time.^3,4 Here, we sought to 1) examine the feasibility of administering the Timed Motor Section of the PANESS using videoconferencing and 2) assess the reliability between scores obtained via traditional in-person administration and via videoconferencing.

Methods

Participants and Procedure

All study procedures were approved by the Johns Hopkins University School of Medicine Institutional Review Board. Participants from this study were part of a larger longitudinal study examining motor and cognitive recovery in youth following medical clearance from concussion (including a subset of participants which overlap with prior published studies.^3,15 Participants included 25 typically developing youth ages 10–18 years old at the time of their study visit (Mean age= 14.49; SD= 2.31) who were administered the Timed Motor section of the PANESS both in-person and via videoconferencing. 11 youth in the sample had history of concussion status post medical clearance for full return to activities (Mean days since injury = 478.73, SD = 76.22). See Table 1 for full demographic details.

Table 1.

Participant Demographics

	n=25
Age at Visit, M (SD)	14.49 (2.31)
Sex, n (%)
Female	13 (52%)
Male	12 (48%)
Race, n (%)
Black or African American	2 (8%)
White	20 (80%)
More Than One Race	3 (12%)
Maternal Education, n (%)
Some College, No Degree	1 (4%)
Undergraduate Degree1	12 (48%)
Master’s Degree	8 (32%)
Doctoral Degree	4 (16%)
Handedness, n (%)
Right Hand	24 (96%)
Left Hand	1 (4%)
Youth with History of Concussion, n (%)	11 (44%)
Days since injury, M (SD)	478.73 (76.22)

¹Undergraduate degree includes associate degree (n=2), professional school (n=1), and bachelor’s degree (n=9)

Participants were administered the Timed Motor section of the PANESS in-person as part of the third and last study visit for the broader longitudinal study. To address our research question related to the equivalence of in-person versus videoconference-administered Timed Motor section of the PANESS, youth were also administered the Timed Motor section of the PANESS via videoconference at the same study visit timepoint. Importantly, youth had previously been exposed to the PANESS at the enrollment and 3-month timepoints for the broader longitudinal study, which allowed for us to mitigate practice effects in our evaluation of reliability across administration contexts.

Measures

The Timed Motor section of the PANESS was administered in-person using standard administration procedures. Participants were seated directly across from the administrator within a laboratory testing room. The Timed Motor section of the PANESS was completed, where participants engage in repetitive and sequential movements as quickly as possible, with the administrator providing brief demonstrations of each movement and ensuring participant comprehension.

For the videoconference-based administration of the Timed Motor section of PANESS, participants either went to a private testing room within our research center to join a Zoom videoconference or joined from a private location within their home within one week of the in-person study visit. Participants were asked to join the Zoom videoconference via a smartphone device and were instructed to sit in a chair without armrests or wheels. Participants who were on site for the videoconference joined using an iPhone SE provided by the study team. Those who were at home joined the videoconference using a personal/family-member smartphone device. The examiner worked with the participant to adjust the positioning of the participant and/or device as necessary so that the examinee’s entire body (including both feet) were visible over Zoom. The administrator pinned the participant’s video feed within the Zoom videoconference to maximize visibility. The examiner then followed the same procedures for the Timed Motor section as in-person administration; first, demonstrating each action (though over Zoom), and then having the participant practice briefly to ensure comprehension. For the foot tapping and heel-toe-rocking items, the examiner used words to describe the movement and used a hand rather than a foot to demonstrate the movement.

Both in-person and videoconference PANESS assessments for the Timed Motor section were scored in real-time by a trained study staff member (either a masters-level research assistant or a clinical neuropsychologist). To uphold study quality control and data verification procedures, both in-person and videoconference PANESS administrations were recorded so that the examiner could refer to video recordings as needed to confirm live scores. All summary scores were calculated using the most current scoring conventions for PANESS which are detailed elsewhere.³

Statistical analysis

Descriptive statistics were conducted to characterize the study sample. Paired sample t-tests (or Wilcoxon signed rank tests in the case of significant normality threats) were used to preliminarily examine if PANESS scores varied significantly by administration context and Cohen’s d (or r in the case of Wilcoxon signed rank test) was reported as a measure of effect size considering our relatively small sample size. To evaluate the reliability between scores obtained via in-person PANESS administration versus via videoconferencing, intraclass correlation coefficients (ICCs) were yielded using a two-way random-effects absolute agreement model as defined by Koo and colleagues.²⁷ Confidence intervals (CI; 95%) were generated and upper and lower limits were interpreted using standard conventions, with 0.5 classified as poor, 0.5 to 0.75 moderate, 0.75 to 0.9 good, and greater than 0.90 excellent.²⁷

Results

All participants were able to complete the PANESS Timed Motor Section assessment in person and via videoconferencing, resulting in data from 50 total administrations (25 obtained in-person and 25 obtained via videoconferencing). For all participants, in-person and videoconference administrations occurred within a week of each other, with the majority occurring on the same day (Median days between administrations= 0; Mean= 1.16, SD = 1.92). Most participants joined the Zoom videoconference from a private office/testing room within our research center (n=16), with the remainder joining from a private location within their home environment (n=9). Most of the sample (n=22) were administered the PANESS Timed Motor section in-person first; however, 3 youth completed PANESS via videoconferencing first due to unforeseen scheduling conflicts.

Descriptive statistics for the PANESS Timed Motor Section summary scores across administration context (i.e., in-person vs. via videoconferencing) are presented in Table 2. No statistically significant differences were found between scores obtained in-person and via telehealth administration for any of the traditional PANESS summary scores nor for the summary metric from the PANESS Timed Motor Raw Speed Score, Total Timed Motor Speed and effect sizes were small (<.2) with the exception of Total Dysrhythmia which was small to moderate (Cohen’s d= .365; see Table 2).

Table 2:

Descriptive Statistics for PANESS Timed Motor Section Summary Scores

	Mean (SD; Range)
	In-Person	Videoconference	P	Effect size
Total Timed Overflow (Out of 25)	6.20 (3.41; 1–12)	6.24 (3.40; 1–13)	.949	−.013
Total Dysrhythmia (Out of 13)	4.48 (2.43; 0–10)	3.80 (3.39; 0–8)	.081	.365
Total SFA1 (Out of 26)	5.60 (4.45; 0–18)	6.36 (4.94; 0–19)	.3492	.187
Total Timed (Out of 64)	16.28 (6.19; 5–29)	16.40 (6.85; 4–34)	.910	.023
Total Timed Motor Speed	65.35 (13.6; 49.6–100.8)	65.1 (11.2; 53.3– 99.4)	.9892	.003

¹SFA: Standard deviation from average based on sex- and age- normative data

²Wilcoxon signed rank test

Reliability of the PANESS Timed Motor section summary scores across administration contexts are displayed in Table 3. ICCs for Total Timed, Total SFA, and Total Dysrhythmia were moderate to excellent whereas Total Timed Overflow ranged from poor to good. ICCs for PANESS Timed Motor Raw Speed Scores are presented in Table 4. Reliability for Repetitive Speed was moderate to excellent whereas ICCs for other raw speed scores ranged from good to excellent. Item-level analyses revealed strong cross-context reliability across items with excellent ICC ranges for several items (i.e., Foot Tapping, Heel-Toe Rocking, Finger Sequencing; see Table 3).

Table 3:

Cross-Context Reliability for PANESS Timed Motor Section Summary Scores

	ICC (95% CI)	F	p
Total Timed Overflow	.743 (.409 – .887)	3.778	< .001
Total Dysrhythmia	.789 (.527 – .907)	5.094	< .001
Total SFA	.873 (.715 – .944)	7.977	< .001
Total Timed1	.811 (.567 – .917)	5.123	< .001

¹Sum of Total Timed Overflow, Total Dysrhythmia, and Total SFA

Table 4:

Cross-Context Reliability for PANESS Timed Motor Section Raw Speed Scores – Repetitive and Sequential Tasks

	ICC (95% CI)	F	p
Repetitive Speed	.911 (.672 – .968)	16.32	< .001
Foot Tapping	.948 (.905 – .975)	20.555	< .001
Hand Patting	.802 (.631 – .904)	5.944	< .001
Finger Tapping	.941 (.891 – .972)	18.675	< .001
Sequential Speed	.921 (.806 – .966)	14.593	< .001
Heel-Toe Rocking	.971 (.947 – .986)	34.603	< .001
Hand Pronate/Supinate	.851 (.725 – .927)	7.415	< .001
Finger Sequencing	.957 (.922 – .979)	23.798	< .001
Repetitive + Sequential 1	.955 (.899 – .980)	21.592	< .001
Total Timed Motor Speed 2	.955 (.898 – .980)	21.466	< .001
Tongue	.899 (.771 – .955)	9.611	< .001

¹Sum of Repetitive Speed and Sequential Speed

²Sum of Repetitive + Sequential and Tongue item

Discussion

The present study is the first to demonstrate the feasibility of administering the Timed Motor section of the PANESS via videoconferencing. Using the procedures described, we were able to successfully administer this portion of the PANESS via videoconferencing in all cases where it was attempted, including in cases where the examinee was remotely located, at home. Importantly, findings suggest consistency between scores obtained in-person versus videoconferencing. Specifically, no statistically significant differences were observed between scores obtained in-person versus via telehealth and we observed moderate to excellent reliability between these scores. Of note, we observed the highest reliability point estimates (all ICCs exceeded .9) for the Raw Speed Scores, suggesting that these metrics may be most stable when altering the context of administration form in-person to via videoconferencing. We observed the lowest cross-contextual reliability for the Total Timed Overflow and Total Dysrhythmia summary scores, likely a result of the restricted score ranges for these metrics which capture the presence of neuromotor phenomenon observed at relatively low rate in this neurotypical sample of youth. Though not statistically significant, there was a small to moderate effect size for the group difference between Total Dysrhythmia summary scores obtained via in-person versus videoconference modalities, with ~1 point higher dysrhythmia scores (reflecting approximately one more instance of dysrhythmia) on average in the in-person modality. Future work should evaluate the extent to which this may be the product of an ordering effect (most participants were administered the measure in-person first in this study) versus other factors such as a slight decrease in the scorer’s ability to perceive dysrhythmia with videoconferencing.

The current study was not without limitations including the fact that there was some variability in the duration of time between in-person and videoconference PANESS administrations across participants and in the ordering of administration. Although most participants completed their in-person and videoconference PANESS evaluations on the same day and completed the in-person administration before the videoconference administration, there were exceptions to this due to scheduling limitations which may have introduced added variability. Despite this, ICC point estimates were all within acceptable ranges, suggesting that scores obtained in-person and via videoconferencing were stable overall, even with this added variability.

Importantly, the current study was conducted in typically developing youth and therefore results may not generalize to clinical populations. Follow up studies should be conducted in clinical populations^4,9 and in younger children (e.g., ages <7),² more likely to demonstrate a broader range of neuromotor phenomena on the PANESS. Further, while we conducted this study in research participants with prior exposure to PANESS to mitigate practice effects in our evaluation of cross-contextual reliability, this does limit our ability to speak to feasibility of telehealth administration in children who are naive to the PANESS (e.g., in a child being evaluated for an initial telehealth clinical visit who has not previously been exposed to the PANESS as part of their clinical care) which represents a significant study limitation. Finally, generalizability of findings was limited in that White youth were overrepresented (80%) in our sample compared to the national average (62%, US Census Bureau, 2020), participants on average had highly educated mothers, and data were collected from one (urban) geographic location within the United States. Future work in a larger and more heterogeneous sample should evaluate the role of sociodemographic and environmental factors such as education level and geographic setting (urban vs. rural) on the feasibility of telehealth administration of the PANESS.

Conclusion

This study represents an important first step in demonstrating the feasibility of administering the PANESS remotely via videoconference and to our knowledge is the first evaluation of the reliability of scores obtained in-person versus via videoconferencing methods. Our findings suggest that it may be feasible to administer the Timed Motor section of the PANESS as part of research protocols with remote data collection procedures or as part of telehealth neuropsychological and neurologic examinations. Importantly, our data suggest that the results obtained via telehealth should be consistent with those obtained in-person. Future work should seek to examine the feasibility of telehealth-based PANESS administration in clinical telehealth context and in a larger and more generalizable sample.