Reliability and Acceptability to Caregivers of Telehealth Administration of the Pediatric Evaluation of Disability Inventory – Computer Adaptive Test (PEDI-CAT) for Brazilian Youth with Down Syndrome

Abstract

Purpose:

To estimate test-retest reliability of the two versions of the PEDI-CAT administered via telehealth to caregivers of Brazilian young people with DS, to compare scores on the two versions, and to determine caregiver acceptance of telehealth administration of the assessment.

Method:

A methodological study approved by the research ethics committee. Data collection was performed online, with a mean duration of 45.0 minutes for the content-balanced version of the PEDI-CAT and 17.5 minutes for the speedy version.

Results:

In total, 28 caregivers of individuals with DS up to age 21 years participated (mean = 5.9 years; SD = 4.9 years). Intra-class correlation coefficients for the four domains of the PEDI-CAT content-balanced version and four domains of the PEDI-CAT speedy version ranged from 0.77 to 0.97. There was a statistical difference between the versions in the scores of the social-cognitive domain (p < 0.05). A mean of 105 items (SD = 21) was administered in the content-balanced version and a mean of 51 items (SD = 8) in the speedy version. All the caregivers found the method of administration of the PEDI-CAT acceptable.

Conclusions:

This study demonstrated that either version of the Brazilian version of the PEDI-CAT can be used by telehealth in clinical practice to assess children, adolescents, and young adults with DS.

Individuals with DS often have impairments in mental and neuromuscular functions, with some individuals also experiencing impairments in cardiovascular and sensory system functions.^1,2 Children with DS tend to have delayed development of motor performance in self-care, and are more dependent on their parents.^3,4 In Brazil, a low-income country, there is a need for valid instruments to access functioning of children with DS. Information about involvement and participation in activities is relevant to aid planning of evidence-based interventions aligned with the goals of the children and their families.⁵

The Pediatric Evaluation of Disability Inventory (PEDI) is a valid and highly reliable measure to assess performance in activities and participation of children with different health conditions.⁶ The new version, the PEDI-CAT (Computer Adaptive Test) was recently translated and culturally adapted in Brazil;⁷ however, the acceptance by caregivers and psychometric properties of the new version have not yet been tested for Brazilian youths with DS.⁷ The questionnaire has two versions, a faster one (speedy version), and a longer one (content-balanced version), both with excellent psychometric properties in children with and without disabilities (i.e., cerebral palsy).^7,8 A study showing that both versions will yield reliable test-retest scores when administered remotely has not been performed in individuals with DS. Although it can be administered remotely, all studies that have used the PEDI-CAT in Brazil, including two validation studies in children with cerebral palsy, administered the questionnaire in person.^7–11

The onset of the COVID-19 pandemic has increased the need for assessments by telehealth.¹² To perform a pediat-ric assessment accurately by telehealth, it is necessary to use a quantitative measure that is valid, reliable, and easy to apply.¹³ The aim of this study was 1) to estimate test-retest reliability between scores of the two versions of the PEDI-CAT administered via telehealth to caregivers of Brazilian young people DS; 2) to compare scores of the two versions, and 3) to determine caregiver acceptance of telehealth administration of the assessment.

Method

Participant recruitment and consent

This methodological study was approved by the research ethics committee of the Universidade Federal de Juiz de Fora (UFJF) (CAAE: 09581119.1.0000.5133). The participants (DS and their caregivers) were recruited from the University Hospital of UFJF, and on social networks. Primary caregivers of children, adolescents, and young adults with DS (0-21 years) were invited to participate.

A sample size estimate of around 30 participants was based on the sample size estimation table by Walter and colleagues.^{1 4} Because this was a parameter estimation study, we removed the reference to the probability of a type II error from Walter’s formula and applied an expected ICC value of 0.80,¹⁵ a lower 1-sided 95% confidence limit value of 0.65, and two measured values. This sample size estimate is similar to previous reliability studies of children with motor disabilities.¹⁶

Instrument

The PEDI-CAT is a questionnaire that assesses performance in activities and participation of individuals between the ages of approximately 1–21 years.^6–9 The bank of 276 items is distributed into 4 domains: activities of daily living; mobility; social/cognitive; and responsibil-ity.^6–9 The questionnaire is administered using PEDI-CAT software for Windows or via Q-global web-based platform that requires Internet throughout the administration of the assessment, with images that facilitate understanding of the items, and can be completed through reports by caregivers, health professionals, or educators familiar with the child.^6–9 The responsibility domain should only be completed by caregivers of children over 3 years of age.^6–9

In the speedy version, approximately 15 items are administered per domain, and in the content-balanced version, approximately 30 items. Although the latter version has more items and provides more detailed information about the child’s skills, which may help plan individual rehabilitation programs, the speedy version takes approximately half the time to complete, perhaps making it more clinically feasible to administer.^6–9 Each version yields two scores: normative and scaled criterion-referenced. The normative scores, standardized using age percentiles and T-scores, based on the child’s chronological age, are intended to be used by clinicians to interpret the functioning of a particular child in comparison with others of the same age. T-scores between 30 and 70 represent + /- 2 SD and therefore are average for age.^6–9 The scaled criterion-referenced scores are based on the functioning profile of the child over time, are not related to age, and are measured on a scale from 20–80.^6–9 Higher scores indicate greater functioning ability.^6–9

Procedures

The researcher ( JC) was trained in the administration of the PEDI-CAT instrument, through reading and discussing the instrument manual with the advisor (PC), and by practicing the administration in three children that did not participate in this study. Recruitment of the participants was made through the university hospital and social media, and those that agreed to participate were selected as participants. Caregivers were contacted by telephone by JC and informed about the objectives and procedures of the study. After agreeing to participate, they were asked to read and sign the consent form. The caregivers then received an online questionnaire for socio-demographic data collection and the PEDI-CAT evaluation was scheduled.

In sequence, the PEDI-CAT was administered to the care-givers twice, with a fifteen-day interval. They were first given the content-balanced version, followed by the speedy version. All participants were tested in the same order. The caregivers completed the questionnaire online during an interview, using a video call on WhatsApp, turning the cell phone camera to the PEDI-CAT software screen, or in a Zoom meet with screen sharing. PEDI-CAT software was used in all evaluations (PEDI-CAT version 1.4.3, CREcare, LLC).

The caregivers received the report from the PEDI-CAT, with the results of the assessments as a thank you for volunteering their time to the study. At the end of the assessment, a Google forms link to a survey satisfaction questionnaire, prepared by the researcher, was sent to the caregivers requesting them to answer three yes or no questions: “Did you find the approach acceptable?”; “Did you find it a good way to apply the instrument?”; and “Would you participate in another survey in this format?” If they answered no to any of the questions, they were asked why.

Data analysis

The characterization of the participants (DS and caregiver) was performed through descriptive statistics, using measures of central tendency and dispersion (mean and standard deviation - SD) for quantitative variables, and frequency for categorical variables. To analyze the differences between the T-scores and scaled criterion-referenced scores between the two versions, a paired t-test was used. In addition, the administration time was computed to assess the difference between the two versions. To analyze the test-retest reliability between the two versions, Intraclass Correlation Coefficient (ICC) 2.1 were calculated, and the following criteria were used to describe the strength of relationships: ≤ 0.25 very weak correlation; 0.25–0.49 weak; 0.50–0.74 moderate; > 0.75 strong correlation.¹⁵ The statistical package SPSS for Windows (version 22.0, 2017) was used to perform the statistical analyses.

To analyze the caregivers’ opinions of the acceptability of the administration of the instrument by telehealth, the following data were analyzed: mean (SD) items that were administered, mean estimate of administration time, preferred mode of administration of the instrument (Zoom or WhatsApp), and responses to the satisfaction survey questions.

Results

Forty-eight caregivers were contacted, of whom 28 (58%) agreed to participate. The other caregivers contacted did not show interest in participating in the study. Most of the DS participants were male (n = 15; 54%), with a mean age of 5.9 years (SD = 4.9 years). The caregivers were mainly mothers (n = 27; 96%), over 40 years of age (n = 18; 64.3%), with a higher level of education (n = 16;57%).

There was a mean of 105 items (SD = 21) administered, with a mean duration of 42.5 (40–45) minutes for the content-balanced version, and a mean of 51 items (SD = 8), with a mean of 17.5 (15–20) minutes’ duration for the speedy version. The mean (SD) numbers of items tested per domain are displayed in Table 1. Twenty-seven (96.43%) caregivers chose the administration of the instrument by WhatsApp video call, and one caregiver opted for Zoom. All (100%) caregivers reported acceptability of the online administration of the PEDI-CAT and the explanation they received about it.

Table 1.

Number of Items Administered and Test-retest Reliability Between Scoresfor PEDI-CAT Content-Balanced and Speedy Versions

		Number of Items tested per domain		Content-balanced version score	Speedy version score	ICC 2,1	T-test between versions
		Mean (SD)					Mean (SD)
PEDI-CAT		C-B	S	Mean (SD)	Mean (SD)	(95% CI)	{p-value}
Scaled criterion-referenced scores	Daily activities	32.79 (3.21)	15.71 (1.44)	50.36 (6.46)	50.46 (6.45)	0.97 (0.94–0.99)	0.07 (1.63){0.819}
	Mobility	26.43 (7.05)	13.68 (2.16)	60.71 (7.50)	60.92 (7.58)	0.97 (0.94–0.99)	−0.21 (1.87){0.550}
	Social-cognitive	30.79 (1.20)	14.43 (2.03)	59.50 (4.86)	58.46 (5.64)	0.95 (0.80–0.98)	1.04 (1.45){0.001}
	Responsibility*	30.00 (0.00)	14.29 (2.16)	49.00 (6.21)	47.50 (6.74)	0.85 (0.60–0.95)	1.50 (3.41){0.124}
T-Score	Daily activities			37.46 (12.68)	37.25 (14.03)	0.97 (0.94–0.99)	0.21 (3.22){0.728}
	Mobility			36.67 (12.11)	37.17 (12.40)	0.88 (0.76–0.94)	−1.50 (5.91){0.191}
	Social-cognitive			37.25 (15.91)	35.67 (15.21)	0.97 (0.93–0.99)	1.57 (3.29){0.018}
	Responsibility*			41.78(8.55)	39.42 (9.30)	0.77 (0.44–0.92)	2.35 (5.85){0.156}

^* n = 14, only children over 3 years of age completed this domain; PEDI-CAT = Pediatric Evaluation of Disability Inventory — computer adaptive testing; SD = standard deviation; C-B = Content-balanced version; S = Speedy version; ICC = Intra-class correlation index.

Table 1 presents the results of the analysis of test-retest reliability (ICC) and differences between scores from the two versions. Point estimates of the ICC’s varied from 0.77 to 0.97, with the lowest value and greatest variability in the responsibility domain, indicating strong correlations between the two administrations of the measure and very good test-retest reliability. Scores did not vary between the two versions of the measure except for the scores in the social cognitive domain, which had a mean difference of approximately 1 point. Because half of the children were under 3 years of age, only 50% (14 caregivers) completed the responsibility domain.

Discussion

This study demonstrated that both versions of the PEDI-CAT had very good test-retest reliability when administered by telehealth, and the caregivers of Brazilian youths with DS found this method acceptable. The long version took twice as long to apply, and the scores of the social-cognitive domain showed a statistical difference between versions.

The two versions of the PEDI-CAT were developed based on the same database of items.⁸ The manual reports that the content-balanced version presents excellent discriminant validity between children with and without disabilities (p < 0.001) and strong test-retest reliability (> 0.95). The speedy version can be used as an accurate measure of functioning in clinical outcome measurement and clinical trials, reducing the burden typically placed on both caregivers’ respondents and research protocols when full item banks are administered.⁸ To date, no studies have compared scores on the two versions using the online Brazilian version in young people with DS, which makes it difficult to compare our results with other studies.

The statistical difference observed in the scores of the social-cognitive domain between versions may have been related to the population included in this study. Children and youths with DS tend to demonstrate more concerns in the social-cognitive domain over the years than in the mobility and daily activities domains.^{1 2} Due to the lower number of items administered when applying the speedy version, our results suggest it is probably better to use the content-balanced version of the PEDI-CAT when social-cognitive issues are a concern in the evaluation of children and youths with DS. This way, a better planning of the intervention can be made according to the limitations presented.

In the current study, the administration time was more than twice as long for the content-balanced version compared to the speedy version. The study of Dumas and col-leagues^{1 7} compared the PEDI-CAT administration time in parents of children with and without disabilities; the mean times to complete the speedy version were 11.85 minutes and 14.61 minutes, with no significant difference between the two groups. This result is in line with the administration time of the speedy version in the current study (15–20 minutes). Regarding the administration time of the content-balanced version, no studies were found with this population or with other children with disabilities.

Caregivers of Brazilian young people with DS were positive about the administration of the PEDI-CAT by tele-health. This is probably due to the ease of administration of the instrument via WhatsApp calling, which is widely used in Brazil,¹⁸ and makes it possible to contact people from different locations in the country and with different levels of socioeconomic status. Although, this group had access to the Internet and a phone/computer. In addition, during the assessment, it was possible to show the software screen to the caregivers and answer questions using this mode. Another point that deserves attention is that most mothers in this study had a high level of education, which may have facilitated understanding of the remote administration of this assessment instrument.

Telerehabilitation has grown recently, as it enables physiotherapists to interact with patients at a distance through information and communication technologies to provide rehabilitation services.¹⁹ In addition, it may be particularly suitable for implementing best practices for children with disabilities when the focus of therapies is to support them and their families.²⁰

It is recommended that therapists use outcome measures with valid psychometric properties to document a child’s functional status and progress, such as the PEDI-CAT.^5,12 Considering the results of this study and the evidence in the literature, it is possible to apply the PEDI-CAT versions by telehealth both in the context of research and in clinical practice. The content-balanced version is more complete and is recommended, especially when needing to evaluate the child’s social skills. Conversely, the speedy version is also valid, reliable, and faster to apply.

Limitations

One limitation is that only families that had Internet access were included in this study. Although this could have left out families that don’t have access to this technology, this measurement was developed to be used this way. Another limitation is that almost half of the families that we contacted didn’t show interest in participating in our study. They didn’t provide the researchers the reasons, but we know that the number of studies performed online during the pandemic of the COVID-19 increased a lot, probably turning difficult to manage all the invitations. A third limitation is that the responsibility domain was only relevant for 14 children with DS of our study. And maybe our results are underpowered and should be interpreted with caution.

Conclusion

Both versions of the PEDI-CAT showed very good test-retest reliability, and the scores between the two versions were not statistically different except for the social-cognitive domain. Also, the PEDI-CAT showed good acceptance by caregivers when administered by tele-health and can be used to assess functioning in clinical practice and in studies with Brazilian children, adolescents, and young adults with DS.