Clinician's Commentary on Chagas et al.

The PEDI-CAT is the outcome of almost two decades of effort to develop a more accurate and clinically useful measure of the performance of daily activities by children and youth with disabilities.¹ Although this development work provided a strong foundation for clinical and research use of the instrument, it was only the first step in what is ideally an ongoing process to evaluate how well the instrument does what it is supposed to do. Properties like reliability and validity are not fixed features of an instrument. Although careful design of the instrument plays a significant role in determining these characteristics, it is not possible to develop the “perfect instrument” that will be equally accurate and valid for all persons. Responses to items, and thus scores, are also influenced by variations in people and context, and we need information about the impact of these variations in order to make informed use of an instrument. This study provides evidence related to several important potential sources of variation.

First, the authors evaluate reliability in a population not previously studied, Down syndrome. As the authors note, much of the previous research with the PEDI-CAT has focused on children and youth with cerebral palsy or autism. The strengths and impairments of these three populations are quite different and therefore their impact on performance of daily activities is also likely to be different. The question addressed in the study is whether caregivers can reliably rate the performance of children and youth with Down syndrome when responding to the PEDI-CAT items. Ordinarily, the approach to examining this question would be a study where the caregiver responds to the PEDI-CAT on two separate occasions, usually a week or two apart. However, in this study, the caregiver responded to two different versions of the PEDI-CAT, one that yields a quick estimate of scores (Speedy CAT) and the other that administers more items across a range of content areas (Content-balanced CAT). Thus, what the results show is that these two versions of the instrument yield equivalent scores. While there is no good reason why the typical re-test design would yield different results, it is important to note that we do not know that from the current findings. Nevertheless, it is important for potential users of the instrument to know that they should obtain the same scores regardless of the option they choose. It is also good to know that the reliability estimates for the PEDI-CAT from this sample of children and youth with Down syndrome are highly similar to those obtained from other studies,^2,3 indicating that variation in clinical profile across populations does not appear to have a major impact on score reliability. This question should continue to be examined in other populations.

A second source of potential variation in the study was the remote method of administration. The PEDI-CAT was designed for the caregiver to complete the items remotely via computer or tablet. However, administration in this study was via Zoom or WhatsApp with caregivers in a different location from the evaluator, which was not the case in previous studies. The strong reliability estimates suggest that this type of variation in administration does not significantly affect consistency of responses. What we still do not know is the extent to which the presence of the evaluator (either in person or remote) affects caregivers’ responses compared to fully independent administration. For example, are caregivers more reflective or thoughtful in their answers if the evaluator is present? Do they take more time and are they more likely to complete all the four domains?

Finally, the study also provides evidence on whether reliability estimates remain as strong when an instrument developed in one culture and language has been translated to a different culture and language. Cultural variations in the form and materials of daily life activities may mean that some items are irrelevant or confusing; alternatively nuances in the language used to describe activities or the rating scale may not translate readily. The version of the PEDI-CAT used in this study had been carefully translated to Brazilian Portuguese as well as culturally adapted to ensure activities described in the items were culturally relevant.⁴ Re-test reliability estimates for the translated version were very strong (>0.90), but they were obtained from a sample of children without disabilities. Although estimates in the present study were obtained from a different design, they remained strong, especially the scaled scores, for this sample with Down syndrome.

Although the reliability estimate was strong, there was a small but statistically significant difference in scores for the Social/Cognitive domain. As the authors note, young people with DS tend to show more difficulties in this domain than in mobility or daily activities. However, it is useful to ask why these difficulties might affect score reliability, that is, what characteristics might make it more difficult for caregivers to rate their performance? One possibility is that half of the children in the sample were under 3 years of age, suggesting that many of them may be in the early stages of language acquisition. Skills tend to be more variable during the acquisition stage, which can make it harder for a caregiver to decide which performance rating is most appropriate. A second, overlapping possibility, is that the articulation difficulties commonly seen in Down syndrome interfere with effective communication,⁵ which again makes it difficult to assign a rating in this domain with confidence. As these examples illustrate, the PEDI-CAT scores by themselves only provide an indicator of whether or not a problem exists. Other information, including careful clinical observation, will be needed to understand why the problem is occurring.

The example just discussed illustrates an important point about the relevance of sample characteristics for interpreting results from psychometric studies like this one. A sample is intended to be representative of the population, thus we need to ask how good is that representation? One consideration in this study is that the sample is a bit small (n = 28): the smaller the sample, the less likely that the full range of variation that is typical for the population has been represented. As noted above, over half of the sample is under 3 and the mean age was 5.9 years, which means that the reliability results can most confidently be applied to younger children. Overall, the average of T-scores was in the lower end of the Average range, but with wide variation (SD > 10), which is similar to those reported on the PEDI in a study of 5.9-year-old Norwegian children with Down syndrome.⁶ Additional study with a greater proportion of older children and adolescents would be valuable, particularly to examine reliability for the Responsibility domain, which only had 14 respondents.

Accumulating evidence on the characteristics of the PEDI-CAT under varying conditions is essential for a tool that was designed to measure function in daily life. We cannot assume that the diverse profiles of strengths and limitations across clinical populations, cultural diversity in the activities themselves, and variations in environmental facilitators and barriers will have no impact on caregiver responses, nor can we always predict what kind of impact they may have. Studies like the one described in this Brief Report make an important contribution to building this important body of evidence.