Initial validation and reliability of the CDKL5 Deficiency Disorder Hand Function Scale (CDD-Hand)

Abstract

Pathogenic variants in the CDKL5 gene result in CDKL5 Deficiency Disorder (CDD), which is characterised by early-onset epilepsy, severe developmental delay and, often, cortical visual impairment (CVI). Validated clinical outcome measures are needed for future clinical trials to be successful. This study aimed to adapt the Rett Syndrome Hand Function Scale (RSHFS) for CDD and evaluate its feasibility, acceptability, content validity and reliability. Consultation with a CVI experienced specialist and the Consumer Reference Group informed modifications to the instructions of the RSHFS for children with CDD (CDD-Hand). Eighty-six families registered with the International CDKL5 Disorder Database provided video clips of their child’s hand function and provided feedback about the measure. Video data were coded by two researchers to evaluate intra- and inter-rater reliability. This study provides initial evidence of validation and reliability. The scale appears to be suitable for a range of ages and functional abilities for CDD.

Introduction

CDKL5 Deficiency Disorder (CDD) is a rare genetic condition that is classified as a developmental and epileptic encephalopathy (DEE). Clinical characteristics of CDD include early-onset seizures generally occurring within the first 3 months of life, and severe neuro-developmental impairments [1 2]. Other comorbidities associated with CDD include cortical visual impairment (CVI), breathing abnormalities, altered muscle tone and poor sleep [2–4].

Functional abilities in CDD are severely limited in the domains of gross motor, hand function and communication although there is some variability. Difficulties with hand function and the inability to grasp objects can impact the child’s independence by rendering daily life activities, such as self-feeding, difficult or impossible to perform. For hand function, data from the International CDKL5 Disorder Database (ICDD) has indicated that approximately one third of 103 females were not able to grasp objects, slightly fewer (about a quarter) could hold or pick up large objects, and a slightly greater proportion (nearly 40%) could pick up large and small objects [5]. In 16 males in the same study, the majority (n=12) could not grasp objects, whereas two could grasp and pick up large objects and two could pick up small objects as well [5]. The true variability in patterns of functioning may only be identified in adequately sized studies using more precise methods of measurement. For example, a much smaller sample that used a blunter measure (functional or non-functional hand use) reported that 21 of 28 individuals with CDD had no functional hand use, painting a starker picture [6].

Prior to recognition of CDD as an independent clinical entity with unique characteristics [1], such as CVI [3 4], many individuals with CDD were considered to have an atypical form of Rett syndrome (RTT) [7 8]. Despite the overall more severe phenotype, hand grasping in CDD has been reported to have some similarities with RTT, with two thirds of individuals demonstrating some purposeful hand function skills in both disorders [5 9]. We previously developed a protocol to demonstrate hand function in RTT [10] based on the Hand Apraxia Scale [11]. Videos (n=144) were collected from parent caregivers of individuals with RTT and the hand function skills were classified into eight levels to form the Rett Syndrome Hand Function Scale (RSHFS) [9]. The RSHFS demonstrated good construct validity with expected relationships between genotype and clinical severity in the RTT population [9]. However, there are currently no validated hand function outcome measures for CDD, or any DEE that considers CVI, which would require strategies to optimise capacity for the individual to demonstrate hand function irrespective of visual function.

Because of similarities between hand function abilities in individuals with CDD and RTT, the RSHFS could also be appropriate to measure the range of hand function in CDD. In the first instance, establishing feasibility, acceptability and reliability form part of the psychometric foundation when developing outcome measures, and is especially important for evaluation of measures to be used in clinical trials [12]. Feasibility considers the extent to which the outcome measure can be successfully implemented within existing contexts (e.g. time taken to complete) [13]. Acceptability is the perception that the measure is satisfactory, based on whether respondent needs, preferences, or expectations are met [13]. Adequate assessment of content validity provides evidence that the conceptual framework and content of items are consistent with the perspective of the population of interest [14]. Intra-rater reliability refers to the consistency of the data recorded by one rater over several trials whereas inter-rater reliability refers to the consistency of data recorded by two or more raters. Evaluation of rater reliabilities assist in establishing confidence that observed changes are not due to rater inconsistencies [15].

Clinical trials that will evaluate novel therapeutics for CDD and capabilities for disease modifying therapies, such as gene therapy, are on the horizon. However, a potential barrier to successful clinical trial protocols for CDD is the lack of validated outcome measures capable of demonstrating meaningful interventional changes [16]. Our aims in the current study were to adapt and pilot test the RSHFS for individuals with CDD and evaluate its feasibility, acceptability, content validity and rater-reliability.

Patients and Methods

Participants

The ICDD was established in 2012 to collect natural history information from families of a child with CDD [17]. Families complete a baseline questionnaire on initial registration which is ongoing, and, in 2019, a follow up questionnaire was administered to 172 families who had already competed a baseline questionnaire [18]. For the current study, parents or caregivers who had previously contributed to the ICDD and whose child had a CDKL5 variant considered to be pathogenic or likely pathogenic were invited to participate, commencing July 2020. Families who agreed to participate were emailed a detailed hand function protocol (Appendix 1). Ethics approval for this study was provided by the Human Research Ethics Committee at The University of Western Australia (RA/4/20/6198) and primary caregivers provided informed written consent to participate.

Measurement of hand function

Development and administration:

The CDD Hand Function Scale (CDD-Hand) protocol for describing hand function was based on the RSHFS [9]. To account for the high prevalence of CVI in CDD [3], additional instructions were created through consultations with a specialist experienced in evaluating CVI in children with severe disability. These modifications included use of strategies such as placing objects on a black background or ensuring the object was visible in the child’s visual field, to ensure available vision was used. Feedback was also sought from an ICDD Consumer Reference Group (n=6), and the CDD-Hand was piloted with two caregivers (child age: 1.7y and 11.5y), whereby minor changes to the instructions were made (Appendix 1).

The protocol for the CDD-Hand is shown in Appendix 1. Parent caregivers were asked to present their child a selection of large objects (e.g. toy, small ball, cup, utensil) and small objects (e.g. small pieces of food). They then encouraged their child to grasp, pick up and hold each of the objects, providing assistance as needed. Parent caregivers were asked to present objects to the child’s dominant and then non-dominant hand.

Upload of videos:

Caregivers were provided with a ShareFile link whereby they could upload video clips to a secure online folder housed on a secure server at Telethon Kids Institute. Once video clips were uploaded to the ShareFile folder, a research team member, where necessary, identified any missing tasks and requested further video clips from the parent.

Feasibility, acceptability, and content validity:

A parent survey to evaluate feasibility, acceptability and content validity was administered through REDCap™ to parent caregivers who had provided video. Items queried the clarity of instructions, access to required equipment, assistance, time to completion and facilitators and barriers of the process to evaluate feasibility. To evaluate acceptability, items explored the parent caregiver’s view on using video to demonstrate hand function and whether they felt it could be a suitable method of measurement in a clinical trial. To assess content validity, the survey explored the suitability of tasks, missing grasping activities, grasping importance to overall functioning and the additional CVI instructions. Parents who were not able to complete hand function videos were also sent the survey for them to identify the reason they could not complete the video (e.g., lack of time, instructions too complicated) and whether they would be willing to provide video of their child’s hand function in a clinical trial setting.

Coding of hand function:

One rater experienced in the coding for the RSHFS (JD) provided training to the second rater (JS) using a bank of 54 video clips collected from previous studies for RTT [9 19]. Both raters then scored the hand function of the first 54 videos showing hand function in CDD on two occasions (four weeks apart).

As for the scoring for the RSHFS, the best grasping ability demonstrated on the videoclips was coded to one of eight levels. The elements of grasping, picking up, and holding the large and small objects were coded as able to do independently, with assistance or unable. Grasping a small object was coded as using a raking or radial grasp. A radial grasp involved using the thumb and included scissor, inferior pincer, and superior pincer grasps. Pre-shaping skills of the hand were observed as the hand approached each object, with hand orientation and size recognition coded as closely approximating the object or not. The ability to transfer an object from one hand to another was coded as ‘present’ or ‘absent’.

Statistical Analysis

Descriptive statistics were used to characterize the features of the individuals with CDD and to describe the distributions of parent caregiver ratings of feasibility and acceptability. Inter-rater reliability compared the scoring of two independent raters on recorded hand function videos. Intra-rater reliability compared the scores of the same two raters with their own scores on the same videos after a 4-week interval. Chance-corrected agreements were calculated using Cohen’s Weighted Kappa statistic [20]. Kappa coefficients above 0.8 were interpreted as excellent agreement, 0.6–0.8 as substantial, 0.4–0.6 as moderate and <0.4 as poor [17].

Results

Study sample

We contacted 156 caregivers of children with CDD who agreed to receive the study information. Of those, 108 (69.2%) provided electronic consent to participate and 86 (79.6%) returned a complete set of videos of hand function. Our overall completion fraction from initial contact was 55.1%. More than half of the participants were from the United States (60.5%) and a smaller percentage were from Australia (12.8%), Germany (7.0%), Canada (3.5%), United Kingdom (2.3%), Russia (7%) or other (Finland, Ireland, Italy, Norway, Ukraine; United Arab Emirates, 7%) (Table 1). The mean age of the individuals with CDD was 9 y (SD 6.7y; range 1.1–28.8) and 82.6% were female (Table 1). Data from the most recently completed ICDD questionnaire confirmed that 22 (25.6%) walked independently, 15 (17.4%) communicated with words, and the majority (67.4%) had at least daily seizures (Table 1). Ten caregivers (6.4%) chose not to participate and their child was on average 15.3 y (SD 8.1; range 1.6–22.2). The minority of children in this group were able to walk independently (n=1) and communicate with words (n=1).

Table 1:

Participant characteristics and level of hand function by mutation group, gender, age, seizure frequency, walking ability and communication (n=86)

	N (%)	Level of hand function Median(range)
Mutation group
No functional protein	24 (27.9)	3.5 (1–8)
Missense/in-frame within the catalytic domain	20 (23.3)	4 (1–7)
Truncations between aa172 and aa781a	26 (30.2)	4 (1–8)
Truncations after aa781a	12 (14.0)	3 (1–8)
Not grouped	4 (4.7)	3.5 (2–4)
Gender
Male	15 (17.4)	1 (1–8)
Female	71 (82.6)	4 (1–8)
Age, y
<8	44 (51.2)	3.5 (1–8)
8-<13	19 (22.1)	2 (1–7)
13-<19	16 (18.6)	2.5 (1–8)
>19	7 (8.1)	4 (2–8)
Seizure frequency
≥5 daily	26 (30.2)	2 (1–8)
1–4 daily	32 (37.2)	4 (1–7)
Weekly	17 (19.8)	5 (1–8)
Monthly	6 (7.0)	5.5 (1–7)
None	5 (5.8)	4 (2–8)
Walking ability
Unable	52 (60.5)	2 (1–7)
Moderate assistance	9 (10.5)	4 (2–7)
Minimal assistance	3 (3.5)	4 (2–7)
No assistance	22 (25.6)	7 (3–8)
Communication
Simple/no communication	17 (19.8)	3 (1–7)
Gestures, signs or vocalisations	54 (62.8)	3 (1–7)
Words	15 (17.4)	5 (1–8)

Levels of hand function

The distribution of hand function levels was right skewed (Figure 1). Half had no or poor hand function (level 1, 2 and 3), nearly one third (30.3%) could independently grasp large or small objects (level 4, 5 or 6) and 19.8% had additional hand function skills (level 7 or 8). Table 1 illustrates the levels of hand function for gender, age, mutation group, walking ability and communication level.

Figure 1:

Distribution of hand function scores (n=86)

Feasibility, acceptability and content validity

The time to submit completed video from the time of initial contact was a median of 76 days (range 3–350 days). Sixty-two (72.1%) parents who completed the hand function videos also completed a hand function feedback form which included questions on feasibility, acceptability and content validity (Table 2). Time (minutes) taken to film the hand function tasks was an average of 15.1 min (SD+−12.9, range 1–60, median 15). The majority found the tasks clear and easy to understand (95.2%), with nearly one third (30.6%) of respondents able to complete video collection without an assistant. Most (85.5%) were comfortable using video to show their child’s hand function and nearly all (96.8%) expressed willingness to provide hand function video in a clinical trial. Further, 74.2% felt the activities were suitable to show their child’s level of grasping ability and 83.9% indicated that hand function was an important area to their child’s overall functioning.

Table 2:

Feasibility, acceptability and content validity feedback items reported by parents/caregiver (n=62)

Item		Positive n(%)	Neutral n(%)	Negative n(%)
Acceptability
Comfortable showing child’s hand function using video		53 (85.5)	7 (11.3)	2 (3.2)
Willing to provide video of child’s hand function in a clinical trial		60 (96.8)	1 (1.6)	1 (1.6)
Feasibility
Clear and easy to understand instructions		59 (95.2)	1 (1.6)	1 (1.6)
Easy to organise the room and objects for the video		52 (83.9)	7 (11.3)	3 (4.8)
Easy to access equipment to video hand function		56 (90.3)	3 (4.8)	3 (4.8)
Ability to upload video		54 (87.1)	2 (3.2)	6 (9.7)
		Yes n(%)	Sometimes n(%)	No n(%)
A second person present to assist		33 (53.2)	10 (16.1)	19 (30.6)
Content Validity		Positive n(%)	Neutral n(%)	Negative n(%)
Activities suitable for child to show grasping ability		46 (74.2)	13 (21.0)	3 (4.8)
Any missing grasping activities		52 (83.9)	-	10 (16.1)*
Importance of hand function to overall functioning		52 (83.9)	7 (11.3)	3 (4.8)
	N/A n(%)	Positive n(%)	Neutral n(%)	Negative n(%)
Cortical visual impairment suggestions	17 (27.4)	29 (46.8)	15 (24.2)	1 (1.6)

^*Functional movements, dressing, cutting and colouring, non-applicable answers

Feedback was also collected from 10 parents who were unable to complete the videos. Parents were able to provide more than one answer about the inability to complete hand function video at this time, which included: lack of time (n=4), the child’s clinical severity and co-occurring health issues (n=3), personal circumstances (n=2), lack of help (n=1) and privacy concerns (n=1). Two responders were willing to provide video for evaluation in a clinical trial whereas three would be reluctant. Although not withdrawn from the study, some parents (n=14) have noted the difficulties which have delayed the completion of their hand function videos. They reported several difficulties including: lack of time (n=5), the severity of their child’s impairments and co-occurring health issues (n=5), personal circumstances (n=3) and lack of help (n=3).

Reliability

Inter-rater reliability for 54 videos of unique individuals showed excellent agreement between the two raters; Kappa (95%CI) = 0.895 (0.836 – 0.947). Intra-rater reliability also showed excellent agreement for both raters with Kappa (95%CI) statistics of 0.966 (0.926 – 0.992) and 0.949 (0.911 – 0.985).

Discussion

Validated outcome measures are essential for accurate evaluation and interpretation of responses to treatment in clinical trials. This study sought to adapt and pilot the RSHFS for individuals with CDD and evaluate its feasibility, acceptability, content validity and reliability. The CDD-Hand is a simple observational measure of object-related hand function which allows collection of hand function data using everyday materials. The CDD-Hand appeared to be suitable for a range of ages and functional abilities in CDD and reported promising results for the initial validation and reliability. This is a necessary step towards outcome measurement development. As the potential for new therapeutics such as gene therapy is accelerating, the validation of outcome measures that are important to caregivers [21] and relate to child functioning [16] is required for successful clinical trials.

The CDD-Hand was adapted from the previously developed RSHFS, an eight-level hand function scale adapted from the Hand Apraxia Scale and derived from cross sectional video footage of 144 individuals with RTT [9]. In the current study, a therapist with extensive experience in evaluating children with CVI was consulted in the modifications for children with CDD. Adaptations were made primarily to the administration instructions due to the high prevalence of CVI in the CDD population [3 4]. Furthermore, the video protocol allows for video to be completed at home. Home-filmed video provides additional perspectives on the functioning and behaviour of children with CDD as familiar settings have the potential to demonstrate greater detail as well as reduce participant stress [22]. This is potentially beneficial as the child’s typical everyday behaviours and movements, which can sometimes be missed in clinic assessments, can be captured. Also, video can be collected from families from all geographical locations and can be completed in the safety of their own home, which is important during the COVID-19 pandemic.

There is a lack of validated hand function outcome measures available for use with the CDD population. We based CDD hand on the RSHFS which enabled a finer level of evaluation compared to previous broader categorical scales. CDD-Hand incorporates data on three of the six distinct categories in hand function, described by Chien et al. [23]. These include adaptive skilled hand use object-related, bimanual use object-related hand skills and general quality of the hand skills. CDD-Hand does not evaluate manual gestures, body contact hand skills and object-related hand skills involving arm-hand use. Rather it focuses on specific functional hand skills that can be readily videoed. Other scales of early hand function skills, such as the Peabody Developmental Motor Scale (PDFMS-2), require cognitive engagement as well as visual motor integration, which makes the measure less suitable for use in individuals with severe cognitive impairments and CVI. Assessment using the CDD-Hand was feasible within the context of severe intellectual disability and CVI. The 8-point scale provides greater characterization of hand function than previously used Likert scales and focuses on current function, which may be responsive to change with therapeutic interventions.

In this CDD sample, the highest proportion of children demonstrated no grasping function (23.3%), however; we also captured partial skills of grasping objects such as holding (15.1%) or grasping with assistance (11.6%). Nearly 20% (19.8%) could pick up large object independently and a smaller proportion could pick up small items using a raking grasp (5.8%) or radial grasp (4.7%). A slightly larger proportion (15.1%) were able to transfer and object from one hand to another and a smaller portion (4.7%) could additionally pick up small items with precision. A floor effect is often observed with severe disorders and, although the distribution of hand function scores is somewhat right skewed, CDD-Hand offers granularity within the lower levels of grasping. Except for the most severely affected children who showed no grasping function, the scale was able to discriminate between children with early rudimentary skills. This discrimination minimised the floor effect that would have been observed with less granular rating scales. The distribution of scores was broadly comparable to what we have previously observed in RTT [9], another severe DEE.

Some parents found the instructions and filming protocol manageable even when only one parent was available to collect data, though it appeared to be generally easier with two adults. Some feedback indicated that additional support was valuable when the child was more severely affected. However, the use of a phone tripod or stand helped some caregivers to conduct the protocol without assistance. Acceptability of CDD-Hand was strong with a majority of participating parents comfortable showing their child’s hand function using video and willing to provide video of their child’s hand function in a clinical trial. In terms of content validity, the addition of instructions specific to CVI were considered helpful from 64.4% of parents of a child with CVI and no additional suggestions were made by parents. Further, a majority found the activities suitable to show their child’s skills and felt that their child’s hand function was an important part of their overall functioning. Finally, the Kappa values indicate excellent intra and inter-rater reliability. Excellent rater reliabilities are consistent with a previous study using the measure in the RTT population [9] and will assist in confidence in the changes occurring in scores [15].

Strengths and limitations

Grounded in a scale validated for RTT, a strength of our study was the thorough preliminary evaluation of the instrument, including consultation with a CVI experienced specialist and piloting with the Consumer Reference Group, which enabled suitable adaptations. Further, the ICDD is the only global CDD database developed specifically to collect comprehensive information on this disorder. Recruiting families registered with the ICDD allowed for a range of clinical presentations to be represented in the study. This observational study adds to the understanding of hand function in children with CDD and is the first to collect video data of hand function abilities (information that has previously been collected through questionnaire data). The scale could be used in clinical settings to rate direct observations, but our study paves the way for caregivers to provide video of their child’s hand function, in their familiar setting, to their clinicians on an ongoing basis as a part of their ongoing clinical relationship.

This was an observational study and response fractions were perhaps lower than what one would expect to be achieved in a clinical trial, where there is potential benefit to be derived from the intervention. We acknowledge that there is some burden for families to complete this measure, which is shown by the time to return and participation rate from recruitment. Phenotype data describing other functional abilities and frequency of seizures were collected between 2018 and 2022, prior to the video collection. We note that small changes in functional abilities can occur with time in CDD and that seizures typically remain refractory [18]. Additionally, time to return would likely be improved and we think would be improved even further with repeat testing as families get even more comfortable with the measure. This theory will be assessed in future work. Further, the CDD-Hand protocol is feasible as conventional technologies reduce the burden of filming and uploading videos. Additional data is needed to further validate CDD-Hand and this is ongoing.

Conclusion

The lack of validated outcome measures for CDD creates a critical gap in clinical trial readiness for this condition. There is urgency in addressing this gap with the prospect of gene therapy on the horizon. The CDD-Hand offers an easy and practical collection of hand function footage for assessment. Active participation by caregivers in video-based research may also have empowering outcomes by facilitating further opportunities for involvement in their child’s intervention and therapy programs [24]. The CDD-Hand allows for data to be collected from all geographical locations, which is beneficial for rare disorders and even for common disorders during extraordinary conditions, such as the COVID-19 pandemic. Additionally, use of remote video assessments will allow research to reach more individuals ultimately improving care for all. Further validation studies are required for the measure to be considered suitable by the FDA for clinical trial use, including more extensive evaluation of validity and responsiveness to change.