Content validation of clinician-reported items for a severity measure for CDKL5 Deficiency Disorder

Abstract

CDKL5 deficiency disorder (CDD) results in early onset seizures and severe developmental impairments. A CDD clinical severity assessment (CCSA) was previously developed with clinician and parent-report items to capture information on a range of domains. Consistent with FDA guidelines, content validation is the first step in evaluating the psychometric properties of an outcome measure. The aim of this study was to validate the content of the clinician-reported items in the CCSA (CCSA-Clinician). Eight neurologists leading the USA CDD Center of Excellence clinics were interviewed using the ‘think aloud’ technique to critique 26 clinician-reported items. Common themes were aggregated and a literature search of related assessments informed item modifications. The clinicians then participated in two consensus meetings to review themes and finalise the items. A consensus was achieved for the content of the CCSA-Clinician. Eight of the original items were omitted, eleven items were added, and the remaining 18 items were revised. The final 29 items were classified into two domains: functioning and neurological impairments. This study enabled refinement of the CCSA-Clinician and provided evidence for its content validity. This preliminary validation is essential before field testing and further validation, in order to advance the instrument towards clinical trial readiness.

Introduction

Cyclin-dependent kinase-like 5 (CDKL5) deficiency disorder (CDD) is a rare developmental epileptic encephalopathy resulting from pathogenic variants in the CDKL5 gene [1]. The median age of epilepsy onset for CDD patients is six weeks, with onset by three months in most children [2]. CDD is associated with significant neurodevelopmental cognitive and fine and gross motor impairments and other comorbidities such as respiratory dysfunction, disrupted sleep, and gastrointestinal issues [3 4]. Cortical visual impairment affects >75% of patients, with poor eye contact and impaired visual tracking [5].

Current treatments do not resolve symptoms for CDD and epilepsy responds poorly to antiepileptic medications [6 7]. Efforts are underway to develop therapeutic strategies to treat CDD, including gene therapy. However, prior to assessing disease modifying treatment, including gene therapy, clinical outcome measures for CDD need to be formalised that are capable of demonstrating interventional changes using objective measures and how a patient feels and functions across the spectrum of severity [8]. Therefore, while capability for disease modifying therapies is accelerating, a critical barrier for clinical trial readiness is the lack of disease specific validated outcome measures, which may result in failure of these therapies.

A severity assessment specific to CDD was developed based on the clinical and research experience of an international expert panel, including clinicians from the International Foundation for CDKL5 Research Centers of Excellence (COE) consortium and the National Institutes of Health Rett and Rett-Related Disorders Natural History Study (U54 HD061222; ClinicalTrials.gov: NCT00299312/NCT02738281) consortium [9]. The items in the CDD Clinical severity assessment (CCSA) span neurological impairments, functional abilities and comorbidities of CDD. With further development and validation, this instrument is intended to assess the outcomes of interventions in clinical trials. The assessment comprised 53 items, 27 reported by parents (CCSA-Parent) and 26 reported by clinicians (CCSA-Clinician). The CCSA was evaluated through four cycles of anonymous modified Delphi comment and an initial consensus was provided by an international panel of clinicians, researchers, industry, patient advisory groups and parents of a child with CDD [9].

Establishing content validity for new clinical outcome measures is central to scientifically sound instrument development processes [10]. Adequate assessment of content validity provides evidence that the conceptual framework, content of items and overall measurement approach are consistent with the perspective of the population of interest and is necessary to meet FDA requirements for the development of outcome measures [8 11]. Content validation of the CCSA is fundamental to validate activities and provides an essential foundation before testing reliability, convergent and divergent validity, and responsiveness to change.

The CCSA-Clinician comprises 26 items that describe domains of motor, cognition, behaviour, vision, speech and autonomic function. For this study, we evaluated the CCSA-Clinician for content validity. In a separate study the CCSA-Parent will follow a similar content validation process with parents of CDD children and will be revised to be complimentary to the design and structure of the CCSA-Clinician.

Materials and Methods

Participants

Eight US COE child neurologists with a median of 10 (range=4–29) years’ experience, six of whom participated in the initial CCSA development process, comprised the expert panel. The COEs are located in eight children’s hospitals funded by the International Foundation of CDKL5 Research, are investigators in the International CDKL5 Clinical Research Network, and provide specialist clinical and research follow-up to children with CDD. Each COE clinician had used the CCSA with their CDD patients (median=20, range=10–70) and were invited to participate individually in an interview to evaluate the clinician portion of the instrument. Two group meetings were then held to discuss findings and develop a consensus of the modified instrument. Ethical approval was granted by the Human Research Ethics Office at the University of Western Australia (RA/4/20/6198) and the University of Colorado (COMIRB19-2756) and all participants provided verbal consent for their involvement in the study.

Content validation processes

The content validation processes were guided by the development processes for outcome measures described by Brod et al.[10] and Epstein et al.[12]; see Figure 1.

Figure 1:

CDKL5 Deficiency Disorder (CDD) Clinician Clinical Severity Assessment (CCSA-Clinician) development process including preliminary concepts, content validation and further psychometric testing towards a final measure suitable for clinical trials. The parent reported CCSA (CCSA-Parent) will undergo the same process and combine with the CCSA-Clinician for use in clinical evaluation and trials.

Part 1: ‘Think Aloud’ Interviews

‘Think aloud’ is a semi-structured cognitive interviewing method that assesses unique higher-level thinking processes and identifies individual differences in thought [13 14]. Prepared with patient examples of different severities, the clinicians were asked by two interviewers (JD, JS) to share their thoughts out loud by verbalizing any words or ideas that came to mind as they completed the assessment [15]. The think aloud interview schedule included probing questions for each item, including: “Why did you give that rating?” “How would you restate this question in your own words?” “Was there anything that you found confusing or did not understand?” The interviewers guided the discussion more proactively by asking additional, direct questions about the basis for the responses. Salient wording in the items was also selected and further probed. For example, the interviewers asked, “What does ‘some trouble’ mean to you in this response category?”

Part 2: Thematic analysis

All interviews were audio recorded and transcribed verbatim by a research team member (JS). Thematic analysis was used to analyse the interviews. Beginning with a within-case analysis, transcribed data pertaining to each clinician for each assessment item were combined and key phrases or statements relevant to item interpretation were extracted. Second, common themes were explored and aggregated across interviews. Key findings for each item were then refined and operationalised into summary statements for each theme and differences between the intended interpretation and that of respondents were identified [16]. Suggestions of added, omitted or revised items based on emerging trends were compiled and reviewed by the investigative team (JD, JS) to establish an iterative set of statements. A literature search for related items in other questionnaires was also conducted to inform item modifications, particularly regarding the areas of movement disorders [17–20] and attention [19 21 22], and a neuro-ophthalmologist (GH) was consulted in regards to the vision and eye movement items.

Part 3: Consensus methods

All participating clinicians took part in two one-hour online consensus meetings, led by an experienced facilitator (JD), and were prepared with the suggested item modifications prior to the first meeting. Each of the items and suggested modifications were discussed systematically and discussion continued until a consensus was reached. The clinicians were then provided with opportunity, by email, to make further comments or suggestion.

Results

The median duration of the individual interviews was 1 hr 43 min (range 1 hr 19 min to 2 hr 51 min, total 14.6 hr). The original CCSA-Clinician comprised 26 items that describe domains of motor (13), cognition, behaviour, vision, speech (12) and autonomic function (1). After reviewing all interview data, we proposed that (1) eleven items be added; (2) eight items be removed; and (3) revisions be made to the remaining 18 items. Also, the questionnaire construct was revised, including reporting the child’s state, weighting of motor function items, and scoring. The revised CCSA-Clinician comprised 29 items, devised into dimensions of functional abilities (n=13; items for gross and hand function, communication, vision and attention) and neurological impairment (n=16; items for abnormal eye alignment and movement, altered muscle tone, dystonia, dyskinesias, stereotypies, autonomic function and behaviour); see appendix 1.

Addition of Items

The team added items to develop additional granularity of measurement and ensure items were restricted to a single concept (Table 1). For example, additional items (head control and supine to sit) were added to allow the broader gradation needed for younger children and to allow greater granularity in scoring for more severely affected children. There was consensus that age was a limiting factor when administering the original gross motor function items (sitting, standing, walking) as these items are not tasks that are expected to be completed by all ages. Age ranges were allocated for the gross motor items.

Table 1:

Example of changes made to original items based on commonly endorsed clinician responses during cognitive interviews and consensus meetings

	Original item	Commonly endorsed responses	New item
Addition	Stands 0 Stands normally (including: goes from sit to stand) 1 Stands, but some trouble, > 20s 2 Stands 10–20s only 3 Stands < 3s (no assistance) 4 Stands only with assistance >3s 5 Not standing (less than 3s with assistance)	Level of assistance needs to be defined ‘Sit to stand’ is a different concept to standing, it is a confounder in this item. 20 sec is reasonable, easy and efficient Interpretation of ‘some trouble’ A 6-month-old would not be expected to stand, even if typically developing	Sit on chair to stand- extent of testing based on parent report of known motor skills, eg, if unable to get up from floor, start with getting up from chair 0 – moves from sitting on the floor to stand independently, without using hands and stabilising independently 1 – moves from sitting on the floor to stand but uses own hand(s) for support on self or furniture 2 – cannot stand from floor, but stands from chair on own (may use own hands to get out of chair but independently) 3 – stands from chair with minimal assist (one hand, one hand from assistant, light touch) 4 – stands from chair with moderate assist (two hands, trunk support) 5 – Maximal assist, unable to stand from chair
Removal	Curvature and scoliosis None = 0 Less than 10 = 1 10–20 = 2 20–40 = 3 > 40 = 4 Repaired = 5	Spinal fusion is less likely to change with time Fits the notion of natural history Overlapping boundaries for the categories Timing of x-rays may not coincide with exam Scoliosis and surgery are age dependent	Removed from the severity assessment and placed in a natural history component
Revision	Alertness and interaction during visit Alertness and interaction during visit (minimum 20 minutes) 100 %, all of visit = 0; Not all of visit but more than half = 1 Half of visit = 2 Less than half of visit= 3, Not interactive (awake but “shut down”) or sleepy for nearly all of the visit but not entirely =4 Not interactive (awake but “shut down”) or asleep during all of the visit = 5 Per the parent, was this typical (yes/no):	All categories may not be necessary The child can be awake and not interactive Concept of ‘shutdown’ not defined The state of the child can affect alertness Literature search: Munde (2009): Alertness and attention is observed in the child’s engagement with the environment	Alertness/attention –For this scale, alertness and attention is observed in the child’s activities in relation to interactions and engagement with the environment (Munde 2009) 0 - Alert/attentive and engaged in sensory activities with a person (eg watching, listening, smiling) consistently when offered, normal 1 - Alert/attentive and engaged in sensory activities with an object (eg watching video on ipad, listening) 2 - Awake but attention has to be drawn by examiner 3 - Awake but ‘shutdown’ or dazed for most of the exam 4 - Drowsy or asleep for most of the exam

Some items were added to assure that different concepts were not being evaluated within a single item. For example, originally, sit to stand was incorporated in the stand item. Interview data revealed a consensus that sit to stand was a different functional concept to standing and therefore should be a separate item. Another example was the addition of a vision item so that optokinetic response, fixing, and following were not examined within a single item.

Removal of Items

The team omitted items when cognitive interview data indicated that they were not consistent with the purpose of a CDD specific clinical severity assessment (Table 1). For example, “curvature and scoliosis” was removed from the severity assessment since it is age-dependent and unlikely to be observed in infants and young children. If scoliosis is present in an older child, any alteration in scoliosis would likely be apparent only over a longer assessment period than the usual course of a clinical trial. While prone examination subjectively determines curvature, specific evaluation requires X-ray and this would not always be indicated during regular clinical assessments. As such, the item was considered relevant to natural history rather than responsive to new therapeutics over the timeframe of a clinical trial. Items describing ‘contractures’ were also removed from the CCSA-Clinician because of age dependency and relationships with access to other physical therapy interventions. However, these items provide important information and were therefore moved to a separate natural history component which does not contribute to the clinical severity score.

The original “impact of dystonia/rigidity, chorea/athetosis and stereotypies” item was removed since this information is covered in the gross motor and fine motor items. This item is also associated with some age dependency because “an extremely hypotonic young child will receive a score indicating low severity”. The item describing “two object choice during minimum 30 sec” was removed from the CCSA-Clinician as it was considered very difficult to collect consistently good quality data in this population within the clinician exam setting. Also, cortical visual impairment and hand function were additional confounders in the item that would have an impact on scoring choice making.

“Self-injury” and “aggression” were omitted as they were rarely observed in the CDD population. Further, if present, these items are more consistently observed by parents and were moved for consideration to be added to the CCSA-Parent.

Revision of Items

The remaining items in the CCSA-Clinician were revised due to identified themes of concern. Items were revised if the clinicians agreed that the item still reflected the underlying conceptual framework of the CCSA-Clinician and that the problems identified could be addressed through rewording.

Item Content

Analysis of interviews revealed diverse and competing interpretations of some items due to lack of definition or instruction for the clinician (Table 1). For instance, the terms of movement disorders (i.e. dystonia) were accompanied by definitions derived from the NIH [23]. Also, instructions for administering items, such as ‘optokinetic response,’ were added to ensure the mode of completion was standardized between clinicians to improve inter-rater reliability.

Response categories

For some items, ambiguous wording in the response categories were identified. For example, a response category in the original walking item included the phrase ‘reduced ability’, which was interpreted differently by the clinicians. Similarly, the wording and number of response categories of items, such as ‘speech’, were altered to define certain concepts and create mutually exclusive categories. These clarifications aimed to increase clarity and improve reliability between users. Otherwise, the response categories for the communication items were suitable to the range of skills that could be achieved for children over 18 months of age. We therefore adjusted the highest response category level for children younger than 18 months to base scoring on what was developmentally appropriate. Irrespective of the child’s age, the total possible score for each item was 100.

Analysis of cognitive interviews identified that CDD patients often had both hypotonia and hypertonia simultaneously and that the pattern rather than the presence of abnormal tone changed with time. This nuance was not captured in the original hypotonia or hypertonia items. The group consensus was to combine hypotonia and hypertonia into a single item describing altered tone and displayed in a grid format, documenting the type and severity of altered tone for each main joint, where severity of the upper extremity, lower extremity and axial tone was scored based on the most severe component (Table 2). Definitions of mild and severe hypotonia and hypertonia were provided based on Barker [20].

Table 2:

Example children of different ages illustrating similar severity of altered muscle tone but changing distribution with increasing child age, as was commonly described in the cognitive interviewing responses. Grey cells below indicate scoring for sample children. Different severities of tone would yield different altered tone scores rather than whether muscle tone is hypotonic or hypertonic.

Questionnaire Construct

Child’s state

A majority of the clinicians reported that understanding and recording the child’s state is important to provide context for a severity assessment in a clinical trial. The child’s state will be recorded and the items that are not observable will be acknowledged at the end of the assessment. The clinicians noted that it would be important to record if the child was postictal or unwell as well as their current medications. Often, the child has travelled many hours to the clinic appointment which is also important to document.

Weighting and scoring

For the gross motor items, the number of items varies depending on the age of the patient. Head control and supine lying to sitting are items that will be completed by children of all ages. Otherwise, items for sitting (≥7 months), sit to stand (≥15 months), standing (≥15months) and walking (>18months) will be administered depending on the age of the child. Scores for gross motor function items and other groups of items (e.g. communication, stereotypies, autonomic function) will be averaged over the number of items administered.

Calculating dimension and total scores will occur by first transforming each item to a 100-point scale. The total score for each domain (e.g. gross motor, communication, stereotypies) will be averaged by the number of grouped items to give a functional abilities and neurological dimension scores. Dimension scores will then be averaged to give a total score.

Discussion

Our systematic interview and consensus methods assessed the content validity of the clinician reported items in the CCSA-Clinician and support its feasibility and practicability. Consensus for the finalised item set was achieved across the group of specialist clinicians using individual and group consultations. This study refined the CCSA-Clinician and improved content validity, clarity and usability of items, and their administration within a feasible time frame. Items were added, removed or revised to enhance the relevance and specificity of the CCSA-Clinician.

Establishing content validity is central to instrument development processes [10] and required by the FDA in developing new outcome measures [8]. The evaluation of the content validity of the CCSA-Clinician lays a strong foundation to then assess reliability and other aspects of validity. The CCSA-Clinician is ready for formal field testing and stands apart from other severity scales previously developed but not content validated, such as the severity assessments for RTT [24–28], FOXG1 [29 30], tuberous sclerosis [31] and other DEEs [32]. Other severity assessments have been content validated with a form that rated items on relevance [33]. In contrast, our novel approach and iterative methods allowed us to explore individual and group thought processes to fine-tune the items and structure of the CCSA-Clinician and optimise content validity.

The CCSA is designed to be completed for individuals of all ages. However, age was a limiting factor when implementing the gross motor items in the original instrument, which included items for walking, standing and sitting. For a typically developing child, the World Health Organisation gross motor milestones identify the average age for independent sitting as 6 months (range 4–9 months) [34]. However, for CDD, motor impairment is delayed and independent sitting is typically acquired at a median age of 3 years (range 6 months-5 years) [35]. Standing and walking are often delayed or not achieved in the CDD population, with independent standing attained by 26% (27/105) and independent walking attained by 22% of children (24/109) [35]. Having only these three gross motor items in the CCSA-Clinician, in a population with motor development difficulties, could not distinguish different gross motor abilities between children, nor could it measure lesser changes. To resolve this issue, additional items (head control and supine to sit) were incorporated into the gross motor item set to allow the gradation needed for younger or more severely affected children. Age ranges were also specified for testing each of the gross motor items, to ensure a child was not evaluated for a skill before its acquisition would be outside the normal range for children in the general population [34]. By contrast, head control and supine to sit are fundamental skills and will be assessed across all ages.

Hypotonia is commonly observed across the CDD spectrum. However, CDD patients can have hypertonia simultaneously with elements of muscle spasticity and rigidity. Analysis of clinician interviews revealed that clinicians did not think that this mixed tone aspect of CDD was adequately addressed in the original CCSA-Clinician and that severity of tone abnormalities also had to be captured. The group consensus was to combine hypotonia and hypertonia into a single item assessing altered tone displayed in a grid format. This allows the clinician to record the severity of hypotonia or hypertonia, as well as which body parts are affected. Barker’s definitions of mild/severe hypotonia/hypertonia and normal tone are provided for clinicians when scoring which may improve inter-rater reliability [20].

The clinicians reported that understanding the child’s state is very important for severity assessment in a clinical trial and the relationship of the assessment to the child’s last seizure (i.e., was the child postictal?), wellness or current medications should be recorded. Also, many US parents wake the child early and drive hours to their clinician, which may affect the child. Other factors may impact the child’s function on a given day (e.g., upper respiratory tract infection). Capacity to note items that were difficult to observe during the exam was provided at the end of the assessment. It was also of value to look carefully at the scoring of items to make sure that groups of items (e.g. gross motor function, communication, autonomic function) were scored equally as there is no clear evidence in the literature that, for severity, one domain is more important than another. For example, for particular groups of items in the CCSA-Clinician, more than one item may be required to assess severity (e.g. stereotypies = 3 items) compared to others where one item is sufficient (e.g. alertness).

Strengths and limitations

A strength of the study was that participation included all of the USA COE network of clinicians. Although the network includes only eight child neurologists, they have extensive experience in evaluating CDD patients, having assessed approximately 190 patients with CDD and have published extensively on CDD (over 15 distinct publications, many co-authored by this group). The interview schedule included probing questions, ranging from general to specific in focus. This allowed us to compare and contrast patients of different severities to widen the clinician’s lens and explore an item with more depth. There is the possibility that there may have been some bias in the specific probes, however; they directed the clinicians to specific aspects in the items. The group consensus method allowed expression of a wide range of direct knowledge and experience, consideration of multiple options, debate that challenged received ideas and stimulated new ones, and individual idiosyncrasies to be filtered out [36]. We have created a broad evaluation of clinical severity and acknowledge that specific domains will profit from additional specialist evaluation. For example, there is an important role for a neuro-ophthalmologist within a multi-disciplinary setting for the evaluation of cortical visual impairment.

The content validation procedure strongly argues for two domains which are clinically justified. This means that an exploratory factor analysis is not required because the dimensionality of these items for the CCSA-Clinician is not in question as the function and neurological impairment items could not be grouped otherwise. However, a confirmatory factor analysis, using standardised item loading coefficients to account for different numbers of response categories, will be indicated to assess the psychometric properties of the scale when applied to the CDD population. Convergent and divergent validity statistics will be calculated for the domains. Reliability and stability will be assessed with test retest reliability and longitudinal studies.

Conclusion

We propose that the consensus methods enabled the refinement of the CCSA-Clinician and provided satisfactory evidence of content validity. This preliminary validation is essential for further advances in instrument development towards clinical trial readiness. The next step for the CCSA-Clinician will include further validation through field testing in COE clinics and testing of reliability and sensitivity to change through the use of video footage. A separate study is underway for the content validation of the CCSA-Parent using a similar methodology but with differences in consultation methods, and which will supplement the CCSA-Clinician on topics of epilepsy, gastrointestinal issues and behaviours. The parent and clinician reported assessments will ultimately combine to result in a total severity score.