The Children’s Hospital of Philadelphia Infant Test of Neuromuscular Disorders (CHOP INTEND): Test development and reliability

Abstract

The motor skills of patients with spinal muscular atrophy, type I (SMA-I) are very limited. It is difficult to quantify the motor abilities of these patients and as a result there is currently no validated measure of motor function that can be utilized as an outcome measure in clinical trials of SMA-I. We have developed the Children’s Hospital of Philadelphia Infant Test of Neuromuscular Disorders (“CHOP INTEND”) to evaluate the motor skills of patients with SMA-I. The test was developed following the evaluation of 26 infants with SMA-I mean age 11.5 months (1.4–37.9 months) with the Test of Infant Motor Performance and The Children’s Hospital of Philadelphia Test of Strength in SMA, a newly devised motor assessment for SMA. Items for the CHOP INTEND were selected by an expert panel based on item mean and standard deviation, item frequency distribution, and Chronbach’s alpha. Intra-rater reliability of the resulting test was established by test–retest of 9 infants with SMA-I over a 2 month period; Intraclass correlation coefficient (ICC) (3,1) = 0.96. Interrater reliability was by video analysis of a mixed group of infants with neuromuscular disease by 4 evaluators; ICC (3,4) = 0.98 and in a group of 8 typically developing infants by 5 evaluators ICC (3,5) = 0.93. The face validity of the CHOP INTEND is supported by the use of an expert panel in item selection; however, further validation is needed. The CHOP INTEND is a reliable measure of motor skills in patients with SMA-I and neuromuscular disorders presenting in infancy.

1. Introduction

Spinal Muscular Atrophy (SMA, OMIM 253300) is a neurodegenerative disorder of motor neurons presenting usually in infancy and childhood, with an incidence of approximately 14 of every 100,000 live-born infants [1]. Homozygous deletion of the gene for this autosomal recessive disorder, SMN1, is responsible for the disease [2], while a near-identical homologue copy, SMN2, rescues an otherwise lethal phenotype. The normal SMN1 gene product, full length SMN protein, is important in RNA metabolism and is expressed in peripheral nerve and muscle. Due to a critical difference in SMN2 at the exon 7 splice site, a reduced amount of full length protein is translated [3]. The number of copies of SMN2 correlates in general with the functional status of SMA [4] and targeted treatment strategies are focused on increasing the amount of the normal SMN protein [5].

The classification of clinical types of SMA is dependent upon age of onset and maximal level of motor function achieved [6,7]. Patients with SMA type I (SMA-I) are infants diagnosed usually by 6 months of age who never achieve independent sitting. Those with SMA type II (SMA-II) present usually between six and eighteen months of age and achieve sitting but never walk independently. Those who achieve independent ambulation have either SMA type III (SMA-III), presenting usually after eighteen months of age, or SMA type IV (SMA-IV), which presents after age 30 years. Due to progressive muscle weakness, loss of this maximal level of function occurs over time, for example, a SMA-II child who loses the ability to sit can act functionally as an infant with SMA-I. This loss of function is more rapid for SMA-I and is more indolent for SMA-II and III and correlates also with general morbidity and mortality [8,9]. In patients with SMA-II or -III, some gain of motor function can be observed early after onset of the disease, a result of ongoing normal motor development in children of their age, although the disease is progressive later on nevertheless SMA patients exhibit a wide range of motor function which from very weak infants who are unable to sit to teenagers who are able to play sports. This creates a significant challenge in designing quantitative motor assessments to evaluate these patients and necessitates scales that are specific to their unique motor behaviors and age related cognitive development. In particular, patients with SMA-I have posed significant obstacles to motor testing because of their limited repertoire of motor behaviors, limited tolerance to handling and fragile medical state [10]. Clinical trials in SMA-I requires such a scale for demonstrating change in motor status. Our prior work in this area identified the absence of an ideal motor scale for this population [10]. This lead us to develop a new motor function scale for weak infants and young children, targeting SMA-I and other similar neuromuscular disorders.

In the initial stages of this project we reviewed existing motor scales for infants and selected for evaluation those having a high proportion of items likely to be useful in patients with SMA-I. These included the Neurological Assessment of the Preterm and Full-term Newborn Infant [11], the Alberta Infant Motor Scale [12], The Peabody Developmental Motor Scales II [13], the Bayley Scales of Infant Development [14], and the Test of Infant Motor Performance (TIMP) [15]. From among these tests many items were not well tolerated and did not effectively capture the motor repertoire of SMA-I (unpublished observations, AMG and RSF). The TIMP was selected as the best test measure among this group that would best reflect motor skill in this population. While being examined in a pilot study, the TIMP was selected as an outcome measure for the AmSMART riluzole trial for SMA-I[16]. As a part of this trial the reliability of the TIMP was established [10]. Yet there were several limiting issues, discussed below, which prompted us to explore a more ideal item set for this population.

2. Methods

2.1. Test development

Development of the CHOP INTEND was performed in three parts. In the first part, we designed several new test items to capture specific targeted motor skills that are clinically significant for the SMA-I population and developmentally appropriate for infants, but where no items existed in the TIMP or The Peabody Developmental Motor Scales II to reflect those skills. These new items were combined with selected items from the TIMP. Both observational and elicited motor activity is captured. Behavioral state of the infant was captured using the Brazelton Neonatal Behavior Assessment Scale, to identify whether response to specific test items was state-dependent [17]. We termed the pilot version of the test The Children’s Hospital of Philadelphia Test of Strength in SMA (CHOP TOSS). In the second part we administered the full TIMP, the fine motor section of the Peabody Developmental Motor Scales II and the CHOP TOSS. Then, in the third part, each item from the TIMP and CHOP TOSS was examined for its clinical utility in capturing a salient aspect of motor activity; redundant items were eliminated and the items were reordered from least to most difficult and stressful to the infant. Scoring for each item was revised to a uniform 0–4 scale: no response (0), minimal (1), partial (2), nearly full (3) and complete (4) level of response. An approach for item selection, which combined descriptive statistics and an expert panel, has been previously reported [18] and was used to identify which items best met the objectives of motor assessment scale for very weak infants. This newer version was designed to meet five main goals: (1) to be applicable for both weak infants and older children having an infant’s repertoire of motor skills (2) to be useful in a broad variety of neuromuscular disorders presenting in infancy or early childhood, in addition to SMA-I (3) to be administered in a short period of time and can be tolerated by weak infants (4) be responsive to change over time and (5) capture both increase and decrease in motor function without a ceiling or floor effect in the SMA-I population. The development process was designed to assure that the resulting test measure would contain items that are internally consistent and reliable when used in the target population of SMA-I. This resulted in construction of a new test measure, which we have named the “CHOP INTEND” to represent The Children’s Hospital of Philadelphia Infant Test of Neuromuscular Disorders (Fig. 1).

Fig. 1.

CHOP INTEND.

Population

A group of 26 infants with typical clinical features of SMA-I and genetic confirmation were evaluated on the full TIMP and CHOP TOSS and a subset of these patients were tested on the fine motor section of the Peabody Developmental Motor Scales II. Mean age was 11.5 months (1.4–37.9 months). Four cases had missing age data.

Statistical analysis

The initial step in the data analysis from the assessment of 26 infants with SMA-I used the Cronbach’s alpha test. This is a test of internal consistency among grouped data where a value of greater than 0.70 generally demonstrates that items relate well with one and other [19]. In this way, the items are identified that most reflect the motor skills and strength that the tests were designed to evaluate. Items that showed lower alphas were interpreted to be independent of gross motor development and strength and this was used as a guide for item selection.

The second step in the review process was to select items based on the potential that they would contribute to the sensitivity of the final test in infants with SMA-I. The mean and standard deviations were calculated for each of the items. The mean item score of each item was used to determine which items were clearly too easy or too difficult for this population and to assure that the range of items would reflect the skills demonstrated by the full range of infants in our cross sectional sample. Many items had low means, with a large proportion of subjects scoring the minimum score of 0, reflecting this set of subjects’ motor skills. The frequency distribution was also used to assure that both difficult and easy items were included, providing assurance that the final test would not have a floor or ceiling effect. In addition items that demonstrated a primarily bimodal distribution and might not best contribute to the sensitivity of the final test were identified and avoided. The standard deviation was used with the frequency distribution to evaluate the potential sensitivity of each item. Items with a high standard deviation demonstrate a larger spread of scores in our cross sectional sample. These items, when included in the final test, provided greater potential to differentiate among subjects with different levels of motor ability. In this way a set of items that captured the entire range of skills in a representative cross-sectional population had the best likelihood of being sensitive to change over time in the context of illustrating disease progression or response to drug treatment. Cut scores were established as a guide for item selection for each of the statistical measures and an expert panel (AMG, RSF, EM, EB, MM, EM, SM) was established to select and edit the final item set based their clinical experience and each items statistical profile.

The intraclass correlation coefficient (ICC) was used to assess the inter- and intra-rater reliability of the CHOP INTEND scale. Three populations were used for this analysis the first was a group of 10 SMA-I patients where intra-rater reliability was established. The second was a group of 9 infants with a mixture of neuromuscular diagnoses including SMA-I, myotubular myopathy, infant botulism, nemaline myopathy, and central core myopathy where interrater reliability was established. The third group was a group of 8 typically developing infants ranging in age from 36 weeks gestation to 8 months of age. These subjects were evaluated by five evaluators.

3. Results

3.1. Test development

The Cronbach’s alpha, standard deviation, item mean, and item frequency distribution were used to guide item selecton. By using Cronbach’s alpha, the TIMP was narrowed from 42 to 27 items and the CHOP TOSS items were narrowed from 12 to 9 items. Next the standard deviation was applied and the TIMP items were narrowed to 21 items and the CHOP TOSS was narrowed to 8 items. The mean and the frequency distribution of each item was then reviewed by our expert panel to assure that the full range of severity was covered in the final item set so that both difficult and easy items were selected to minimize the chance that the final test would suffer from a ceiling or floor effect. The items underwent a final editing process. Some items were combined and the scoring reordered with the goal of decreasing the stress on the subject (placing the least stressful items first), minimizing the impact of developmental issues, and extending the range of each item to best reflect the range of performance of our sample population of SMA-I. The first two observational items were adaptations of several of the observational item on the TIMP and represent a combination of the observational finger and ankle movement items and the kicking items. The handgrip item was the result of combining the traction response item originally adapted from the Neurological Assessment of the Preterm and Full-term Infant with the grasp item we had developed for the CHOP TOSS. Three of the TIMP items (#22, 23 and 24) captured adjoining portions of the SMA sample population based on the frequency distribution and were combined to form the “head in midline” item in the INTEND (#4). The four elicited rolling items (#28–31) were kept largely intact from the TIMP with only minor adaptation. The knee extension and the shoulder flexion in sitting (#10, 8) and sidelying (#7) items were retained from the CHOP TOSS. The placing item (#9) was adapted to remove the reflexive/developmental component. The head control item in the INTEND (#12) reflects a combination of several of the head control and head rotation items from the TIMP (#15–17). The scoring of the head control item was adapted and the criteria reordered to eliminate the stress of placing the head in extension or flexion when the infants head control is limited. Finally, the pull to sit, Landau and Galant items on the INTEND (#13–16) were adapted from their descriptions on the Neurological Assessment of the Pre-term and Full-term Infant, TIMP and the CHOP TOSS. Two of the items (INTEND #11, 16) were reworked to diminish the impact of developmental and reflexive behaviors and allow the scoring of active goal directed movement in the older patient. In this way the item set was narrowed from the full 42 item TIMP and 12 item CHOP TOSS to the final 16 items. Two items from the CHOP TOSS (#6, 12) were eliminated as they proved less informative and six items from the TIMP were added (#15, 22, 28–31), These final 16 selected items were then reordered to limit position change and to place the items most likely to cause irritability toward the end of the final version of the CHOP INTEND.

By including items with both active and elicited reflexive movement we aimed to construct a test that reflected the strength of the infant, which is the primary impairment in SMA-I, and not biased, defined or limited by neurodevelopmental status. Each item is constructed to capture the movement of one body segment against another or against gravity. It is the strength of the related muscle groups and the ability to move the body segment that creates the basis for this test. At each developmental level these movements might be reflexive, spontaneous, or goal directed and in each instance reflects the presence of sufficient muscle strength and endurance to fully, partially or minimally complete the item.

3.2. Reliability

Intra-rater reliability was established by test retest over a 2-month period by the same evaluator (AMG) in a group of 9 infants with SMA-I. Interclass correlation coefficient (ICC) (3,1) was 0.96. The interrater reliability was established by evaluating 10 infants with a variety of neuromuscular diseases. Because of the fragility of our patient population and the tendency toward fatigue we chose to have one evaluator administer the CHOP INTEND on video and then have four evaluators score the INTEND by video tape without knowledge of the other evaluators scores. Interclass correlation coefficient (3,4) of 0.98 used to establish reliability. Finally we establish the interrater reliability in a group of 8 typically developing subjects evaluated by five evaluators, ICC (3,5) was 0.93.

4. Discussion

Patients with neuromuscular disease present unique challenges in motor assessment. These patients typically have limited tolerance to handling or positioning because of poor respiratory reserve and difficulty in handling secretions. Infants with SMA-I often do not tolerate the prone position due to respiratory compromise secondary to their reliance on diaphragmatic breathing and abdominal expansion for inspiration. Patients that present in infancy often have a severely limited repertoire of motor skills and as a result a limited set of movements on which to base an assessment. Many current infant motor assessments rely heavily on prone and head control items and, as a result, examination with many of these existing standardized assessments is poorly tolerated. These evaluations are unsatisfactory because of the respiratory demands of a lengthy assessment or the positioning requirements of the items. Additionally, many items designed to assess typical infant motor development result in a floor effect due to the motor skill needed to achieve a minimal item score and as a result have limited sensitivity in this population. Here we have presented the development of a motor assessment designed to be tolerable and reliable in infants with neuromuscular disease and a severe degree of weakness.

The test development process was designed to accommodate the fragile nature of infants with SMA-I. The statistical analysis utilized for evaluating the utility of each test item was combined with an expert panel review of the selected items that spanned the range of skills observed in our cross sectional sample and which were well tolerated. In this way the panel was guided in item selection by the statistical characteristics of each item as well as their clinical judgment about the item’s ability to quantify motor behavior in our sample of patients with SMA-I and the observed tolerance to item administration in our population. The items were then ordered in such a way as to position the least stressful items first and to limit changes in position to optimize the patient’s tolerance of the testing process.

The reliability of the CHOP INTEND was established in a population of children with SMA-I, in a mixed group of subjects with neuromuscular disease and weakness and in a group of typically developing infants. Despite the test development process being directed by our sample of 26 infants with SMA- I the resulting items reflect skills that might be found in a wide range of weak infants. The high reliability found in both the SMA and myopathy subjects presents the opportunity for the use of the CHOP INTEND in a wider population of infants with neuromuscular diseases. The reliability in the typically developing group of infants was least robust although still excellent. This is most likely due to the ceiling effect that we saw in this group of subjects and the resulting limited variability in the scores between subjects. This likely diminished the resulting ICC.

Validation of the CHOP INTEND is necessary for both infants with SMA and other neuromuscular diseases. The face validity is supported by the use of an expert panel in the development process, however, having concurrent, construct and convergent validity with other measures of disease progression and motor skill in those populations would add to the strength of the test as an outcome measure in this population. Additionally, it is necessary to establish the predictive and discriminant validity of the CHOP INTEND if it is to effectively measure the extent of motor impairment and function and serve as an outcome measure in a clinical trial.

Several items from the TIMP were deleted because they proved too stressful or irritating for the children, other items, felt to be developmentally sensitive were excluded if they did not mesh effectively with our goal of creating an assessment of strength and motor function in a broad age range of patients with SMA-I. For example, the two defensive reaction items (#25, 26) are dependent on the development of object permanence, which typically develops at the age of 8 months. Similarly, with the inhibition of neck righting items (#19, 20), the immature response (rolling to the side) differs from the more mature response (to actively resist rolling to realign the trunk with the head) [15]. In addition the Pea-body Developmental Motor Scales II fine motor items were not felt to be optimal for this population because they were developmentally dependent and relied on a mix of fine motor coordination and proximal strength for successful completion. As a result the Peabody Developmental Motor Scales II items were abandoned and no consistent data was collected for analysis.

In conclusion, the CHOP INTEND is a reliable, easily administered and well tolerated motor test measure for SMA-I and similarly weak infants with neuromuscular disease. The CHOP INTEND can provide a useful measure of motor skill in this population both for clinical monitoring and for research trials. Further research is needed to establish the sensitivity of the measure to change over time and to further establish its validity as a measure of motor function in the target population.

Abbreviations

SMN 1

survival of motor neuron gene 1

SMN 2

survival of motor neuron gene 2

SMA-I

spinal muscular atrophy type I

SMA-II

spinal muscular atrophy type II

SMA-III

spinal muscular atrophy type III

SMA-IV

spinal muscular atrophy type IV

TIMP

Test of Infant Motor Performance

CHOP INTEND

The Children’s Hospital of Philadelphia Infant Test of Neuromuscular Disorders

CHOP TOSS

The Children’s Hospital of Philadelphia Test of Strength in SMA

ICC

intraclass correlation coefficient