Baby Observational Selective Control AppRaisal (BabyOSCAR): Construct validity and test performance

Abstract

Aim:

To investigate the construct validity of the Baby Observational Selective Control AppRaisal (BabyOSCAR), an assessment of independent joint motion in infants with cerebral palsy (CP).

Method:

BabyOSCAR was scored for 75 infants (45 with CP and 30 without CP). Rasch analysis was used in combination with classical test theory to assess areas of strength or improvement. Overall fit and precision, unidimensionality, local independence, reliability indices, Wright’s child-item map, and differential item functioning were examined as part of Rasch analysis to investigate the item properties, internal construct validity, and reliability of BabyOSCAR. Cronbach’s α was used to evaluate items’ internal consistency.

Results:

Analysis demonstrated good fit to the Rasch model, with only one erratic item. Unidimensionality results suggest two dimensions, split between arm and leg items. Item calibration reliability was between 0.84 and 0.86, with three distinct item difficulty levels. Infant measure reliability was between 0.82 and 0.91, separating infants into three ability levels. Together, the two subscales covered the full range of skills, with redundancy mostly between the same motion on both sides of the body. Cronbach’s α was between 0.90 and 0.95.

Interpretation:

BabyOSCAR’s construct validity was supported. Arm and leg subscales can be translated to a logit scale.

Graphical Abstract

There is a significant amount of redundancy in the body’s movement system, with multiple degrees of freedom at each joint and many joint combinations that can be selected by the nervous system to meet end-point coordinates. A unique aspect of the corticospinal tracts of the nervous system is the ability to individually control and isolate movements at single joints.¹ This skill has been shown to be significantly compromised in children with cerebral palsy (CP),^2,3 resulting in a reduced number of possible joint movements and combinations, and less variability of movement possibilities. Selective motor control (SMC) is the term that describes the ability to move one joint in an isolated way⁴ and is usually tested in older children.

We created an observational test to quantify the presence of SMC in infants 12 to 16 weeks of age, called the Baby Observational Selective Control AppRaisal (BabyOSCAR). It quantifies an infant’s capacity to perform isolated joint movements in the extremities while spontaneously moving in supine, identifying which joints infants do and do not move in isolation (out of 32 possible arm and leg items). Because spontaneous infant motor behavior at 12 to 16 weeks is highly predictive of a diagnosis of CP,⁵ we used this time point to study SMC with BabyOSCAR. In the accompanying studies to this one,^6,7 BabyOSCAR shows excellent convergent, discriminant,⁶ and predictive⁷ validity. In this study, we evaluated the construct validity of BabyOSCAR using Rasch analysis⁸ and classical test theory.

METHOD

Data collection

BabyOSCAR scores (at 10–16 weeks post-term) were obtained from 75 infants born preterm and at term, of whom 45 were diagnosed by a physician as having spastic CP and classified in Gross Motor Function Classification System (GMFCS) levels at a corrected age of at least 2 years. A detailed description of the sample was previously presented.⁶ There were 32 possible points for the total BabyOSCAR score: 14 in the leg and 18 in the arm (more details on scoring are available in Sukal-Moulton et al.⁶). Infants were scored during a 1-minute video of spontaneous activity in supine, receiving a point for each joint that demonstrated independent motion (moved in isolation without motion at other joints or mirrored to the opposite side of the body). All parents gave written permission for the use of videos through institutional review board-approved studies.

Rasch analysis

Rasch analysis,⁸ used in the development of other infant motor assessment tools,^9,10 helps to determine the quality of a test by using a probabilistic model to evaluate both item difficulty and participants’ abilities simultaneously (placing both items and infants along an ability continuum). In addition, Rasch analysis transforms ordinal data into interval-level measures (reported in logits) addressing important scale assumptions of a test’s measurement properties and allowing subsequent statistical analyses to be applied. If the data fit the model, a hierarchical order of infant ability levels and item difficulty combine to measure a unidimensional construct (i.e. measuring a single attribute at a time) and support validity.

The following parameters were used to evaluate BabyOSCAR item properties, internal construct validity, and some aspects of reliability using a Rasch dichotomous model in Winsteps (version 3.8).¹¹

Overall fit and precision

Fit and precision metrics were calculated to determine how well each item ‘fitted’ the underlying construct. Both infit and outfit mean-square greater than 1.30 and t greater than 2.0 were considered as misfitting.¹² The model was considered valid if less than 5% of items misfitted.¹³

Unidimensionality

We evaluated unidimensionality to determine whether the construct of BabyOSCAR had more than one dimension¹⁴ using the following criteria: (1) number of misfitting items less than 5%;¹⁵ (2) point-biserial correlation between items and total score greater than 0.30;¹⁶ and (3) principal component analysis of standardized residuals had both its main component explaining at least 50% of the variance in the measures^15,17 and the eigenvalue of the first residual component less than 2 and explaining less than 10% of the variance.¹⁸

Local independence

We evaluated the possibility of redundancy and degrees of dependence between the responses to different items using local independence.¹⁴ Yen’s Q₃ statistic¹⁹ values of at least 0.30 delineated pairs of items in possible violation of local independence.

Reliability indices

We used reliability indices as a measure of internal consistency. Values of at least 0.80 represented good consistency for items and infants.^11,12 Separation values (G) were used to calculate the strata in which the items’ difficulty level and infant’s ability measures are divided using ([4G + 1]/3).¹²

Wright’s child-item map

Wright’s child-item map was used to (1) visualize the relationship between item difficulty and infant ability along the continuum of SMC skill and (2) investigate how well the item difficulty calibrations were distributed across the measurement range to describe all infant ability measures (targeting). Poor targeting was defined when there was a (1) large difference (>1.0 logits) between infant and item means; (2) skewed infant distribution relative to the item distribution; (3) wide range of infant measures where few items were located; or (4) large number of infants located above the most challenging or below the easiest item on the distribution map.¹²

Differential item functioning

Differential item functioning (DIF) was used to determine whether item difficulty remained stable across the infants with and without CP, who were matched on their overall ability at the time of testing. The DIF analysis focused on comparisons of items that were more difficult for one group but easier for others who had similar scores.¹² A sample of 30 infants per group is sufficiently powered to say there is DIF when the contrast is robust (>1.0).²⁰ Both statistical significance (p < 0.05) and substantive difference (DIF contrast > 0.5) were used to indicate bias.¹⁸

Classical test theory

To confirm reliability, the BabyOSCAR’s internal consistency coefficient (Cronbach’s α) was calculated, with expected values between 0.70 and 0.90.²¹ SPSS version 28 (IBM Corp., Armonk, NY, USA) was used for classical test theory analysis.

RESULTS

Infants

The sample used for validity testing comprised 75 infants: 30 without CP and 45 who received a CP diagnosis and GMFCS level at a corrected age of at least 2 years (16 with unilateral CP and 29 with bilateral CP).

Rasch analysis

Table 1 summarizes the psychometric properties of BabyOSCAR using Rasch analysis.

TABLE 1.

Psychometric values of Baby Observational Selective Control AppRaisal (BabyOSCAR) using Rasch analysis

Number of items in the analysis	32 (all items)	14 (leg items)	18 (arm items)
Infant measures
Infants with extreme scores	1 minimum extreme score (1%)	11 minimum extreme score (14.6%) 5 maximum extreme score (6.6%)	2 minimum extreme score (2%) 3 maximum extreme score (4%)
Raw score range	0 to 32	0 to 14	0 to 18
Raw score mean (SD)	17.3 (9.6)	7.4 (4.5)	10.3 (4.9)
Measure range, logits	−3.74 to 3.79	−2.90 to 2.95	−3.10 to 3.20
Measure mean, logit (SD)	0.23 (1.89)	0.18 (2.05)	0.42 (1.70)
Infant fit, %	96	100	100
Misfitting infants, n	3	0	0
Reliability	0.91	0.83	0.82
Separation index	3.24	2.19	2.14
Strata	4.65	3.25	3.18
Item measures
Measure range, logits	−1.60 to 1.69	−1.31 to 1.48	−1.27 to 1.93
Measure mean, logit (SD)	0 (0.81)	0 (0.97)	0 (0.87)
Item fit, %	97	100	95
Misfitting item (name)	1 (left shoulder horizontal ab/adduction)	None	1 (left shoulder horizontal ab/adduction)
Reliability	0.86	0.84	0.86
Separation index	2.45	2.32	2.50
Strata	3.6	3.42	3.66
Point-biserial correlation
	28 items > 0.50	14 items > 0.50	17 items > 0.50
	4 items > 0.30	—	1 item > 0.30
Principal component analysis
Variance explained by measure, %	43.6	49.5	39.4
Variance explained by first contrast, %	7.0	8.2	10.3
Loading	Upper vs lower extremities	Left vs right side	Left vs right side
Yen Q3
	0.50 (right with left shoulder horizontal ab/adduction)	0.32 (left ankle inversion/eversion with left toe flexion/extension)	0.43 (right shoulder internal/external rotation with right elbow flexion/extension)
	0.46 (right shoulder internal/external rotation to right elbow flexion/extension)		0.39 (right to left shoulder horizontal ab/adduction)
	0.43 (right to left shoulder flexion/extension)		0.33 (right elbow flexion/extension to right wrist ulnar/radial deviation)
DIF
	—	None	Left shoulder horizontal abduction/adduction (DIF contrast = 2.11, p = 0.02), easier for infants with CP Left wrist flexion/extension (DIF contrast = 2.99, p = 0.03), easier for infants without CP

Abbreviations: CP, cerebral palsy; DIF, differential item functioning; SD, standard deviation.

Overall fit and precision

Only three infants (<0.5% of the sample) and one of 32 items (left shoulder horizontal abduction/adduction) misfitted the model, supporting adequate goodness of fit.

Unidimensionality

All items had a positive point-biserial correlation greater than 0.30, with 28 out of 32 being greater than 0.50. Principal component analysis of residuals, on the other hand, indicated that the items explained only 43.6% of the variance in the measurements, not meeting the 50% variance criteria. The eigenvalue of the first residual factor was 4.0 (>2 eigenvalue criteria) but explained less than 10% of the variance of the first residual factor (7%), partly meeting the criteria for unidimensionality. The standardized residuals showed separation between arm and leg items. Together, these results suggest the possibility of two dimensions or subscales: one for arms and the other for legs.

To further evaluate the impact of the arm and leg items, we plotted infant measures that were derived using the arm or leg items separately (Figure 1). This showed distinct infant ability measures for 29% of the sample, depending on which set of items were used. Considering that the criteria for unidimensionality were only partly met, this analysis supports further evaluation using BabyOSCAR as two separate subscales. We repeated the fit analysis for the two subscales. When arms and legs were analyzed separately, no infants misfitted, but the item left shoulder horizontal abduction/adduction continued to misfit.

FIGURE 1.

Infant Baby Observational Selective Control AppRaisal (BabyOSCAR) ability measures (n = 75) where arms and legs items were calibrated separately. Each point represents the relationship between arms and legs; larger diamonds signify more than one infant had the same scores. The dotted line represents the error-free Rasch-modeled invariance of infant ability measures when tested by different items. The solid lines represent the 95% confidence interval control lines. The infant measures are invariant within the limits of pairs of error estimates for each infant’s ability measure. Points outside the solid lines represent infants whose measures were variable depending on which set of items were used.

Local independence

The legs subscale had only one item pair with Yen Q₃ greater than 0.30, suggesting no local dependency, while the arms subscale showed three pairs of related items (Table 1).

Reliability indices

The item calibration reliability (Table 1) for all items analyzed together was 0.86, with a separation index of 2.45 and a spread of 3.6. These values were relatively stable for the subscale analyses. The reliability of the arm items was 0.86 (re-analysis of the data with the misfitting item deleted had a similar reliability value, 0.84), with a separation index of 2.5 or a spread of 3.66 levels of item difficulty. The reliability of the leg items was 0.84, with a separation index of 2.32 or a spread of 3.42 levels of item difficulty. These values indicate that there were three levels of item difficulty for both subscales.

When all items were analyzed together, the infant measure reliability (Table 1) was excellent (0.9), with a separation index of 3.24, representing 4.65 strata, and indicating four distinct ability levels of SMC. Re-analysis by arms and legs showed lower but still good infant measure reliability (0.82 and 0.83), with a separation index of 2.14 and 2.19, representing 3.18 and 3.25 strata for the arm and leg items respectively. This indicates three different levels of infant ability for each subscale.

Wright’s child-item map

The BabyOSCAR measures for the infants ranged from −2.90 to 2.95 in the leg subscale and from −3.10 to 3.20 in the arm subscale (Table 1), with a mean infant ability measure of 0.42 in the arms and 0.18 in the legs, indicating good test-item targeting. To facilitate the practical interpretation of scores for clinical use, these measures were transformed into a 0 to 100 logit-based scale (Figure 2). Overall, item difficulty calibrations for the arm and leg subscales on their own did not cover the end ranges of infants’ abilities. The arm subscale had a broader distribution, with both higher and lower measures than the leg subscale. There was no ceiling effect for either subscale, as far less than 15% of the children received a maximum score;²² but there was a borderline floor effect for the leg subscale, with 14.6% of infants receiving a minimum score. This indicates less precision in identifying infants with lower SMC if only the leg subscale is used.

FIGURE 2.

Distribution of the ability measures for all infants (n = 75) and the difficulty calibrations for all items in (a) legs and (b) arms (14 and 18 items respectively) on the 0 to 100 Baby Observational Selective Control AppRaisal (BabyOSCAR) unit logit scale, illustrating the targeting of items and infants. The most able infants and the most difficult items are at the top. The left-hand columns locate the infant ability measures; each ‘X’ is one infant. The right-hand columns locate the item difficulty measures; each item is placed at its mean calibration, corresponding to a 50% probability of being rated a score of 1.

Abbreviations: M, the mean infant measure and item calibration; S, 1 standard deviation (SD) from the mean; T, 2 SDs from the mean. Items are labeled with the convention side–joint–direction. Sides: L, left; R, right. Joint: shoulder; elbow; forearm; wrist; finger; hip; knee; ankle; toe. Direction: AA, abduction/adduction; FE, flexion/extension; HAA, horizontal abduction/adduction; IE, internal/external rotation; IF, individual finger movement; IT, individual toe movements; NV, inversion/eversion; PD, plantarflexion/dorsiflexion; PS, pronation/supination; RU, radial/ulnar deviation.

Figure 2 also shows the item hierarchy for the arm and leg subscales. Items located at the top are the more difficult ones in both scales. The pattern of item difficulty demonstrated some gaps around the middle ranges of item measures in the leg subscale, while the arm subscale had more items in the middle and upper ranges, with fewer in the lower ranges. There was also some redundancy in the item difficulty level in each subscale because items that repeated on both sides of the body tended to be next to each other in the difficulty hierarchy, without consistency of which side was more difficult. The two most difficult arm items were shoulder flexion/extension. In the leg subscale the most difficult items were knee flexion/extension and hip flexion/extension (Figure 2).

DIF

Infants with CP had overall lower arm and leg scores that than those without CP. No DIF was found in the leg items, but a large DIF contrast was found for two arm items: left shoulder horizontal abduction/adduction (DIF contrast = 2.11, p = 0.018) was easier for infants with CP; and left wrist flexion/extension (DIF contrast = 2.99, p = 0.028) was easier for infants without CP. Because these items were found in opposing groups, the effects were unlikely to have had an overall impact or bias on the infants’ total measures.

Classical test theory

Cronbach’s α for the total BabyOSCAR score was 0.95. After removal of each item individually, Cronbach’s α was between 0.948 and 0.951, indicating high internal consistency between items with no items measuring markedly different constructs. For the leg and arm score, Cronbach’s α was 0.941 (0.934–0.939 with individual times deleted) and 0.903 (0.894–0.905 with individual times deleted) respectively.

DISCUSSION

In this study, we found that BabyOSCAR demonstrated excellent construct validity with adequate goodness of fit, excellent item and infant measure reliability, good test-item targeting, no significant DIF, and high internal item consistency. These detailed psychometric analyses indicate that BabyOSCAR is well constructed and has adequate ability to measure an infant’s SMC performance.

Considering our initial hypothesis about redundancy in degrees of freedom in the nervous system, we did find some degree of dependency between the responses to different items in both subscales. While item redundancy may be a reason to remove a test item in some circumstances, in this case we did not because these items captured different ability levels, represented different joints or body sides (which may be particularly important in identifying asymmetries), and had good psychometric values (i.e. item fit, point-biserial correlation).

After conducting Rasch analysis on the test items, we identified areas for improvement in the BabyOSCAR test, leading to two design changes. The first change was to revise the description of the shoulder abduction and adduction item in the manual to reduce potential ambiguity in scoring among raters. Although the left shoulder abduction/adduction item initially misfitted the model, the criterion of less than 5% of items misfitting was still met and removing the item did not significantly improve the overall fit of the BabyOSCAR test to the Rasch model. Therefore, the item was retained because of the clinical importance of assessing both shoulders. The second change was the definition of arm and leg subscales. It is not surprising that these two different regions of the body represent different dimensions of the test with unique ordering of item difficulty. The different dimensions suggest that the scores should be interpreted separately, particularly when using the interval-level measure-unit (logit) produced by Rasch analysis. However, the two dimensions identified are unlikely to reflect different underlying neuromechanisms and the use of the total raw score (all items together) did demonstrate high internal item consistency. For clinical purposes, separate interpretation of two subscales makes prediction of future outcome more difficult.⁷ Because this is a tool that can be used both for clinical purposes and for research, we suggest using (1) a total raw score for clinical purposes of interpretation and (2) the Rasch-transformed arm and leg subscale measures (included in the manual⁶) with parametric statistical analyses when interval-level measures are needed for research. The separate interpretation of BabyOSCAR arm and leg subscales may also be useful in future comparisons with the Test of Arm Selective Control² and the Selective Control Assessment of the Lower Extremity,³ which measure SMC in the arm and leg separately.

In our sample, we found that the BabyOSCAR tool can infer an infant’s ability based on a continuum of skills (i.e. a high child and item reliability index) and, importantly, that the observed differences in ability groups were valid and free of bias (minimal DIF). Furthermore, we found that Rasch analysis separated the infants into three ability levels, consistent with our previous findings where BabyOSCAR scores were predictive of three groups we created on the basis of diagnosis at 2 years of age or older (no CP diagnosis, spastic CP classified in GMFCS level I or II, and spastic CP GMFCS levels III–V).⁷ Moreover, because BabyOSCAR test items differentiate infants’ abilities, the separation of test items into different difficulty levels may be particularly valuable in a quick screening of infants. For example, if an infant can achieve the hardest items (shoulder flexion/extension, knee flexion/extension, and hip flexion/extension), this is more likely to represent an infant with better SMC who might either not have CP or have CP classified in GMFCS levels I or II.

Because each item on the BabyOSCAR represents an independent movement that an infant can produce spontaneously, the hierarchy of items difficulty derived from Rasch analysis may have implications about the neurophysiological substrates that generate these movements and for guiding intervention. For instance, the items that were ranked as hardest included those requiring proximal joint movements against gravity, particularly hip and shoulder flexion. Maintaining isolated limb movements in an antigravity position requires more strength than a movement that can be completed while supported on a surface and may be more difficult for infants with medical complexity who already have increased energy expenditures²³ compared with their typically developing peers. Furthermore, children who have injury to the corticomotor regions of the brain have more difficulty performing isolated joint movements.^4,24,25 Therefore, antigravity movements that require recruitment of more muscle fibers, using more cortical resources, may overtax the limited resources of the impaired corticospinal system, requiring movements to be generated from subcortical structures that activate multiple joints at a time. This observation may be supported by the higher correlations found between joints identified as abnormal synergy patterns in children with CP (shoulder rotation and elbow flexion/extension).⁴

Interestingly, the most difficult BabyOSCAR leg item for infants was isolated knee extension and flexion. To receive credit for this item on BabyOSCAR, an infant needed to hold their hip in a stable antigravity position while extending their knee, requiring a dissociation of joints. Infants who are born preterm or at term with typical development develop this skill around the second to third month of post-term life;^26,27 however, infants with significant white matter injury^28–30 or who had been diagnosed with bilateral spastic CP³¹ were not able to demonstrate this skill when observed during spontaneous kicking activity. Notably, in our sample of 45 infants with spastic CP (n = 90 lower extremities), independent knee flexion/extension was rarely observed in paretic legs (n = 3, 3%). Additionally, two infants who were able to perform this item had unilateral CP classified in GMFCS level I and a history of focal brain injury due to ischemic stroke.⁷ The underlying injury type may be related to the ability to isolate joint movements because a more focal injury may allow for greater plasticity of intact corticospinal connections during a time of initial organization. The other BabyOSCAR leg item that was most difficult for infants was isolated hip flexion. Our findings support previous work which found significantly fewer isolated hip flexion movements in infants with bilateral spastic CP than in a comparison group without it.³¹ In summary, the hardest leg items on the BabyOSCAR test reflect known movements that are difficult for infants who have CP compared with infants without it.

The BabyOSCAR arm items showed a wider distribution of difficulty than leg items. This may reflect the stronger influence and greater number of projections from the cortex to the upper extremities for dexterous motor control of the human hand.¹ In addition, there may be a wider distribution of difficulty simply because there are more arm items representing the increased degrees of freedom available in the upper extremities. Because there are more items, and consequently more movement complexity, this may also explain the lower reliability in the arms compared with the legs.

CONCLUSION

The BabyOSCAR has strong psychometric properties and is a valid method to assess SMC in young infants considering a continuum of item difficulty and infant ability. This assessment shows promise as a prognostic indicator of impaired SMC and complements other clinical assessments^32,33 that are recommended in international guidelines for early diagnosis of CP³⁴ and inform evidence-based intervention in early development.

What this paper adds.

• Baby Observational Selective Control AppRaisal (BabyOSCAR) has excellent construct validity with good overall fit and precision.

• Individual BabyOSCAR items contribute and work well together, forming an interval-level assessment.

• BabyOSCAR has two separate subscales, arms and legs, that complement each other.

• BabyOSCAR’s items represent a continuum of skills with three distinct difficulty levels.

• BabyOSCAR’s continuum of skills reliably separates infants into three ability levels.

Abbreviations:

BabyOSCAR

Baby Observational Selective Control AppRaisal

DIF

differential item functioning

SMC

selective motor control