Brief Assessment of Motor Function: Content Validity and Reliability of the Upper Extremity Gross Motor Scale

Abstract

Content validity and reliability of the Brief Assessment of Motor Function (BAMF) Upper Extremity Gross Motor Scale (UEGMS) were evaluated in this prospective, descriptive study. The UEGMS is one of five ordinal scales designed for quick documentation of gross, fine and oral motor skill levels. Designed to be independent of age and diagnosis, it is intended for use for infants through young adults. An expert panel of 17 physical therapists and 13 occupational therapists refined the content by responding to a standard questionnaire comprised of questions which asked whether each item should be included, is clearly worded, should be reordered higher or lower, is functionally relevant, and is easily discriminated. Ratings of content validity exceeded the criterion except for two items which may represent different perspectives of physical and occupational therapists. The UEGMS was modified using the quantitative and qualitative feedback from the questionnaires. For reliability, five raters scored videotaped motor performances of ten children. Coefficients for inter-rater (0.94) and intra-rater (0.95) reliability were high. The results provide evidence of content validity and reliability of the UEGMS for assessment of upper extremity gross motor skill.

The Brief Assessment of Motor Function (BAMF) was developed in response to the need for criterion-referenced measures of motor skills that do not take long to complete and can be used to evaluate change over time or in response to an intervention. The BAMF consists of five separate scales. Four have been validated: a) Lower Extremity Gross Motor Scale (Cintas, Siegel, Furst, Gerber, 2003); b) Upper Extremity Fine Motor Scale (Parks, Cintas, Chou Chaffin, Gerber, 2007); c) Oral Motor Deglutition; and d) Oral Motor Articulation Scales (Sonies et al., 2009). Each is a hierarchical 0-10 item scale that evaluates performance capability, rather than limitations or impairments. One of the motivations for its development was to be able to provide a fast, reliable assessment of motor skill level, independent of age and diagnosis.

The BAMF scales are designed to provide a quick assessment of motor skill capability, rather than disability. Notably, the most recent disablement classification, the World Health Organization International Classification of Function and Disability (2009) represents progress in this same direction, with an emphasis on capability rather than limitations. The conceptual framework for the BAMF scales is based on what is generally accepted as a developmental progression. This progression assumes that lower levels are less difficult in terms of the need for strength, coordination, sequencing or synchrony of motion. We believe that this is the only way to construct a scale that offers a fast, reliable assessment of motor function, independent of age and diagnosis that is recognized as a hierarchy of competencies, rather than as a series of impairments. The BAMF scales are not linked to age and can be used for infants through young adults. Age and UEGMS scores are unrelated (r_S = 0.21) for infants, age two months, to adults, age 28 years.

The BAMF Upper Extremity Gross Motor Scale (Appendix) was designed to evaluate UE gross motor behaviors essential to elevating the body and moving on a support surface. Upper extremity gross motor skill is defined as multi-joint anti-gravity movements of the upper extremity, principally used to lift and propel other body segments. It includes activities which require large muscles, most frequently those of the arm and shoulder girdle. These muscles support mobility and transfer, such as crawling or pulling to stand, and moving objects in and out of the plane of the body. Upper extremity (UE) gross motor function has not typically been considered an independent domain for assessment and expands on the perspective that the role of the arm is to place the terminal device (the hand) so that it can perform optimally (Schott, Brenner & Smeets, 2010; Pehoski, 2005). The assumption our research team has made is that there is heuristic value in measuring UE gross motor function because it requires the coordination and effective mechanical linkage of the large muscles of the arm and shoulder girdle. These muscles support mobility and transfer, such as crawling and transfer from floor to upright; and reaching for, pulling and pushing objects out of the plane of the body. For example, infants bear weight on their arms to elevate their body off the floor for prone locomotion and to assume standing. Children, adolescents, and adults with lower body weakness depend on upper extremity gross motor skill for transfers and locomotion.

With the purpose of the BAMF Upper Extremity Gross Motor Scale in mind, four independent literature searches on the prone progression and upper extremity gross motor development were carried out by the first author, a full time occupational therapy summer research student, a medical librarian, and a medical informationist. Each used classical sources to find more contemporary publications.

Influenced by the work of McGraw, Horowitz and Sharby (1988) described the prone progression in 20 typically developing infants, intending to document cephalocaudal motor development. They were unable to do this because individual lower extremity (LE) extension behaviors typically preceded those of the UE. However, the serendipitous findings were useful. Forearm support correlated significantly with head elevation and LE extension behaviors, but no significant correlations were found between prone-on-hands postures and the development of capital, LE or UE extension behaviors. This suggests that head elevation and LE extension behaviors may be time-linked to early UE competence, but they are no longer linked by the time infants achieve quadrupedal support with the abdomen elevated.

McGraw (1941) recorded the prone progression in 82 infants using cinematography. She described nine phases beginning with capital elevation, progressing through palms-knees locomotion, and culminating with palms-feet locomotion. She reported considerable variation in the trajectory of ascent and age of onset of a specific skill. Some infants completed one phase before their age-mates even started it, and none of the infants who began with abdominal propulsion progressed directly to bipedal locomotion without first going through an abdomen-elevated locomotor stage. Adolf and colleagues (Adolph, Vereijken & Denny, 1998) reported the same finding in 28 infants during the interval from the onset of prone locomotion to walking. One infant never crawled, and proceeded directly to walking. The remaining 27 infants began locomotion by pivoting around an umbilical axis in prone. Thirteen of those infants progressed directly to crawling without abdominal contact. The remaining infants began with abdominal propulsion and progressed through palms-knees propulsion before walking, as previously reported by McGraw (1941).

VanSant and colleagues expanded McGraw’s approach across the lifespan, recording the UE, axial, and LE components of the transition from supine to standing with cinematography. They began with a cohort of 32 men and women, mean age 28.6 years, (VanSant, 1988a) whose UE support patterns consisted primarily of pushing straight up with the arms from supine to flex and elevate the trunk forward over the legs. However, 12% of the supine-to-standing transitions in these young, healthy adults included a bilateral UE prone support phase incorporating dynamic elbow extension. Based on these findings, they proposed the existence of a lifespan UE developmental sequence which includes this prone elevation strategy as a fundamental element.

VanSant (1988b) then applied the same approach to 120 children, 4-7 years, and 60 toddlers, 15-47 months (Marsala & VanSant, 1998). They observed the prone support pattern identified for adults, however, 50-60% of the children in each age group demonstrated more than one UE support strategy during 10 trials. The greatest variation occurred in the youngest children. This suggests a developmental continuum in which the UE prone support pattern is prevalent in youngest children (1-2 year olds) and is present, but occurs considerably less often in older children and adults. Marsala and VanSant also identified two linked UE-LE prone support patterns in toddlers that previously had been identified in middle and older adults (VanSant, 1990). Their data suggest that these patterns are evident at both ends of the lifespan as alternative strategies, representing increased reliance on the UE’s to compensate for trunk and LE weakness, lack of stability, or developmental immaturity.

Based on the studies reviewed, UE prone support behaviors, particularly forearm support and palm support, are most apparent in infants and younger children, but they also occur within transitions performed by older children and adults, increasingly in later adulthood. This suggests that the prone support behaviors which are fundamental to UE gross motor development represent a lifespan capability that can be described as a U-shaped developmental curve. The objectives of this prospective, descriptive study were to evaluate the content validity and reliability of the Upper Extremity Gross Motor Scale (UEGMS).

METHODS

Content Validity

When a gold standard does not exist to evaluate concurrent validity, establishing content validity is an appropriate first step in instrument development (Dunn, 1989). Evaluating content validity in the process of instrument development is encouraged by the standards for test and measurement development of the American Psychological Association (1999) and the American Physical Therapy Association (2010). The typical approach is to convene a panel of experts for a specific domain, provide them with a list of objectives and the test items, and elicit their feedback in a standard manner. The information is then used by the test developers to refine the content and/or the format of the assessment. Previous studies have validated this approach for motor skill assessment (Exner, 1993; Harris & Daniels, 1996; Haley, Coster & Fass, 1991).

We convened an Expert Panel consisting of 17 physical therapists and 13 occupational therapists with earned doctoral degrees and at least 20 years of experience in their field. All responded affirmatively to a standard email letter asking if they wished to participate in the content validity study for the UEGMS. None had previous exposure to the UEGMS. Each returned the questionnaire by the deadline. Their demographics are: 25 females, 5 males. All have earned doctoral degrees (26 PhD, 3 ScD, 1 DPT). Twenty-five have 25 or more years of professional experience, the remaining 5 have 20-24 years. Twenty-five are employed in university settings, 5 in other contexts, including private practice and research facility.

Expert Panel Members were asked to complete a questionnaire in which they were asked to rate the BAMF UEGMS by responding to six standard format questions for each of the 11 skill levels (Table 1). The range of possible responses were 1 = Disagree to 4 = Agree. Panel members were also invited, but not required to provide additional written comments directly on the questionnaire.

TABLE 1.

The Six Standard Questions for Each BAMF Item

		Disagree		Agree	Score
1. This item should be included	1	2	3	4	_________
2. The item is clearly worded	1	2	3	4	_________
3. Item should be reordered higher on scale     _______		1	2	3	4
4. Item should be reordered lower on scale	1	2	3	4	_________
5. This is a functionally relevant motor behavior	1	2	3	4	_________
6. This behavior is easily discriminated from others on the scale	1	2	3	4	_________

Reliability

Ten children (ages 11 months to 16 years; 7 females, 3 males; 5 Caucasian, 3 Black, 2 Hispanic) were videotaped performing upper extremity gross motor activities after informed consent for participation was obtained from their parents, and assent from children 7 years and older. One of the children whose motor skill level progressed over a several month interval during the study was videotaped a second time, generating 11 performances used for reliability assessment. For those children who could not follow directions, the environment was structured to elicit their best UE GM motor behaviors.

This purposeful sample was intentionally selected to represent a wide range of ages, diagnoses, and motor skill capabilities. Diagnoses include osteogenesis imperfecta, mucolipidosis type IV, achondroplasia, and Smith-Lemli Opitz syndrome. Individual performances were videotaped under the Rehabilitation Medicine Department’s IRB-approved protocol permitting data collection to promote instrument development. The performances were then randomly ordered on a master videotape for the purpose of scoring by the raters.

Following a standardized introduction to the objectives and format of the UEGMS, and the opportunity to ask questions prior to viewing the videotape, two occupational therapists, two physical therapists, and one physical therapy student independently rated the children’s videotaped performances on two occasions, two weeks apart.

Except for the physical therapy student, all raters were employees in the Rehabilitation Medicine Department of a large clinical research facility. They were selected to represent a broad range of expertise, anticipating use of the UEGMS by students and new graduates as well as experienced evaluators. Their experience in pediatrics ranged from a recent graduate to 20 years of clinical practice, including a board-certified pediatric therapy specialist. Once rating sessions were underway, no discussion among raters was permitted. However, raters were allowed to view performances on the video tape again if requested.

Videotaping performances may limit their generalizability to clinical environments in which assessments would not be videotaped. However, this was the means by which we could ensure all raters would see and evaluate exactly the same performance, given the difficulties of simultaneously assembling five raters and the child and family.

Data Analysis

Descriptive statistics were used to describe average agreement and range of responses for each item on the questionnaire. On the 1-4 rating scale for each BAMF item, a mean value (average agreement) of 3.0 or higher for standard statements 1,2,5,6 was considered high agreement. For standard statements 3 and 4 (this item should be reordered higher or lower), a mean value of 2.0 or lower was considered high agreement. Reliability was evaluated using Kendall’s W (coefficient of concordance) for inter-rater reliability (Zar, 1996) and Spearman’s Rho for intra-rater reliability.

RESULTS

Content Validity

For the question this item should be included, the mean ratings (average agreement) were 3.1 or higher for all items except items 4, 7, and 10. For the question this item is clearly worded, mean ratings were 3.2 or higher for all items except items 4 and 7. For the question this item should be ordered higher or lower on the scale, mean ratings were ≤ 2.1 for all items. For the question this item is functionally relevant, mean ratings were 3.2 or higher for all items except items 7 and 10. For the question this behavior is easily discriminated, mean ratings were ≥ 3.3 for all items.

Expert Panel members (EPM) were not required to provide written responses on the questionnaire, but were given the opportunity to do so. In some instances, EPM provided written feedback in lieu of a quantitative response. This item-specific feedback was helpful to refine the format and content of the UEGM items in addition to the quantitative feedback. Revisions based on the quantitative and qualitative feedback were incorporated into the scale prior to its use for the reliability trials.

Changes to refine the UEGMS were based on the quantitative data and in response to 117 written comments on the questionnaire according to the following predetermined criteria:

1. The change would result in describing more clearly the task identifier or the criterion behavior.

2. The change would increase the specificity of the task by modifying environmental factors (i.e., specifying the child’s body position in relation to the support surface).

3. The change would make the behavior more generalizable to all possible test environments.

4. The change would be suggested by at least two respondents.

Examples of Expert Panel Members’ comments are included in Table 2

TABLE 2.

Examples of Comments by Panel Members

UEGMS Item 1: Elevates head > 45° with evidence of forearm weight bearing in prone. Criterion: Elevates head more than 450 while bearing weight on one or both forearms.
Rater 2: “ Delete ‘placed in prone’ from criterion and add ‘in prone’ to the end of the descriptor.	Rater 9: “Supports weight on one or both forearms in prone lying?”

UEGMS Item 7: Fully elevates trunk in sitting by extending elbows Criterion: In sitting, arms at sides, extends elbows to lift body off support surface so there is no contact between the pelvis and the support surface.
Rater 6: “This seems like pushups in sitting; I don’t know how functionally relevant this is.”	Rater 9: “This item is a task that emphasizes UE weight support. I think a good item.”

UEGMS Item 10: Pushups	Criterion: Completes five full body pushups in prone, elbows and knees extended completely.
Rater 5: “Good item if emphasis UE strength.”	Rater 6: “Have not seen this item on other developmental assessments.

Reliability

Raw values, range, means and standard deviations for reliability are reported in Table 3. Ratings from the first session were used to calculate inter-rater reliability (Kendall’s W = 0.94, p < .0001). Ratings from the first and second sessions were used to calculate intra-rater reliability (Spearman’s Rho = 0.95, p < .0001).

TABLE 3.

Raw Data and Descriptive Data for Inter-rater and Intra-rater Reliability of the BAMF Upper Extremity Gross Motor Scale

First Rating Session – Inter-rater Reliability Kendall’s W =.0.94 (p < .0001)
Performance Score
Rater	1	2	3	4	5	6	7	8	9	10	11
1	9	8	5	2	1	0	7	3	6	4	10
2	9	8	5	2	3	0	7	3	6	4	10
3	9	8	5	2	1	0	7	3	6	4	10
4	9	8	5	2	1	0	7	3	1	4	10
5	9	8	5	2	1	0	7	3	1	4	10

First & Second Rating Session – Intra-rater Reliability Spearman r = 0.95, p < .0001
Performance Score
Rater	1	2	3	4	5	6	7	8	9	10	11
1	9	8	5	2	1	0	7	3	6	4	10
2	9	8	5	2	3	0	7	3	6	4	10
3	9	8	5	2	1	0	7	3	6	4	10
4	9	8	5	2	1	0	7	3	1	4	10
5	9	8	5	2	1	0	7	3	9	4	10

DISCUSSION

Since no gold standard exists for UE gross motor skill, establishing concurrent validity with an existing instrument was not an option for upper extremity gross motor assessment. Thus, we selected an Expert Panel to provide systematic feedback to refine the instrument. We chose physical therapists and occupational therapists with 20 or more years of experience. High agreement among the raters occurred for 59 out of 66 questions. Among the remaining seven questions, two relate to clarity of the wording of specific items, and suggestions were easily incorporated. The other questions where agreement among experts did not reach the criterion pertain to the inclusion and functional relevance of items 7 and 10.

Item 7 describes the ability to elevate the trunk off the support surface in sitting by extending the arms downward and bearing weight on the hands. Item 10 describes the ability to perform five prone push-ups with extended elbows, hips and knees. Item 10 generated considerable feedback from Expert Panel Members. Several respondents questioned its inclusion, stating that it is not included in conventional developmental schedules. Actually, it is item 30 in the PDMS-2 (Folio & Fewell, 2003); the highest skill listed for the Stationary (non-locomotor) cluster.

During development of the BAMF UEGMS, we solicited input from intramural and extramural colleagues and considered several possibilities for Item 10. After multiple reviews, two remained for further consideration in addition to full body extension pushups: use of the arms to elevate the body to standing, and the ability to fully support the body over the arms by doing a hand stand. The former is a functional life skill, part of the normal progression to standing for typical children. However, the children who inspired development of the UEGMS may have strong arms, but insufficient leg and/or trunk strength to stand. For these children, the ability to push up to standing is not an index of UE gross motor function. The second option, a hand stand sustained for a standard period of time, is a “pure” and easily quantified challenge of UE gross motor function, but it is not generalizable to transfers or other essential life skills. It also poses an undesirable risk of falling. Therefore, the outcome of this process was the selection of full body extension push-ups for item 10, representing a high level of UE performance conceptually analogous to running for the lower extremity. We remain open to and welcome other suggested possibilities and anticipate pursuing this further using Rasch methodology (Bond & Fox, 2001).

Based on our experience, the information provided by the UEGMS may be particularly useful for children with transfer difficulties due to UE, LE or trunk weakness. In a scoring time of 2 minutes or less, it provides information on what the child can do, and the potential for achieving independence in transfers. It also provides information on prone locomotor capability and the potential for UE assisted upright ambulation that can be directly linked to specific clinical interventions. The UEGMS can be used independently in this fashion, but it may be valuable to use it in conjunction with more comprehensive assessments. For example, an infant with a storage disease such mucolipidosis-type IV can be followed longitudinally with the Peabody Developmental Motor Scales-2 and the BAMF UE and LE Gross Motor Scales. The BAMF scales provide immediate information on the child’s best demonstrated skill in these two motor skill domains at each testing event, while the PDMS-2 provide norm-based, comprehensive information across a range of skills through age eighty-three months.

To follow this child through age 18 years in a natural history study, another comprehensive assessment would be required at age 83 months, possibly the Gross Motor Function Measure, a criterion-based assessment across five dimensions of gross motor skill (Russell, Rosenbaum, Avery & Lane, 2002). The Bruininks-Oseretsky Test of Motor Proficiency, 2^nd Edition (Bruininks & Bruininks, 2005), with significant emphasis on hand-eye coordination and balance, is another possibility for a child with high gross motor function who can follow directions. Linking assessments in this way provides consistency over time and the rapid retrieval of longitudinal information from the BAMF scales with the comprehensive information available from age-interval scales which evaluate skills in multiple domains.

Limitations

Some respondents questioned the inclusion of the items Elevating the body in sitting by extending the arms (Item 7) and Full body extension pushups (Item 10), citing their absence in measures of child development. For many children with movement disorders, however, the ability to elevate the core body mass is the portal to transferring in and out of a wheelchair, and scooting among different support surfaces. The ability to perform 5 full body extension pushups also generated considerable feedback from Expert Panel Members. The study did not assess change over time and so the sensitivity to change of the UEGMS was not examined. This is an objective for further research.

CONCLUSION

An Expert Panel composed of 30 physical therapists and occupational therapists provided feedback to refine the content of the BAMF UEGMS. The majority of items were rated as easily discriminated, clearly worded, appropriately ordered, functionally relevant. Although not fully supported by expert consensus, the items Elevating the body in sitting by extending the arms and Full body extension pushups were retained because Item 7 is the basis for transfer behaviors for non-ambulatory individuals and Item 10 represents a challenge conceptually analogous to running for the lower extremities. Intra-rater and inter-rater reliability of the UEGMS were high. The results provide evidence of content validity and reliability of the UEGMS. We recommend use of the UEGMS when the objective is rapid, reliable, determination of the individual’s best upper extremity gross motor performance with minimal dependence on vision and balance.

APPENDIX

Brief Assessment of Motor Function (BAMF)

Upper Extremity Gross Motor Scale

Holly Cintas PT, PhD, Rebecca Parks MS, OTR/L, Lynn Gerber MD

Rehabilitation Medicine Department, National Institutes of Health, Bethesda, MD

Score = highest observed item completed

Example: Child can crawl hands and knees, but cannot lift body weight using arms in sitting: Score = 9

Example: Child can commando crawl, but cannot elevate head > 45 degrees: Score = 5

Item	Criterion
O: Head elevated < 45 degrees, little or no forearm weightbearing in prone	Elevates head less than 45 degrees, with minimal or no forearm weightbearing, in prone
1: Head elevated > 45 degrees, with evidence of forearm weightbearing in prone	Elevates head more than 45 degrees while bearing weight on one or both forearms in prone
2: Fully extends elbows to grasp an object while in supine	Extends both elbows fully to grasp an object positioned in midline above the nose, in supine
3: Shifts weight laterally in prone in order to reach	Shifts body weight to one side, freeing opposite arm to reach toward an object, positioned within reaching distance, in prone
4: Rotates around umbilical axis in prone	Using arms, rotates body to the right or left at least 30 degrees in prone position
5: Commando Crawl	Locomotes 5 feet forward or backward in prone, abdomen in contact with support surface
6. Weightbearing on extended arms	Extends elbows fully to elevate body while in prone position
7. Fully elevates trunk by extending arms in sitting	In sitting, arms at sides, extends elbows to lift body off support surface so there is no contact between the pelvis & the support surface
8. Maintains static quadruped position	Sustains hands-knees weightbearing position with abdomen elevated above the support surface for 15 seconds
9: Crawls hands, knees	Crawls at least 3 feet with trunk elevated off the support surface, using hands and knees, pattern may be reciprocal or symmetrical
10: Pushups	Completes 5 full-body pushups in prone, elbows and knees extended completely

Inclusion Criteria for all BAMF items: Gross Motor, Fine Motor, and Oral Motor Subscales

1. Every task represents observed performance

2. Performance can be judged unequivocally as present or absent

3. Tasks are adaptive behaviors typically used in daily function across cultures

4. Items chosen to test behaviors are readily available in all potential test settings

5. Lowest and highest items on each scale represent, respectively, most primitive and most advanced levels of performance for that domain

6. Hierarchical progression of motor behaviors is designed to be independent of age