The Pediatric/Adolescent Shoulder Survey (PASS): A Reliable Youth Questionnaire With Discriminant Validity and Responsiveness to Change

Abstract

Background:

Health-related quality of life tools for shoulder pathology in adults have been developed, such as the Disabilities of the Arm, Shoulder, and Hand (DASH) or QuickDASH. However, some content is not applicable to or readily grasped by a younger population.

Purpose:

To evaluate the psychometric properties of a new 13-item pediatric/adolescent shoulder survey (PASS).

Study Design:

Cohort study (diagnosis); Level of evidence, 2.

Methods:

An initial development cohort of children was given the PASS, QuickDASH, and single assessment numeric evaluation (SANE) in a prospective fashion. Internal reliability and concurrent validity were evaluated. Subsequent to this, the PASS, along with the QuickDASH, was incorporated into routine clinical evaluation of shoulder injuries. The secondary cohort was established via a retrospective review of patients (age <18 years) evaluated in our sports clinics for shoulder complaints. Discriminant validity and floor/ceiling effects were evaluated. A subset of patients with scores available both pre- and within 6 months postoperatively was analyzed to assess responsiveness to treatment.

Results:

Overall internal reliability of the shoulder survey was good in the development cohort of 127 patients, with Cronbach α = 0.86. Total score averaged 59.7% ± 17.9% in this cohort (range, 18.75%-100%). Significant correlations between all surveys were observed (r = 0.51-0.79, P < .05). A total of 127 surveys were available for the secondary cohort. The QuickDASH demonstrated significant differences between sexes with regard to scores (P < .05). Both tools were able to discriminate between patients with acute injuries and diminished clinical examinations (P < .05). A significant correlation was observed between duration of symptoms and scores on the PASS. The subset of 25 patients with pre- and postoperative data showed significant change in total score and the symptoms/limitations domain of the PASS, with significant correlation with length of time since surgery.

Conclusion:

This 13-question pediatric/adolescent-focused shoulder survey shows excellent psychometric properties, good discriminant ability, and early evidence of responsiveness to change, especially when compared with an adult-formatted questionnaire.

Safely and effectively caring for children requires specialized training for clinicians, adjustments to medication and pharmacology, and altered devices and medical supplies to account for dramatic size variations. Similarly, assessing health-related quality of life in a patient population with such wide variation in cognitive understanding poses unique challenges.^9,11,18,20 Age appropriateness of the content and formatting of the items on the outcome tool must be considered in developing pediatric patient-reported outcome tools (PROs). Utilizing PROs developed for adults, like the Disabilities of the Arm, Shoulder, and Hand (DASH) outcome measure and its abbreviated version (QuickDASH), in a pediatric population could result in unreliable data that have limited content validity or applicability. A lack of sensitive PROs for pediatric patients with shoulder injuries could hinder advances in treatment of such injuries and could also put providers at a disadvantage within the new payment model proposed by the Department of Health and Human Services.

While there are validated PROs specific to upper extremity function in existence (DASH and QuickDASH), the general indications for use are in patients aged 18 to 65 years,^3,14,16,19 and the reading level is just under the 10th grade. Despite the lack of validation in a younger population, the QuickDASH has been utilized in pediatric upper extremity research due to its ease of administration and availability.^2,6,25 Recently, Quatman-Yates et al²⁴ evaluated the internal consistency and validity of the QuickDASH in a population of 8- to 18-year-old patients with upper extremity injuries. While overall the tool demonstrated good internal reliability and concurrent validity, 5 items of the tool demonstrated bimodal distribution, and the interitem correlations ranged from low to moderately high (0.24-0.68). This suggests lack of uniform comprehension of some of the items on the QuickDASH in this pediatric/adolescent population.

To assess and predict outcomes of upper extremity treatment in the pediatric/adolescent population accurately, it is critical to utilize a reliable and validated tool that is sensitive to the activity level of young patients of both sexes. The purpose of this study was to develop and establish internal reliability and concurrent validity of a pediatric/adolescent shoulder survey (PASS) in a developmental cohort. The secondary purpose was to evaluate the discriminant validity and responsiveness to change in a follow-up cohort. The hypothesis of the study was that a dedicated outcome questionnaire designed for assessing shoulder injuries in children would have better concurrent and discriminant validity than an adult age–based questionnaire.

Methods

Developmental Cohort

Patients presenting to our freestanding children’s hospital for evaluation, diagnosis, and/or treatment for chief complaints related to the shoulder were asked to participate. Packets were given to each family and included the newly developed shoulder survey (PASS), the QuickDASH (including the sports and work modules), and the single assessment numeric evaluation (SANE). They were also given a second packet, with a stamped addressed envelope, which they were instructed to complete and return 2 weeks after completion of the initial packet. Two-week delays were used because the children would not have had surgery or started physical therapy prior to that time in our local system. Therefore, there would not be iatrogenic changes in their reported scores on the outcome measures. The newly developed shoulder survey consists of 13 questions that assess (in child-friendly language) symptoms, limitations, need for compensatory mechanisms, and emotional distress related to shoulder dysfunction (Figure 1). The PASS was developed because most of the adult-age questionnaires ask questions that are not age appropriate (carrying a briefcase, sexual activity, etc). Word choice and grammar were set at the fourth-grade reading level after combining the concepts for shoulder pathology found in the DASH score and the Western Ontario Shoulder instability score. Other elements that were added included concepts from other activity-related scores (University of California–Los Angeles [UCLA] and Marx) since this age group places a heavy interest in this realm. After a first draft of the instrument was compiled and written by a committee of pediatric orthopaedic surgeons who specialize in sports medicine, it was then sent to our institution’s institutional review board committee, who also made changes to better reflect the population being questioned and improve readability.

Figure 1.

The pediatric/adolescent shoulder survey (PASS).

Responses were provided on a 0 to 5 or 0 to 10 scale, with a score calculation normalized to the number of total possible points (100 indicates no/minimal impact on quality of life from shoulder dysfunction). The direction of the responses was varied among the questions to allow for identification of response sets. Items 1 through 9 are reverse scored, such that a response of 1 is recorded as a response of 10 (on a 10-point item) or 5 (on a 5-point item) and vice versa (Table 1). For example, question 1 asks the patient to rate their pain on a scale of 1 (no pain) to 10 (extreme pain). This is reverse scored to fit the overall schema of 100 points being full function (ie, a score of 1 for no pain would count as 10 points and a score of 10 for extreme pain receives 1 point). Questions 11 through 13 are not reverse scored, and the actual circled response is utilized in calculating the subject’s domain and overall scores. For this study, data were entered into Excel (Microsoft) as the subject responded, and the VLOOKUP function was utilized to convert the scores for items 1 through 9. VLOOKUP is a formula in Excel that allows for automation of this reverse scoring and is characterized as a lookup/reference function. It scans a table of preset values so that the appropriate value is returned to the cell to correspond with the value you are looking up. This automates the process of determining what the reverse score should be and eliminates the need for individual recoding of scores. After this, the reverse scores for 1 to 9 and 10 through 13 are summed. The total summed score is then divided by the total number of possible points, and a total score is generated. The alteration in directionality assists in identifying acquiescence bias in responses by the subject/patient.^17,23

TABLE 1.

Reverse Scoring System for Items 1 Through 9 and Tables Used for VLOOKUP Function^a

Items 1, 3, 5, 6, 7, 8		Items 2, 4, 9
Actual Response	Reverse Score	Actual Response	Reverse Score
1	10	1	5
2	9	2	4
3	8	3	3
4	7	4	2
5	6	5	1
6	5
7	4
8	3
9	2
10	1

^aOnce reverse scoring is applied to items 1 through 9, the reverse scores from items 1 through 9 are summed together with the actual scores from items 10 through 13. If all 13 items are completed, the formula for total score would be as follows: =SUM(rev1-9, 10-13)/100.

Analysis of Internal Reliability and Convergent Validity

Cronbach α coefficients were utilized to determine internal consistency reliability of the items in the new shoulder questionnaire. Item-item correlations were also evaluated to identify any potential items that do not contribute to the overall reliability of the questionnaire. Factor analysis in the form of principal components analysis (PCA) was undertaken to identify relevant domains within the questionnaire. Domain scores are calculated similar to the total score (summation of points divided by total number of possible domain points). Varimax (orthogonal) rotation was performed to identify optimal independent factors within the components of the survey. A general rule of thumb²⁷ is that an absolute loading of >0.3 indicates that the item is relevant to that factor; however, final selection is chosen based on the largest loading. Change in scores during test-retest was assessed by comparing average scores from time 1 to time 2, and test-retest reliability was assessed using intraclass correlation coefficients (ICCs). Guidelines for interpreting ICCs were taken from Cicchetti’s criteria: less than 0.40 = poor, 0.40 to 0.59 = fair, 0.60 to 0.74 = good, and 0.75 to 1.00 = excellent.⁷ Pearson correlation coefficients were utilized to establish convergent validity with the existing tools.

Secondary/Validation Cohort

After determining that the PASS had internal reliability, the PASS and the QuickDASH were added to routine care for patients with appointments at our orthopaedic sports clinic for initial complaints or follow-up of shoulder injuries. The validation cohort was established via a separate institutional review board protocol, which involved retrospective review of patients who had completed the PASS during at least 1 visit. Total score of the PASS was calculated as well as domain scores. A maximum score of 100 indicates full function. A total score was calculated for the QuickDASH and QuickDASH sports module. A maximum score of 100 points indicates total disability. Ceiling and floor effects were evaluated by identifying the percentage of patients with the top and bottom scores in both tools. Ceiling/floor effects are determined to be present if greater than 15% of the cohort reports the highest/lowest score.²⁸ Discriminant validity was evaluated based on acute injury, duration of symptoms, and diminished range of motion (ROM) or strength on clinical examination as available in the charts. Acute injury was defined as a first-time visit for a recent-onset (<4 weeks) problem with the shoulder. Diminished ROM was defined as reduction in any shoulder ROM measure of at least 10° as compared with the contralateral side, and diminished strength was defined as a decrease in strength of 1 point compared with the contralateral side in at least 1 or more strength measure available (supraspinatus, infraspinatus, subscapularis, or teres major; 5-point motor score). ROM was determined by the treating surgeons independent of this study but acquired after the visit from the chart review by an independent researcher. A subset of patients had scores available both preoperatively and within 6 months postoperatively that were evaluated to assess responsiveness to treatment.

Analysis of Discriminant Validity and Responsiveness to Change

Discriminant validity was analyzed either via Pearson correlation or analysis of variance (ANOVA) based on the format of the discriminating variable (continuous discriminators such as age and duration of symptoms were analyzed via Pearson correlation; categorical discriminators such as diminished ROM/strength as a yes or no were analyzed via ANOVA). Cohen’s suggestion for interpreting correlation values were utilized: 0.10 indicates small effect, 0.3 indicates a moderate effect, and 0.5 indicates a large effect.⁸ Responsiveness to change was analyzed using repeated-measures ANOVA. All assumptions underlying the parametric statistics were evaluated (normality for all and homogeneity of variances for ANOVA). Alpha was set at P < .05 to declare significance. All statistics were performed using SPSS version 12 (IBM Corp).

Results

Developmental Cohort

A total of 132 families agreed to have their child participate. Five children (3.8%) were missing responses to almost half of the survey (6 questions, 46%) and were excluded, resulting in 127 surveys available for analysis. Total mean PASS score was 59.7 ± 17.9 in this cohort (range, 19-100). Overall reliability of the shoulder survey was good, with Cronbach α = 0.86. Individual question item-total correlations ranged from 0.37 to 0.72. Two items yielded scores less than 0.40; however, their removal did not decrease the overall alpha. A subset of 17 children returned the second set of surveys. The mean PASS score was not significantly different from time 1 (57.5) to time 2 (57.6) (P = .98), and the ICC was excellent at 0.75. Similarly, the mean QuickDASH score was not significantly different from time 1 (32.4) to time 2 (31.6) (P = .81), and the ICC was excellent at 0.85. PCA analysis demonstrated that the survey questions were loaded (or grouped) on 2 factors/domains. Initially, 7 questions were loaded on factor 1 and 6 questions on factor 2. However, for the final question, the difference in loading between factors was not large (factor 1, 0.46; factor 2, 0.36). In examining the constructs by hand rather than via PCA analysis, question 13 appeared to more closely relate to the constructs assessed by items in factor 2 and it was therefore grouped with those questions that were more strongly loaded on the PCA for factor 2 for the next analysis.

Questions 4 through 6 and 10 through 12 were loaded on factor 1, which addressed compensatory mechanisms and function related to the shoulder. Analysis of this combination of questions indicated ideal internal reliability for a compensatory mechanism/function domain (Cronbach α = 0.82). The remaining questions (1-3, 7-9, and 13) comprised the symptom/social-emotions domain (Cronbach α = 0.74). Significant correlations for total and both domain scores of the shoulder survey were observed with the QuickDASH score, sports module score, work module score, and SANE scores (Table 2).

TABLE 2.

Pearson Correlateion Coefficients (r) for Total and Domain Scores of the Shoulder Survey With Existing Questionnaire Scores in the Development Cohort^a

	PASS Total Shoulder Score	Symptom/Limitation Domain Score	Compensatory Mechanism/Emotional Domain Score
QuickDASH total	−0.79	−0.71	−0.71
QuickDASH sports	−0.65	−0.51	−0.69
QuickDASH work	−0.69	−0.62	−0.63
SANE	0.61	0.55	0.54

^aAll r values significant, P < .05. QuickDASH, abbreviated Disabilities of the Arm, Shoulder, and Hand outcome measure; SANE, single assessment numeric evaluation.

Secondary/Validation Cohort

Our secondary chart review identified a new set of 132 children between January 2013 and April 2014 who completed the shoulder outcome packets as part of routine clinical care. Of these, 73% completed the survey at their first visit for the shoulder complaint and therefore had not yet undergone any treatment at the time of survey completion. Sixteen percent were undergoing physical therapy or a period of recommended rest, and 11% were postoperative. The cohort was 70% male, with a mean age of 16 ± 2 years (range, 12-19 years). Mean scores for both questionnaires are seen in Table 3. No patients in the cohort responded with the minimum total score of 0 or maximum total score of 100 on the PASS. Unimodal distributions were observed on all questions. Two patients responded with the minimum score on the QuickDASH (0, or no disability). On the QuickDASH sports module, 29 patients (33%) responded with a score of 100, which is indicative of total disability and meets criteria for ceiling effect; 3 patients responded with the minimum score of 0. For the PASS, 79.5% of patients completed all 13 questions (range, 11-13 questions), and for the QuickDASH, 72.7% of patients completed all 11 questions (range, 6-11 questions) (binomial probability, P = .012).

TABLE 3.

Descriptive Statistics for the PASS and QuickDASH Tools in the Validation Cohort^a

	Minimum	Maximum	Mean	SD
PASS score, %	21	99	57	17
Symptoms/emotions domain	20	100	63	19
Compensatory/function domain	13	100	55	21
QuickDASH score	0	92.5	36.3	22.3
QuickDASH sports moduleb	0	100	68.6	30.9

^aPASS, pediatric/adolescent shoulder survey; QuickDASH, abbreviated Disabilities of the Arm, Shoulder, and Hand outcome measure.

^bDue to symptom limitations or physician recommendations, many patients indicated lack of participation in sports due to their shoulder and did not complete the sports module of the QuickDASH (n = 87).

The QuickDASH demonstrated significant differences between sexes with regard to scores, with female respondents having a higher mean score (more disability) than males (P < .05, Table 4). There were no significant differences between sexes on the PASS total score or domains. Both tools were able to discriminate between patients with acute injuries and diminished clinical examinations (P < .05). A significant small-medium effect correlation was observed between duration of symptoms and scores on the PASS (0.23, P < .05); however, the correlation did not reach significance for the QuickDASH (r = −0.13, P > .05). A subset of 25 children had both pre- and postoperative surveys. The time since surgery ranged from 3 days to 16 weeks postoperative. The results demonstrated a significant mean improvement in total score and 1 domain (symptoms/emotion) of the PASS (P < .05, Table 4). All 3 (PASS, QuickDASH, and QuickDASH sports) showed a significant correlation with length of time since surgery (Table 4).

TABLE 4.

Discriminant Validity and Responsiveness to Change for the Shoulder Surveys^a

	PASS Total Score	Symptoms/Emotion Domain Score	Compensatory Mechanism/Function Domain	QuickDASH	QuickDASH Sports
Sex
F (n = 39)	55 ± 17	59 ± 18	50 ± 19	45 ± 23	78 ± 27
M (n = 93)	58 ± 17	64 ± 17	51 ± 19	33 ± 21	65 ± 32
P	.34	.11	.81	.006	.069
Acute injury
Yes (n = 33)	53 ± 16	58 ± 17	46 ± 18	44 ± 28	72 ± 35
No (n = 88)	59 ± 16	64 ± 17	54 ± 17	33 ± 19	67 ± 30
P	.048	.12	.03	.025	.61
Duration of symptoms
r	0.23	0.15	0.28	–0.13	–0.18
P	.01	.12	.002	.2	.11
Diminished ROM or strength
Yes (n = 75)	54 ± 15	59 ± 16	48 ± 17	41 ± 21	76 ± 27
No (n = 47)	63 ± 17	67 ± 17	58 ± 18	26 ± 18	54 ± 30
P	.018	.011	.001	.001	.001
Responsiveness to change
Preoperative	56 ± 16	61 ± 17	50 ± 17	33 ± 21	75 ± 30
Postoperative	67 ± 19	77 ± 18	56 ± 21	27 ± 29	55 ± 33
P	.015	.001	.13	.44	.17
Time since surgery
r	0.59	0.46	0.6	–0.47	–0.59
P	.002	.033	.003	.02	.02

^aSignificant findings are in boldface. F, female; M, male; PASS, pediatric/adolescent shoulder survey; QuickDASH, abbreviated Disabilities of the Arm, Shoulder, and Hand outcome measure; ROM, range of motion.

Discussion

Our hypothesis was partially upheld in that the PASS had overall discriminant reliability similar to the QuickDASH for this pediatric population, but in contrast to the adult-focused outcome questionnaire, the PASS did not demonstrate significant differences between sexes in those scores. Both tools were able to discriminate between patients with acute injuries and diminished clinical examinations (P < .05). A significant correlation was observed between duration of symptoms and scores on the PASS. The pre- and postoperative data showed significant improvements in total score and the symptoms/emotions domain of the PASS after surgery.

Over the past few decades, significant efforts have been made within various fields of medicine to assess and quantify patients’ perceptions of their well-being and function as it relates to their health. Standardized evaluations of health-related quality of life were historically utilized largely for research purposes; however, advances in patient-centered care and a culture shift toward value-based reimbursement models have placed increased importance on PROs in the clinical setting.¹⁰ A press release in 2015 by the US Department of Health and Human Services announced the goal of linking 85% of Medicare fee-for-service payments to quality or value and 30% of Medicare payments through alternative payment systems by the end of 2016.^4,29 The same release announced a plan to expand these goals beyond Medicare, to the private sector, with the creation of a new network. The motivation behind this shift is to reduce costs, improve outcomes, and include more of a consumer focus in healthcare delivery by associating “rewards” with positive patient outcomes.

While Quatman-Yates et al²⁴ recently found the QuickDASH to have internal reliability in patients aged 8 to 18 years with upper extremity injuries, several limitations of this score were identified in younger patients. First, 5 of the 11 items on the tool demonstrated a bimodal response distribution, which could potentially indicate discrepancy in how the respondents interpreted the question. Second, the range of item-total correlations found in their study on the QuickDASH had a minimum value of 0.24. A small value suggests little relationship between that item and others in the tool, suggesting that it is measuring a different construct.¹² It has been suggested that items with an item-total correlation less than 0.30 should be dropped from the outcome tool.^12,13,26 The current study identified several other limitations of the QuickDASH, including its ceiling effect and the fact that some items on the QuickDASH are of questionable relevance within a pediatric population. For example, it was noted on occasion that respondents left QuickDASH items blank with a note that the question did not apply to them or their child (ie, wash your back, carry a shopping bag or briefcase, and use a knife to cut your food). For these reasons and others, we believe it was necessary to develop a better shoulder score for this young population. An example of our improvement was that all 13 items and item-total correlations were greater than the 0.30 threshold for the new PASS score. We were also able to establish 2 different factor loadings, ultimately representing 2 different domains of assessment related to impacts to quality of life due to shoulder impairment. Even though not every child completed the entirety of the PASS test, none of the surveys had free text comments written in by the patient/parent on the form citing a lack of relevance to them (or their child) as the reason for omission, in contrast to the QuickDASH.

The ability to discriminate between symptomatic and nonsymptomatic patients is a critical component of a PRO (discriminant validity). Both the PASS and the QuickDASH demonstrated the ability to discriminate between patients who were acutely injured and those with an abnormal clinical examination, assessed by a decrease in range of motion or strength compared with the uninjured arm. However, the QuickDASH also discriminated between male and female respondents, with females reporting greater disability. While differences between sexes are not an unexpected reality in outcomes tools,^1,22 their presence is not optimum when data are to be aggregated for the purposes of performance measures and will complicate analyses associated with research studies with mixed-sex cohorts. If this finding of sex discrepancy on the QuickDASH existed in solitude, it would be reasonable to explore the possibility of true differences between sexes regarding function due to shoulder injury. However, given the evidence to suggest misinterpretation of some items of the QuickDASH in the youth population and a lower rate of 100% completion of the tool as compared with the PASS, it is possible that these differences are a reflection of differing social and cognitive development between the sexes in this age group and simply a further extension of lack of comprehension or applicability of items in a youth population.^5,21 Further research into evaluating normative values in a large population of males and females would help in interpreting scores and score differential based on sex.

Responsiveness to change is another important psychometric property of an outcome tool and it may in fact be tied to validity.¹⁵ When an outcome tool demonstrates properties that indicate it is indeed measuring what it was intended to measure, then the tool is deemed valid. In the case of the outcome tools examined in this study, shoulder function and disability was the operational definition of the outcome of interest. The expectation of the instruments was such that any intervention aimed at improving shoulder function/ability would produce a resultant change in outcome scores. The PASS did demonstrate statistically significant improvements in mean score after operative intervention, further providing evidence of its validity and ability to detect change in children receiving treatment for shoulder ailments. The clinically significant changes in the PASS outcome will need to be studied for individual shoulder pathology now that the outcome score has been validated.

One limitation of this study was the utilization of a heterogeneous population. Patients were being seen for a multitude of shoulder ailments at various phases of treatment. The number of patients with both pre- and postoperative completed surveys was limited. A second limitation to the study is not knowing whether the overall health status of the patients had changed in the 2-week duration of the test-retest assessment because we were blinded to patient identity. A third limitation is that our validation cohort was represented by children aged 11 years and older; thus, the generalizability to children younger than 11 years is yet to be determined. However, the results of this 2-part study indicate the PASS has good internal reliability with evidence of concurrent and discriminant validity and is a viable, youth-focused alternative to assessing shoulder function and disability. The PASS also demonstrated appropriate sensitivity to postoperative changes. Future research in a larger, more homogenous operative population, with appropriate anchoring, is needed to develop an understanding of thresholds of meaningful or minimally clinically important differences in PASS scores, and further exploration of the applicability to a younger patient cohort is warranted.

Conclusion

The PASS is a 13-question shoulder survey specifically developed for the pediatric/adolescent population that shows excellent psychometric properties, internal reliability, concurrent validity, discriminant validity, and early evidence of responsiveness to change. The lack of perfect correlation (1.0) with the QuickDASH score and even lower magnitude correlation with the sports and work modules indicate that while the tools are measuring a similar construct, there is adequate divergence to justify utilization of this pediatric outcome tool. Moreover, the PASS is not affected by sex (or other demographic variables) concerning validity in reporting patient outcomes and demonstrated a higher rate of total completion. Utilization of a reliable shoulder-specific outcome tool specifically designed and validated in a pediatric/adolescent population may provide more sensitive and specific information related to the impact of treatment in this younger cohort.