Can the Kerlan-Jobe Orthopaedic Clinic Shoulder and Elbow Score Be Reliably Administered Over the Phone?: A Randomized Study

Abstract

Background:

The Kerlan-Jobe Orthopaedic Clinic (KJOC) shoulder and elbow outcome score is a functional assessment tool for the upper extremity of the overhead athlete, which is currently validated for administration in person.

Purpose/Hypothesis:

The purpose of this study was to validate the KJOC score for administration over the phone. The hypothesis was that no difference will exist in KJOC scores for the same patient between administration in person versus over the phone.

Study Design:

Cohort study (diagnosis); Level of evidence, 2.

Methods:

Fifty patients were randomized to fill out the KJOC questionnaire either over the phone first (25 patients) or in person first (25 patients) based on an a priori power analysis. One week after the patients completed the initial KJOC on the phone or in person, they then filled out the score via the opposite method. Results were compared per question and for overall score.

Results:

There was a mean ± SD of 8 ± 5 days between when patients completed the first and second questionnaires. There were no significant differences in the overall KJOC score between the phone and paper groups (P = .139). The intraclass correlation coefficient comparing paper and phone scores was 0.802 (95% CI, 0.767-0.883; P < .001), with a Cronbach alpha of 0.89. On comparison of individual questions, there were significant differences for questions 1, 3, and 8 (P = .013, .023, and .042, respectively).

Conclusion:

The KJOC questionnaire can be administered over the phone with no significant difference in overall score as compared with that from in-person administration.

Athletic injuries to the shoulder and elbow are increasing in frequency, especially among adolescent pitchers.^{9,10,15–18,29,30} The frequency of surgical treatment has also increased.⁸ With the rise in the number of surgical procedures performed among adolescents and adults, it is imperative that the results of these operations be scrutinized. Standardized patient-centered outcome scores are one method whereby the outcomes of various procedures can be compared and contrasted, allowing surgeons to measure their success and compare and communicate these outcomes among practices.^{1–3,12,21,26} This allows surgeons to understand how successful or unsuccessful a certain treatment is. The Kerlan-Jobe Orthopaedic Clinic (KJOC) Shoulder and Elbow Score is one of the most widely used validated clinical outcome measures specific to the shoulder and elbow of athletes, especially overhead athletes (Appendix). This measure has been validated for use in person, however no prior studies have determined whether administration over the phone is equally valid.¹

When filling out the KJOC questionnaire, the participant must mark a specific spot on a line (which starts at 0 and extends to 10) as the answer to each of the 10 questions. The distance from the end of the line to the mark is then measured, and the measurement is recorded as the score for that question.¹ One major disadvantage of this questionnaire is that it must be actively filled out by the patient, which reduces follow-up rates and potentially introduces selection bias. This score could theoretically be obtained passively over the phone (passive because the patient is not actively filling out the score by hand), as there is no physical examination portion to the measure. Outcome measures that allow phone follow-up and do not require in-person administration can increase rates of follow-up and reduce bias attributed to loss of follow-up.²² Phone validation of this outcome measure thus has the potential to decrease bias in future literature as well as improve the quality of research and clinical practice outcomes. The questionnaire and information on scoring are publically available in a previously reported study.¹

Therefore, the purpose of this study was to determine whether the KJOC Shoulder and Elbow Score can be reliably administered over the phone without altering the overall score. We hypothesized that there will be no significant difference in the score when the questionnaire is administered over the phone or in person.

Methods

This study was approved by the institutional review board at Rush University Medical Center. All potentially eligible patients were under the care of 2 authors (A.A.R., G.P.N.). Eligibility for inclusion was age ≥13 years and participation in any overhead sport, such as baseball, weightlifting, football, swimming, basketball, boxing, water polo, lacrosse, volleyball, tennis, softball, cheerleading, rock climbing, martial arts, and gymnastics. Patients seen for operative and nonoperative problems and those with a history of surgery were all eligible for inclusion. The clinic schedule for these 2 authors exactly 1 week ahead of time was reviewed. Patients who were scheduled to come into the office were contacted via phone to determine eligibility and, if eligible, to obtain written informed consent. No special trips were made to the clinic to fill out the questionnaire; that is, all patients who were included had been previously scheduled to come to clinic. Those who agreed to participate were then randomized with a random number generator into 1 of 2 groups. Patients in group 1 were administered the KJOC questionnaire over the phone. They came to the clinic 7 days later and were asked to complete the KJOC in person. The questionnaire was administered to them before they saw the attending surgeon, and no physical examination maneuvers or treatments were rendered before they had completed the questionnaire. The 2 scores were then compared.

Patients who were assigned to group 2 were administered the KJOC in person when they came into clinic 7 days after consent was obtained by phone. They were then contacted via phone 7 days later and readministered the questionnaire. A 7-day period was selected because it was considered a long-enough time frame for patients to forget the answers provided between administrations but not so long that the underlying symptoms would change. The 2 scores were then compared. Thus, one-half of the overall group completed the questionnaire on the phone first and then in person, and one-half completed the questionnaire in person first and then over the phone. At no point was any patient instructed to attempt to match his or her previous answers to the questions on the second round of completing the KJOC. Patients were simply instructed to answer the questions on the basis of their current symptoms.

Over the phone, the patients were instructed to quantify their answers for each question from 0 to 100. The question was read aloud, and the answer choices on either end of the spectrum were read to the patient so that she or he understood what a score of 0 or 100 would mean. For example, when asked question 2 (“How much pain do you experience in your shoulder or elbow?”), the patient was given the response that pain at rest would correlate with a score of 0 and no pain with competition would correlate with a score of 100. The patient was then instructed to give a score as a whole number between 0 and 100. The answer was subsequently divided by 10 to obtain a score for that question (eg, an answer of 85 would be a score of 8.5; the maximum score for each question was 10). The scores for the 10 questions were then summed to arrive at a maximum total score of 100. In person, the score was administered per Alberta et al,¹ also with a maximum score of 10 per question and an overall score of 100.

Two a priori power analyses were conducted. First, we planned to determine whether scores obtained via phone significantly differed from scores obtained in person. For this analysis, we had planned to conduct a paired t test. One of the largest prior studies with the KJOC, with 99 noninjured swimmers, described an SD of 18.7.³² To be conservative with regard to sample size, we used this SD instead of one from studies describing values of 6.7 and 12.1.^13,19 We determined that a change of 50% for this SD would be clinically significant. As no prior studies are available to describe the minimum clinically significant difference for the KJOC, 50% was selected, as it is among the most commonly selected differences for power analyses for reliability studies of this type.^6,7,24 To detect a 50% difference in SD and an SD of 18.7, 36 patients would be necessary to achieve a power of 80% with alpha set at 0.05.

In our second analysis, we used the methods of Walter et al,²⁸ who developed a methodology for an a priori power analysis when intraclass correlation coefficients (ICCs) are planned as a method of statistical analysis. Our aim was for an ICC of 0.9, which would be similar to the ICCs for phone versus in-person administration of other orthopaedic outcome measures.^4,6,7,24 Our minimum ICC would be 0.75, which is similar to the generally considered limits between excellent and fair/good reproducibility. Given these ICC limits and a total of 2 ratings, we need 45.8 participants. Allowing for attrition, we thus planned to include 50 participants within our study to meet both power analyses.

Statistical Analysis

All analyses were conducted in Excel X (Microsoft) and SPSS (v 23; IBM). P values <.05 were considered significant. Phone and paper scores were compared with paired Student t tests, given that scores were normally distributed as determined by the Kolmogorov-Smirnov test. Phone and paper scores were compared with ICCs via a 2-way random-effects model as well as Cronbach alpha. We decided a priori that only ICC values >0.75 would be considered acceptable for our purpose.¹¹ Individual test question answers were compared with related-samples Wilcoxon signed-rank tests, given that these data were not normally distributed as determined by the Kolmogorov-Smirnov test.

Results

Fifty participants were included, with 25 randomized to complete the KJOC questionnaire on paper first and 25 randomized to completing it over the phone first. The time between the first and second administration was a mean ± SD of 8 ± 5 days (median [interquartile range], 7 [2] days; range, 1-30 days). There were no significant differences in KJOC scores between the phone and paper groups (P = .139) (Table 1 and Figure 1). The mean KJOC score was 59.01 ± 23.1 as obtained over the phone versus 56.72 ± 21.0 in person. For those patients randomized to receive in-person administration first, KJOC paper scores ranged from 19 to 100, as opposed to 10 to 94 over the phone. For those randomized to receive phone administration first, KJOC paper scores ranged from 12 to 97, as opposed to 5 to 98 over the phone. The ICC value comparing paper and phone scores was 0.802 (95% CI, 0.767-0.883; P < .001), with a Cronbach alpha of 0.89. On comparison of individual questions, there were significant differences on questions 1, 3, and 8 (P = .013, .023, and .042, respectively), with the answers given via the phone higher than those given in person.

TABLE 1.

Median (Interquartile Range) for the Total KJOC Score Between Initial Paper and Phone Administration^a

	Initial Phone Administration		Initial Paper Administration
Variable	Paper Score	Phone Score	Paper Score	Phone Score	P Value
Total score	54 (31)	60 (28)	54 (30)	62 (36)	.139
Question
1	64 (52)	70 (30)	49 (66)	77 (34)	.013
2	50 (44)	60 (48)	59 (55)	73 (50)	.186
3	47 (33)	72 (50)	40 (47)	55 (48)	.023
4	75 (55)	80 (44)	80 (54)	90 (35)	.142
5	93 (89)	80 (100)	92 (38)	73 (60)	.051
6	66 (65)	80 (82)	41 (60)	45 (74)	.856
7	32 (41)	50 (59)	64 (50)	50 (57)	.672
8	42 (25)	50 (60)	51 (45)	65 (32)	.042
9	69 (62)	72 (42)	60 (57)	75 (53)	.068
10	40 (46)	25 (60)	35 (54)	45 (74)	.846

^aStatistically significant differences are in bold. KJOC, Kerlan-Jobe Orthopaedic Clinic.

Figure 1.

This boxplot displays score distributions for both paper and phone scores. Whiskers display minimum and maximum scores; the box displays the interval between the first and third quartiles; and the line displays the median score. KJOC, Kerlan-Jobe Orthopaedic Clinic.

Of the patients who participated, 44 (88%) had a shoulder issue, while 6 (12%) had an elbow issue. There were 39 male (78%) and 11 female (22%) patients, with a mean age of 27 ± 13 years (range, 14-59 years). In the phone-first group, there were 5 females, as opposed to 6 in the paper-first group. In the phone-first group, the mean patient age was 27.44, as opposed to 26.72 in the paper-first group. Baseball was the sport most commonly played, with 16 patients (32%) participating (Table 2). Of those who played baseball, 5 (31%) were pitchers.

TABLE 2.

Sports Participation of the Study Patients

	Patients, n
Sport	Total	Phone First	Paper First
Baseball	16	7	9
Weightlifting	7	3	4
Football	5	5	0
Swimming	5	3	2
Basketball	3	1	2
Boxing	2	0	2
Water polo	2	1	1
Lacrosse	2	1	1
Volleyball	2	1	1
Tennis	1	1	0
Softball	1	1	0
Cheerleading	1	0	1
Rock climbing	1	0	1
Martial arts	1	0	1
Gymnastics	1	1	0

Discussion

One of the many difficulties with performing clinical outcomes research is contacting patients and having them return to the office for evaluation. The ability to perform outcome measures over the phone can substantially improve rates of follow-up and greatly alleviate these difficulties. Our hypothesis was confirmed, as there was no difference in overall KJOC scores administered over the phone versus in person, and the 95% CIs for the ICC were above the lower limit of acceptability set a priori. This study provides information on the concurrent/equivalent forms of validity of the KJOC score whether administered in the person or over the phone.

The KJOC has become one of the most commonly used outcome scores for the overhead athlete.^{5,13,14,20,32} While there are several shoulder and elbow outcome scores that are frequently used (eg, American Shoulder and Elbow Surgeons; Disability of the Arm, Shoulder, and Hand), the KJOC is unique to the overhead athlete and asks higher-level activity questions specific to the shoulder and elbow.¹ This measure thus avoids the substantial ceiling effects encountered with other scores in examination of overhead athletes. Hence, this questionnaire is ideal for reporting clinical outcomes for overhead athletes, including baseball players, tennis players, swimmers, and so on.^25,32 Patients participating in this study were not limited to baseball pitchers, thus ensuring that the results were translatable across sports. The ICC for the responses from the phone and in person was 0.802, with a Cronbach alpha of 0.89. The Cronbach alpha, which ranges from 0 to 1, is used to provide a measure of the internal consistency of an assessment, while the ICC is a measure of the reliability of measurements and is useful for test-retest scenarios.^27,31 This study indicated very high internal consistency and reliability of the answers to the questions, regardless of whether the KJOC questionnaire was administered over the phone or in person.

Three questions did show significant differences in scores when patients responded to them over the phone versus in person: question 1, “How difficult is it for you to get loose or warm prior to competition or practice?”; question 3, “How much weakness and/or fatigue (ie, loss of strength) do you experience in your shoulder or elbow?”; and question 8, “What limitation do you have in endurance in competition due to your arm?” The responses to all of these questions were significantly higher over the phone than in person. It is unclear exactly why the answers to only these 3 questions differed between the phone and in-person questionnaire. It may be that patients overestimate their flexibility, strength, and endurance over the phone. It could also be that a potential bias exists when a patient passively fills out a form on paper, as opposed to actively answering questions in a phone interview. When patients are asked to fill out a paper survey, they may not concentrate on the task at hand and may be more focused on finishing the questionnaire than providing accurate responses. Conversely, when a patient must engage with a person via the phone and listen to and respond to questions, she or he may provide more accurate responses. Furthermore, patients can clarify the meaning of questions when taking a survey over the phone versus in person, if there is no assistant present to aid them in filling out the paper survey. Future modifications of the KJOC score could focus on improving the psychometrics of these questions. However, this did not have a net effect on the overall score, as there was no difference seen between the mean score obtained in person and that registered over the phone. This study thus showed that the KJOC can be reliably administered over the phone, minimizing the need for patients to visit the office to obtain this score.

Limitations

This study was constructed to minimize as many limitations as possible. Patients were randomized to fill out the survey first either over the phone or in person to minimize the risk of reporting and recall bias. Furthermore, the study was adequately powered to achieve its primary outcome. There were differences seen between phone and in-person responses for 3 of the 10 questions, although this did not affect the overall mean score. This finding may indicate the need to alter these 3 questions of the KJOC score going forward to avoid any issues when results for each question are reported separately. Although done purposefully, there was heterogeneity of the players who were included (ie, not all of the participants were pitchers). Furthermore, the number of days between questionnaire administration has not been validated, and to our knowledge, no high-level studies have determined what the ideal time is between questionnaires. Seven days was chosen, as it was believed to be enough time to allow patients to forget their answers but not long enough for their symptoms to change. However, some patients may have remembered their answers. A 50% SD was used for determination of clinical significance, but no studies have assessed minimal clinically important differences. It is possible that the results of this study would have changed if a different SD had been used. Furthermore, prior work^19,23,32 in this area was on high-functioning athletes (collegiate swimmers and minor league pitchers) and thus may not be reflective of lower-level athletes. Last, type II error is possible; however, we conducted 2 separate a priori power analyses, and based on these analyses, a large-enough sample size was achieved to ensure that our study was adequately powered.

Conclusion

This study showed that the KJOC questionnaire can be administered over the phone with no significant difference in overall score compared with in-person administration.

APPENDIX

Kerlan-Jobe Orthopaedic Clinic Score Questionnaire