Abstract
Background:
Our primary purpose was to evaluate the reliability of telephone administration of the PROMIS Upper Extremity (UE) Computer Adaptive Test (CAT) version 2.0 in a hand and upper extremity population, and secondarily to make comparisons to the QuickDASH.
Methods:
Patients more than 1 year out from hand surgeries performed at a single tertiary institution were enrolled. Half of the patients completed telephone UE CAT and QuickDASH surveys first, followed by computer-based surveys 1-10 days later, and the other half completed them in the reverse order. Telephone surveys were re-administered 2-6 weeks later to evaluate test-retest reliability. Concordance correlation coefficients (CCC) were used to assess agreement between telephone and computer-based scores, and intraclass correlation coefficients (ICC) were used to assess test-retest reliability. The proportion of patients with discrepancies in follow-up scores that exceeded estimates of the minimal clinically important difference (MCID) was evaluated.
Results:
For the 89 enrolled patients, the UE CAT CCC was 0.82 (83% CI 0.77-0.86; ‘good’), which was significantly lower than 0.92 (83% CI 0.89-0.94; ‘good’ to ‘excellent’) for the QuickDASH. The UE CAT ICC did not differ significantly from the QuickDASH (0.85 and 0.91 respectively). Differences in telephone versus computer scores exceeded 5 points (MCID estimate) for the UE CAT in 34% of patients, versus 5% of patients exceeding 14 points (MCID estimate) for the QuickDASH.
Conclusions:
Significantly better reliability was observed for the QuickDASH than the PROMIS UE CAT when comparing telephone to computer-based score acquisition. Over one third of patients demonstrated a clinically relevant difference in scores between the telephone and computer administered tests. We conclude that the PROMIS UE CAT should only be administered through computer-based methods.
Keywords: Computer adaptive test (CAT); PROMIS (Patient-Reported Outcomes Measurement Information System), Reliability; Test-retest; Upper Extremity
Introduction:
In today’s evolving health care landscape, there continues to be emphasis placed upon providing quality, high-value health care.1 Recently, the Hand Surgery Quality Consortium identified patient-reported outcome measures (PROMs) as an important component in defining quality,2 and PROMs data are likely to influence reimbursement of healthcare services under proposed pay-per-performance payment models.3
In 2004, the National Institute of Health initiated the Patient-Reported Outcomes Measurement Information System (PROMIS) with the goal of developing and validating PROMs to assess a diverse array of health domains.4,5 These instruments were designed to be administered to patients in digital format (desktop, laptop, or tablet computers) to allow for Computer Adaptive Testing (CAT). CAT involves the administration of a subset of questions from a testing bank to a patient, while tailoring subsequent questions based upon responses to previous items.4,6 The PROMIS Upper Extremity (UE) Computer Adaptive Test (CAT) has recently received increasing attention in the hand and upper extremity literature. Specifically, UE CAT scores have been demonstrated to correlate with traditional legacy scales including the DASH and its abbreviated version, QuickDASH 7-9, which were designed to be completed on a paper questionnaire form.10,11
Situations exist in which verbal administration of follow-up PROMs would be advantageous. Previous studies have demonstrated greater rates of response and survey completeness using telephone-acquired PROMs, as compared to other methods of administration.12,13 Furthermore, outcome assessment for patients lacking computer access, those who are illiterate, or those who do not regularly use e-mail, may be more successful by telephone. Requesting patients return to the office for collection of mid- to long-term PROMs may also be impractical. Since PROMs are often obtained via telephone in clinical research, it is important to understand the implications of changing the administration format because the long-term efficacy of various treatments may be based upon this method of collection.
Despite the advantages of telephone acquisition of PROMs, authors have cautioned against administering surveys in a format other than that for which the measure was originally designed and validated.14 As the performance of an instrument administered in such a way cannot be predicted without prior investigation,14 our main purpose was to assess the reliability of the PROMIS UE CAT administered by telephone compared to administration using the intended computer-based format. Secondarily, we aimed to compare the reliability of the PROMIS UE CAT to the QuickDASH.
Materials and Methods:
This IRB-approved study was performed at a single-center tertiary academic center. Patients were identified who were more than 1-year post following surgery performed by one of five fellowship-trained orthopaedic hand surgeons. We excluded non-English speaking patients, minors (< 18 years), and patients undergoing nonoperative procedures. Additionally, we excluded patients indicating on a screening questionnaire that they had developed an upper extremity injury, or had undergone additional upper extremity treatment, since the listed date of their index surgery. This exclusion, and requirement of a minimum of 1-year postoperative follow-up, were implemented in an effort to exclude patients who could be expected to demonstrate changes in PROMs for upper extremity function or disability. In this specific sample of patients meeting the selection criteria, we assumed that any changes in PROM scores between telephone surveys would be due to variation in how the patient rates their function (which would reflect instrument reliability), rather than actual variation in their function.
Postoperative patients were identified by surgical Current Procedural Terminology (CPT) codes performed by the five surgeons. Surgical procedures, postoperative diagnoses, and baseline demographic data were verified and recorded through manual chart review. Eligible patients were mailed opt-out consent forms. After excluding those who opted out, patients were informed that study participation would involve three contacts for survey completion over a several week period. Email addresses were confirmed, and all participants were asked the screening question regarding new upper extremity conditions or treatments. Patients were informed that their responses would be helpful in understanding how to measure their surgical outcome without specific discussion of the planned comparisons.
Remaining patients were electronically randomized to one of two study arms in blocks of 4: 1) Arm 1 received the telephone survey first, followed by the computer-based survey, and 2) Arm 2 received the computer-based survey first, followed by the telephone survey. Specifically, Arm 1 patients were administered the PROMIS Upper Extremity CAT v2.0 followed by the QuickDASH during this first contact, which was by telephone. Computer-based administration of both instruments was solicited using email with a personalized link within a day following the telephone survey. The link was valid 1-10 days from completion of the initial telephone survey. Reminder emails were sent daily until completion, or until the window of eligibility had lapsed. Arm 2 patients were emailed a personal link for completion of the PROMIS Upper Extremity CAT and QuickDASH, with reminder emails every other day until completed. Once this computer-based administration was completed, attempts to contact each participant by telephone were made the next day to complete the same surveys verbally and up to ten days after. Attempts to contact participants were made daily until completed or until outside of the eligibility window.
All participants in both study arms were then contacted by telephone between 2 and 6 weeks following the completion of their second set of surveys, as previously recommended for test-retest analysis.15 Verbal administration of both the PROMIS Upper Extremity CAT and QuickDASH was performed during this third contact.
Basic descriptive statistics were calculated. Continuous variables were compared using the Student’s t-test, and categorical variables were compared using either chi-squared or Fisher’s exact tests. Response rate was calculated as the number of patients who responded to the PROMIS UE CAT and QuickDASH for both telephone and computer-based administrations, divided by the total number of patients for whom contact was attempted.
Differences in PROMIS UE CAT and QuickDASH scores between the first two contacts were calculated as the first telephone contact score minus the computer-based score. The concordance correlation coefficient (CCC) was used to assess agreement between the first verbal score and the computer-based score. Ninety-five percent confidence intervals (CIs) for the CCC were calculated for all participants, and separately for Arms 1 and 2. For comparisons of scores between the two instruments and between the two arms, we used 83% CIs to assess statistical significance because in this situation overlapping CIs does not indicate a lack of statistical significance.16 The reason is because there are two sets of CIs indicating variability among two different sample means, and thus narrower 83% (or 83.7%) CIs are needed to equate overlap of CIs with a lack of statistical significance at the 0.05 level.16-18
Pearson’s correlation coefficients were calculated to evaluate the relationship between patient age and difference in telephone versus computer-based scores in which p-value calculation utilized the Fisher’s z-transformation. Univariate linear regression was used to model the change in the differences between scores at the first two contacts (first telephone score minus computer-based score) for both instruments. Predictor variables included study arm, age, sex, race, anesthetic type, BMI, surgical location (ambulatory surgery center versus main operating room), diagnosis, provider, ASA score, and marital status. Due to the potential for unreliable estimates, we did not provide coefficients for predictor variable categories with less than 5 observations.
Test-retest reliability was used to assess agreement between the two telephone scores using intraclass correlation coefficients (ICC). Differences between scores collected at the two telephone contacts were calculated as the second telephone score minus the first telephone score for both instruments. Estimates and 95% CIs were calculated for the entire cohort, and separately for Arms 1 and 2. For statistical comparison of ICCs between the two instruments and between the two arms, 83% CIs were calculated.
Bland-Altman plots were created to illustrate differences between telephone and computer-based scores for both instruments. Bland-Altman plots were also created to demonstrate differences between the two telephone scores. The number and percentage of patients with differences in scores exceeding estimates of the minimal clinically important difference (MCID) were calculated for each plot, as has been done previously in orthopaedic outcomes studies.19-22 For the QuickDASH, an MCID estimate of 14 was used.23 As the MCID for the PROMIS UE CAT v2.0 has not yet been determined, an estimate of 5 was used based upon the ½ standard deviation method (the expected standard deviation is 10 in a normative population).24 One-sided paired t-tests and associated 95% confidence intervals were used to determine whether absolute differences in telephone versus computer-based scores, or the pair of telephone scores, were different within the tolerance of the MCID estimate for the study population.
CCCs and ICCs were interpreted as follows in terms of strength: poor reliability (< 0.50), moderate reliability (0.50 to 0.75), good reliability (0.75 to 0.90), and excellent reliability (> 0.90).25,26 Absolute values of Pearson correlation r-values were interpreted as follows in terms of strength of the association: negligible (r < 0.3), low (0.3 ≤ r < 0.5), moderate (0.5 ≤ r < 0.7), high (0.7 ≤ r < 0.9), and very high (≥0.9).27,28 A significance level of α = 0.05 was used throughout.
An a priori sample size estimate was performed based upon a previously-published ICC between the written and verbal QuickDASH (0.91; 95% CI: 0.84-0.95).12 We conservatively calculated the sample size needed to differentiate between the lower bound of this 95% CI (0.84) from the upper limit of good/fair reproducibility (0.74).29 A sample size of 86 subjects, with 2 observations per subject, achieves 80% power to detect a difference between an ICC of 0.84 and an ICC of 0.74 using an F-test and significance level of 0.05.29
Results:
Out of 465 reviewed patients, 112 were excluded due to injury, or additional upper extremity treatment or surgery that occurred after their index surgery. A total of 15 patients opted out (10 in Arm 1, 5 in Arm 2). An additional 8 patients were excluded due to recent additional surgery (3 in both arms) or injury (2 in Arm 2) after the initial survey was completed. Among the remaining 330 patients, only 89 (27%) completed all three sets of questionnaires. Of the 89 included patients, the mean age was 50.6 ± 15.9 years and 54 (59%) were female. Baseline characteristics for patients in Arm 1 and Arm 2 are provided in Table 1.
Table 1 –
Baseline Patient Characteristics
| Variable* | Type/Level | Summary | Arm 1 (N=43) | Arm 2 (N=46) |
|---|---|---|---|---|
| Age | [mean (SD)] | 50.6 (15.9) | 48.2 (16.4) | 52.9 (15.4) |
| Anesthesia Type | ||||
| General | 62 (68%) | 29 (64%) | 33 (72%) | |
| Local | 1 (1%) | 0 (0%) | 1 (2%) | |
| MAC | 22 (24%) | 13 (29%) | 9 (20%) | |
| Regional | 6 (7%) | 3 (7%) | 3 (7%) | |
| ASA Score | ||||
| Healthy | 24 (26%) | 13 (29%) | 11 (24%) | |
| Mild Systemic Disease | 54 (59%) | 25 (56%) | 29 (63%) | |
| Severe Systemic Disease | 13 (14%) | 7 (16%) | 6 (13%) | |
| BMI | [mean (SD)] | 27.8 (6%) | 27.9 (7%) | 27.7 (6%) |
| Diagnosis Category | ||||
| Osteoarthritis | 20 (22%) | 7 (16%) | 13 (28%) | |
| Nerve Compression | 24 (26%) | 13 (29%) | 11 (24%) | |
| Tendinitis | 8 (9%) | 3 (7%) | 5 (11%) | |
| Fracture | 12 (13%) | 6 (13%) | 6 (13%) | |
| Dupuytrens | 3 (3%) | 0 (0%) | 3 (7%) | |
| Ligament/Tendon Injury | 10 (11%) | 6 (13%) | 4 (9%) | |
| Mass/Cyst | 9 (10%) | 3 (7%) | 6 (13%) | |
| Nerve Laceration | 5 (5%) | 2 (4%) | 3 (7%) | |
| Ulnar Impaction | 3 (3%) | 2 (4%) | 1 (2%) | |
| Infection | 3 (3%) | 3 (7%) | 0 (0%) | |
| Other | 7 (8%) | 4 (9%) | 3 (7%) | |
| Insurance Type | ||||
| Commercial | 69 (76%) | 33 (73%) | 36 (78%) | |
| Medicaid | 2 (2%) | 1 (2%) | 1 (2%) | |
| Medicare | 18 (20%) | 11 (24%) | 7 (15%) | |
| Workers Compensation | 2 (2%) | 0 (0%) | 2 (4%) | |
| Marital Status | ||||
| Single | 20 (22%) | 15 (33%) | 5 (11%) | |
| Married | 61 (67%) | 25 (56%) | 36 (78%) | |
| Divorce | 5 (5%) | 2 (4%) | 3 (7%) | |
| Other/Unknown | 5 (5%) | 3 (7%) | 2 (4%) | |
| Race | ||||
| White/Caucasian | 84 (92%) | 39 (87%) | 45 (98%) | |
| Other | 7 (8%) | 6 (13%) | 1 (2%) | |
| Sex | ||||
| Female | 54 (59%) | 26 (58%) | 28 (61%) | |
| Male | 37 (41%) | 19 (42%) | 18 (39%) | |
| Surgeon | ||||
| Provider A | 8 (9%) | 6 (13%) | 2 (4%) | |
| Provider B | 12 (13%) | 4 (9%) | 8 (17%) | |
| Provider C | 21 (23%) | 10 (22%) | 11 (24%) | |
| Provider D | 37 (41%) | 19 (42%) | 18 (39%) | |
| Provider E | 13 (14%) | 6 (13%) | 7 (15%) | |
| Surgery Location | ||||
| Ambulatory Surgery Center | 84 (92%) | 42 (93%) | 42 (91%) | |
| Main OR | 7 (8%) | 3 (7%) | 4 (9%) |
Missing values: BMI = 1
Comparison of telephone and computer-based PROMIS UE CAT scores revealed a CCC of 0.82 (83% CI 0.77 to 0.88) in the ‘moderate’ to ‘good’ reliability range, which was significantly lower compared to a QuickDASH CCC of 0.92 (83% CI 0.89 to 0.94) in the ‘good’ to ‘excellent’ range (Table 2). Given no overlap between respective 83% confidence intervals, the CCC for the QuickDASH should be considered significantly greater than that for the PROMIS UE CAT at a 0.05 significance level. Bland-Altman plots illustrating differences between telephone and computer-based scores are provided in Figure 1 (PROMIS UE CAT) and Figure 2 (QuickDASH). For the UE CAT, 34% of patients had a score difference that exceeded an MCID estimate of 5, and for the QuickDASH 5% had score differences beyond an MCID estimate of 14. One-sided paired t-test results demonstrated that the overall difference in telephone and computer-based scores were below MCID estimates for both the PROMIS UE CAT (mean difference 4.17 95% CI 0 to 4.99; p = 0.049) and the QuickDASH (mean difference 4.76; 95% CI 0 to 5.63; p < 0.05).
Table 2 –
Comparison of Telephone and Computer-Based Scores
| Instrument* | Type/Level | Arm 1 (N=43) | Arm 2 (N=46) | Summary |
|---|---|---|---|---|
| PROMIS UE CAT | ||||
| Phone | [mean (SD)] | 51 (10.3) | 48.9 (10.4) | 49.9 (10.4) |
| [mean (SD)] | 47.2 (10.9) | 47.4 (10.3) | 47.3 (10.6) | |
| Difference | [mean (SD)] | 3.7 (5.5) | 1.5 (5.8) | 2.6 (5.8) |
| CCC Estimate | 0.81 | 0.83 | 0.82 | |
| CCC 95% CI CI | (0.70, 0.89) | (0.72, 0.90) | (0.75, 0.88) | |
| CCC 83% CI CIa | (0.74, 0.87) | (0.76, 0.89) | (0.77, 0.86) | |
| QuickDASH | ||||
| Phone | [mean (SD)] | 15.8 (19.8) | 13.2 (13.2) | 14.5 (16.7) |
| [mean (SD)] | 16.4 (19.7) | 14.6 (14.1) | 15.5 (17) | |
| Difference | [mean (SD)] | −0.4 (7.2) | −1.4 (6.5) | −0.9 (6.8) |
| CCC Estimate | 0.93 | 0.88 | 0.92 | |
| CCC 95% CI CI | (0.88, 0.96) | (0.80, 0.93) | (0.88, 0.94) | |
| CCC 83% CI CIa | (0.90, 0.96) | (0.83, 0.92) | (0.89, 0.94) | |
| Days Elapsed | [mean (SD)] | 2.6 (2.3) | 2.5 (1.9) | 2.5 (2.1) |
| 1 to 7 | 41 (91%) | 46 (100%) | 87 (96%) | |
| 8 to 10 | 4 (9%) | 0 (0%) | 4 (4%) | |
| Response Rate | N/Total (%) | 45/171 (26%) | 46/154 (30%) | 91/325 (28%) |
CCC = concordance correlation coefficient
Missing values: QuickDash Email = 1, QuickDash Difference = 1
Lack of overlap between 83% confidence intervals indicates that the QuickDASH CCC is significantly greater than the PROMIS UE CAT CCC.
Figure 1 –
PROMIS UE CAT Bland-Altman Plot: Telephone versus computer-based administration. The mean score for the x-axis label represents the mean of the first telephone score and the computer-based score.
Figure 2 –
QuickDASH Bland-Altman Plot: Telephone versus computer-based administration. The mean score for the x-axis label represents the mean of the first telephone score and the computer-based score.
In the PROMIS UE CAT univariate regression analysis, increasing age, American Society of Anesthesiologists (ASA) class 2 and class 3, MAC anesthesia, and Medicare insurance status were significantly associated with a greater score difference between telephone and computer-based modalities (p < 0.05 for all comparisons; Table 3). These factors were not significant for the QuickDASH (p = 0.30 to 0.95). A diagnosis of a ligament/tendon injury was significantly associated with a greater score difference for the QuickDASH (p < 0.05). Differences in all other baseline patient characteristics under study were not significant (all p > 0.05).
Table 3 –
Univariate Regression of Differences in Telephone and Computer-Based Scores
| Variable | Level | PROMIS UE CAT Difference |
QuickDASH Difference Estimate | ||
|---|---|---|---|---|---|
| Coefficient (95% | p-value | Coefficient (95% CI) | p-value | ||
| Arm | Arm 1 | - Reference - | - | - Reference - | - |
| Arm 2 | −2.19 (−4.53, | 0.07 | −0.94 (−3.77, 1.89) | 0.51 | |
| Age | [mean (SD)] | 0.14 (0.07, 0.21) | < 0.05 | −0.05 (−0.14, 0.04) | 0.30 |
| Anesthesia Type | |||||
| General | - Reference - | - | - Reference - | - | |
| Local | * | * | * | * | |
| MAC | 3.45 (0.72, 6.18) | < 0.05 | −0.41 (−3.75, 2.92) | 0.81 | |
| Regional | −1.95 (−6.65, | 0.42 | 4.55 (−1.19, 10.28) | 0.12 | |
| ASA Score | |||||
| Healthy | - Reference - | - | - Reference - | - | |
| Mild Systemic Disease | 2.86 (0.16, 5.55) | < 0.05 | 0.73 (−2.59, 4.05) | 0.67 | |
| Severe Systemic Disease | 4.73 (0.95, 8.51) | < 0.05 | 0.15 (−4.50, 4.80) | 0.95 | |
| BMI | [mean (SD)] | 0.00 (−0.20, | 0.97 | 0.22 (−0.02, 0.45) | 0.07 |
| Diagnosis Category** | |||||
| Osteoarthritis | −2.04 (−4.88, | 0.16 | 0.87 (−2.60, 4.33) | 0.63 | |
| Nerve Compression | 0.83 (−1.86, | 0.55 | −1.20 (−4.39, 2.00) | 0.47 | |
| Tendinitis | 2.88 (−1.28, | 0.18 | 1.64 (−3.34, 6.61) | 0.52 | |
| Fracture | 0.70 (−2.81, | 0.70 | −1.45 (−5.61, 2.71) | 0.50 | |
| Ligament Tendon Injury | 1.13 (−2.66, | 0.56 | −5.10 (−9.48, −0.72) | < 0.05 | |
| Mass Excision | −0.67 (−4.65, | 0.74 | 3.55 (−1.12, 8.22) | 0.14 | |
| Nerve Laceration | −2.60 (−7.80, | 0.33 | 1.94 (−4.24, 8.12) | 0.54 | |
| Other | −1.67 (−6.12, | 0.46 | −3.23 (−8.48, 2.03) | 0.23 | |
| Insurance Type | |||||
| Commercial | - Reference - | - | - Reference - | - | |
| Medicaid | * | * | * | * | |
| Medicare | 4.35 (1.47, 7.23) | < 0.05 | 0.85 (−2.70, 4.40) | 0.64 | |
| Workers Compensation | * | * | * | * | |
| Marital Status | |||||
| Single | - Reference - | - | - Reference - | - | |
| Married | 1.77 (−1.12, | 0.23 | −1.65 (−5.10, 1.80) | 0.35 | |
| Divorce | 3.21 (−2.40, | 0.27 | 3.64 (−3.04, 10.31) | 0.29 | |
| Other/Unknown | 5.25 (−0.36, | 0.07 | −0.45 (−7.13, 6.22) | 0.89 | |
| Race | |||||
| White/Caucasian | - Reference - | - | - Reference - | - | |
| Other | 1.52 (−2.94, | 0.51 | −1.46 (−6.75, 3.82) | 0.59 | |
| Sex | |||||
| Female | - Reference - | - | - Reference - | - | |
| Male | −2.22 (−4.59, | 0.07 | 1.67 (−1.19, 4.53) | 0.26 | |
| Surgeon | |||||
| Provider A | - Reference - | - | - Reference - | - | |
| Provider B | −2.66 (−7.83, 2.51) | 0.32 | 0.38 (−5.84, 6.60) | 0.91 | |
| Provider C | 1.00 (−3.71, 5.70) | 0.68 | −1.02 (−6.72, 4.68) | 0.73 | |
| Provider D | 0.26 (−4.16, 4.68) | 0.91 | −0.69 (−6.00, 4.62) | 0.80 | |
| Provider E | 0.20 (−4.89, 5.29) | 0.94 | 0.74 (−5.38, 6.87) | 0.81 | |
| Surgery Location | |||||
| Ambulatory Surgery Center | - Reference - | - | - Reference - | - | |
| Main OR | −1.15 (−5.60, 3.31) | 0.62 | −0.41 (−5.70, 4.88) | 0.88 | |
Coefficient not provided due to small counts (<5) in the given category.
Diagnosis categories with less than 5 counts were not included in this analysis.
Bold, italicized p-values are significant.
Test-retest results comparing the two telephone scores are depicted in Table 4. The PROMIS UE CAT ICC was 0.85 (83% CI 0.80 to 0.89) in the ‘good’ to ‘excellent’ range. The QuickDASH ICC was 0.91 (83% CI 0.88 to 0.93), which was not significantly different, and also in the ‘good’ to ‘excellent’ range. Given overlap between 83% confidence intervals, there was no significant difference between PROMIS UE CAT and QuickDASH ICCs at a 0.05 significance level. Bland-Altman plots illustrating differences between telephone and computer-based scores are provided in Figure 3 (PROMIS UE CAT) and Figure 4 (QuickDASH). For the UE CAT, 29% of patients had a score difference that exceeded an MCID estimate of 5, and for the QuickDASH 6% had score differences beyond an MCID estimate of 14. One-sided paired t-test results demonstrated that the overall difference in telephone and computer-based scores were below MCID estimates for both the PROMIS UE CAT (mean difference 3.58 95% CI 0 to 4.36; p < 0.05) and the QuickDASH (mean difference 4.89; 95% CI 0 to 5.85; p < 0.05).
Table 4 –
Test-Retest: Comparison of First and Second Telephone Scores
| Instrument* | Type/Level | Arm 1 (N=43) | Arm 2 (N=43) | Summary |
|---|---|---|---|---|
| PROMIS UE CAT | ||||
| Phone | [mean (SD)] | 51.0 (10.3) | 48.9 (10.4) | 49.9 (10.4) |
| [mean (SD)] | 48.4 (10.4) | 48.1 (10) | 48.3 (10.2) | |
| Difference | [mean (SD)] | −2.4 (5.0) | −1.1 (5.7) | −1.7 (5.4) |
| ICC Estimate | 0.86 | 0.84 | 0.85 | |
| ICC 95% CI | (0.72, 0.93) | (0.73, 0.91) | (0.77, 0.90) | |
| ICC 83% CIa | (0.77, 0.91) | (0.77, 0.89) | (0.80, 0.89) | |
| QuickDASH | ||||
| Phone | [mean (SD)] | 15.8 (19.8) | 13.2 (13.2) | 14.5 (16.7) |
| [mean (SD)] | 16.9 (20.4) | 13.1 (12.9) | 15.0 (17.1) | |
| Difference | [mean (SD)] | 0.7 (8.5) | 0.1 (5.8) | 0.4 (7.3) |
| ICC Estimate | 0.91 | 0.90 | 0.91 | |
| ICC 95% CI | (0.84, 0.95) | (0.83, 0.95) | (0.87, 0.94) | |
| ICC 83% CIa | (0.87, 0.94) | (0.86, 0.94) | (0.88, 0.93) | |
| Days Between Score | ||||
| Collections | [mean (SD)] | 20.4 (5.4) | 19.6 (6.1) | 20.0 (5.8) |
| Response Rate | N/Total (%) | 43/171 (25%) | 43/154 (28%) | 86/325 (26%) |
ICC = Intraclass correlation coefficient
Missing values: QuickDash Email = 1, QuickDash Difference = 1
Overlap between 83% confidence intervals indicates that the QuickDASH ICC does not significantly differ from the PROMIS UE CAT ICC.
Figure 3 –
PROMIS UE CAT Bland-Altman Plot: Telephone test-retest. The mean score for the x-axis label represents the mean of the first and second telephone scores.
Figure 4 –
QuickDASH Bland-Altman Plot: Telephone test-retest. The mean score for the x-axis label represents the mean of the first and second telephone scores.
Pearson correlations between age and difference in scores are provided in Table 5. The only statistically significant finding was a low correlation between increasing age and score difference between the first telephone score and the computer-based score for the PROMIS UE CAT (p < 0.05). This association is illustrated as a scatterplot in Figure 5.
Table 5 –
Correlations between Age and Difference in Scores
| Comparison | First Telephone versus Computer- Based Score |
First versus Second Telephone Score (Test-Retest) |
||
|---|---|---|---|---|
| Pearson Coefficient | p-value | Pearson Coefficient | p-value | |
| PROMIS UE CAT versus Age | 0.38 | < 0.05 | −0.09 | 0.39 |
| QuickDash versus Age | −0.11 | 0.30 | 0.03 | 0.82 |
Bolded, italicized p-values are statistically significant.
Figure 5 –
Scatterplot of Differences in PROMIS UE CAT telephone and computer-based scores by age
Discussion:
The main finding of this study was that, when verbally-administered, the PROMIS UE CAT had significantly lower reliability than the QuickDASH in comparison to computer-based administration. Although the UE CAT reliability was considered ‘good’, it is noteworthy that the difference between the telephone and computer scores exceeded the UE CAT MCID estimate one third of the time (versus 5% for the QuickDASH). These findings illustrate the clinical relevance of the differing modes of survey administration.
Although test-retest reliability did not significantly differ between instruments, the difference between the two telephone scores for the PROMIS UE CAT exceeded an MCID estimate in 29% of cases as opposed to only 6% for the QuickDASH. As a final concern, a weak but significant correlation was observed such that increasing age was associated with greater differences between telephone and computer-based PROMIS UE CAT scores – this association was absent for the QuickDASH.
In light of these observations regarding the proportion of patients with scores exceeding a difference of an MCID estimate upon repeated surveys, it is noteworthy that applying the MCID concept to individual patients may be controversial. However, multiple studies across several orthopaedic specialties have published analyses of this nature.19-22 The optimal way to apply the MCID and the most appropriate way to report outcomes results is yet to be determined and is the focus of an increased research effort recently. Despite these findings, it should be noted that the paired differences between repeat survey scores were less than the chosen MCID estimate for the PROMIS UE CAT and QuickDASH comparisons of telephone to computer scores, and for comparisons between the two telephone scores. Since the MCID for the PROMIS UE CAT version 2.0 has yet to be elucidated (e.g. the chosen estimate of 5 is hypothetical), it is possible that an estimate lower than the value of 5 used here may lead to significant differences for the UE CAT. Mean differences between score acquisitions for the UE CAT were in the range of 3.6 to 4.2 on the paired t-test analysis, and therefore our conclusion that the overall difference in scores is lower than an MCID, could be reversed should new estimates arise that fall within or below these values. Although scores from different versions of the PROMIS UE CAT are not interchangeable nor can the values be directly compared,30 one recent study provided MCID estimates for the PROMIS UE CAT v1.2 in the range of 4.2 to 8.0 in a carpal tunnel population.31 Although we could not identify a publication with PROMIS UE CAT v2.0 MCID values, one published abstract suggested this value falls in the range of 3.0 to 4.1.32 This range of values, which were obtained in a general hand and upper extremity population similar to that of the current study, are interestingly below the paired differences between computer and telephone scores. It is possible that the differences between telephone and computer scores observed in the current study actually exceed that MCID estimate, which raises further concerns about telephone acquisition of the UE CAT. In contrast, paired differences for the QuickDASH were in the range of 4.8 to 4.9, which is subjectively far from even the lower of the MCID estimates in the literature (6.8 to 19).21,23,33-35 Although further investigation is warranted, we hypothesize that a negative trade-off for scale brevity for the PROMIS UE CAT could be slightly lower reliability. That is, if a patient completes a second survey and provides a slightly different response to even the same initial question, the computer adaptive algorithm may place that patient on a different track of subsequent questions, which may pertain to higher or lower overall function than the track of questioning for their initial survey. It should be noted that this is a hypothesis and formally commenting upon this is beyond the scope of the study results.
Although our findings for the PROMIS UE CAT are unique, study findings with regard to the QuickDASH may be compared to prior literature. London et al utilized ICCs to compare QuickDASH scores obtained through telephone and paper form 1 day apart. This yielded a value of 0.91 and a 95% confidence interval in the ‘good’ to ‘excellent’ reliability range (0.84 to 0.94),12 which is subjectively similar to the value of 0.92 and 95% CI spanning the ‘good’ to ‘excellent’ range in the current study. Similarly, the authors did not observe a correlation between patient age and difference in QuickDASH scores. Minor differences between the studies should be noted, however. The current study differs in its use of concordance correlation coefficients (although they are very similar to the ICC), and by comparing telephone-based scores to computer-based scores rather than the written version of the QuickDASH. London et al observed ‘good’ test-retest reliability, as opposed to ‘good’ to ‘excellent’ reliability noted in the current study for the QuickDASH. Similarly, the ICCs for test-retest reliability observed in the current study were subjectively greater (0.91 versus 0.68). It is possible that our shorter time frame between the test and retest administrations (mean 20 days versus 5 months), or comparison to computer-based scores rather than those from paper forms, could contribute to these differences. Nonetheless, as in the current study, the authors concluded that these differences were not clinically relevant.
Although the QuickDASH demonstrated superior performance over the UE CAT in certain aspects of reliability, both telephone and computer-based acquisition methods may potentially be suitable for both instruments to be utilized in clinical research. Frost et al have recommended a minimum reliability threshold of 0.70 for an instrument to be used for clinical trials outcomes, and both instruments have exceeded this when comparing telephone to computer-based scores and for test-retest reliability.36 Furthermore, other studies have evaluated reliability of select PROMIS instruments with favorable results, but we were unable to locate prior studies on the PROMIS UE CAT for comparison. Deyo et al observed ‘good-excellent’ test-retest reliability for the PROMIS short 29-item form for chronic musculoskeletal pain patients.37 Bjorner et al evaluated multiple modes of administration (interactive voice response, paper form, handheld computer, or personal computer) for 8-item short forms derived from the PROMIS Physical Function, Fatigue, and Depression item banks.38 High levels of reliability (average ICC of 0.90) between different modes of administration were noted with no clinically relevant effect attributed to the mode of administration within a score tolerance of ± 2 points. Broderick et al collected several PROMIS instruments (pain intensity, pain interference, physical function, and fatigue) in CAT and short-form formats for a large cohort of patients with osteoarthritis.39 ‘Good’ to ‘excellent’ test-retest performance for both modes of administration were observed, and less than 25% of patients had differences in CAT and short-form scores that exceeded MCID estimates. For a sample of spinal cord or traumatic brain injury patients, Kisala et al observed no effect of mode of administration for multiple PROMIS fixed-length short forms (Physical Function, Fatigue, Pain Interference, Anger, Anxiety, Depression), which were administered by an interviewer (in-person form or telephone) or by a computer-based format.40 Lastly, Magnus et al observed similarity between telephone and computer-based PROMIS Depressive Symptoms, Fatigue, and Mobility short form scores in a pediatric population.41
Our study limitations deserve mention. Although our findings should not be generalized to all PROMIS instruments or all study populations, we agree with the notion that non-standard administration methods should be evaluated before clinical or research use.14 Our study sample age should be considered when interpreting the results because it is possible that the results could differ with a younger population more comfortable with technology. Our study did not investigate the reasons why the PROMIS UE CAT performs less favorably than the QuickDASH. It is unclear if a higher response rate would affect our results. It remains possible that the advantage of fewer questions for UE CAT completion could decrease precision or reliability, although future work is needed to elucidate and potentially improve upon this. Additionally, it remains unclear what level of reliability is acceptable, which makes it more challenging to interpret the study results with regard to reliability between different modes of survey administration. Our finding that patients with diagnoses related to tendon/ligament pathology demonstrated a greater score difference between telephone and computer administration of the QuickDASH on univariate analysis likely represents alpha error – that is erroneously concluding there is a difference when one does not actually exist – versus the possibility that these patients were older. However, this study is limited in that it was not designed to specifically evaluate for reliability in outcome collection between diagnostic categories. Lastly, it is unclear if our response rate of 26-28% had an impact on the study findings because it is possible that responders and non-responders were different in terms of treatment or patient factors.
In conclusion, the QuickDASH and PROMIS UE CAT demonstrated similar test-retest reliability for telephone administration in the ‘good’ to ‘excellent’ range relative to computer-based administration. However, the QuickDASH demonstrated significantly better reliability than the PROMIS UE CAT when comparing telephone and computer-based administration. Similarly, we observed a high rate of clinically important differences with the PROMIS UE CAT in both the verbal test-retest and telephone vs computer portions of this study. Based on these findings, we recommend that the PROMIS UE CAT only be administered in a computer-based format.
Clinical Relevance Statement: These findings suggest that differences in collection methods for the PROMIS UE CAT may systematically affect the scores obtained, which may erroneously influence the interpretation of postoperative scores for hand surgery patients.
Acknowledgements:
This investigation was supported by the University of Utah Study Design and Biostatistics Center, with funding in part from the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health, through Grant UL1TR002538.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References:
- 1.Lee VS, Kawamoto K, Hess R, et al. Implementation of a value-driven outcomes program to identify high variability in clinical costs and outcomes and association with reduced cost and improved quality. JAMA. 2016;316(10):1061–1072. [DOI] [PubMed] [Google Scholar]
- 2.Kamal RN, Hand Surgery Quality C. Quality and value in an evolving health care landscape. J Hand Surg Am. 2016;41(7):794–799. [DOI] [PubMed] [Google Scholar]
- 3.Clough JD, McClellan M. Implementing macra: Implications for physicians and for physician leadership. Jama. 2016;315(22):2397–2398. [DOI] [PubMed] [Google Scholar]
- 4.Cella D, Yount S, Rothrock N, et al. The patient-reported outcomes measurement information system (promis): Progress of an nih roadmap cooperative group during its first two years. Medical care. 2007;45(5 Suppl 1):S3–S11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Brodke DJ, Saltzman CL, Brodke DS. Promis for orthopaedic outcomes measurement. The Journal of the American Academy of Orthopaedic Surgeons. 2016;24(11):744–749. [DOI] [PubMed] [Google Scholar]
- 6.Cella D, Riley W, Stone A, et al. The patient-reported outcomes measurement information system (promis) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. J Clin Epidemiol. 2010;63(11):1179–1194. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Tyser AR, Beckmann J, Franklin JD, et al. Evaluation of the promis physical function computer adaptive test in the upper extremity. The Journal of hand surgery. 2014;39(10):2047–2051 e2044. [DOI] [PubMed] [Google Scholar]
- 8.Doring AC, Nota SP, Hageman MG, Ring DC. Measurement of upper extremity disability using the patient-reported outcomes measurement information system. J Hand Surg Am. 2014;39(6):1160–1165. [DOI] [PubMed] [Google Scholar]
- 9.Beckmann JT, Hung M, Voss MW, Crum AB, Bounsanga J, Tyser AR. Evaluation of the patient-reported outcomes measurement information system upper extremity computer adaptive test. The Journal of hand surgery. 2016;41(7):739–744 e734. [DOI] [PubMed] [Google Scholar]
- 10.Beaton DE, Wright JG, Katz JN, Grp UEC. Development of the quickdash: Comparison of three item-reduction approaches. J Bone Joint Surg Am. 2005;87a(5):1038–1046. [DOI] [PubMed] [Google Scholar]
- 11.Gummesson C, Ward MM, Atroshi I. The shortened disabilities of the arm, shoulder and hand questionnaire (quickdash): Validity and reliability based on responses within the full-length dash. BMC Musculoskelet Disord. 2006;7(44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.London DA, Stepan JG, Boyer MI, Calfee RP. Performance characteristics of the verbal quickdash. J Hand Surg Am. 2014;39(1):100–107. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Schwartzenberger J, Presson A, Lyle A, O'Farrell A, Tyser AR. Remote collection of patient-reported outcomes following outpatient hand surgery: A randomized trial of telephone, mail, and e-mail. J Hand Surg Am. 2017;42(9):693–699. [DOI] [PubMed] [Google Scholar]
- 14.Cella D, Hahn EA, Jensen SE, et al. Patient-reported outcomes in performance measurement. Research Triangle Park, North Carolina USA: RTI International; 2015. [PubMed] [Google Scholar]
- 15.Streiner D, Norman G, Cairney J. Health measurement scales: A practical guide to their development and use. Fifth Edition ed. New York, New York: Oxford University Press; 2015. [Google Scholar]
- 16.Austin PC, Hux JE. A brief note on overlapping confidence intervals. J Vasc Surg. 2002;36(1):194–195. [DOI] [PubMed] [Google Scholar]
- 17.Goldstein H, Healy MJR. The graphical presentation of a collection of means. J R Stat Soc a Stat. 1995;158(175–177. [Google Scholar]
- 18.Payton ME, Greenstone MH, Schenker N. Overlapping confidence intervals or standard error intervals: What do they mean in terms of statistical significance? J Insect Sci. 2003;3(34. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Cvetanovich GL, Gowd AK, Liu JN, et al. Establishing clinically significant outcome after arthroscopic rotator cuff repair. J Shoulder Elbow Surg. 2019;28(5):939–948. [DOI] [PubMed] [Google Scholar]
- 20.Nwachukwu BU, Fields K, Chang B, Nawabi DH, Kelly BT, Ranawat AS. Preoperative outcome scores are predictive of achieving the minimal clinically important difference after arthroscopic treatment of femoroacetabular impingement. Am J Sports Med. 2017;45(3):612–619. [DOI] [PubMed] [Google Scholar]
- 21.Franchignoni F, Vercelli S, Giordano A, Sartorio F, Bravini E, Ferriero G. Minimal clinically important difference of the disabilities of the arm, shoulder and hand outcome measure (dash) and its shortened version (quickdash). J Orthop Sports Phys Ther. 2014;44(1):30–39. [DOI] [PubMed] [Google Scholar]
- 22.Liu JN, Gowd AK, Redondo ML, et al. Establishing clinically significant outcomes after meniscal allograft transplantation. Orthop J Sports Med. 2019;7(1):2325967118818462. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Sorensen AA, Howard D, Tan WH, Ketchersid J, Calfee RP. Minimal clinically important differences of 3 patient-rated outcomes instruments. J Hand Surg Am. 2013;38(4):641–649. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Healthmeasures - interpret scores: Promis. Northwestern University; 2019. [February/22/2019]; Available from: http://www.healthmeasures.net/score-and-interpret/interpret-scores/promis. [Google Scholar]
- 25.Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155–163. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Portney L, Watkins M. Foundations of clinical research: Applications to practice. 3rd Edition ed. Upper Saddle River, New Jersey USA: Prentice Hall; 2009. [Google Scholar]
- 27.Mukaka MM. Statistics corner: A guide to appropriate use of correlation coefficient in medical research. Malawi Med J. 2012;24(3):69–71. [PMC free article] [PubMed] [Google Scholar]
- 28.Hinkle D, Wiersma W, Jurs S. Applied statistics for the behavioral sciences. Boston, MA: Houghton Mifflin; 2003. [Google Scholar]
- 29.Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Stat Med. 1998;17(1):101–110. [DOI] [PubMed] [Google Scholar]
- 30.University N. Healthmeasures - interpret scores: Promis. 2019. [January/25/2019]; Available from: http://www.healthmeasures.net/score-and-interpret/interpret-scores/promis.
- 31.Bernstein DN, Houck JR, Mahmood B, Hammert WC. Minimal clinically important differences for promis physical function, upper extremity, and pain interference in carpal tunnel release using region- and condition-specific prom tools. J Hand Surg Am. 2019;44(8):635–640. [DOI] [PubMed] [Google Scholar]
- 32.Kazmers NH, Qiu Y, Yoo M, Stephens AR, Zeidan M, Yue Z. Establishing the minimal clinically important difference for the promis upper extremity computer adaptive test version 2.0 in a nonshoulder hand and upper extremity population. 74th Annual Meeting of the American Society for Surgery of the Hand; San Antonion, TX: American Society for Surgery of the Hand; 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Mintken PE, Glynn P, Cleland JA. Psychometric properties of the shortened disabilities of the arm, shoulder, and hand questionnaire (quickdash) and numeric pain rating scale in patients with shoulder pain. J Shoulder Elbow Surg. 2009;18(6):920–926. [DOI] [PubMed] [Google Scholar]
- 34.Polson K, Reid D, McNair PJ, Larmer P. Responsiveness, minimal importance difference and minimal detectable change scores of the shortened disability arm shoulder hand (quickdash) questionnaire. Man Ther. 2010;15(4):404–407. [DOI] [PubMed] [Google Scholar]
- 35.Kazmers NH, Qiu Y, Yoo M, Stephens AR, Tyser AR, Zhang Y. The minimal clinically important difference of the promis and quickdash instruments in a nonshoulder hand and upper extremity patient population. J Hand Surg Am. 2020; [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Frost MH, Reeve BB, Liepa AM, Stauffer JW, Hays RD, Mayo FDAP-ROCMG. What is sufficient evidence for the reliability and validity of patient-reported outcome measures? Value Health. 2007;10 Suppl 2(S94–S105. [DOI] [PubMed] [Google Scholar]
- 37.Deyo RA, Katrina R, Buckley DI, et al. Performance of a patient reported outcomes measurement information system (promis) short form in older adults with chronic musculoskeletal pain. Pain Med. 2016;17(2):314–324. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Bjorner JB, Rose M, Gandek B, Stone AA, Junghaenel DU, Ware JE Jr. Method of administration of promis scales did not significantly impact score level, reliability, or validity. J Clin Epidemiol. 2014;67(1):108–113. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Broderick JE, Schneider S, Junghaenel DU, Schwartz JE, Stone AA. Validity and reliability of patient-reported outcomes measurement information system instruments in osteoarthritis. Arthritis Care Res (Hoboken). 2013;65(10):1625–1633. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Kisala PA, Boulton AJ, Cohen ML, et al. Interviewer- versus self-administration of promis(r) measures for adults with traumatic injury. Health Psychol. 2019;38(5):435–444. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Magnus BE, Liu Y, He J, et al. Mode effects between computer self-administration and telephone interviewer-administration of the promis((r)) pediatric measures, self- and proxy report. Qual Life Res. 2016;25(7):1655–1665. [DOI] [PMC free article] [PubMed] [Google Scholar]