Abstract
Background
Questionnaires are often used to quantify the subjective aspects of illness such as disability, coping strategies, or symptoms of depression. Most questionnaires were validated in a paper-based format, but direct entry into a computer is becoming commonplace. The aim of this study was to assess differences in outcome of questionnaires pertinent to hand and upper extremity illness when they were administered in a computer or paper format.
Methods
Ninety-nine patients completed both paper and web versions of the short forms of the Disabilities of Arm, Shoulder, and Hand; the Pain Catastrophizing Scale; the Short Health Anxiety Index (SHAI-6); the Patient Health Questionnaire Depression Scale; and a pain scale and the Pain Self-Efficacy Questionnaire sequentially during a single visit. We alternated starting with the paper or the web version after every five patients.
Results
The cohort consisted of 46 females and 53 males with a mean age of 49 years. There were significant differences in SHAI-6 and the pain scale, but not in the other measures. The intraclass correlation was high.
Conclusions
The use of these questionnaires in electronic format is valid provided that the small differences we observed are not important to the primary study question. In our opinion, for most studies, the advantages of using an electronic format outweigh the small additional variation that might be introduced in the measures in an electronic format.
Keywords: Questionnaires, Validation, Paper, Web
Introduction
Questionnaires are often used to quantify the subjective aspects of illness such as disability, coping strategies, or symptoms of depression [15, 21, 22]. Most questionnaires were validated in a paper-based format, but direct entry into a computer is becoming commonplace. Electronic questionnaires are more efficient and obviate the need for secondary data entry [19].
Previous studies comparing paper-based and computer-based completion of questionnaires document comparable results for the SF-36 and Consumer Quality Index Breast Care questionnaire [2, 10, 23]. Ritter and colleagues documented comparable results for 16 different instruments [16]. On the other hand, some studies have found that patients assigned to computer-based questionnaires tended to have higher scores [4, 5, 7].
In the realm of musculoskeletal illness, Shervin and collegues asked patients undergoing hip arthroplasty to complete the SF-36, EQ-5D, UCLA, WOMAC, and Harris Hip Score in touch screen, paper-based, and web-based formats and found comparable outcomes for the questionnaires in this crossover study [19]. The same was true for the SF-36 in healthy and chronic pain patients [17]. In patients with rheumatoid arthritis, small differences were found when patients completed both pen-and-paper and online version of questionnaires such as self-efficacy and pain visual analogue scale [1].
The aim of this study was to assess differences in outcome of questionnaires pertinent to hand and upper extremity illness when they were administered in computer or paper format. Our null hypothesis was that there were no differences in the short forms of the Disabilities of Arm, Shoulder, and Hand (QuickDASH); Pain Catastrophizing Scale (PCS-6); Short Health Anxiety Index (SHAI-6); Patient Health Questionnaire Depression Scale (PHQ-2); and the Pain Self-Efficacy Questionnaire (PSEQ) and ordinal pain scale when administered web based or paper based.
Materials and Methods
Study Design
One hundred twenty-four new or follow-up patients presenting to a hand surgeon were asked to enroll in this Institutional Review Board-approved study. Twenty-four patients declined participation because of the time requirement (about 30 min total). One hundred patients were enrolled in the study in March 2012. Informed consent was obtained before enrollment in the study. One patient withdrew when he saw the amount of work required, leaving 99 patients for analysis.
Patients completed both a paper-based version and a web-based version (completed on a laptop computer) of the QuickDASH [3, 9], the PCS-6 [20], the SHAI-6 [18], an ordinal pain scale, the PHQ-2 [11–13], and the PSEQ [6, 14] questionnaires. We alternated starting with the paper-based or the web-based version of the questionnaires after every five patients. The web-based questionnaires were completed on a laptop.
The data were collected and managed using Research Electronic Data Capture (REDCap) electronic data capture tools [8]. REDCap is a secure, web-based application designed to support data capture for research studies, providing (1) an intuitive interface for validated data entry, (2) audit trails for tracking data manipulation and export procedures, (3) automated export procedures for seamless data downloads to common statistical packages, and (4) procedures for importing data from external sources.
Patient Characteristics
There were 46 women (47 %) and 53 men (53 %), with a mean age of 49 years (range, 18 to 95 years). Sixty-five percent of visits were follow-up appointments. The majority of patients were married, worked full time, and sought advice for traumatic pathology (Table 1). There were no significant differences between patients starting with web-based questionnaires compared to patients who started with the paper-based version (Table 1).
Table 1.
Patient demographics
| Parameter | Cohort (n = 99) | Web start (n = 49) | Paper start (n = 50) | p values | |||
|---|---|---|---|---|---|---|---|
| Mean | SD | Range | Number | Mean | Mean | ||
| Age (years) | 49 | 18 | 18–95 | 50 | 48 | 0.58 | |
| Months since first pain experience | 16 | 44 | 0.6–350 | 19 | 14 | 0.56 | |
| Patient self-rating of health | 8 | 2 | 0–10 | 7.9 | 8.1 | 0.63 | |
| Education (years) | 16 | 3.5 | 0–29 | 16 | 15 | 0.087 | |
| Sex | 0.13 | ||||||
| Female | 46 (47 %) | 19 | 27 | ||||
| Male | 53 (54 %) | 30 | 23 | ||||
| Visit type | 0.36 | ||||||
| First | 34 (34 %) | 19 | 15 | ||||
| Follow-up | 65 (66 %) | 30 | 35 | ||||
| Prior surgery | 0.27 | ||||||
| Yes | 14 (14 %) | 5 | 9 | ||||
| No | 85 (86 %) | 44 | 41 | ||||
| Other pain conditions | 0.56 | ||||||
| Yes | 31 (31 %) | 14 | 17 | ||||
| No | 68 (69 %) | 35 | 33 | ||||
| Smoking | 0.83 | ||||||
| Yes | 17 (17 %) | 8 | 9 | ||||
| No | 82 (83 %) | 41 | 41 | ||||
| Marital status | 1.0 | ||||||
| Single | 32 (32 %) | 14 | 18 | ||||
| Living with partner | 8 (8.1 %) | 4 | 4 | ||||
| Married | 42 (42 %) | 22 | 20 | 0.99 | |||
| Separated/divorced | 15 (15 %) | 8 | 7 | ||||
| Widowed | 2 (2.0 %) | 1 | 1 | ||||
| Diagnosis | 0.15 | ||||||
| Fracture/crush/amputation/laceration | 44 (44 %) | 18 | 26 | ||||
| Carpal/cubital tunnel syndrome | 8 (8.0 %) | 4 | 4 | ||||
| Osteoarthritis | 1 (1.0 %) | 1 | 0 | ||||
| Trigger finger | 3 (3.0 %) | 3 | 0 | ||||
| Nonspecific arm pain | 6 (6.0 %) | 5 | 1 | ||||
| All other diagnoses | 37 (37 %) | 19 | 18 | ||||
| Working status | 0.76 | ||||||
| Full time | 56 (57 %) | 27 | 29 | ||||
| Part time | 11 (11 %) | 7 | 4 | ||||
| Homemaker | 3 (3.0 %) | 1 | 2 | ||||
| Retired | 12 (12 %) | 4 | 8 | ||||
| Unemployed-able to work | 8 (8.1 %) | 4 | 4 | ||||
| Unemployed-unable to work | 3 (3.0 %) | 2 | 1 | ||||
| Workers compensation | 1 (1.0 %) | 1 | 0 | ||||
| Currently on sick leave | 3 (3.0 %) | 2 | 1 | ||||
| Missing | 2 (2.0 %) | 1 | 1 | ||||
Evaluation
For more details concerning the questionnaires, we refer to an earlier work [21]. One patient did not complete the QuickDASH in both the paper-based and web-based version [3, 9]. There were four missing questions in four different patients in the web-based PSEQ; we imputed the mean of the patient’s other questions for these missing questions [6, 14].
In order to create the PCS-6, which is a shorter version of the PCS-13, we used the questions (2) “I feel I can’t go on”, (3) “It’s terrible and I think it’s never going to get any better”, (6) “I become afraid that the pain may get worse”, (8) “I anxiously want the pain to go away”, (11) “I keep thinking about how badly I want the pain to stop”, and (13) “I wonder whether something serious may happen” of the original 13-item PCS scale [20]. There was one patient who did not complete the PCS in both formats. Another patient had an invalid web-based PCS questionnaire, and we imputed the group mean for this missing value. One patient had one missing question for the web-based PCS; we imputed the mean of the patient’s other questions for this missing value.
We used the following questions of the 18-item SHAI to construct the shorter version, SHAI-6: (2) “I notice aches/pains… compared to other people”, (3) “As a rule I am… aware of bodily sensations or changes”, (9) “If I hear about an illness I… think I have it myself”, (12) “I… think I have a serious illness”, (15) “If I had a serious illness I would still… to enjoy life”, and (17) “A serious illness would ruin… aspect of my life” [18]. One patient did not complete the SHAI-6 in both formats. One patient had one missing question in the web-based SHAI; four patients had one missing question in the paper-based SHAI, and we imputed the mean of the patient’s other questions for these five missing questions. One patient did not complete the paper version of the PHQ-2, and we imputed the mean of the group for this missing value [11–13].
Statistical Evaluation
An a priori power analysis with two-tailed paired t test for our primary study question determined that 90 patients were needed to provide 80 % power to detect a 0.30-standard deviation difference between the web-based and the paper-based version of the QuickDASH with alpha 0.05. To correct for a possible 10 % loss to follow-up, 99 patients were enrolled.
Based on our sample size, we used parametric tests for further analysis. Paired t test was used to assess differences between continuous outcomes in the questionnaires. For the baseline parameters, we used independent t tests, and chi-square test was used to assess differences between dichotomous outcomes. To measure the correlation between web-based and paper-based questionnaires, we calculated the intraclass correlation coefficient (ICC) by using a two-way random model with absolute agreement.
Results
Differences in the Web-Based and Paper-Based Questionnaire Scores
Forty-nine patients started with the web-based questionnaires, and 50 with the paper-based format. There were small but significant differences between the mean SHAI in the web-based and the paper-based format (3.8 ± 2.9 vs. 4.2 ± 3.2, p = 0.030) and between the mean pain scale (3.1 ± 2.9 vs. 2.6 ± 2.6, p = 0.007), but not between the mean QuickDASH, PHQ-2, PCS-6, and PSEQ. The ICC was highest for PCS-6 (0.97) and lowest for the PHQ-2 (0.81) (Table 2).
Table 2.
Differences in questionnaires, n = 99
| Questionnaires | Mean | SD | Range | p value | ICC |
|---|---|---|---|---|---|
| QuickDASH | 0.72 | 0.96 | |||
| Paper | 30.0 | 23.8 | 0–93 | ||
| Web | 29.6 | 23.7 | 0–100 | ||
| PHQ-2 | 0.79 | 0.81 | |||
| Paper | 0.64 | 1.4 | 0–6 | ||
| Web | 0.62 | 1.0 | 0–4 | ||
| PCS-6 | 0.49 | 0.97 | |||
| Paper | 3.0 | 4.3 | 0–16 | ||
| Web | 3.1 | 4.0 | 0–17 | ||
| SHAI-6 | 0.030 | 0.92 | |||
| Paper | 4.2 | 3.2 | 0–18 | ||
| Web | 3.8 | 2.9 | 0–18 | ||
| Pain scale | 0.007 | 0.86 | |||
| Paper | 2.6 | 2.6 | 0–10 | ||
| Web | 3.1 | 2.9 | 0–10 | ||
| PSEQ | 0.069 | 0.94 | |||
| Paper | 48.5 | 14.1 | 6–60 | ||
| Web | 49.8 | 13.6 | 4–60 |
DASH Disabilities of Arm, Shoulder, and Hand, PHQ Patient Health Questionnaire, PCS Pain Catastrophizing Scale, SHAI Short Health Anxiety Index, PSEQ Pain Self-Efficacy Questionnaire, ICC intraclass correlation coefficient
Discussion
We rejected the null hypothesis for SHAI and the pain scale, but not for the other questionnaires. However, the differences in the mean scores of the SHAI and pain scale are small and likely clinically meaningless for most study questions. Outcomes for QuickDASH, PHQ-2, PCS-6, and PSEQ were comparable when administered in web-based or paper-based format. The ICC was high for most comparisons, which indicates a high correlation between the measures when they are administered in either web-based or paper-based format.
Differences in SHAI and the pain scale were statistically significant, but small, and their influence on clinical practice seems limited. This is in line with findings in literature [1, 4, 7].
This study has several points of limitation. We did not randomize whether the patients would start with the paper-based or web-based format of the questionnaires, but alternated after every five patients. Although this method is inferior to randomization, the cohorts were comparable. Twenty-four patients declined and one withdrew. These patients might be different from the participants, but it is a relatively small percentage and probably would not change our findings. We imputed the group mean of a questionnaire in two patients (one in each group) and used the mean of a patient’s other questions when there were individual questions missing for a couple of patients. This might have influenced our results. Our data for pain was complete, but we did use imputations of the patient’s other questions for the SHAI. The comparison for QuickDASH, PCS-6, and SHAI-6 were made based on 98 questionnaire pairs in 98 patients. For each of those questionnaires, the same patients did not complete both web-based and paper-based formats. We did not assess whether the patients preferred the web-based or paper-based format, and we also did not record the time necessary to complete the questionnaires. All patients completed the questionnaires at the same time, one after another, which was a practical decision. If there would be a longer timeframe between the administration time points of completion, this would have lowered the chance that patients could recall the answers. Finally, we found some small differences, but it is not possible to determine accuracy with subjective aspects of illness.
In contrast to the study of Zuidgeest and colleagues, we had more missing data in the web-based questionnaires than in the paper-based format [23]. Patients were not obligated to answer questions in the web-based format, which might be an explanation for the differences. The number of missing questions can also be related to the amount of time needed to complete all surveys since patients completed all questionnaires twice.
In line with earlier studies, we found very small differences between web-based and pen-and-paper questionnaires in most questionnaires [2, 10, 16, 19, 23]. The pain scale was higher and the SHAI-6 was lower in the electronic format. This is in contrast to other studies where, if there were differences, scores were higher in the online version [1, 4, 7].
We interpret this data to indicate that the use of these questionnaires in electronic format is valid provided that the small differences we observed are not important to the primary study question. Web-based questionnaires are more efficient and do not require secondary data entry. In our opinion, for most studies, the advantages of using an electronic format outweigh the small additional variation that might be introduced in the measures in an electronic format.
Acknowledgments
Conflicts of interest
The authors declare that they have no conflict of interest related to this work.
References
- 1.Athale N, Sturley A, Skoczen S, et al. A web-compatible instrument for measuring self-reported disease activity in arthritis. J Rheumatol. 2004;31:223–228. [PubMed] [Google Scholar]
- 2.Basnov M, Kongsved SM, Bech P, et al. Reliability of short form-36 in an internet- and a pen-and-paper version. Inform Health Soc Care. 2009;34:53–58. doi: 10.1080/17538150902779527. [DOI] [PubMed] [Google Scholar]
- 3.Beaton DE, Wright JG, Katz JN. Development of the QuickDASH: comparison of three item-reduction approaches. J Bone Joint Surg Am. 2005;87:1038–1046. doi: 10.2106/JBJS.D.02060. [DOI] [PubMed] [Google Scholar]
- 4.Carini RM, Hayek, J.C., Kuh, G.D., Kennedy, J.M., Ouimet, J.A. (2003) College student responses to web and paper surveys: does mode matter? Research in Higher Education 44.
- 5.Davis RN. Web-based administration of a personality questionnaire: comparison with traditional methods. Behav Res Methods Instrum Comput. 1999;31:572–577. doi: 10.3758/BF03200737. [DOI] [PubMed] [Google Scholar]
- 6.Estlander AM, Vanharanta H, Moneta GB, et al. Anthropometric variables, self-efficacy beliefs, and pain and disability ratings on the isokinetic performance of low back pain patients. Spine. 1994;19:941–947. doi: 10.1097/00007632-199404150-00013. [DOI] [PubMed] [Google Scholar]
- 7.Fouladi RT, McCarthy CJ, Moller NP. Paper-and-pencil or online? Evaluating mode effects on measures of emotional functioning and attachment. Assessment. 2002;9:204–215. doi: 10.1177/10791102009002011. [DOI] [PubMed] [Google Scholar]
- 8.Harris PA, Taylor R, Thielke R, et al. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377–381. doi: 10.1016/j.jbi.2008.08.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Hudak PL, Amadio PC, Bombardier C. Development of an upper extremity outcome measure: the DASH (disabilities of the arm, shoulder and hand) [corrected]. The Upper Extremity Collaborative Group (UECG) Am J Ind Med. 1996;29:602–608. doi: 10.1002/(SICI)1097-0274(199606)29:6<602::AID-AJIM4>3.0.CO;2-L. [DOI] [PubMed] [Google Scholar]
- 10.Kongsved SM, Basnov M, Holm-Christensen K, et al. Response rate and completeness of questionnaires: a randomized study of Internet versus paper-and-pencil versions. J Med Internet Res. 2007;9:e25. doi: 10.2196/jmir.9.3.e25. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16:606–613. doi: 10.1046/j.1525-1497.2001.016009606.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Kroenke K, Spitzer RL, Williams JB. The Patient Health Questionnaire-2: validity of a two-item depression screener. Med Care. 2003;41:1284–1292. doi: 10.1097/01.MLR.0000093487.78664.3C. [DOI] [PubMed] [Google Scholar]
- 13.Lowe B, Kroenke K, Grafe K. Detecting and monitoring depression with a two-item questionnaire (PHQ-2) J Psychosom Res. 2005;58:163–171. doi: 10.1016/j.jpsychores.2004.09.006. [DOI] [PubMed] [Google Scholar]
- 14.Nicholas MK. The pain self-efficacy questionnaire: taking pain into account. Eur J Pain. 2007;11:153–163. doi: 10.1016/j.ejpain.2005.12.008. [DOI] [PubMed] [Google Scholar]
- 15.Niekel MC, Lindenhovius AL, Watson JB, et al. Correlation of DASH and QuickDASH with measures of psychological distress. J Hand Surg Am. 2009;34:1499–1505. doi: 10.1016/j.jhsa.2009.05.016. [DOI] [PubMed] [Google Scholar]
- 16.Ritter P, Lorig K, Laurent D, et al. Internet versus mailed questionnaires: a randomized comparison. J Med Internet Res. 2004;6:e29. doi: 10.2196/jmir.6.3.e29. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Ryan JM, Corry JR, Attewell R, et al. A comparison of an electronic version of the SF-36 General Health Questionnaire to the standard paper version. Qual Life Res. 2002;11:19–26. doi: 10.1023/A:1014415709997. [DOI] [PubMed] [Google Scholar]
- 18.Salkovskis PM, Rimes KA, Warwick HM, et al. The Health Anxiety Inventory: development and validation of scales for the measurement of health anxiety and hypochondriasis. Psychol Med. 2002;32:843–853. doi: 10.1017/S0033291702005822. [DOI] [PubMed] [Google Scholar]
- 19.Shervin N, Dorrwachter J, Bragdon CR, et al. Comparison of paper and computer-based questionnaire modes for measuring health outcomes in patients undergoing total hip arthroplasty. J Bone Joint Surg Am. 2011;93:285–293. doi: 10.2106/JBJS.I.01104. [DOI] [PubMed] [Google Scholar]
- 20.Sullivan MPJ. The Pain Catastrophizing Scale: development and validation. Psychol Assess. 1995;7:525–532. doi: 10.1037/1040-3590.7.4.524. [DOI] [Google Scholar]
- 21.Vranceanu AM, Jupiter JB, Mudgal CS, et al. Predictors of pain intensity and disability after minor hand surgery. J Hand Surg Am. 2010;35:956–960. doi: 10.1016/j.jhsa.2010.02.001. [DOI] [PubMed] [Google Scholar]
- 22.Vranceanu AM, Safren S, Zhao M, et al. Disability and psychologic distress in patients with nonspecific and specific arm pain. Clin Orthop Relat Res. 2008;466:2820–2826. doi: 10.1007/s11999-008-0378-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Zuidgeest M, Hendriks M, Koopman L, et al. A comparison of a postal survey and mixed-mode survey using a questionnaire on patients' experiences with breast care. J Med Internet Res. 2011;13:e68. doi: 10.2196/jmir.1241. [DOI] [PMC free article] [PubMed] [Google Scholar]