Abstract
Objective:
This study compared response rates of paper and electronic versions of the Knee injury Osteoarthritis and Outcome Score questionnaire and examined the characteristics of patients who responded to each survey method.
Methods:
A total of 1486 patients registered by the Kaiser Permanente Anterior Cruciate Ligament Reconstruction Registry between 2005 and 2010 were included in this study. Response rates by survey modality for the overall cohort, by the specific time periods, and by age and sex at time of anterior cruciate ligament reconstruction were compared using χ2 tests or the Fisher exact test when appropriate. Independent Student t tests were used to compare the Knee injury Osteoarthritis and Outcome Scores of survey respondents.
Results:
The overall survey response rate was 42%. The 36% response rate in the electronic-survey group was significantly higher than the 22% response rate in the paper-survey group (p < 0.001). The electronic response rate was also significantly higher than the paper response rate at all follow-up times (35% vs 25% at 1 year, p = 0.004, 38% vs 20% at 2 years, p < 0.001, and 35% vs 21% at 3 years; p < 0.001) and among all age groups 19 years and older.
Conclusion:
Although the electronic survey produced higher response rates, it is not sufficient alone to replace the traditional paper version among this Kaiser Permanente population.
Introduction
Patient-reported outcome measures are tools often used before and after medical interventions to assess patients’ perspectives of symptoms, function, treatment preferences, and general well-being. These tools are important for evaluating the effectiveness of treatments and changes in disease trajectory.1 The Knee injury and Osteoarthritis Outcome Score (KOOS), a patient-reported outcome measure developed in 1995 by Ewa Roos and colleagues,2,3 evaluates both short- and long-term symptoms and function after knee injury and osteoarthritis progression. The questionnaire consists of 42 items in 5 separately scored subscales: “Pain, other Symptoms, Function (activities) in daily living (ADL), Function in Sport and Recreation (Sport/Rec), and knee-related Quality of Life (QOL).” The KOOS has been validated for a number of orthopaedic interventions, including anterior cruciate ligament (ACL) reconstruction, meniscectomy, and total knee replacement.4,5 The KOOS has also been validated within a number of populations consisting of varying ages, activity levels, and diseases.4 Traditionally, patient-reported outcome measures, like the KOOS, have been administered using paper questionnaires, with data collected, recorded, and computerized manually. This time-consuming and costly process can also compromise data quality.6
Today, patients are increasingly using the Internet to obtain and to exchange health-related information, allowing new modalities for the collection of patient-reported outcomes. Electronic questionnaires are desirable in both the health care and clinical research fields because they often allow for the collection of good-quality data, without missing or problematic responses. In addition, results from electronic questionnaires are often compiled automatically and are immediately available for use. Numerous studies have shown comparable results between patient-reported outcome measures administered via paper and those administered electronically.7–10 Among certain populations, several studies have reported electronic questionnaires to be preferred over traditional pen and paper methods.7,8
The objective of this study was to develop an Internet-based electronic KOOS questionnaire, to investigate its feasibility in replacing the traditional paper version, and to examine the characteristics of the patients who respond to the electronic and paper surveys.
Methods
A cross-sectional evaluation of two methods of data collection processes for the KOOS questionnaire was conducted. The KOOS questionnaire was implemented by the Kaiser Permanente (KP) Anterior Cruciate Ligament Reconstruction Registry (ACLRR) to track the current health status of its registered cohort of patients who underwent ACL reconstruction. This KP registry, developed in 2005, tracks surgical procedures, techniques, graft types, fixation types, surgical outcomes, complications, and patient-reported outcome measures. The registry’s data collection process and population has been previously described.11,12 In brief summary, the registry collects information from multiple Medical Centers located in six geographical Regions, all part of a large integrated health care system. The KP ACLRR began using the KOOS questionnaire in 2008 in a subset of participating Medical Centers located in Southern California (San Diego, Orange County, and Baldwin Park).
A total of 1486 patients registered by the KP ACLRR between October 1, 2005, and September 30, 2010, who were seen at the 3 participating Medical Centers were included in this study. A total of 26 surgeons contributed to the study, and institutional review board approval was obtained before its execution.
Data Collection Tools and Procedures
Paper Survey
The routine ACLRR KOOS data collection process involved both preoperative and postoperative administration of paper forms. Postoperative data collection occurred at 1, 2, and 5 years after surgery. At each participating Medical Center, the KOOS form was completed during regular office visits both before and after ACL reconstruction. The surgeons’ staff members were responsible for data collection and mailed all completed KOOS forms to the KP ACLRR data repository center. Because most patients did not return for office visits at 1, 2, and 5 years after surgery, a paper postoperative KOOS questionnaire was mailed to all patients. Approximately 6 months after their ACL reconstruction, all 1486 patients included in this study were mailed a letter informing them that at 1, 2, and 5 years after their ACL reconstruction, they would receive a questionnaire regarding their knee. Paper questionnaires along with self-addressed stamped envelopes were then mailed to all 1486 patients at the designated follow-up times. All patients were instructed to complete the survey even if they were no longer KP members. Patients were also informed that all questions were strictly confidential and would not be shared outside the organization.
Three weeks later, patients who had not yet responded to the initial survey were mailed a second survey. Finally, three weeks after the second survey was mailed, a voice message reminding patients to complete their survey was sent to all nonrespondents.
Electronic Survey
An electronic version of the KOOS was created internally at KP. After thorough pretesting for usability and comparability with the paper survey, the electronic survey was also adopted for follow-up at 1, 2, and 5 years after ACL reconstructive surgery. Of the 1486 patients included in this study, 830 (56%) had an e-mail address on file. Those patients were sent an electronic survey in addition to the mailed paper survey. A week before receiving the electronic survey, patients were sent a prenotification e-mail informing them that in a week they would receive an e-mail containing a link to a questionnaire regarding their knee. A week later, patients were sent an e-mail invitation containing the link to the questionnaire. All patients were informed that the survey would take 5 minutes to complete and were also given the option to opt out. Similar to the paper questionnaire, patients were instructed to complete the survey even if they were no longer KP members. They were also informed that their answers were strictly confidential and would not be shared.
A week after the invitation e-mail was sent, a follow-up reminder e-mail containing the link to the questionnaire was sent to all nonrespondents. Two weeks later, a second reminder e-mail was sent to nonrespondents. Finally, three weeks after the invitation e-mail was sent, a voice message was sent to patients who had not yet responded. All questions of the electronic survey were presented in the same order and with the same instructions as the paper KOOS questionnaire. Patients were unable to progress through the questionnaire without answering each question, but they could go back to modify previous responses. Identifiable information (ie, patients’ name, medical record numbers, and dates of surgery) was autopopulated into the first page of the questionnaire, but patients could make changes if for any reason the information was incorrect. The electronic KOOS survey was not connected to the patients’ electronic medical records.
Statistical Analysis
Frequencies, proportions, means, and standard deviations (SDs) were employed to describe the study sample and response rate of the surveys. Response rates by survey modality for the overall cohort, by the specific time periods, by age, and by sex at the time of ACL reconstruction were compared using χ2 tests or the Fisher exact test when appropriate. Independent Student t tests were used to compare the KOOS scores of survey respondents. SAS (Version 9.2; SAS Institute, Cary, NC) was used to analyze the data, with p < 0.05 as the statistical threshold.
Results
A total of 1486 patients were included in the study. The average age of study participants was 28 years (SD = 11) and 67% of participants were male (Table 1). Overall, the 36% response rate in the electronic group was significantly higher than the 22% in the paper group (p < 0.001). The electronic response rate was significantly higher than the paper response rate at 1 year (35% vs 25%, p = 0.004), 2 years (38% vs 20%, p = 0.004), and 5 years (35% vs 21%) after surgery (Table 2).
Table 1.
Study sample characteristics by type of survey received
| Characteristic | Paper, no. (%) | Electronic, no. (%) |
|---|---|---|
| Total number | 1486 | 830 |
| Men | 996 (67.0) | 537 (64.7) |
| Women | 490 (33.0) | 293 (35.3) |
| Mean age, years (SD) | 27.7 (11) | 28.3 (11) |
| Age group | ||
| < 19 years | 395 (26.6) | 86 (10.4) |
| 19–29 years | 497 (33.4) | 320 (38.6) |
| 30–39 years | 310 (20.9) | 212 (25.5) |
| 40–49 years | 216 (14.5) | 157 (18.9) |
| ≥ 50 years | 68 (4.6) | 55 (6.6) |
| Follow-up time | ||
| 1 year | 507 (34.1) | 291 (30.1) |
| 2 years | 597 (40.2) | 361 (43.5) |
| 5 years | 382 (25.7) | 178 (21.4) |
SD = standard deviation.
Table 2.
Survey response rate by follow-up time and sample characteristics
| Response rate | Total participants, mail | Paper response rate, no. (%) | Total participants, e-mail | Electronic response rate, no. (%) | p value |
|---|---|---|---|---|---|
| Follow-up time | |||||
| Overall | 1486 | 328 (22.1) | 830 | 301 (36.3) | < 0.001a |
| 1 year | 507 | 127 (25.0) | 291 | 101 (34.7) | 0.004a |
| 2 years | 597 | 121 (20.3) | 361 | 137 (38.0) | < 0.001a |
| 5 years | 382 | 80 (20.9) | 178 | 63 (35.4) | < 0.001a |
| Patient characteristics | |||||
| Men | 996 | 206 (20.7) | 537 | 182 (33.9) | < 0.001a |
| Women | 490 | 122 (24.9) | 293 | 119 (40.6) | < 0.001a |
| Age group | |||||
| < 19 years | 395 | 118 (29.9) | 86 | 21 (24.4) | 0.312 |
| 19–29 years | 497 | 92 (18.5) | 320 | 102 (31.9) | < 0.001a |
| 30–39 years | 310 | 49 (15.8) | 212 | 84 (39.6) | < 0.001a |
| 40–49 years | 216 | 57 (26.4) | 157 | 64 (40.8) | 0.003a |
| ≥ 50 years | 68 | 12 (17.6) | 55 | 30 (54.5) | < 0.001a |
Statistically significant.
Thirty-four percent of men in the electronic questionnaire group responded, and 21% of men in the paper survey group responded (p < 0.001). The corresponding response rate among women was 41% and 25%, respectively (p < 0.001) (Table 2). There was a significant difference in the response rates to electronic and paper surveys in participants between the ages of 19 and 29 years old (32% vs 19%, p < 0.001), those 30 to 39 years old (40% vs 16%, p < 0.001), those 40 to 49 years old (41% vs 26%, p = 0.003), and those 50 years and older (55% vs 18% p < 0.001). There was no significant difference in response rates to paper and electronic KOOS surveys in participants under 19 years of age.
With the exception of the Symptoms scale, there were no significant differences between any of the KOOS subscales in patients who responded to the electronic or paper survey (Table 3).
Table 3.
Average Knee injury and Osteoarthritis Outcome Score subscale score by survey method and follow-up time
| Survey method | Follow-up time | Total response, no. | Function in daily living score (SD) | Pain score (SD) | Quality of Life score (SD) | Symptoms score (SD) | Function in sport and recreation score (SD) |
|---|---|---|---|---|---|---|---|
| Paper | 1 year | 127 | 93.6 (8.6) | 87.5 (12.7) | 61.3 (23.9) | 65.8 (14.7) | 72.7 (20.7) |
| 2 years | 121 | 90.9 (15.0) | 85.6 (16.8) | 65.3 (25.6) | 65.3 (15.0) | 72.9 (24.6) | |
| 5 years | 80 | 94.0 (11.1) | 89.6 (14.7) | 77.6 (23.4) | 67.6 (14.9) | 79.1 (22.6) | |
| Paper total | 328 | 92.7 (11.9) | 87.3 (14.8) | 66.7 (25.2) | 66.0 (14.9) | 74.3 (22.8) | |
| Electronic | 1 year | 101 | 92.7 (11.0) | 86.9 (12.3) | 62.9 (22.3) | 80.7 (14.0) | 70.9 (22.4) |
| 2 years | 137 | 91.2 (13.1) | 85.3 (15.3) | 63.8 (24.8) | 79.6 (15.2) | 73.2 (23.3) | |
| 5 years | 63 | 93.7 (11.1) | 87.3 (16.0) | 71.7 (22.3) | 82.3 (15.6) | 75.6 (22.3) | |
| Electronic total | 301 | 92.2 (12.0) | 86.3 (14.5) | 65.1 (23.6) | 80.6 (14.9) | 72.9 (22.8) | |
| Total | 629 | 92.5 (12.0) | 86.8 (14.7) | 66.0 (24.5) | 73.0 (16.5) | 73.7 (22.8) |
SD = standard deviation.
Discussion
This study found that an Internet-based electronic KOOS questionnaire is at present not sufficient to replace the paper version among the KP ACL reconstruction population. Although the Internet-based electronic KOOS questionnaire yielded a significantly higher response rate than the paper version, it cannot yet completely replace the paper questionnaire. In this study, e-mail addresses were available for only 56% of study participants. Until this proportion increases to include nearly all eligible respondents, the paper version of the KOOS questionnaire will remain a necessary supplement to the data collection process.
The low overall response rate of 42% including both the electronic and paper methods could be partially because of the age of this study population. Numerous studies have found that younger people are less likely to participate in surveys.13–15 Dunn et al15 examined data from 7 general-population pen and paper surveys conducted in the United Kingdom between 1996 and 2002, to which a total of 27,797 people responded. They found that the overall survey response rate was lowest in the youngest age group of 18 to 29 years and increased with age until age 70 years in women and 80 years in men. The response rate among women age 18 to 29 years was 52%, whereas the response rate among the same age range in males was 32%. These are comparable response rates obtained in our sample.
In addition to the average age of our study cohort, the length of the questionnaire (42 questions, 1198 words, and 5 pages on paper) may have contributed to the low overall response rate. Although it is unknown how many patients started the paper survey but did not finish because of its length, we found that 3% (24/830) of participants who received the electronic survey failed to finish and were considered incompletes. Jepson et al16 examined associations between questionnaire length and response rate of mailed surveys among 1700 physicians and found that questionnaires above a threshold of 1000 words had lower response rates than those below it (38% vs 59%). The KOOS-Physical Function Shortform (KOOS-PS), a shortened KOOS questionnaire containing only 7 questions, may produce a higher response rate among the ACLRR population.
In this study, financial incentives were not used to motivate patients to complete the KOOS questionnaire. In a systematic review of 481 randomized controlled trials, Edwards et al17 found that the use of monetary incentives for completing mailed paper questionnaires can double the odds of response. In a meta-analysis of Web-based questionnaires, Göritz18 found that material incentives increase both response and retention rates. They found that material incentives increase the odds of a person responding to an electronic survey by 19% over the odds without incentives. They also found that an incentive increased retention by 4.2% on average. Beebe et al19 found that even a $2 cash incentive helped increase participation rates (54% response rate in an incentive group and 45% in a nonincentive group). The use of a small incentive may have helped increase the overall response rate in our specific population.
Our study found that regardless of a patient’s sex, time from surgery, and age older than 19 years, the electronic KOOS questionnaire produced significantly higher response rates than the paper survey. There are many advantages to using the computerized KOOS questionnaire, including improved data quality, faster data output, and reduced workloads. The electronic KOOS survey was designed to allow only complete responses, so no data elements were missing and more than one answer was never selected. Data entry was eliminated and results were ready for analysis the day after the survey was closed for participation. Manual preparation of the paper questionnaire is expensive and requires exhaustive work, which is greatly reduced with the use of the electronic questionnaire. Another benefit of the electronic KOOS is that all entries are date and time stamped. Velikova et al7 found significant problems with inaccurate entry of names, dates of birth, and postal codes when entered manually. To avoid such errors in the present study, we autopopulated the patient name, medical record number, and date of surgery into the first page of the electronic KOOS questionnaire.
In our study, 4 of the 5 average KOOS subscale scores were not different by survey modality or time from surgery. However, there was a difference in the Symptoms subscale between patients who completed the electronic survey and those who completed the mailed survey (80.6 vs 66.0, respectively). This difference in scores could be because of a selective response that may have occurred. It is possible that patients who were more likely to respond to the electronic survey did so because they were experiencing worse symptoms after surgery.
A major limitation to this study is the high percentage of patients whose e-mail addresses were not available. Of the total study population, 44% received only the paper questionnaire because their e-mail addresses were not on file. Because the electronic response rate was significantly higher than the paper response rate, the overall study response rate may have been higher if all participants had the option of answering either the paper or the electronic questionnaire. Several studies have demonstrated the use of hybrid data collection for increasing participation rates. Fowler et al20 achieved a 46% response rate to a mailed survey among 800 health care members, but they increased the response rate to 66% after following up with nonrespondents by phone. Similarly, Beebe et al19 achieved a 45% response rate to a mailed survey, which increased to 64% after completion of telephone follow-ups. McMahon et al21 used postal mail, fax, and e-mail to distribute a vaccine-related questionnaire to pediatric physicians. Their overall response rate increased from 39% before mixed mode of contact to a final of 53% after mixed modes. Although multimodal data collection has been found to increase participation rates, it also has the potential to cause measurement error and create challenges when combining the data obtained.22 Therefore, in this study, questions to the electronic survey were presented in the same order and with the same instructions as in the paper questionnaire. The paper and the electronic surveys were both scored in the same manner following instructions from the KOOS User’s Guide 2003 and scoring software (www.koos.nu). As the use of patient-reported outcome measures increases, organizations should focus their efforts on obtaining and maintaining all current modes of contact for their members.
Results from this study show that use of an Internet-based KOOS questionnaire among the younger ACL reconstruction population is promising, but not sufficient to replace the paper version at this time. As the proportion of e-mail addresses on file increases, replacing paper questionnaires with electronic versions may become more feasible. In addition, other electronic survey delivery modes, such as via mobile devices, may be more attractive to this population, thereby increasing the reach of the electronic survey. Future research should investigate the use of various devices for the delivery of the electronic KOOS questionnaire combined with a shortened version of the survey. Additional focus on the underlying characteristics of patients who opt to complete the electronic version would aid in designing an attractive questionnaire.
Acknowledgments
Kathleen Louden, ELS, of Louden Health Communications provided editorial assistance.
Footnotes
Disclosure Statement
The author(s) have no conflicts of interest to disclose.
An Art or Craft
Scientific research is not itself a science; it is still an art or craft.
— The Scientist in Action, William H George, London, UK: Williams and Norgate Ltd; 1938
References
- 1.Revicki D, Hays RD, Cella D, Sloan J. Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. J Clin Epidemiol. 2008 Feb;61(2):102–9. doi: 10.1016/j.jclinepi.2007.03.012. DOI: http://dx.doi.org/10.1016/j.jclinepi.2007.03.012. [DOI] [PubMed] [Google Scholar]
- 2.Roos EM, Roos HP, Ekdahl C, Lohmander LS. Knee injury and Osteoarthritis Outcome Score (KOOS)—validation of a Swedish version. Scand J Med Sci Sports. 1998 Dec;8(6):439–48. doi: 10.1111/j.1600-0838.1998.tb00465.x. DOI: http://dx.doi.org/10.1111/j.1600-0838.1998.tb00465.x. [DOI] [PubMed] [Google Scholar]
- 3.Roos EM, Roos HP, Lohmander LS, Ekdahl C, Beynnon BD. Knee injury and Osteoarthritis Outcome Score (KOOS)—development of a self-administered outcome measure. J Orthop Sports Phys Ther. 1998;28(2):88–96. doi: 10.2519/jospt.1998.28.2.88. DOI: http://dx.doi.org/10.2519/jospt.1998.28.2.88. [DOI] [PubMed] [Google Scholar]
- 4.Roos EM, Lohmander LS. The Knee injury and Osteoarthritis Outcome Score (KOOS): from joint injury to osteoarthritis. Health Qual Life Outcomes. 2003 Nov 3;1:64. doi: 10.1186/1477-7525-1-64. DOI: http://dx.doi.org/10.1186/1477-7525-1-64. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Roos EM, Toksvig-Larsen S. Knee injury and Osteoarthritis Outcome Score (KOOS)—validation and comparison to the WOMAC in total knee replacement. Health Qual Life Outcomes. 2003 May 25;1:17. doi: 10.1186/1477-7525-1-17. DOI: http://dx.doi.org/10.1186/1477-7525-1-17. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Rolfson O, Salomonsson R, Dahlberg LE, Garellick G. Internet-based follow-up questionnaire for measuring patient-reported outcome after total hip replacement surgery—reliability and response rate. Value Health. 2011 Mar-Apr;14(2):316–21. doi: 10.1016/j.jval.2010.08.004. DOI: http://dx.doi.org/10.1016/j.jval.2010.08.004. [DOI] [PubMed] [Google Scholar]
- 7.Velikova G, Wright EP, Smith AB, et al. Automated collection of quality-of-life data: a comparison of paper and computer touch-screen questionnaires. J Clin Oncol. 1999 Mar;17(3):998–1007. doi: 10.1200/JCO.1999.17.3.998. [DOI] [PubMed] [Google Scholar]
- 8.Wilson AS, Kitas GD, Carruthers DM, et al. Computerized information-gathering in specialist rheumatology clinics: an initial evaluation of an electronic version of the Short Form 36. Rheumatology (Oxford) 2002 Mar;41(3):268–73. doi: 10.1093/rheumatology/41.3.268. DOI: http://dx.doi.org/10.1093/rheumatology/41.3.268. [DOI] [PubMed] [Google Scholar]
- 9.Cook AJ, Roberts DA, Henderson MD, Van Winkle LC, Chastain DC, Hamill-Ruth RJ. Electronic pain questionnaires: a randomized, crossover comparison with paper questionnaires for chronic pain assessment. Pain. 2004 Jul;110(1–2):310–7. doi: 10.1016/j.pain.2004.04.012. DOI: http://dx.doi.org/10.1016/j.pain.2004.04.012. [DOI] [PubMed] [Google Scholar]
- 10.Gwaltney CJ, Shields AL, Shiffman S. Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: a meta-analytic review. Value Health. 2008 Mar-Apr;11(2):322–33. doi: 10.1111/j.1524-4733.2007.00231.x. DOI: http://dx.doi.org/10.1111/j.1524-4733.2007.00231.x. [DOI] [PubMed] [Google Scholar]
- 11.Paxton EW, Inacio MC, Kiley ML. The Kaiser Permanente implant registries: effect on patient safety, quality improvement, cost effectiveness, and research opportunities. Perm J. 2012 Spring;16(2):36–44. doi: 10.7812/tpp/12-008. DOI: http://dx.doi.org/10.7812/TPP/12-008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Maletis GB, Inacio MC, Funahashi TT. Analysis of 16,192 anterior cruciate ligament reconstructions from a community-based registry. Am J Sports Med. 2013 Sep;41(9):2090–8. doi: 10.1177/0363546513493589. DOI: http://dx.doi.org/10.1177/0363546513493589. [DOI] [PubMed] [Google Scholar]
- 13.Burg JA, Allred SL, Sapp JH., 2nd The potential for bias due to attrition in the National Exposure Registry: an examination of reasons for nonresponse, nonrespondent characteristics, and the response rate. Toxicol Ind Health. 1997 Jan-Feb;13(1):1–13. doi: 10.1177/074823379701300101. [DOI] [PubMed] [Google Scholar]
- 14.Eagan TM, Eide GE, Gulsvik A, Bakke PS. Nonresponse in a community cohort study: predictors and consequences for exposure-disease associations. J Clin Epidemiol. 2002 Aug;55(8):775–81. doi: 10.1016/s0895-4356(02)00431-6. DOI: http://dx.doi.org/10.1016/S0895-4356(02)00431-6. [DOI] [PubMed] [Google Scholar]
- 15.Dunn KM, Jordan K, Lacey RJ, Shapley M, Jinks C. Patterns of consent in epidemiologic research: evidence from over 25,000 responders. Am J Epidemiol. 2004 Jun 1;159(11):1087–94. doi: 10.1093/aje/kwh141. DOI: http://dx.doi.org/10.1093/aje/kwh141. [DOI] [PubMed] [Google Scholar]
- 16.Jepson C, Asch DA, Hershey JC, Ubel PA. In a mailed physician survey, questionnaire length had a threshold effect on response rate. J Clin Epidemiol. 2005 Jan;58(1):103–5. doi: 10.1016/j.jclinepi.2004.06.004. DOI: http://dx.doi.org/10.1016/j.jclinepi.2004.06.004. [DOI] [PubMed] [Google Scholar]
- 17.Edwards P, Roberts I, Clarke M, et al. Increasing response rates to postal questionnaires: systematic review. BMJ. 2002 May 18;324(7347):1183. doi: 10.1136/bmj.324.7347.1183. DOI: http://dx.doi.org/10.1136/bmj.324.7347.1183. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Göritz AS. Incentives in Web studies: methodological issues and a review. International Journal of Internet Science. 2006;1(1):58–70. [Google Scholar]
- 19.Beebe TJ, Davern ME, McAlpine DD, Call KT, Rockwood TH. Increasing response rates in a survey of Medicaid enrollees: the effect of a prepaid monetary incentive and mixed modes (mail and telephone) Med Care. 2005 Apr;43(4):411–4. doi: 10.1097/01.mlr.0000156858.81146.0e. DOI: http://dx.doi.org/10.1097/01.mlr.0000156858.81146.0e. [DOI] [PubMed] [Google Scholar]
- 20.Fowler FJ, Jr, Gallagher PM, Stringfellow VL, Zaslavsky AM, Thompson JW, Cleary PD. Using telephone interviews to reduce nonresponse bias to mail surveys of health plan members. Med Care. 2002 Mar;40(3):190–200. doi: 10.1097/00005650-200203000-00003. [DOI] [PubMed] [Google Scholar]
- 21.McMahon SR, Iwamoto M, Massoudi MS, et al. Comparison of e-mail, fax, and postal surveys of pediatricians. Pediatrics. 2003 Apr;111(4 Pt 1):e299–303. doi: 10.1542/peds.111.4.e299. DOI: http://dx.doi.org/10.1542/peds.111.4.e299. [DOI] [PubMed] [Google Scholar]
- 22.de Leew ED. To mix or not to mix data collection modes in surveys. Journal of Official Statistics. 2005;21(2):233–55. [Google Scholar]