Abstract
Background
Long-term follow-up studies are an important tool in the evaluation of orthopaedic illness and its treatment options. However, a patient’s participation in a follow-up study may be affected by several factors, leading to variability in response rates and the risk of selection bias.
Questions/purposes
(1) What is the average response rate in hand surgery questionnaire studies? (2) What factors are associated with higher and lower response rates to research questionnaires? (3) What factors are associated with higher and lower contact, initial participation, and completion rates?
Methods
We included 798 adult patients who were enrolled in one of 12 questionnaire follow-up studies in the hand and upper extremity service of our institution. All included studies evaluated patient-reported outcomes for the surgical treatment of upper extremity conditions using questionnaires and all used the same enrollment design. Patients were invited by letter to ask if they would be willing to participate, and we informed them that they would be contacted by telephone at least three times if they did not respond to the letter. Patients were contacted at a median of 6.6 years (interquartile range [IQR] 3.7 to 11) after surgery. The successful response rate was 49% (390 of 798 patients). We manually reviewed records to collect data on patient characteristics, and we performed bivariate analysis and multivariable logistic regression analysis to identify factors associated with the contact rate (percentage of patients reached by either mail, phone, or email), initial response rate (percentage of reached patients who initiated participation), completion rate (percentage of patients who initiated participation and completed the entire follow-up questionnaire), and our primary outcome successful response rate (percentage of patients who were contacted and who completed the entire questionnaire).
Results
The average response rate in hand surgery questionnaire studies was 49% (390 of 798 patients). In the multivariable analysis, enrollment of women (odds ratio 1.43 [95% confidence interval 1.03 to 1.97]; p = 0.031) was independently associated with higher response rates. On the contrary, a longer follow-up time from surgery (OR 0.95; 95% CI 0.92 to 0.99]; p = 0.015) and multiple researchers contacting patients (OR 0.51 [95% CI 0.37 to 0.71]; p < 0.001) were independently associated with lower response rates. The contact rate was higher for women (OR 1.46 [95% CI 1.03 to 2.06]; p = 0.034) and patients with higher income (OR 1.000007 [95% CI 1.000001 to 1.000013]; p = 0.019). The contact rate was lower in patients with a longer follow-up time from surgery (OR 0.93 [95% CI 0.90 to 0.97]; p = 0.001). The initial participation rate was lower when patients were contacted by multiple researchers (OR: 0.34 [95% CI 0.23 to 0.52]; p < 0.001). Studies with a lower number of questions (36; IQR 22 to 46) were completed more frequently than studies with a higher number of questions (51; IQR 39 to 67; p = 0.044).
Conclusions
Studies assessing long-term outcomes that have a large proportion of men and longer follow-up time tend to have lower response rates. When performing a follow-up study, it seems beneficial to have one researcher contact the patients and use a shorter questionnaire. Results of this study can help clarify the response rates in hand surgery follow-up questionnaire studies and help with the planning of future follow-up studies.
Level of Evidence
Level II, prognostic study.
Introduction
Long-term follow-up studies are an important tool to evaluate treatment outcomes after upper extremity and orthopaedic surgery [20]. Long-term in-person follow-up is challenging and is becoming more difficult because of the cost of parking, lost time from work or personal time, and patients moving to other locations. Subsequently, studies are becoming more reliant on questionnaires that can be administered online or over the telephone. In prior studies, patient questionnaire response rates varied between 24% and 80%, and it is important to determine the expected response rate before study initiation for feasibility purposes [2, 6, 13, 21, 22, 29]. Additionally, differences between responders and non-responders in their response to follow-up questionnaires may lead to selection or reporting bias influencing the generalizability and validity of a study [20-22, 24].To date, there is no consensus on the factors associated with response to follow-up studies [2, 6, 13, 21, 22, 24, 29, 31].
Prior studies found patients who are men, have less education, and who are younger less frequently respond to research questionnaires by phone or mail [2, 6, 13, 21]. However, other studies did not observe a difference in response rate between gender and education, and reported that younger patients and employed patients less frequently respond to research questionnaires by either mail, email, or phone [22, 29]. Studies that assessed the difference in response rate to follow-up in clinic for research purposes found that both employed and unemployed patients attend follow-up visits, and that older patients less frequently attend these visits [24, 31]. However, all of these studies used different methods to collect follow-up data and assessed different conditions. Some only collected data by mail and enrolled patients in clinic, whereas others enrolled patients by mail, phone, and/or email that were retrospectively identified. The use of multiple modalities such as mail, email, and phone may help increase the response rate and overcome barriers to follow-up, such as travel time, travel costs, and parking costs. Also, it may be helpful to understand at what point of contact do patients stop responding and which factors are associated with no response. In addition, we wanted to clarify whether most patients are unwilling to participate or are researchers unable to get in contact with patients.
Therefore, we asked: (1) What is the average response rate in hand surgery questionnaire studies? (2) What factors are associated with higher and lower response rates to research questionnaires? (3) What factors are associated with higher and lower contact, initial participation, and completion rates?
Patients and Methods
We included 12 institutional review board-approved hand surgery studies that used questionnaires to evaluate patient-reported outcomes that were conducted by the clinical research team of the hand and upper extremity service of our institution. All included studies enrolled English-speaking adult patients with an upper extremity condition who were not pregnant and were treated at one of five affiliated hospitals between 2002 and 2017 (see Appendix 1; Supplemental Digital Content 1, http://links.lww.com/CORR/A356). The 12 included studies retrospectively identified 821 eligible patients who were contacted to complete a follow-up research questionnaire. Twenty-three patients participated in two different studies and were contacted twice. To prevent overcounting, we only assessed the first study the patient was enrolled in (see Fig. 1; Supplemental Digital Content 2, http://links.lww.com/CORR/A357). A total of 798 patients were analyzed.
We retrospectively reviewed the medical records of these 798 patients to collect information on patient and study characteristics (age, sex, race, diabetes, tobacco use, marital status, and occupation). We used the patient’s ZIP code on file at the time of follow-up as a surrogate for the median household income using the most current income statistics from the United States Census Bureau [7, 32] The ZIP code was also a surrogate to calculate how far patients lived from the hospital of treatment. We identified which patients were contacted, initially participated, and completed the questionnaires using enrollment logs and outcome sheets from Research Electronic Data Capture (REDCap, Nashville, TN, USA). In addition, we reviewed at what time and day questionnaires were completed.
Study-specific characteristics were the number of questions per study (see Appendix 2; Supplemental Digital Content 3, http://links.lww.com/CORR/A358), the number of researchers involved in enrollment, and the modality by which the questionnaires were completed. The median time to follow-up was defined as the number of days between the date of treatment (for example ganglion excision or Guyon canal release) and the date the questionnaire was conducted.
Outcome Variables
Standard definitions of the American Association for Public Opinion Research were used to give a detailed description of questionnaire participation and define the contact rate and response rate [34]. Our primary outcome of interest was the response rate, which was defined as the percentage of eligible patients who were contacted (n = 798) and completed the entire follow-up questionnaire (Table 1). Additionally, we assessed the following outcomes: contact rate, initial participation rate, and completion rate. The contact rate was defined as the percentage of eligible patients who were contacted and reached, either by standard mail, telephone, or email. The initial participation rate was defined as the percentage of patients who were reached and decided to participate and completed at least one component of the questionnaire. The completion rate was defined as the percentage of patients who initiated participation and completed the entire follow-up questionnaire [20]. Data regarding the time and day of completion were available for 73% (286 of 390) of the patients who completed the questionnaire.
Table 1.
Overview of the contact rate, initial participation rate, completion rate, and response rate per follow-up studies
| Diagnoses per study | Eligible patients | Response rate (primary outcome) | Contact rate | Initial participation rate | Completion rate |
| Carpal arthropathy, flexor carpi ulnaris enthesopathy | 61 | 51% (31 of 61) | 70% (43 of 61) | 72% (31 of 43) | 100% (31 of 31) |
| Digital neuroma | 85a | 32% (27 of 85) | 46% (39 of 85) | 73% (28 of 39) | 97% (27 of 28 ) |
| Essex-Lopresti injury | 15 | 67% (10 of 15) | 80% (12 of 15) | 83% (10 of 12) | 100% (10 of 10) |
| Guyon's canal syndrome | 64 | 47% (30 of 64) | 73% (47 of 64) | 64% (30 of 47) | 100% (30 of 30) |
| Hypothenar hammer syndrome | 25 | 56% (14 of 25) | 76% (19 of 25) | 79% (15 of 19) | 93% (14 of 15) |
| Pisotriquetral arthrosis, pisiform non-union, pisiform subluxation | 45a | 67% (30 of 45) | 76% (34 of 45) | 88% (30 of 34) | 100% (30 of 30) |
| Post-traumatic distal radioulnar join dysfunction | 83 | 61% (51 of 83) | 73% (61 of 83) | 85% (52 of 61) | 98% (51 of 52) |
| Proximal biceps tendon tear | 42 | 52% (22 of 42) | 74% (31 of 42) | 74% (23 of 31) | 96% (22 of 23) |
| Radial sensory nerve neuroma | 36a | 50% (18 of 36) | 73% (27 of 36) | 69% (18 of 27) | 100% (18 of 18) |
| Severe necrotizing soft tissue disease | 84a | 43% (36 of 84) | 54% (45 of 84) | 84% (38 of 45) | 95% (36 of 38) |
| Triangular fibrocartilage complex tear | 84 | 57% (48 of 84) | 64% (54 of 84) | 89% (48 of 54) | 100% (48 of 48) |
| Wrist ganglia | 174 | 42% (73 0f 174) | 74% (129 of 174) | 57% (74 of 129) | 99% (73 of 74) |
| All 12 studies | 798 | 49% (390 of 798) | 68% (541 of 798) | 73% (397 of 541) | 98% (390 of 397) |
Patients who were previously enrolled in one of these 12 studies were substracted from the total number of eligible patients per study to prevent overcounting in our analysis. The total number of eligible patients was 86 for the digital neuroma study, 53 for the pisotriquetral arthrosis study, 49 for the radial sensory nerve neuroma study, and 85 for the severe necrotizing soft tissue disease study.
Enrollment Design of the 12 included Follow-up Studies
Each of the 12 included studies used a similar enrollment design. An invitation letter explaining the study and a postcard on which the patient could accept or refuse participation were mailed by standard mail (see Fig. 2; Supplemental Digital Content 4, http://links.lww.com/CORR/A359). We included a pre-stamped envelope in which the postcard could be returned free of charge. If a patient did not respond to the letter within 2 weeks, they were contacted by telephone to ask if they were willing to participate in the study. Patients who were willing to participate could complete the questionnaire either over the telephone or by email. Patients had the option to stop participation at any time throughout the study, with partial questionnaire completion as a consequence. All questionnaires were administered in REDCap, which is a secure electronic data capturing tool hosted on a secure server at our institution [18]. Researchers made at least three attempts at different times of the day to reach patients over the telephone. The goal of this study design was to evaluate long-term outcome in patients who have been treated in the past and were not enrolled in a prospective follow-up study at time of their treatment or outpatient visit. All patients included in these 12 studies were identified retrospectively and the request to participate in a follow-up research questionnaire was unexpected and not consented at the time these patients were treated.
Study Population
The 798 patients had a median follow-up duration of 6.6 years (interquartile range [IQR] 3.7 to 11). The patient population consisted of 383 men and 415 women with a median (range) age of 54 years at the time of follow-up (19 to 96) (Table 2). Upper-extremity diagnoses consisted of wrist ganglion, hypothenar hammer syndrome, digital neuroma, Guyon’s canal syndrome, radial sensory nerve neuroma, proximal biceps tendon tear, Essex-Lopresti injury, severe necrotizing soft-tissue disease, post-traumatic distal radioulnar joint dysfunction, carpal arthropathy, triangular fibrocartilage complex tear, pisotriquetral arthrosis, flexor carpi ulnaris enthesopathy, pisiform nonunion, and pisiform subluxation. Based on the ZIP code, patients had a median income of USD 81,040 (IQR 62,933 to 101,229) and lived at a median of 17 miles (IQR 6.6 to 38) from one of the hospitals [32].
Table 2.
Factors associated with response to a follow-up questionnaire: bivariate analysis
| Patient characteristics | All 100% (798) | Response | p value | |
| Yes 49% (390) | No 51% (408) | |||
| Age (years), median (IQR) | 54 (41 to 64) | 57 (46 to 66) | 51 (38 to 61) | < 0.001 |
| Sex, % (n) | 0.005 | |||
| Female | 52% (415 of 798) | 57% (223 of 390) | 47% (192 of 408) | |
| Race, % (n)a | 0.144 | |||
| White | 81% (622 of 765) | 84% (316 of 378) | 79% (306 of 387) | |
| Asian | 2% (17 of 765) | 2% (8 of 378) | 2% (9 of 387) | |
| Black | 7% (56 of 765) | 8% (29 of 378) | 7% (27 of 387) | |
| Hispanic | 7% (54 of 765) | 6% (21 of 378) | 9% (33 of 387) | |
| Other | 2% (16 of 765) | 1% (4 of 378) | 3% (12 of 387) | |
| Marital status, % (n)b | 0.061 | |||
| Single | 36% (275 of 766) | 32% (120 of 374) | 40% (155 of 392) | |
| Married | 53% (406 of 766) | 57% (214 of 374) | 49% (192 of 392) | |
| Divorced | 9% (67 of 766) | 8% (29 of 374) | 10% (38 of 392) | |
| Widowed | 2% (18 of 766) | 3% (11 of 374) | 2% (7 of 392) | |
| Tobacco dependency reported in chart, % (n) | 10% (76) | 10% (40) | 9% (36) | 0.547 |
| Diabetes, % (n) | 10% (79) | 11% (41) | 9% (38) | 0.636 |
| Occupation, % (n)c | 0.163 | |||
| Employed | 54% (390 of 726) | 51 (191 of 378) | 57% (199 of 348) | |
| Unemployed | 19% (137 of 726) | 19% (72 of 378) | 19% (65 of 348) | |
| Student | 2% (11 of 726) | 1% (4 of 378) | 2% (7 of 348) | |
| Retired | 23% (163 of 726) | 25% (95 of 378) | 20% (68 of 348) | |
| Unable to work | 3% (25 of 726) | 4% (16 of 378) | 3% (9 of 348) | |
| Median income (in USD) based on ZIP code, median (IQR)d | 81,040 (62,933 to 101,229) | 84,847 (65,476 to 102,342) | 78,343 (61,375 to 96,857) | 0.015 |
| Distance to hospital (in miles) based on ZIP code, median (IQR)e | 17 (6.6 to 38) | 15 (6.1 to 37) | 18 (7.4 to 38) | 0.884 |
| Time from surgery to follow-up (years), median (IQR) | 6.6 (3.7 to 11.2) | 6.0 (3.7 to 10.7) | 7.2 (3.8 to 11.6) | 0.109 |
| More than one researcher calling, % (n) | 45% (359 of 798) | 38% (148 of 390) | 52% (211 of 408) | < 0.001 |
Number of values in variables with missing data:
n =765;
n = 766;
n = 726;
n = 786;
n = 793.
Statistical Analysis
Missing data was non-differential and completely at random. Pair-wise deletion was used to handle missing data in bivariate analysis and listwise deletion in multivariable analysis. Fisher’s exact test and a t-test were used to study the association between explanatory variables and outcome variables (response rate, contact rate, initial participation rate, and completion rate). For nonparametric, continuous data, we used Wilcoxon’s rank sum test. To mitigate confounding, variables with a p value less than 0.20 in the bivariate analysis were entered in a multivariable logistic regression model to identify factors independently associated with response, contact, and participation. A p value of less than 0.05 was considered statistically significant.
Results
The overall response rate of all studies combined was 49% (390 of 798 patients) (Table 1).
In the multivariable analysis, enrollment of women (odds ratio 1.43 [95% confidence interval 1.03 to 1.97]; p = 0.031) was independently associated with higher response rates (Table 3). On the contrary, a longer follow-up time from surgery (odds ratio 0.95 [95% CI 0.92 to 0.99]; p = 0.015) and patient enrollment by multiple researchers (OR 0.51 [95% CI 0.37 to 0.71]; p < 0.001) were independently associated with lower response rates. Bivariate analyses showed that patients who successfully responded to the follow-up questionnaire were more often older, women, less frequently contacted by more than one researcher, and more frequently had a higher median household income (Table 2).
Table 3.
Factors associated with response to a follow-up questionairre: multivariable analysis
| Patient characteristics | OR (95% CI) | p value |
| Age (years) | 1.01 (0.996 to 1.03) | 0.134 |
| Female sex | 1.43 (1.03 to 1.97) | 0.031 |
| Racea | ||
| Asian | 0.92 (0.30 to 2.79) | 0.881 |
| Black | 1.38 (0.73 to 2.59) | 0.332 |
| Hispanic | 0.83 (0.43 to 1.59) | 0.569 |
| Other | 0.42 (0.13 to 1.38) | 0.152 |
| Marital statusb | ||
| Single | Reference | |
| Married | 1.24 (0.86 to 1.81) | 0.255 |
| Divorced | 0.61 (0.33 to 1.15) | 0.124 |
| Widowed | 1.14 (0.37 to 3.47) | 0.819 |
| Occupationc | ||
| Unemployed | 1.10 (0.72 to 1.69) | 0.659 |
| Student | 0.49 (0.12 to 2.05) | 0.330 |
| Retired | 0.96 (0.56 to 1.66) | 0.893 |
| Unable to work | 2.54 (0.99 to 6.55) | 0.054 |
| Median income based on ZIP coded | 1.000005 (0.9999998 to 1.000011) | 0.058 |
| Time from surgery to follow-up (years) | 0.95 (0.92 to 0.99) | 0.015 |
| Study characteristics | ||
| More than one researcher calling | 0.51 (0.37 to 0.71) | < 0.001 |
Number of values in variables with missing data:
n = 765;
n = 766;
n = 726;
n = 786.
Higher income (OR 1.000007 [95% CI 1.000001 to 1.000013]; p = 0.019) and enrollment of women (OR 1.46 [95% CI 1.03 to 2.06; p = 0.034) were independently associated with a higher contact rate (Table 4); whereas a longer follow-up time from surgery (OR 0.93 [95% CI 0.90 to 0.97]; p = 0.001) was independently associated with a lower contact rate. Bivariate analyses showed that patients who were reached (successfully contacted) by either phone or mail were more frequently older, women, retired, less frequently single, more frequently earned a higher median household income, more often lived closer to the hospital, and had more frequently a shorter follow-up time (see Appendix 3; Supplemental Digital Content 5, http://links.lww.com/CORR/A360). Multiple researchers enrolling patients (OR 0.34 [95% CI 0.23 to 0.52]; p < 0.001) and a race other than white, Asian, black, or Hispanic (OR 0.25 [95% CI 0.08 to 0.94]; p = 0.040) were independently associated with a lower initial participation rate (Table 5). Bivariate analyses showed that patients who participated were less frequently called by more than one researcher, more frequently lived further from the hospital, and more often reported a race other than white, black or Hispanic (see Appendix 4; Supplemental Digital Content 6, http://links.lww.com/CORR/A361). Studies with a lower number of questions (median 36; IQR 22 to 46) were more frequently completed than studies with a higher number of questions (median 51; IQR 39 to 67; p = 0.044) (Table 6).
Table 4.
Factors associated with the ability to contact a patient: multivariable analysis
| Patient characteristics | OR (95% CI) | p value |
| Age (years) | 1.01 (0.997 to 1.03) | 0.105 |
| Female sex | 1.46 (1.03 to 2.06) | 0.034 |
| Marital statusa | ||
| Single | reference | |
| Maried | 1.30 (0.88 to 1.94) | 0.188 |
| Widowed/Divorced | 1.23 (0.65 to 2.32) | 0.521 |
| Occupationb | ||
| Employed | reference | |
| Unemployed | 1.47 (0.93 to 2.34) | 0.102 |
| Student | 0.998 (0.26 to 3.80) | 0.998 |
| Retired | 1.42 (0.77 to 2.62) | 0.255 |
| Unable to work | 1.59 (0.62 to 4.11) | 0.337 |
| Median income based on ZIP codec | 1.000007 (1.000001 to 1.000013) | 0.019 |
| Distance to hospital based on ZIP coded | 1.0003 (0.9993 to 1.001) | 0.591 |
| Time from surgery to follow-up | 0.93 (0.90 to 0.97) | 0.001 |
Number of values in variables with missing data:
n = 766;
n = 726;
n = 786;
n = 793.
Table 5.
Factors associated with participation in a follow-up questionnaire: multivariable analysis
| Patient characteristics | OR (95% CI) | p value |
| Racea | ||
| Asian | 1.38 (0.27 to 7.01) | 0.697 |
| Black | 1.03 (0.49 to 2.19) | 0.934 |
| Hispanic | 0.63 (0.30 to 1.33) | 0.227 |
| Other | 0.25 (0.08 to 0.94) | 0.040 |
| Distance to hospital based on ZIP codeb | 1.001 (0.9995 to 1.003) | 0.167 |
| Study characteristics | ||
| More than one researcher calling | 0.34 (0.23 to 0.52) | < 0.001 |
Number of values in variables with missing data:
n = 525;
n = 537.
Table 6.
Factors associated with completion of a follow-up questionnaire: bivariate analysis
| Patient characteristics, % (n) | All 100% (397) | Completion | p value | |
| Yes 98% (390) | No 1.8% (7) | |||
| Demographics | ||||
| Age (years), median (IQR) | 54 (41 to 64) | 57 (46 to 66) | 46 (35 to 59) | 0.095 |
| Sex, % (n) | 0.247 | |||
| Female | 57% (225 of 397) | 57% (223 of 390) | 29% (2 of 7) | |
| Race, % (n)a | 0.228 | |||
| White | 83% (321 of 385) | 84% (316 of 378) | 71% (5 of 7) | |
| Asian | 2% (8 of 385) | 2% (8 of 378) | 0% (0 of 7) | |
| Black | 8% (29 of 385) | 8% (29 of 378) | 0% (0 of 7) | |
| Hispanic | 6% (23 of 385) | 6% (21 of 378) | 29% (2 of 7) | |
| Other | 1% (4 of 385) | 1% (4 of 378) | 0% (0 of 7) | |
| Marital status, % (n)b | 0.576 | |||
| Single | 32% (123 of 381) | 32% (120 of 374) | 43% (3 of 7) | |
| Married | 57% (217 of 381) | 57% (214 of 374) | 43% (3 of 7) | |
| Divorced | 8% (30 of 381) | 8% (29 of 374) | 14% (1 of 7) | |
| Widowed | 3% (11 of 381) | 3% (11 of 374) | 0% (0 of 7) | |
| Tobacco dependency reported in chart, % (n) | 10% (40 of 397) | 10% (40 of 390) | 0% (0 of 7) | 1.00 |
| Diabetes, % (n) | 10% (41 of 397) | 11% (41 of 390) | 0% (0 of 7) | 1.00 |
| Occupation, % (n)c | 0.292 | |||
| Employed | 51% (194 of 384) | 51% (191 of 378) | 50% (3 of 6) | |
| Unemployed | 20% (75 of 384) | 19% (72 of 378) | 50% (3 of 6) | |
| Student | 1% (4) | 1% (4 of 378) | 0% (0 of 6) | |
| Retired | 25% (95 of 384) | 25% (95 of 378) | 0% (0 of 6) | |
| Unable to work | 4% (16 of 384) | 4% (16 of 378) | 0% (0 of 6) | |
| Median income (in USD) based on ZIP code, median (IQR)d | 81,040 (62,933 to 101,229) | 84,847 (65,476 to 102,342) | 71,576 (53,778 to 75,404) | 0.063 |
| Distance to hospital (in miles) based on ZIP code, median (IQR)e | 17 (6.6 to 38) | 15 (6.1 to 37) | 33 (13 to 58) | 0.201 |
| Time from surgery to follow-up (years), median (IQR) | 6.6 (3.7 to 11) | 6.0 (3.7 to 11) | 2.9 (1.5 to 9.2) | 0.176 |
| Study characteristics | ||||
| Number of researchers calling, % (n) | 0.715 | |||
| > 1 researcher | 38% (150 of 397) | 38% (148 of 390) | 29% (2 of 7) | |
| Number of questions, median (IQR) | 39 (22 to 46) | 36 (22 to 46) | 51 (39 to 67) | 0.044 |
| Modality of completion (phone or email) | 0.447 | |||
| Phone | 60% (237 of 397) | 60% (234 of 390) | 43% (3 of 7) | |
Number of values in variables with missing data:
n = 385;
n = 381;
n = 384;
n = 392;
n = 395.
Other Findings
The contact rate was 68% (541 of 798 patients) and if a patient was reached, the initial participation rate was 73% (397 of 541 patients) (Table 1). Ninety-eight percent of the participating patients (390 of 397) completed all questionnaires (Table 1). Questionnaires were completed over the telephone by 60% of the patients (237 of 397) and 40% of the patients (160 of 397) completed the questionnaires by email. The questionnaires were completed on Monday (15% of the patients; 42 of 286 patients), Tuesday (23%; 66 of 286), Wednesday (20%; 58 of 286), Thursday (21%; 61 of 286), or Sunday (21%; 59 of 286), and no questionnaires were completed on Fridays or Saturdays. The percentage of questionnaires completed by telephone or email was comparable between each day. Many questionnaires were completed in the afternoon (49%), followed by the evening (26%), morning (22%), and night (2%). Questionnaires completed at night were completed by email (Fig. 1).
Fig. 1.
This graph shows the percentage of completed questionnaires by modality per day of the week and time of day.
Discussion
Long-term follow-up studies are an important tool to evaluate treatment outcomes in upper extremity and orthopaedic surgery [20]. Long-term, in-person, follow-up remains challenging because it is resource intensive and requires patients to travel to the hospital, which involves time and cost for travel and/or parking. For that reason, follow-up research has become more reliant upon questionnaires that can be administered online or over the phone. To date, there is no consensus on the factors associated with response to long-term research follow-up questionnaires studies using these different contact modalities in orthopaedic upper extremity patients [2, 6, 13, 21, 22, 24, 29, 31]. In this study, we aimed to describe the response rate of hand surgery follow-up questionnaires for patients with upper extremity issues. In addition, we assessed what factors are associated with higher response, contact, initial participation, and completion rates. Women, patient enrollment by a single researcher, and a shorter follow-up time were independently associated with a higher response rate.
The results of this study need to be interpreted in the context of its limitations. First, as with any retrospective study, the validity depends on completeness of the medical record and coding accuracy. Missing data were random and nondifferential. To help minimize nondifferential classification bias, we used pairwise deletion in our analysis. In addition, we manually reviewed the medical records to minimize the amount of missing data based on coding errors. Second, some patients may have been deceased at time of the study but were not recorded as deceased in the hospital registry. Deceased patients might have incorrectly been considered eligible, which would have lowered the contact and response rates. However, age did not negatively impact the response rate, suggesting this was not a major issue. Third, our study included patients with different upper extremity conditions treated at one of five affiliated academic urban hospitals in the northeastern United States and may not be generalizable to other geographic locations. A prior report suggested that patients treated at academic hospitals have increased satisfaction, possibly due to the specialized care these patients received and the prestige of the hospital [4]. This may have had a positive impact on our response rate. Nonetheless, the included patients had common upper extremity conditions that could be treated in academic and nonacademic settings, making the findings of this study generalizable. Lastly, although we did not consider the variable of education level, based on prior studies at our institution, we assume the education level of our patients is comparable to the national average [12, 30, 35].
This study found that women had higher response and contact rates than men. Prior studies also reported that enrollment of women was associated with a higher response rate [2, 6, 21]. We do not have sufficient information to explain these differences. We did not find any difference in employment between men and women in our cohort or an association between employment and contact rate. Some statistics have documented that in the United States, 16% of total male employment works part time compared with 28% of the total female employment [33]. Although we could not make a distinction between part-time and full-time employment in our study, it is possible that some of the follow-up differences could be related to this observation. However, we also observed that the median household income of the women that were reached was higher than in the women who could not be reached (respectively USD 84,847 versus USD 75,565; p = 0.01), which may be contradictory to the part-time employment explanation. Another explanation could be that men are less likely to discuss health issues compared with women, negatively impacting the study response rate [5, 11, 16, 17, 23]. Our study and others suggest that men are more difficult to contact than women [17, 27, 34].
This study showed that a shorter interval from the last visit to follow-up was independently associated with higher response and contact rates. Patient contact information may have changed over time and patients may have moved to other locations. A prior study showed that 51% of patients changed at least one form of contact information within 5 years and 66% within 10 years [20]. In addition, they found that younger patients and women changed their contact information more frequently than men and older patients [20]. Over time, patients are more likely to move or change contact information such as postal address, e-mail address, and telephone number. This highlights the constant challenge in reporting long-term outcomes after orthopaedic surgery, suggesting that a larger loss to follow-up may need to be accepted when the time to follow-up increases. Gathering and updating as much contact information as possible at the time of the patient’s clinical visit may help increase the contact rate of these patients when they are enrolled in future research studies.
With regard to the performance of a study, this study showed that involvement of more than one researcher in the enrollment of patients was independently associated with a lower response and initial participation rate than when only one researcher enrolled patients. In most of our studies, multiple researchers are involved in patient enrollment, including researchers with less experience. We believe that inexperienced researchers may be less consistent and efficient in their patient enrollment because these researchers are at the beginning of their learning curve. This may affect the interaction between the patient and interviewer and explain why multiple researcher enrollment is associated with lower response rates. Also, involvement of multiple researchers may reduce the sense of ownership [3]. In studies where only one researcher was involved in enrollment, all enrollments were conducted by the first author, who in general has more research experience and is responsible for most of the study. In all cases, patients were enrolled and informed by either a medical student or a PhD student. When only a single researcher was involved with patient enrollment, the researcher was a PhD student or a fourth-year medical student with experience in research and interviewing.
Our study showed that a higher median income based on ZIP code seemed to be associated with a higher contact rate. Although we found a small statistical association (OR 1.000007; 95% CI 1.000001 to 1.000013; p = 0.027), the median income of responders (USD 84,847) and nonresponders (USD 78,343) differed by USD 6504; but it is important to note that this difference is considered within the same socioeconomic class. The median income of responders and nonresponders would be considered middle-class income [14, 19]. Another finding was that race classified as “other” appeared to be independently associated with a lower initial participation rate; however, this may be spurious because it was based on only four patients with a race other than white, black, Asian, or Hispanic.
We found a relatively even distribution of responses over most days of the week. No questionnaires were completed on Fridays and Saturdays, in contrast to the other days of the week. Although it is unknown whether patients were called with the same frequency on Fridays and Saturdays, we do know patients were called by our researchers on all days of the week including Friday and Saturday. These results imply that patients are less willing to complete a questionnaire on Friday and Saturday. We think that patients do not wish to participate in research questionnaire at the end of a working week when they seek rest. In addition, no patient chose to complete the questionnaire by email on these days, supporting our idea. Contacting patients in the evening and on Sundays may help increase the response rate of a follow-up study. Similar trends were seen in a review of more than 100,000 online survey participants [36, 37]. People who participated, most frequently participated at the beginning of the week and less frequently on Fridays, and most people participated in the morning and afternoon, with a substantial group of people responding outside the office hours. A report looking at the cooperative contact rate of phone interviews by landlines showed that calling after 5 pm on a weekday or Sunday are efficient moments to call [26]. The day of the week that a questionnaire is sent by mail does not influence the response rate [1, 8, 25].
Although most questionnaires were completed by telephone (59%) a substantial amount of the people (41%) chose to complete the questionnaire by email. Offering both modalities gives patients the freedom to participate at their convenience and use their preferred method. Email might be ideal for people with little time during the day, while telephone calls might be ideal for those who are not comfortable with Internet surveys or do not have access to the Internet [35]. Prior research showed that the use of multiple communication techniques such as postal mail, telephone, and email increases the responses to research questionnaires and is worth considering in future studies [10, 15]. Based on our findings and experience performing these studies, we recommend that researchers administer questionnaires over the telephone if possible because patients may feel more committed to finish the questionnaire when they are talking to a researcher on the phone. In addition, a researcher can explain how many more questions are coming and encourage the patient during completion of the questionnaire by phone. Most research questionnaires consist of multiple measurement tools (sub-questionnaires) and people who participate by email may get discouraged finishing questionnaires containing a larger number of questions or delay participation even after reminder emails.
Finally, studies using a questionnaire with a large number of items had lower completion rates than studies using shorter questionnaires. Prior research showed that shorter questionnaires sent by postal mail led to a higher response rate than lengthy questionnaires [9, 18, 28]. The number of questions used in the follow-up studies included in this report ranged from 15 to 71 questions and all questionnaires took less than 12 minutes to complete, which led to relatively high completion rates. In our center, we always aim to keep our questionnaires shorter than 10 minutes to help the patient stay focused and prevent alteration of responses and skipping of questions. To improve the response and completion rates of a study, we therefore recommend keeping the questionnaire as short as possible and calculating how much time it takes to complete a questionnaire while preparing the study.
When considering study design and interpreting its results, men and participants with a longer follow-up time have lower response rates than women and those with shorter follow-up. From a practical standpoint, patient enrollment by a single researcher may increase the response rate and should be the target when allocating resources. In addition, improving the length of the questionnaires and careful collection of contact information at the time of the patient’s clinical visit, and contacting patients outside traditional work hours in the evening and on Sundays is worth considering.
Acknowledgments
We thank Daniel Baker BSc, Rachel Gottlieb BSc, Daphne van Hooven MD, Femke Nawijn BSc, and Svenna Verhiel MD, for sharing the data of their studies that made this review possible.
Footnotes
Each author certifies that neither he or she, nor any member of his or her immediate family, has funding or commercial associations (consultancies, stock ownership, equity interest, patent/licensing arrangements, etc) that might pose a conflict of interest in connection with the submitted article.
All ICMJE Conflict of Interest Forms for authors and Clinical Orthopaedics and Related Research® editors and board members are on file with the publication and can be viewed on request.
Each author certifies that his or her institution approved the human protocol for this investigation and that all investigations were conducted in conformity with ethical principles of research.
References
- 1.Akl EA, Gaddam S, Mustafa R, Wilson MC, Symons A, Grifasi A, McGuigan D, Schünemann HJ. The effects of tracking responses and the day of mailing on physician survey response rate: Three randomized trials. PLoS One. 2011;6:e16942. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Bot AGJ, Anderson JA, Neuhaus V, Ring D. Factors associated with survey response in hand surgery research. Clin Orthop Relat Res. 2013;471:3237-3242. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Campbell Pickford H, Joy G, Roll K. Psychological ownership: Effects and applications. 2016. Available at: http://eureka.sbs.ox.ac.uk/6266/1/2016-32.pdf. Accessed February 22, 2020.
- 4.Chen AS, Revere L, Ratanatawan A, Beck CL, Allo JA. A comparative analysis of academic and nonacademic hospitals on outcome measures and patient satisfaction. Am J Med Qual. 2019;34:367-375. [DOI] [PubMed] [Google Scholar]
- 5.Chinn DJ, White M, Howel D, Harland JOE, Drinkwater CK. Factors associated with non-participation in a physical activity promotion trial. Public Health. 2006;120:309-319. [DOI] [PubMed] [Google Scholar]
- 6.Dallosso HM, Matthews RJ, McGrother CW, Clarke M, Perry SI, Shaw C, Jagger C. An investigation into nonresponse bias in a postal survey on urinary symptoms. BJU Int. 2003;91:631-636. [DOI] [PubMed] [Google Scholar]
- 7.Datasheer LLC. Free distance calc. Available at: https://www.zip-codes.com/search.asp?selectTab=5. Accessed September 1, 2019.
- 8.de Fine Olivarius N, Andreasen AH. Day-of-the-week effect on doctors’ response to a postal questionnaire. Scand J Prim Health Care. 1995;13:65-67. [DOI] [PubMed] [Google Scholar]
- 9.Edwards PJ, Roberts I, Clarke MJ, Diguiseppi C, Wentz R, Kwan I, Cooper R, Felix LM, Pratap S. Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2009;3:MR000008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Feigelson HS, McMullen CK, Madrid S, Sterrett AT, Powers JD, Blum-Barnett E, Pawloski PA, Ziegenfuss JY, Quinn VP, Arterburn DE, Corley DA. Optimizing patient-reported outcome and risk factor reporting from cancer survivors: A randomized trial of four different survey methods among colorectal cancer survivors. J Cancer Surviv. 2017;11:393-400. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Glass DC, Kelsall HL, Slegers C, Forbes AB, Loff B, Zion D, Fritschi L. A telephone survey of factors affecting willingness to participate in health research surveys. BMC Public Health. 2015;15:1017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Golkari S, Teunis T, Ring D, Vranceanu A-M. Changes in depression, health anxiety, and pain catastrophizing between enrollment and 1 month after a radius fracture. Psychosomatics. 2015;56:652-657. [DOI] [PubMed] [Google Scholar]
- 13.Graaf Rd, Bijl RV, Smit F, Ravelli A, Vollebergh WAM. Psychiatric and sociodemographic predictors of attrition in a longitudinal study the netherlands mental health survey and incidence study (nemesis). Am J Epidemiol. 2000;152:1039-1047. [DOI] [PubMed] [Google Scholar]
- 14.Guzman GG. Household income: 2017. 2018. Available at: https://www.census.gov/content/dam/Census/library/publications/2018/acs/acsbr17-01.pdf. Accessed February 21, 2020.
- 15.Harewood G, Yacavone RF, Locke GR, 3rd, Wiersema MJ. Prospective comparison of endoscopy patient satisfaction surveys: E-mail versus standard mail versus telephone. Am J Gastroenterol. 2001;96:3312-3317. [DOI] [PubMed] [Google Scholar]
- 16.Harrington KM, Nguyen X-MT, Song RJ, Hannagan K, Quaden R, Gagnon DR, Cho K, Deen JE, Muralidhar S, O’Leary TJ, Gaziano JM, Whitbourne SB, Gaziano JM, Ramoni R, Breeling J, Chang K-M, Huang G, Muralidhar S, O’Donnell CJ, Tsao PS, Muralidhar S, Moser J, Whitbourne SB, Brewer JV, Concato J, Warren S, P D, Argyres DP, Tsao PS, Gaziano JM, Stephens B, Brophy MT, Humphries DE, Do N, Shayan S, Nguyen X-MT, O’Donnell CJ, Pyarajan S, Tsao PS, Cho K, Pyarajan S, Hauser E, Sun Y, Zhao H, Wilson P, McArdle R, Dellitalia L, Harley J, Whittle J, Beckham J, Wells J, Gutierrez S, Gibson G, Kaminsky L, Villareal G, Kinlay S, Xu J, Hamner M, Haddock KS, Bhushan S, Iruvanti P, Godschalk M, Ballas Z, Buford M, Mastorides S, Klein J, Ratcliffe N, Florez H, Swann A, Murdoch M, Sriram P, Yeh SS, Washburn R, Jhala D, Aguayo S, Cohen D, Sharma S, Callaghan J, Oursler KA, Whooley M, Ahuja S, Gutierrez A, Schifman R, Greco J, Rauchman M, Servatius R, Oehlert M, Wallbom A, Fernando R, Morgan T, Stapley T, Sherman S, Anderson G, Tsao P, Sonel E, Boyko E, Meyer L, Gupta S, Fayad J, Hung A, Lichy J, Hurley R, Robey B, Striker R. Gender differences in demographic and health characteristics of the million veteran program cohort. Womens Health Issues. 2019;29:S56-S66. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Jay GM, Liang J, Liu X, Sugisawa H. Patterns of nonresponse in a national survey of elderly japanese. J Gerontol. 1993;48:S143-152. [PubMed] [Google Scholar]
- 18.Jepson C, Asch DA, Hershey JC, Ubel PA. In a mailed physician survey, questionnaire length had a threshold effect on response rate. J Clin Epidemiol. 2005;58:103-105. [DOI] [PubMed] [Google Scholar]
- 19.Kochhar R. The American middle class is stable in size, but losing ground financially to upper-income families. Pew Research Center. Available at: https://www.pewresearch.org/fact-tank/2018/09/06/the-american-middle-class-is-stable-in-size-but-losing-ground-financially-to-upper-income-families/. Accessed February 21, 2020.
- 20.London DA, Stepan JG, Goldfarb CA, Boyer MI, Calfee RP. The (in)stability of 21st century orthopedic patient contact information and its implications on clinical research: A cross-sectional study. Clin Trials. 2017;14:187-191. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Neuner B, Fleming M, Born R, Weiss-Gerlach E, Neumann T, Rettig J, Lau A, Schoenfeld H, Kallischnigg G, Spies C. Predictors of loss to follow-up in young patients with minor trauma after screening and written intervention for alcohol in an urban emergency department. J Stud Alcohol Drugs. 2007;68:133-140. [DOI] [PubMed] [Google Scholar]
- 22.Nota SPFT, Strooker JA, Ring D. Differences in response rates between mail, e-mail, and telephone follow-up in hand surgery research. Hand (N Y). 2014;9:504-510. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Oliffe J, Mróz L. Men interviewing men about health and illness: Ten lessons learned. J Mens Health Gend. 2005;2:257-260. [Google Scholar]
- 24.Ootes D, Buijze GA, Ring D. Predictors of missed appointments in prospective hand surgery research. Hand (N Y). 2012;7:177-180. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Pressley MM, Tullar WL. A factor interactive investigation of mail survey response rates from a commercial population. J Mark Res. 1977;14:108. [Google Scholar]
- 26.Reimer B, Roth V, Montgomery R. Optimizing call patterns for landline and cell phone surveys. Proc Am Stat Assoc. 2012;2012:4648-4660. [PMC free article] [PubMed] [Google Scholar]
- 27.Rönmark EP, Ekerljung L, Lötvall J, Torén K, Rönmark E, Lundbäck B. Large scale questionnaire survey on respiratory health in sweden: Effects of late- and non-response. Respir Med. 2009;103:1807-1815. [DOI] [PubMed] [Google Scholar]
- 28.Sahlqvist S, Song Y, Bull F, Adams E, Preston J, Ogilvie D. Effect of questionnaire length, personalisation and reminder type on response rate to a complex postal survey: Randomised controlled trial. BMC Med Res Methodol. 2011;11:62-62. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Solberg TK, Sørlie A, Sjaavik K, Nygaard ØP, Ingebrigtsen T. Would loss to follow-up bias the outcome evaluation of patients operated for degenerative disorders of the lumbar spine? Acta Orthop. 2011;82:56-63. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Teunis T, Thornton ER, Guitton TG, Vranceanu A-M, Ring D. Coaching of patients with an isolated minimally displaced fracture of the radial head immediately increases range of motion. J Hand Ther. 2016;29:314-319. [DOI] [PubMed] [Google Scholar]
- 31.Thomas MC, Walker M, Lennon LT, Thomson AG, Lampe FC, Shaper AG, Whincup PH. Non-attendance at re-examination 20 years after screening in the british regional heart study. J Public Health Med. 2002;24:285-291. [DOI] [PubMed] [Google Scholar]
- 32.U.S. Census Bureau. 2018. American community survery 1-year estimates subject tables, income in the past 12 months (in 2018 inflation-adjusted dollars). 2017. Available at: https://data.census.gov/cedsci/table?q=income&hidePreview=false&tid=ACSST1Y2018.S1901&t=Income%20%28Households,%20Families,%20Individuals%29. Accessed April 22, 2020.
- 33.van Osch Y, Schaveling J. The effects of part-time employment and gender on organizational career growth. J Career Dev. 2017;20:1-16. [Google Scholar]
- 34.Wagner M, Kuppler M, Rietz C, Kaspar R. Non-response in surveys of very old people. Eur J Ageing. 2019;16:249-258. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Westenberg RF, Zale EL, Heinhuis TJ, Özkan S, Nazzal A, Lee S-G, Chen NC, Vranceanu A-M. Does a brief mindfulness exercise improve outcomes in upper extremity patients? A randomized controlled trial. Clin Orthop and Relat Res. 2018;476:790-798. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Wronski L. Who responds to online surveys at each hour of the day? Available at: https://www.surveymonkey.com/curiosity/responds-online-surveys-hour-day/. Accessed February 21, 2020.
- 37.Zengh J. What day of the week should you send your survey? Available at: https://www.surveymonkey.com/curiosity/day-of-the-week/. Accessed February 21, 2020.