Enhancing Response Rates in Physician Surveys: The Limited Utility of Electronic Options

Abstract

Objective

To evaluate the utility of offering physicians electronic options as alternatives to completing mail questionnaires.

Data Source

A survey of colorectal cancer screening practices of Alabama primary care physicians, conducted May–June 2010.

Study Design

In the follow-up to a mail questionnaire, physicians were offered options of completing surveys by telephone, fax, email, or online.

Data Collection Method

Detailed records were kept on the timing and mode of completion of surveys.

Principal Findings

Eighty-eight percent of surveys were returned by mail, 10 percent were returned by fax, and only 2 percent were completed online; none were completed by telephone or email.

Conclusions

Offering fax options increases response rates, but providing other electronic options does not.

Gathering quality data on physician practices and patterns of care is crucial to enhancing efficiency in health care delivery while continuing to improve patient outcomes. One of the more common approaches to gathering such data has been survey research in the form of mail questionnaires. This approach has significant advantages over other modes of data collection: reasonably accurate contact information is readily available; survey delivery and return are relatively inexpensive; and no technologically advanced skills or capabilities are required for the researchers or the respondents.

The only significant disadvantage to mail questionnaires involves the unwillingness of some physicians to participate, resulting in low response rates, which in turn raise issues of nonresponse bias and problems of generalizability. This article describes an attempt to boost response rates in a survey of primary care physicians in Alabama by offering a range of electronic options for survey completion.

MAXIMIZING PARTICIPATION IN PHYSICIAN SURVEYS

It is beyond the scope of this paper to provide an extensive review of literature on the various approaches to increasing response rates in mail surveys of physicians. It is also unnecessary in that it has been done admirably elsewhere (Kellerman and Herold 2001; Field et al. 2002; VanGeest, Johnson, and Welch 2007; Dillman, Smyth, and Christian 2009; Sprague, Quigley, and Bhandari 2009). This literature is relatively definitive when it comes to the importance of maximizing the professionalism and personalization of survey materials, providing incentives for respondents to participate, and repeatedly encouraging participation with follow-ups and reminders. The literature also demonstrates that the sole use of electronic surveys, as alternatives to mail questionnaires, typically results in unacceptably lower response rates (Braithwaite et al. 2003; Leece et al. 2004; Sprague, Quigley, and Bhandari 2009; Kim et al. 2010). Even so, it has been suggested that offering electronic options as a supplemental means of survey completion might increase participation (VanGeest, Johnson, and Welch 2007; Dillman, Smyth, and Christian 2009). The validity of this suggestion was tested in a survey of Alabama primary care physicians regarding colorectal cancer screening practices.

When it comes to the typical mail survey, there are five electronic alternatives for completion that might be offered: (1) the survey embedded in the text of an email; (2) the survey delivered and returned as an attachment to an email; (3) the survey loaded on a website, with respondents invited to participate by mail, telephone, fax, or email; (4) the survey delivered and/or returned by fax; or (5) the opportunity to complete the survey by telephone. The most effective means of determining which option or combination of options is most effective would be an experimental design where randomly selected subgroups of respondents would receive different options, so that subgroup response rates would reveal the optimal approach. Unfortunately, that would significantly complicate the research process, thereby increasing costs. It would also insure that some subgroups would receive suboptimal options, thereby decreasing overall response rates. Another approach would be to offer all the options to the full target population and then track which modes are most commonly used. The latter approach was used in the survey of Alabama physicians.

SURVEY OF ALABAMA PHYSICIANS

The primary goal of the survey was to gather information to serve as a guide to developing an effective program to educate physicians regarding current best practices in the area of colorectal cancer prevention and detection. The target population was primary care physicians currently practicing in Alabama in the areas of Family Medicine, Internal Medicine, and Obstetrics & Gynecology. A database of contact information for 2,642 physicians was obtained from an outside vendor. Removal of obvious duplicate entries resulted in a mailing list of 2,624 physicians. During the response phase of the project, a number of surveys were returned as “undeliverable”; in other cases, we received notifications that physicians had died, retired, or moved out of state. Removal of these cases resulted in an adjusted population of 2,378 Alabama primary care physicians. From this adjusted population, we received 609 completed surveys, constituting a participation rate of 26 percent.

The primary survey approach was a traditional mail questionnaire. The instrument consisted of four pages of 46 items in 18 numbered questions, plus two open-ended opportunities for suggestions and comments. Survey packets included stamped, postage-paid return envelopes, a cover letter from the clinical director of the Mitchell Cancer Institute who had previously served as director of the Division of Cancer Prevention and Control at the Centers for Disease Control in Atlanta, an information request card, and a promotional incentive (a glass magnifier). All survey materials were branded with a robotic stick figure looking through a magnifying glass accompanied by some version of the tag line, “Taking a Closer Look at Colorectal Cancer Screening.” Before mailing the surveys, a postcard was sent to all physicians in the target population to encourage participation.

In addition to the mailed questionnaire, electronic, fax, and telephone versions of the instrument were prepared. Subsequent to the mailed questionnaire, two rounds of follow-up postcard reminders were sent to the target group. To further encourage participation, follow-up telephone calls were made to physicians who did not return completed surveys, offering them these options to facilitate completion: (1) to provide another copy of the survey by mail, email, or fax; (2) to accept return of the completed survey by mail, email, or fax; (3) to complete a web-based version of the survey on the Internet; or (4) to conduct the survey over the telephone.

Approximately 1 week after the initial mailing, follow-up postcards were sent to the full mailing list to remind physicians of the importance of the survey and to encourage participation. The postcard included the survey project team's telephone, fax, and email contact information to facilitate respondents' requests for a replacement copy of the instrument or for inquiries regarding any other aspect of the survey project. Ten days later, another postcard was mailed to those physicians who had failed to return the survey. This second reminder also included telephone and email contact information, as well as a link to a web-based survey site where the questionnaire could be completed online. In addition, beginning 2 weeks after the initial mailing, all nonresponding physicians were contacted by telephone and offered the full range of mail, telephone, fax, email, and web-based options for completing the survey. Finally, 1 month after the initial mailing, a survey team attended the annual meeting of the Alabama Academy of Family Physicians, providing information about the survey, generally encouraging participation, and offering all options for completing the survey, including doing so online at the convention.

Because the timing of these follow-up efforts overlapped extensively, it is very difficult to parse out the comparative effectiveness of the individual approaches. For example, if a physician is called and a message is left with office staff regarding the survey, the subsequent return of the survey might be due to the reminder, or it might have been returned regardless of the reminder. Even so, one can get some sense of the practical utility of the follow-up based on the ultimate timing and mode of completion.

In demonstrating this utility, the first step is to review the timing of survey completions; Table 1 provides the relevant breakdown. As can be seen in the table, the bulk of completed surveys were returned after the follow-up phase of the project began, demonstrating the utility of the reminder effort. However, almost three-fourths were returned before the beginning of the telephone follow-up. Even so, without the telephone follow-up, our participation rate would likely have been below 20 percent.

Table 1.

Timing of Survey Completions

	Frequency	Percent	Cumulative Percent
Before initial postcard reminder	169	27.8%	27.8%
Between initial reminder and second postcard reminder	270	44.3	72.1
After second reminder, during telephone follow-up	163	26.8	98.9
Missing/undetermined	7	1.1	100%
Total	609	100%

A second approach to demonstrating the utility of the multimodal follow-up is to review the reaction of respondents to the offer of various options, as well as the ultimate mode of completion. In the telephone follow-up phase, almost 2,000 physicians' offices were called. The outcomes of these calls are summarized in Table 2. This table reveals the difficulty of contacting physicians' offices by telephone, as contact could not be made in 42 percent of the cases (sum of the last five categories). In many of these cases, messages left on voice mails were not returned. It should also be noted that virtually all contacts were with office staff, rather than with the physicians themselves.

Table 2.

Initial Outcomes of Telephone Calls

	Frequency	Percent of Total Calls	Percent of Actual Contacts
Request for faxed survey	729	36.7%	63.1%
Request for web-based survey link	40	2.0	3.5
Request for remailing of survey	23	1.2	2.0
Request for emailed survey	0	0.0	0.0
Request for telephone completion	0	0.0	0.0
Refusal	202	10.2	17.5
Contact made, outcome pending	130	6.5	11.3
Physician not available	30	1.5	2.6
Duplicate identified by call	16	0.8	100% (n = 1154)
Incorrect telephone numbers	160	8.1
Busy	114	5.7
No answer	52	2.6
Answering machine	491	24.7
Total	1987	100%

These results demonstrate that cutting-edge technological options are not the preference for the overwhelming majority of Alabama primary care physicians at this time. Only 3.5 percent requested the web-based survey link and none asked for an email version of the questionnaire. On the other hand, almost two-thirds of the offices contacted requested a faxed copy of the instrument. This should not be particularly surprising considering that the use of faxes for records transfers and referrals is such common practice in doctors' offices.

When interpreting these results, one must also be mindful that requesting a faxed copy of the questionnaire does not necessarily reflect a good faith commitment to complete the survey. Table 3 provides a breakdown of the ultimate mode of survey completion. When comparing Tables 2 and 3 results on the fax option, we find that <10 percent of those requesting a fax (n = 769) actually returned a completed questionnaire by fax (n = 62). For many of these physicians' offices, the request for a fax may have been a soft refusal. Rather than refuse outright, office staff would simply free themselves from the telephone interaction by allowing a fax to be sent.

Table 3.

Mode of Survey Completion

	Frequency	Percent of Total	Percent of Electronic Options
Fax completions	62	10.2%	78.5%
Web-based completions	11	1.8	13.9
Remailed and returned	6	1.0	7.6
Emailed completions	0	0.0	0.0
Telephone completions	0	0.0	0.0
Survey returned by mail	530	87.0	100% (n = 79)
Subtotal	609	100%

Another important finding in Table 3 is the apparent lack of interest in electronic versions of the survey. While there were six requests to remail hard copies of the questionnaire, there were no requests for email versions of the survey instrument. Also, <2 percent (n = 11) of the 609 surveys were completed online. The inclusion of those 11 surveys increased the response rate by less than one-half of 1 percent. Yet offering the web-based version required hours of additional time in investigating options, formatting, testing the web questionnaire, and retrieving the online data. It also complicated the data aggregation process in creating the need to adjust and merge data files. In addition, web-based survey providers are for-profit operations and their charges are not insignificant. Based upon these considerations, if one focuses solely on incremental increases in response rates, the costs of offering electronic versions of the survey cannot be justified.

CONCLUSION

Physician surveys will continue to be an important source of data for health care researchers. As an approach to surveying physicians, mail questionnaires tend to have advantages over other modes of data collection. They are relatively inexpensive, convenient for respondents, generally accepted, and widely utilized. Given the increasing reluctance of many physicians to participate in such surveys, however, maximizing response rates will continue to be a critical ingredient for ensuring valid and reliable results. There are a number of tried and true strategies for increasing participation, the more important of which include maximizing the professionalism of all aspects of the survey process, providing physicians with monetary incentives to participate, and implementing an effective reminder system. Another potential strategy suggested in the literature is to attempt to maximize convenience for respondents by offering a number of alternative electronic modes for returning a completed survey. Testing the effectiveness of this strategy in a survey of Alabama physicians, however, did not yield encouraging results. While offering a faxed option had a small positive impact on response rates, providing email and online options had no practical utility.

Supporting Information

Additional supporting information may be found in the online version of this article:

Appendix SA1: Author Matrix.

Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.