Survey Says? A Primer on Web-based Survey Design and Distribution

Abstract

The internet has changed the way in which we gather and interpret information. While books were once the exclusive bearers of data, knowledge is now only a keystroke away. The internet has also facilitated the synthesis of new knowledge. Specifically, it has become a tool through which medical research is conducted. A review of the literature reveals that in the past year, over one-hundred medical publications have been based on web-based survey data alone.

Due to emerging internet technologies, web-based surveys can now be launched with little computer knowledge. They may also be self-administered, eliminating personnel requirements. Ultimately, an investigator may build, implement, and analyze survey results with speed and efficiency, obviating the need for mass mailings and data processing. All of these qualities have rendered telephone and mail-based surveys virtually obsolete.

Despite these capabilities, web-based survey techniques are not without their limitations, namely recall and response biases. When used properly, however, web-based surveys can greatly simplify the research process. This article discusses the implications of web-based surveys and provides guidelines for their effective design and distribution.

INTRODUCTION

From Google^a to eMedicine,^b the internet has changed the way in which medical information is gathered and interpreted. While books were once the exclusive bearers of data, knowledge is now only a keystroke away. The internet has also facilitated the synthesis of new knowledge. Specifically, it has become a tool through which medical research is conducted. Web-based survey tools, as a prime example, facilitate efficient communication between investigators and participants. Accordingly, survey reach and sample size are augmented, and cost is markedly reduced.¹

Evidence also suggests that subjects on the web may be more likely to take a survey to completion; this indicates that participants find web-based surveys more appealing.² Increased appeal is likely due to the ease of submission: clicked responses and a “Submit” button are more facile than written responses and postal mail. In accordance, web-based survey response rates have been shown to be comparable to traditional survey methods.^3,4,5

As internet technology has emerged, web-based surveys can now be launched with little computer knowledge. The surveys may also be self-administered, eliminating personnel requirements. Ultimately, an investigator may build, implement, and analyze survey results with speed and efficiency, obviating the need for mass mailings and data processing. All of these qualities have rendered telephone and mail-based surveys increasingly outdated.^6,7

Web-based survey techniques are not without their limitations.⁸ The fundamentals of research design should not be lost in a rush to adopt this novel approach. Fastidious adherence to traditional survey principles has particular importance in online surveys, where verbal communication and guidance is noticeably absent.⁹ Despite their limitations, appropriate use of web-based surveys can greatly simplify the research process. This article discusses the implications of web-based surveys and provides guidelines for their effective design and distribution.

DESIGNING A WEB-BASED SURVEY

Web-Based Survey Providers

While there are several online applications dedicated to customized survey solutions, Zoomerang^c and SurveyMonkey^d are the leading providers. Their services are nearly identical, each providing a step-by-step approach to customized survey design. A limited account is available for free through either service. Premium plans range from $20 to $350, depending on account requirements. Expert assistance is also available (for an additional fee) in cases of complex survey development.

Creating a Survey

Like any computer interface, web-based surveys must be both easy to navigate and to understand.¹⁰ This principle of web design is referred to as usability. Online survey applications solve much of the usability problem by presenting survey questions and responses in a standardized format. Online survey providers allow each question to be customized by type. For example, responses may be displayed in formats such as multiple choice, yes/no, rating scale, etc. The caveat is that large variations in format may cause confusion in respondents. Graphics, sound, video, colors, and fonts may then be added to improve communication and create a uniquely branded appearance.

Based on established principles, the general language of a survey should mimic a verbal dialogue.¹¹ Schwarz contends that survey respondents conduct themselves as if involved in conversation with the investigator.¹² This “dialogue” should be clear, concise, honest, and not repetitive.¹³ The absence of verbal cues imparts further importance to survey design, with appearance and syntax carrying more weight.¹⁴

Deployment and Accessibility

There are two major ways in which to deploy an online survey: via e-mail or on the web. Both of these means may be used concurrently.

Web-based survey applications enable investigators to reach subjects via mass e-mail. A contact importer allows an investigator to upload subjects’ e-mail addresses to a private account. Subsequently, a survey invitation may then be sent automatically to all subjects simultaneously. Reminder emails may also be sent, which has been shown to increase response rates (figure 1).^1,25

Figure 1.

Total number of responses to the web-based BreastPE.com survey over time.²⁶ Arrow indicates timing of “reminder” email which was sent to non-respondents fourteen days after the initial survey was launched.

A unique web address, or URL (Uniform Resource Locator), is generated by the survey application. The URL can then be posted on a website or sent to subjects via e-mail. Subjects may simply click the link to be redirected. This URL is often long and difficult to verbally communicate (e.g. http://www.zoomerang.com/Survey/WEB229R27WJVCD) and respondents may have difficulty retyping the URL into the web browser. In order to resolve this problem, a separate web address may be purchased through a domain name registrar, such as Godaddy.com.^e This simplified domain name (e.g. http://www.PRSsurvey.info) can then forward to your survey. The new domain name is much easier to convey, both verbally and in print. This approach is particularly useful if a mail-based solicitation is desired. Notably, a preliminary mailing prior to web release has been shown to improve survey response rates.¹⁵

Before deployment, a small test group should be physically observed taking your survey. This “pilot testing” or “usability testing” can reveal stumbling blocks in design and formatting. Items that are ambiguous can be identified and altered. Changes may then be made without detriment to the survey's integrity. An improved, user-friendly version of the survey can then be sent to the true research population.

Participation Incentives

Researchers commonly use incentives to encourage survey participation and increase the volume of data generated. Singer and Couper contend that incentives are influential but not coercive.¹⁶ They found that the willingness of survey respondents to divulge personal information was not related to increasing incentives. The authors further argued that an Institutional Review Board (IRB) should not regulate incentive size, as this methodology does not pertain to the protection of subjects from harm.

Mail surveys commonly provide incentives through cash distribution prior to survey completion. Pre-payment is possible in web-based surveys using money distributed via PayPal.^f Pre-paid incentives, however, have not been shown to significantly increase participation rates when compared to incentives provided upon survey completion.¹⁷ Studies evaluating multiple means of web-based survey incentives have shown that entry into a “prize draw” will significantly increase willingness to participate when compared with a smaller, guaranteed cash incentive.¹⁷ Evidence also suggests that monetary compensation may introduce bias, with a disproportionate increase in respondents from lower socioeconomic groups.¹⁸

Even more important than financial incentives is conveying the importance of the research to the respondents. Similarly, investigator gratitude should be conveyed. These methods engender trust between the participant and researcher, while creating a sense of cooperation for survey participation.¹⁹ Even without a financial incentive, potential respondents may be more likely to participate if they perceive that doing so will have an impact on an issue they find important.

PRIVACY AND CONFIDENTIALITY

Data Security

Medical information is inherently sensitive. Similarly, clinician behaviors, practice patterns and outcomes data should be treated as privileged, confidential information. The movement to health care computing has required an enormous overhaul of hospital information technology (IT) systems in order to keep such information protected. Web-based surveys often require participants to disclose highly private information, and the principal investigator is responsible for upholding the participants’ right to privacy. Zoomerang,^g for example, uses both physical and technological means to protect survey data. In addition to an individual account password, Zoomerang’s central servers (upon which data is stored) require card key access and are under constant video surveillance. They are also protected from “hacking” by firewalls and SSL (Secure Sockets Layer) encryption. These means have become the industry standard, and are consistent the mechanisms presently in place within hospital computer mainframes.

Institutional Review Board (IRB) Approval

Research involving any human subject requires a full IRB application that demonstrates a favorable risk/benefit ratio to study subjects and/or society. As such, all surveys at the authors’ institution are reviewed by the IRB prior to launch. Consultation with an IRB prior to initiating survey research is mandatory. In general, the IRB will require that subject data is de-identified. With de-identification, response data cannot be linked to an individual survey participant.

Patient-reported data clearly falls under the realm of protected health information (PHI). Survey data that reflects the attitudes and/or practice patterns of physicians—although not considered PHI—still fall under the realm of the IRB. No distinction between patient and clinician is made during IRB evaluation; each survey respondent is treated as a subject.

DATA ANALYSIS

Response Rates

The existing literature gives conflicting reports on the differential response rates for mailed and web-based surveys.^3,4,5 While some authors have demonstrated higher initial survey response rates with mailed versus web-based surveys (58% vs. 45%, respectively),⁴ other authors have demonstrate the opposite (36% vs. 73%, respectively).⁵ Recruitment methods (e.g. an advertisement in a print journal) may also affect relative response rates to a web-based survey.²⁰ Regardless of recruitment methods, when regular reminders are sent to subjects, the difference in final survey response rates between mailed and web-based surveys are small.^3,4,5 Reminder e-mails to non-respondents has specifically been shown to augment web-based survey response rates.^3,4,5,21 The primary authors recently conducted a web-based survey of practice patterns for venous thromboembolism prophylaxis after autogenous tissue breast reconstruction. Through the use of a single reminder e-mail, sent two weeks after initial survey deployment, the total number of responses was increased by over 40% (figure 1).²¹

Perhaps more important than the actual response rate, however, is whether the respondents comprise a representative sample of the population of interest. If a sufficient number of subjects respond (from a statistical standpoint), and it can be demonstrated that the respondents are representative, then it is reasonable to make generalizations from the data back to the population. Conversely, even with a response rate of 50%, if the respondents are not representative of the population, generalizations are difficult. This underscores the importance of obtaining pertinent demographic information.

Data Collection

Responses to online surveys may be viewed in real-time. Data may be viewed either in aggregate or individually. Survey results may also be provided to participants on the “thank you” page at the end of a survey. While this would affect the validity of a formal study, the technique may be used in an informal survey as a way to normalize patient experiences with those sharing the same condition.¹⁴

To prevent “loading the ballot,” responses from a specific computer (IP address) may be limited to a single survey completion. Alternatively, a computer may be repeatedly used and refreshed, as a kiosk. Kiosks are ideal for recruiting patient participants in clinic or for surveying surgeons at national meetings. Survey results may be exported to almost any statistical software package (e.g. Microsoft Excel, SAS, Stata) for further analysis. Advanced analytical capabilities, such as filtering and cross-tabulation, are also possible online. In general, an unlimited amount of survey responses may be interpreted, although the investigator may incur a fee for each respondent after a certain number (usually $.05 for each response above 1,000).

An advantage of web-based survey methodology is that, depending on the sophistication of the software, the instrument may be designed using “logic,” such that items may be chosen based responses to previous items. This can certainly be done with paper questionnaires using directions such as “if you answered ‘no’ to this item please proceed to item number X.” However, if these directions become too complicated, drop out is likely. In a web-based instrument, this all occurs “behind the scenes:” the respondent sees only a smooth transition from one item to the next, and a shorter appearing instrument overall. This may lead to higher completion rates. Qualtrics™ allows for sophisticated logic, with items being chosen randomly from a pool.^h The particular pool itself is determined by responses to previous items. In this way, data can be gathered across a large number of topics without having any one respondent forced to answer items on all topics. The most sophisticated software systems use item response theory to choose successive items. Based on responses to several initial items with broad subject coverage, the respondent’s awareness of the subject is determined. Subsequent items are then selected, with continual adjustment throughout the remainder of the survey.

Bias

Innovation in survey modalities should always be mitigated by concerns about bias. Bias is a broad topic that has been extensively reviewed elsewhere.^21,22 A brief review of recall and response biases, however, is appropriate for this discussion of survey research.

Recall bias refers to the phenomenon that personal experience may color an individual’s response. If a surgeon has recently experienced a specific complication, for example, he or she may be more likely to report a higher incidence of said complication. Stringent inclusion criteria or stratified analysis, in which similar subjects are grouped prior to analysis, may minimize the influence of recall bias.

Systematic differences may be present between subjects who choose to respond to a survey compared to those who do not respond. This is known as response bias. As another example, younger surgeons, who have less experience but are more computer-savvy, may be more likely to respond to a web-based survey. Internet browser compatibility may also be an issue for web-based surveys. Most software allows for tracking of browser data without tracking respondents’ identities. Multiple instances of drop out, particularly at a consistent item, should be investigated for browser compatibility issues. This would best be discovered and corrected in the pilot phase. Response bias can be quantified through collection of demographic data from respondents and non-respondents. A data comparison can help to determine if a representative sample has been obtained. For web-based surveys, consideration of subjects without internet access is intrinsically critical to one’s ability to generalize the results of the study.²³

DISCUSSION

While the body of research pertaining to web-based surveys is still in its infancy, increasing attention has been given to this modality as a valid means of conducting formal research.^24,25 A review of the literature reveals that in the past year, over one-hundred medical publications have been based on internet survey data alone. While this article has presented a basic review, established guidelines for the use of the internet in medicine is a topic of central interest: the U.S. Department of Health and Human Services has published a treatise on this subject.ⁱ

Using the principles outlined in this article, the authors have completed survey research on a variety of topics, including practice patterns for venous thromboembolism prophylaxis in breast reconstruction²⁶ and post-bariatric body contouring,³⁰ cost-utility and decision making in rheumatoid hand surgery,^27,28 hand transplantation,²⁹ and the management of lower extremity fractures.³⁰ A sample survey for Plastic and Reconstructive Surgery readers has been created at www.PRSsurvey.info. Please explore an example of this exciting new research tool.

CONCLUSION

Web-based surveys can be created, deployed, and analyzed with increasing ease. Here, we have provided clinicians with a practical guide to the design and distribution of web-based surveys for medical research.