Effects of Sequential Prepaid Incentives on Response Rates, Data Quality, Sample Representativeness, and Costs in a Mail Survey of Physicians

Abstract

While collecting high quality data from physicians is critical, response rates for physician surveys are frequently low. A proven method for increasing response in mail surveys is to provide a small, prepaid monetary incentive in the initial mailing. More recently, researchers have begun experimenting with adding a second cash incentive in a follow-up contact in order to increase participation among more reluctant respondents. To assess the effects of sequential incentives on response rates, data quality, sample representativeness, and costs, physicians (N = 1,500) were randomly assigned to treatments that crossed the amount of a first ($5 or $10) and second ($0, $5, or $10) incentive to form the following groups: Group $5/$5; Group $5/$10; Group $10/$0; Group $10/$5; and Group $10/$10. Overall, second incentives were associated with higher response rates and lower costs per completed survey, and while they had no effect on item nonresponse, they increased sample representativeness.

Introduction

Surveys of physicians are crucial in informing health care research and understanding the health care practices, attitudes, and behaviors of clinicians (Klabunde et al., 2012). Successfully recruiting physicians to participate in surveys, however, continues to be challenging with studies indicating that response rates for physician surveys are declining (Cho et al., 2013; McLeod et al., 2013). Along with mode of data collection and number of follow-up attempts, incentives are among the most effective strategies used to increase participation in physician surveys (Cho et al., 2013; Dykema, Jones, et al., 2013a). Summarizing results from studies that test the impact of incentives in surveys of clinicians, researchers report that a small, prepaid monetary incentive is particularly effective in increasing response rates when compared to not providing an incentive (see reviews in Cho et al., 2013; Flanigan et al., 2008; VanGeest et al., 2007).

Findings for the effectiveness of small, prepaid incentives are consistent with the theory of social exchange, which posits that individuals are more likely to respond positively to a request when they trust the originator of the request and perceive the ratio of rewards to costs to be personally acceptable (Dillman et al., 2014). In contrast to not offering an incentive, these small, prepaid amounts are likely viewed as a “token of appreciation” that motivates the recipient to respond in a reciprocal manner. Drawing on the principles of social exchange, Dillman et al. (2014, p. 424) recently extended their recommendations about incentives to advise researchers to include a second cash incentive in a follow-up contact in order to increase the likelihood that “later communications will be read, and hopefully acted upon, thereby increasing overall response.” Few studies, however, have directly experimented with including additional prepaid incentives to convert nonresponders in a standard, three-contact mail survey. This study aims to address that gap by examining the effect of sequential incentives on response rates, survey data quality, sample representativeness, and costs in a survey of physicians.

Effects of Sequential Incentives on Response Rates

Several research syntheses and meta-analyses (e.g., Edwards et al., 2009; Mercer et al., 2015; Singer & Ye, 2013), including a systematic review (Flanigan et al., 2008) and meta-analysis (Cho et al., 2013) of research focused on health care providers demonstrate the overall effectiveness of incentives in increasing response rates. However, while incentives are particularly effective in increasing response rates when compared to groups that are not offered incentives, their effectiveness tends to vary along several important dimensions, including their liquidity, timing, and amount (Dykema et al., 2013a). With regard to their liquidity, monetary incentives are typically more effective than nonmonetary incentives. In terms of their timing, prepaid (not contingent on participation) incentives are usually more effective than promised (contingent) incentives in increasing survey participation. Overall, while studies indicate that larger amounts are often associated with higher response rates, the relationship between these variables is likely nonlinear (Edwards et al., 2005), with studies among physicians often offering non-significant or contradictory results (see Dykema et al., 2013a). At this time, recommendations about the optimal amount for a monetary incentive are not possible as the research available is insufficient to draw definitive conclusions (Klabunde et al., 2012; McLeod et al., 2013).

While research consistently demonstrates small, prepaid incentives increase response rates, particularly for mail surveys (Mercer et al., 2015), few studies examine the effect of a second, prepaid incentive on survey participation. Two studies explore the combined effect of sequential incentives and mailing types (e.g., priority mailing versus not). In a mail survey of physicians, Moore and An (2001) varied the amount of a prepaid incentive ($0 versus $10) and type of mailing (first class versus priority) included in an initial and follow-up mailing. The response rate achieved for the only group that received two $10 incentives—both of which were sent using priority mailings—was nearly double the response rate for a group that included no incentives and only first class mailings (i.e., response rates of 80% versus 43%, respectively), and roughly 17 percentage points higher, on average, than for four groups that received only a single $10 incentive or no incentives but a priority mailing (e.g., response rates of 80% versus 63%, respectively). Messer and Dillman (2011) experimented with methods to push sample members in the general population to complete surveys online using postal mailings, comparing web/mail methods to more traditional mail modes. They included a set of experiments in which nonresponders were sent a second $5 incentive in a priority mailing. Findings indicated the incentive significantly increased response rates by 9 percentage points, in contrast to sending either the initial $5 incentive only or the follow-up priority mailing only. While these studies are informative their implications are limited because their designs make it impossible to completely disentangle effects due to the implementation of the sequential incentive from effects due to the type of mailing.

Teisl et al. (2006) conducted a large national mail survey of the general population in which the survey was administered in two mailings. For each mailing, sample members were randomly assigned to receive one of four incentives including $1 cash, $2 cash, a $2 phone card, or a $5 phone card, resulting in 16 experimental conditions. Unfortunately, the authors did not include a no-incentive control group, making it impossible to determine whether the inclusion of a second incentive improved response rates over not including one. Overall, response rates, evaluated after each mailing, indicated the $2 cash incentive significantly increased response rates compared to the other incentive conditions, but these conditions did not differ from each other. For the groups receiving the cash incentives, the difference in response rates between the $2/$2 group and the $1/$1 was significant (e.g., response rates of 46% versus 37%, respectively), but there were no differences among the $1/$1, $1/$2, and $2/$1 groups (response rates of 37%, 42%, and 42%, respectively), and the $1/$2 and $2/$1 were not different from the $2/$2 group.

In two health-related studies of the general population, Dykema et al. (2015) also manipulated the amounts of a second incentive and included a $0 control group. In the first study, all sample members received either $2 or $5 in the initial mailing, and nonresponders to the initial mailing were sent either no incentive or $2 in a second mailing. Contrary to expectations, the inclusion of the second incentive—valued at the same amount or less than the first incentive—did not increase response rates. Informed by these findings, the researchers conducted a second experiment in which all sample members received $2 in the first mailing, but then nonresponders to the first mailing were assigned to receive no incentive or a second incentive of $2 versus $5. Results indicated that only the inclusion of a second prepaid incentive of higher value than the initial incentive—the $5 incentive—was associated with a significant increase in response of 6 points. In a recent study of the general population, Dykema et al. (2019) experimented with sequential incentive combinations of $1 in the initial mailing followed by $0 versus $2 in the second mailing to nonresponders. Overall, providing a second incentive of $2 increased response rates to that mailing by 6 points over not providing a second incentive.

Taken together, findings from past research offer support for sending nonresponders a second incentive, particularly if the amount is larger than the initial incentive in surveys of the general population. It is unknown what the effect of a sequential incentive would be for a survey of physicians using a standard mail protocol that does involve sending the incentive and questionnaire in a priority mailing.

Research Questions Concerning Sequential Incentives and Response Rates

To examine the relationship between the effect of a second incentive and response rates among physicians, we address the following questions:

• Response rate question 1 (RRQ1): Keeping the total amount a respondent could receive flat (e.g., $10), is it more effective to divide the amount into two incentives of equal value or provide the full amount in the initial mailing? We address this question by comparing response rates among physicians receiving $5 in the first mailing and $5 in the second mailing (Group $5/$5) versus those receiving $10 in the first mailing and no incentive in the second mailing (Group $10/$0).

• Response rate question 2 (RRQ2): Keeping the total amount a respondent could receive flat (e.g., $15), what effect does the amount of the second incentive relative to the first incentive have on response rates? We address this question by comparing response rates for a group receiving $5 in the first mailing and $10 in the second mailing (Group $5/$10) to a group receiving $10 in the first mailing and $5 in the second mailing (Group $10/$5).

• Response rate question 3 (RRQ3): Keeping the amount of the first incentive constant, what effect does increasing the amount of second incentive relative to the first incentive have on response rates? We address this question by comparing response rates for Group $5/$10 to Group $5/$5.

• Response rate question 4 (RRQ4a-c): What is the effect of a second incentive when the first incentive is more substantial in terms of its absolute value? We address this question by comparing response rates among the three groups formed by receiving $10 in the first mailing and $0 (Group $10/$0), $5 (Group $10/$5), or $10 (Group $10/$10) in the second mailing; thus contrasting Group $10/$5 versus $10/$0, Group $10/$10 versus $10/$0, and Group $10/$10 versus $10/$5.

Effects of Sequential Incentives on Survey Data Quality, Sample Representativeness, and Costs

While high response rates are a primary goal of data collection, equally important is ensuring that the data collected are of high quality, representative of the target population, and gathered in a cost-effective manner. In addition to examining the effects of sequential incentives on response rates, we also explore their effect on item nonresponse, sample representativeness, and costs. We do not offer specific hypotheses for these outcomes as the analyses conducted were exploratory and not motivated by specific research questions.

Survey data quality.

With regard to the effect of incentives on survey responses, Singer and Ye (2013) offer two opposing views on the likely effect of an incentive on response quality. First, in addition to increasing their likelihood to participate, an incentive might motivate respondents to work harder and provide better data while answering the survey questions. In contrast, by inducing participation among reluctant and late-responding sample members, incentives may be associated with lower data quality if those members ultimately provide poorer quality data (Fricker & Tourangeau, 2010). While few studies have examined the effect of incentives on data quality, available evidence for mail surveys indicates that they have little effect on survey responses (Singer & Ye, 2013). Where differences have been found, however, they favor the hypothesis that incentives increase data quality. For example, respondents receiving a cash incentive provided higher reports of sensitive behaviors such as smoking (indicating more accurate responding) in a survey of new mothers (Dykema et al., 2012), and respondents receiving larger incentives provided more open-ended answers and wrote more comments in a mail survey of consumers (James & Bolstein, 1990).

In their research using sequential incentives, Dykema et al. (2015) operationalized data quality by calculating respondent-level item-missing scores. They found no effect of a second incentive on data quality when mailed envelopes did not contain a message on the outside of the envelope, the standard practice for mail surveys. However, when envelopes contained a message alerting potential respondents to the presence of a monetary incentive, item nonresponse was lower for respondents who received an initial $5 incentive followed by a $2 incentive as opposed to an initial $2 incentive followed by another $2 incentive. These results point to a need for caution in the implementation of sequential incentives. We examine how item nonresponse varies across experimental incentive groups.

Sample Representativeness.

Response rates are just one indicator of overall data quality and a “low” response rate does not necessarily indicate inferior data. Even with a high response rate, nonresponse bias may be large if those surveyed differ substantially from those who are not. Conversely, nonresponse bias may be small, even with a low response rate, if respondents are similar to nonparticipants on the characteristics of interest (Biemer & Lyberg, 2004). Nonresponse bias may be lower in surveys of physicians than in surveys of other populations, possibly because physicians are relatively homogeneous (Flanigan et al. 2008; Kellerman & Herold, 2001). When documented using traditional mail survey methods, nonresponse bias has been suggested in studies that find women, younger physicians, nonspecialists, and recently licensed physicians may be more likely to respond (Barclay et al., 2002; Cull et al., 2005). Because the profile of those who fail to respond to a request for participation may be associated with nonresponse bias, especially when survey responses are associated with demographic characteristics (see, for example, Messer & Dillman, 2011), we examine whether the distribution of characteristics of respondents in the experimental groups is less representative of the population for any of the experimental groups. The characteristics we examine that were available in the administrative data are gender, geographic location, practice size, and patient volume.

Costs.

A primary objective of researchers is to achieve the highest response rate in the most cost effective manner possible. While the inclusion of an incentive in the first contact of a mail survey increases initial costs, research indicates the incentive may ultimately reduce costs by decreasing the number of nonrespondents that require additional mailings (Dykema et al., 2012). Survey designers continue to recommend sending nonresponding sample members up to three full mailings (i.e., three copies of the paper questionnaire) (Dillman et al., 2014). For a protocol that involves mailing respondents up to three questionnaires, including a second pre-paid incentive in the second mailing could reduce overall survey costs if response to the second mailing is large enough that the amount saved by having to send fewer mailings in the third mailing is larger than the total costs incurred by including the second incentive. In the analysis, we examine total costs and costs per completed survey for the incentive groups.

Methods

Participants

The population for the current study was practicing pediatricians in the United States. A sample of 1,500 physicians was randomly selected from a sample frame that consisted of a national listing of U.S. Pediatricians (N = 33,476; PEDs only), purchased from a private vendor. Eligibility criteria included: having pediatrics listed as their primary specialty; currently practicing in an office setting in the U.S.; and having a valid postal mailing address.

Survey Instrument

The survey sought to measure pediatricians’ attitudes, training needs, and practices regarding advising parents about child discipline and related parenting issues. The paper-and-pencil questionnaire consisted of 84 items formatted on eight pages.

Survey Administration

Sampled physicians received up to four contacts by mail. The initial mailing packet contained a cover letter, cash preincentive, questionnaire, and a self-addressed first-class-stamped return envelope. Materials were sent in a 10″ × 13″ envelope with a first-class stamp. Approximately a week after the initial mailing, all sample members received a reminder postcard. Roughly 1 month after the initial mailing, all nonresponders were sent a second mailing packet containing a cover letter, second incentive (if relevant), questionnaire, and return envelope. Approximately 3 weeks later, all remaining nonresponders were sent a final mailing packet that contained a cover letter, questionnaire, and return envelope. The field period extended from March to June, 2016.

Experimental Design

Experimental factors included the amount of the cash incentive sent in the first mailing ($5 or $10) and second mailing ($0, $5, or $10). We selected from among the six groups formed by crossing these factors (see Figure 1). We selected groups based on which combinations the literature indicated would be most effective and with a focus on testing the effect of different amounts of a second incentive. Consequently, we omitted the treatment of $5 in the first mailing and $0 in the second mailing. The study was approved by the institutional review board at the University of Wisconsin–Madison.

Figure 1.

Overview of the experimental design and number of cases sampled in each group.

Statistical Analysis

Response rates, calculated after each contact attempt, were computed as the number of completed questionnaires divided by the total number of sampled physicians (RR1; American Association for Public Opinion Research [AAPOR] 2016). Eight cases were deemed ineligible and removed from the denominator; for seven of these cases the sampled physician reported they were no longer practicing medicine or where not currently seeing patients, and an additional case was removed because the sample member contacted us to learn more about the incentive experiment.

Comparisons between groups were analyzed using logistic regression. To test for significant differences between groups, we fitted a baseline model in which we regressed an indicator for whether the respondent completed the survey or not on indicators for the experimental groups, omitting Group $5/$5, which served as the reference group. Tests of the pairwise differences of log odds of participating for group comparisons (e.g., Group $5/$5 versus Group $10/$0, Group $5/$5 versus $10/$5, etc.) were performed using the postestimation command pwcompare in Stata Version 15 and results using unadjusted outcomes are presented.

To evaluate item nonresponse, we computed a score for each respondent that indicated the total number of items for which the respondent’s answer to the question was “missing” divided by the total number of items in the questionnaire (n = 84). A respondent’s answer to a given question was classified as missing if the respondent left blank a question that they should have answered or if the respondent’s value for the question was undetermined because they left blank a filter question that determined the respondent’s eligibility for the current question. We also coded as missing the few cases in which respondents wrote “don’t know,” “refused,” or the equivalent as their answer to a question. Comparisons between groups were analyzed using ordinary least squares (OLS) regression. To test for significant differences between groups, we fitted a baseline model in which we regressed the respondent’s item-missing score on indicators for the experimental groups, omitting Group $5/$5, which served as the reference group. Tests of pairwise comparisons of marginal linear predictions were performed using the postestimation command pwcompare in Stata Version 15.

To test for sample representativeness, we compared the proportion of respondents with various characteristics to the distribution of the characteristics in the administrative data, which included both respondents and nonrespondents, comparing across each of the experimental groups. Differences were tested using one sample z-tests for proportions. The characteristics we examined were physicians’ gender, geographic location, practice size, and patient volume. We hand coded gender based on the gender of the physician’s first name. Geographic location was assessed using zip codes from mailing addresses. We created a dummy variable for practice size based on the variable “size” from the administrative file that ranged in value from 1 to 78 with a mean of 9.06. Our dummy variable for practice size had a value of 1 if the physician had a practice that included more than the average number of physicians and 0 otherwise. We used a similar approach to create a dummy for patient volume, such that 1 indicated the physician’s patient volume was higher than the average value of 120.2 versus 0 otherwise.

Our analysis of total costs and cost per completed survey across experimental groups included only variable costs, such as mailing costs (e.g., postage, printing, labor for mailing package assembly), data entry costs, and costs associated with the incentives (e.g., their monetary value and administration). We omitted fixed costs that were consistent across the experimental groups, such as for questionnaire design, project management, and data delivery.

Results

Response Rates

Just over a third of the surveys, 535 were returned after the first mailing for an initial response rate of 35.9% (Table 1). Respondents completed an additional 165 questionnaires in response to the second mailing, raising the response rate to 46.9%. After the final mailing, a total of 779 questionnaires were returned for an overall response rate of 52.2%.

Table 1.

Sample Sizes, Response Rates, Item Nonresponse Rates, and Costs by Experimental Groups.

	Experimental Groups
	Group $5/$5	Group $5/$10	Group $10/$0	Group $10/$5	Group $10/$10	Total
Sample Sizes
N	300	300	300	300	300	1,500
Not eligible	3	5	0	0	0	8
Completed surveys
After initial mailing	88	112	102	108	125	535
After second mailing	129	149	124	140	158	700
After final mailing	145	171	131	156	176	779
Response Rates (%)
After initial mailing	29.6	38.0	34.0	36.0	41.7	35.9
After second mailing	43.4	50.5	41.3	46.7	52.7	46.9
After final mailing	49.0	58.0	43.7	52.0	58.7	52.2
Item-missing Scores
Mean	0.020	0.021	0.016	0.022	0.012	0.018
Standard Deviation	0.050	0.065	0.038	0.048	0.026	0.047
Variable Costs
Incentives only	$2,540	$3,350	$3,000	$3,959	$4,690	$17,539
Total	$5,705	$6,363	$6,106	$6,988	$7,602	$32,764
Per completed survey	$39	$37	$47	$45	$43	$42

Overall, the response rate was the lowest for the group that did not receive a second incentive, Group $10/$0, and highest for the group that received $10 initially and as a second incentive, Group $10/$10. We expected response rates after the initial mailing to be roughly equivalent for the two groups that received $5 in the first mailing (Group $5/$5 and Group $5/$10) and for the three groups that received $10 in the first mailing (Group $10/$0, Group $10/$5, and Group $10/$10). However, due to chance, response in Group $5/$10 was significantly higher than Group $5/$5 and marginally significantly higher for Group $10/$10 than Group $10/$0 (Table 1). When we collapse across the groups receiving $5 versus $10 initially, we find the average response rate for the three groups receiving $10 in the initial mailing is higher than the average response rate for the two groups receiving $5 in the initial mailing, 37.2% versus 33.8%, but the difference is not statistically significant (p > .05).

To address our research questions about response rates (e.g., RRQ1 to RRQ4c), we provide pairwise comparisons of the results from logistic regression analyses (Table 2). We compare response rates among physicians in Group $5/$5 to Group $10/$0 to evaluate the effectiveness of dividing the amount into two incentives of equal value instead of providing the full amount in the initial mailing (RRQ1). While the final response rate was five points higher for Group $5/$5 than Group $10/$0, the magnitude of the difference was not statistically significant.

Table 2.

Logistic Regression Analyses of Survey Response.

Response rate questions	Pairwise comparisons	After initial mailing		After second mailing		After final mailing
		Odds Ratio	95% CI	Odds Ratio	95% CI	Odds Ratio	95% CI
RRQ1	Group $5/$5 vs Group $10/$0	0.82	0.58–1.15	1.09	0.79–1.51	1.23	0.89–1.70
RRQ2	Group $5/$10 vs Group $10/$5	1.09	0.78–1.52	1.17	0.85–1.61	1.27	0.92–1.76
RRQ3	Group $5/$10 vs Group $5/$5	1.45*	1.03–2.05	1.33+	0.96–1.84	1.44*	1.05–2.00
RRQ4a	Group $10/$5 vs Group $10/$0	1.09	0.78–1.53	1.24	0.90–1.72	1.40*	1.01–1.93
RRQ4b	Group $10/$10 vs Group $10/$0	1.39+	1.00–1.93	1.58**	1.14–2.18	1.83***	1.32–2.53
RRQ4c	Group $10/$10 vs Group $10/$5	1.27	0.92–1.76	1.27	0.92–1.75	1.31	0.95–1.81

Note.

⁺p < .10;

^*p < .05;

^**p < .01;

^***p < .001.

We compare Group $5/$10 to Group $10/$5 to examine the effect the amount of the second incentive relative to the first incentive has on response rates when the total amount a respondent might receive is flat (e.g., $15) (RRQ2). Although the response rate is higher for Group $5/$10 than Group $10/$5, the difference between the response rates is not significant.

Previous research indicated the amount of a second incentive might need to exceed the amount of the first to be effective. To examine the effect of the amount of the second incentive relative to the first (RRQ3), we compare Group $5/$10 to Group $5/$5. Results indicate a significantly (or marginally so) higher response for Group $5/$10 group after each mailing. However, these results are tempered by the artificially high response for Group $5/$10 after the initial mailing.

Finally, we explore the use of a second incentive among physicians when the first incentive is more substantial in terms of its absolute value (i.e., when the first incentive is $10) (RRQ4a-c). Decreasing the value of the second incentive relative to the first—comparing Group $10/$10 to Group $10/$5—did not have a significant effect on response rates. However, not including a second incentive at all did affect response rates when the starting value for the first incentive was $10. After the final mailing, response rates were significantly higher for the $10/$5 and $10/$10 groups than the $10/$0 group.

For completeness, statistical tests for the remaining pairwise group comparisons are presented in Online Appendix A. These comparisons were not motivated by specific research questions and are not discussed further.

Item-Missing Scores

The lower panel in Table 1 provides the means and standard deviations of the item-missing data scores by experimental group. Levels of missing data were low overall, ranging from 1.2% for Group $10/$10 to 2.2% for Group $10/$5. Regression results indicated no significant differences between the groups at the p < .05 level.

Sample Representativeness

Table 3 presents results for tests of sample representativeness. Overall, we found few differences in the distribution of characteristics between the administrative data and the experimental groups. We found no differences by gender or based on the volume of patients seen at the physician’s practice. In contrast, physicians in Group $10/$0 were less likely to participate if they resided in the South or were from larger practices, while physicians in Group $5/$5 were more likely to respond if they were from the South.

Table 3.

Analysis of Sample Representativeness: Distributions of Key Variables From the Administrative Data and by Experimental Group.

	Administrative Data (n = 1,492)	Experimental Groups
		Group $5/$5 (n = 145)	Group $5/$10 (n = 171)	Group $10/$0 (n = 131)	Group $10/$5 (n = 156)	Group $10/$10 (n = 176)
Gender (%)
Female	911 (61.1)	80 (55.2)	98 (57.3)	79 (60.3)	101 (64.7)	104 (59.1)
Male	581 (38.9)	65 (44.8)	73 (42.7)	52 (39.7)	55 (35.3)	72 (40.9)
p Level		n.s.	n.s.	n.s.	n.s.	n.s.
Geographic location
Northeast/Midwest/West	1,039 (69.6)	89 (61.4)	126 (73.7)	106 (80.9)	113 (72.4)	124 (70.4)
South	453 (30.4)	56 (38.6)	45 (26.3)	25 (19.1)	43 (27.6)	52 (29.6)
p Level		*	n.s.	**	n.s.	n.s.
Practice size
At or below the mean	1,101 (73.8)	107 (73.8)	129 (75.4)	109 (83.2)	119 (76.3)	133 (75.6)
Above the mean	391 (26.2)	38 (26.2)	42 (24.6)	22 (16.8)	37 (23.7)	43 (24.4)
p Level		n.s.	n.s.	*	n.s.	n.s.
Patient volume
At or below the mean	971 (65.1)	89 (61.4)	103 (60.2)	81 (61.8)	105 (67.3)	124 (70.5)
Above the mean	521 (34.9)	56 (38.6)	68 (39.8)	50 (38.2)	51 (32.7)	52 (29.5)
p Level		n.s.	n.s.	n.s.	n.s.	n.s.

Note. The sample for the administrative data includes all respondents and nonrespondents. Comparisons are made for each experimental group and the administrative data using one sample z-tests for proportions. Significant differences are indicated as follows:

^*p < .05;

^**p < .01.

The abbreviation “n.s.” indicates the difference was not significant.

Costs

Total costs and costs per completed survey are provided in lower panel of Table 1. Total costs were lowest for Group $5/$5 and highest for Group $10/$10. Overall, costs per completed survey were lower for the groups receiving $5 versus $10 in the initial mailing. Group $5/$5 was the most cost effective treatment with a cost of $37 per completed survey and Group $10/$0 was the least cost effective treatment with a cost of $47 per completed survey.

Discussion

To date, a substantial number of studies demonstrate the effectiveness of a using a small, prepaid incentive to increase response rates in surveys of physicians and other clinicians (Cho et al., 2013; VanGeest et al., 2007). More recently, a few studies of the general population have incorporated the recommendation of Dillman et al. (2014) to include a second cash incentive in follow-up contacts in order to boost participation among initially reluctant sample members (e.g., Dykema et al., 2015, 2019). To the best of our knowledge, this is the first study to experiment with different amounts of sequential incentives in a mail survey of physicians.

With regard to their effects on response rates, our results suggest a sequential incentive is likely to increase response over not including a second incentive when the initial incentive is valued at $10. In comparison to Group $10/$0, response rates were significantly higher for Group $10/$5 and $10/$10, but these two groups did not differ from each other. Our results also lend some support to previous research that finds the amount of the second incentive might need to exceed the amount of the first to be effective. The final response rate for Group $5/$10 was significantly higher than Group $5/$5, although these results need to be viewed in the context of the artificially high response for Group $5/$10 after the initial mailing. Finally, for a fixed incentive amount of $10 or $15 dollars overall, our results also suggest an incentive strategy that uses a smaller amount first may be more likely to be effective. Although not statistically significant, owing to the relatively small sample sizes, response rates were higher for Group $5/$5 versus Group $10/$0 and for Group $5/$10 versus Group $10/$5.

Cost analyses must be considered in the context of response rates. With regard to costs, including the more expensive $10 initial incentive increased costs per completed survey over a $5 initial incentive. The inclusion of a second incentive, however, appeared to save on costs per completed survey. Comparing Group $5/$5 to Group $5/$10, an increase in the amount of the second incentive from $5 to $10 was associated with a savings of $2 per completed survey (i.e., $39 per completed survey for Group $5/$10 versus $37 per completed survey for Group $5/$5). A similar pattern was shown for the groups receiving an initial incentive of $10. Comparing Group $10/$0, Group $10/$5, and Group $10/$10, each increment of $5 for the second incentive was associated with a reduction in costs per completed survey of $2, with costs per completed survey for the groups at $47, $45, and $43, respectively. Thus, while the second incentives increased overall costs, because they also increased response rates, they lowered costs per completed survey.

Past research indicates low nonresponse bias in surveys of physicians (Flanigan et al., 2008; Kellerman & Herold, 2001), likely owing to fact that physicians as a group are relatively homogeneous, and researchers frequently employ methods known to reduce bias such as including incentives and contacting sample members multiple times (McFarlane et al., 2007). Although we find little evidence of nonresponse bias across the incentive groups, our results indicate that sequential incentives may reduce bias by bringing in sample members with a lower likelihood to participate. Focusing on the two groups that received a pre-paid incentive of $5, we find Group $5/$5 under-represents the proportion of physicians residing in the northeast, Midwest, or west. In contrast, there are no significant differences between the characteristics of the physicians in the administrative data and the resulting sample for the group with the larger second incentive, Group $5/$10. Looking just at the groups that received a pre-paid incentive of $10, the recruiting strategy that was not associated with a second incentive—Group $10/$0—was the only group that failed to achieve a sample distribution that matched the population on the variables included in the administrative file. Group $10/$0 was associated with an underrepresentation of physicians from the south and with a larger practice size.

Our experimental design also allows us to offer some recommendations for best practices. First, we consider recommendations for how to distribute incentive amounts of $10 and $15 across mailings. Keeping the total amount a respondent might receive flat, in this case $10, both overall costs and costs per completed survey were lower for Group $5/$5 versus Group $10/$0. Further, this cost savings did not come at the expense of response rates as response rates were higher (although not significantly so)—49.0% versus 43.7%—for Group $5/$5 than Group $10/$0. Again, keeping the total amount a respondent might receive flat, in this case $15, both overall costs and costs per completed survey were lower for Group $5/$10 versus Group $10/$5, and this cost savings did not come at the expense of response rates as response rates were higher—58.0% versus 52.0%—for Group $5/$10 than Group $10/$5. Overall, Group $5/$10 achieved a response rate as high as Group $10/$10 (response rates did not significantly differ between these groups; Online Appendix A), was associated with the lowest costs per completed survey, and produced a sample with characteristics similar to the administrative data. In contrast, Group $10/$0 was associated with the lowest response rate, the highest cost per completed survey, and the least representative sample. It is worthwhile to note that Group $10/$0 is the only group in our study that reflects the standard practice of including the entire incentive amount in the initial mailing.

We note several limitations with our design and methods. First, while we expected response rates after the initial mailing to be roughly equivalent for the two groups that received $5 in the first mailing and the three groups that received $10 in the first mailing, due to chance, response in Group $5/$10 was significantly higher than Group $5/$5 and marginally significantly higher for Group $10/$10 than Group $10/$0. These unequal response rate values temper the conclusions we draw. Second, sample sizes for our experimental groups were relatively small, limiting our ability to detect statistically significant differences. This was most frustrating for the comparison of response rates for Group $5/$5 versus $10/$0 and Group $5/$10 versus $10/$5. While our cost analysis favored Group $5/$5 and Group $5/$10 and final response rates were higher for these groups, the differences in response rates were not significant, and our recommendations regarding these groups must be evaluated with this limitation. Third, because of costs, we were limited in the number of cases we could afford to field. In designing the experiment, we placed priority on testing the effect of different amounts of a second incentive and omitted a treatment of $5 in the first mailing and $0 in the second mailing. Because this combination might reflect what is closer to common practice for mail surveys of physicians, including this group in our design would have been informative. Fourth, our respondents included physicians with pediatrics as their primary specialty, and our results may not be generalizable to physicians in other specialties. Fifth, while we increased the representativeness of our data with regard to several sample characteristics for the $5/$10, $10/$5, and $10/$10 groups, we cannot conclude with certainty that doing so reduced nonresponse bias for other measures in the survey.

Given the proliferation of surveys conducted with clinicians online, we encourage research aimed at exploring how sequential incentives could be employed in a fully web-based design or in a web-mail mixed-mode design. When data collection moves from a postal to a web administration, significant constraints to delivering incentives are encountered. Prepaid incentives are difficult to deliver effectively in web surveys that use e-mail to recruit participants. For example, it is not possible to administer cash using e-mail and alternatives, such as electronic gift certificates or online banking, may not be as effective (Birnholtz et al., 2004; Bosnjak & Tuten, 2003). Further, methods found to be successful with mail surveys may not directly transfer to web surveys (see Dykema et al., 2011). For web surveys, this complexity in incentive administration may require an increase in the amount of the incentive in order to have a strong effect on participation. When researchers have access to postal addresses in addition to email addresses, a design that includes inviting clinicians to participate in an online survey in a postal letter that includes a cash incentive and manual-entry URL, followed by a postal reminder with a second cash incentive and URL, may be more effective at increasing participation than using only emailed invitations and follow-up contacts (Dykema et al., 2013b).

While larger incentive amounts are often associated with higher response rates, a recent meta-analysis of household surveys reports a strong, nonlinear relationship between incentive amounts and participation (Mercer et al., 2015). In their study of sequential incentives in a survey about health administered to general population sample, Dykema et al. experimented with sequential incentive combinations of $2/$2, $5/$2, and $2/$5. They found only the $2/$5 combination to be effective. Teisl et al. also reported no differences in response rates among groups receiving $1/$1, $1/$2, or $2/$1. These findings raise important questions about both the absolute and relative size of the first incentive compared to the second one, since the sequential incentive always has an effect conditional on the amount of the first incentive.

Future research should continue to explore methods to increase participation among physicians in mail surveys. Given declining response rates among physicians (Cho et al., 2013), sequential incentives offer a promising and cost-effective method for improving survey response. Additional studies are needed to see if the patterns found here are replicated for other topics and across medical specialties. We encourage studies that evaluate the cost-effectiveness of different incentive combinations and examine not just response rates but other measures of data quality.