Effect of Small Monetary Incentive and Demographic Characteristics on Response Rate of Self-Administered Questionnaire Mailed to Rural Women

Abstract

Objectives:

The National Children’s Study (NCS) is a longitudinal observational study that will examine the effects of genetics and environment on the health and development of children in the United States. The NCS is in a Vanguard, or pilot phase, so it is important to determine the feasibility, acceptability and cost of different data collection methods. The purpose of this study was to determine whether demographic characteristics differed in self-administered questionnaire (SAQ) completion rates, and to examine response rates when a $2 incentive was included with the mailed questionnaire.

Methods:

Subjects for the study (~500) were being followed by a local call center and were mailed SAQs. Women who did not complete their last telephone event were eligible. Women were randomized to receive a $2 bill incentive with their SAQ (approximately 250 women in each group).

Results:

Approximately 450 women comprised the final sample. In the incentive group, 35% of SAQs were returned, which was significantly higher than the 27% returned in the non-incentive group (p<0.001). No significant differences in completion rates based on demographic characteristics were seen, except in divorced women. The final cost per completed SAQ was $12.51 in the incentive group compared to $7.57 for the non-incentive group.

Conclusions:

Adding a $2 bill as an incentive significantly increased response rates of hard-to-reach rural women completing mailed questionnaires about pregnancy information. Future research should focus on different incentive amounts and types and how they influence response rates.

Introduction

The National Children’s Study (NCS) is a longitudinal observational study that will examine the effects of genetics and environment on the health and development of children in the United States. Participants are followed from before birth through 21 years of age.¹ During the initial vanguard phase of the study, eligible households were enumerated to determine if women between the ages of 18 and 49 years resided in them. These women then completed a pregnancy screener to determine pregnancy status and probability of becoming pregnant. Those women who were not pregnant at the time of screening, but were age-eligible, were followed-up periodically through a local Call Center to collect information on pregnancy status and update contact information. Approximately 4,000 rural women were followed at the South Dakota State University Study Center (SDSU SC) Call Center.

A paper-based version of the Call Center questionnaire was developed for women who were difficult to contact by phone. This one-page self-administered questionnaire (SAQ) collected the same information as the telephone interview, but could be completed at the participant’s leisure. A business-reply paid envelope was included for returning the SAQ. Anecdotal evidence from other Study Centers suggested mailing $5 with the SAQ increased response rates. Published research also has shown that including an incentive with a questionnaire increases response rates.²

The NCS is in a Vanguard phase, so it is important to determine the feasibility, acceptability and cost of different methods of data collection. Specifically, it is important to examine whether women who are hard to reach over the phone are easier to reach via postal mail. Poor response rates for a certain mode of data collection may lead to a biased sample and ultimately limit generalizability of study findings. Achieving high response rates can be time-consuming and expensive; therefore it is valuable to know how influential a small monetary incentive can be.

The purpose of this study was to determine whether there were differences in SAQ completion rates by demographic characteristics (age, race/ethnicity, marital status), and to compare response rates when a $2 incentive was included with the mailed questionnaire. We hypothesized that those receiving the incentive would have significantly higher response rates.

Methods

Subjects

Subjects for the incentive study (520) were a subset of nearly 2,300 rural women being followed by the SDSU SC Call Center. This includes women from Brookings County, South Dakota and Pipestone, Lincoln and Yellow Medicine Counties in Minnesota. Women who did not complete their last telephone event were eligible to receive the mailed SAQ. Approval for the study was obtained from the South Dakota State University Institutional Review Board and from the NCS Program Office.

Procedures

Mailed SAQs were prepared according to protocol. This involved printing a questionnaire with the local SDSU SC logo and contact information and attaching a label with the woman’s ID. A cover letter addressed to the woman was also generated that contained information about the questionnaire as well as thank you message. Additionally, a brochure about the NCS and a magnet indicating household eligibility with the SDSU SC toll-free number also was included. Women were randomized to either receive a $2 bill incentive with their SAQ (approximately 250 women in each group) or no incentive at all.

Data Analysis

Data were analyzed using SAS Version 9.2 software (SAS Institute, Inc., Cary, North Carolina, USA). A chi-square test was calculated to determine if including an incentive significantly increased the completion rate of the SAQ. Differences in completion rates were assessed among different age groups, marital statuses, ethnicities and races. PROC FREQ procedures were used to determine returned percentages of the different categories that were analyzed.

Results

Table 1 gives an overview of the status of all SAQs.

Table 1.

Final status and description of SAQs, by Incentive or No Incentive

Final Status	Description	Incentive (n=269)	No Incentive (n=251)
Complete*	The SAQ was completed and returned within 75 days.	30%	23%
Moved, Unknown Address**	Return information indicated a move, but no forwarding address.	1%	1%
Moved, Out of Study**	Return information indicated a move to an ineligible address, so the woman was removed from the study.	13%	12%
Telephone Research Center Event**	Call Center information obtained before SAQ was returned.	<1%	<1%
Other	Did not return SAQ.	55%	64%

^*Includes a small number that were initially returned by the Postal Service and then re-mailed

^**Not included in final analysis, ineligible

Overall, 30% of completed SAQs were received from the incentive group and 23% were received from the non-incentive group while 14% from the incentive group and 13% from the non-incentive group were returned by the postal service. Completed SAQs included those that, based on return information received from the postal service, were re-mailed (11% in the incentive group and 6% in the non-incentive group) and subsequently returned completed (3% and 1%, respectively). In some cases, the Call Center was able to contact the woman and complete the questionnaire over the phone (<1% in both groups). In other cases, a participant had moved, making her no longer eligible for the study (13% in the incentive group and 12% in the non-incentive group). Only 1% of the women moved and did not provide a forwarding address. More than half (55%) of the women in the incentive group and 64% in the non-incentive group did not return the SAQ and did not fall into any of the previous categories.

Only those in the ‘Complete’ and ‘Other’ groups were considered for final analysis, which resulted in 229 in the incentive group and 218 in the non-incentive group (Table 2).

Table 2.

Description of SAQs included in final analysis, by Incentive or No Incentive

Final Status	Description	Incentive (n=229)	No Incentive (n=218)
Complete*	The SAQ was completed and returned within 75 days.	35%	27%
Other	Did not return SAQ.	65%	73%

^*Includes a small number that were initially returned by the Postal Service and then re-mailed

In the incentive group, 35% of SAQs were returned which was significantly higher than the 27% returned in the non-incentive group (p<0.001).

The cost to initially mail the SAQ was $1.59 plus the incentive, if applicable (including printing, labeling, postage and envelopes). Thus it cost approximately $966 to mail the SAQ’s with an incentive and $399 to mail the SAQs without an incentive (Table 3).

Table 3.

Summary of costs for mailed SAQs and returned envelopes

	Original Cost	Returned Incentives	Returned Envelopes	Re-mailed SAQs	Completed & Returned	Final Costs
Cost	$1.59 +incentive	$2 each	$0.44 each	$1.59 +incentive	$0.44
Incentive	$965.71 (269)	−$78.00 (39)	$17.16 (39)	$104.11 (29)	$3.96 (9)	$1,013
No Incentive	$399.09 (251)	$0.00 (32)	$14.08 (32)	$23.85 (15)	$1.32 (3)	$439

If an SAQ status wasn’t ‘Complete’ or ‘Other’, the envelope had been returned, at a charge of $0.44 for return service. Additionally, with returned mail, incentives were also returned. Those in the ‘Re-mailed’ group incurred charges both for returning of the first envelope and mailing of the second. The final cost per completed SAQ was $12.51 in the incentive group compared to $7.57 for the non-incentive group. Based on these figures and including only those women ultimately found to be eligible for the study, it cost 2.3 times more to include the $2 incentive and the response rate increased by 1.3 times if the incentive was included.

Demographic information was collected on all women who were sent SAQs. No significant differences in completion rates based on demographic characteristics were seen, except that divorced women in the incentive group returned significantly more than divorced women in the non-incentive group (p=0.001).

Discussion

The purpose of this study was to examine response rates and associated costs when a $2 incentive was included with a mailed questionnaire and to determine whether differences in demographic characteristics between responders and non-responders were present. In general, the mailed SAQ is used when telephone contacts have been unsuccessful. It was hypothesized that although these women are hard to reach, adding a monetary incentive might increase response rates.

Previous research has suggested ways to increase response rates of mailed questionnaires such as following up with a postcard or changing questionnaire color³ or hand-addressing envelopes and sending them via certified mail.⁴ Completion rates for mailed surveys have been shown to be increased when telephone contact was made prior to the mailing⁵ and when stamped return envelopes were used instead of business reply.⁶

In addition to cosmetic and logistical alterations, research has shown that providing study participants with incentives generally increases response rates, especially when the incentive is non-contingent or is some form of direct payment.⁷ It is not clear what magnitude of monetary incentive is most effective at eliciting a response, and if that response rate differs by demographic characteristics of the study participants.

Various studies have examined how incentives impact clinicians responding to surveys, including lotteries versus fixed incentives⁸ and contingent versus non-contingent cash and checks⁹, with immediate cash payments being most successful. Hawley and colleagues examined how differing values of non-contingent incentives affected response rates among mental health clinicians.¹⁰ The clinicians received a magnet, $1, $2 or $5 bill or no incentive. The highest response rate was from those who received the $5 bill. Response rates increased between each category: no incentive (39.8%), magnet (41.8%), $1 bill (51.0%), $2 bill (60.6%), and $5 bill (64.6%). Analyses determined that type of clinician did not affect response rates. These results are similar to the current study, which indicated that including a monetary incentive significantly increased response rates. The response rates in this study are lower, however, which could be due to the hard-to-reach nature of these participants. Additional research could assess whether a great monetary incentive elicits a significantly greater response rate in women between the ages of 18 and 49 years. We chose to use a $2 bill because of its uniqueness and to see if a relatively small dollar amount could elicit significant differences in response rates.

Similarly, Kanaan and colleagues explored survey response rates when including an immediate incentive, a promised incentive or no incentive.¹¹ Those who received the incentive immediately responded at a significantly higher rate (61%) than those who were promised the incentive upon completion (48%). Response rates were not different between the no incentive group (43%) and the promised incentive group (48%). The authors cited guilt as a possible reason for the effectiveness of the immediate incentive; however it may not be applicable to all groups of people. Future research could explore the effectiveness of immediate versus promised incentives in the NCS study population.

Although previous research has shown that college students who received a $5 or $2 incentive with a questionnaire were more likely to respond (41% and 36%, respectively) than those who receive no incentive (23%)¹², we did not find any significant differences in response rates among age groups or within the college-aged individuals.

Additional forms of incentives could be explored as well. Liu and Geidenberger found African American women of childbearing age had increased response rates when given a gift card as opposed to a calling card.¹³ Whiteman and colleagues found that middle-aged women who received either a scratch-off lottery ticket and a postcard, or a $1 bill had higher response rates for a mailed questionnaire than those who received no incentive.¹⁴ In this case it seems that a potential monetary incentive acts just as well as a monetary incentive in eliciting higher response rates.

Conclusions

Adding a $2 bill as an incentive significantly increased response rates of hard-to-reach rural women completing mailed questionnaires about pregnancy information. Completion rates did not vary by demographic factors, however divorced women in the incentive group returned significantly more than divorced women in the non-incentive group. Although response rates were increased overall, cost must be taken into consideration and the benefits of mailing a monetary incentive must outweigh the costs. Future research should focus on different incentive amounts and types and how they influence response rates in young to middle-aged women, as well as the optimal method for data collection in this population.

Biography

Lacey A. McCormack

Lacey A. McCormack, MPH, RD, LN is Program Director with the EA Martin Program at South Dakota State University and a Registered Dietitian with a Master of Public Health in Public Health Nutrition from the University of Minnesota (2009). She is in the final year of her doctoral program. During graduate school she worked on projects and published articles in the area of food and beverage advertising and youth and weight-related teasing in youth. Her professional experience includes the management and execution of the National Children’s Study at the SDSU Vanguard Center, which involved management of electronic-, telephone- and paper-based data collection. Her current responsibilities center on developing project and proposal ideas and applying for funding along with analyzing and interpreting data for publication. Her research interests include: determinants of body weight and composition in rural children, adolescents and young adults; eating behaviors and food environments; and program evaluation.