Participation Willingness in Web Surveys: Exploring Effect of Sponsoring Corporation's and Survey Provider's Reputation

Abstract

Prior research involving response rates in Web-based surveys has not adequately addressed the effect of the reputation of a sponsoring corporation that contracts with a survey provider. This study investigates the effect of two factors, namely, the reputation of a survey's provider and the reputation of a survey's sponsoring corporation, on the willingness of potential respondents to participate in a Web survey. Results of an experimental design with these two factors reveal that the sponsoring corporation's and the survey provider's strong reputations can induce potential respondents to participate in a Web survey. A sponsoring corporation's reputation has a greater effect on the participation willingness of potential respondents of a Web survey than the reputation of the survey provider. A sponsoring corporation with a weak reputation who contracts with a survey provider having a strong reputation results in increased participation willingness from potential respondents if the identity of the sponsoring corporation is disguised in a survey. This study identifies the most effective strategy to increase participation willingness for a Web-based survey by considering both the reputations of the sponsoring corporation and survey provider and whether to reveal their identities.

Introduction

Compared to traditional survey modes, Web-based surveys often confer a number of advantages.^1–3 These advantages help explain the exponential expansion of the use of Web-based surveys, especially in corporate research.⁴ However, easy access to the Internet to collect data may not translate into a high response rate simply because potential survey participants may not be willing to respond. Considerable research has studied strategies to increase a potential respondent's willingness to participate in Web-based surveys by focusing on improvements in survey design and by identifying psychological constructs to minimize nonresponse to Web surveys.^5–10

Among the potential factors affecting Web survey response, three of them—the survey topic, the sponsoring corporation, and the survey provider—are the primary determinants influencing a potential respondent to participate and typically appear on the survey's welcome page or in the e-mail invitation or possibly on the survey's banner advertisement. Since information acquired earlier in the decision process plays a more significant role than information acquired later,¹¹ potential respondents formalize their initial participation intention based on the assessment of these three factors. Hence, the survey topic and the names of the sponsoring corporation and/or survey provider have an important role in a potential respondent's decision to participate in a Web-based survey.

Prior research^4,12 has focused primarily on investigating the effects of topic saliency factors, that is, topic involvement and topic sensitivity on participation willingness in surveys. These studies have not rigorously examined the effect of either the sponsorship or the contract provider on Web-based surveys, nor have these studies fully explained how the prominence of either the sponsor or contract provider influences participation willingness in Web surveys. Limited prior research^13,14 does examine the role of a sponsoring corporation's logo or prominence in Web surveys on response rates. However, the results from these prior studies are inconsistent and generally indicate that only slight changes in participation willingness are due to these factors. Furthermore, no study extends this line of research to a distinction between the effect on participation willingness due to the reputation levels of a sponsoring corporation and the identity of a survey provider.

A recent comprehensive study¹⁵ systematically reviewed the factors influencing the response rate of Web survey, but did not assess the differential role of sponsoring corporation and survey provider on participation rates. A sponsoring corporation of a Web survey typically initiates an investigation into available external providers and then selects one based on their efficiency and experience as well as their reputation. Clearly, research evaluating the effects of the reputation of both the sponsor and the provider may assist in explaining the importance of these factors in soliciting participation in Web-based surveys. It is possible that the roles of the sponsoring corporation and survey provider differ in their impact on a participant's intention to respond to a Web-based survey. A study of the effect of the levels of prominence of both the sponsoring corporation and survey provider has pragmatic implications that would be beneficial in assessing the feasibility of an online survey.

The prominence of the sponsorship represents an indirect measure of trust that may easily influence the response rate to a survey. Although the trustworthiness of the sponsoring corporation and/or survey provider is not readily observable to potential respondents in the online environment, research in signaling theory suggests that it can be evaluated by signals such as the reputation and status.¹⁶ Reputation is a characteristic that evolves over time, and considerable investment is required to establish a positive valence and this valence cannot be changed instantaneously.¹⁷ Thus, signaling positive reputation features serves to influence a potential participant's perception of the sponsor's and the provider's trustworthiness.

Due to the predominantly commercial nature of Web survey practice, this study focuses solely on corporate research (a corporation uses a third party to conduct research). This study explored the effect of the reputation level of a sponsoring corporation and/or survey provider on promoting participation in a survey and examined the effect of different levels of reputation for the sponsoring corporation and survey provider of a survey. This study investigated the hypothesis that strategies that include revealing only the identity of either the sponsoring corporation or the survey provider, or revealing both of them during surveys will have different effects on participation willingness. In addition, this study examines the effect of combinations of reputation levels of the sponsoring corporation and survey provider on participation willingness.

Methods

Design and procedure

Separate samples are selected to test each hypothesis. Since participants are asked to respond to items measuring participation willingness on two different scenarios for each hypothesis, a randomized block design or, equivalently, a paired t test is used for data analysis.

Experiment 1 (sponsoring corporation: strong reputation vs. weak reputation) investigates the effect of the reputation for the sponsoring corporation on participation willingness promotion. Experiment 2 (survey provider: strong reputation vs. weak reputation) explores the effect of the reputation for the survey provider on participation willingness promotion. Experiment 3 is a 2 (sponsoring corporation: strong reputation vs. weak reputation)×2 (survey provider: strong reputation vs. weak reputation) factorial design, and examines the effect of sponsoring corporation reputation and survey provider reputation on participation willingness promotion. Results from these experiments are used to test the proposed hypotheses. For each experiment, a different set of survey respondents is selected.

China Mobile and Sina.com are selected as the reputable sponsoring corporation and survey provider, respectively, for the experiments since each has a strong reputation. China Mobile, established in 2000, has become the largest telecom operator in the world. The total number of customers of China Mobile has exceeded 500 million. China Mobile is one of China's most admired companies and is recognized as the top employer by a poll conducted by Chinahr.com, the largest human resources Web site in China. Sina.com, a National Association of Securities Dealers Automated Quotations (NASDAQ)–listed company, acts as a leading online media company and mobile value-added service provider in China, which has won a China's Most Admired Companies Award for eight consecutive years since 2003. Another two ostensible firms (one is articulated as a telecom device manufacturer and the other is expressed as an online media firm) are fabricated as the sponsoring corporation, and survey provider with weak reputations, respectively.

The three experiments recruit different participants from undergraduate and postgraduate classes. To control the impact of topic involvement on the results, this study selects the “Smart Phone Application Services Survey” as the survey topic and devises 10 items included in the survey. A randomized block design method is adopted to avoid the influence of personality and other individual-related variables. The fabricated survey instruction and the identity of the sponsoring corporation and the survey provider are placed at the beginning of the survey or concealed in the instruction section according to the context manipulation.

The procedures of the three experiments are similar. First, participants are asked to read the general instructions provided on the screen carefully. After the general instruction, the following screens randomly display a series of “smart cell phone application services” surveys in order to eliminate the order effects. Participants are asked to browse each survey carefully. After browsing the survey, each participant is requested to complete a questionnaire on the next screen that involves either the dependent variable or the participation willingness measure. Participants answer a series of questions about demographic information (gender, age, education level, and hours spent on using the Web) after the last survey design.

Three items (“I am likely to participate in this Web survey”; “I am willing to participate in this Web survey”; “I will try to participate in this Web survey if I received a request”), adapted from the items used in Fang et al.,⁸ measure participation willingness for the satisfactory measurement properties. Participants rate each of these items on a 7-point scale (strongly disagree, disagree, somewhat disagree, neutral, somewhat agree, agree, and strongly agree).

Cronbach's alpha is used to measure the reliability of the participation willingness scale. In experiment 1, the two Cronbach's alphas for the three measurement items are 0.91 and 0.94 for the two scenarios. In experiment 2, the respective Cronbach's alphas are 0.93 and 0.94. In the last experiment, the Cronbach's alpha values in four scenarios: (1) weak reputation sponsoring corporation and weak reputation survey provider, (2) weak reputation sponsoring corporation and strong reputation survey provider, (3) strong reputation sponsoring corporation and weak reputation survey provider; (4) strong reputation sponsoring corporation and strong reputation survey provider are 0.94, 0.93, 0.92, and 0.89, respectively. All of these values are above the recommended threshold of Cronbach's alpha for establishing internal consistency.¹⁸ Thus, items are sufficiently reliable to be averaged into a composite index.

Manipulation checks

To ensure that the manipulations of the treatments align with the authors' intention, a pilot study with 20 participants is conducted. Four items adapted from Kim et al.¹⁹ are used for measuring reputation: “This firm is well known”; “This firm has a good reputation”; “This firm has a reputation for being honest”; “I am familiar with the name of this firm.” Participants rate these items on a 7-point scale (1=strongly disagree, 7=strongly agree). A randomized block analysis of variance shows that no significant difference exists in reputation between Sina.com and China Mobile [F(1, 19)=0.09, p=0.77]. A similar finding exists between the two ostensible firms [F(1, 19)=0.32, p=0.58]. However, with a p value <0.01, the reputation for either Sina.com or China Mobile is significantly higher than either of the ostensible firms. In summary, the manipulations of the treatments are successful.

Results

Experiment 1

Out of a sample of 100 participants, one respondent is eliminated because of missing data. Therefore, the final sample includes 99 complete and valid responses. The participants consist of 64 (64.6 percent) men and 35 (35.4 percent) women. For 81 percent of participants, their ages range from 20 to 29 years.

The authors conduct a paired t test to investigate the impact of the sponsoring corporation having a strong reputation (M=4.55, SD=1.53) and sponsoring corporation having a weak reputation (M=4.27, SD=1.70) on participation willingness promotion. The results show a significant mean difference, t(98)=2.20, p=0.03, Cohen's d=0.17. Therefore, a sponsoring corporation having a strong reputation is more effective in increasing survey participation than a sponsoring corporation having a weak reputation.

Experiment 2

Out of a sample of 100 survey participants, total usable responses are 90 (39 women and 51 men). Participants' age ranged from 20 to 29 years for 87.8 percent of the respondents.

A paired t test is conducted to investigate the impact of the survey provider's reputation on participation willingness promotion. For survey providers with strong reputations, participants demonstrate higher participation willingness (M=4.73, SD=1.67) than survey providers with a weak reputation (M=4.23, SD=1.74). The mean difference shows significance, t(89)=4.21, p<0.01, Cohen's d=0.30. Thus, the reputation of the survey provider has a positive association with the participation willingness.

Experiment 3

A total of 200 participants (71 women and 129 men) are used in experiment 3. Participants' age ranged from 20 to 29 years for 81 percent of the participants. A Scheffé multiple comparison test reveals a significant difference in the mean of participation willingness across the four combinations of reputation type. Levene's Robust Test for the equality of variances shows that standard deviations in the four situations do not significantly differ [W=0.19, df=(3, 796), p=0.90].

The significance of the main effects and the interaction of the two factors, sponsoring corporation reputation and survey provider reputation, are tested using an analysis of variance. The two main effects, sponsoring corporation reputation [F(1, 199)=30.42, p<0.01] and survey provider reputation [F(1, 199)=12.43, p<0.01], are significant. The interaction effect also shows significance [F(1, 199)=4.28, p=0.04]. Figure 1 illustrates the interaction effect with the vertical bars representing the standard error of the mean.

FIG. 1.

Effects of sponsoring corporation's and survey provider's reputation on participation willingness.

The results presented in Figure 1 show that compared to the combination of weak reputation sponsoring corporation and strong reputation survey provider, the participation willingness significantly increased for the combination of strong reputation sponsoring corporation and weak reputation survey provider. The sponsoring corporation's reputation moderates the effect of the survey provider's reputation on participation willingness. Thus, the combination of a strong reputation sponsoring corporation and a survey provider with a weak reputation is more effective than the combination of a strong reputation survey provider and a weak reputation sponsoring corporation in promoting participation willingness.

Conclusion

Considerable research has been conducted on promoting the willingness of survey respondents to participate in Web surveys. Limited research has been conducted on Web-based surveys investigating the effect of the reputation of a sponsor or provider. These studies typically assume that surveys are conducted by sponsoring corporation-survey providers, and few studies make a distinction between the sponsoring corporation and the survey provider. This study provides a framework for future studies to empirically examine the different effects of the sponsoring corporation's and survey provider's identities (reputations) on participation willingness.

This study provides several important findings. First, the sponsoring corporation's or survey provider's reputation has a positive association with the participation willingness when only the sponsoring corporation's or the survey provider's identity appeared in the survey.

Second, when both the sponsoring corporation's and survey provider's identities appear in a survey, their identities demonstrate different salience on increasing the participation willingness. A significant interaction exists between the reputation levels of the sponsoring corporation and the survey provider. The sponsoring corporation's reputation moderates the effect of the survey provider's reputation on participation willingness, and the sponsoring corporation's identity shows more prominence than that of the survey provider's. This result may help to explain theoretically the previous inconsistent results for the response rate, while the reputation of the survey provider does not change. Moreover, the interaction figure illustrates that the combination of a strong reputation sponsoring corporation and a weak reputation survey provider is more effective than the combination of a strong reputation survey provider and a weak reputation sponsoring corporation in promoting participation willingness.

Third, this research implies that a sponsoring corporation having a weak reputation who contracts with a survey provider having a strong reputation does not ensure the promotion of participation willingness. Only under the condition that the identity of a weak reputation sponsoring corporation stays concealed and the survey provider's identity appears during a survey, participation willingness increases. On the other hand, displaying a sponsoring corporation's identity that has a strong reputation is a recommended strategy to increase participation willingness. A strong reputation sponsoring corporation contracting with a weak reputation survey provider will not significantly reduce participation willingness, if the sponsoring corporation's identity appears in surveys. A strong reputation sponsoring corporation can further increase participation willingness through presenting both the strong reputation survey provider's identity and the sponsoring corporation's identity in the survey.

Practically, this study highlights the importance of a sponsoring corporation selecting a fitted survey provider to obtain the desired effect on participation willingness for a survey. Different combinations of reputation levels for the sponsoring corporation and the survey provider involving identity display strategies will incur distinct results. When the sponsoring corporation has a weak reputation, this study's results suggest that contracting with a strong reputation survey provider might not guarantee a high response rate. To achieve a high level of participation willingness, this study recommends concealing the sponsoring corporation's identity during surveys. In contrast, when a sponsoring corporation has a high reputation, the sponsoring corporation can choose to display or conceal the survey provider's identity, provided that the sponsoring corporation's identity appears in surveys. Either strategy will not impair the participation willingness. The sponsoring corporation can further increase the participation willingness by contracting with a strong reputation survey provider and by revealing both of their identities.

There are several limitations to the research. First, this study used student participants. Second, this study assumes that the sponsoring corporation's and survey provider's reputation affect participation willingness directly. However, it is possible for the direct and indirect effects to coexist in reality. Third, this study focuses solely on corporate research. The results of the present study may apply only to situations in which a corporation uses a third party to conduct research and these findings may not generalize to other research contexts (e.g., academic research).

Acknowledgments

The authors are grateful to LeAnn Kitchens and Chris Dubelaar for their insightful comments that have resulted in an improved study. This work was supported by the NSFC under Grant 71101018 and the Fundamental Research Funds for the Central Universities under Grant ZYGX2010J131.

Disclosure Statement

No competing financial interests exist.