Statistical Application and Cost Saving in a Dental Survey

Abstract

Objective

To effectively achieve a robust survey response rate in a timely manner, an alternative approach to survey distribution, informed by statistical modeling, was applied to efficiently and cost-effectively achieve the targeted rate of return.

Design

A prospective environmental scan surveying adoption of health information technology utilization within their practices was undertaken in a national pool of dental professionals (N=8000) using an alternative method of sampling. The piloted approach to rate of cohort sampling targeted a response rate of 400 completed surveys from among randomly targeted eligible providers who were contacted using replicated subsampling leveraging mailed surveys.

Methods

Two replicated subsample mailings (n=1000 surveys/mailings) were undertaken to project the true response rate and estimate the total number of surveys required to achieve the final target. Cost effectiveness and non-response bias analyses were performed.

Results

The final mailing required approximately 24% fewer mailings compared to targeting of the entire cohort, with a final survey capture exceeding the expected target. An estimated $5000 in cost savings was projected by applying the alternative approach. Non-response analyses found no evidence of bias relative to demographics, practice demographics, or topically-related survey questions.

Conclusion

The outcome of this pilot study suggests that this approach to survey studies will accomplish targeted enrollment in a cost effective manner. Future studies are needed to validate this approach in the context of other survey studies.

Historically, assessments of response rates associated with surveys of health-related professionals have reported comparatively lower response rates, eliciting a body of research on how to improve response rates among these professionals.¹ Investigators often have no prior knowledge to inform projection of a response rate before a survey mailing. The usual approach for conducting a survey is to pre-determine the achievable “targeted survey sample size” (‘N’), which is contingent on best estimates of rate of return, and then send out ‘N’ surveys, regardless of cost. There are several potential issues with this ‘all-or-none’ approach. First, the investigators have little prior knowledge to inform estimation of the actual survey return rate. Second, resources may be invested unnecessarily in too many survey mailings if the rate of return exceeds the projected response rate. Third, ultimate decisions surrounding extensiveness of sampling may be bounded by resources available to conduct the survey research, potentially predisposing study outcomes to bias.

Our research team recently undertook an environmental scan of dentists across the continental USA to assess their utilization of Health Information Technology (HIT) utilizing a survey study approach applying best practices for conduct of survey research (manuscript under review). In order to effectively achieve a targeted response rate at lowest cost in a timely manner across a representative sample of dental providers, a statistical modeling approach was piloted to project the number of mailings required to achieve the targeted sample size most cost effectively. Specifically, the alternative approach included application of binomial proportion confidence interval statistics. Informed by two initial replicated subsample surveys conducted on a random subset of the total population, statistical modeling was applied to project the total number of completed surveys required from the residual population to achieve the targeted response rate.

Methods

Estimating Number of Mailings Required to Achieve Targeted Sample Size

The goal of this parallel sub-study was to define an algorithm to project the true number of surveys actually required to achieve a projected survey return rate and estimate the cost-saving associated with the approach. Specifically, the approach applied random replicated subsampling² and statistical modeling, applying binomial proportion analysis bounded by a confidence interval, to project the number of residual surveys to be mailed to achieve the targeted sample size, informed by the subsample response rates achieved. Further, cost savings associated with applying this approach were modeled. Modeling was undertaken as follows:

We assumed that ‘n’ is the targeted number of survey returns, ‘R’, the projected conservative survey return rate, and ‘N’ is the required number of total surveys, n=N*R, where 0<R<1, and 0<n<N. We further assumed that ‘k’, which represents the total number of random replicate subsamples and mailing, is set by the investigator. In general, k is much smaller than N, and the information obtained from the first k–1 survey mailings will be used to determine the required number of additional surveys for achieving the targeted number of surveys, n. We designated N_i to be the total number of surveys that will be sent out on the i^th mailing, and n_i to be the number of return surveys for the i^th mailing, and therefore, R_i = n_i/N_i, is the survey return rate for the i^th mailing, where i = 1, 2, …, k–1. If n_sum= n₁+n₂+…+n_i ≥ n, then the study is complete (that is, we have reached the n targeted number of survey returns at the i^th mailing), and we calculate the cost-saving as follows: N_sum= N₁+N₂+…+N_i, for I = 1, 2, …, k–1. C_s = (N–N_sum)*(M_c+P_c), where C_s denotes the total cost saving, M_c denotes the estimated material cost (e.g., envelope, stamp) per unit of survey, and P_c denotes the estimated personnel cost per unit of survey. If n_sum = n₁+n₂+…+n_i < n, that is, we have not reached the n targeted number of survey returns after the ith survey mailing, then we calculate the required final number of survey mailings as follows: N_sum= N₁+N₂ +…+N_i <N, for i = 1, 2, …, k–1. R_a, the average survey return rate from the i mailings (i = 1, 2,…, k–1), is R_a = (R₁+R₂+…+R_i)/i. LCLR_a = R_a–[R_a*(1–R_a)/min(N₁, N₂,…, N_i)]^1/2*1.96, where min(N₁, N₂,…, N_i) is the minimum value of N₁, N₂,…, N_i, for i = 1, 2,…, k–1. Lastly, N_k, the k^th and the last number of survey mailings, will be: N_k = (n–n_sum)/LCLR_a (1), where LCLR_a^3,4 represents the estimated conservative 95% lower confidence limit for the average survey return rate based on i mailings as a whole.

Assuming a response rate of R = 5% and a targeted n=400 for our study, the total projected number of surveys required to be mailed out was N = 400/5% = 8000 using traditional approaches. Based on cost and time constraints, the alternative approach described in this paper was applied and the frequency of mailings, k, was set to 3, applying a random selection of a targeted subset of 1000 potential participants (N₁ = N₂ = 1000) among the 8000 total available potential participants for each of the first two mailings.

The return rate for our first set of mailings was R₁ = n₁/N₁ = 84/1000 = 8.4%, and R₂ = n₂/N₂ = 69/1000 = 6.9% for our second set of mailings. Therefore, the average return rate for the first two (=k–1 = 3–1 = 2) mailings turned out to be R_a = (R₁+R₂)/2 = (8.4%+6.9%)/2 = 7.6%.

The estimated conservative 95% lower confidence limit of R_a was: LCLR_a = R_a–[R_a*(1–R_a)/min(N₁, N₂)]^1/2*1.96 = 7.6%−[7.6%*(1–7.6%)/min(1000, 1000)]^1/2*1.96 = 6%. The total responses from the first two sets of mailings were as follows: n_sum = n₁+n₂ = 84+69 = 153. Therefore, the additional required responses to achieve target was projected to be:

n–n_sum = 400–153 = 247. To arrive at the projected number of mailings that would still be required to achieve the residual target based on response to the previous mailings, we applied equation 1: N_k = N₃ = (n–nsum)/LCLR_a = 247/6% = 4117 mailings (or 4120 mailings, rounding to the nearest tens).

For the scenario in which, with a fixed N, it is not possible to reach the targeted n after k–1 mailings, two options exist: (1) stop further mailing after k – 1 mailings and calculate C_s via equation 2; or (2) increase N to N* so that n becomes achievable by the k^th mailing. In this case, it can be argued that C_s = N*(M_c + P_c), because the entire project has been saved from failure that would be attributable to not reaching the targeted ‘n’.

Cost Analysis

To estimate cost-saving, we modeled costs associated with materials, mailings, and labor. We know that the number for the three sets of mailings is N_sum = N₁+N₂+N₃=1000+1000+4120 = 6120, the cost of printing material and postage cost for each mailing was approximately M_c = $2.13, and the personnel cost (preparing mailing) was approximately P_c = $.50/per survey mailed. Thus, we can calculate the cost-saving (C_s) figure for this scenario as follows: C_s = (N–N_sum)*(M_c+P_c) (2) where: N_sum = N₁+N₂+…+N_k, and C_s, M_c and P_c are defined as before. Finally non-response bias was estimated using wave analysis as a surrogate measure.

Non-Response Bias Analysis

Responses of the initially-targeted replicate subsample were compared to responses of all submitted surveys, to randomly selected questions across the questionnaire including demographic and practice demographic questions, and topical questions related to HIT adoption and knowledgeability surrounding health information exchange program.

Results

In addition to the two random replicate subsamples of 2000 surveys sent out in two separate earlier mailings in order to more accurately gauge response rate, the modeled calculations projected that a total of 4120 additional survey mailings was required to achieve our targeted return rate of “400” survey completed by the end of our designated survey recruitment period. This projected number of additional survey mailings (N = 4120) required to achieve the goal of 400 captured surveys was 23.5% lower than the original estimated N of 8000, representing the entire cohort of eligible dentists. The final number of surveys received (n = 484) exceeded the projected enrollment by 18%. Based on equation (2), the estimated total cost savings associated with determining a more accurate response rate was:

$${\mathrm{C}}_{\mathrm{s}}=\left(\mathrm{N}-{\mathrm{N}}_{\text{sum}}\right)*\left({\mathrm{M}}_{\mathrm{c}}+{\mathrm{P}}_{\mathrm{c}}\right)=\left(8000-6120\right)*\left(\$2.13+\$.50\right)=\$4,944.40$$

To assess potential for non-response bias, a comparison of responses to randomly-selected questions across the questionnaire including demographic and practice demographic questions, and topical questions related to HIT adoption and knowledgeability surrounding health information exchange program, revealed no significant difference between the subset of respondents targeted by the first 1000 questionnaires (n = 84) and overall responses of the entire cohort of respondents.

Discussion

High variability in response rates to survey-based research studies conducted each year in the United States among biomedical professionals has been noted, with relatively low response rates ranging from < 20% to 32% reported in studies by Danhauer et al⁵ and Grava-Gubins & Scott⁶ to as high as > 80% as exemplified in response rates of 84.2%, 87% and 94.5%, respectively, reported by Karlsen et al,⁷ Munoz et al,⁸ and Thompson et al.⁹ Even when best practices in survey research were applied, more modest response rates have historically been associated with survey studies targeting medical/dental professionals compared to other targeted populations. A study monitoring response rates of physicians across a 10-year observational period between 1985 and 1995 reported response rates ranging from 11% to 39%.¹⁰ Another study by Asch et al¹¹ noted that response rates to surveys published in medical trade journals in 1991 achieved response rates 13% lower than those achieved among non-provider groups.

In response to suboptimal response rates reported in the literature, investigators have researched and developed sets of best practices to maximize return rates when conducting surveys including mixed mode surveys, and the most prominently-applied Dillman Total Design Survey Methods.¹² Notably, however, application of these approaches has met with mixed success.¹³ For example, despite application of strategic approaches to enhance response to the Canadian National Physician Survey, the response rate in 2007 was lower than the prior survey conducted in 2004.⁶ By contrast, in an earlier study, Hoddinott & Bass¹⁴ reported achievement of a return rate of nearly 93% in an earlier survey conducted among a cohort of Canadian practitioners. Similarly, Thorpe et al¹⁵ reported response rates of approximately 74% in three consecutive years among physicians applying a modified Dillman approach, compared to 48% achieved in a fourth survey that did not implement gift certificate distribution and use of recorded/registered mail delivery, the latter being a component of the Dillman approach.¹⁵ A 2014 systematic review of randomized controlled trials (RCTs), cluster RCTs, and factorial trials by Pit SW et al¹⁶ reported that optimization of survey response rates among physicians was most strongly influenced by provision of incentives, pre contact, use of registered mail and personalization of the survey packet, mailed or multi-modal survey.¹⁶ The 2007 Cochrane Systematic Review by Edwards et al¹⁷ reported the odds ratios of improving the likelihood of return of a completed survey through application of components of the Dillman approach based on analyses of 372 randomized controlled trials.

Apropos to the current study, Funkhouser et al¹⁸ recently reported achievement of an 87% response rate in a survey of 1,716 eligible dentists affiliated with the National Dental Practice-Based Research Network (NDPBRN) through application of rigorous recruitment using a mixed mode strategy. The authors noted that the robust participation rate was possibly attributable to some degree to the targeted dental group who projected a pro-active interest in research by virtue of their membership in the NDPBRN. By contrast, a 2007 survey of dentists and physicians regarding antibiotic prophylaxis in the dental setting resulted in statistically significant differences in response rates of 32% and 17% among dentists and physicians, respectively, applying a traditional approach of distributing the one-page survey by mail with no follow-up.¹⁹ Differences in responses were potentially attributable to the greater perceived relevance and interest of the survey to dental practitioners. Notably, a study among Japanese dentists applying the Dillman Total Design Method and multi-modal survey delivery only achieved a 47% response rate in their population, compared to some surveys among dentists in the United States that had employed the Dillman approach to achieve comparably higher response rates, suggesting that portability may not apply in all settings.²⁰

A concern with respect to lower response rates is the increased risk of bias. While robust response rates are generally thought to provide a more accurate reflection of the larger population and minimize potential for bias, no definitive ‘gold standard’ response rate has been advanced. Notably, studies suggest that surveys achieving low response rates may still be valid if non response can be effectively evaluated.^21–24 However, a systematic review of survey literature reports low rates of non-response analyses among studies.²⁵ Particularly, Parashos et al²⁶ reported that despite no detection of non-response bias among early and late survey responders with respect to demographics and practice demographics, a bias on response to the topical question among early and late responders was detected, despite achieving a robust sample size of 87%. Taken together, these studies emphasize that demonstration of non-response bias may be more critical than the actual return response rate.

Collectively, these findings prompted the study team to employ the strategy of projecting a targeted response rate posited to provide a representative sampling of the provider population for a recent survey-based environmental scan surrounding HIT adoption by dental practices. The study design also incorporated current best practices for survey research.¹² Proceeding on the premise that mailed survey response rates can still be representative if non-response bias is assessed,²³ the current study proceeded with the approach of achieving a targeted fixed return rate, complemented by replicative subsampling to project the residual number of surveys that needed to be sent informed by the response rate to the two random mailings to a subset of eligible recipients. The residual number of mailings, which was statistically modeled, represented a subset of the remaining eligible cohort and was associated with considerable cost saving due to not having to mail surveys to the entire eligible cohort.

Specifically, we received a total of 484 survey responses, exceeding our target for (n = 400) final returns, thus demonstrating the success of this approach. The final response rate for our survey study was 7.9% (= 484/6120). Our study also illustrated that our proposed method resulted in a 23.5% (= [8000–6120]/8000) overall cost saving to our general survey operating budget, which leveraged nearly $5000 in savings for our survey study. Given that 6,120 were randomly selected from among 8,000 potential participants, demonstration that 484 responses were representative and not associated with bias was undertaken applying non response analysis. Wave analysis²⁷ as the surrogate approach for analyzing non-response was applied, comparing responses of participants across both demographic and topical domains of the survey differentiated by temporality of the response. No statistically significant difference was detected among the survey responses between surveys distributed in the initial mailing and those distributed in subsequent mailing, reflecting no detection of bias.

A number of binomial confidence interval methods have been well established such as Normal approximation interval,⁴ Wilson score interval,²⁸ Jeffrey’s interval,²⁹ Clopper-Pearson interval,³⁰ Agresti-Coull interval,³¹ and Arcsine transformation.³² However, all these are valid only for one sample (n), which cannot be applied to covering the scenario of our survey mailings that include more than one sample (n₁, n₂, etc). The only comparative existing method is Normal approximation interval, but its computation becomes somewhat unwieldy for two, three, or more samples, compared to our simple approach. To the best of our knowledge, the application of binomial proportion analyses bounded by confidence intervals as described in the present study to cost effectively achieve a targeted response rate has not been previously reported in survey literature, and our team is the first group to implement the proposed method, with satisfactory outcomes based on cost-effective achievement of enrollment goals as demonstrated in the Results section. Although our survey data collection only showed the scenario with k=3 (ie, 3 survey mailings), we predict that this methodological approach has portability to projects conducting survey research requiring more than three mailings. The proposed method is further predicted to cost-effectively achieve desired enrollment targets, as supported by results of the theoretically-derived statistical modeling demonstrated in the present study. We further posit that this approach has applicability to any type of mail survey.

Study Limitations

The proposed survey approach has only been piloted in the context of one dental survey study. The proposed method represents a conservative approach, and further methodological refinement may be possible to improve its applicability and cost effectiveness in the context of other survey-based studies. While non-response bias testing did not detect a selection bias among the respondents in the current study, while unlikely, it cannot be completely ruled out that the total sample of the respondents in our study was not systematically biased, since the sample surveyed represented < 10% of the eligible population. In addition, the actual cost savings may vary from one mail survey study to another. For example, in our institution, there is no bulk mailing postage rate, so all letters cost the same to send, irrespective of the number of letters mailed. Thus, cost savings may differ from those where institutions offer bulk mailing rates. Further, the present study did not factor in administrative costs that may be associated with multiple mailings versus a one-time mailing, but we proceeded with the assumption that the collective mailing of what amounted to fewer questionnaires overall would likely offset nominal administrative costs associated with multiple mailings. Moreover, our approach cannot be applied when the estimated confidence lower bound is negative. Lastly, while the issue of selection bias can arise when the sample survey response rate is small, such as < 10%, our analyses suggested that the sample surveyed was indeed representative of the larger sample.

Conclusion

In summary, we have proposed an easy to adopt, cost-effective approach to survey dissemination. Further testing of generalizability of this approach in various applications is warranted to further test the range of its applicability and effectiveness.