The Importance of Survey Research Standards

Abstract

Every discipline within fields of research has instituted guidelines and templates for research endeavors and subsequent publications of findings, with the ultimate result being an increase in quality and acceptance by researchers within and across disciplines. These significant efforts are by nature ongoing, as well they should. These enhancements and guideline developments have been instituted in basic science disciplines, clinical pharmacy, and pharmacy administration relevant and related to subsequent scholarly publication of research findings. Specific research endeavors have included bench research, clinical trials and randomized clinical trials, meta analyses, outcomes research, and large scale database analyses. A similar need for quality and standardization also exists for survey research and scholarship. The purpose of this paper is to clarify why this is important and crucial for the Journal and our academy.

INTRODUCTION

In the Research Standards section of Instructions to Authors (http://archive.ajpe.org/instructions.asp), the Journal provides guidelines for authors to consider when preparing a manuscript for submission to the Journal. These standards are important for a number of reasons, and may be seen as unique and groundbreaking with regard to other academic health professions journals. This paper is intended to add clarity to this sometimes controversial set of Journal guidelines.

Whether referring to sampling texts such as Cochran’s Sampling Techniques, 3rd edition,¹ or Kish’s Survey Sampling,² or using guidelines or tables generated based on these classics as found in Krejcie and Morgan,³ Salant and Dillman,⁴ Bartlett and colleagues,⁵ and Dillman,⁶ the researcher will find that small populations require a high number of data elements (ie, high response rates) to confidently generalize results because of the potential for sampling error. The recommended minimum sample size for a study depends upon desired confidence level (typically 95%) and how varied the population is with respect to the variable(s) of interest.

Using the conservative approach of a 50/50 split (in other words, an equal chance of one response versus another) on a dichotomous variable of interest at the conventional 95% confidence level for a population of 100, we would need a sample of 80 to ensure a sampling error of no more than +/- 5% at the 95% confidence level. For a population of 100, if a response rate of 50% was achieved for an item with a simple yes/no answer (eg, “Do you have a full-time biostatistician employed by the college?”) and responses were evenly split (50% yes and 50% no), it would not be prudent to extrapolate those findings to the overarching population (100) because the range of possible true percentages would be 25%-75% (that is, all, some, or none of the 50 nonrespondents could have a biostatistician at their college.)^{7 (p55)}

For a variable with a smaller standard deviation in response to a survey item, say an 80/20 split (eg, 80% agree, 20% disagree), a sample size of only 71 (rather than 80) would be required to maintain the same precision as in the previous example, ie, a 95% confidence level. However, according to Salant and Dillman,^{4 (p55)} “unless we know the split ahead of time, it is best to be conservative and use 50/50.” Continuous data sets may not require as many data points, however, “if a categorical variable will play a primary role in data analyses…the categorical sample size formulas should be used.”^5(p46) To estimate the sample size required for a continuous variable would necessitate a measure of variability in the population, which may not be easily discerned, thus “the sample size for the proportion is frequently preferred.”^8(p4) As well, “the effect of nonresponse on one variable can be very different than for others in the same survey.”^{7 (p54)}

Others have simply called for a census in small populations, again necessitating high response rates. ^8,9 These considerations supported the rationale for the expectations set forth in the Viewpoint by Fincham.¹⁰

There are 129 doctor of pharmacy degree programs in academic pharmacy in 1 of 3 classifications of accreditation: 109 full accreditation, 15 candidates, and 5 precandidates.¹¹ The recommended sample size for N=129 at +/- 5% sampling error and 95% confidence level is 97, or a 75% response rate for a 50/50 split. Modeling on a variable with an 80/20 split (ie, less variability in the population) would result in a recommended sample size of 85 or a 66% response rate. Because of the increase in the number of colleges and schools of pharmacy in the United States, the Journal will now accept a 70% response rate threshold for those survey projects collecting data on multiple variable types with the intent of generalizing results to the entire population.

The paper by Draugalis and Plaza¹² provides several examples of the importance of striving for a census and how much confidence readers would have in a published study with a data set with less than optimal response rates, including the annual AACP Faculty Salary Survey. As an example of the potential effects of nonresponse on specific variables in a study, consider the following from a published study on career planning and preparation strategies of pharmacy deans.¹³ The subjects were 53 “new” deans with less than 5 years’ experience and 40 “experienced” deans previously in the database with greater than 5 years’ experience, for a cohort of 93 sitting permanent deans (ie, acting and interim deans were excluded) in 2009. Descriptive findings were presented for the total cohort as well as for separate groups on a number of variables when contrasts were desired. “Newly named deans spent an average of 17.1 +/- 8.7 years in the professoriate prior to assuming their first deanship, compared with established deans who had spent an average of 19.0 +/- 5.1 years (p = 0.006).” If just 3 of the new dean respondents with no or few years in the professoriate had not participated in the study, the mean would have increased to 18.1, the comparison would not have been significant, and an important finding would have been missed. In the career path ladder variable, 9 of the 53 new deans fell in the nontraditional category. If any number of these subjects had actually been nonrespondents, and the closer to actually all 9 of them not participating, this would have skewed descriptive findings and obscured longitudinal comparisons.

High response rates to a research survey do not ensure the validity of the findings as there are other potential sources of error to consider. While attaining a high response rate is a necessary first step, it is not sufficient in and of itself. The specific research question determines the acceptable research methods. For example, in some inquiries, a survey of all colleges and schools of pharmacy may not be necessary or desirable. Depending on the research question, interviews or focus groups may be useful, but the results cannot be generalized to all institutions. Some projects may be intended to gather information only from certain types of institutions, such as private entities, or programs affiliated with a health sciences center. A demonstration project with descriptive findings may be useful to others and in a sense, the argument would be for a methodological development, with the method being generalizable and useful to others, but not the specific institutional findings pertinent to their research. Also, the accepted tools of modeling and decision analytic methods may be appropriate alternatives.

IMPORTANCE OF RESEARCH GUIDELINES AND STANDARDS

In several other research arenas, standards for research methods have been proposed, implemented, and well accepted. Other journals have set standards for research and publications appearing in such. In the 1990s, an international collaboration set in motion a process whereby research standards were developed to enhance the quality and validity of results from clinical trials. A thorough scrutiny of refereed journals accessed through MEDLINE, Embase, Cochrane Central, and associated reference lists was accomplished, and then experts determined the CONSORT checklist, which was subsequently proven to improve the methodology, quality, and external validity aspects of reports of randomized clinical trials.^14,15

Similarly a checklist has been published for qualitative research in hopes of promoting explicit, comprehensive reporting of such research.¹⁶ A Canadian group has proposed developing a survey reporting guideline for health research beginning in 2013 (David Moher, Director, Evidence-based Practice Centre, University of Ottawa, Canada, personal communication, May 17, 2012).

The EQUATOR network (the resource center for good reporting of health research studies) also has been developed to address and make recommendations dealing with the “growing evidence demonstrating widespread deficiencies in the reporting of health research studies.”¹⁷ The EQUATOR Web site provides a list of collected tools and guidelines available for assessing health research issues (www.equator-network.org).

Poor reporting guidelines lead to subsequent deficient outcome segments in written summaries of research. Bennett and colleagues have summarized this problem as follows: “There is limited guidance and no consensus regarding the optimal reporting of survey research. As in other areas of research poor reporting compromises both transparency and reliability, which are fundamental tenets of research.”^{18 (p.8)}

In addressing their concerns over established response rates, Mészáros and colleagues¹⁹ point to the Journal of Dental Education and Academic Medicine as similar publications to the Journal that do not specify response rate criteria. Actually, the issue of response rates has been addressed repeatedly and specifically in these journals. As early as 1983, Creswell and Kuster²⁰ writing in the Journal of Dental Education noted that at that juncture, 40% of papers published over the previous 5 years were survey studies. Thirty years ago, they called for increased diligence in assessing appropriate sample sizes, adequate attention paid to survey response rates, and greater effort in improving the quality of survey-related research in the Journal of Dental Education.

In 2009, in an excellent analysis of survey research issues in the Journal of Dental Education, Chambers and Licari suggest that: “Evidence that is not grounded in theory is just data. There is a natural pull on the authors of surveys to interpret their findings as supporting policies or positions they favor.”^21(p288) The authors also speak to the importance of adequate response rates: “…that the precision of any claim based on a survey is strongly affected by sample size.” ^21(p294) The authors point to sample saturation as a technique to reduce the impact of bias in surveys. This technique directly addresses the response rate issue by noting that the larger the sample size and the higher the response rate, the more accuracy can be attributed to the study results. A built in assumption is that even unknown missing data adversely affect the conclusions of the analyses. Subsequently, even contrary results that may have potentially come from the nonrespondents would result in a less likely scenario. In effect, the results would be different from what was obtained from the analyses of the data in hand.

Response rates matter a great deal, and this point has been made in the Journal of Dental Education over a 30-year period. The issue is not that the Journal of Dental Education has chosen not to develop standards for survey research papers, but rather that the American Journal of Pharmaceutical Education has taken a leadership role in this regard.

Although it is true that Academic Medicine does not explicitly list an acceptable response rate, the October 2011 issue provided summary guidance for survey research published in their journal.²² In this excellent summary of good research practices relative to survey design and reporting, 5 references are listed.^23-27 These seminal references provide explicit information regarding sampling, research design, response rates and associated problems with biases, and acceptability indices in other components of survey research. In one of these “gold standard” references, Krosnick notes that: “It is important to recognize the inherent limitations of nonprobability sampling methods and to draw conclusions about populations or differences between populations tentatively when nonprobability sampling methods are used.”^25(p541) This point becomes even more significant when low response rates are achieved in nonprobability samples.

GUIDELINES AND STANDARDS AS A QUALITY CONTROL MECHANISM

Setting standards and suggesting guidelines are in no way a move on the part of the Journal editors to stifle research or unfairly limit the reporting of research findings; nor are they intended in any manner to arbitrarily curtail creativity. Many fine survey research papers are published in the Journal and contribute to the academy. There are simply no published studies that have pointed out the negative impact of such standard-setting processes on the research endeavors related to clinical, health services research, or sociological research.