Table 2.

Item-writing “best practices” based on scientific evidence from questionnaire design research.

Pitfall	Survey example(s)	Why it’s a problem	Solution(s)	Survey example(s)	References
Creating a double- barreled item	– How often do you talk to your nurses and administrative staff when you have a problem?	Respondents have trouble answering survey items that contain more than one question (and thus could have more than one answer). In this example, the respondent may talk to his nurses often but talk to administrative staff much less frequently. If this were the case, the respondent would have a difficult time answering the question. Survey items should address one idea at a time.	When you have multiple questions/premises within a given item, either (1) create multiple items for each question that is important or (2) include only the more important question. Be especially wary of conjunctions in your items.	– How often do you talk to your nurses when you have a problem? – How often do you talk to your administrative staff when you have a problem?	Tourangeau et al. 2000; Dillman et al. 2009
Creating a negatively worded item	– In an average week, how many times are you unable to start class on time? – The chief resident should not be responsible for denying admission to patients.	Negatively worded survey items are challenging for respondents to comprehend and answer accurately. Double negatives are particularly problematic and increase measurement error. If a respondent has to say “yes” in order to mean “no” (or “agree” in order to “disagree”), the item is flawed.	Make sure “yes” means yes and “no” means no. This generally means wording items positively.	– In an average week, how many times do you start class on time? – Should the chief resident be responsible for admitting patients?	Dillman et al. 2009
Using statements instead of questions	I am confident I can do well in this course. • Not at all true • A little bit true • Somewhat true • Mostly true • Completely true	A survey represents a conversation between the surveyor and the respondents. To make sense of survey items, respondents rely on “the tacit assumptions that govern the conduct of conversation in everyday life” (Schwarz 1999). Only rarely do people engage in rating statements in their everyday conversations.	Formulate survey items as questions. Questions are more conversational, more straightforward and easier to process mentally. People are more practiced at responding to them.	How confident are you that you can do well in this course? • Not at all confident • Slightly confident • Moderately confident • Quite confident • Extremely confident	Krosnick 1999; Schwarz 1999; Tourangeau et al. 2000; Dillman et al. 2009
Using agreement response anchors	The high cost of health care is the most important issue in America today. • Strongly disagree • Disagree • Neutral • Agree • Strongly agree	Agreement response anchors do not emphasize the construct being measured and are prone to acquiescence (i.e. the tendency to endorse any assertion made in an item, regardless of its content). In addition, agreement response options may encourage respondents to think through their responses less thoroughly while completing the survey.	Use construct-specific response anchors that emphasize the construct of interest. Doing so reduces acquiescence and keeps respondents focused on the construct in question; this results in less measurement error.	How important is the issue of high healthcare costs in America today? • Not at all important • Slightly important • Moderately important • Quite important • Extremely important	Krosnick 1999; Tourangeau et al. 2000; Dillman et al. 2009
Using too few or too many response anchors	How useful was your medical school training in clinical decision making? • Not at all useful • Somewhat useful • Very useful	The number of response anchors influences the reliability of a set of survey items. Using too few response anchors generally reduces reliability. There is, however, a point of diminishing returns beyond which more response anchors do not enhance reliability.	Use five or more response anchors to achieve stable participant responses. In most cases, using more than seven to nine anchors is unlikely to be meaningful to most respondents and will not improve reliability.	How useful was your medical school training in clinical decision making? • Not at all useful • Slightly useful • Moderately useful • Quite useful • Extremely useful	Weng 2004

Adapted with permission from Lippincott Williams and Wilkins/Wolters Kluwer Health: Artino et al. 2011. AM last page: Avoiding five common pitfalls in survey design. Acad Med 86:1327.