Effective questionnaire design: How to use cognitive interviews to refine questionnaire items

Abstract

Background:

Cognitive interviewing is a technique that can be used to improve and refine questionnaire items. We describe the basic methodology of cognitive interviewing and illustrate its utility through our experience using cognitive interviews to refine a questionnaire assessing parental understanding of concepts related to preterm birth.

Methods:

Cognitive interviews were conducted using current best practices. Results were analyzed by the multidisciplinary research team and questionnaire items that were revealed to be problematic were revised.

Results:

Revisions to the questionnaire items were made to improve clarity and to elicit responses that truly reflected the participants understanding of the concept.

Conclusion:

Cognitive interviewing is a useful methodology for improving validity of questionnaire items, we recommend researchers developing new questionnaire items design and complete cognitive interviews to improve their items and increase confidence in study conclusions.

Background

Questionnaires are a relatively low-cost and efficient data collection method and are thus attractive for health research and educational projects. However, writing new questionnaire items can be tricky, and a questionnaire of poor quality can preclude meaningful results. Cognitive interviewing is a formal technique to evaluate whether questionnaire items are understandable to respondents and will be useful for making conclusions [1]. It is a cost-effective approach for improving questionnaire-based research and provides evidence for the content validity of items. Prior research across various disciplines has demonstrated how cognitive interviews can be used to detect issues related to clarity, comprehension, ambiguity, cognitive recall burden, timeframe, missing answer categories, inaccurate instructions, and relevance of questionnaire items [2–4]. Herein we offer an overview of cognitive interviewing methodology, drawing on our experience using cognitive interviews during the development of a questionnaire designed to assess knowledge of concepts related to preterm birth in parents of premature infants.

Methods

We used cognitive interviews to test and improve a knowledge questionnaire meant to assess participants’ understanding of concepts related to preterm birth. In preparation for the study, the objectives were outlined and clarified among team members: to test a questionnaire related to concepts related to preterm birth, with the goal of removing or improving problematic items to ensure understandability of the questionnaire and enhance its validity. Established methods and best practices were followed to draft each item of the initial preterm birth questionnaire [5]. We developed a cognitive interview guide consisting of the questionnaire items to be tested followed by probing questions [1]. Probing questions are designed to assess the participants’ understanding of each item, the mental process used to answer, how they mapped their answer to one of the response options provided, and any judgments or decisions related to responding (e.g. social desirability). Scripted probes for each item ensured standardization of analyses across interviews. Common probes included, “Can you rephrase the question in your own words?” or “How did you decide on that answer?” The research team took care in designing and updating the interview guide to ensure that probes on earlier items did not contaminate participant interpretation of later items and ensured that sufficient time was allocated within the interview for the items of greatest concern to be probed adequately. Probing may be conducted using either the concurrent or retrospective technique [1]. The concurrent technique involves the interviewer using targeted probes immediately after the participant responds to each questionnaire item; whereas, with the retrospective technique, the interviewer asks the respondent probing questions after the full questionnaire has been completed. The concurrent technique was used for this study, probing the participant on how they understood and answered each questionnaire item immediately after they provided their answer. Prior to beginning interviews, approval was obtained from an institutional review board. Informed consent was obtained from participants, and it was emphasized with each participant that they could refuse to answer any question or stop participation at any time.

In preparation for the cognitive interviews, team members participated in training, which included a review of literature related to cognitive interviewing and an overview of best practices in the field, such as those outlined in Willis’ Cognitive interviewing a ‘how-to’ guide [6], and Boeije & Willis’ cognitive interviewing reporting framework (CIRF) [7].

Prior to beginning the study, interviewers practiced role playing cognitive interviews, and held detailed discussion with the project investigators to understand the intent of each questionnaire item. Novice team members were observed by experienced team members and received feedback to improve their technique.

Participants were recruited from a NICU (six participants) and a high-risk OB clinic (four participants) to obtain a sample similar to our population of interest. However, to obtain a sample naïve to the topic of the questionnaire items and responses (knowledge of prematurity), we excluded parents of infants who had been admitted for issues of prematurity. We aimed to reflect the diversity among our target population by interviewing mothers and support-persons, aiming for racial diversity and a range of literacy levels. Because as few as four interviews may be sufficient to identify problematic questions [8] our goal was for each item to be reviewed by at least five parents, at least two of whom had less than a high school degree or less than a ninth grade reading level. This purposeful sampling promotes a heterogenous sample and helps achieve a type of sensitivity and specificity [8].

Participants provided written consent and received an incentive of $50. Each interview lasted approximately one hour and was conducted in person in a quiet and comfortable private space. To encourage uninhibited feedback, interviewers distanced themselves from the investigators by using phrases, as suggested by Willis, such as: “I didn’t write these questions, so don’t worry about hurting my feelings if you criticize them. It’s my job to find out what’s wrong with them” [6]. For each interview, one team member led the interview while an additional team member took notes, which served as the primary source of data for analysis. Four members of the team participated in the interviewing and note take process, one who only interviewed, one who only took notes, and two who did at different times did both. Interviews were recorded but not transcribed. Participants were provided a paper copy of the questionnaire and directed to silently read each item, mark their answer, and then read the item aloud along with their chosen response. The interviewer asked probing questions to elucidate the participant’s comprehension of each item before moving on to the next. The interview notes were compiled in a spreadsheet organized by item, which served as the primary source of data.

The full research team- representing expertise in neonatology, nursing, survey research methods, sociology, and psychology [9], met after each set of five interviews was conducted and took a reparative approach for analysis of the results. Careful inspection of each item and response allowed identification of “dominant trends” (problems that repeatedly emerged), as well as “discoveries” (problems that may have emerged only in one interview but still posed a threat to valid data) [8]. The team collectively decided the best way to improve flawed items to reduce response error. Substantially revised items were tested in additional interviews.

Results

During the cognitive interviews, issues emerged that led to the revision of certain items. For the sake of brevity, results from every individual questionnaire item are not systematically described here, but a selection of questionnaire items that were revealed to be not-well understood or otherwise problematic are described in Table 1. For example, multiple participants interpreted the phrase “at risk” as a fact or expected outcome, rather than a potential outcome. We clarified questions that used the phrase “at risk” by adding a concrete comparison. For example, the questionnaire item: “A baby born before 25 weeks of pregnancy is at risk of having problems learning due to prematurity. True or False?” was revised to: “Compared to a baby born after 37 weeks, is a baby born before 25 weeks of pregnancy more likely to have problems learning?”

Table 1:

Original Questionnaire Item	Cognitive Interview Probe(s)	Participant Response	Issues Identified	Correction	Revised Questionnaire Item
A baby born before 25 weeks of pregnancy is at risk of having problems learning due to prematurity. True or False.	How did you come up with your answer?	Answered “False”, stating that learning problems are “case by case.”	“At risk” was interpreted as equivalent to “will happen” rather than “has increased potential to happen.”	Questions using “at risk” were revised to include a concrete comparison group.	Compared to a baby born after 37 weeks, is a baby born before 25 weeks of pregnancy more likely to have problems learning?
Premature baby boys have a better chance of being healthy than premature baby girls. True or false.	Can you reword this question in your own words? (If problems): Is there a better way to ask this question?	Although surprised by this question, it was generally understood. However, participant did indicate it seemed too much like a statement and would prefer it to be worded like a question.	Preference of participants for survey items to be in the form of a yes/no question rather than a true/false statement.	Modify from statement to question format.	Do premature boys have a better chance of being healthy than premature girls? Yes or no.
Most women who go into preterm labor will deliver within the next week. True or false.	How did you come up with your answer?	Patient responded correctly: false. However her thought process was “I hope not… I’d hope it would be within the next day or two.”	Although participant answered the question correctly, it did not reflect her understanding of the topic.	Revised item to remove misleading language.	Most women who go into preterm labor will deliver in the next day or two. True or false.
If a premature baby is healthy when they leave the NICU, then they are not at risk of health problems later in life. True or false.	What is this question asking you? What kinds of things come to mind when you see the phrase “health problems later in life”? How did you come up with your answer?	Almost all participants answered the question correctly with “false,” but cited reasons such as “any baby is at risk of problems later in life,” and “there are people who are born healthy who still have issues later in life.”	Answered correctly but for the wrong reasons.	Despite attempts to revise, question was ultimately removed.
Antibiotics can affect hearing in premature babies. True or false.	Can you rephrase this question in your own words? What does “affect hearing” mean to you?	Participant was unsure if “affect hearing” meant help or harm hearing.	Ambiguous word choice caused confusion.	Revised question to make direction of “affect” clear.	Antibiotics can damage hearing in premature babies. True or false.

While many of our original items were true/false statements, through the cognitive interviews we discovered that some participants found this presentation unhelpful, even confusing. Thus, the team revised some questionnaire items to a traditional question format with a yes/no answer, instead presenting a “true or false” statement that the participant was asked to evaluate.

During the cognitive interviews, there were multiple instances when the participant gave the right answer, but probing revealed it was for the wrong reason. For example, when presented with the true or false statement: “Most women who go into preterm labor will deliver within the next week”, the correct answer would be false (only one in ten delivers within a week). However, one participant interpreted this to mean preterm labor could last a week and responded: “I hope not. I’d say false. I’d hope it would be within the next day or two.” While she provided the correct answer (false), probing revealed that meaningful understanding of the topic had not been achieved. To clarify, the question was revised: “Most women who go into preterm labor will deliver within the next day or two”. The answer remains false; however, the confusion that the preterm labor lasts for one week was resolved.

Discussion

The overarching goal in our research is for parents to have critical information about preterm birth in order to participate in shared decision making, so understanding their underlying thought process for selecting a response was essential. For example, learning that some participants may have selected the right answer for the wrong reason allowed the research team to recognize the participant may not have fully understood the concept, thus undermining their ability to leverage the information when making healthcare decisions for their child. Cognitive interviews highlighted problems with our questionnaire items and allowed us to revise or eliminate unclear items and gain confidence in using them as an outcome measure. The main limitation of our study was that participants came from a single institution and may not represent the whole population, though the sample size was appropriate for a qualitative study, [6] and it is generally assumed that even a modest sample size will expose the most critical problems [8].

For studies collecting data through questionnaires, the validity of the questionnaire items is crucial. We recommend that researchers (including student researchers) developing new questionnaire items utilize available resources [1,7,8] to design and complete cognitive interviews, thus improving their items and increasing confidence in study conclusions.

Abbreviations:

NICU

Neonatal intensive care unit

OB

Obstetric

CIRF

Cognitive interviewing reporting framework