Abstract
Objectives
To determine if respondents differed by their demography (age group, race or ethnicity, and insurance status) in their tendency to correctly answer knowledge-based questions when they were in an agree-disagree instead of a multiple-choice format.
Methods
Women aged 18–55 years seeking treatment in the emergency department completed surveys of preventive health knowledge on three cancer screening tests (Pap smears, breast self-examinations, and mammograms) and two contraceptive measures (birth control pills and emergency contraception) that contained paired versions of questions in two different formats (agree-disagree format and multiple-choice format). Linear and ordinal regressions were used to determine demographic correlates of correctly answering the agree-disagree questions more often than the corresponding multiple-choice questions.
Results
Of the 570 women included in this analysis, 64.6% were younger than 35 years, 62.1% were white, and 54.8% had private health care insurance. Older, white, and privately insured women demonstrated greater comprehension of all topics. Younger women, nonwhite women, and women without private health care insurance were more likely to respond to items correctly when they were in an agree-disagree format compared with a multiple-choice format.
Conclusions
This study demonstrated that survey responses are influenced by the format of the questions, particularly in certain demographic groups. Policy makers and researchers might draw false conclusions about the baseline knowledge and need for education of patients, especially in these populations. The use of agree-disagree format questions in preventive health knowledge surveys should be avoided whenever possible.
Keywords: emergency medicine, questionnaires, women’s health, preventive health services, survey methods, psychometrics, acquiescence bias
The Preventive Health Task Force of the Society for Academic Emergency Medicine and other groups advocate for the implementation of selected preventive health screening procedures in emergency departments (EDs).1,2 Patient questionnaires are frequently used to assess knowledge about and need for these procedures. Policy decisions, such as the selection of interventions to create opportunities to provide these procedures in the ED, are partially based on presumptions about the quality of information that is obtained. Flaws, inadequacies, and limitations of question design affect the accuracy of participant responses, and thus the interpretation and understanding of respondent knowledge on the topics studied. Incorrect information impedes effective intervention design.
Two common question formats utilized in patient surveys are the agree-disagree and the multiple-choice formats. Both formats have inherent problems that limit their effectiveness in evaluating respondent comprehension of the concepts being assessed. Agree-disagree questions restrict responses to a narrow range of extremes (agree vs. disagree) that do not allow respondents to fully express their beliefs, knowledge, or opinions; moreover, respondents have a tendency to choose “agree” when they do not fully understand the question.3,4 For multiple-choice question formats, limitations include the propensity to encourage guessing through selection of any response, the tendency of respondents to select either the first or last response options dependent on question length and complexity, and the inclination of respondents to select an incorrect response when the response option has part of the correct answer embedded in the response.3,4 Despite the limitations of multiple-choice format questions, they have a number of advantages. Multiple-choice format questions tend to be more reliable than true-false and open-ended response questions, allow for greater items of measurement within a single question, tend to be better predictors of performance than open-ended response questions, and can force the examinee to discriminate among alternatives to show degree of understanding of concepts.5–8
Acquiescence bias (also known as agreeing-response bias, agreement bias, or “yea-saying”) is the propensity for participants to agree with a statement or assertion presented to them regardless of its contents.9,10 This type of bias is believed to be an important limitation of agree-disagree and other Likert-scale format questions. Several study investigators have observed that the level of acquiescence bias in surveys varies by demographic group. Acquiescence bias in surveys has been shown to be more prominent among persons of lower socioeconomic class,11,12 nonwhite persons,13–17 those with fewer years of formal education,18,19 older persons,12 and among some cultures.10,12 Most of these studies use agree-disagree or other Likert scales to examine attitude and opinions and utilization of services instead of knowledge-based assessment. Some studies offer conflicting conclusions on the importance of demographic characteristics, and investigators debate whether or not acquiescence bias truly affects interpretation of survey results.9,20,21 It is at least likely that the role of acquiescence bias varies by type of survey, context of the survey, nature of the question, response format, and form of administration of the survey.
We were interested in examining whether female ED patients were more likely to respond correctly to questions in a preventive health knowledge survey relevant to women when they were in an agree-disagree format instead of a multiple-choice format. The survey included an assessment of participant knowledge on three cancer screening (Pap smears, breast self-examinations [BSEs], and mammograms) and two contraceptive (birth control pills [BCPs] and emergency contraception [EC]) measures. The survey evaluated participant knowledge on these topics using paired versions of questions in two formats: the agree-disagree format and the multiple-choice format. We also investigated if patient demography (age group, race/ethnicity, and insurance status) was associated with a tendency for participants to correctly respond to knowledge-based questions when they were in an agree-disagree format instead of a multiple-choice format. We hypothesized that if an agree-disagree formatted survey was relied on to obtain participant knowledge about the preventive health measures, respondent knowledge would be overestimated, that is, acquiescence bias would be a limiting factor in assessing knowledge when an agree-disagree response format was used. Further, important differences in participant knowledge according to patient demography would be masked.
METHODS
Study Design and Population
Female ED patients 18–55 years of age were surveyed using a self-administered, written, anonymous, multiple-choice and agree-disagree questionnaire developed by the study authors. The hospital institutional review board deemed the study exempt from informed consent.
The women surveyed were awaiting medical care at an urban, northeastern U.S., adult, academic, tertiary referral center ED. Women 18–55 years of age were chosen for this sample because older women have different health concerns (e.g., menopause) and are less likely to use the contraceptive measures addressed in our survey. Women were excluded from participating if they could not read or write in English; were being evaluated in the critical care, psychiatric section, or substance abuse holding areas of the ED; were not awake; or could not complete the questionnaire because of a physical or medical impairment.
Survey Content and Administration
The survey included three demographic (age group, racial/ethnic group, and insurance status) and 13 preventive health knowledge questions (Table 1). The knowledge-based questions comprised five subject areas: three on cancer screening (Pap smears, BSEs, and mammograms) and two on contraceptive knowledge (BCPs and EC). The knowledge-based portion of the survey contained three paired questions on each topic, except for EC, which only had one paired question. There was a multiple-choice format question paired with a corresponding agree-disagree format question for each of these topics. These questions concerned by whom, why, and how often the preventive health measures were typically used. The multiple-choice knowledge-based questions were asked in a section of the survey according to the order shown in Table 1, and the corresponding agree-disagree format questions were asked in a subsequent section of the survey in the order shown in Table 1. There were some additional agree-disagree questions on respondent opinions regarding the preventative health topics that were interspersed within the agree-disagree section of the survey. These opinion questions were not a part of this analysis. Table 1 shows the underlined and bolded response that we considered to be correct for each of the questions. The correct answer to all of the agree-disagree questions was “agree.” All respondents received an identical version of the questionnaire. Given the controversy over some of the recommendations regarding how often some of these preventive health measures should be used, we considered as the correct response the most commonly accepted or typical interval for average use in the general population.
Table 1.
Preventive Health Knowledge Questions and Responses
| Multiple-choice Format | Answer Choices* | Agree-disagree Format | Answer Choices* |
|---|---|---|---|
| Pap smear questions | |||
| Who should have a Pap smear/Pap test? |
|
All adult women should have a Pap smear/Pap test. |
|
| Why do you think a Pap smear/Pap test is done? |
|
Pap smears/Pap tests help tell if a woman may have cervical cancer. |
|
| How often should most people go for a Pap smear/Pap test (unless otherwise directed by a doctor)? |
|
Adult women should have Pap smears/Pap tests each year, unless otherwise instructed by their doctor. |
|
| Breast self-examination questions | |||
| Who should examine their own breasts (breast self-exams)? |
|
All adult women should examine their breasts (breast self-exam). |
|
| Why do you think a woman should examine her own breasts (breast self-exam)? |
|
Women should examine their breasts to check for breast cancer (breast self-exam). |
|
| How often should a woman examine her own breasts (breast self-exam)? |
|
Adult women should examine their breasts each month (breast self-exam). |
|
| Mammogram questions | |||
| Who should have a mammogram? |
|
Women over age 40 and/or those instructed by their doctors should have mammograms. |
|
| Why do you think a woman should have a mammogram? |
|
Mammograms help determine if a woman has breast cancer. |
|
| How often should a woman typically have a mammogram (unless otherwise directed by her doctor)? |
|
Most women over age 40 and/or instructed by their doctors should have a mammogram every year. |
|
| Birth control pill questions | |||
| Who may take birth control pills? |
|
Only women can take birth control pills. |
|
| Why do you think someone might take birth control pills? |
|
Birth control pills can prevent pregnancy. |
|
| How often should someone take birth control pills? |
|
Birth control pills should be taken everyday in order to prevent pregnancy. |
|
| Emergency contraception questions | |||
| If a woman has had vaginal sexual intercourse with a man (without using birth control) can she take birth control pills AFTERWARDS to prevent pregnancy? |
|
A woman can take birth control pills shortly AFTER having vaginal intercourse with a man to prevent pregnancy. |
|
The correct answers are underlined and in bold.
The reading level of the survey was at a Flesch-Kincaid grade level of 5.4 (Microsoft Word; Microsoft Corp., Redmond, WA). We created a draft questionnaire and pilot tested it in June 2002 on a convenience sample of 20 female patients, and we revised the survey based on their comments and our observations.
Research assistants and trained volunteers administered the survey during eight- and four-hour shifts in three two-month blocks between July 2002 and May 2003. Twenty-five percent of the shifts were between 7 and 11 AM, 50% between 11 AM and 11 PM, and 25% between 11 PM and 7 AM. Each day of the week was sampled equally. The times selected for survey administration mirrored the patient volume at this ED. All seasons, days of the week, and hours of the day were sampled to avoid temporal-related biases in the sample selection.
Data Analysis
The analyses included tabulation of the percentage of responses to the demographic and preventive health knowledge questions by question format. The percentage of participants who correctly answered both questions on a given topic by question format was also calculated. Because we were comparing responses from the same respondents, we conducted McNemar’s test to compare differences in the percentages of participants who correctly answered the questions by question format type. Differences were considered significant at an α level of 0.05.
We conducted univariable and multivariable linear regression analyses to identify demographic factors associated with tendencies to correctly respond to knowledge-based questions when they were in an agree-disagree instead of a multiple-choice format. Because the univariable analyses were consistent with the multivariable analyses, only the multivariable analyses are presented. The demographic covariates used in the regression models were age (categorical; by age group), race (bivariate; white vs. nonwhite), and health care insurance (categorical; no insurance, Medicare/Medicaid, private insurance). We first constructed a 26-point scale that measured the difference in number of questions answered as correct. For this scale, the number of questions answered correctly for the agree-disagree format was compared with those answered correctly in the multiple-choice format. Because there were 13 preventive health knowledge questions, the scale ranged from +13 to −13. A +13 represented a woman who answered all 13 questions correctly in the agree-disagree format but none of the multiple-choice questions correctly, and conversely −13 represented a woman who answered all 13 of the multiple-choice questions correctly but none of the agree-disagree format questions correctly. A score of zero was assigned for those whose responses did not favor one type of question over another. We performed summary statistics of the distribution of these scores by demographic group and non-parametric tests (Wilcoxon rank sum test and k-sample equality of medians test) to compare the scores within each group. Next, we used the 26-point linear scale as the outcome in linear regression models and the three demographic factors (age group, race/ethnicity, and insurance status) as covariates.
Finally, we constructed ordinal logistic regression models to identify demographic factors associated with participant comprehension of the five preventive health knowledge topics. The outcome was concordance of answers between the two question formats for each preventive health knowledge topic. Concordance was considered a proxy measure of participant comprehension. The underlying assumption was that respondents who could correctly answer both questions of both formats demonstrated greater comprehension of that topic. Because there were three questions for each of the preventive health topics (except EC), concordance was determined on a four-point scale (from 0 to 3). On this scale, 3 (complete concordance) corresponded to answering all three paired multiple-choice and agree-disagree questions correctly for a given topic. For example, for Pap smears, complete concordance would entail correctly answering all of the three multiple-choice and all of the three agree-disagree “who, why, and how often” questions. A participant who had complete concordance on a topic was considered to be someone who demonstrated the highest level of comprehension for that topic. Zero concordance represented someone who exhibited the lowest level of comprehension for that topic. This person would have answered the paired questions differently for a given topic. Further, this means that the respondent answered either the multiple-choice or the agree-disagree question incorrectly for all three paired questions. Likewise, on this scale, a concordance of 1 meant the participant answered only one of the three sets of questions correctly and a concordance of 2 meant the participant answered two of the three sets of questions correctly. There were no participants who answered all agree-disagree and all multiple-choice questions incorrectly for a given topic. For the ordinal logistic regression models, we constructed univariable and multivariable models. Because the univariable analyses were consistent with the multivariable analyses, only the multivariable analyses are presented. Odds ratios and corresponding 95% confidence intervals were calculated. The data were analyzed using Stata 9.0 (Stata Corp., College Station TX).
RESULTS
Respondent Demographics
Of 1,100 women eligible to participate in this study, 769 (69.9%) completed all or part of the survey. Failure to answer questions in the survey was related to its length. Patients who did not complete the questionnaire had left the ED before its completion. Of the 769 women completing at least part of the survey, the data from the 570 women who answered all three demographic and 26 preventive health knowledge questions relevant to this analysis are presented in this report. Of the 570 women, 36.0% were aged 18–35 years, 28.6% were 26–35 years, 21.9% were 36–45 years, and 13.5% were 46–55 years old. In terms of self-identified race/ethnicity, 62.1% self-identified as being white, 10.4% as African American, 15.4% as Hispanic, and 12.1% as other. More than half (54.8%) had private health insurance; 30.5% had Medicare, Medicaid, or both; and 14.7% had no insurance.
Response to Knowledge-based Questions
Table 2 shows the percentage of women who correctly answered the individual knowledge-based questions (who, why, and how often) within each of the five topics. Table 2 also shows the percentage of respondents who answered all questions correctly for each of the five topics (“all correct”). The percentage of women answering the questions correctly is displayed for the two question formats (agree-disagree and multiple-choice formats). In addition, the percentages of women who answered the paired questions correctly in both formats (“MC and AD correct”) are also given in Table 2.
Table 2.
Percentage of Correct Responses by Topic and Question Format (n = 570)
| MC and AD Correct (%) | MC Correct (%) | AD Correct (%) | McNemar’s test p-value |
|
|---|---|---|---|---|
| Pap smears | ||||
| Who | 74.6 | 75.4 | 97.5 | ≤0.000 |
| Why | 75.3 | 76.5 | 94.2 | ≤0.000 |
| How often | 74.0 | 76.0 | 93.9 | ≤0.000 |
| All correct | 54.4 | 55.8 | 90.0 | ≤0.000 |
| Breast self-examinations | ||||
| Who | 92.6 | 93.9 | 98.1 | ≤0.000 |
| Why | 91.4 | 96.8 | 94.0 | ≤0.026 |
| How often | 84.4 | 85.4 | 94.9 | ≤0.000 |
| All correct | 76.7 | 81.4 | 89.8 | ≤0.000 |
| Mammograms | ||||
| Who | 51.4 | 51.6 | 94.9 | ≤0.000 |
| Why | 89.0 | 90.0 | 94.9 | ≤0.000 |
| How often | 63.3 | 65.1 | 90.0 | ≤0.000 |
| All correct | 37.9 | 38.3 | 87.9 | ≤0.000 |
| Birth control pills | ||||
| Who | 81.8 | 89.5 | 86.0 | ≤0.021 |
| Why | 93.0 | 97.5 | 94.7 | ≤0.011 |
| How often | 91.6 | 93.7 | 95.8 | ≤0.065 |
| All correct | 74.7 | 85.3 | 80.2 | ≤0.003 |
| Emergency contraception | ||||
| Why | 15.6 | 27.4 | 19.7 | ≤0.000 |
MC = multiple-choice format; AD = agree-disagree format.
We considered participants who answered both the agree-disagree and the multiple-choice questions correctly to exhibit greater comprehension than those who answered only one of the paired questions correctly. This measure of participant comprehension is first introduced in Table 2 as “MC and AD correct.” According to this assessment of comprehension, the respondents showed better overall comprehension of BCPs and BSEs than the other topics, that is, the percentage of “all correct” responses was higher for these two topics. The women surveyed demonstrated the least amount of comprehension for EC. For all five topics, the women were more likely to answer the individual questions (who, why, or how often) correctly than to answer all questions correctly (“all correct”). The percentage of correct responses to the individual questions varied within all topics, except for the Pap smear questions.
Table 2 also provides the percentage of respondents correctly answering the multiple-choice and the agree-disagree versions of each of the 13 preventive health knowledge questions. The results of McNemar’s test comparing the percentage of questions answered correctly for each pair of questions are also shown. Respondents were more likely to answer eight of the agree-disagree and four of the multiple-choice questions correctly than the questions in the other format. There was no difference for one paired set of questions (“how often” for BCPs). For Pap smears and mammograms, participants answered all of the agree-disagree correctly more often than the corresponding multiple-choice questions. For BSEs, participants answered two of the agree-disagree questions correctly more often than the corresponding multiple-choice questions, and answered the remaining multiple-choice question correctly more often than the corresponding agree-disagree question. For BCPs, participants answered two of the multiple-choice questions correctly more often that the agree-disagree questions. For EC, women were more likely to correctly answer the multiple-choice question correctly.
Demographic Correlates of Response Format
The summary statistics by demographic group and for all respondents for the distribution of values on the +13 to −13 linear scale comparing propensity to answer agree-disagree questions correctly with the corresponding multiple-choice questions are presented in Table 3. Because the middle value of the linear scale was zero, the range for all participants of −3 to 8 indicates a rightward distribution of values. This rightward distribution signifies that respondents had a tendency to answer agree-disagree questions correctly more often compared with multiple-choice questions. For the demographic groups, the nonparametric test results indicate that there was variation in the values within each group. Younger women, nonwhite women, and women without private health care insurance showed a tendency to correctly answer the agree-disagree questions more often than multiple-choice questions.
Table 3.
Scale of Tendency to Correctly Answer Agree-disagree Format Questions
| n | Range | Median | ||
|---|---|---|---|---|
| All respondents | 570 | (−3, 8) | 1 | |
| Age (yr) | ≤0.000 | |||
| 18–25 | 205 | (−3, 7) | 2 | |
| 26–35 | 163 | (−3, 7) | 1 | |
| 36–45 | 125 | (−3, 8) | 1 | |
| 46–55 | 77 | (−3, 5) | 1 | |
| Race/ethnicity | ≤0.000 | |||
| White | 354 | (−3, 7) | 1 | |
| Nonwhite | 216 | (−1, 8) | 2 | |
| Health insurance | ≤0.000 | |||
| None | 84 | (−1, 6) | 1 | |
| Medicare/Medicaid | 174 | (−3, 8) | 2 | |
| Private | 312 | (−3, 7) | 1 |
For each of the five topics, Table 4 displays the results of the linear regression analyses that examined demographic factors (age group, race/ethnicity, and insurance status) associated with a tendency to correctly answer agree-disagree questions more often than the corresponding multiple-choice questions. As shown by the β coefficients from the linear regression models, there was a greater tendency for participants to answer agree-disagree format questions correctly if they were younger, nonwhite, and without private health care insurance. For example, comparing nonwhite with white respondents for all preventive health topics, white respondents on average had a 0.63 (β) lower value on the linear scale that measured tendency to respond correctly to agree-disagree questions than multiple-choice questions, as adjusted for age and insurance status (Table 4). When considering each of the five topics, these relationships were also true for Pap smears and mammograms. However, there was no clear trend for BSEs, BCPs, and EC. We conjectured that the relationship observed for all preventive health topics for these factors was probably driven by the Pap smear and mammogram questions. We conducted a separate linear regression analysis removing these two topics and found that private health care insurance remained a factor in the multivariable analysis while age group and race/ethnicity were less strongly associated with the outcome.
Table 4.
Multivariable Linear Regression (n = 570)
| All Preventive Health Topics | Pap Smears | Breast Self-examinations | Mammograms | Birth Control Pills | Emergency Contraception | All Topics Except Pap Smears and Mammograms | |
|---|---|---|---|---|---|---|---|
| Age (yr) | |||||||
| 26–35 | −0.55 (−0.93, −0.17) | −0.31 (−0.48, −0.13) | −0.06 (−0.19, 0.06) | −0.28 (−0.46, −0.11) | 0.04 (−0.07, 0.15) | 0.06 (−0.02, 0.15) | −0.01 (−0.51, 0.50) |
| 36–45 | −0.43 (−0.86, −0.01) | −0.22 (−0.42, −0.03) | −0.07 (−0.20, 0.05) | −0.22 (−0.40, −0.04) | 0.04 (−0.08, 0.15) | 0.05 (−0.03, 0.14) | 0.45 (0.02, 0.87) |
| 46–55 | −0.79 (−1.21, −0.38) | −0.38 (−0.59, −0.16) | −0.07 (−0.22, 0.08) | −0.35 (−0.54, −0.15) | −0.03 (−0.15, 0.09) | 0.03 (−0.07, 0.13) | 0.32 (−0.21, 0.84) |
| Race | |||||||
| White vs. nonwhite | −0.63 (−0.98, −0.28) | −0.34 (−0.49, −0.18) | −0.04 (−0.15, 0.07) | −0.28 (−0.42, −0.13) | 0.01 (−0.10, 0.11) | 0.01 (−0.06, 0.09) | 0.37 (−0.04, 0.78) |
| Health insurance | |||||||
| Medicare/Medicaid | 0.13 (−0.40, 0.66) | 0.11 (−0.15, 0.37) | 0.10 (−0.07, 0.26) | −0.14 (−0.35, 0.08) | 0.07 (−0.08, 0.22) | −0.02 (−0.12, 0.08) | 0.28 (−0.45, 1.00) |
| Private | −0.65 (−1.10, −0.20) | −0.25 (−0.48, −0.03) | −0.10 (−0.22, 0.03) | −0.23 (−0.43, −0.04) | −0.05 (−0.17, 0.07) | −0.01 (−0.10, 0.08) | 1.47 (0.82, 2.12) |
All values are expressed as β (95% CI). Reference groups: age 18–25 years, white, no insurance.
Correlates of Knowledge-based Question Comprehension
Table 5 presents the results of the ordinal logistic regression analysis that examined if age, race, and health care insurance status were related to greater respondent comprehension of the topics. Greater comprehension for these analyses was defined as higher concordance in answering the two question formats correctly for each topic. Except for EC, concordance was measured on a 0–3 scale, with 0 representing no concordance between answering the paired agree-disagree and multiple-choice questions correctly and 3 representing complete concordance. In the analyses for all preventive health topics, white women and women with private health care insurance had greater concordance in their responses to the questions and therefore greater comprehension of these topics compared with other women. When the topics were analyzed separately, women with private health insurance demonstrated greater comprehension with regard to Pap smears, mammograms, BSEs, and BCPs. Older age was associated with greater comprehension for Pap smears and mammograms only. White race was associated with greater comprehension for Pap smears and mammograms and was possibly associated for BSEs and BCPs. There were no clear trends for EC.
Table 5.
Multivariable Ordinal Logistic Regression (n = 570)
| Pap Smears Knowledge | Breast Self-examination Knowledge | Mammogram Knowledge | Birth Control Pill Knowledge | Emergency Contraception Knowledge | |
|---|---|---|---|---|---|
| Age | |||||
| 26–35 | 1.93 (1.29, 2.89) | 1.04 (0.63, 1.71) | 2.04 (1.34, 3.09) | 0.67 (0.42, 1.08) | 0.74 (0.42, 1.30) |
| 36–45 | 2.40 (1.53, 3.78) | 1.26 (0.74, 2.16) | 2.43 (1.61, 3.66) | 0.64 (0.39, 1.07) | 0.90 (0.49, 1.68) |
| 46–55 | 2.71 (1.56, 4.72) | 0.81 (0.44, 1.48) | 3.20 (1.98, 5.18) | 0.71 (0.37, 1.36) | 0.54 (0.24, 1.22) |
| Race | |||||
| White vs. nonwhite | 2.37 (1.68, 3.36) | 1.46 (0.96, 2.22) | 1.88 (1.34, 2.63) | 1.39 (0.91, 2.11) | 0.67 (0.40, 1.11) |
| Health Insurance | |||||
| Medicare/Medicaid | 0.97 (0.58, 1.61) | 0.80 (0.44, 1.43) | 1.51 (0.92, 2.48) | 0.87 (0.50, 1.50) | 0.54 (0.24, 1.23) |
| Private | 2.45 (1.68, 3.86) | 1.86 (1.06, 3.25) | 2.91 (1.86, 4.57) | 1.82 (1.06, 3.12) | 0.67 (0.40, 1.11) |
All values are expressed as odds ratio (95% CI). Reference groups: age 18–25 years, white, no insurance.
DISCUSSION
In this study, we observed that women were more likely to correctly answer agree-disagree format questions compared with corresponding multiple-choice questions on women’s preventive health topics. We believe this finding has several implications for preventive health care research, particularly in the ED setting. Consistent with other studies, we found that our survey participants were more apt to choose the “agree” rather than the “disagree” response, whether or not they truly comprehended the underlying concept that the question tried to address. In other words, we believe that acquiescence bias affected the observed results. Therefore, studies of preventive health knowledge using the agree-disagree format may not accurately assess respondent comprehension. As a result, public health policy makers and others conducting this type of research may make flawed decisions based on inaccurate assessments of participant knowledge, which might ultimately affect delivery of preventive health services. To our knowledge, this is the only study examining respondent selection behaviors among ED patients. We believe that this and similar studies are important for the emergency medicine setting because there is considerable interest in implementing preventive health programs for ED patients.
We found that younger women, nonwhite women, and women without private health care insurance were more likely to answer agree-disagree format questions correctly but were more likely to demonstrate lesser comprehension on the topics we queried them. It appears that these groups of women are particularly inclined to answer “agree” even though they might not know the answer to the concept the question represents. These findings suggest that this type of question format is not useful in predicting knowledge for these groups of women. However, we believe that the observed results for all preventive health topics were probably driven by two of the five topics (Pap smears and mammograms). Nevertheless, the multivariable analyses support the finding that at least insurance status was associated with a tendency to favor answering questions with “agree.” In regard to comprehension of the topics, we are unable to make inferences on the specific effects of the question format on understanding versus knowledge of the questions themselves. That is, regardless of the question format, the respondents might have understood the question well but might not have known the answer. In this case, the findings at least suggest that certain respondents are more consistent in the way they answer the same question presented to them in different formats than other respondents.
The tendency of certain demographic groups to choose the “agree” option in an agree-disagree format question has been shown in several studies.9–13,18–20 We know of no other studies that have examined insurance status as a factor in identifying differences in respondent behavior on surveys that examine acquiescence bias. Prior studies have used other proxies of socioeconomic status, such as occupation or head of household income.11,12 Insurance status as a proxy for socioeconomic status is imperfect, however. It is likely that women with private health care insurance are more likely to benefit from educational initiatives about and offers for preventive health care. It might be these influences that explain the observed relationships.
We must consider an alternative explanation for the findings in this study. It is possible that the implicit assumption that the multiple-choice question format was a better assessor of respondent knowledge than the agree-disagree question format might not be correct. If the multiple-choice questions were, for example, not as well constructed as the agree-disagree questions, then the agree-disagree questions might have been easier to understand and to answer. As a consequence, lower-literacy respondents might have answered the agree-disagree questions more easily. Even with this alternative explanation, the findings present a cautionary reminder to researchers that the type of question format used in a survey can greatly affect the results obtained and subsequently the interpretation and actions taken based on the results.
LIMITATIONS
The sample included only English-speaking women, ages 18–55 years, who were visiting a single urban ED. Given our sampling of subjects all days of the week and all times of the day according to patient volume, and our year-round collection of data, we believe that we have a reasonably representative sample of those eligible to participate in the study. As a result, we believe that our observations, in general, are valid. We are hopeful that future studies can be conducted in multiple EDs and among even more diverse patient populations. Although we had a large sample, larger sample sizes might have improved our ability to show significance where our analyses merely revealed trends.
The order in which questions were presented to respondents, multiple-choice questions followed by agree-disagree questions on the same topic, undoubtedly had a priming effect. It is possible that the higher proportion of participants correctly answering the agree-disagree questions resulted from priming from the corresponding multiple-choice question. As a result, our estimates for the percentage answered correctly for agree-disagree questions were inflated. However, there were some questions where the percentage of participants answering the multiple-choice questions was higher than the corresponding agree-disagree format questions. This observation argues against a uniform priming effect.
All of the correct answers to the agree-disagree questions were “agree.” It is possible that respondents recognized this trend for these questions and correctly surmised that all questions would be correctly answered as “agree.” If so, the agree-disagree format questions overestimate women’s knowledge. However, not all participants answered all of these questions correctly. There was variability within each topic for the percentage of questions answered correctly. Furthermore, there were additional questions not included in this analysis regarding respondent opinions about preventive health measures that were interspersed within the agree-disagree format questions that broke up the pattern of questions. Regardless, a future study might vary the order of question format type and correct answers to the responses.
There is some disagreement among clinicians and public health policy makers on the suggested intervals for some of the preventive health measures we examined (e.g., mammograms). As a result, participants may have selected an incorrect response (per our judgment) or the “I am not really sure” response that might actually be correct according to other groups. However, few respondents chose the “I am not really sure” response, so the results would not have been significantly affected by these responses. Moreover, we believe our questions and responses attempted to target the most common or typical suggested intervals that would seem reasonable in light of the response options provided. We do not believe that the central point of the study to compare question formats was unduly influenced by this simplification.
CONCLUSIONS
Although patient knowledge on public health and preventive health is assessed through surveys, the responses are dependent on question format. This has many implications for public health and preventive health. Younger women, nonwhite women, and women without private health insurance displayed a tendency to correctly answer agree-disagree formatted questions more often than corresponding multiple-choice formatted questions. This tendency might reflect issues with participant test literacy, psychological tendencies to choose “agree” in the face of confusion or ignorance, or actual deficits in preventive health knowledge. Older women, white women, and women with private health insurance showed a higher degree of concordance across question type, which suggests better comprehension of the topics. On the other hand, if the multiple-choice formatted questions were more difficult to understand or not well constructed, agree-disagree formatted questions might be better assessors of knowledge for some women. For either interpretation, the study findings nevertheless demonstrate the influence that question format yields on the observed results.
The results of surveys, such as the one we conducted, can influence the decisions of policy makers and researchers, thus impacting expenditures, programs, and, ultimately, the state of public health. If our assumptions regarding acquiescence bias and variability of respondent behavior by demographic group are correct, investigators should avoid agree-disagree formats whenever possible to avoid overestimating respondent knowledge of preventive health topics. Continuing research should be conducted to determine improved methods of asking public and preventive health knowledge questions, particularly in the ED setting.
Acknowledgments
Dr. Merchant was supported by a National Institutes of Health training grant through the Division of Infectious Diseases, Brown Medical School, the Miriam Hospital, from the National Institute on Drug Abuse (5 T32 DA13911) and a Resident Research Training Award from the Society for Academic Emergency Medicine, and Ms. Vuittonet was supported by a Student Research Assistantship Award from Brown Medical School.
The authors thank Lyn Robillard, Kristina Casadei, and Robert Andreozzi for their data entry expertise, the volunteers from the Society for Clinical Research for Undergraduates at the Brown University for their help in survey administration, and CVS, Inc., for providing a $500 donation of gift cards.
References
- 1.Rhodes KV, Gordon JA, Lowe RA. Preventive care in the emergency department, part I: clinical preventive services—are they relevant to emergency medicine? Society for Academic Emergency Medicine Public Health and Education Task Force Preventive Services Work Group. Acad Emerg Med. 2000;7:1036–41. doi: 10.1111/j.1553-2712.2000.tb02097.x. [DOI] [PubMed] [Google Scholar]
- 2.Babcock Irvin C, Wyer PC, Gerson LW. Preventive care in the emergency department, part II: clinical preventive services—an emergency medicine evidence-based review. Society for Academic Emergency Medicine Public Health and Education Task Force Preventive Services Work Group. Acad Emerg Med. 2000;7:1042–54. doi: 10.1111/j.1553-2712.2000.tb02098.x. [DOI] [PubMed] [Google Scholar]
- 3.Aday LA. Designing and Conducting Health Surveys: A Comprehensive Guide. 2. San Francisco, CA: Jossey-Bass; 1996. [Google Scholar]
- 4.Fowler FJ. Survey Research Methods. 3. Vol. 1. Thousand Oaks, CA: Sage Publications; 2002. [Google Scholar]
- 5.Hopkins KD, Stanley JC. Educational and Psychological Measurement and Evaluation. Englewood Cliffs, NJ: Prentice-Hall; 1981. [Google Scholar]
- 6.Haladyna T. Developing and Validating Multiple Choice Test Items. Hillsdale, NJ: Lawrence Erlbaum Associates; 1994. [Google Scholar]
- 7.Frisbie DA. Multiple choice versus true-false: a comparison of reliabilities and concurrent validities. J Educ Meas. 1973;10:297–304. [Google Scholar]
- 8.Hancock GR, Thiede KW, Sax G, Michael WB. Reliability of comparably written two-option multiple-choice and true-false test items. Educ Psychol Meas. 1993;53:651–9. [Google Scholar]
- 9.Schuman H, Presser S. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. New York, NY: Academic; 1981. [Google Scholar]
- 10.Javeline D. Response effects in polite cultures: a test of acquiescence in Kazakhstan. Public Opin Q. 1999;63:1–28. [Google Scholar]
- 11.Lenski GE, Leggett JC. Caste, class, and deference in the research interview. Am J Sociol. 1960;65:463–7. [Google Scholar]
- 12.Ross CE, Mirowsky J. Socially-desirable response and acquiescence in a cross-cultural survey of mental health. J Health Soc Behav. 1984;25:189–97. [PubMed] [Google Scholar]
- 13.Bachman JG, O’Malley PM. Yea-saying, naysaying, and going to extremes: black-white differences in response styles. Public Opin Q. 1984;48:491–509. [Google Scholar]
- 14.Johnson TP, Cho YI, Holbrook AL, Rourke DO, Warnecke RB, Chavez N. Cultural variability in the effects of question design features on respondent comprehension of health surveys. Ann Epidemiol. 2006;16:661–8. doi: 10.1016/j.annepidem.2005.11.011. [DOI] [PubMed] [Google Scholar]
- 15.Warnecke RB, Johnson TP, Chavez N, et al. Improving question wording in surveys of culturally diverse populations. Ann Epidemiol. 1997;7:334–42. doi: 10.1016/s1047-2797(97)00030-6. [DOI] [PubMed] [Google Scholar]
- 16.Warnecke RB, Sudman S, Johnson TP, O’Rourke D, Davis AM, Jobe JB. Cognitive aspects of recalling and reporting health-related events: Papanicolaou smears, clinical breast examinations, and mammograms. Am J Epidemiol. 1997;146:982–92. doi: 10.1093/oxfordjournals.aje.a009226. [DOI] [PubMed] [Google Scholar]
- 17.Johnson TP, O’Rourke DP, Burris JE, Warnecke RB. An investigation of the effects of social desirability on the validity of self-reports of cancer screening behaviors. Med Care. 2005;43:565–73. doi: 10.1097/01.mlr.0000163648.26493.70. [DOI] [PubMed] [Google Scholar]
- 18.Bishop GF, Oldendick RW, Tuchfarber AJ. Effects of presenting one versus two sides of an issue in survey questions. Public Opin Q. 1982;46:69–85. [Google Scholar]
- 19.Schuman H, Presser S. Question working as an independent variable in survey analysis. Sociol Methods Res. 1977;6:151–70. [Google Scholar]
- 20.Gove WR, Geerken MR. Response bias in surveys of mental health: an empirical investigation. Am J Sociol. 1977;82:1289–317. doi: 10.1086/226466. [DOI] [PubMed] [Google Scholar]
- 21.Calsyn RJ, Winter JP. Understanding and controlling response bias in needs assessment studies. Eval Rev. 1999;23:399–417. doi: 10.1177/0193841X9902300403. [DOI] [PubMed] [Google Scholar]