Improving abortion underreporting in the USA: a cognitive interview study

Abstract

Abortion is a difficult-to-measure behaviour with extensive underreporting in surveys, which compromises the ability to study and monitor it. We aimed to improve understanding of how women interpret and respond to survey items asking if they have had an abortion. We developed new questions hypothesised to improve abortion reporting, using approaches that aim to clarify which experiences to report; reduce the stigma and sensitivity of abortion; reduce the sense of intrusiveness of asking about abortion; and increase respondent motivation to report. We conducted cognitive interviews with cisgender women aged 18–49 in two US states (N = 64) to assess these new approaches and questions for improving abortion reporting. Our findings suggest that including abortion as part of a list of other sexual and reproductive health services, asking a yes/no question about lifetime experience of abortion instead of asking about number of abortions, and developing an improved introduction to abortion questions may help to elicit more accurate survey reports. Opportunities exist to improve survey measurement of abortion. Reducing the underreporting of abortion in surveys has the potential to improve sexual and reproductive health research that relies on pregnancy histories.

Introduction

In the USA, about one in four women will have an abortion in their lifetime (Jones and Jerman 2017). Nevertheless, abortion stigma—’a negative attribute ascribed to women who seek to terminate a pregnancy’—remains a significant barrier to the access and provision of abortion services (Kumar, Hessini, and Mitchell 2009; Norris et al. 2011; Cutler et al. 2021) while hindering efforts to measure abortion in individual-level surveys worldwide (Scott 2017). The Turnaway Study, conducted at University of California San Francisco by the Advancing New Studies in Reproductive Health (ANSIRH) research group, found that more than half of respondents seeking an abortion perceived that people close to them or in their community would look down on them for seeking care (Biggs, Brown, and Foster 2020). Prior research has documented abortion underreporting of 60–70% by respondents in multiple US surveys (Yan 2021; Lindberg et al. 2020). Abortion underreporting has implications for researchers, service providers and policy makers, threatening the quality of the pregnancy and fertility data on which many service provision and budget allocations depend. In the absence of reliable self-reported data, researchers rely on health facility data to estimate abortion incidence in the USA (Jones, Witwer, and Jerman 2019); accurate data from facilities will likely be increasingly difficult to obtain directly as laws restricting abortions expand (Jones et al. 2021).

There has been increased attention to improving data accuracy in settings where abortion is highly restricted. Some new international approaches rely on indirect abortion reporting, such as self-reporting in list experiments and social network reporting, but these have not consistently generated reliable estimates of abortion incidence (Sully, Giorgio, and Anjur-Dietrich 2020; Sedgh and Keogh 2019; Moseson et al. 2015; Bell et al. 2020). Other research has explored how ambiguity about pregnancy status can improve reporting by reducing stigma compared to conventional abortion questions (Bell and Fissell 2021; Dixon-Mueller 1988). But within the USA, new approaches to measuring abortion have been limited (Kissling and Jackson 2022; Moseson et al. 2019; Cowan et al. 2016).

In this paper, we attempt to understand and improve abortion underreporting based on the cognitive stages of survey response, including comprehension, retrieval, judgement and reporting (Groves et al. 2009). Survey methodologists generally focus on reporting to improve measurement of sensitive behaviours. Misreporting is thought to be the result of respondents’ deliberate attempts to provide socially desirable answers (Tourangeau and Yan 2007). Misreporting may also be due to the perceived intrusiveness of questions, asking for information that respondents may consider too personal to share with an interviewer (Tourangeau, Rips and Rasinski 2000). The combined influence of comprehension, retrieval and judgement on abortion reporting has not received adequate research attention; one recent study found response latencies associated with direct abortion reporting, as respondents hesitate to decide whether and how to respond to the question (Bell and Bishai 2021). Other research exploring question wording found that respondents find the term ‘abortion’ to be more distressing than ‘termination of pregnancy’ (Norris et al. 2011).

This understanding of the survey response process informed our development of new survey questions and approaches that might improve the accuracy of self-reported abortion histories. We used cognitive interviews to assess new questions, gain insight into how respondents understand and evaluate abortion questions, and learn about respondents’ preferences for reporting their abortion histories. Cognitive interviewing is ‘an evidence-based, qualitative method specifically designed to investigate whether a survey question … fulfills its intended purpose’, (Willis and Artino 2013, 353) by identifying issues that arise in the response process (Hickman et al. 2021). Cognitive interviews test participants understanding of survey questions while allowing researchers to probe participants’ responses and identify comprehension, retrieval, judgement and reporting issues (Willis 1999). This feedback can guide the design and implementation of new survey items. Through cognitive interviewing, we evaluated survey questions exploring potential data quality improvement in the self-report of abortion in individual-level surveys.

Methods

We conducted 64 cognitive interviews with cisgender women¹ in two US states between January and February 2020. We selected Wisconsin (a state in the upper Midwest USA) and New Jersey (a Mid-Atlantic state adjacent to New York City) due to their differing abortion climates (abortion rates, clinic numbers, distance to clinics, population opinions on abortion, and legal context) (Nash 2019) and to avoid geographically specific findings. We conducted 35 interviews in suburban Wisconsin and 29 in urban New Jersey. Data collection in each state occurred over five days. Following Willis (2005), we aimed to interview respondents with diverse racial/ethnic backgrounds and ages to ensure our interviews captured a range of responses, and we modified questions in response to respondent feedback midway through data collection, after determining that additional interviews were not resulting in new information.

All respondents were recruited for an interview about sexual and reproductive health by the same third-party recruiting agency. The agency posts flyers and recruits individuals from public settings to join their database and participate in future research. When recruiting for this study, it contacted participants that previously indicated a willingness to participate in research on sensitive topics. Eligible participants were aged 18–49 years, assigned female at birth, identified as women, spoke English, lived in one of the two study states, and had ever had penile-vaginal sex. Participants were asked if they had ever had an abortion during the screening process to guarantee respondents included those who did and did not report an abortion. This ensured we received input from some respondents who would be able to speak to their experiences reporting an abortion. In all but one case, respondents’ self-report at screening matched their self-report during the interview.

Interviews lasted approximately 90 min, were audio recorded, and were conducted in English in private rooms at conference centre and market research locations. All participants provided verbal consent prior to the interview during which they were told they could stop the interview at any time or decline to answer any question. The Guttmacher Institute’s federally registered Institutional Review Board (DHHS identifier IRB00002197) approved the study.

We developed six sets of abortion survey questions (Table 1) and ten question introductions (Table 2) that we hypothesised could improve abortion reporting. We adapted two question series from the National Survey of Family Growth (National Center for Health Statistics (NCHS) 2020) that ask about abortion in the context of other pregnancies. Since the National Survey of Family Growth is the primary national survey that attempts to measure self-reported abortion incidence, these questions serve as the standard version for our investigation. We adapted third party abortion reporting methods (Sully, Giorgio, and Anjur-Dietrich 2020; Sedgh and Keogh 2019) to explore if asking about friends and family members’ abortion histories before asking about their own might improve retrieval and reduce social desirability bias within the US context. To further attempt to reduce social desirability bias, we adapted a National Survey of Family Growth question asking respondents to report their abortion on a list of less stigmatised sexual and reproductive health services. Due to concerns that respondents may miscategorise medication abortion, we asked a question that explicitly identified different abortion methods to improve comprehension, and following Udry et al. (1996), we separated questions about any abortion experience from the number of abortion experiences to reduce intrusiveness. Finally, we evaluated destigmatising question introductions, including the current National Survey of Family Growth introduction, to increase response motivation and reduce social desirability bias.

Table 1.

Initial question wordings used in the cognitive interviews.

Approach	Question wording
1 Pregnancy context	Sometimes women are reluctant to tell an interviewer about some of their pregnancies, especially those pregnancies that ended in abortion or with babies they no longer live with. In the next set of questions, please give a complete count of all your pregnancies. In your lifetime, how many pregnancies have you had that resulted in a live birth, that is, in a baby born alive? [#]. In your lifetime, how many pregnancies have you had that ended in miscarriage, stillbirth, or ectopic pregnancy? [#] In your lifetime, how many pregnancies have you had that ended in abortion? [#]
2 Pregnancy context	How many times have you been pregnant in your life? [#]
3 Priming using friends and family members abortions	As far as you know, have any of your friends or family ever had an abortion? [Yes/No]
4 SRH services context	Have you ever received any of the following birth control services shown on this card from a doctor or other medical care provider? [Yes/No] ● A method of birth control or a prescription for a method? [Yes/No] ● A check-up or medical test related to using a birth control method? [Yes/No] ● Counselling or information about birth control? [Yes/No] ● A sterilising operation? [Yes/No] ● Counselling or information about getting sterilised? [Yes/No] ● Emergency contraception, also known as “Plan B” or the “Morning-after pill,” or a prescription for it? [Yes/No] ● Counselling or information about Emergency contraception, also known as “Plan B” or the “Morning-after pill”? [Yes/No] We’re also interested in where women go to get other kinds of reproductive health care. Please look at this new sheet. In your lifetime, have you received any of the following medical services from a doctor or other medical care provider? ● A pregnancy test? [Yes/No] ● An abortion? [Yes/No] ● A Pap test – where a doctor or nurse puts an instrument in the vagina and takes a sample to check for abnormal cells that could turn into cervical cancer? [Yes/No] ● A pelvic exam – where a doctor or nurse puts one hand in the vagina and the other on the abdomen? [Yes/No] ● Prenatal care? [Yes/No] ● Post-pregnancy care? [Yes/No] ● A test for a sexually transmitted disease? [Yes/No]
5 Specifying abortion methods	People think about abortion differently. When we say abortion, we’re including people who have a surgical procedure, and people who take medications that end a pregnancy. Using that definition, have you ever had an abortion? [Yes/No]
6 Separating any and number of abortions	Have you ever had an abortion? [Yes/No] How many abortions have you had in your lifetime? [#]

Table 2.

Initial introduction wordings used in the cognitive interviews.

Option		Introduction wordings
1	Reluctance	Sometimes women are reluctant to tell an interviewer about some of their pregnancies, especially those pregnancies that ended in abortion or with babies they no longer live with.
2	Helping	The following question is one of the most important in this interview because it will help to improve family planning and health services for all women.
3	Common	Abortion is a common experience and there are a lot of reasons why people get abortions.
4	Statistic	1 in 4 American women will have an abortion in their lifetime. Research shows that 1 in 4 American women will have an abortion in their lifetime.
5	Legal	Abortion is a legal medical procedure in the United States.

Two members of the research team [authors AV and JM] conducted the interviews. Questions and response options were presented to respondents in a random order, except for the friends and family primer which was always presented as a first question when asked. Questions were displayed on laminated cards and read aloud by the interviewer. Respondents answered questions verbally and/or by marking their response on the card. After respondents had answered a question, interviewers probed for feedback about response formation, definitions of selected terms, and feelings toward the question. Once a respondent had been asked all the questions, they selected their most and least preferred questions as well as whether and why they thought those questions would encourage or discourage accurate abortion reporting. Respondents were also asked to reflect on their preferences for answering abortion questions on a survey. Immediately following the interview, participants completed a short sociodemographic questionnaire and received $150 cash as remuneration. Interviewers debriefed on findings and revised the interview guide to improve question clarity and wording daily. After completing the interviews in Wisconsin, questions and probes were added, edited and removed to improve comprehension and incorporate respondent feedback and preferences before conducting the interviews in New Jersey.

Audio recordings of the interviews were sent to a professional transcription service before being checked for accuracy and stripped of identifying information by Guttmacher staff. Deidentified transcripts were uploaded to NVivo12 for systematic content analysis, organising responses by themes and assigning codes to meaningful segments of the transcript. The research team (JM, MK, AV) iteratively developed a deductive coding scheme based on the interview guide and existing literature (Sedgh and Keogh 2019; Udry et al. 1996; Kumar, Hessini, and Mitchell 2009). Transcripts were initially triple coded by all members of the research team to test and refine the coding scheme. After finalising the coding scheme, transcripts were divided among the team members and coded. The team met regularly to resolve questions and ensure alignment between coders’ application of the coding scheme. Once coding was complete, we identified salient themes. We explored differences in major themes by respondent abortion report.

Results

Table 3 reports the samples’ demographic characteristics. We conducted slightly fewer interviews in New Jersey compared to Wisconsin, and approximately half of the interviews were conducted with respondents who reported prior abortions during screening. Three-quarters of the sample was between the ages of 28 and 49. Almost half of the respondents were non-Hispanic White, one-quarter were Hispanic, less than one-fifth were non-Hispanic Black, and less than one-fifth reported other or multiple races. Participants were highly educated, and most reported an annual income of at least 200% over the US federal poverty level. All respondents identified as heterosexual or straight (data not shown).

Table 3.

Respondents by demographic characteristic (N = 64).

Characteristic		n	%
State	New Jersey	29	45
	Wisconsin	35	55
Abortion	Yes	33	52
	No	31	48
Age	18–27	14	22
	28–38	23	36
	39–49	27	42
Poverty status	<100% Federal Poverty Level	1	2
	100–199% Federal Poverty Level	11	17
	200–299% Federal Poverty Level	29	45
	300+% Federal Poverty Level	23	36
Race/ethnicity	Non-Hispanic White	26	41
	Non-Hispanic Black	11	17
	Hispanic	16	25
	Other/multiple races	10	16
Marital status	Living with partner	8	13
	Married	28	44
	Other	28	44
Education	High school graduate or Tests of General Educational Development	3	5
	Some college or associate degree	34	53
	College graduate or above	27	42
Previous births	None	21	33
	One or more	43	67

We analysed respondent data based on their abortion report, although responses were generally consistent between those who did and did not report an abortion. All quotes come from respondents who reported an abortion unless otherwise stated. We did not find major differences by state. Results are organised by question approach and how stigma influenced reporting, as described by respondents. (Tables 1 and 2 show the initial language used, while the online supplemental Appendix shows the modified wordings).

Pregnancy context

We tested the National Survey of Family Growth approach of asking about abortion within the context of pregnancy (Table 1, Approaches 1 and 2). Respondents who reported an abortion consistently expressed a desire to keep their abortions separate from other pregnancies.

I feel like for my situation, if someone asked me, you’ve been pregnant three times but they see you only have two kids, they say, where is the third one? So, then you kind of have to explain. (NJ-11)

A few respondents did not include their abortions when providing their total number of pregnancies, associating ‘pregnancy’ with live births or wanted pregnancies, suggesting that comprehension could be an issue for using the context of pregnancies to ask about abortion.

The word “pregnant” seems to suggest that it’s wanted and voluntary, maybe. So, if the goal is to get information on abortion, focusing on words like pregnant might put off some people. (WI-31)

However, others felt that separate questions for different pregnancy outcomes could be helpful for correctly cataloguing pregnancy experiences.

I’m okay with it. It doesn’t bother me. I think it’s very specific. I liked that they kind of put it, you know, miscarriage/stillbirth/ectopic. I think it’s important that you specify because I think that can be a very like big question otherwise. (WI-20, Did not report an abortion)

When asked about pregnancies that ‘ended in abortion’, some respondents suggested that this wording removed implications of ‘fault’, which could facilitate reporting.

You could have said, “how many pregnancies have you had that you aborted” or that you have had an abortion, something with you, but you are not asking me personally, if it was my choice, I did it, but it ended in abortion. For some reason or the other. So, it could be maybe my life was in danger and the doctor decided. Of course, I would have had to go along with him, but it doesn’t put the [pause] all the pressure on me. (NJ-05)

Other respondents felt this wording was ‘loaded’, judgemental, and potentially ‘triggering’ because it emphasised that the pregnancy had ended. Respondents who did not report an abortion described how the question reminded them of a question from a healthcare provider and felt it would be easy for anyone to answer.

Priming using friends and family members abortions

To prime women to report sensitive abortion information, we asked respondents to report either whether or how many people they knew who had an abortion before asking about their own abortion experiences (Table 1, Approach 3). After respondents saw the rest of the questions, we asked how reporting these experiences affected answering questions about their abortion history. Some reported that thinking of their friends and family made them feel less alone.

It’s nice to know that there [are] people and that you’re not alone. It makes it a little bit easier. (NJ-22)

However, a few respondents reported that thinking of others affected their willingness to report.

I mean it doesn’t really affect how I answer any other questions having to think about it because, I mean, my answer is my answer. So, I had it done. So, it’s not going to change like how I answer or what I answer so it didn’t really affect my thought processes how to answer the questions at all. (WI-33)

Sexual and reproductive health services context

To reduce the possibility of social desirability effects, we contextualised abortion care within other sexual and reproductive health services one could receive from a healthcare provider (Table 1, Approach 4). This was the most preferred approach for respondents who did and did not report abortions. Participants reported that abortion was normalised when cushioned with other topics and the format was similar to a medical form.

When it’s on this list, it seems like it’s just another thing, another medical procedure that someone would have done, as normal as getting a pregnancy test or prenatal care, that kind of thing. (WI-27)

Respondents also liked the fact that this approach allowed them to report their abortion history in a checklist format, as it was quick, straightforward, and limited ‘dwell[ing]’ on abortion.

[The list] is not really digging. This is just yes/no, yes/no, and that’s it. This doesn’t really bring up any feelings to me. (NJ-03)

Specifying abortion methods

We attempted to improve comprehension by naming medication and surgical abortion methods when asking respondents for their abortion history (Table 1, Approach 5). We found that many respondents were unfamiliar with medication abortion or did not know the name of the method used in their abortion, resulting in comprehension issues.

I can see where it can confuse people because it’s like, well I just had an abortion; I don’t know if it was surgical or medication. [ … ] I think it can confuse people, like I said, if they don’t understand there’s two different types of abortions. They may say, “Well, I had an abortion, I don’t know what I had. I just had it done.” (NJ-29)

Mentioning medication abortion highlighted several respondents’ uncertainty about the difference between medication abortion and emergency contraceptive pills.

I think it’s a good question because as it states, people think differently about abortion. And in the back of my mind, like I didn’t really consider like the morning after pill a form of abortion. So, this question explains that and says, “Okay, now that you have that information; have you had it, ever had an abortion?” (WI-12)

Feedback on this question was largely negative. Some participants felt the question was unnecessarily detailed, intrusive, ‘clinical’ and ‘technical’, and evoked unwanted memories of their abortion experiences.

It makes you think about the actual … because surgical, you know how it’s done, but you also know how medication works. So, you think about it, and you get like, surgical, it’s a horrible way. (NJ-14)

Separating any and the number of abortions

To explore how intrusiveness influences willingness to report abortions, we asked respondents whether they had an abortion separately from asking about the number of abortions (Table 1, Approach 6). When asked to compare these two options, almost all respondents preferred the yes/no question.

Respondents generally reported that the yes/no question was straightforward, but they provided conflicting feedback as to whether this question felt intrusive. Some respondents expressed that this was ‘not anyone’s business’ or that it was ‘too personal;’ in contrast, others felt that the question was direct, general, and involved less ‘digging’ into their personal lives as compared to other abortion questions.

It’s not digging too deep into your medical history because it doesn’t ask the specifics about well, you have had an abortion. Well, how many times? And, you know … This is just – it’s information that’s provided or given and then you had just have an option to answer very generally, yes or no. (NJ-09)

Respondents overwhelmingly felt that asking about the number of abortions they had had was too intrusive and required too much information. Many respondents expressed social desirability anxieties, particularly when reporting more than one abortion.

[Any] would definitely be easier [to answer], because if I have to put a number here, that’s like, “oh my god, I had like 10 abortions. What wrong with me?” [ … ] Yes, I had and that’s it. I can have one or 20 and nobody is going to know nothing about me, but here … like I said, even if you put down two, it is more personal. (NJ-03)

A few respondents felt asking about the number of abortions was less judgemental than asking about any abortion experience since the question could be read as assuming the respondent had an abortion. Respondents suggested that they would prefer only to be asked to report the number of abortions if this information was needed for a specific purpose. Respondents preferred to only be asked one question rather than first the yes/no question and then the number.

In an alternate approach, respondents were asked a series of questions requesting information about the number of pregnancies they had, followed by questions about how each pregnancy ended. Some respondents who had had abortions felt uncomfortable about the request for the number of pregnancies.

I think that, again, the difficulty is just what, you know, what you live with after the fact, what you feel after the fact. Like I made this decision four times, um, which also, if I can be honest is very irresponsible. (NJ-09)

Notably, most respondents answering these questions named that it was not challenging to remember how many abortions they had ever had.

Destigmatising and motivating introductions

We also presented different abortion question introductions to increase response motivation and reduce social desirability bias. We asked respondents whether they would want to see an introduction before a question about their abortion history; the majority felt that an introduction should be used to ease respondents into the question while explaining its purpose and convey that the respondent is not alone. Those who reported not wanting an introduction felt that providing an introduction could have a negative effect by drawing attention to being asked about abortion.

The current National Survey of Family Growth introduction, which we term the ‘Reluctance’ introduction (Table 2, Option 1), was frequently reported as the least favourite introduction. Regardless of abortion history, respondents did not like the phrase ‘babies they no longer live with’, commenting that this phrasing was insensitive to the complexities of choosing whether to parent.

But at the time, the only right thing to do was give them up for adoption. So, it’s just the way it says “or with babies they no longer live with.” That just seems to me that that’s kind of an insensitive way of putting that. (WI-10)

Respondents labelled the language ‘negative’, despite it resonating with others. Some voiced concern that mentioning feeling ‘reluctant’ would encourage misreporting.

This would make me more probably not want to say, because I feel like it’s like, they know I’m not going to tell anyway. So, I might as well not tell. They don’t trust me anyway, so I’m not going to be honest. (NJ-03)

Respondents most preferred the ‘Helping’ introduction (Table 2, Option 2). Many respondents reporting an abortion felt this was motivational, explanatory and refocused the question away from the respondents’ experience.

This one seems like they’re asking for purposes of not just personal information, but for information to help all women and to do something about women’s health services and family planning … (WI-10)

Some respondents were sceptical that their answer could help other women and felt the introduction was misleading or gave too much importance to the question.

See, I would have to question this. [ … ] How does it improve family planning if I say yes, I had an abortion, or no, I had[n’t had] an abortion? (NJ-01)

In response to the ‘Common’ introduction (Table 2, Option 3), most respondents who did not report an abortion (and some who did) disagreed with the statement or felt that the commonality should not be highlighted.

It’s a common experience, but nobody is going to see it as a common experience. You’re not breaking your leg. (NJ-14)

Many respondents selected the ‘Common’ introduction as their favourite, citing that the language highlighted that they were ‘not the only one’ that had had an abortion.

I think it would probably be encouraging to respond in a truthful manner because it’s kind of allowing, it’s taking away any kind of judgment. It’s saying that there’s, you know, it’s a shared experience or it’s something that happens to a lot of people or a lot of women and that there are reasons I think that a lot of women judge themselves on their choices or feel others judge them without understanding that there are so many different reasons why pregnancies are terminated. (WI-35)

Feedback on the ‘Statistic’ introduction centred in part on whether ‘one in four’ was an accurate statistic; some assumed this overstated the prevalence of abortion while others viewed it as an underestimate (Table 2, Option 4). Respondents also focused on whether ‘one in four’ made abortion seem common or uncommon, with some concerned it suggested abortion was ‘cool’ or trendy.

Some respondents who reported an abortion felt ‘singled out’ by the mental imagery of being the one among four, while it made others feel like they were ‘not alone’.

Because I wouldn’t want to be in that statistic. I wouldn’t want to be the one of four. (NJ-26)

It’s more like, it’s okay. It’s not like you are the only person on this earth that’s ever had an abortion, and it’s showing there has been research on it. (NJ-18)

The ‘Legal’ introduction was the least preferred introduction as many respondents felt it was unhelpful, impersonal, polarising or encouraged abortion (Table 2, Option 5). Several respondents questioned whether abortion was legal everywhere in the USA.

It becomes a political thing, in this sense, to me at least. It feels more like a political question than it does a medical question about reproductive health. (WI-18)

Some felt the introduction might be helpful to alert respondents, particularly in US states hostile to abortion, that they would not be in ‘trouble’ for reporting an abortion.

It feels more comfortable because I am not going to be afraid of answering yes or no, because I am not doing something illegal. (NJ-15)

Influence of abortion stigma on reporting

As participants shared feelings and thoughts during the cognitive interview, stigma frequently arose as a potential influence on reporting. Many respondents who reported an abortion described feelings of shame, guilt, fear of judgement, and generally not being ‘proud’ of their decision. Experiences of stigma made them hesitant to discuss having had an abortion.

I think I am reluctant sometimes to tell interviewers about pregnancies that ended in abortion, because I don’t know their thoughts and I don’t know if they’ll judge me. (NJ-05)

Similarly, language that evoked detailed memories of abortion experiences was not viewed positively by respondents who reported an abortion.

I think it’s actually their experience, like, was it painful? Did you have somebody with you when you went? Why did you have to get an abortion? So, it’s bringing back up those emotions and the memory of that. (WI-03)

Some respondents preferred the phrase ‘termination’ to ‘abortion’, as they felt the word itself was too jarring. Some respondents were uncomfortable using the word ‘abortion’ at all.

It should be “termination” or “terminate.” “Have you ever had a termination?” Rather than “abortion” – it’s the word itself that could be a little too, um, just in your face, you know. (NJ-06)

There is such a stigma about [abortion] in our society, and that’s why I never used the word before because I felt like termination sounded a little better, because it wasn’t like we screwed up and I got pregnant accidentally. (NJ-11)

Discussion

Few advances have been made in assessing abortion underreporting, despite it being a well-documented phenomenon. This is one of the first studies to qualitatively and systematically investigate comprehension of, and responses to, abortion questions posed in surveys. Cognitive interviews are a valuable step in developing and implementing question items to improve survey measurement of abortion. Our qualitative analysis identified several promising approaches to measuring abortion in surveys while finding evidence that certain approaches are unlikely to yield meaningful improvements.

Many participants responded positively to having abortion as part of a list of less stigmatised sexual and reproductive health services. Placing abortion in the context of more socially acceptable health services appears to reduce social desirability bias around reporting of abortion. This is a notable shift from the conventional context of abortion in surveys, which situates abortion experiences alongside other pregnancies or stigmatised behaviours (2017–2019 NSFG Codebooks 2021). We encourage further research exploring how best to refine the services provided on this list. Interestingly, the National Survey of Family Growth has a section on health care services which includes abortion but limits the question to the past 12 months, which constrains its analytic use when it comes to estimating lifetime abortion incidence (2017–2019 NSFG Codebooks 2021).

A second approach that appeared successful was to solicit reports of any prior abortion rather than the number of abortions respondents had. Our findings support prior research that suggests asking a yes/no item about abortion reduces stigma associated with reporting multiple abortions (Kopplin, Desai, and Lindberg 2017). Respondents suggested they would prefer only to be asked to report the number of abortions if this information was needed for a specific purpose. The single question item about any abortion may meet many researchers’ needs. Researchers should be cautious in collecting more detailed information than this if it reduces data quality or completeness.

Most respondents supported the use of a question introduction. However, we found inconsistent reactions to the introductions that we designed. Some respondents may find an introduction motivating or destigmatising, while those having a negative response to the text may be less prone to report. It is important to grapple with the findings that the ‘Reluctant’ introduction modified from the National Survey of Family Growth typically garnered negative feedback, and future work should further examine developing and utilising improved introductions before asking about abortion.

Several approaches investigated in these interviews did not seem likely to improve reporting. We found naming abortion methods explicitly did not improve comprehension and may have introduced comprehension issues while seeming unnecessarily intrusive to respondents. We plan to explore this further in subsequent analyses. Our efforts to first ask about friends’ and family members’ abortions did not seem to prime women to report their own abortions, despite having intentionally broadened the question to include a wider range of people than close confidantes to increase the likelihood that respondents would answer affirmatively. Furthermore, participants responded negatively to the contextualisation of abortion among fertility and childbearing, as supported by prior literature suggesting that associating abortion with other pregnancies increases concerns of stigma (Kumar, Hessini, and Mitchell 2009).

Limitations

Further efforts to improve survey items for measuring abortion should be responsive to two factors not fully examined in this study. First, the sensitivity of and willingness to report an abortion is differentially experienced by women depending on personal and community characteristics (Maddow-Zimet, Lindberg, and Castle 2021) and there may not be a one-size-fits-all approach to improve reporting for all respondents. In this study, participants were generally highly educated with incomes ≥200% above the US federal poverty level, and only cisgender women were sampled. Given sample size constraints, we were not able to examine responses across demographic subgroups, so more investigation of variability is needed.

Similarly, while this study was designed to explore reporting across state contexts, we did not find major differences between respondents in Wisconsin and New Jersey. Responses may have been different in states with even greater abortion stigma or politicisation of abortion access, and other contextual factors should also be considered (Nash 2019).

A further limitation of the study was the inclusion of a question asking respondents “Have you ever had an abortion?” during eligibility screening which may have discouraged the involvement of respondents who were uncomfortable with the topic. Additionally, the pool of respondents initially contacted may have been biased as they were selected from a sample of people who had opted into being contacted about research on sensitive topics. As a result, respondents may have a different level of willingness to disclose their abortions. By including people who did not report an abortion in the screener (but may not have been truthful), we hoped to mitigate the effect of having a potentially skewed sample and include respondents who had abortions but would typically not report their experience. Interview responses from participants who did and did not report a prior abortion were generally similar. One notable difference was that respondents who did not report a prior abortion more often described abortion as highly stigmatised. Future research should bear in mind that respondents unwilling to report abortions may be missing from the sample.

Our findings cannot speak to question order since we randomised the order of questions and introductions. We recognise too that this study used a modified cognitive interview technique, as we did not test an entire questionnaire but attempted to identify improvements in abortion history questions. Further research is needed to determine if our findings can lead to improvements in reporting within the context of a larger survey as well as how applicable these findings are to survey reporting of other sensitive behaviours. We will be testing this in an experimental study that will adapt the most promising approaches and test them against each other to ascertain which are most effective in improving abortion reporting.

Conclusions

Abortion is extensively underreported on surveys, which undermines the provision of health care services and the quality of research on pregnancy experiences. Within the US context, our findings highlight the influence of question wording on accurate self-reporting of abortions. We found respondents most preferred answering a single yes/no question about whether they had ever had any abortions and preferred abortion questions posed in the context of sexual and reproductive health services, rather than pregnancy outcomes. Finally, respondents indicated that introductions to abortion questions could play a role in motivating respondents to answer accurately. This work can inform efforts to improve abortion measurement in other contexts, as well as inform efforts to improve the reporting of other sensitive behaviours.

Data availability statement

Participants were asked a separate, optional consent to allow their data to be made publicly available. De-identified data from the study is hosted by the Qualitative Data Repository at Syracuse University; researchers can apply for access through the site.