Relying on Surveys to Understand Abortion Behavior: Some Cautionary Evidence

Abstract

Objectives. The reliability of abortion self-reports has raised questions about the general usefulness of surveys in research about abortion behavior; however, the extent of underreporting remains a subject of some debate. This study sought to examine abortion reporting in a sample of welfare mothers and to determine factors in underreporting.

Methods. In New Jersey, which covers abortions requested by welfare recipients under its Medicaid program, the responses of a randomly drawn sample of 1236 welfare mothers about abortion events were compared with the Medicaid claims records of these women.

Results. Only 29% of actual abortions were self-reported by the women in the sample. This finding varied dramatically by race, with substantially higher rates of underreporting by Blacks than by Whites or Hispanics.

Conclusions. Although race is the most consistent predictor of underreporting behavior, attitudinal factors and survey technology also help in explaining abortion reporting behavior.

Survey research has become a staple for policy analysts over the past several decades and has always served as a critical data source in the study of many issues of interest to sociologists, demographers, and political scientists. The usefulness of surveys in studying highly personal or sensitive individual characteristics, such as income, crime victimization, mental health, or sexual behavior, however, has been questioned many times.1–4 It would be difficult to imagine a more sensitive topic than induced abortion, yet surveys have commonly been used to study abortion behavior.

Accurate reporting of induced abortions is important for 2 reasons. First, accurate reports are crucial to estimating the incidence of unintended pregnancies, which is of immense interest to reproductive health professionals and researchers. Data on unintended pregnancies, in turn, are critical to assessing the failure rate of contraceptives and the need for contraceptive services. Underreporting of abortions introduces a downward bias in the estimation of contraceptive failure rates calculated from survey data5–7 (also E. F. Jones and J. D. Forrest, unpublished report, 1989). Second, an unreported abortion is also an unreported pregnancy. Demographers rely heavily on pregnancy data to estimate fecundity and elucidate the fertility dynamics of populations.8,9

Many studies were undertaken in the 1980s to assess the reliability of survey data on abortions. These studies, however, relied on average or aggregate characteristics culled from external data sources to gauge the quality of survey data. Only 1 previous study has compared self-reported abortion data with external medical records for the same individuals.10

Three major national surveys have contributed information on abortion at an individual level in recent years: (1) the National Survey of Family Growth, (2) the National Surveys of Young Women, and (3) the National Longitudinal Surveys of Work Experience of Youth. The principal sources of external data used by researchers in assessing the reliability of survey estimates of abortions have been the Alan Guttmacher Institute's provider surveys and the Centers for Disease Control and Prevention's compilation of state health department and other abortion data. Findings on underreporting from various studies show that the rate of underreporting ranges from 40% to 65%; that is, only 35% to 60% of actual abortions are reported in surveys.

Previous researchers have identified some factors that influence underreporting of abortions. Udry et al. provide a useful taxonomy that classifies these factors as fertility-related, demographic, or survey implementation factors.10 Hammerslough adds a further dimension, namely, psychologic repression.5

Underreporting of abortions has been found to vary with fertility-related factors, such as the method of contraception used and the number of abortions a woman has had.11,12 Demographic factors that have been identified as most consistently influencing underreporting of abortions are race and marital status of the respondent. Non-White and unmarried women are significantly more likely than others to underreport.5,13,14

Interviewer characteristics, the directness of the abortion question, and the interview method have also been found to be useful predictors of underreporting. London and Williams found that women of all races are more likely to underreport having had an abortion to an interviewer who is not of their own race.15 There is some evidence that abortion questions that are asked indirectly—through prefacing, filtering questions, or randomized response technique—result in a higher rate of abortion reporting.5,16

In the present study, abortions reported by a sample of New Jersey welfare clients surveyed in 1995 were compared with abortion data on the same individuals from Medicaid claims files. Demographic information for the analysis was drawn from New Jersey public welfare administrative records. Data from these 2 sources, while not perfect, are generally considered the most reliable administrative data available because they are directly linked to welfare check issuance or vendor payments. Both data sources are subject to rigorous internal quality control edits and external sampling audits.

It should be pointed out that none of the previous studies that have examined abortion underreporting have included respondents' attitudes toward abortion and, more generally, attitudes toward childbearing experience. In the present study I used a set of 3 scales developed by W. B. Miller (unpublished paper, 1993) that probe these attitudes. Miller's Abortion Attitude Index (AAI) measures the respondent's overall attitude toward abortion and attempts to tap the physical, social, psychologic, and moral aspects of pregnancy termination. The second scale measures the respondent's positive childbearing motivation (PCM), and the third measures negative childbearing motivation (NCM). (The items that constitute the 3 scales are described in an appendix that is available from the author.)

I hypothesized that women who have a less restrictive attitude toward abortion would be more likely to report abortions, regardless of demographic, interviewer, or fertility-related factors, and that the same would be true of women with a more negative (less positive) attitude toward childbearing.

METHODS

Data and Sample

In this study I used data from a survey of welfare clients in New Jersey conducted in 1995 as part of a larger evaluation of New Jersey's welfare program, called the Family Development Program. The evaluation used an experimental design in which 8379 welfare clients were randomly assigned to either the Family Development Program group or a control group. A random sample of 3018 of the 8379 clients were selected to be surveyed; 1236 surveys were completed, for a response rate of just over 40%. The primary reason for nonresponse was the inability to locate women who had moved. The 40% response rate is comparable to rates for other surveys targeted at welfare clients in which limited resources precluded an extensive follow-up effort.17,18

The client survey contained an extensive fertility component that recorded respondents' pregnancy history and their attitudes toward becoming pregnant, carrying a pregnancy to term, and terminating or avoiding a pregnancy. To elicit pregnancy history, the survey used the format employed by the 1988 National Survey of Family Growth—respondents were asked about each pregnancy and its outcome, that is, whether the pregnancy ended in miscarriage, stillbirth, abortion, or live birth. The survey also contained the AAI and the attitudinal questions on childbearing that were adapted from W. B. Miller (unpublished paper, 1993).

The survey was administered by a nationally known professional survey research firm between July 1995 and January 1996. Because all of the survey targets were women, only women interviewers were used. Of the 1236 women interviewed, 1087 were interviewed via computer-assisted telephone interviewing techniques, and 149 were interviewed in their homes via computer-assisted personal interviewing techniques.

Two versions of the survey were used: a short version that typically took 45 to 50 minutes to administer and a long form that required an average of 70 minutes to complete. Although both versions contained the same detailed fertility component, the long form contained more detailed probes on employment history, family and youth functioning, and health care coverage.

Although it is possible that nonrespondents were systematically more or less likely than respondents to underreport their fertilityrelated behavior, this comparison of actual and reported abortion behavior nonetheless offers valuable insight into the usefulness of attempting to gather sensitive information through surveys. The representativeness of this survey is indicated in Table 1 ▶, which shows that the survey respondents closely mirrored nonrespondents on experimental status, length of welfare receipt, marital status, mean age, and mean number of children eligible for Aid to Families With Dependent Children. There was a high correspondence between the 2 groups in county of residence, and the distribution of number of abortions across the 2 groups was similar.

TABLE 1—

Characteristics of Respondents and Nonrespondents to a Survey on Pregnancy, Childbirth, and Abortion: New Jersey Welfare Clients, July 1995–January 1996

	Respondents	Nonrespondents
Experimental status, %	(n = 1223)	(n = 1782)
Experimental group	53.0	55.0
Control group	47.0	45.0
Length of welfare receipt, %	(n = 1223)	(n = 1782)
Long-term	53.2	55.2
Short-term	47.7	44.8
County of residence, %	(n = 1223)	(n = 1782)
Atlantic	5.2	6.1
Camden	14.4	13.0
Cumberland	6.2	4.8
Essex	22.7	25.7
Hudson	13.1	16.4
Mercer	6.0	6.1
Passaic	8.6	7.8
Union	7.4	7.2
All others	16.4	12.9
Education, %	(n = 1229)	(n = 1740)
<High school	35.5	41.8
All others	64.5	58.2
Marital status, %	(n = 1232)	(n = 1752)
Never married	59.2	64.1
All others	40.8	35.9
Race, %	(n = 1232)	(n = 1752)
White	22.8	18.0
Black	53.5	48.8
Hispanic	23.7	32.2
Other	0.0	1.0
Age, mean, y	(n = 1232)	(n = 1782)
	32.2	31.4
Employed, %	(n = 1233)	(n = 1782)
	29.7	28.9
No. of AFDC-eligible children, mean	(n = 1233)	(n = 1782)
	1.8	1.8
No. of abortions reported, %	(n = 1233)	(n = 1782)
0	81.7	80.7
1	12.5	10.7
2	3.9	5.3
3	1.4	2.0
4	0.4	0.8
5	0.1	0.5

Note. AFDC = Aid to Families With Dependent Children.

It is apparent that Hispanics were underrepresented in the surveyed group, as were women with less than a high school education. Respondents may differ from nonrespondents in unmeasured ways, as well. One of the mechanisms through which these unmeasured differences may surface is clients' work behavior. The close correspondence between respondents and nonrespondents on the key characteristic of employment—a behavior that would be expected to have a high correlation with unmeasured characteristics such as motivation and self-directedness—provides some further evidence for the representativeness of the sample's opinions and attitudes.

Variables

Four sets of variables were used to describe and predict accuracy in abortion reporting: demographic variables, attitudinal variables, survey implementation variables, and fertility-related variables.

The first set of variables contains typical demographic measures such as age, race, marital status, and education. In addition, this set includes a measure of whether the respondent was a long-term welfare recipient or a short-term recipient.

The second set of variables consists of 3 scales, alluded to earlier, that tap respondents' motivations with regard to childbearing and their attitudes about abortion: the PCM scale, the NCM scale, and the AAI. The PCM scale contains 21 items that describe the desirability of getting pregnant and having a child. The respondent is asked how important each of these items was in her decision to get pregnant or to have a baby. The items are coded 1 (important) or 2 (not important). Following Miller,19 responses to all 21 items are added to form a scale score. However, the items are recoded prior to summing so that higher scores indicate a more positive motivation toward childbearing.

The NCM scale contains 19 items that describe the “undesirability” of having a pregnancy or giving birth. These items are coded 1 (important) or 2 (not important). The responses to the 19 items are then summed to form a scale score, with low scores indicating a more negative motivation toward childbearing.

The AAI assesses how respondents feel about terminating a pregnancy. This index contains 13 items, which are coded 1 (an acceptable reason to terminate a pregnancy), 2 (not sure), or 3 (not acceptable). Higher scores on this scale indicate a more restrictive attitude toward abortion.

The third group of variables, survey implementation measures, describes the mode of administration (by telephone or in person) and whether the respondent was given the long or short version of the interview. As in Anderson et al.,12 the measure used to represent fertility-related variables in this study is the number of abortions the respondents had during the study period according to Medicaid data.

The external data used to validate reported fertility events came from 2 administrative data systems. The welfare department's database (New Jersey Department of Human Services, Family Assistance Management Information System, 1992–1996) provides demographic data. The Medicaid claims files (New Jersey Department of Human Services, Division of Medical Assistance, 1992–1996) provide data on abortions.

Analytic Strategy

Abortions as reported in the survey for the period January 1993 through the end of the survey (January 1996) were matched with quarterly Family Assistance Management Information System and Medicaid claims files by means of the respondent's welfare case number and interview date. If the data were matched on a quarterly or yearly basis, a misreported abortion might be categorized as a nonreported event; therefore, data for the entire time period (January 1993 through January 1996) were matched.

Two kinds of analyses were undertaken: descriptive comparisons of reported and actual abortions, and multivariate analysis of the determinants of underreporting of abortions. The dependent variable for the latter analysis was defined only for women who had had at least 1 abortion (according to administrative records) during the study period (n = 224). This variable captures whether or not a client reported accurately at least 1 abortion that she had had during the study period; it was coded 1 if the respondent reported an abortion that was also found in the administrative data system and 0 if she did not. Logistic regression models were used to estimate a respondent's accuracy in reporting abortions.

RESULTS

Descriptive Results

Table 2 ▶ shows a comparison of reported vs actual abortions for all survey respondents and for different subgroups. Overall, there is a wide difference between the 2 data sources in the number of abortions reported—only 29% of actual abortions were reported by survey respondents. White women reported about 71% of their actual abortions, whereas Black women reported only 24% of their actual abortions. The rate of abortion underreporting for Hispanics, 34%, lies in between that of Whites and Blacks.

TABLE 2—

Self-Reported vs Actual Abortions, by Respondents' Characteristics and Survey Implementation Factors: New Jersey Welfare Clients, July 1995–January 1996

	n	No. of Abortions Reported	No. of Actual Abortionsa	% of Actual Abortions Reported
Race
White	281	17	24	71.0
Black	659	59	251	24.0
Hispanic	282	17	50	34.0
Age, y
16–24	244	39	142	27.5
25–28	246	25	80	31.3
29–33	246	20	55	36.4
34–38	243	8	31	25.8
>38	246	1	17	5.9
Marital status
Married	127	5	16	31.3
Separated/divorced	365	17	55	30.9
Never married	726	70	253	27.7
Education
<High school	435	33	110	30.0
High school	564	39	137	28.5
>High school	194	21	72	29.2
Mode of interview
In person	143	4	40	10.0
Telephone	1090	89	285	31.2
Length of interview
Long	452	23	120	19.2
Short	780	70	205	34.1
Overall	1233	93	325	29.0

^aAccording to Medicaid claims files.

Abortion reporting generally increased with age, except for the 2 oldest age groups. Never-married women were the least likely to report abortions. No specific pattern emerged with respect to education, however. Survey implementation factors indicate that in-person interviews and long interviews were less successful in eliciting the truth about abortions than interviews conducted over the telephone and shorter interviews.

Multivariate Results

Table 3 ▶ presents descriptive statistics on the variables used in the multivariate analysis. As noted, only women of childbearing age who, according to Medicaid claims records, had had at least 1 abortion during the study period were included in this analysis. The racial breakdown of the sample was 77% Black, 15% Hispanic, and 8% White. About 76% of these women had never been married, and their ages ranged from 17 to 49 years. Average education was about 12 years.

TABLE 3—

Description of Variables Used in Multivariate Analysis of the Determinants of Underreporting of Abortions: New Jersey Welfare Clients, July 1995–January 1996

	Mean (SD)
Dependent variable: truthful abortion reporters	0.29 (0.45)
Demographic measures
Black	0.77 (0.42)
Hispanic	0.15 (0.36)
Never married	0.76 (0.43)
Age, y (range, 16.7–48.76)	27.64 (6.57)
Education, y (range, 1.0–18.0)	11.76 (2.34)
Short-term welfare receipt	0.38 (0.49)
Attitudinal measures
Abortion Attitude Index (range, 15.0–39.0)	28.41 (5.81)
Positive childbearing motivation scale (range, 4.95–20.0)	11.67 (4.95)
Negative childbearing motivation scale (range, 4.23–18.0)	10.47 (4.23)
Survey implementation measure: interviewed in person	0.10 (0.30)
Fertility measure: no. of abortions	1.45 (0.77)

Note. The analysis was restricted to women who had had at least 1 abortion during the study period (n = 224).

Table 4 ▶ displays the results of logistic regression analyses. Model 1 contains only the demographic variables; attitudinal measures, survey implementation measures, and fertility measures are added incrementally in models 2, 3, and 4. We see from model 1 that the only demographic variables that were good predictors of accuracy in abortion reporting were race and age. Education, marital status, and length of welfare receipt did not appear to be significant predictors of accuracy in abortion reporting.

TABLE 4—

Coefficients (SEs) From Logistic Regression of Truthful Abortion Reporters on Demographic, Attitudinal, Survey Implementation, and Fertility-Related Factors: New Jersey Welfare Clients, July 1995–January 1996

Regressor	Model 1	Model 2	Model 3	Model 4
Intercept	1.1562 (1.2524)	6.0040 (1.8715)	7.1614 (1.9631)	6.7615 (1.9963)
Demographic measures
Black race	–1.0726** (0.5308)	–1.1769** (0.5537)	–1.1531** (0.5571)	–1.1939** (0.5601)
Hispanic race	–0.4806 (0.6221)	–0.7404 (0.6524)	–0.4683 (0.6633)	–0.5275 (0.6671)
Never-married status	0.1618 (0.4010)	0.2825 (0.4256)	0.0840 (0.4338)	0.0449 (0.4364)
Age	–0.0598** (0.0278)	–0.0708** (0.0297)	–0.0669** (0.0314)	–0.0603* (0.0317)
Education	0.0140 (0.0686)	–0.0257 (0.0718)	–0.0222 (0.0733)	–0.0299 (0.0741)
Short-term welfare receipt	0.3350 (0.3201)	0.1747 (0.3394)	0.0787 (0.3484)	0.1980 (0.3626)
Attitudinal measures
Abortion Attitude Indexa	. . .. . .	–0.1035** (0.0317)	–0.1212** (0.0338)	–0.1235** (0.0343)
Positive childbearing motivation scaleb	. . .. . .	–0.0568* (0.0339)	–0.0641* (0.0346)	–0.0616* (0.0347)
Negative childbearing motivation scaleb	. . .. . .	–0.0446 (0.0410)	–0.0887** (0.0450)	–0.0885** (0.0454)
Survey implementation measure:	. . .. . .	. . .. . .	–1.9493* (0.7800)	–2.0682** (0.7857)
interviewed in personc
Fertility measure: no. of abortions	. . .. . .	. . .. . .	. . .. . .	0.2535 (0.2201)
Pseudo R2	0.05	0.11	0.14	0.15

Note. The analysis was restricted to women who had had at least 1 abortion during the study period (n = 224).

^aHigher values indicate a more restrictive attitude.

^bHigher values indicate a more positive attitude.

^cBecause all in-person interviews used the long version of the questionnaire, being interviewed in person and the length of the interview were collinear. Therefore, only 1 of the 2 variables could be used in the regressions. Using the length-of-survey variable yields results that are qualitatively the same as results of using the in-person interview indicator.

^*P < .10; ^**P < .05.

Model 2 shows that in addition to demographic factors, women's attitudes toward abortion and childbearing in general had a significant bearing on whether or not they accurately reported abortions. Women with more restrictive attitudes were significantly less likely than other women to tell the truth about having had an abortion. Also, women who held a less positive attitude toward childbearing were more likely to report an abortion.

Model 3 adds the mode of survey administration. Respondents who were interviewed in their homes were significantly less likely than respondents who were interviewed by telephone to admit they had had an abortion.

Model 4, in which the fertility-related measure of actual number of abortions is added, contributes little to model 3 by way of statistical fit. All the significant variables from the earlier models, however, remain significant in the final model, and their effects typically become stronger. Model 4 shows that the odds of accurate reporting for Blacks were 70% lower than those for Whites. A 1-unit increase on the AAI reduced the odds of accurate reporting by 12%, a 1-unit increase on the PCM scale reduced the odds by 6%, and a 1-unit increase on the NCM scale reduced the odds by 8%. In-person interviews reduced accurate reporting by 87%.

An effort was made in these multivariate analyses to test whether abortion attitudes had an impact on underreporting that was moderated by race. This was done by including an interaction term between race and the AAI in models 2, 3, and 4. The interaction term did not reach statistical significance in these models, providing no evidence for the racial dependence of the AAI in this sample.

DISCUSSION

This study attempted to gauge the extent of abortion underreporting in a sample of welfare mothers and sought to explain underreporting behavior through demographic, attitudinal, survey implementation, and fertility-related factors. The comparison of self-reported fertility data with individual-level information from an external data source (Medicaid administrative data files) for this low-income population is one contribution of this study. It is worth noting that this is only the second study in the United States to use an individual-level abortion matching methodology.

In addition, the study examined the role of an expanded set of explanatory variables in influencing reporting of abortions. This expanded set includes attitudinal factors as well as the demographic, survey implementation, and fertility-related factors that have previously been used to explain abortion underreporting.

Finally, a perspective is introduced that could help our understanding of racial differences in underreporting—one that indicates that improvements in survey technology may have only a limited impact in closing the racial gap. The key findings of this study are as follows:

• Only a small percentage of actual abortions are reported.

• Race is the largest single demographic factor affecting accuracy of abortion reporting; Black women are the least likely to report having had an abortion.

• Underreporting of abortions generally decreases with age.

• Respondents' general disposition toward childbearing exerts a significant influence on whether or not they report an abortion; women whose attitude toward childbearing is more positive are more likely to underreport abortions.

• Women who have more restrictive attitudes toward abortion are more likely to be underreporters.

• Longer interviews and interviews conducted in person are less likely to elicit truthful abortion reporting.

How does one account for the enormous difference in underreporting of abortions between White and Black women? There are at least 3 possible explanations: (1) race may interact with abortion attitudes to produce differences in underreporting; (2) there may be differences across racial groups in other attitudinal measures (not included in the analyses) that contribute to differences in underreporting; or (3) there may be differences in the degree of mistrust with which different racial groups regard the survey process.

The first possibility was tested empirically in this study, but the evidence did not support the notion that the relationship between abortion attitudes and underreporting varies by race. Differences in other attitudes or trust levels, however, may provide a partial answer to the racial differences in abortion underreporting. Smith and Seltzer, for example, in their analyses of more than 50 national, state, and local public opinion polls, reported that on a series of questions measuring religiosity, the differences between Blacks and Whites consistently ranged from a “gap” of 10 to 19 percentage points to a “chasm” of more than 40 percentage points, with Blacks viewing religion as more important.20,21 They also showed that the difference between Blacks and Whites in response to statements such as “People cannot be trusted” approached almost 50%, with Blacks much less trusting.

Although Smith and Seltzer did not address the issue of abortion, the case can be made that this behavior—abortion—occurs at the very point where religious guilt and the need for concealment intersect. Perhaps carefully worded questions, more sensitive questionnaire contexts, different survey modes, or racially matched interviewers could help overcome at least the element of mistrust and improve abortion reporting among Blacks.

This study's findings on interview length and mode of survey administration offer some hope in improving reporting. Shorter interviews and interviews that put some psychologic distance between the respondent and the survey administrator appear to induce respondents to be more truthful reporters of abortion. Other studies also show that self-administered questionnaires, with or without computerization, appear to offer the best results in reporting.7,14,15,22–27

The generalizability of the survey data used in this study may, of course, be limited by a low response rate, concerns about sample representativeness, and the population under study. The comparison of survey respondents and nonrespondents on key measured factors indicates that nonresponse bias may be small and that estimates of underreporting may be conservative. The study's findings, moreover, are not generally at variance with those of similar studies that have used nationally representative samples of women at different childbearing ages. For example, the percentage of actual abortions reported that was uncovered in this study for non-White women is comparable to those found in studies by Hammerslough5 and Jones and Forrest,14 both of which used nationally representative data from the National Survey of Family Growth, the National Longitudinal Surveys of Youth, or both.

The most significant difference between the sample used in this study and the samples in other studies cited in this article is in socioeconomic level. The present study was limited to recipients of Medicaid or public assistance, a factor that signals uniformly high levels of poverty. The overall percentage of actual abortions reported in national surveys is somewhat higher than the 29% reported here, ranging between 35% and 59%.7,14 However, when socioeconomic status is controlled for in these national studies, the differences in actual reporting percentages between different samples of low-income women are small.

This study serves to point up the difficulties of relying solely on survey responses to study induced abortions. This problem poses a serious dilemma for researchers attempting to study the pressing issues of pregnancy, birth, and abortion decisions. These findings indicate that survey data require an external source of validation before they can be used to make legitimate inferences or projections about abortions.