Reducing under-reporting of stigmatized health events using the List Experiment: results from a randomized, population-based study of abortion in Liberia

Abstract

Background: Direct measurement of sensitive health events is often limited by high levels of under-reporting due to stigma and concerns about privacy. Abortion in particular is notoriously difficult to measure. This study implements a novel method to estimate the cumulative lifetime incidence of induced abortion in Liberia.

Methods: In a randomly selected sample of 3219 women ages 15–49 years in June 2013 in Liberia, we implemented the ‘Double List Experiment’. To measure abortion incidence, each woman was read two lists: (A) a list of non-sensitive items and (B) a list of correlated non-sensitive items with abortion added. The sensitive item, abortion, was randomly added to either List A or List B for each respondent. The respondent reported a simple count of the options on each list that she had experienced, without indicating which options. Difference in means calculations between the average counts for each list were then averaged to provide an estimate of the population proportion that has had an abortion.

Results: The list experiment estimates that 32% [95% confidence interval (CI): 0.29-0.34) of respondents surveyed had ever had an abortion (26% of women in urban areas, and 36% of women in rural areas, P-value for difference < 0.001), with a 95% response rate.

Conclusions: The list experiment generated an estimate five times greater than the only previous representative estimate of abortion in Liberia, indicating the potential utility of this method to reduce under-reporting in the measurement of abortion. The method could be widely applied to measure other stigmatized health topics, including sexual behaviours, sexual assault or domestic violence.

Key Messages

• Under-reporting of sensitive health events plagues our ability to accurately plan for and develop needed health services and policies.

• Alternative methods of measurement have been developed, but many rely on strong assumptions or require logistical complexity and time in implementation.

• This study introduces to the public health literature a novel method for measuring sensitive health events, and compares it with other existing methods.

• We find that this method, the List Experiment, may dramatically reduce under-reporting of a sensitive health event as compared with alternative methods.

• Results indicate that this is a valuable new method that should be added to the epidemiologist’s toolbox to generate improved population estimates of sensitive health events and exposures worldwide.

Background

Validation studies suggest that the degree of under-reporting on self-reported health exposures and outcomes can be substantial, particularly for sensitive topics in sexual and reproductive health.^1,2 This is particularly true for abortion, as indicated by studies that compare survey responses to patient medical records.^3-–6 These studies find that directly asking participants about their abortion experience, in contexts where abortion is legal, results in under-reporting of up to 70%.⁷ In countries where abortion is illegal, under-reporting may be even greater. Unsafe abortion contributes to maternal mortality around the world,⁸ particularly in places where abortion is illegal. But without accurate estimates of the size of the population affected, effective policy and programmes cannot be developed or targeted.

Alternative methods for measuring abortion aim to improve accuracy by better protecting the privacy and confidence of the respondent. Some of these methods include the random-response technique (RRT), the Anonymous Third Party Reporting (ATPR) method, the Sealed Envelope or Ballot-Box method and various other question administration strategies.^9–16 Each of these strategies, however, has its limitations due to the time required for administration or the complexity of design, and they continue to result in abortion under-reporting.¹²

The list experiment is used in political science and social psychology to increase disclosure of truthful answers on sensitive subjects.^17–19 The method ensures that the interviewer cannot know whether a respondent has exhibited the behaviour of interest, because the only information reported is a number. In this way, the list experiment overcomes barriers to reporting that result from women’s reluctance to explicitly acknowledge having had an abortion, due to the presence of shame or perceived judgment by others.²⁰ Also known as the ‘item count technique’, list experiments have been used to measure illicit drug use,¹⁷ racism,¹⁹ risky sexual behaviour,²¹ drug violence and gang affiliation,²² among other subjects. However, to the best of our knowledge, this method has not been used to measure abortion or any health issues other than risky sexual behaviour.

We tested the list experiment as a new method for measuring abortion in Liberia, a country where maternal mortality is high,^23,24 contraceptive use is low²⁴ and abortion is illegal.²⁵ Poor access to contraceptives suggests that unintended pregnancy may be common in this setting, creating a need for abortion. We know that unsafe abortion contributes substantially to maternal mortality—13% of maternal deaths worldwide are attributed to unsafe abortion.²⁶ Abortion incidence is thus likely to be high in contexts like Liberia, and improved data could have a large impact on improving women’s health. The only previous representative estimate of abortion in Liberia is captured in the Demographic and Health Survey (DHS) data from 2007, where 6% of the female population reported having had an abortion²⁷ when asked directly in a face-to-face survey.⁵ Given the Ebola epidemic that has overwhelmed the country’s already weak health system over the past year, there is an even greater need for accurate information about women’s health needs to effectively target the rebuilding of health services begun in earnest in Spring 2015.

Methods

This study was conducted in June and July 2013 before the Ebola epidemic, by six female and six male Liberian enumerators among a representative sample of women from Liberia’s capital, Monrovia, as well as from Bomi County, a neighbouring rural area. Using the sample() function in the R software platform [http://www.R-project.org/], 50 enumeration areas (EAs) (2·5%) out of the 1967 in Monrovia and 126 EAs (46%) out of the 273 in Bomi County were randomly selected with probability proportional to size. EAs are spatially defined, and were created for the 2008 National Census in Liberia.²⁸ Within each sampled EA, survey teams listed all households using mobile global positioning system (GPS)-enabled maps, and conversations with local residents. Starting from a random household in the EA, enumerators then selected every fourth household for interview. A household, defined as a group of individuals who eat from the same pot, was eligible if it contained at least one woman of reproductive age (15–49 years). Within each household, one woman was randomly selected for interview based on the most recent birthday.²⁹ Verbal informed consent was obtained from both the household head and the female respondent. The target sample size of 3 200 was arrived at based on previous conservative estimates of induced abortions in Liberia,³⁰ and to allow for estimation of population levels and associated characteristics within a margin of error of 10%. This study was approved by the ethical review boards of the Liberian Ministry of Health and the University of California, San Francisco.

Questionnaire design and administration

The questionnaire was designed on the Open Data Kit (ODK) platform, and assessed basic demographic characteristics including age, ethnicity, education, income, religion, marital status and parity, followed by several embedded list experiments. The universal format of the list experiment entails a sample of respondents that is split into two groups, treatment and control. The control group is read a list of non-sensitive experiences and asked to report a count of how many of the experiences they have had, without indicating the specific events. The treatment group is read the same list of non-sensitive items, with an added sensitive item (e.g. abortion). They are similarly asked to report only how many of the events they have experienced, not which ones. Then, a difference-in-means calculation between the average numbers provided by both groups provides an estimate of the population proportion of individuals who have experienced the sensitive event.

In an effort to increase the precision of this estimate (and to better utilize the full sample), an extension to this method has been developed called the ‘double list experiment,’¹⁸ which was used in this study. This method is the same as the basic method, except that there are two control lists. Both groups are asked both lists, with the sensitive item randomly added to one list for each group. Under this design, each group functions as the control for the other, and together, the two difference-in-means estimates can be averaged to provide a more precise estimate of the population proportion.¹⁸

The first list in the Liberia survey was a practice list asking about foods eaten in the past week, to ensure respondent comprehension of the list experiment method, followed by the two abortion-specific experimental lists as follows:

List A: ‘Here is a list of three things that some people have done. Please listen to them and then tell me HOW MANY you have experienced. Do not tell me which ones, just tell me how many. Here are the three things:

1. I have been to a prenatal (‘big belly’) clinic.

2. I have had an Xray in a city hospital.

3. I have had malaria.

Now, how MANY of these have you experienced? None, one, two, or all?’

List B: ‘Now I am going to read you another list. Please listen to all of the things and then tell me HOW MANY you have done. Again, not which ones, just how many.

1. I have taken a vaccination.

2. I have used an ambulance to reach the city hospital.

3. I have visited a health clinic.

Now, how MANY of these are true for you? None, one, two, or all?’

Half of the sample received the control version of List A as above, as well as a treatment version of List B with abortion (‘take out belly’) added as a fourth item. The other half of the sample received List B as above, and a treatment version of List A with abortion added as a fourth item. At the time each survey began, the tablet generated a random number that was then used to allocate the set of lists for that respondent. In this way, neither the researcher nor the enumerator could influence (intentionally or not) the randomization process. The expectation is that randomization of the lists will result in the balanced distribution of characteristics that could influence experience of these events—sensitive and non-sensitive alike—between the two groups.

Several design principles are important to consider in reducing the variance of the resulting population proportion estimate of induced abortion.¹⁸ The items on the two lists were deliberately chosen to be positively correlated with one another—i.e. for any item on List A that a respondent would have experienced, there is a corresponding item on List B that would also likely be true for that respondent, or vice versa. In contrast, items within a list are negatively correlated such that it is relatively unlikely that a respondent will answer ‘yes’ to all items (‘ceiling effect’), or ‘no’ to all items (‘floor effect’).¹⁹ This is to protect the confidentiality of the response, such that any one respondent will ideally say yes to approximately half of the list items, and therefore study personnel cannot know which specific items are true for any individual respondent. In the lists used above, items 1 and 3 in both are thought to be true of many women in Liberia, whereas item 2 in both lists is thought to be true for few women.³¹ It is not necessary to pick non-sensitive list items with high and low prevalence—in fact, it may be preferable to avoid this given the consequent risks of ceiling or floor effects.¹⁸ The most important considerations are the within-list negative correlation and between-list positive correlation. However, due to the limited number of routine women’s health services offered in Liberia, the candidate list of items for each list was constrained in this study.

To provide some estimate, however imperfect, of the relative performance of the list experiment measure, a small comparison study was conducted within a sub-sample of respondents. A randomly selected sample of 600 participants was drawn from the sample of respondents who had provided their mobile number for future contact (1548 respondents, 48% of full sample). Two study enumerators called all 600 mobile phone numbers over the course of 2 weeks. Each number was tried up to three times, resulting in a final comparison sample of 275 respondents (46% response) who were asked verbally, over the phone, about each of the list items individually.

Data analysis

The lifetime cumulative incidence of abortion can be estimated for women of reproductive age in Liberia using the average of two difference-in-means calculations, one for List A and one for List B:

$$\pi =({\Sigma \text{Y}}_{\text{T}=\text{1},\text{i}})/{\mathrm{N}}_{\text{1}}–({\Sigma \text{Y}}_{\text{T}=0,\text{i}})/{\text{N}}_0$$

where π represents the proportion of the population that has experienced the sensitive event (i.e. abortion), Y is the number of items reported by each respondent, T represents the version of the list each individual received (T = 1 is the treatment list, T = 0 is the control list), N₁ represents the number of individuals that received the treatment version of the list, and N₀ represents the number of individuals that received the control version of the list (N₀ = N – N₁). A limitation of this formula is that, although unbiased, it can produce estimates that are outside the 0 to 1 range.¹⁸ Several alternative estimators have been developed to account for this, including a truncated estimator, a piecewise estimator and a maximum likelihood estimator.^18,32 Each of these estimators performs differently in terms of bias and precision, depending on the prevalence of the sensitive item. The confidence interval for the combined list estimate was calculated using linear mixed models with nested random effects for each level of clustering, to generate an estimate of the variability of the population parameter that accounts for the multi-stage sampling process. Differences in list estimates were also assessed by age of respondent, to generate estimates of abortion incidence by age. Only four respondents had missing data for list questions, and these women were excluded from analysis. Data management and analyses were conducted in the R statistical platform and in Stata version 13.

Ethics approval

This study was approved by the ethical review boards of the Liberian Ministry of Health and the University of California, San Francisco (IRB# 13-11055). Due to low literacy in the population, verbal informed consent was obtained from all household heads and from all female survey participants.

Results

A total of 3291 women between the ages of 15 and 49 years participated in the survey (1917 rural, 1374 urban), for a response proportion of 95%. On average, women in the sample were 30 years old, had three children and most were married or living with a partner (Table 1). Several key demographic characteristics differed by urban and rural residence, including parity, religion, educational attainment, and relationship status (Table 1), and thus we present results stratified by place of residence.

Table 1.

Demographic characteristics of full study sample, overall and stratified by urban or rural residence, as well as by validation sub-sample

Characteristic	Overall(n = 3291)	Urban sample (Monrovia)(n = 1374)	Rural sample (Bomi County)(n = 1917)	Validation sub-sample(n = 275)	T-testP-value for validation sub-sample vs overall sample
Means, ± SD
Age, in years	30 ± 0.2	28 ± 0.2	32 ± 0.2	31 ± 0.7	0.17
Parity	3 ± 0.1	2 ± 0.1	4 ± 0.1	4 ± 0.2	0.01
Persons living in household	7 ± 0.4	7 ± 1	7 ± 0.1	8 ± 0.3	0.47
Monthly household income, USD	$59 ± 7	$90 ± 16	$37 ± 3	$52 ± 9	0.06
Proportions, %
Religion, %
Muslim	28	9.5	41	27	0.27
Christian	71	90	58	67	0.19
Other	1	0.5	1	6	0.53
Education, %
None	38	20	51	41	0.27
Some primary	19	16	21	22	0.14
Completed primary	16	19	14	14	0.31
Traditional education	2	1	4	1	0.42
Some/all high or trade school	21	36	9	14	0.10
College or university	4	8	1	1	0.09
Marital status, %
Single	26	37	17	21	0.70
Living with partner	35	37	33	37	0.46
Married	32	21	40	34	0.45
Divorced/separated	3	3	5	4	0.39
Widowed	4	2	5	4	0.99

The list experiment difference-in-means estimates are shown in Table 2. For the entire sample, the list experiment estimates that 32% of women (95% CI: 29-34) have had an abortion at some point in their lives. (Utilizing an alternative estimator, the truncated estimator, to bound the estimate between 0 and 1, the list experiment estimate for the population changes to 32.2%—essentially unchanged.) Among urban women, this estimate is 26% (95% CI: 22-30), and among rural women it is 36% (95% CI: 32-39) (P-value for difference < 0·001). For each list and for each possible number of indicated items, we report the proportion of respondents reporting at least that number in Table 3. Rows 1 and 3 state the proportion of respondents that reported each number of list items for those read the list with abortion (Row 1) and without (Row 3). Rows 2 and 4 report the proportion of respondents reporting at least that number of items for each list. For instance, 100% of respondents reported zero or more items for each list, because zero is the minimum possible. In the table for List A, in Row 2, Column 1, the number is 0·961. This means that 96·1% of respondents reported at least one list item as true for themselves on the list containing abortion. Row 4, Column 3 records that 10.7% of respondents reported that at least three items were true for them on the version of List A without abortion. If there is no under-reporting, we should always observe that the proportion reporting at least x items is larger in the treatment list (Row 2) than the control list (Row 4). The bottom right cell in the ‘Sum’ column reports the difference in means estimator (equivalent to that reported in Table 2). The negative number in List A, Row 5 (−0·008), suggests that some respondents may have misrepresented their answers on the ‘how many’ questions; in other words, there may have been a small degree of under-reporting on one or more list items.¹⁸ This may imply that for some small number of respondents, adding abortion to the list caused them to claim they have not experienced one of the health events listed, when in fact they have. By summing the differences reported in Row 5 of Table 3, one arrives at the same difference-in-means estimate—albeit calculated in an arithmetically alternative manner.

Table 2.

Estimates of the percentage of women who have had an abortion. Results from list experiment estimators, by each list (A and B), as well as combined and by age category; 95% confidence intervals are calculated using linear mixed models that account for clustering at each level of the sampling scheme

	List A estimate	List B estimate	Average of Lists A & B	95% CI	n
Overall	28	35	32	29–34	3291
Urban	24	28	26	22–30	1374
Rural	31	40	36	32–39	1917
By age (in years)
≤ 20	29	13	21	15–28	580
20–29	22	38	30	26–35	1205
30–39	29	44	36	31–42	933
40+	33	39	36	29–43	571

Table 3.

Detailed assessment of response proportions by number of reported items in the entire sample (urban and rural), by list

List A
		Number of reported items
Estimated proportion	Source	0	1	2	3	4	Sum
Row 1	List with abortion	0.039	0.278	0.446	0.168	0.069	1.000
Row 2	Proportion at least*	1.000	0.961	0.683	0.237	0.069	–
Row 3	List without abortion	0.031	0.376	0.486	0.107	0.000	1.000
Row 4	Proportion at least*	1.000	0.969	0.593	0.107	0.000	–
Row 5	Row 2 minus Row 4	0.000	−0.008	0.090	0.130	0.069	0.281

List B
		Number of reported items
Estimated proportion	Source	0	1	2	3	4	Sum
Row 1	Treatment list	0.052	0.241	0.451	0.178	0.078	1.000
Row 2	Proportion at least*	1.000	0.948	0.707	0.256	0.078	–
Row 3	Baseline list	0.055	0.358	0.484	0.103	0.000	1.000
Row 4	Proportion at least*	1.000	0.945	0.587	0.103	0.000	–
Row 5	Row 2 minus Row 4	0.000	0.003	0.120	0.153	0.078	0.354

*Proportion reporting at least this number of items for the specified list.

When list responses were assessed by age of the respondent, the proportion of the population within each age category that is estimated to have experienced an abortion increases with each decade, from 21% among those under age 20, to 36% among those over age 40 (Table 2). Respondents in the comparison sub-sample were similar overall on measured characteristics to the full study sample, differing only in the number of children: validation participants had one additional child, on average, as compared with the full sample (Table 1). Among this sub-group, in which list questions were asked individually via telephone, 43% of women admitted to having had an abortion. When restricted to this same validation sub-group, the list experiment estimate for abortion was 40%. The non-sensitive list items performed mostly as expected, in that a large majority of women had attended a prenatal clinic (79%), had malaria (96%), been vaccinated (97%) and had visited a health clinic (90%). Similarly, fewer women had received an Xray (20%) or used an ambulance (15%). We also observed negative correlation between responses to items within the same list, as expected.

Discussion

We found that implementing the list experiment to measure a sensitive health event, abortion, in a setting where abortion is illegal, proved feasible among a sample of 3219 women in Liberia. Use of this method indicates that a substantial minority of women in our sample had ever had an abortion. The only previous representative estimate of abortion in Liberia was measured via face-to-face interviews with the Demographic and Health Survey, which estimated that 6% of women in Liberia have had an abortion.²⁷ The estimate generated from the list experiment here is an order of magnitude larger, which suggests that the method removes some of the pressure on respondents to under-report. Notably, these results are consistent with estimates the authors obtained from clinicians working in Monrovia and Bomi County health facilities. This estimate may be generalizable to similar contexts elsewhere.

Several important limitations of this study include the lack of a gold-standard reference for validation of the list results, such as comprehensive medical records. This suggests that our estimate may only represent a lower bound on the true population proportion of women who have had an abortion. Another limitation is low literacy in the population,³¹ that may bias the results in unpredictable ways due to miscomprehension of the survey questions. There are also important strengths to this study. The large, randomly selected sample reduces the likelihood of chance or random error as primary determinants of the results, and increases the representativeness of the findings. The estimate that nearly one in three women in Liberia has experienced an abortion is consistent with anecdotal evidence, and further supported through a follow-up sub-sample.

In our sub-sample of participants with cellphones who could be reached to complete a phone survey, 43% reported having had an abortion when asked directly, similar to the list experiment estimate. It is notable that this is much higher than the previously obtained DHS estimate of 6%. The relative privacy and anonymity conferred by the phone modality may explain the greater willingness to report honestly. A study in the USA found that asking about abortion via telephone interview resulted in 22% increased odds of honest reporting as compared with in-person interviews.⁷ Further, respondents may also have been more willing to speak honestly because of the previous face-to-face meeting with an enumerator, which could have established trust. At the time of this survey, mobile phone penetration in Liberia was estimated at 42% of the population.³³ As coverage increases, relying on a direct mobile phone-based methodology for asking about sensitive health experiences may become feasible. However, until then, the estimates obtained may not be representative given that women with cellphones could differ in important ways, unmeasured in this study, from the general female population. Further, the low response rate of this methodology limits its utility. Moving forward, however, validation studies comparing the accuracy of women’s answers via mobile phone would improve our ability to evaluate the usefulness of this modality for abortion measurement in similar contexts.

The estimates of cumulative lifetime incidence of induced abortion suggest a need to reassess the resources currently directed toward abortion in places like Liberia. Abortion, when legal and performed with the requisite training and equipment, is safe and effective; however, when performed illegally, abortion can pose serious health risks.³⁴ As part of the national commitment to reduce the high maternal mortality rate in Liberia,³⁵ particularly in the wake of the Ebola epidemic that further weakened maternal health services, greater attention and resources must be directed toward reducing unsafe abortions. Increasing access to contraception across the country could help toward this goal. Further, a re-evaluation of current policy toward the training of clinicians and resources allocated to them to provide post-abortion care may be in order, given the magnitude of the population affected. This study did not ask about method of abortion, or about abortion-associated morbidity, so assessments of these aspects of abortion in Liberia in the future are important next steps for improving understanding of the scope of the issue, the specific health risks faced by women and the full burden of morbidity and mortality that abortion confers in this population.

In conclusion, researchers’ inability to elicit truthful answers to sensitive questions is a significant and persistent challenge in many areas of research.¹⁸ The list experiment reduces under-reporting of a sensitive experience, abortion, among a representative sample of respondents. The estimator holds promise for measuring the size of specific populations with sensitive health needs, burdens or risk factors, and may prove useful for public health planning and resource allocation. Multivariate regression methods have recently been developed to explore how the probability of experiencing the sensitive item varies by respondent characteristics.³² This will greatly add to the value of the list experiment as an epidemiological tool. Future work should explore the utility of this estimator across diverse public health issues and populations, from history of sexually transmitted infections, experiences with intimate partner violence or other abuse, provider and patient biases and more, with substantive attention paid to validation of the measure.

Funding

This work was supported by the Clinton Health Access Initiative, which received direct funding from the Swiss Agency for Development and Cooperation (SDC) for the implementation of research on maternal mortality, of which this survey was a part. HM was supported by a trainee research award from the Society for Family Planning, and by training grant #R25 GM56847 from the National Institute of General Medical Sciences. Study funders had no role: in the study design; in the collection, analysis and interpretation of data; in the writing of the report; or in the decision to submit the article for publication.

Author contributions

HM conceived of the design. HM, CG and MM secured funding for the project through grants. HM, MM, LB, BD and YZ implemented the survey. HM analysed the results, together with EV. HM drafted the paper, with substantial editorial input from CD, EV, CG and RH. All authors have reviewed and approved the final draft. This material has not been published previously in a substantively similar form.

Conflict of interest: None declared.