Assessing the reliability of the retrospective reproductive calendar: evidence from urban Kenya

Abstract

The reproductive calendar is a data collection tool that collects month-by-month retrospective histories of contraceptive use. This survey instrument is implemented in large-scale demographic surveys, but its reliability is not well understood. Our analysis helps to address this research gap, using longitudinal panel data with overlapping calendars from urban Kenya. Our findings indicate calendar data collected in 2014 underestimated 2012 reports of current use by five percentage points. And while the total number of women reporting at least one episode of contraceptive use was similar across the two calendars (67 versus 70 percent), there was notable disagreement in contraceptive behavior when comparing the histories of individual women; less than 20 percent of women with any contraceptive use reported the exact same pattern of use in both calendars. Low calendar reliability was especially apparent for younger women and those with complicated contraceptive histories. Individual-level discordance resulted in a small difference in 12-month discontinuation rates for the period of calendar overlap; when surveyed in 2014, women reported a 12-month discontinuation rate of 39 percent, compared to a rate of 34 percent reported in 2012. When using retrospective calendar data, attention must be paid to the potential for individual reporting errors.

Background

Access to family planning is critical for the health, well-being, and reproductive autonomy of women and their families across the developing world (Cleland et al. 2006, Cleland et al. 2012, Cohen 2009, Ahmed et al. 2012, Reichenbach 2009). Yet throughout sub-Saharan Africa, contraceptive prevalence remains low and as many as one in four women have an unmet need for family planning (Cleland and Machiyama 2015). Among all women with an unmet need for family planning, a substantial portion - about 38 percent - previously used a contraceptive method but discontinued their method (Jain et al. 2013). The high percentage of unmet need that is comprised of prior contraceptive users reveals the importance of not only facilitating first-time contraceptive use but also helping women to maintain method use over time. It is estimated that between one fifth and nearly two-thirds of all women using modern reversible contraceptive methods discontinue their method within 12 months of initiation (Blanc et al. 2009). The ability to reliably measure discontinuation rates, as well as reasons for discontinuation, is critical to the success of international family planning programs.

The reproductive calendar is an individual level survey instrument that collects month-by-month retrospective histories of contraceptive use, typically for a period of six years prior to the interview (Curtis and Blanc 1997, Bradley, Winfrey, and Croft 2015). Women who discontinue a contraceptive method are also asked the primary reason for discontinuing. The calendar was first developed in the 1980s and is frequently implemented by large-scale population-based surveys such as the Demographic and Health Survey (DHS). Since its development, the quality of data collected by the reproductive calendar has been assessed periodically, with varying results.

The first evaluations of reproductive calendar data quality were conducted in the 1980s in Latin American countries (Goldman, Moreno, and Westoff 1989, Rosero-Bixby and Oberle 1989, Westoff, Goldman, and Moreno 1990). In a 1986 study of approximately 5,000 Peruvian women, the calendar instrument was used to estimate contraceptive use over the previous five years and these data were then compared to current contraceptive use data from a 1981 survey. With the exception of the pill and sterilization, the reconstructed contraceptive histories consistently under-estimated contraceptive use across all method types, although differences between calendar and current use data were very modest (within 1.1 percentage points) for most methods and only statistically significant for a few methods including injectables, diaphragms, and condoms (Goldman, Moreno, and Westoff 1989). A similar study conducted during the same time period in the Dominican Republic among 2,311 women found even less difference in overall contraceptive prevalence between calendar data and previous estimates of use, possibly due to a shorter recall period (3 years versus 5 years) and a different method mix between the two countries, with the Peruvian method mix more dominated by traditional methods (Callahan and Becker 2012, Westoff, Goldman, and Moreno 1990). A third study was conducted during this same time period in Costa Rica among a sample of nearly 900 women and compared data from the reproductive calendar with various other data sources. Consistency between data sources varied; notably, although data quality was reasonable overall, shorter episodes of use were less reliably reported by the calendar than longer episodes of use (Rosero-Bixby and Oberle 1989). All three of these studies, although limited in geographic scope, suggested moderate reliability of the calendar instrument for most women and most methods, and these earliest assessments of the calendar found it to be an improvement over other existing retrospective data collection efforts designed to measure contraceptive dynamics.

As more calendar data were collected in additional regions, the DHS program was able to conduct further assessment of the quality of calendar data. The first of these subsequent assessments, using data from six countries¹ and published in 1997, supported the earlier findings from Latin America; estimates of current use measured in prior surveys were very similar to subsequent retrospective estimates produced by reproductive calendar data. Where differences did exist, there was no evidence of systematic underreporting on contraceptive use in the calendar (Curtis and Blanc 1997). The second DHS assessment, using data from eight countries² and published in 2009, differed from the 1997 report; although Bradley et al did not find large differences between current prevalence estimates and the calendar data, differences were more pronounced than in the 1997 report and Bradley and colleagues found that the reproductive calendar consistently underestimated prior contraceptive use (Bradley, Schwandt, and Khan 2009). In such scenarios, where use is underestimated, the authors concluded that not all contraceptive use will be captured, and discontinuation rates may be underestimated by calendar data. In a subsequent 2015 DHS report analyzing data from 37 countries (some of which contributed multiple calendars), Bradley found far greater discrepancies between current estimates and subsequent calendar data (Bradley, Winfrey, and Croft 2015). In half of all calendars assessed, the calendar significantly underestimated prior contraceptive use. Further, these differences were meaningful; underestimation of prior contraceptive use ranged from approximately from 10 to 50 percent, depending on the region. Overall, calendar data from sub-Saharan Africa were found to underestimate prior use by the largest amount. Further, short-acting contraceptive methods were reported much less reliably in the calendar, compared to long-acting and permanent methods. These prior assessments of calendar data did not measure whether contraceptive episodes were reported at differing times (displacement of reporting rather than omission) or whether participants recalled prior episodes of use but with method substitution (i.e. the contraceptive use is reported, but with a different method in the two data sources). Additionally, we are not aware of prior calendar assessments that found that retrospective measures led to overreported contraceptive use (i.e., instances where a participant reported episodes of contraceptive use in the retrospective calendar that are not found in the comparison data source).

In each of these prior assessments of the quality of calendar data, a current estimate from one sample of women is compared to retrospective reports from a subsequent (and different) sample of women. Bradley and colleagues suggest that the best way to assess calendar reliability is to interview the same women more than once and then compare the two sources of information (Bradley, Winfrey, and Croft 2015). Only two studies to date have been able to assess the reliability of the reproductive calendar by comparing data collected at two different time points within the same sample of women. Strickler, Magnani et al. (1997) assessed reliability using a sub-sample of women (n=1694) who participated in two overlapping DHS surveys from Morocco. Authors found substantial agreement between the two calendars when comparing contraceptive use status at the aggregate level. However, this aggregate-level agreement masked large individual-level differences; among women who reported at least one month of contraceptive use in the first calendar, fewer than a third of participants provided matching reports in both calendars. And among episodes of discontinuation that could be matched between the two calendars, the reasons for discontinuation in more than a third of these episodes were discordant across the two calendars. Of note, for several measures of reporting consistency, older women (ages 40 and above) performed better. And those women with more complicated reproductive and contraceptive histories were significantly less likely to demonstrate strong reporting reliability. Further, Strickler and colleagues report that reliability did not appear to decline over the three-year period for which the two calendars overlapped (Strickler JA et al. 1997).

The second study to assess reliability via overlapping calendars used DHS data collected from a sample of (primarily) rural women in Bangladesh and compared data on contraceptive use measured in 2006 with data from a subsequent panel of the same women conducted three years later (Callahan and Becker 2012). Although the authors were not able to evaluate individual-level reliability due to a short period of calendar overlap (three to five months), aggregate-level agreement of contraceptive use status and reasons for discontinuation were somewhat weaker in the Bangladesh study, as compared to the Morocco study. In agreement with the Morocco study, participants in Bangladesh were less likely to provide consistent retrospective reports if they had a more complicated contraceptive history. Bangladesh participants were far more likely to reliably report past use if using a long-acting or permanent method during the baseline interview month, compared to women not using such methods at baseline, although this result was not statistically significant.

The bigger picture to be drawn from these studies is that calendar data may provide reasonable estimates of contraceptive behavior among specific types of women, for example those who are older, those with less complex histories, and those using long-acting or permanent methods. However, contraceptive users who are younger, those with more complex histories, and those using short-acting methods may be less able to provide accurate recall of previous contraceptive use. Inaccurate measures of discontinuation among younger women are concerning given the knowledge that sexually active adolescent and young women in developing countries face more dire consequences of unintended pregnancy including loss of educational attainment, stigma of an early or out-of-wedlock pregnancy, and increased exposure to sub-optimal peri-natal services (Klein 2005).

The limited number and geography of prior studies with overlapping data from the same sample of women, however, has contributed to on-going debate regarding the quality of reproductive calendar data. As Callahan and Becker (2012) observed, the reliability of the calendar method “remains largely unknown.” The largest hindrance to conducting rigorous research on the reliability of calendar data centers on a dearth of appropriate longitudinal datasets with overlapping panels of calendar surveys. Callahan and Becker suggest a two-year period of overlap is ideal for assessing consistency in reports of method type, use duration, and switching; of all prior studies, only the Morocco study contained a period of overlap greater than two years (Callahan and Becker 2012, Strickler JA et al. 1997).

The present study capitalizes on the availability of recent longitudinal data from a large sample of urban Kenyan women. These women participated in a demographic survey that included use of the reproductive calendar and they were surveyed at two time points, resulting in calendars that overlapped by a period of 33 months. The objective of this study is to assess both aggregate and individual-level reliability of retrospectively reported contraceptive behavior collected with the reproductive calendar. We also aim to investigate predictors of reliable reports of contraceptive use. We hypothesize that younger women, and those women with more complex reproductive histories, will provide less reliable reports of prior contraceptive use. Additionally, this study is broadly modelled after Strickler’s 1997 Morocco assessment, which will allow for comparison of results from a prior study in a contrasting geographic region.

Methods

Study Population

The data for this analysis come from The Measurement, Learning & Evaluation (MLE) Project, implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill. The MLE Project evaluated a Bill and Melinda Gates funded project designed to increase the contraceptive prevalence rate in select urban areas of Kenya, Senegal, Nigeria, and India. This analysis uses data from Kenya. In Kenya, the MLE Project collected individual-level data within a sample of 3,207 women of reproductive age between September and December 2012. Data were collected in three urban areas in Kenya (Nairobi, Mombasa, and Kisumu) and involved a multi-stage sampling design. For each selected household, all eligible women (ages 15 to 49) in the household were asked to participate in a detailed face-to-face interview with a trained female interviewer via an informed consent protocol. Respondents were asked about current contraceptive use, demographic characteristics, fertility desires, and exposure to family planning messages, among other things. A three-year retrospective reproductive calendar was conducted at the end of the woman’s questionnaire.

Another round of data collection was conducted between September 2014 and January 2015 among the same women interviewed in 2012. This survey also included implementation of the calendar instrument. These two calendar instruments – the one implemented in 2012 and the one implemented in 2014 – were identical and overlapped by a period of 33 months. The MLE calendar instrument used in these two surveys was very similar in content and structure to the DHS calendar and was implemented following standard procedures developed by the DHS, a detailed description of which is now available in Module 1 of the online DHS Contraceptive Calendar Tutorial. The enumerators implementing the MLE calendars were trained by a team of demographers from the Kenya National Bureau of Statistics with extensive experience conducting large-scale, population-based data collection who were trained on the MLE questionnaires over a period of seven days. These trainers then conducted an 11-day training of the enumerators, including instruction in interviewing techniques, field procedures, how to use the contraceptive calendar, and a detailed review of each questionnaire item. Data collectors also piloted all instruments in non-study clusters prior to commencing data collection.

Of the 3,207 women participating in the 2012 calendar, 2,421 were found and interviewed again in 2014, leaving 786 women (25 percent of the sample that participated in the first calendar) unmatched due to loss of follow-up between the two data collection efforts. Any women reporting inconsistent background characteristics across survey rounds were excluded from the final MLE dataset in 2014. Therefore, reports of age and education were concordant across surveys. Women included in the 2012 calendar were part of a panel survey that began in 2010, among women ages 15 to 49. Therefore, women in our analysis ranged in age from 17 to 51 in the 2012 panel and 19 to 53 in the 2014 panel.

Data Analysis

In our analysis, which broadly follows the Strickler et al. (1997) Morocco study, we first assess the degree of reliability or concordance observed between the two calendars. To assess reliability, we conducted a cross-tabulation to assess the proportion of respondents who provided the same answer (in both survey rounds) to questions about their contraceptive status and method type for the reference month (September 2012; September and October were the month of interview for most participants during the first calendar³). We assume that the contraceptive behavior reported in September 2012 of the first calendar is accurate because all participants completed the first calendar that month, or shortly thereafter, and are therefore reporting their current (or extremely recent) contraceptive behavior. This comparison of data for the reference month informs Tables 2 and 3. All subsequent tables are informed by the entire period (33 months) of overlap. We also conducted a cross-tabulation of participant responses related to their reason for method discontinuation, restricting to data supplied by participants with one or more occurrence of stopping method use or switching to another method. In order to account for marginal distribution, we calculated a Kappa statistic which considers the observed and expected sum of proportions in agreement. Fleiss, 1981, defines the Kappa statistic as:

$$K=\frac{\left((\text{proportion}\hspace{0.17em}\text{of}\hspace{0.17em}\text{observed}\hspace{0.17em}\text{agreement})-(\text{proportion}\hspace{0.17em}\text{of}\hspace{0.17em}\text{agreement}\hspace{0.17em}\text{expected}\hspace{0.17em}\text{by}\hspace{0.17em}\text{chance}\hspace{0.17em}\text{alone}\right)}{1-(\text{proportion}\hspace{0.17em}\text{of}\hspace{0.17em}\text{agreement}\hspace{0.17em}\text{expected}\hspace{0.17em}\text{by}\hspace{0.17em}\text{chance}\hspace{0.17em}\text{alone})}$$

TABLE 2.

Comparison of contraceptive use status during the “reference month” for the 2012 survey, 2012 and 2014 MLE surveys, urban Kenya

	Contraceptive status reported in 2014
Status reported in 2012	Not using contraceptives	Using contraceptives	Pregnant/pregnancy outcome	Total (Percent)
Not using contraceptives	673	197	24	894
	(75.3% agreement)			36.9%
Using contraceptives	301	1044	39	1384
		(75.4% agreement)		57.2%
Pregnant/pregnancy outcome	12	15	116	143
			(81.1% agreement)	5.9%
Total (Percent)	986	1256	179	2,421
	40.7%	51.9%	7.4%	100.0%

Kappa Statistic: 0.56; Off-diagonal proportion: 24.3%

Table 3.

Comparison of contraceptive method used during the reference month (September 2012) as reported in the 2012 and 2014 calendar, MLE surveys, urban Kenya^*

	Current method reported in 2014
Method reported in 2012	Injectable	Implant	Pill	Female sterilization	IUD	Total (Percent)
Injectable	321					355
	(86.1% agreement)	16	15	1	2	43%
Implant		187				213
	22	(85% agreement)	3	0	1	26%
Pill			102			131
	20	6	(70.8% agreement)	1	2	16%
Female Sterilization				57		62
	2	1	0	(91.9% agreement)	2	8%
IUD					48	58
	4	4	1	1	(82.8% agreement)	7%
Total (Percent)	369	214	121	60	55	819
	45%	26%	15%	7%	7%	100.0%

Kappa Statistic: 0.71; Off-diagonal proportion: 22.6%

^*Calculated among women who reported using a method in the reference month in both surveys.

Landis and Koch (1977) recommend the following interpretation of the Kappa statistic (Landis 1977):

Greater than .80 = excellent reliability

0.41 – 0.80 = moderate to substantial agreement

0.01 – 0.40 = slight to fair agreement

0.00 = no better than chance alone

Additionally, we compare the percentage of women in each survey round who reported no reproductive events (pregnancy, birth, termination, and starting/stopping/switching method) or contraceptive use throughout the 33 months of calendar overlap. We also compare the mean and median number of events, contraceptive episodes (a period of one or more months in which the participant uses the same method), and total months of contraceptive use reported in each survey round. Further, we look at the percentage of these answers that agree in both survey rounds, which serves as a summary measure of individual-level reliability. We then expand this analysis by comparing responses across the two survey rounds if we tolerate incremental differences in the number of events, episodes, and months of use reported. We also consider how these individual-level measures vary based on the participant age, education, and method type reported in the reference month.

We also examine the reliability of the reported reason for contraceptive discontinuation. This portion of the analysis is conducted at the level of individual episodes of contraceptive use rather than individual women and is limited to episodes that matched in both survey rounds. Episodes were matched by searching for a non-blank entry in the column of the 2012 contraceptive calendar where reasons for discontinuation were recorded. For every non-blank entry, which signaled the end of a contraceptive episode, the corresponding month in the 2014 calendar was checked for a non-blank entry. If both calendars contained a non-blank entry for the same month, then that same month’s value for contraceptive method was checked for both calendars. If the methods matched, then the episodes were matched. If the 2014 column where reasons for discontinuation were recorded did not have a non-blank code for the same month as the 2012 calendar, then we searched outward from that month for a non-blank beginning with one month prior, then one month after, then two months prior, then two months after, then three months prior, and lastly three months after, stopping the search only if a non-blank entry was found. Among the 181 matched episodes, a large number of reasons for discontinuation were collapsed into six categories (see Supplemental Table 1A) and a crosstab was run to compare the reported reasons between the two calendars.

Further, we investigated predictors of reliability using logistic regression. We considered the following outcomes in our logistic regression: whether the participant reported the same number of episodes of contraceptive use in both calendars, whether they reported the same total number of months of contraceptive use, and whether their 33 months of overlapping calendar data were an exact match, meaning the exact same pattern of contraceptive behavior was reported in both calendars for all 33 months of overlap. If the reports matched, the outcome variable was coded as “1”; otherwise “0.” Potential predictors that we explore include number of pregnancies reported in the 2012 calendar (0, 1, 2 or more), number of contraceptive episodes reported in the 2012 calendar (0, 1, 2 or more), as well as individual characteristics reported in the 2012 calendar including age (under 30, 30–39, over 40), education (none, primary, secondary), and method type, as reported in the reference month (September) of the 2012 calendar (long-acting and permanent methods (the intrauterine device, the contraceptive implant, and male or female sterilization) were coded as “1”; otherwise “0”).

Finally, to understand how individual-level reporting error might impact estimates of contraceptive discontinuation, we compare 12-month discontinuation rates in the period of overlap produced by the two calendars. The 12-month discontinuation rate is calculated using life table methods, following DHS analysis procedures. We define 12-month discontinuation based on those users who discontinue using a contraceptive method within 12 months of beginning use for each specific episode of use. Users who switch to another method are considered to have discontinued the previous method at the time of switching. All analyses are unweighted.

Results

As seen in Table 1, 25 percent of the sample was lost to follow-up between the first and second calendars. Table 1 provides a comparison of age and education levels between those women who completed both calendars and those lost to follow-up. While the two groups of women appear similar in terms of education, they vary in age distribution, with the matched sample appearing to have an under-representation of women between the ages of 20 and 29. Women in this age group may be more mobile due to college or university attendance, relocation for jobs following completion of education, or relocation following marriage; as a result of higher mobility, these women may have been more difficult to locate two years after the first calendar was implemented. The potential impact of age bias in the sample is described in the discussion section.

TABLE 1.

Age and education distribution of women who completed both calendars and the 25 percent of women who were lost to follow-up, 2012 and 2014 MLE surveys, urban Kenya

Characteristics	Completed both calendars	Lost to follow-up
Age in 2012
Under 20	6%	7%
20–29	42%	53%
30–39	33%	29%
40 or older	20%	11%
Education in 2012
None	4%	5%
Primary	47%	46%
Secondary +	49%	49%
(N)	2421	786

Table 2 provides a comparison of data on contraceptive status during the reference month (September 2012) in the two calendars, described using three mutually exclusive categories: Not using contraceptives, using contraceptives, and pregnant/pregnancy outcome⁴. As shown, the marginal distributions of responses from the two surveys vary across these three categories, with the largest discordance seen for women reporting contraceptive use. The Kappa statistic of .56 suggests moderate agreement. The 2014 calendar responses underestimated prior contraceptive use by more than five percentage points relative to the 2012 calendar responses. In other words, women were less likely to report they were using contraception in the reference month when reporting in 2014 (52 percent reported use) versus reporting use in the reference month in 2012 (57 percent reported use); only 75 percent of respondents reporting use in the reference month in 2012 gave the same report two years later. Those women who reported a pregnancy or pregnancy outcome during the reference month had the highest reliability at follow-up, with 81 percent of women who reported a pregnancy/pregnancy outcome during the reference month in 2012 also reporting that they were pregnant during that month when interviewed in 2014. Across all categories, nearly one in four women reported a different status between the two data collection points (off-diagonal proportion of 24.3 percent).

Of note, part of the observed window in the 2012 calendar contains the few months immediately preceding the interview. Therefore, it is possible that some of the discrepancies observed between the two calendars could be due to a pregnancy that was not yet discovered. However, few women (n=63) reported they were pregnant in the reference month in 2014 but not in 2012 and these women only account for 11% of the off-diagonal responses in Table 2. Therefore, we suggest undetected pregnancy plays a relatively minor role in the low level of overall agreement between the two calendars.

Among the 1044 women who reported using a contraceptive method during the reference month in both survey rounds, we examined the consistency of reporting the same specific method. Table 3 presents a comparison of reports of the specific method used during the 2012 reference month. There is moderate to substantial agreement in the proportion of women reporting the same method in both survey rounds, as demonstrated by a Kappa statistic of 0.71. However, as in Table 2, the off-diagonal proportion suggests some degree of inconsistency between the two calendars. More than one in five women reported use of a different contraceptive method for the reference month, when comparing the two calendars (off-diagonal proportion of 22.6 percent). The least reliable reports came from women who - during the reference month - reported using condoms (53 percent concordance) or other methods (47 percent concordance), which included female condom, lactational amenorrhea, standard days method, rhythm, withdrawal, and emergency contraception, in 2012. Women who reported in 2012 that they were using long-acting or permanent methods had much higher reporting reliability (between 83 and 93 percent concordance). Women who reported in 2012 that they were using pills provided consistent reports in 2014 among 71 percent of participants. Among women using the most popular contraceptive method (the injectable), agreement between reporting periods was 86 percent. Despite an off-diagonal proportion of 23 percent,, the marginal distributions of the different contraceptive methods were fairly similar from the two calendars..

We examine whether women consistently report the same number of reproductive events (such as pregnancies, births, terminations, and initiating/switching/stopping contraceptive use), episodes of contraceptive use (a period of one or more months in which the participant uses the same method), or total number of months of contraceptive use in Table 4. These data provide a picture of the aggregate level of agreement across both calendars. As seen in Table 4, in both calendars, 45 percent of women reported no reproductive events. Among women with at least one reproductive event, women reported a median of two events in both calendars. Slightly more women reported no contraceptive use when surveyed in 2014; 30 percent reported no use at all during the period of overlap when surveyed in 2012 while 33 percent reported no use when surveyed in 2014. The mean number of episodes of contraceptive use in both calendars was 1.4 Also in close agreement, the median number of months of contraceptive use (among those women reporting any contraceptive use) was 25 months in 2012 and 24 months in 2014.

Table 4.

Comparison of behavior reported during the overlapping calendar period, 2012 and 2014 MLE surveys, urban Kenya

	2012	2014
Events	N=2421
Number reporting none	1084	1101
Percent reporting none	45%	45%
Among women reporting 1+ events	(n=1337)	(n=1320)
Mean number of events	2.4	2.5
Median of events	2	2
Episodes of contraceptive use	N=2421
Number reporting none	730	793
Percent reporting none	30%	33%
Among women reporting 1+ episodes	(n=1691)	(n=1628)
Mean number of episodes	1.4	1.4
Median of episodes	1	1
Months of contraceptive use	(n=1691)	(n=1628)
Mean number of months of use	23	23
Median number of months of use	25	24

^*The two contraceptive calendars overlapped by a period of 33 months

While the data in Table 4 convey a high degree of agreement across aggregate reports, these data mask substantial individual-level disagreement, as seen in Table 5. Only a little more than half (56 percent) of participants reported the same number of events in both calendars. This percentage rises to 87 percent if we tolerate an error of plus or minus one event and to 96 percent with an error of two events. A larger percent (62) of women were able to reliably report the same number of contraceptive episodes and this percent increases to 94 if we allow for a reporting error of one episode and to 99 percent if allowing an error of two episodes. Only 43 percent of women reported the same number of months of contraceptive use in both calendars. The percent of agreement on the months of contraceptive use rises steadily as we tolerate one or more months of error; however, even allowing for an error of plus or minus six months, only about two thirds of participants reliably reported their total number of months of contraceptive use.

Table 5.

Summary measures of the consistency of reporting contraceptive behavior during the overlapping calendar period, 2012 and 2014 MLE surveys, urban Kenya

	(N)	Percent
Reported same number of events	2412
Exact	1353	56%
Tolerance +/− 1 events	2090	87%
Tolerance +/− 2 events	2312	96%
Reported same number of episodes of contraceptive use	2412
Exact	1504	62%
Tolerance +/− 1 episodes	2261	94%
Tolerance +/− 2 episodes	2394	99%
Reported same number of months of contraceptive use	2412
Exact	1041	43%
Tolerance +/− 1 months	1180	49%
Tolerance +/− 2 months	1296	54%
Tolerance +/− 3 months	1410	58%
Tolerance +/− 6 months	1630	68%
Exact month-by-month record match among all women	2412
Exact match	772	32%
Exact month-by-month record match among women reporting 1+ episode of contraceptive use in 2012	1691
Exact match	303	18%

^*The two contraceptive calendars overlapped by a period of 33 months

A more stringent criteria is to require an exact month-by-month match between the calendars implemented in the two survey rounds. The bottom of Table 5 displays these results. Approximately one third of participants reported an exact record match across the two calendars, meaning the exact same pattern of contraceptive behavior was reported in both calendars for all 33 months of overlap. However, bearing in mind that 30 to 33 percent of our sample report using no contraception at any point during the 33 months of overlap, Table 5 also presents exact match data among a sub-sample of women who reported at least one month of contraceptive use during the 33 months when surveyed in 2012. Among this smaller sample of 1,691 women, only 18 percent (n=303) reported an exact record match between the two calendars. Of these 303 women, 101 were using a long-acting or permanent method and 129 were using injectables.

In Tables 6 and 7 we consider how background characteristics may impact the reliability of calendar data. Table 6 presents the percent of women reporting the same number of episodes and months of use and those with an exact match, stratified by age, education, and method type, as reported during the 2012 survey, with method type determined by the method reportedly used in September 2012 and calculated only among those women (n=1384) reporting use for that month. As shown, reporting reliability improves significantly as women ascend age categories. For example, 58 percent of women under 30 years of age consistently reported the number of contraceptive episodes, compared with 71 percent of women ages 40 and older. As another example, while only one in five women in the youngest age category had an exact record match, more than half of women 40 and above achieved this level of consistency. Notably, women over 40 were also more likely to be using no method, less likely to be using a short-acting method, and less likely to have a complex reproductive history (compared to women under 40) and this may explain, in part, their greater reporting reliability (data not shown). Those women using a long-acting or permanent method in the reference month in 2012 were also more likely to report the same total number of months of use across both calendars and were more likely to provide an exact record match, compared to women using short-acting methods. Participant education did not appear to impact reporting reliability in our sample.

Table 6.

Summary measures of the consistency of reporting contraceptive behavior during the overlapping calendar period, by select respondent characteristics, 2012 and 2014 MLE surveys, urban Kenya

	Percent who reported
Characteristic	(N)	Same # of episodes	Same # of months	Exact match
Age reported in 2012		***	***	***
under 30	1153	58%	33%	22%
30–39	794	62%	46%	32%
40 and above	474	71%	63%	54%
Education reported in 2012
None	96	64%	50%	34%
Primary	1135	61%	42%	31%
Secondary and above	1190	63%	43%	33%
Method type reported in 2012			**	**
Long acting or permanent method	385	59%	41%	26%
Short acting method	999	56%	33%	20%

^**Significant at p ≤ 0.01

^***p≤ 0.001

Note: Age, education, and method type are based on responses given during the 2012 survey, with method type determined by the method reportedly used in September 2012.

Table 7.

Adjusted odds ratios predicting reliable reporting of selected reporting outcomes among 2421 women, 2012–2014 calendar period, MLE surveys, urban Kenya

	Adjusted Odds Ratio (95% confidence interval)
Predictor	Reported same # of episodes	Reported same # of months	Reported an exact month-by-month match
# of pregnancies reported in 2012
0	ref	ref	ref
1	0.71*** (0.58, 0.86)	0.23*** (0.18, 0.29)	0.11*** (0.08, 0.15)
2 or more	0.71 (0.48, 1.05)	0.34*** (0.22, 0.54)	0.06*** (0.02, 0.14)
# of episodes of contraceptive use reported in 2012
0	ref	ref	ref
1	0.74*** (0.60, 0.92)	0.15*** (0.12, 0.19)	0.14*** (0.11, 0.17)
2 or more	0.19*** (0.14, 0.24)	0.11*** (0.08, 0.15)	0.01*** (0.005, 0.019)
Age in 2012
Under 30	ref	ref	ref
30–39	1.1 (0.91, 1.35)	1.8*** (1.43, 2.21)	1.6*** (1.25, 2.10)
40 or older	1.2 (0.90, 1.49)	1.7*** (1.28, 2.17)	1.6*** (1.24, 2.19)
Education in 2012
No education	ref	ref	ref
Primary	1.2 (0.74, 1.83)	1.2 (0.74, 1.99)	1.6 (0.93, 2.82)
Secondary	1.2 (0.76, 1.89)	1.2 (0.73, 1.95)	1.6 (0.90, 2.73)
Method type in 2012
Short acting method or non-use	Ref	Ref	ref
Long acting or permanent method	1.4** (1.12, 1.85)	2.3*** (1.75, 2.98)	3.23*** (2.27, 4.46)

^**Significant at p ≤ 0.01

^***p≤ 0.001

To assess the extent to which low reliability in younger women or those using long-acting/permanent methods might be a result of confounding, Table 7 presents adjusted odds ratios. In this table, we assess associations across several different determinants and outcomes. In addition to the background characteristics included in Table 6, we also explore the complexity of participant reproductive histories including the number of pregnancies and contraceptive episodes reported in 2012. The outcomes assessed in Table 7 (which increase in reporting stringency when reading from left to right) include whether participants reported the same number of contraceptive episodes, the same number of months of contraceptive use, and whether they provided an exact record match.

The results of our logistic regression indicate that reporting reliability is strongly associated with both number of pregnancies and number of contraceptive episodes. Women with one pregnancy are significantly less likely to provide consistent reports, across all three outcome measures, compared to women with no reported pregnancies. For example, women with one reported pregnancy in 2012 had one-ninth the odds (AOR=0.11; 95% CI=0.08, 0.15) of providing an exact match, compared to those women who reported no pregnancy in the overlapping period. And those with two or more pregnancies were less likely to report the same number of months of contraceptive use and less likely to report an exact match, compared to women reporting no pregnancies. For both pregnancy categories, the magnitude of the adjusted odds ratio increases as the stringency of the outcome increases, with the highest magnitude of association seen for the outcome of exact record match. Similar results were seen when examining the association between number of contraceptive episodes and the outcomes of interest. Women with one episode of use were significantly less likely to provide a reliable report across all three outcomes, compared to women with no contraceptive use reported in 2012. This association increased in magnitude for women with two or more contraceptive episodes and, as seen with pregnancy, the magnitude of the association also increased with greater stringency of the outcome.

In addition to assessing the role of complex reproductive and contraceptive histories, Table 7 displays results examining participant characteristics include age, education, and method type. We examined age by comparing women in the upper two age categories (30–39 and 40 or older) to women under 30 years of age. Age was not associated with reliably reporting the same number of contraceptive episodes but was significantly associated with reporting the same number of months of use and providing an exact record match. Women in both of the older age categories had about 70 percent higher odds of reporting the same number of months or providing exactly matched records, compared to women under 30 years of age. Those women using long-acting or permanent methods of contraception were also significantly more likely to report the same number episodes (AOR=1.4; 95% CI=1.12, 1.85), report the same number of months of use (AOR=2.3; 95% CI=1.75, 2.98) and provide an exact record match (AOR=3.23; 95% CI=2.27, 4.46), compared to women not using this method type. Participant level of education did not emerge as having a significant association with any of the outcomes of interest.

We also assessed whether reliability of reporting contraceptive status diminished with increased time from the reference month. We found little evidence that the level of agreement was meaningfully reduced as women recalled prior behavior farther back in time. Approximately 91 percent of women were in agreement in both calendars in terms of whether they were using a contraceptive method or not in September 2012, the most recent month of overlap. A similar percent (86 percent) gave concordant reports of contraceptive use for the furthest month of December 2009.

This paper also considers whether women reliably report their reasons for discontinuation, as seen as in Table 8. Across all reasons for discontinuation, a Kappa statistic of 0.47 suggests moderate reporting reliability. However, an off-diagonal proportion of 43 percent reveals large percentages of episodes for which women provided discrepant reasons for discontinuing their contraceptive method. Across all reasons for method discontinuation, there was highest consistency among women reporting that they discontinued their method due to a desire to become pregnant (76 percent agreement, with the 2012 report as the denominator). The remaining four categories – contraceptive failure, side effects/health concerns, still in need, and reduced need – had only about 50 percent agreement between the two calendars. Notably, in nearly half of all 2014-reported instances of discontinuation for which the women indicated a desire to become pregnant as her reason for stopping, she previously reported having discontinued due to side effects, contraceptive failure, or other reasons (still in need). This might suggest post-event rationalization of an unintended pregnancy. Additionally, we reconfigured Table 8, collapsing the categories of side effects/health concerns and still in need, given that side effects and health concerns are a specific type of discontinuation while in need (data not shown). When we collapsed these two similar categories, we found that the off-diagonal proportion for Table 8 was reduced from 43 to 37 percent and there was 61 percent agreement within the collapsed category, compared to 51 and 46 percent agreement using two separate categories. This suggests that, among all women who discontinue while still in need, they may not recall the exact reason for discontinuation, but 61 percent consistently recall that their reason for discontinuing was not related to contraceptive failure, a desire to become pregnant, or reduced need for family planning.

Table 8.

Comparison of reasons reported for contraceptive discontinuation for matched episodes, 2012 and 2014 MLE surveys, urban Kenya

	Reason reported in 2014
Reason reported in 2012	Contraceptive failure	Wanted to become pregnant	Side effects/health problems	Other - Still in need	Other - Reduced or no need	Total (Percent)
Contraceptive failure	14 (48.3% agreement)	11	2	1	1	29 16.0%
Wanted to become pregnant	2	41 (75.9% agreement)	4	6	1	54 29.8%
Side effects/health problems	4	14	27 (50.9% agreement)	6	2	53 29.3%
Other - Still in need	4	10	5	17 (45.9% agreement)	1	37 20.4%
Other - Reduced or no need	0	1	0	3	4 (50% agreement)	8 4.4%
Total (Percent)	24 10%	77 33%	38 16%	33 14%	9 4%	181 100.0%

Note: Kappa statistic: 0.47; off-diagonal proportion: 43.1%

Finally, Table 9 presents 12-month discontinuation rates for the overlapping time-period, stratified by method type and reason for discontinuation, as reported in 2012 and again in 2014. As shown, participants reported a higher rate of discontinuation (39 percent) when surveyed in 2014 compared to their reports in 2012 (34 percent). This difference appears to be largely driven by differences in discontinuation rates for condoms and ‘other’ methods (primarily composed of traditional methods). When reporting in 2014, women were more likely to report discontinuation of other methods due to fear of side-effects or reduced need and were more likely to report discontinuation of condoms while still in need due to reasons unrelated to side effects or health concerns. A similar pattern was seen with switching between contraceptive methods. Participants reported a higher rate of switching (17 percent) when surveyed in 2014 compared to their report in 2012 (12 percent). A much higher percentage of women using condoms or pills reported switching in 2014 compared to the percent using these methods who reported switching in 2012.

Table 9.

Comparison of 12-month contraceptive discontinuation rates by method and by reason for discontinuation, for the overlapping period, as reported in 2012 and 2014, MLE surveys, urban Kenya

	Reasons for discontinuation
	As reported in 2012
Contraceptive method	Wanted to become pregnant	Contraceptive failure	Side-effect/health concerns	Other-still in need	Other-reduced need	All reasons	Switching	Unweighted N
Pill	8.9	6.9	16.1	12.7	1.8	46.5	16.7	156
Injectables	6.2	1.0	18.4	8.7	2.6	36.8	12.0	349
Implant	1.2	0.0	3.8	0.0	0.0	5.0	3.2	169
Condom	3.4	2.4	3.0	10.5	16.7	36.0	7.3	116
Other	1.7	5.8	1.8	28.3	5.9	43.5	19.9	225
All methods	4.5	2.9	10.2	12.2	4.4	34.2	12.4	1027
	As reported in 2014
Pill	7.3	7.4	17.1	15.0	1.6	48.4	25.1	150
Injectables	2.5	1.5	15.9	10.3	2.3	32.4	13.3	348
Implant	3.0	0.0	6.0	0.0	0.0	8.9	1.9	185
Condom	5.8	2.4	2.8	23.0	19.7	53.7	19.1	132
Other	3.6	3.0	7.2	29.6	12.8	56.3	26.7	226
All methods	3.8	2.6	10.7	15.1	6.3	38.6	16.7	1050

Discussion

In this study, we assess the reliability of reproductive calendar data by using overlapping calendars implemented among a large sample of women of reproductive age living in urban Kenya. For a variety of outcomes of interest (contraceptive status, specific method used, and reasons for discontinuation), our analysis reveals moderate to substantial reliability of calendar data at the aggregate level. However, this high aggregate agreement masks considerable individual-level discordance. Large percentages of women were unable to reliably report the number of reproductive events, contraceptive episodes, or months of contraceptive use, and less than one in five contraceptive users had an exact match of data reported during all 33 months of the overlapping calendar period. A complex reproductive/contraceptive history was associated with lower odds of reliable reporting, and older women and those using long acting and permanent methods were significantly more likely to give reliable reports across several outcomes.

This individual-level discordance resulted in different estimates of 12-month discontinuation rates between the two survey rounds, with higher all-method discontinuation and higher method switching reported in 2014 compared to 2012 reports. This finding is in contrast with Bradley et al’s 2009 hypothesis that women remembering back further in time are more likely to underestimate (not overestimate) their prior contraceptive discontinuation. These differences in rates of discontinuation were driven primarily by differences in reported condoms and other method discontinuation. Although the differences in calculated discontinuation rates resulting from individual-level error were modest, it’s possible that the large individual-level error could result in biased estimates of relationships between contraceptive discontinuation and important predictors such as the quality of the service environment (in surveys for which facility and individual data can be linked).

While we found some evidence that women were more likely to report no contraceptive use in the overlapping calendar when reporting in 2014 on events further back in time, the difference was small (three percentage points); further, agreement between the two calendars on whether women reported using any contraceptive method in a given month only varied by five percentage points when comparing the most recent and most distant month of the overlapping calendars. In combination, these results do not support the hypothesis that reliability is meaningfully diminished as women recall behavior that is further back in time in this population.

Given the greater reliability seen among women reporting long-acting methods, we re-ran our analysis with a subset of data that excluded women using condoms and ‘other’ methods in the reference month, as reported in either 2012 or 2014 (total number of women excluded=226; data not shown). In this subset of the data that excluded condom and ‘other’ method users, reliability of reporting any contraceptive use improved with observed agreement increasing from 76% to 83% and the Kappa statistic improving from 0.56 to 0.70. Agreement in terms of the specific method reported also improved, with observed agreement increasing from 77% to 87% and a Kappa statistic that increased from 0.71 to 0.82. Additionally, in this subset of the data, the same percent reported no prior contraceptive use in both calendars (35% in both calendars compared to 30% in 2012 and 33% in 2014 in the full sample). The improved reliability seen in this subset of the data suggest that the calendar instrument may produce more reliable data in settings with relatively low condom use; those countries with a large percentage of their method mix coming from condoms may be advised to use the calendar with caution. Dropping condom and other users did not meaningfully change other results in this analysis.

In comparison to Strickler et al.’s prior study conducted in Morocco (1997), our findings from urban Kenya demonstrate relatively less reliability at the aggregate level, with consistently lower Kappa statistics seen in our cross-tabulations. Our study also found lower reliability at the individual level, demonstrated by larger off-diagonal proportions in our cross-tabulations compared to those presented by Strickler et al. The differences in reliability between the Morocco and Kenya studies may reflect generally poorer reporting in sub-Saharan Africa, as seen in Bradley et al’s 2015 DHS report. However, despite differences in reliability, our results are in line with Strickler’s finding that moderate agreement at the aggregate level occurs side-by-side with sizeable individual-level discordance. And, although Strickler did not find that age significantly predicts reporting reliability – as we found – their finding that complex reproductive and contraceptive histories reduce reliability was strongly confirmed in our sample. Callahan and Becker’s more recent reliability assessment, conducted in Bangladesh in 2013, found aggregate level results more similar to our results, although the authors were less able to explore individual-level variation in terms of reported number of contraceptive episodes, months, and exact record match due to the short period of overlap of only three to five months. Additionally, when regressing individual characteristics against the outcome of reliably reporting contraceptive use status in the reference month, Callahan and Becker did not find age to play a significant role (in agreement with Strickler et al. but in contrast to our findings) but did find that complex histories and method type impacted reporting reliability, bringing all three studies into consensus on the impact of complicated prior behavior.

These results have implications for future data collection efforts designed to capture patterns of contraceptive use. In populations dominated by younger women, women with complex histories, or settings with a high prevalence of short-acting methods, the reproductive calendar is unlikely to provide reliable reports of individual-level behaviors. One potential solution is to engage additional efforts to improve the quality of calendar data. Such efforts might include use of electronic data collection for which prior pregnancy and birth data are automatically populated based on previous questionnaire responses. This strategy is already in effect to a degree: The DHS currently instructs enumerators using paper-based calendars to fill in sections of the calendar as they gather data in the women’s questionnaire on births, pregnancies, terminations, and current contraceptive use; however, a digital tool that automates this process could reduce opportunity for errors. Yet, as Bradley et al point out, collecting calendar data via digital means has been problematic to date. Electronic data collection efforts may inadvertently omit visual cues customarily used in the paper questionnaire; therefore, additional research should investigate the most effective way to aid women in more accurately recalling their contraceptive histories when participating in digital calendars. In addition, Bradley et al suggest investigating whether a shorter calendar (three years instead of five) might result in higher quality data, although our results suggest little difference in the percentage of women providing discrepant reports of contraceptive use whether women are recalling two versus nearly five years prior to the interview. Finally, Bradley et al suggest looking to countries that have demonstrated high quality calendar data to investigate whether they implement strategies that could be adopted in additional settings to improve data quality.

Another possible strategy to reduce individual-level reporting error is to explore prospective data collection of contraceptive use patterns. Such data collection techniques would enable respondents to avoid potential recall bias and offer more reliable reports of contraceptive use dynamics over time. As technology increases and expands its reach within low-income countries, opportunities for frequently collected (i.e. monthly) panel data also increase in the form of surveys that are pushed to respondents via SMS text to their cellular or smart phones. However, such prospective data collection techniques are not without problems. There is greater potential for missing data to arise when women cannot be found each and every month of the survey. It may also be the case that it will not be feasible or acceptable for women in low-income countries to provide sensitive data on a monthly basis over a long period of time, especially if doing so via an unsecure text messaging platform. Additionally, implementing changes to the way in which calendar data are collected may impact our ability to make comparisons with historical data on contraceptive use dynamics.

The primary limitation of this study is that there was a 25 percent loss to follow-up between implementation of the first and second calendar. This resulted in a different age distribution in our sample, compared to those who were lost between surveys. Our sample included considerably fewer women of the ages 20 to 29 and more women who were over the age of 40. Therefore, the discordance we note is not applicable to the entire 2012 sample as it appears that older women demonstrate greater reporting reliability; however, the fundamental conclusions of this paper are not likely to change if we were able to conduct the analysis without loss to follow up. Additionally, the data that inform this analysis were only collected in urban areas of Kenya, limiting generalization within rural populations. Finally, it is important to acknowledge that weak reliability of calendar data cannot be attributed entirely to the instrument itself and may be partially attributable to the way in which the instrument is implemented and the skill and training of the enumerator administering the calendar. It is not possible within this analysis to determine whether weaknesses in calendar implementation may have contributed to discordance between the 2012 and 2014 reports.

Regarding future research, as concerns over the reliability and validity of reproductive calendar data continue to arise in our field, additional analysis of overlapping calendars in other settings would be useful, although it’s notable that the results of the three studies with overlapping calendars that have been conducted to date (Strickler et al. 1997; Callahan and Becker 2012; present manuscript 2021) are broadly similar across three fairly different cultural contexts (Morocco, Bangladesh, and Kenya). Additionally, it would be useful to explore how more stringent data quality measures – such as more useful visual aids - might impact calendar reliability. Further, prior to considering retiring this retrospective approach in favor of a prospective measure, it is imperative to gather more data on the logistical feasibility and the data quality issues associated with prospective calendar efforts so that the relative merits and challenges of different data collection approached can be assessed.

Conclusion

Our analysis addresses a critical research gap by using overlapping calendars to explore the reliability of retrospective reproductive calendar data within a large sample of women. Findings suggest moderate reliability of the reproductive calendar at the aggregate level; however, when analyzing at the individual level, large percentages of women have discordant reports across the two calendars, particularly younger women and those with more complex histories. This suggests that results of the calendar instrument will be less reliable in populations dominated by younger women, women with complex histories, or settings with a high prevalence of short-acting methods. In such settings, additional work is needed to strengthen the quality of calendar data. Further, prospective measures of contraceptive use may offer a more reliable picture of contraceptive use dynamics but may present logistical and methodological challenges of their own.

Data availability statement:

Data for this analysis come from the Measurement, Learning & Evaluation Project and can be accessed via the data portal of the Carolina Population Center.