Using Evidence to Drive Impact: Developing the FP Goals Impact Matrix

Abstract

When designing a family planning (FP) strategy, decision‐makers can choose from a wide range of interventions designed to expand access to and develop demand for FP. However, not all interventions will have the same impact on increasing modern contraceptive prevalence (mCP). Understanding the existing evidence is critical to planning successful and cost‐effective programs. The Impact Matrix is the first comprehensive summary of the impact of a full range of FP interventions on increasing mCP using a single comparable metric. It was developed through an extensive literature review with input from the wider FP community, and includes 138 impact factors highlighting the range of effectiveness observed across categories and subcategories of FP interventions. The Impact Matrix is central to the FP Goals model, used to project scenarios of mCP growth that help decision‐makers set realistic goals and prioritize investments. Development of the Impact Matrix, evidence gaps identified, and the contribution to FP Goals are discussed.

Since the Family Planning Summit in 2012 and the resulting Family Planning 2020 (FP2020) Initiative, there has been an increased global focus on family planning, including demand for greater use of evidence to drive strategic decision‐making around family planning policy and investments (Askew and Brady 2013; Silverman and Glassman 2016). For countries to set realistic family planning goals and define strategies that can support the achievement of those goals, it is critical to understand the comparative effectiveness of different types of family planning interventions in different contexts. This understanding can support the selection of effective and appropriate interventions for inclusion in national strategies. Further, as resources are limited and the need is great, ensuring that efforts and resources are allocated in ways that prioritize impact and cost‐effectiveness can help countries provide for the family planning needs of more women.

The Impact Matrix presented in this article is the first to comprehensively summarize findings related to the impact of a full range of family planning interventions on increasing modern contraceptive use using a single comparable metric. While there have been efforts to document the impact of family planning interventions, including a number of systematic reviews (Samara, Buckner, and Tsui 1996; Bauman 1997; Phillips, Greene, and Jackson 1999; Peters, Mirchandani, and Hansen 2004; Mwaikambo et al. 2011) and initiatives such as the High‐Impact Practices (HIPs) in family planning,¹ the effort described in this article focuses on consolidating evidence of the impact of different types of family planning interventions specifically on contraceptive use, and calculates impact estimates that decision‐makers can use to compare the potential impact of their programmatic choices on the prevalence of current use of modern contraceptives, or modern contraceptive prevalence (mCP).²

The intention of this work is to document the full range of findings on the effectiveness of different types of family planning interventions regardless of the magnitude or directionality of the assessed impact. In addition, by including both published and gray literature, the results are intended to be representative of the range of potential outcomes that could be expected during actual implementation (versus a pilot or study). The Impact Matrix summarizes the impact of interventions that directly increase access to contraception (e.g., community health workers), and interventions focused on generating demand (e.g., mass media). Where possible, intervention categories have been aligned to HIPs to ensure continuity with wider efforts.

The Impact Matrix was developed as a component of the FP Goals model, developed by Avenir Health's Track20 Project (Avenir Health 2017). FP Goals overlays the Impact Matrix with country‐specific information on demographics to model the impact of introducing or scaling up different sets of family planning interventions on a country's mCP. FP Goals can support improved strategic decision‐making and more effective prioritization of family planning investments.

A number of models exist (e.g., FamPlan Impact2, ImpactNow, RealityCheck) to estimate the impact of family planning programs or to estimate the number of commodities needed to achieve a specified level of mCP (Stover, Heaton, and Ross 2006; Health Policy Project, USAID, and Marie Stopes International 2014; RESPOND Project 2014; Marie Stopes International 2019). FP Goals is the first model that enables countries to identify how specific investments in family planning interventions can lead to increases in modern contraceptive use; these projections are underpinned by the Impact Matrix. A validation exercise conducted in two countries shows the ability of FP Goals to accurately project mCP changes as well as the utility of attributing change to specific interventions (Weinberger et al. 2017). Given that funding for family planning is limited in many countries, it is critical that decisions be evidence‐based when setting national goals, developing national strategies and Costed Implementation Plans (CIPs), and designing national and regional programs.

This modeling approach is new in family planning, but there is a long history of similar strategic modeling in other health areas. Silverman and Glassman (2016: p. 20) note that “current modeling capacity for family planning lags behind recent advances in other fields, most notably HIV.” The development of the FP Goals model, as well as the Impact Matrix, builds on the experience of the Goals Model for HIV‐related strategic resource allocation (Bollinger, Cooper‐Arnold, and Stover 2004; Bollinger 2008). Within the HIV field, evidence‐based modeling led to a shift in the effort to combat the HIV epidemic, encouraging evidence‐based planning and investments through model applications that supported the development of national plans and strategies (Forsythe, Stover, and Bollinger 2009). Despite the long history and successes of family planning programming, there is much that can be learned from the innovations and experiences of HIV programs (Silverman and Glassman 2016).

This work explored findings on the impact of the following broad intervention categories: postpartum family planning (PPFP), post‐abortion family planning (PAFP), community health workers (CHWs), mobile outreach, social franchising, family planning vouchers, HIV/FP integration, youth‐focused programs, and social and behavior change (SBC) including mass media, comprehensive community engagement (CCE), and interpersonal communication (IPC). While many of these intervention categories are widely accepted as effective means of increasing access to and use of contraception, not all had sufficient findings to support their inclusion in the final Impact Matrix, specifically mobile outreach, social franchising, and HIV/FP integration. The FP Goals model also includes impact from additional intervention categories such as expanding access to specific methods, introducing new methods, and reducing stock‐outs based on existing analysis and modeling work outside of this review (Karim, Bieze, and Chimnani 2008; Wang et al. 2011; Ross and Stover 2013; RHSC 2017).

The intervention categories included in the Impact Matrix may not comprise all the interventions being implemented in family planning; however, these were the areas for which there were findings available quantifying the relationship between an intervention and changes in contraceptive use. This is not meant to imply that other interventions or outcomes are not important, but rather that sufficient findings do not exist measuring the link between these interventions and changes in mCP. The Impact Matrix, and its inclusion within the FP Goals model, can help support better use of evidence in decision‐making and development of policies and strategies in family planning, as has occurred in the HIV field. The FP Goals model is not meant to influence decision‐making in a vacuum, and model outputs should be considered alongside other factors such as country priorities, rights‐based programming considerations, and the political and policy environment.

METHODOLOGY

Literature Review

A review of the literature was conducted through formal (literature databases) and informal (organizational reports/websites) sources to identify findings related to the effectiveness of family planning programming on increasing mCP.

Between June and July 2012, we conducted searches in both PubMed/Medline and Popline using the following main search terms: program evaluation, contraceptive use or behavior, family planning, fertility. The following free text combinations were also used with the term “interventions”: evaluation, outcome, impact. We conducted supplementary searches on the websites for the projects of Engender Health, FHI360, Johns Hopkins, MSH, Palladium, the Population Council, PSI, and other global health organizations along with abstracts from the 2011 International Conference on Family Planning held in Dakar, Senegal (November 29 to December 2, 2011). More than 2,400 citations, presentations, studies, and reports were reviewed for relevancy based on titles and abstracts. An initial review eliminated studies that did not include a quantitative measure of program effectiveness or looked only at trends in contraceptive prevalence in the absence of a program evaluation, resulting in 649 remaining studies. We conducted additional limited searches between January 2016 and July 2017 to seek out further studies on key topics or to identify more recently published literature, resulting in the addition of 31 citations. Overall, 680 studies, program reports, and evaluations were identified for potential use.

We developed inclusion and exclusion criteria to narrow down the identified studies. Studies were considered for inclusion if they (1) defined and evaluated a family planning intervention in a developing country, (2) included an outcome measure related to modern contraceptive use, and (3) provided quantitative pre‐post or intervention‐control outcome measures. Studies were excluded if they (1) did not include adequate documentation, (2) presented substantial methodological issues,³ (3) evaluated an intervention that was not family planning focused (e.g., HIV interventions), (4) did not feature a clearly defined intervention to which impact could be attributed, (5) were based in very specific contexts that made the results nongeneralizable (e.g., conflict zones), (6) reported only results related to changes in ideation or intention (not contraceptive use), (7) produced results limited to changes in use of one kind of contraceptive method (e.g., condom use, long‐acting reversible contraceptives [LARCs]), or (8) reported only on “referrals” for contraceptive services as the outcome, without subsequent measures of contraceptive use.

Of the 680 studies identified, 115 were excluded because adequate documentation could not be located (only abstracts or citations found); an additional 469 were excluded for either failing to meet inclusion criteria (e.g., literature reviews, policy documents, summaries of best practices) or for meeting exclusion criteria (e.g., outcome limited to condom use, methodological issues, evaluations of non‐family planning interventions). Overall, 96 studies were identified for inclusion, based on the criteria (Figure 1).

Figure 1.

Flowchart of included studies

As noted in the inclusion criteria, studies were required to have quantitative results related to changes in contraceptive use with at least two groups (e.g. pre‐post within intervention group or post only comparing intervention control) to allow for calculation of the impact of the intervention. By not limiting inclusion to study designs that included both a pre‐post measure and a control group, a broader set of quantitative findings could be included. Further, since tests of statistical significance were not always available, statistical significance was not used as a screening criterion for inclusion. At this time, no weighting has been done based on the strength of the results, however results are categorized by the type of study design to allow visibility into the strength of study design.

Producing Impact Estimates

We selected odds ratios (ORs) as the impact measure to produce comparable estimates across intervention categories. ORs are a relative measure of effect, in this case measuring the association between exposure to an FP intervention and modern contraceptive use. Aside from allowing easy comparison of impact between and across interventions, several of the characteristics of ORs make them ideal for this work: ORs can estimate the joint effects of interventions, and they can be multiplied by different levels of exposure to scale the expected impact.

The 96 included studies were reviewed in detail, and intervention inputs (e.g., program type, specific intervention components, intervention location, etc.) and the outcomes measured (e.g., current contraceptive use, modern contraceptive prevalence, contraceptive use at last sex) were extracted. Study information and results related to contraceptive use were organized into a database. For studies with published ORs, these results were entered as published to be used as impact estimates. For studies without published ORs, outcomes of comparative contraceptive use (pre‐post, control‐intervention, pre‐post with control) were documented and used to calculate ORs as described below.

First, the odds of each outcome o for each result r (o_r) as follows:

$$\mathrm{Odd}{\mathrm{s}}_{\mathrm{r}}={\mathrm{o}}_{\mathrm{r}}/\left(1-{\mathrm{o}}_{\mathrm{r}}\right)$$

Where r includes pre‐intervention (prei), post‐intervention (posti), pre‐control group (prec), and post‐control group (postc). The outcome o was generally current contraceptive use (modern), but some other outcomes were also included as discussed below.

Next, an OR was calculated based on the study design:

• Intervention only with pre‐post: OR = Odds_posti/Odds_prei

• Post only with intervention and control: OR = Odds_posti/Odds_postc

• Pre‐post with intervention and control: OR = [Odds_posti/Odds_prei]/[Odds_postc/Odds_prec]

A number of studies contributed multiple ORs as they featured outcomes from multiple study arms or intervention sites or included evaluations of more than one type of intervention. In addition, two cross‐country initiatives, the African Youth Alliance and the Urban Reproductive Health Initiative, produced numerous results that were included and make up a substantial portion of the findings used for a few intervention categories (youth and mass media).

Eleven ORs across five interventions were based on outcome measures that are related but not identical to current modern contraceptive use: adopt modern method from baseline, client left with modern method, client received modern method, and ever use of modern contraception. To ensure comparability, these 11 ORs were adjusted to account for the fact that these measures would overestimate impact on current use (see Appendix 1). The majority (6/11) of these adjusted ORs relate to post‐abortion family planning; for this intervention, adjusted ORs represent half of the ORs included. Sensitivity testing shows a minimal impact of excluding these adjusted ORs on results as shown in Appendix 1.

Developing Intervention‐Specific Impacts

We classified ORs by intervention categories and subcategories where sufficient findings existed. Intervention categories were aligned to existing HIPS when possible.⁴ Categorization was based on the ORs rather than the studies.

Once the initial OR data was compiled and organized into intervention categories, an expert meeting was convened in January 2016 to review the findings, validate the intervention categories, and fill in gaps with any further studies that had been missed in the review. Representatives from Abt Associates, Chemonics, FHI360, FP2020, Georgetown IRH, IntraHealth, IPPF JHU, Palladium, Pathfinder, the Population Council, the PRB, PSI, UNFPA, USAID, and WHO participated and provided feedback on the findings, categorization, and methodology. Additional studies provided by participants were reviewed and added to the database if they conformed to the established criteria (these are included within the 31 additional studies identified, noted above). Based on discussions and feedback from the expert review meeting, categorization and definitions used in HIPs, and careful review of the findings, final intervention categories were assigned.

Where sufficient findings existed and where there were clear distinctions between specific intervention strategies, we segmented intervention categories into more detailed subcategories. For example, the number of findings related to CHWs was large enough to segment between those models where only condoms, pills, and/or referrals were provided and those models in which injectables were also provided. For a full summary of the identified intervention categories and subcategories see Table 1.

Table 1.

Summary of interventions

Broad Intervention Category	Intervention Subcategory	Definition
Community Health Workers (CHWs)	CHW provision of pills and condoms	Provision of FP by lay health workers (CBDs, CHWs) in a community (rather than health‐care facility) setting, limited to pills and condoms. (Note: Must provide FP methods to qualify as CHW.)
	CHW provision of pills and condoms + injectables	Provision of FP by lay health workers (CBDs, CHWs) in a community (rather than health‐care facility) setting; provision includes at least pills, condoms, and injectables
Mobile Outreach	n/a	Provision of a wide range of contraceptives, including long‐acting reversible and permanent methods, through Mobile Outreach teams
Social Franchising	n/a	Organization of private providers into branded, quality‐assured networks to increase access to provider‐dependent contraceptive methods and related services
Family Planning Vouchers	n/a	Distribution of vouchers for free or subsidized FP services, generally in the interest of providing access to FP for low‐income or other underserved populations (e.g., youth)
Post‐Abortion Family Planning (PAFP)	n/a	Integration of FP counseling and provision with abortion services or post‐abortion care
Postpartum Family Planning (PPFP)	Facility‐Based Integration of PPFP into Antenatal (ANC) and/or Delivery Care	Provision of PPFP counseling during antenatal care in facilities, and provision of PPFP counseling and services at the same time and location as facility delivery
	Facility‐Based Integration of PPFP with Postpartum Care	Provision of PPFP counseling and services in a facility during postpartum care (not including delivery)
	Community‐Based/SBC PPFP Promotion	Promotion and provision of PPFP counseling, and sometimes services, outside of facility, generally through community health workers (CHWs)
	Integration of PPFP with Childhood Immunization Services	Provision of FP counseling and services (or referrals) at the same time and location where childhood immunizations are provided
HIV/FP Integration	Integration of Family Planning into Routine HIV/AIDS Treatment (ART)	Provision of FP counseling and services or referrals at the same time and location where people living with HIV receive care related to the treatment of HIV and AIDS
	Integration of Family Planning into Voluntary Counseling and Testing (VCT) for HIV	Provision of FP counseling and services or referrals at the same time and location where VCT is offered
Youth‐Focused Programming	Curriculum‐based Sexual and Reproductive Health (SRH) Education	Educational programming for youth focused on improving knowledge, attitudes, and behavior around sexual and reproductive health, including contraceptive use
	Multi‐Component Youth Programming	A combination of demand‐side and access‐side interventions designed to improve social norms around youth contraceptive use and increase youth knowledge of and access to FP services
	Multi‐Component Youth Programming with Youth‐Friendly Service (YFS) Provision	A combination of demand‐side and access‐side interventions designed to improve social norms around youth contraceptive use and increase youth knowledge of and access to FP services, specifically accompanied by the provision of Youth‐Friendly FP service provision
Social and Behavior Change Programming	Comprehensive Community Engagement (CCE)	Promotion of FP (and related topics, e.g., healthy birth spacing) through community engagement and events
	Community‐Based Interpersonal Communication (IPC)	Promotion of FP (and related topics, e.g., healthy birth spacing) through individual or group conversations
	Mass Media Campaigns	Promotion of FP (and related topics, e.g., healthy birth spacing) through mass media channels including television, radio, and print materials

n/a = Category could not be divided into subcategories.

A minimum of five ORs were required to include a given category or subcategory in the Impact Matrix to prevent one result from overly affecting the impact of a given intervention category. The median of all ORs in a given category was used as the impact factor of that category to avoid estimates being strongly influenced by outliers.

RESULTS

Overview of Odds Ratios

Overall, we extracted or calculated 148 ORs from the 96 identified studies. Below we present information on the volume and trends, by time, intervention category, and geographical location of the studies from which these ORs were derived. Because some individual studies contributed multiple ORs that may have represented multiple countries or intervention categories, summary statistics below are based on ORs, rather than studies.

Of the intervention categories and subcategories identified, mass media had the greatest number of findings, with 33 ORs describing the impact of these types of programs. The bulk of these results are derived from an evaluation of the Urban Reproductive Health initiative, where results were disaggregated by individual intervention components (including television, radio, leaflets). Next, both PPFP and youth‐focused programs had 25 individual ORs identified. In both cases, the large number of findings allowed for segmentation into subcategories providing a more detailed understanding of the differential impact of specific models within these overarching intervention categories. For mobile outreach, social franchising, and HIV/FP integration, the minimum threshold of five ORs was not met and therefore these intervention categories are not included in the final Impact Matrix. However, the ORs for these three intervention categories have been included in summary results for transparency and discussion.

Time Period Distribution of Odds Ratio Sources

The included studies covered a wide time frame; the earliest included study was published in 1989, while the most recent studies were published in 2016. Figure 2 illustrates the trends in the timing of research and evaluations of different types of interventions.

Figure 2.

Count of ORs by intervention and publication year

Among the access‐focused intervention categories, findings for mobile outreach do not appear to be time‐dependent, with the first study occurring in the late 1990s and the most recent study from 2013. Evaluations of social franchising occurred more recently, with the first evaluation occurring in 2005. Studies on CHWs were conducted over all six time periods, however there is an uptick in studies focused on this intervention area after 2000, with the majority occurring after 2010.

Among those intervention categories where FP is integrated with other health services (PPFP, PAFP, FP Vouchers, and HIV/FP Integration), studies on PPFP appear across all six time periods while studies on PAFP appear across five time periods. There appears to be a shift in the research published on these two topics, with the bulk of the PAFP research published in the 2000s and tapering off after 2010, while publications related to PPFP increased substantially in the late 2000s and continue to the present. Findings on voucher programs and HIV/FP integration appear more recently, with the majority published after 2010.

The bulk of the findings on youth‐focused programs (16 of 25 ORs) come from studies published in the early 2000s. Publications on SBC interventions (comprehensive community engagement [CCE] and interpersonal communication [IPC]) saw two spikes, in the early 2000s and then again in the early 2010s. The findings on mass media comprise studies from the late 1990s through 2014, with the majority (17 of 32) of ORs originating from one study published in 2014 evaluating the various mass media components included in the Urban Reproductive Health Initiative implemented in four countries.

Geographic Distribution of Odds Ratio Sources

The ORs identified covered interventions implemented in a diverse set of 39 developing countries (Figure 3). Out of the total 148 identified ORs, 24 resulted from studies conducted in India, followed by Nigeria, Pakistan, and Kenya. For India, Kenya, and Nigeria, these findings were largely the result of the Urban Reproductive Health Initiative, which contributed 3 of the 24 ORs from India, half of the ORs from Kenya (6 of 12), and about three‐quarters (8 of 13) of the ORs from Nigeria.

Figure 3.

Count of ORs by country

When examining the volume of findings by subregion (Figure 4), South Asia and West Africa have the largest representation, followed by East Africa. Findings related to PAFP were the most geographically widespread, with interventions evaluated in all regions except South Asia. The findings were most geographically narrow for social franchising, mobile outreach, and HIV/FP integration, with all studies related to social franchising conducted in South Asia, mobile outreach findings from Southern and East Africa, and HIV/FP integration findings based only on studies conducted East and West Africa. However, these intervention categories also have the fewest numbers of ORs, which contributes to the more limited geographic diversity. Evaluations of CHWs were more geographically diverse, with findings from a total of three regions in Africa and South Asia. Findings on the impact of youth‐focused programs were largely based on studies conducted in West Africa, East Africa, and Latin America & the Caribbean.

Figure 4.

Count of ORs by subregion and intervention

Type of Odds Ratios

In addition to the geographic and publication year distribution of ORs, another important consideration is the type of ORs that are utilized by each intervention or sub‐intervention. Table 2 shows a summary of ORs by intervention and type (published versus calculated); for calculated ORs, the distribution by study design is shown. Overall, about half of the ORs were published and half were calculated by the authors. Among those calculated, there was a fairly even split across the three study designs (18 percent for control‐intervention only, 16 percent for pre‐post only (no control) and 16 percent from pre‐post with control).

Table 2.

OR by intervention and type

				Distribution by Type within Calculated ORs
Intervention Categories/Subcategories		Published: Total (%)	Calculated: Total (%)	Control‐ Intervention Only (%)	Pre‐Post Only (%)	Pre‐Post + Control (%)	Total Number of ORs
Total		49	51	18	16	16	148
CHW	Pills, Condoms only	50	50	0	17	33	6
	+ Injectables	55	45	0	18	27	11
Mobile outreach		100	0	0	0	0	2
Social franchising		0	100	0	25	75	4
FP vouchers		15	85	0	46	38	13
PAFP integration		25	75	33	42	0	12
PPFP	Facility: ANC/Delivery	67	33	17	17	0	6
	Facility: Postpartum	20	80	60	20	0	5
	Community/SBC	22	78	44	33	0	9
	Immunization Integration	20	80	0	0	80	5
HIV/FP Integration	ART Integration	0	100	33	0	67	3
	VCT Integration	100	0	0	0	0	1
Youth	SRH Education	20	80	60	0	20	5
	Multi‐Component	50	50	20	20	10	10
	Multi‐Component + YFS	40	60	40	0	20	10
SBC	CCE	83	17	0	17	0	6
	IPC	86	14	0	0	14	7
	Mass Media	82	18	15	3	0	33

While the overall distribution between published and calculated ORs is fairly equal, there is a large variation in published versus calculated ORs by intervention, with the exception of CHW, which has a more even split. For mass media and SBC, the vast majority of ORs were published, while most other interventions rely more heavily on calculated ORs.

Impact Matrix

The Impact Matrix summarizes the findings presented above to provide an aggregate picture of what is known about the effect size of different types of FP interventions. The Impact Matrix is organized by broad intervention category (and subcategory where appropriate) and presents the median, range (minimum to maximum), and count of the included ORs. The intervention categories are also segmented based on those that reach a general population versus those that focus on a specific subpopulation.

The Impact Matrix excludes three intervention categories (mobile outreach, social franchising, and HIV/FP integration) that did not meet the minimum threshold of at least five ORs. Therefore, the Impact Matrix only draws on a subset of 138 ORs from 87 studies.

Figure 5 is a visual illustration of the Impact Matrix. For some intervention categories and subcategories the available findings are largely consistent in terms of the impact estimates, with ORs clustering around the median. This can be seen in particular for comprehensive community engagement (“SBC: CCE”) and youth sexual and reproductive health education (“Youth: SRH education”). For other intervention categories, a much wider range is seen across the available ORs. The largest spread is seen for community‐based postpartum family planning (“PPFP: Community/SBC”) where there is a spread of 4.1 between the lowest OR (1.08) and highest (5.19) OR.

Figure 5.

Distribution of ORs by intervention for interventions included in the Impact Matrix

The full Impact Matrix is shown in Table 3, while Table 4 shows the sources contributing to each section of the Impact Matrix. The detailed ORs that contribute to this matrix are available in Appendix 2.⁵

Table 3.

Impact Matrix—Evidence of impact of family planning interventions resulting in changes in contraceptive use, shown as medians of ORs, range of ORs, and count of ORs, by type of intervention and general versus targeted population focus

		Population		Impact Factor (OR)
Intervention/Sub‐Intervention		General	Specific	Median	Range	Count
CHW	Pills and Condoms Only	X		1.56	(1.39–2.18)	6
	+ Injectables	X		1.70	(1.04–4.68)	11
Family Planning Vouchers		X		1.33	(0.59–2.08)	13
PAFP Integration			X	1.68	(1.11–3.22)	12
PPFP
	Facility: ANC/Delivery		X	1.67	(1.47–2.15)	6
	Facility: Postpartum Care		X	2.03	(1.56–2.31)	5
	Community/SBC		X	1.96	(1.08–5.19)	9
	Immunization Integration		X	1.15	(0.85–1.82)	5
Youth‐Focused Programming	SRH Education		X	1.24	(1.14–1.53)	5
	Multi‐Component		X	1.35	(0.56–3.84)	10
	Multi‐Component + YFS		X	1.44	(0.55–2.78)	10
SBC	CCE	X		1.29	(1.13–1.62)	6
	IPC	X		1.51	(0.90–2.40)	7
	Media	X		1.29	(0.93–2.75)	33

Excluded for Insufficient Evidence
Mobile Outreach		X				2
Social Franchising		X				4
HIV/FP Integration	ART/FP Integration		X			3
	VCT/FP Integration	X				1

Table 4.

Source Matrix—Studies included in matrix that evaluate impacts of family planning interventions resulting in changes in contraceptive use, by type of intervention and general versus targeted population focus

		Included Studies
Intervention/Sub‐Intervention		Reference
CHW	Pills and Condoms Only	Kambo et al. 1994; Stoebenau and Valente 2003; Douthwaite and Ward 2005; Kalanda 2010; Jayachandran et al. 2011; Viswanathan et al. 2011
	+ Injectables	Thomas and Maluccio 1996; Luck et al. 2000; Debpurr et al. 2002; Sultan et al. 2002; Stanback et al. 2007; Huber et al. 2010; Lutalo et al. 2010; Khan et al. 2011
Family Planning Vouchers		Meuwissen, Gorter, and Knottnerus 2006; Agha 2011; Das et al. 2011; Mishra et al. 2011; IFPS Technical Assistance Project (ITAP) 2012; Bajracharya et al. 2016
PAFP Integration		CEFOREP 1998; Population Council 2000; Benson and Huapaya 2002; Johnson et al 2002; Savelieva et al. 2003; Alemayehu et al. 2009; Zhu et al. 2009; Romero, Zamberlin, and Gianni 2010; Senlet et al. 2011; Smith et al. 2015
PPFP	Community/SBC	Bashour et al. 2008; Khan et al. 2008; Toth 2008; Ahmed et al. 2011; Baqui et al. 2011; Sebastian et al. 2012; Ayiasi et al. 2015
	Facility: ANC/Delivery	Bolam et al. 1998; Medina et al. 2001; Barber 2007; Akman et al. 2010; Achuyut et al. 2015
	Facility: Postpartum Care	Omu et al. 1989; Varkey et al. 2004; Abdel‐Tawab and Saher 2011; Achuyut et al. 2015
	Immunization Integration	Herrin et al. 2012; Speizer et al. 2013; Vance et al. 2014; Dulli et al. 2016
Youth‐Focused Programming	SRH Education	Kane et al. 1993; Eggleston et al. 2000; Murry et al. 2000; Villarruel et al. 2010
	Multi‐Component	Brieger et al. 2001; Levitt‐Dayal et al. 2001; Speizer, Tambashe, and Tegang 2001; Agha 2002; Tegang and IRESCO 2002; Daniel, Masilamani, and Rahman 2008; Santhya et al. 2008; Undie et al. 2012
	Multi‐Component + YFS	Kim et al. 2001; Magnani et al. 2001; Askew et al. 2004; Vernon and Durá 2004; Williams et al. 2007; Karim et al. 2009
SBC	CCE	Debpurr et al. 2002; Roy, Dham, and Loomba 2011; Speizer et al. 2014
	IPC	Kincaid 2000; Luck et al. 2000; Odeyemi and Ibude 2011; Shattuck et al. 2011; Decat et al. 2012; Speizer et al. 2014
	Mass Media	Bankole and Adewuyi 1995; Kim et al. 1996; Weinman 1997; Kane et al. 1998; Jato et al. 1999; Rogers et al. 1999; Storey et al. 1999; Valente and Saba 2001; Boulay, Storey, and Sood 2002; Gupta, Katende, and Bessinger 2003; Sood et al. 2004; Babalola and Vonrasek 2005; Hutchinson and Wheeler 2006; Hutchinson and Meekers 2012; Speizer et al. 2014

Excluded for Insufficient Evidence
Mobile Outreach		Thomas and Maluccio 1996; Casey et al. 2013
Social Franchising		Hennink and Clements 2005; Agha et al. 2007; Azmat et al. 2013; Azmat et al. 2016
HIV/FP Integration	ART/FP Integration VCT/FP Integration	Ngure et al. 2009; Kosgei et al. 2011; McCarraher et al. 2011; Bradley et al. 2010

DISCUSSION

The Impact Matrix comprehensively summarizes existing findings on the impact of broad categories of FP interventions on contraceptive use using a single comparable metric. The medians allow for comparisons of the relative effectiveness of different intervention categories, while the ranges illustrate the potential variability in impact within each category. In addition, within the Impact Matrix, intervention categories are divided between those that can be implemented among the general population of women of reproductive age and those that are focused on a specific subpopulation (e.g., postpartum women). This distinction contextualizes the potential impact of a given intervention category. For example post‐abortion family planning interventions have one of the highest impact factors, however they can only effect contraceptive use among a small subset of the population (women who have recently undergone an abortion or presented for post‐abortion care services). As a result, the overall impact of PAFP programming on increasing national contraceptive use may be lower than an intervention with a lower impact factor that could reach a wider population.

This issue is addressed when using the Impact Matrix within the FP Goals model, through which population data, along with data on the health system and current FP program context, is incorporated into estimates of an intervention's impact on mCP. In addition to accounting for the populations that can be affected by given interventions, the model also accounts for existing implementation. For example, if an intervention is already being implemented at wide scale, its impact is already contained within the current mCP, meaning there may be limited scope for further impact (despite the size of the impact factor). As a component of the FP Goals model, the Impact Matrix helps to provide decision‐makers with contextualized information to support strategic, evidence‐based program investments.

For example, in Rwanda, an FP Goals application, informed by the Impact Matrix, was instrumental in the development of their Family Planning/Adolescent Sexual and Reproductive Health (FP/ASRH) Strategic Plan. Results helped the country to set a realistic goal, and led to a strategic focus on postpartum family planning given the country context and the high potential for impact as shown by the model results (Sayingoza et al. 2018). In Lao PDR, the FP Goals model application formed the basis of their FP Costed Implementation Plan. Model results and subsequent discussion led to a more targeted focus of demand‐generation investments that prioritized provinces where these investments could have the greatest impact (Track20 2017).

In addition to providing the global evidence base for the FP Goals model, the process of developing the Impact Matrix also highlighted interesting findings related to perceptions and gaps within the evidence base, and challenges in defining and distinguishing specific intervention categories.

Availability of Documented Evidence Is Overestimated

During the review process it was found that perceptions and reality of available evidence often did not match. Experts in the field frequently overestimated the volume of evidence that was available. For example, there was a feeling that robust evidence existed for mobile outreach services, but only two studies that met inclusion criteria could be identified. While program data on mobile outreach exists on the volume of clients served by individual programs or in specific countries, little was found that assessed the impact of these programs on changing contraceptive use at a community or population level. Similarly, the evidence related to youth‐focused programs is overestimated. While a large number of youth‐focused studies (18 studies, 25 ORs) were included in the Impact Matrix, at least 20 additional studies related to youth could not be used. These excluded studies either focused solely on outcomes related to knowledge or attitudes and did not measure changes in contraceptive use, or were evaluations of HIV/AIDS‐focused interventions for which outcomes were limited to delays in sexual activity or condom use specifically for HIV prevention. In both cases, opportunities to evaluate the impact of interventions on changing contraceptive use for pregnancy prevention among youth were missed because relevant outcomes were not measured. While there are clear challenges in collecting sometimes sensitive data on sexual activity and contraceptive use among young people, studies seeking to evaluate programs focusing on improving youth sexual and reproductive health should include measures of contraceptive use beyond “condom use at last sex.”

Significant Evidence Gaps Persist

It is useful to consider the gaps in the evidence base when examining where future research funding should be directed. For both social franchising and mobile outreach the authors could not identify sufficient findings to include these intervention categories in the Impact Matrix, although they are routinely implemented. Supplementary work (not included in this article) is underway to expand the evidence base for these two interventions using routine program data and modeling. Going forward, building evaluations into program implementation of these types of interventions could help strengthen the evidence base, especially examining whether these service‐delivery models simply provide alternate sources of family planning services or actually increase access and, subsequently, contraceptive prevalence.

The wide range of impacts seen for both CHWs and community‐based PPFP calls for further research to understand why these types of interventions have such disparate outcomes, and to identify key elements likely to lead to greater impact.

Finally, for some intervention categories, especially those related to youth and comprehensive community engagement, the available evidence largely draws from large‐scale multicomponent interventions including community engagement, provider training, peer education, multichannel SBC campaigns, and other types of components, making it impossible to tease out which components led to changes in contraceptive use. For example, understanding the impact of youth‐friendly services as a stand‐alone intervention is a challenge because studies generally only looked at this intervention in conjunction with a range of community‐based engagement activities. More detailed evaluations seeking to understand specific pathways to impact are needed to develop better youth‐focused programming, as, in many contexts, this type of multicomponent intervention may not be feasible, scalable, or sustainable given the wide range of inputs and often limited budgets.

Definitions and Terminology Used in Evaluation Are Inconsistent

The process through which we developed the Impact Matrix necessitated not only a detailed examination of evaluation designs, but also of the terminology and definitions used in documentation. The process of segmenting the available ORs into intervention categories and subcategories helped to clarify key distinctions and definitions of interventions. This was done based on reviewing the intervention designs of included studies and through consultation during the technical review meeting and coordination with HIPs. This process was useful to ensure consistency in how interventions are categorized, and to ensure that available findings supported these distinctions. Two notable discussions that arose were related to CHWs and social marketing. For the Impact Matrix, inclusion under CHWs required provision of at least some family planning methods by community‐based distributors. Models where CHWs provide only counseling and referrals were counted as SBC interventions, rather than as CHWs. Rather than relying on the terms “CBD” or “CHW,” the classification of ORs was based on the description of the intervention. For social marketing, it was ultimately decided not to include this as a stand‐alone intervention category—rather, studies were segmented between supply‐side and demand‐side interventions based on the details of the intervention. For example, some of the findings for mass media interventions may include demand‐generation components traditionally classified as part of “social marketing,” while other findings categorized under CHW or vouchers may include supply‐side components featured in “social marketing” interventions.

Some Key Aspects of Programming Remain Difficult to Evaluate

It is important to note areas that have not been covered explicitly within the Impact Matrix; two notable areas are quality improvements and male engagement. There has been widespread interest in the FP community on understanding the potential impact of quality improvement as a means of increasing mCP through improving client experience and counseling, expanding contraceptive choice, increasing uptake of contraceptives, and reducing discontinuation. Through this review, three studies examining the impact of “quality” on contraceptive use were identified (Sanogo et al. 2003; Jain et al. 2011; Casey and Tshipamba 2017). These studies evaluated diverse types of quality‐improvement interventions (provider training, facility improvements, increased supporting supervision, better management/data systems, and so forth) and changes in uptake or use of contraceptives. The results of these studies were mixed, and often there was not a clear intervention (but rather a range of system changes), meaning that none of the studies met the inclusion criteria.

Further, there was concern that identifying “quality‐improvement” as a distinct intervention category for which impact can be estimated could imply that provision of quality services is optional or an “add‐on” to existing programming, rather than be treated as an integral part of all family planning programming. Finally, male engagement was not considered as a stand‐alone intervention because multiple channels and interventions have been used to reach and include men in family planning. Rather, studies that include aspects of male engagement are included in the relevant intervention categories—for example, some of the ORs used within PPFP and SBC include elements related to male engagement. The examples of male engagement and quality improvement demonstrate that while it is useful in some cases to link decision‐making about investments in interventions to the presence of documented evidence of impact, not all interventions can be neatly defined and evaluated, and that a lack of documented evidence does not necessarily mean that an intervention is not important to an overall program, or a worthwhile investment. Male engagement and a range of other interventions that support quality services and rights‐based approaches may never have sufficient evidence for inclusion in the matrix as distinct intervention categories but are nonetheless important and should be considered in the country's own context.

Study Limitations

As noted when the original HIV Goals impact matrix was developed, “the quality of the data is limited by the quality of the studies” reviewed (Kunz and Oxman 1998: p. 1188; Bollinger, Cooper‐Arnold, and Stover 2004). Because the Impact Matrix draws on both peer‐reviewed literature and program reports and evaluations, it is not always possible to assess the quality of the study design. A rigorous evaluation design may not have been feasible or appropriate for some of the interventions included in this review. Further, even in cases of strong study design there could be factors that contaminate the impact of the intervention on the measured outcome that could not be accounted for in the study results. While impossible to fully address all these potential shortcomings of the findings included, a decision was made to be inclusive of a wider body of evidence from a range of sources and methodologies as long as at least two comparable quantitative measures were available (e.g., pre‐post or control‐intervention). By then using the median value from among this wider set of findings, the Impact Matrix limits the influence of any single finding on the overall results.

Among the 138 included ORs, 77 percent (106 ORs) included a test for significance for the OR (or results used to calculate the OR). Among these, 39 percent (41 ORs) were found not to be statistically significant. The nonsignificant ORs were generally close to 1 or below 1 (median = 1.14), implying little to no impact from the evaluated intervention. The inclusion of these findings ensures that the Impact Matrix does not overstate the impact of intervention categories. Had inclusion criteria limited findings to only statistically significant results, the impact factors would be larger given that the significant results identified in this review had, on average, larger effect sizes. The remaining 23 percent of included findings (32 ORs) did not include a test for significance of the OR, or results that could be used to determine significance of the calculated OR. Based on the justification for including nonsignificant results, the decision was also made to include these results for which significance was not tested. In addition, the inclusion of these findings allows for the disaggregation of broad intervention categories into more detailed subcategories within the Impact Matrix that may not have otherwise met the minimum inclusion criteria of 5 ORs. Overall, the inclusion of both nonsignificant findings and findings without a significance test is intended to more accurately represent real‐world implementation of FP interventions, understanding that under some circumstances or in certain contexts interventions may have limited impact.

For studies that did not include published ORs, the calculations done for this article were not able to control for confounding factors and thus may not be fully comparable to ORs produced from regression analysis. This bias has the potential to work in both directions. Each intervention area included in the Impact Matrix includes a mix of both published and calculated ORs, limiting dependence of the results on one type of OR. For studies that had outcomes related to ever use and clients leaving with a method, further adjustments were made. However, sensitivity testing (Appendix 1) showed that including these ORs in the Impact Matrix had a minimal impact on the results.

ORs were also drawn from different implementation contexts. In some cases, interventions are being evaluated as they are first being introduced or being brought to scale, or with different levels of investments and political will. The magnitude of the impact achieved is dependent on these contextual factors. Much of the published literature may evaluate new and well‐implemented interventions; this combined with publication bias against negative outcomes may skew results toward larger impacts. However, the inclusion of gray literature reflecting routine program implementation was intended to capture the range of potential outcomes that could be expected during actual implementation. Sensitivity testing can be conducted with the FP Goals model using the minimum and maximum ORs for each intervention area to understand the potential range of impact that could be achieved.

In addition, this review was designed around a very specific outcome, modern contraceptive use, and a very specific result type, odds ratios or data from which odds ratios could be calculated. As a result, these findings are not representative of all evidence for the effectiveness of family planning programs, but instead aim to be representative of the evidence of the effectiveness of the identified categories of interventions specifically on increasing modern contraceptive use.

While drawing on a wide variety of evidence is a strength of the Impact Matrix, as noted previously, for youth‐related interventions and mass media, two cross‐country initiatives—the African Youth Alliance and the Urban Reproductive Health Initiative—make up a substantial portion of the findings used. The high reliance on these multi‐country initiatives may limit the generalizability of these results. However, within each initiative findings reflect impact across a range of country contexts.

Finally, while the Impact Matrix and the FP Goals model help inform decision‐making, they should not be used in isolation. Some family planning interventions that are critical to program success do not lend themselves to inclusion in the Impact Matrix or do not yet have adequate findings for inclusion. In addition, interventions may be prioritized for reasons other than increasing contraceptive use; decision‐makers must consider results from this work alongside other priorities and considerations.

CONCLUSIONS

The Impact Matrix presented in this article is the first to comprehensively summarize the available findings related to the impact of a full range of family planning interventions on increasing modern contraceptive use using a single comparable metric. This allows for an understanding of the range of impact seen within intervention categories, as well as a comparison of the relative impact of different types of intervention categories.

The Impact Matrix synthesizes a large body of findings, incorporating a total of 138 ORs from 87 studies. Despite the substantial number of ORs included, the review revealed significant evidence gaps, signaling potential areas for further research and program evaluations. Three intervention categories could not be included in the Impact Matrix due to limited numbers of studies: mobile outreach, social franchising, and HIV/FP integration. Work is underway to fill the evidence gaps for mobile outreach and social franchising utilizing routine program data and modeling. Even for those intervention categories that had sufficient findings, the numbers of relevant findings vary greatly; within SBC, mass media includes 33 odds ratios while comprehensive community engagement (CCE) and interpersonal communication (IPC) include only 6 and 7, respectively, signaling the need for continued research in this area.

While the Impact Matrix has utility on its own by documenting the volume of findings and the range of outcomes seen in this evidence, and showing the comparative effectiveness of FP intervention categories, the ultimate aim of this work was to develop impact estimates that could be used within the FP Goals model. The inclusion of the Impact Matrix within FP Goals allows for ORs to be applied while considering relevant contextual factors such as the size of the population being reached, baseline levels of contraceptive use, and how much scale‐up can reasonably be achieved. This allows the model to estimate impact drawing both on the global evidence base and the specific country context. This kind of strategic modeling is an important contribution to evidence‐based programming.

Overall, as more rigorous evidence becomes available, the Impact Matrix will be updated and improved. In the future, a decision could be made to exclude studies with less rigorous designs or to weight studies based on their design methodology or sample size. However, even with the current volume of findings available, the use of the Impact Matrix presented here in conjunction with the FP Goals model represents a large step forward for the family planning sector to better incorporate evidence‐based planning into the design of family planning policies and strategies.

Supporting information

Supporting Information

APPENDIX 1.

Adjusting for Outcomes Other Than Current Contraceptive Use

Several studies included outcomes related to contraceptive use but not directly comparable to measures of mCP:

• Client left with method/client received method

• Ever use of family planning method

• Adopt family planning from baseline

All of these outcomes risk overestimating the impact of the interventions on current contraceptive use. Some clients who left with a method may not actually use the method, and this is especially true for pills and condoms. Left with a method also does not account for any discontinuation that would be picked up in a study that followed up clients at 6 or 12 months. Measures of ever use are likely to overestimate impact since, where comparable data exists, ever use is higher than current contraceptive use.

While these outcome measures are different, the same adjustment factor was applied to all due to limited data available to develop adjustments. This adjustment factor was created based on the relationship between mCP and ever use of a modern method as seen in all available DHS surveys. Note that DHS no longer asks questions about ever use, so recent DHS surveys are not included in this analysis. The “Relationship Between mCP and Ever Use of a Modern Method” graph shows the relationship seen between mCP and ever use; these outcomes are highly correlated (R² = .927) when considering an exponential relationship between the two variables.

Relationship Between mCP and Ever Use of a Modern Method

While the relationship between ever use and mCP is different at different levels of mCP, for the purposes of this work a single adjustment factor was desired. The intention was for the adjustment factor to be based on the ratio between mCP and ever use of contraception. This ratio is shown in the “Ratio of Current Use to Ever Use, by mCP Level” graph for levels of mCP ranging from 1 percent to 60 percent; across this mCP range the ratio ranges from .2 to .7. To be conservative, and to recognize that the literature being drawn on for this review is largely focused on low prevalence settings, a ratio of .4 was selected.

Ratio of Current Use to Ever Use, by mCP Level

This ratio was multiplied by the OR (published or calculated) that represented the impact on the original outcome to convert this OR into a proxy result for “current contraceptive use.”

In total, 11 ORs were adjusted. The “Summary of Adjusted ORs” table shows the unadjusted and adjusted ORs, as well as the median and range for the intervention category. As can be seen, in most cases, the adjustment factor brings the result more closely in line with those seen from studies that measure an outcome of current use.

Summary of Adjusted ORs

Intervention Category	Outcome	Unadjusted OR	Final OR	Median and Range for Intervention	Study
CHW: Injection	Adopt FP from baseline	3.86	1.54	1.7 (1.04–4.68)	Luck et al. (2000)
Post‐abortion FP	Client left with method	2.79	1.11		Romero, Zamberlin, and Gianni (2010)
Post‐abortion FP	Client left with method	8.06	3.22		Alemayehu et al. (2009)
Post‐abortion FP	Client left with method	4.04	1.62	1.68 (1.11–3.22)	Benson and Huapaya (2002)
Post‐abortion FP	Client left with method	4.38	1.75		Senlet et al. (2001)
Post‐abortion FP	Client left with method	3.68	1.47		Population Council (2000)
Post‐abortion FP	Client left with method	3.07	1.23		CEFOREP (1998)
PPFP: Facility‐Based: Prenatal/Delivery Care	Received modern method	4.43	1.77	1.67 (1.47–2.15)	Medina et al. (2001)
SBC: Community‐based IPC	Adopt FP from baseline	3.77	1.51	1.51 (.9–2.4)	Luck et al. (2000)
Youth: Curriculum‐based SRH	Ever use	3.08	1.23	1.24 (1.14–1.53)	Kane et al. (1993)
Youth: Curriculum‐based SRH	Ever use	3.61	1.44		Kane et al. (1993)

Sensitivity testing shows a negligible impact of excluding these adjusted ORs on final results for nearly all intervention categories. The largest difference is seen in the median results for PAFP; with adjusted values excluded the median OR for this category increases by .2 (from 1.7 to 1.9). (See “Summary of Results with and without Adjusted ORs” table.)

Summary of Results with and without Adjusted ORs

Intervention Category/Subcategory		Adjusted/Total ORs	Median OR Including Adjusted Values	Median OR Excluding Adjusted Values
CHW	Pills and Condoms Only	0/6	n/a	n/a
	+ Injectables	1/11	1.70	1.72
Family Planning Vouchers		0/13	n/a	n/a
PAFP Integration		6/12	1.68	1.92
PPFP	Community/SBC	0/9	n/a	n/a
	Facility: ANC/Delivery	1/6	1.67	1.62
	Facility: Postpartum Care	0/5	n/a	n/a
	Immunization Integration	0/5	n/a	n/a
Youth‐Focused Programming	SRH Education	2/5	1.24	1.24
	Multi‐Component	0/10	n/a	n/a
	Multi‐Component + YFS	0/10	n/a	n/a
SBC	CCE	0/6	n/a	n/a
	IPC	1/7	1.51	1.40
	Media	0/33	n/a	n/a

n/a = Not applicable.