Tailoring Process and Impact Evaluation of a “Cash-Plus” Program: The Value of Using a Participatory Program Impact Pathway Analysis

ABSTRACT

Background

Evaluations are often limited to affirming what impact health and nutrition interventions have, without providing enough insights into “how/why” impacts are achieved.

Objectives

This article describes how a Program Impact Pathway (PIP) analysis was used to tailor theory-driven impact and process evaluation of a “Cash-Plus” program combining unconditional cash transfers with behavior change communication (BCC) activities, which was implemented to improve children's growth in Togo.

Methods

A theoretical PIP diagram was developed using existing literature, program documentation, and interviews with the program's stakeholders at the central level. Next, the PIP diagram was refined through 2 regional participatory workshops, 6 mo after the program began. Workshop participants were multilevel field implementers and were asked to 1) discuss their vision of the program's objectives; 2) describe the “inputs–process–outputs–outcomes–impacts” flow; 3) reflect on modifiers that may arise along the PIP; and 4) report bottlenecks in the program's conception or implementation and suggest corrective actions. The PIP was used to determine research questions that should be investigated during impact or process evaluation and guided the choice of data collection methods and tools.

Results

The PIP analysis identified 3 impact pathways, all based on the synergy between cash and raised women's knowledge. Along these pathways, the motivation and workload of frontline workers, along with issues in cash flow, were identified as factors that may affect the delivery of activities, whereas women's control over resources, time availability, support from relatives, and the presence of markets and health and school services were recognized as factors that may influence the uptake of activities. Improved communication between stakeholders and increased involvement of husbands were suggested for better impact achievement.

Conclusions

The participatory PIP analysis helped implementers and evaluators to share a common vision of the program's objective and logic, encouraged communication across sectors, and facilitated course-adjustments of the program.

This article showcases how a Program Impact Pathway analysis can be operationalized to conduct comprehensive process and impact evaluations of cash transfer programs.

Introduction

Conditional and unconditional cash-based interventions (CBIs) are part of nutrition-sensitive strategies that can reduce the multiple forms of children's undernutrition (1). The number and size of CBIs have increased rapidly over the past 2 decades worldwide—especially cash transfers (CTs), which imply giving money in the form of cash to target groups—and today it is estimated that 1.5 billion individuals in low- and middle-income countries benefit from such interventions (2). First administered in Latin America, this type of assistance has become increasingly popular in Sub-Saharan Africa since the 2000s. In East Africa, CT programs tend to be implemented by governments and on a large scale; conversely, in West African countries, CT programs remain predominantly pilot and small-scale initiatives that are implemented and funded by external agencies and donors (e.g., the World Bank, the World Food Program, or UNICEF). Globally there is evidence of CTs’ impacts on a broad range of outcomes, including poverty, food insecurity, health-seeking behaviors, and school enrollment and attendance. However, the evidence that CTs influence children's nutritional status is weak. Some studies have demonstrated positive impacts on growth retardation, e.g., in Mexico, Colombia, and Nicaragua, in particular when children were exposed to the intervention at an early stage and when the transfer amount represented a significant share of the household's total expenditures (3–5). In 2013, however, a literature review examining 17 CT programs reported a null effect on the height-for-age indicator (6). More recently, a review focusing on Sub-Saharan Africa reported that out of 8 CT programs, only 1 had a significant impact on children's growth, and that this impact disappeared after 2 y of follow-up (7).

Several authors pointed to a number of factors that may explain these heterogeneous and inconclusive results (7–9). These factors, which include inappropriate targeting, too-brief interventions, and insufficient amount of CTs, relate to the conceptual design of the program and suggest a poor match between the program's theory and the problem it attempts to address. However, a program's failure, or its suboptimal success, may also be the result of inadequate implementation or low uptake of interventions by beneficiaries (10, 11). Documenting both the theory of change of the program (i.e., how the program has been designed to work to achieve impact) and the program implementation is required to identify and address any such failure. This encompasses analyzing the program's theoretical pathways; the coverage, quality, and intensity of activities; and the receipt and utilization of transfers by beneficiaries, in addition to an appraisal of the opinions and behaviors of participants (12, 13). Whereas conventional randomized and quasi-experimental evaluation designs determine whether a program does or does not “work” with regard to primary and secondary outcomes, intertwining theory-driven impact and process evaluations is necessary to fill the implementation gap and to answer the “how” and “why” of programs (14, 15). The recognition of the critical role of implementation science in the field of health and nutrition is relatively new (16). Yet a comprehensive approach that combines impact and implementation assessment will have major policy implications by providing actionable evidence to guide the conception and implementation of future programs.

This approach is nonetheless difficult to put into practice (17) and the use of a Program Impact Pathway (PIP) analysis can be of great value. PIP analysis is a practical approach developed for use in research planning, monitoring, and evaluation of complex programs (18). It is meant to engage implementers and evaluators in brainstorming and developing the theoretical chain of causality between a program's activities, outcomes, and impacts, while considering both the impact and process theories and accounting for the contextual factors that might influence the chain (19). As explained in Avula et al.’s (20) study, the PIP analysis has many advantages, which include, but are not limited to, helping evaluators interpret impact results and identifying corrective actions for implementers. Yet there are few published examples of such analyses related to nutrition programs (20–27), and even fewer when it comes to examples related to CT programs (28). Because the focus is generally on documenting what impact interventions have, and because PIP analysis can be time- and staff-consuming, researchers have not necessarily prioritized this approach. When they have, it was often as a simple link of the research work and the methodology used, which can vary greatly across projects, and these were not described in detail. In recent years there have been calls to emphasize implementation- and process-oriented research in order to understand mechanisms of action and optimize the delivery and uptake of public health interventions. Dedicating articles to PIP approaches, including providing a detailed description of the methodology used and a demonstration of its application, would fill an important gap in evaluative research and hence contribute to the emerging field of implementation science.

We used a PIP approach to tailor the process and impact evaluation of an unconditional CT pilot program in Togo primarily aimed at improving children's linear growth. In this article we describe the principle and methodology of the PIP, which included participatory workshops with the program's actors at multiple levels, and we showcase how we operationalized it for the overall evaluation of the program. Our aim is to share our experience and encourage funders and researchers to engage with this approach for the purpose of conducting more useful comprehensive evaluations of CT programs, as well as for evaluating other nutrition and health programs.

Program description

Principle

In 2014, the government of Togo implemented a 30-mo pilot “Cash-Plus” program in the regions of Kara and Savanes, in partnership with the World Bank and UNICEF. The program provided unconditional monthly CTs (the equivalent of ∼9 USD/mo) to women during the “first 1000 days” (from conception to the child's second birthday), combined with behavior change communication (BCC) activities, with the aim of improving children's nutrition, health, and rights. Beneficiary women were encouraged to adopt “good practices” such as birth registration, attending antenatal care visits, school enrollment, participation in BCC activities, and refraining from placing children < 15 y old in foster care. Women who attended BCC activities assiduously received the equivalent of 35 USD as a bonus when they exited the program, i.e., when the child turned 24 mo old. Implementers labelled these criteria “soft conditionalities.” BCC sessions relating to children's rights were organized monthly by community child protection workers (CCPWs) (Supplemental Table 1). CCPWs also visited women at home to provide advice on how the cash should be used and to discuss particular matters within the family. CCPWs were responsible for organizing CT distributions, handling complaints, and monitoring compliance with BCC activities.

Synergy with Integrated Community Case Management of childhood illnesses and undernutrition

The program implementers decided to draw on the existing ICCM-Nut program (Integrated Community Case Management of childhood illnesses and undernutrition), carried out since 2011 by the Ministry of Health and UNICEF in 565 villages in the same regions. This program aimed to treat common childhood illnesses in communities where health services were inaccessible. In those villages, community health workers (CHWs) were trained and equipped to screen children for malaria, diarrhea, pneumonia, and acute malnutrition and deliver treatment. CHWs were also in charge of carrying out monthly BCC sessions on 13 essential family practices (Supplemental Table 1), as well as home visits.

Setting and beneficiaries

The “Cash-Plus” program was implemented in the 5 districts with the highest rates of acute and chronic malnutrition among children under 5 y old. This represented a total of 273 villages already covered by the ICCM-Nut program. Women who were ≥3 mo pregnant and mothers/caregivers of a child <24 mo or a child <5 y old suffering from severe acute malnutrition were the targets of this intervention. All beneficiaries received CTs for a minimum of 12 mo and a maximum of 30 mo. The program targeted 11,500 women and reached 18,270 women eventually.

Roles of actors

The program involved 4 ministries (Figure 1). The Ministry of Social Action, Women's Empowerment and Literacy (hereafter called the Ministry of Social Action) was in charge of the overall program coordination whereas the Community Development Project (PDC—French acronym), under the leadership of the Ministry of Grassroots Development, Handicrafts, Youth and Youth Employment (hereafter called the Ministry of Grassroots Development), ensured the administrative and financial management. The Ministry of Postal Affairs and Digital Economy, via the Postal Agency, was responsible for cash distribution. The Ministry of Health was in charge of the ICCM-Nut program. These ministries received technical or financial support from the World Bank, UNICEF, and the Fund of Japan. The World Bank and UNICEF cofunded the impact and process evaluation of the program, which was under our team's responsibility.

FIGURE 1.

Simplified chart of organization and roles of stakeholders.

Methods

Theory-driven impact and process evaluation

We used a mixed-method approach to conduct a theory-driven impact and process evaluation of the program. The design of the impact evaluation was as a randomized controlled trial in which mother–child pairs received either CTs and BCC talks (intervention) or BCC talks only (control). Mother–child pairs from both groups were surveyed at the baseline and 2 y afterwards. The process evaluation used qualitative data collected 6–8 mo after the program started—an interval which ensured that the program was rolling out sufficiently, yet allowed enough time to make programmatic adjustments—and quantitative process data collected at the endline. Our methodological approach for the process evaluation followed 5 key steps, as inspired by Rawat et al. (23):

Step 1: develop the theoretical PIP before the program's start;

Step 2: confront the theoretical PIP and adjust it to stakeholders’views 6 mo after the program's start;

Step 3: identify research domains and prioritize research questions;

Step 4: link data collection to the PIP;

Step 5: analyze data and provide feedback to program implementers.

Develop a detailed PIP (steps 1 and 2)

The development of the PIP consisted in laying out the program's theory via the sequence of “inputs–process–outputs–outcomes–impact.” This included the impact theory, i.e., the causal pathways through which the program was intended to achieve impact on the outcomes of interest, and the process theory, i.e., the steps through which the program was expected to be implemented and used. The PIP also described the organizational structure of the actors involved in the program and how they interacted. We first developed the PIP before the program started, using gray and published literature on CT programs and theories of change in addition to the program documentation to which we had access, i.e., the manual of operations of the program, guidelines written for frontline workers (FLWs), tools conceived for the monitoring of activities, and several minutes of preparational meetings. We also conducted 7 nonstructured interviews with the program's stakeholders at the central level; more precisely, we interviewed 2 staff members from UNICEF Togo, 2 from the World Bank, 2 from the Ministry of Social Action, and 1 person from the PDC. Six months afterwards, we organized two 2-d participatory workshops in Dapaong (25–26 March 2015) and Kara (27–28 March 2015)—the main cities in the regions of Savanes and Kara, respectively—to compare the PIP with the reality in the field and adjust it accordingly. The workshops gathered key actors from different sectors (Health, Social Action, Postal Agency, and Grassroots Development) and different levels (regions, prefectures, villages) who were involved either directly or indirectly in the program implementation (Dapaong, n = 31 participants; Kara, n = 35 participants) (Table 1). Representatives from the central level (PDC, Ministry of Social Action, Ministry of Health) were purposely not invited to these workshops so that participants could be free to speak up. In either subgroups or plenary sessions, and with the aid of colored sheets of paper, arrows, and other visual materials, participants were invited to 1) discuss their visions and interpretations of the intervention and its objectives; 2) agree on a list of the intervention activities from inputs to impact and their interrelation, and how and by whom they were implemented; 3) identify the effect modifiers along the impact pathways; and 4) report any bottlenecks and discuss adjustments or corrective actions. The process resulted in a detailed PIP diagram that was further analyzed by evaluators (Figure 2). As facilitators (n = 5 at both workshops), we adopted several measures to ensure that all participants would actively contribute, and to limit hierarchical relationships between them. First, the list of participants was carefully drawn up so that representatives of all sectors and all levels (except the central level) would be present with no direct hierarchical links. For example, CHWs and CHWs’ supervisors were both present at the workshops, but they were not working in the same geographical location. Second, we organized activities wherein participants provided their ideas on sheets of paper anonymously; the papers were then collected and put on the wall for discussion and collective analysis. Third, we favored small group work as much as possible, and 1 person/group was chosen to provide a report to other participants. Although subgroups comprised people from different sectors, we tried to make them homogeneous in terms of levels of responsibility.

TABLE 1.

Participants who attended the Program Impact Pathways workshops in Dapaong and Kara, Togo¹

	Kara (n = 35)	Dapaong (n = 31)
Regional level
Representatives of Community Development	2	3
Accounting assistant	1	0
Representatives of Social Action	2	2
Database manager for the cash transfer component	1	1
Representatives of Health	3	1
Focal point in nutrition or IMCI or IEC	1	3
Representative of the Post Office	1	0
Prefectoral level
Prefects	3	2
Representatives of Social Action	3	2
Representatives of Health	2	2
Chief Medical Officer	1	0
Nurse and nutrition focal point	1	0
Focal point in IMCI	1	2
Village level
Representatives of village development committees	3	3
Supervisors of CCPWs	3	3
CCPWs	3	2
Supervisors of CHWs	1	2
CHWs	3	3

¹Values are ns. CCPW, community child protection worker; CHW, community health worker; IEC, Information, Education, Communication; IMCI, Integrated Management of Childhood Illnesses.

FIGURE 2.

Construction of the program's impact pathway diagram with stakeholders during workshops. AGAIB, Regional Agencies for Grassroot Initiatives [in French]; BCC, behavior change communication; CCPW, community child protection worker; CHW, community health worker; DRAS, Regional Directorate of Social Action [in French]; DRS, Regional Directorate of Health [in French]; EFP, Essential Family Practices; ICCM-Nut program, Integrated Community Case Management of childhood illnesses and undernutrition; PDC, Community Development Project [in French].

Identify research domains and prioritize research questions (step 3)

The information that emerged from the PIP diagram was analyzed and used by the research team to identify research domains and, within each domain, to identify and prioritize key research questions to investigate during process evaluation. These questions related to 1) the program's organization and management; 2) the scale and quality of the interventions’ delivery; 3) exposure to interventions; 4) the utilization of interventions by beneficiary women; and 5) the perceived impact by beneficiaries and the community, including any unexpected effects that the program would have.

Linking data collection to the PIP (step 4)

Research protocols and data collection tools were developed to address each research question. We ensured that the data collected would investigate all domains of the PIP diagram and provide a comprehensive picture of the program. We also ensured that mixed methods and multiple sources of information would be used to allow triangulation of data (29). Building on the PIP process, we collected qualitative data while the program was in progress (April–June 2015) and conducted quantitative data collection at the endline (May 2016). Questionnaires regarding the impact evaluation were also linked to the theoretical PIP developed in step 1.

Analyze data and provide feedback to program implementers (step 5)

The PIP diagram that emerged from the workshops was synthesized and presented by the research team to stakeholders at the central level (PDC, Ministry of Social Action, Ministry of Health, UNICEF, and the World Bank). This meeting was held in Lomé in April 2015 and offered participants the opportunity to provide new inputs to the PIP, receive feedback on implementation facilities and challenges, and discuss the pertinence and feasibility of the programmatic adjustments suggested at the workshops.

Results

PIP

The PIP identified 5 domains from inputs to impact and 3 pathways (Figure 3).

FIGURE 3.

The Program Impact Pathways of the pilot cash transfer program in Togo. AGAIB, Regional Agencies for Grassroot Initiatives [in French]; CCPW, community child protection worker; CHW, community health worker; DRAS, Regional Directorate of Social Action [in French]; DRS, Regional Directorate of Health [in French]; IYCF, infant and young child feeding; PDC, Community Development Project [in French].

Organization and management

Stakeholders organize to transmit social information about the program and spur mobilization, identify beneficiaries, make funds available, establish a contract with the postal agency, recruit and train local program staff including CCPWs, and create appropriate tools for monitoring activities (beneficiary cards and payment forms for CTs, messages and visual aids for BCC talks). The Postal Agency schedules payments in the villages according to the program standards. The CCPWs are equipped with the knowledge, tools, and skills to conduct BCC sessions and home visits. CHWs receive information on the program and advice on how to collaborate with CCPWs.

Delivery of activities

Postal agents are able to distribute the correct amount of cash to women every month. Both CCPWs and CHWs carry out high-quality BCC sessions to inform and raise awareness of women and the community on children's rights and nutrition/health care/hygiene. Both CCPWs and CHWs are able to provide adequate counselling to women and households during home visits. CHWs continue to screen and treat sick and severely malnourished children in the community.

Exposure to activities

Beneficiary women and the community are aware of and understand the program. Beneficiaries receive monthly payments. Women and other members of the community attend BCC sessions every month, ask questions, and receive counselling by CCPWs and CHWs at home.

Uptake of activities

Beneficiary women understand how they should use the cash and use it for their offspring and themselves: for dietary, schooling, health, and/or hygiene expenses, to keep children with them at home, to establish birth certificates, or to make some savings or investments. Some women may use the cash for “antisocial spending,” e.g., alcohol or tobacco. Women assimilate key messages and gain knowledge of children's rights and nutrition/health/hygiene.

Impact

Cash and increased knowledge of mothers act in synergy to 1) improve maternal and young children's feeding practices (the food pathway); 2) improve hygiene practices and maternal and children's health care (the health and hygiene pathway); and 3) improve children's level of education and reduce child labor and trafficking (the children's rights pathway). Ultimately these contribute to improving children's linear growth.

Effect modifiers along the PIP

Many factors that could either facilitate or hinder impact achievement were identified and classified into program-related factors or context-related factors (Figure 4).

FIGURE 4.

Facilitating and impeding factors along the Program Impact Pathways of the pilot cash transfer program. AGAIB, Regional Agencies for Grassroot Initiatives; CCPW, community child protection worker; CHW, community health worker; DRAS, Regional Directorate of Social Action [in French]; DRS, Regional Directorate of Health [in French]; IYCF, infant and young child feeding; PDC, Community Development Project [in French].

Factors that are specific to cash distribution

Issues in the cash flow between the PDC and the Postal Agency, or between the Postal Agency and its decentralized post offices, may occur (e.g., delays in the transfer of funds, a lack of small bills for distributions). Poor road infrastructure, long distances to villages, and a high risk of robbery were cited as factors that could hinder the smooth functioning of CTs. Postal agents emphasized that they only had 7 d to perform the distributions in all villages, which required a great deal of anticipation and organization. At the village level, cash distribution was organized in a central location (at the school or at the village head's home); hence, collecting the money could be time-consuming for women who lived far away. Days and times of distributions could also influence participation in the program. Non–market days and mornings were preferable because women were more likely to be available then. Women occasionally had to pay a gongoneur—a person from the community in charge of spreading information throughout the village, in this case the date of the upcoming distribution—in the event CCPWs would not have time to make the announcement themselves. However, for security reasons, the postal agents were reluctant to disclose the days of distributions in advance, so the gongoneur often ended up spreading the word on the day of the distribution itself. The sharing of cash with husbands or other relatives could prevent women from substantially increasing their income and dilute the impact of the program. Participants all agreed that this practice was likely to be minimized in female-headed households. Likewise some husbands may have decided to reduce the amount of money they traditionally gave to their wives (mostly to buy vegetables and condiments for meals) because of the transfer. In that case the income of the household may have increased, but that of women did not. Lastly, the empowerment of women determined to what degree they could spend the cash as they wished.

Factors related to BCC talks and home visits

This component of the program very much relied on the level of education of FLWs, as well as on their motivation, dynamism, and social integration within the community. The training they received (once at the start of the program, with some later refreshers) and supervision were also important. CCPWs were recruited for this program and had higher workloads than CHWs. CHWs had a specific quota of households to visit and several might work in the same village, whereas there was only 1 CCPW per village no matter its size. Sometimes CCPWs had to do the same BCC talk several times in a month to avoid overcrowding. The numerous home visits further increased their workload. CCPWs were not equipped with bicycles and received less remuneration than CHWs. These differences may have generated jealousy, frustration, and demotivation in CCPWs, despite the program's success being partly based on good relationships and close collaboration between CCPWs and CHWs. Women's time availability was likely to influence their involvement in BCC activities, as was their support from relatives, in particular husbands. Other effect modifiers for knowledge achievement included the education of women (and that of their husbands), the frequency and quality of BCC sessions (facilitation skills and quality of tools), and the coherence between the messages delivered by FLWs and those delivered by other programs in the area.

Factors along the impact pathways (CT and BCC components)

Women were expected to use the cash to put what they had learned with FLWs into practice. However, women's choices may have been affected by the regularity of transfers, their skills in budget management, their empowerment, CT-related family conflicts, social norms and beliefs, as well as pressure from the community and program operators on “how to use the money.”

The food pathway

Women may decide to buy food to improve their children's diets. This pathway may be hindered by a lack of availability of nutritious foods, or by poor dietary practices and diet-related taboos. The intrahousehold allocation of foods may also affect the nutritional impact of the program. In rural Togo, children eat meat on rare occasions, either because the household cannot afford it or because meat is given to men. Most often, children receive pieces of meat at the very end of the meal, when they are already full with cereal- or tuber-made paste.

The children's rights pathway

Women may decide to establish birth certificates or to send children to school, therefore reducing the risk of child labor, forced marriage, or trafficking. In Togo, birth registration is a man's duty because a child “belongs to the father.” Therefore, the involvement of husbands is key in this process. The competence and motivation of civil agents are also important; participants reported that civil offices were often closed and that official documents were sometimes invalid because of agents’ low levels of literacy. Regarding schooling, free meals motivate parents to put children in school and also contribute to better learning and academic success, provided that the teaching is of high quality. Low teachers’ salaries, overloaded classes, and a lack of educational materials are, unfortunately, common matters in this area. Gender inequality may also be a problem: boys are generally sent to school whereas girls stay at home to help with domestic chores, except among the Fuhla ethnic group, who prefer to keep boys at home to watch over livestock.

The health and hygiene pathway

Women may decide to increase their uptake of health services for their child and for themselves in order to improve pregnancy outcomes and increase institutional deliveries, vaccination rates, growth monitoring, and prevention and treatment of childhood illnesses. Distances to health facilities, quality of supply (in terms of equipment, staff number, and skills), and the way patients are treated by medical staff are likely to influence the uptake of such services. For traditional and/or economic reasons, people from the community resort to indigenous medicine, including self-administration of plants and herbal teas and seeking care with “witch doctors.” This practice may be reinforced by an overall lack of trust in modern medicine, and altogether these factors may result in reducing the program's impact through this pathway. In Togo, deeply ingrained habits and customs affect delivery, because women who give birth at home are considered braver than those who deliver at a health facility.

Unspecific additional factors

Some actors raised concerns that targeting women during the first “1000 days” may encourage births. This undesired phenomenon could have occurred among beneficiary and nonbeneficiary women, to permit them either to stay in the program longer or simply to enter it in the first place. Participants also agreed that the program greatly relied on a positive dynamic at the community level. The engagement of village heads, women who organized themselves to mutualize, and FLWs’ dynamism were major factors that may influence impact achievement overall.

Operational troubleshooting

Program bottlenecks identified at the workshops and suggested operational adjustments were relayed to stakeholders at the central level to assess their pertinence and feasibility. Stakeholders, who formulated additional recommendations for the greater success of the program, adopted 5 concrete actions (Table 2):

TABLE 2.

Program bottlenecks and suggestions for operational adjustments from stakeholders at the local and central levels¹

Category	Program bottlenecks	Suggestions at the local level2	Responses/suggestions from the central level3
Comprehension and interpretation of the program
Misunderstanding regarding the “soft conditionalities”
	Local actors think that awoman who misses 3BCC sessions shouldbe excluded from theprogram	The concept of “softconditionalities” should beexplained again to all actors	There is no possible exclusion but onlyunforeseen exits (child's death,definitive moving outside the area). Thepurpose of “soft conditionalities” is topromote good behaviors and practices.They only and partly apply in theory tothe bonus transfer, i.e., only womenwho attend BCC sessions regularlyreceive a bonus at the exit of theprogram. This will be explained againto all actors during the second round ofthe census (beneficiary list update)
Misunderstanding regarding the bonus
	Local actors do not knowwhat are the criteria toreceive the bonus	No suggestion on that point	The criterion to receive the bonus is toattend BCC sessions on a “regular”basis; “regular” needs to be furtherspecified
	Local actors think thatwomen from the controlvillages would receive thebonus if they attend BCCsessions	No suggestion on that point	Women from the control villages whoregularly attend BCC sessions will onlyreceive a diploma; additionalinformation will be disseminated incontrol villages
Implementation of the program
Lack of planning and coordination of activities
	Absence of meetings andtools to manage andmonitor activities	Regular meetings with clearobjectives should be organizedwith local actors. Chronogramsof activities and monitoringtools should be provided tolocal actors	No clear answer on that point
Flaws in communication
	Local actors reportedissues of communicationboth vertically andhorizontally	Regular meetings and monthlynewsletters for betterinformation and discussionabout progresses anddifficulties	The newsletter is approved and will beimplemented by the person dedicatedto communication. Regular meetingswill be organized between regionaland central levels
	Poor communication andcollaboration betweensupervisors of CHWsand CCPWs	Regular meetings betweensupervisors and additionaltraining for supervisors ofCHWs about the programbecause they were not invitedto the initial training sessions	A “catch-up” training will be organizedfor CHW supervisors and they will fromnow on be invited to refresher courses
Frontline workers’ workloads
	CCPWs’ workloads arehigh in large villages,with no additionalfinancial compensation.They have too manyhouseholds to cover	Quotas of households to becovered by CCPWs should beimplemented, as is done forCHWs	Quotas of households per CCPW will bedetermined
	Disparities in salarybetween CCPWs andCHWs	No suggestion on that point	Material conditions between CCPWs andCHWs will be harmonized
BCC sessions: content and participants
	There is no specific sessionon budget management	A session on householdeconomy and budgetmanagement should be added	The program does not intend to promotesavings or income-generating activitiesbut rather to increase knowledge andadoption of good practices; no sessionon budget management will be added
	Men should be targeted aswell (especially for birthregistration)	Men should be furtherimplicated in program activities	Husbands will be invited to take part inBCC sessions
	Little information isprovided on familyplanning yet the programtargets pregnant womenand mothers of youngchildren	Further information about familyplanning should be provided atBCC sessions and/or homevisits	Family planning is performed at thehealth center. CHWs mention theimportance of birth spacing in a BCCsession
Cash distribution
	Delays in distribution dueto problems with photoson beneficiary cards	Equip CCPWs’ supervisors with acamera to take the pictures ofwomen when they performtheir supervision rounds	AGAIB has to solve the problem. CCPWs’supervisors are already overloaded
	Delays in distributionbecause some womenare not on the list (delaysin the list update)	Urgently organize the secondround of the census to avoidmisunderstandings andfrustrations	The second round of the census isongoing (May–June 2015)
	Some women live very farfrom the payment point	Organize several payment pointsin specific villages	Payment points must be located <5 kmfrom beneficiaries; distances will bedouble checked
	Delays in distributionbecause the planning ofthe Postal Agency isextremely tight	A collaboration between thePostal Agency and prefecturalactors could optimizescheduling and itineraries whileconsidering village accessibilityand security risks	This collaboration already exists, but thePost Office should inform the CCPWsin case of delays
	Long waiting time at thepayment point	No suggestion on that point	The CCPWs often call women in wellbefore the scheduled time, so waitingtime is even longer
	CCPWs have to informwomen about thepayment but they are notinformed early enough	No suggestion on that point	The program could cover the expensesfor the gongoneur (someone from thecommunity who is in charge ofspreading information throughoutthe village)
	No effective complaintmechanism for theprogram (CCPWs haveto deal with complaints)	A “complaint office” specific tothe program should beimplemented at the villagelevel in addition to the freehotline which is common toall PDC programs	A “complaint office” will be implementedin each village. The village heads willbe in charge of collecting complaintsand dealing with conflicts generated bycash transfers. This will take the weightoff the CCPWs
	Payrolls are not correctlyfilled out by the postalagents (errors in datesand IDs)	Further training should beprovided to postal agents	The Post Office will remind all its agentshow to correctly fill out the forms
Other problems: lack of engagement and motivation
	Heads of villages are notengaged, although theyhave great roles to play(encouraging men to	Heads of villages could befurther involved and rewardedfor a better roll-out andutilization of the program at	Rewarding heads of villages is notplanned; they will be involved whenthey take charge of the “complaintoffices”
	participate, testifying asto the validity ofbeneficiaries’ ID, etc.)	the community level
	Civil agents are not awareof the program	Supervisors of CCPWs couldraise the awareness of civilagents when they do theirmonthly supervision round.Civil agents should beequipped with bicycles foran outreach strategy	It is the role of CCPWs to raise theawareness of women about birthregistration. Raising the awareness ofcivil agents may lead to increases inbirth registration fees (which are notfixed). However, civil agents makeerrors when establishing the certificatesand this should be corrected

¹AGAIB, Regional Agencies for Grassroot Initiatives [in French); BCC, behavior change communication; CCPW, community child protection worker; CHW, community health worker; PDC, Community Development Program [in French].

²Obtained through the regional Program Impact Pathway analysis workshops.

³Obtained through the workshop in Lomé.

1. Improve information and communication through additional refresher trainings of FLWs, monthly newsletters, and more regular meetings between stakeholders;

2. Reduce CCPWs’ workloads by recruiting new agents and applying quotas of households, and improve their work conditions (wage and equipment);

3. Cover the expenses for gongoneurs to help CCPWs inform the public about payment scheduling;

4. Invite husbands/partners to BCC sessions to enhance support to their wives;

5. Implement a “complaint office” at the village level, under the responsibility of the village head, to address grievances and fix conflicts within the community.

Other suggestions were dismissed either because of budget, time, or logistical constraints, or because they were not considered sufficiently pertinent or enough of a priority. As an example, the local level suggested adding a BCC session on family planning; however, the central level decided not to address this topic because it was already handled by the health center (Table 2).

Key research questions and tailored methods for data collection

We articulated process-related research questions within each domain of the PIP. Table 3 presents examples of questions. These questions related to 1) the program's organization and management (coordination, communication, management tools); 2) the scale and quality of the interventions’ delivery (workload, motivation); 3) the exposure to interventions; 4) the utilization of interventions by beneficiary women; and 5) the perceived impact by beneficiaries and the community, as well as any unexpected effects that the program might have. For each question, we determined at which levels data should be collected—“agents,” “beneficiaries,” or “nonbeneficiaries”—as recommended by Guba and Lincoln (30) (Table 3). Regarding the program's organization and management, we decided to focus on multiple levels to obtain a comprehensive picture of the process and perceptions from all stakeholders. For questions relating to service delivery, we planned to collect data from FLWs and their supervisors, and from all types of agents involved in the cash flow, from the central level to the beneficiaries. Data on the program's utilization and impact levels were collected at the household level, including not only beneficiary women but also their husbands and any other members who may influence women or be indirectly affected by the program (husbands, mothers-in-law). We identified husbands as key interview targets, because we learned from the workshops that they had the responsibility of birth certification of children. Women from the control villages were also interviewed because they attended BCC sessions and were also aware of CTs in other villages. We used a variety of data collection methods, which combined qualitative and quantitative approaches to allow triangulation of results and sources. These included a review of the program's documents; semistructured interviews with the program's implementers (n = 79), beneficiary women (n = 60), their husbands (n = 20) and mothers-in-law (n = 10), and nonbeneficiary women (n = 20); and focus group discussions with beneficiary women (n = 15), their husbands (n = 5), and nonbeneficiary women (n = 10). Direct nonparticipatory observations of cash distributions (n = 10), BCC sessions (n = 20), and home visits (n = 10) were also conducted to document the reality of implementation (organization, environment, speakers’ skills, interactions between implementers and beneficiaries, existence of question-and-answer exchanges, etc.) and participation (number of participants, beneficiaries’ behavior, etc.). Quantitative surveys among beneficiary and nonbeneficiary women were also organized (baseline and endline surveys). These complementary methodologies allowed us to assess the program from different angles.

TABLE 3.

Key research questions emerging from the Program Impact Pathways and methodology to address them¹

Research domains	Key research questions	Level of information required	Methods for data collection	Specific topics to address
Organization Management	What is the level ofinformation andengagement of thestakeholders fromdifferent sectors?	Stakeholders, alllevels	Interviews	Individual perceptions of rolesMultisectoralityProgram management
	How are the program'sactivities coordinatedbetween sectors?	Stakeholders, alllevels	Interviews
	Do stakeholders have anymanagement tools(specifications, manualof operations, workplan/timeline) to facilitateimplementation andmonitoring of activities?	Stakeholders, alllevels	Document review,interviews
	How do stakeholderscommunicate? Whatare the communication/reporting flow and tools?	Stakeholders, alllevels	Interview
Delivery of activities	What are theimplementationcharacteristics of theactivities (how frequently,which messages, whichorganization, etc.)?	Stakeholders, centrallevel	Document review,interviews	Work conditions of FLWs and other stakeholdersQuality of delivered activities
	What is the impact ofactivities on CCPWs’workloads? What factorsmotivate and sustain theirperformance?	CCPWs andsupervisors	Interviews
	What is the quality of themessages provided byFLWs to women at BCCsessions? What is thequality of counsellingprovided by FLWs towomen at home visits?	CCPWs andCHWs	Observations
	What are CHWs’perceptions of theaddition of the program'sactivities to their existingportfolio of activities?	CHWs	Interviews
	What is the cash flow—andits difficulties—fromfunders to distributionto beneficiaries?	World Bank, PDC,Postal Agency,CCPWs, beneficiarywomen	Interviews,observations
Exposure to activities	What is the level ofexposure of women (andtheir husbands) to BCCsessions and home visitsby FLWs? What factorsinfluence this exposure?	Women, husbands,FLWs	Document review,interviews, endline	Influencing factors
	What is the level ofexposure of women toCTs? What factorsinfluence this exposure?	Women, husbands,CCPWs, postalagents	Document review,interviews, endline
Utilization	What is the level ofinformation and what arethe perceptions ofbeneficiary women aboutthe program (CT, BCCsessions, and home visits)?	Beneficiary women	Interviews, FG,endline	Program methods, including soft conditionsCash–BCC synergyWomen's empowerment
	What are the perceptionsabout the programamong the community orkey influential familymembers?	Family members	Interviews, FG,endline
	What is the level ofinformation and what arethe perceptions ofwomen from controlvillages about theprogram?	Nonbeneficiarywomen	Interviews, FG,endline
	By whom and how is thecash used within thefamily?	Beneficiary women	Interviews, FG,endline
Impact	What is the impact of theprogram on behaviorsand practices?	Beneficiary women	Baseline and endline	Conflicts or solidarityBirth promotion
	What is the impact of theprogram on nutritional/health/rights outcomes?	Beneficiarywoman–child pairs	Baseline and endline	Interaction with other programsMarket prices
	What is the social impact ofthe program within thefamily? Within thecommunity?	Family members,community	Interviews, FG,endline
	Does the programgenerate undesiredeffects?	All	Interviews, endline

¹BCC, behavioral change communication; CCPW, community child protection worker; CHW, community health worker; CT, cash transfer; FG, focus group; FLW, frontline worker; PDC, Community Development Program.

Discussion

This article demonstrates how a PIP analysis contributed to the process and impact evaluation of an unconditional “Cash-Plus” program in north Togo. This approach proved valuable in clarifying mutual objectives, architecture, logic, and linkages between the program's “boxes,” and was particularly relevant for such a multisectoral nutrition-sensitive strategy. We identified 3 main PIPs, all based on the assumption that CT and BCC components would act in synergy for impact achievement. The “food pathway” implied that the purchase of foods, along with increased maternal knowledge on child feeding practices, would lead to both quantitative and qualitative improvement in children's diets and nutrient intakes. The “health and hygiene pathway” implied a better use of health services and improvement in hygiene practices, leading to better preventive and curative management of health in childhood. The “children's rights” pathway implied better attendance and performance at school, as well as improved legal existence and better-documented ages of children, with a direct impact on their protection and well-being and an indirect impact on their health and nutrition. Along these pathways, the analysis highlighted many program- and context-related factors that may come into play in influencing the program's operation and effectiveness. Intensive communication between stakeholders, smooth cash flow, and adequate training and motivation of FLWs were among the key factors for a successful program implementation; whereas, the irregularity of CTs, women's lack of time availability, a lack of support from family, low availability of nutritious foods, and poor offers of proximate health and school services were identified as potential barriers to utilization of the program. The number and variety of factors show the complexity of implementing this type of program in this context in real-world conditions and point out the risks of the program's failure at multiple levels.

The PIP analysis helped define salient research questions and key indicators that should be investigated during impact and process evaluation. Process-related research questions were guided by 5 key domains: 1) program management; 2) delivery of interventions; 3) exposure to interventions; 4) utilization by beneficiaries; and 5) impact as perceived by beneficiaries and the community. Once these domains were identified, the research team was able to plan data collection activities using mixed methods and multiple data sources. The unexpected effects of the program which surfaced from the PIP—in our case, the risk of increased pregnancies in both the intervention and control villages, or the fact that women may be threatened regarding the use of cash—were also integrated into the research portfolio. Lastly, because we do believe that the PIP analysis and process evaluation were performed at the right time in the program implementation cycle (not too early, not too late), it helped to report programmatic issues from the field and communicate with decision-makers to contemplate program adjustments. A similar approach was used in Nigeria as part of the evaluation of a community infant and young child feeding counselling package (31). However, in that study, the PIP, which was initially drafted before the program implementation, was reviewed and revised 18 mo afterwards (instead of after 6 mo as in our study). The revision used routine monitoring data, information from multiple discussions and meetings, observations of activities, and interviews with community actors and beneficiaries. According to the authors, their goal was to “revise the initial PIP to better reflect the realities on the ground and incorporate new information from the mid-term assessment,” and it seems that the process occurred after programmatic adjustments had been made (32).

The PIP approach was not without challenges and limitations. Because actors from all levels—except the central level—were invited to participate in the workshops, top-down relationships were difficult to avoid. As facilitators, we favored group work and tried to avoid strong hierarchical links within each group so that all participants could speak freely. We paid close attention to ensure that every participant had a chance to express their thoughts over the 2 d. We must confess that some stakeholders from the central level were surprised, and possibly unhappy, at not being invited. We deliberately adopted this bottom-up approach to ensure that actors who were closest to the program's reality were not alienated from the process (33) and to reduce the risk of self-censoring. On the other hand, this did not favor interactive exchanges between those who take action and those who make decisions. We believe that the presence of the central level would have been helpful to justify certain decisions—for example, the fact that there was no BCC session on family planning—or clarify some aspects of the program. Nevertheless, we are confident that we did not miss any major information, because we had the opportunity to talk to the central level both before the workshops and afterwards. In contexts with no budget, time, or distance constraints, it may be wise and valuable to save a half-day at the end of the workshop, or to plan on having an extra day, to invite actors from the central level. Another use of the PIP we did not mention is to define monitoring indicators and to use it as a frame for regular checks on progress or for fidelity assessment of the program (34). Like other authors, we acknowledge that our PIP was not sufficiently updated over the course of the program to fulfill this objective (35), primarily because of budget constraints. Furthermore, there was not enough utilization of the PIP by the program's actors to implement their internal monitoring and evaluation. Anticipating the way results will be used by all and improving communication strategies between evaluators and stakeholders are essential pieces in making the best use of the PIP analysis. However, engagement in the process can be time-consuming for both the evaluators and program implementers because it requires continued dialogue before, during, and after program roll-out. In that sense, a PIP analysis also requires a significant investment in terms of human resources from both sides.

Beyond the PIP diagram itself, and despite these limitations, we do think that the whole process was remarkably useful, both for the program implementers and for the evaluators. It required the early establishment of strong implementer–evaluator engagement and interactions, as recommended by Menon et al. (36), to achieve meaningful evaluation. It also invited evaluators and stakeholders to agree on a common vision of the program's objectives and logic. For example, interpretation of the “soft conditionalities” was somewhat divergent between all parties. At the central level, these “conditionalities” were considered strong recommendations that should be adopted by women to ensure the program's success. From the evaluators’ point of view, the program was unconditional because failure to adopt these recommendations did not result in any sanction or exclusion. At the community level, some actors thought that the recommendations had to be adopted by women; those actors threatened women for legitimate uses of cash, either telling them that they would be excluded from the program or that the police would come and arrest them for misusing the funds. The PIP analysis allowed for the identification of this divergence and generated back-and-forth discussions that helped harmonize perceptions and practices concerning the matter. Moreover, the program's actors were more likely to use the results of the evaluation if they contributed to it personally and understood the purposes and steps of its construction (37). Although there is not a single way or timing for mapping a PIP, most studies have traditionally used desktop reviews of the program's documentation and interviews with stakeholders at the central level (20, 22–24), but very few mentioned the use of participative workshops, and even fewer used this participative approach while the program was already in progress. We chose to use this last method to refine the initial theoretical PIP diagram with information drawn from the reality in the field. The workshops put program implementers from different levels and sectors together, representing opportunities for dialogs around the program's objectives, interpretations, and challenges, and to discuss the means to improve collaboration. The agents from the Postal Agency could, for example, hear why CCPWs were complaining about late announcements of cash distributions; in turn, CCPWs could hear postal agents explain their reluctance to announce distributions in advance (for security reasons). Other participants contributed by sharing their own experiences in their localities, such as their reliance on gongoneurs, which was a worthy solution to the problem of poorly timed announcements. Eventually participants reached a mutual understanding of the program, which generated a feeling of common purpose and fostered programmatic integration across and within sectors (19). In this “Cash-Plus” program, inadequate communication across sectors, burdensome workloads, and low financial compensation of CCPWs were identified as crucial points to be addressed. The workshops were facilitated by 5 experts, which helped to galvanize and supervise the discussions.

In conclusion, this article detailed the application of a PIP analysis to an unconditional “Cash-Plus” program in rural Togo. Our research and methodological choices for the process and impact data collection and interpretation were guided by engaging implementing actors from multiple levels in a participatory and iterative process. The PIP analysis has taught us many lessons. First, the intervention is likely to have multiple possible pathways for impact achievement. We identified the “Food,” “Health/hygiene,” and “Rights” pathways as relevant to this study. This implies that many indicators must be monitored if we want to obtain a global picture of the program's impact and mechanisms of actions. Second, the intervention may have unintended effects—e.g., increased pregnancies, reduced money allocation from husbands, or threats against women made by the community—that should be carefully investigated. Third, although women and young children were the intervention's main targets, it appeared that other community members (husbands, mothers-in-law, village heads) had roles in the program's success and hence should be part of the evaluation. Fourth, a wide range of factors that could compromise activities or outcomes were identified. Among them, a number of factors relating to the conception or implementation of the program could be improved: these include, but are not limited to, the lack of communication between stakeholders, excessive workloads and insufficient incentives for FLWs in charge of delivering activities, the short notice of announcements of CTs in villages, insufficient involvement of husbands/partners, and the absence of an effective complaint system. The avenues for improving these matters that were suggested in this study can certainly be used for replicability (38). Participatory workshops with local actors held after the program had been active for 6 mo were highly valuable for both evaluators and participants. For evaluators, these workshops provided an intimate knowledge of the program in real-world conditions and of the context of implementation; for participants, they promoted staff motivation and commitment, programmatic integration, communication, and solidarity. These benefits may be reinforced when dealing with nutrition-sensitive programs with multiple and indirect pathways for which local field implementers might have difficulty envisioning how the final impact will be achieved. Overall this article calls for a wider use of PIP analysis when evaluating complex and multisectoral programs, particularly CT programs with nutritional objectives for which the “black box” has not yet been fully decoded. This article also invites researchers who use this approach to publish their work so we can build on each other's experiences in order to improve and standardize the methodology.

Notes

Supported by the World Bank and UNICEF Togo. JB received a PhD research grant from the IDeX (University of Bordeaux Initiative of Excellence) scholarship program through the doctoral school SP2 (Sociétés, politique, santé publique, University of Bordeaux), which funded half of her PhD; the other half was funded by the research project, with a complement provided by the Institut de Recherche pour le Développement.

Author disclosures: The authors report no conflicts of interest.

Supplemental Table 1 is available from the “Supplementary data” link in the online posting of the article and from the same link in the online table of contents at https://academic.oup.com/cdn/.

Abbreviations used: BCC, behavior change communication; CBI, cash-based intervention; CCPW, community child protection worker; CHW, community health worker; CT, cash transfer; FLW, frontline worker; ICCM-Nut program, Integrated Community Case Management of childhood illnesses and undernutrition; PDC, Community Development Program [in French]; PIP, Program Impact Pathway.

Ethics

The study, including the PIP analysis and the process and impact evaluation, received ethical clearance from the Ministry of Health of Togo and from the consultative ethics committee of the Institut de Recherche pour le Développement in France.