Lessons learnt from the CERCA Project, a multicomponent intervention to promote adolescent sexual and reproductive health in three Latin America countries: a qualitative post-hoc evaluation

Highlights

• •Large projects like CERCA require a preparatory phase and a thorough analysis of the situation at the country level as well as community-specific context information, to which sufficient time and budget should be attributed.
• •An essential part of community-based intervention and action research is to adapt the monitoring strategy if the interventions are changed and to thoroughly document all activities.
• •Accepting teenage pregnancies as a complex issue, not only means adapting interventions to this complexity, but also the evaluation design. In the case of the CERCA Project an overall evaluation should have included a component for each activity.
• •Further, the evaluation rightly addressed the effectiveness of the intervention package but should have extended this to address their acceptability, and factors that helped or hindered their application among all target groups and other stakeholders.
• •Through the concerted application of a multi-faceted strategy, it is possible to decrease the taboo on adolescent sexuality in a conservative setting.

Abstract

The Community-Embedded Reproductive Health Care for Adolescents (CERCA) Project was implemented in Bolivia, Ecuador and Nicaragua (2011–2014) to test the effectiveness of interventions preventing teenage pregnancies. As the outcome evaluation showed limited impact, a post-hoc process evaluation was carried out to determine if and how CERCA’s design, implementation, monitoring and evaluation affected the results. We did a document analysis and conducted 18 in-depth interviews and 21 focus group discussions with stakeholders and beneficiaries. Transcripts were analyzed using directed content analysis.

Data showed that CERCA sensitized stakeholders and encouraged the discussion on this sensitive issue. In terms of design, a strong point was the participatory approach; a weak point was that the detailed situation analysis was completed too late. In terms of implementation, a strong point was that multifaceted activities were implemented; a weak point was that the activities were not pilot tested for feasibility/acceptability and evolved substantially throughout the Project. In terms of monitoring, strong points were that regular monitoring kept the Project on track administratively/financially; a weak point was that monitoring indicators did not change as the intervention package changed. In terms of evaluation, weak points were the substantial attrition rate and narrow focus on adolescents.

This study provides recommendations for future projects.

1. Introduction

The need for effective interventions to improve adolescents’ sexual and reproductive health (ASRH) in Latin American continues to be pressing. In this region adolescents (10–19 years old) initiate sexual activity at an increasingly earlier age and only a minority is taking any precautions to prevent sexually transmitted infections (STIs) or pregnancy (Ali & Cleland, 2005). Latin America is the only region in the world where births among girls under the age of 15 years are increasing and are projected to rise through 2030 (UNFPA, 2013). As abortion remains highly restricted in the region, young girls faced with unwanted pregnancies often choose to have an illegal termination. The rate of unsafe abortions in Latin America amounts to 25 per 1000 girls aged 15–19 years (Shah & Ahman, 2012).

In response to this, the Community-Embedded Reproductive health Care for Adolescents (CERCA) Project, was implemented in Nicaragua, Bolivia and Ecuador (Decat et al., 2013). The Project aimed to reduce teenage pregnancies by improving adolescent communication with parents, partners and peers on SRH issues, access to accurate SRH information, use of SRH services in primary health settings, and use of modern contraceptives. Interventions were implemented from September 2011 to April 2013 in randomly chosen town districts in Managua (Nicaragua) and in purposively selected secondary schools in Cochabamba (Bolivia) and Cuenca (Ecuador). The implemented activities addressed adolescents and their environment including parents, community members, local authorities and health care providers; they were tailored to the local contexts and needs (Córdova Pozo et al., 2015).

To evaluate the effectiveness of CERCA, a quasi-experimental study was set up. Two cross-sectional surveys were conducted among adolescents before and after the intervention (20 months apart). A nested cohort analysis assessed the changes in selected behaviours among the adolescents from the intervention and control groups. This evaluation demonstrated limited impact. In Ecuador, it found slight improvement in condom use and improved knowledge and use of health care services in the intervention group compared to the control group, while positive changes in communication on SRH topics were reported in Bolivia. No significant changes were found in Nicaragua. These results need to be interpreted with caution, as their validity is questionable due to a high attrition rate of 70% between waves (Decat, 2015). Nevertheless, the results seem to align with other evaluations. Whereas some interventions in the field of ASRH have shown positive results (Jewkes et al., 2006), many other long-term, multi-component interventions have encountered similar problems in demonstrating clear effects of their efforts using (quasi-)experimental designs (Campbell et al., 2013, Doyle et al., 2010; Pettifor, MacPhail, Bertozzi, & Rees, 2007; Wight, Plummer, & Ross, 2012).

To understand the possible reasons for these findings, a qualitative post-hoc process evaluation was conducted in order to answer the main research question: which factors determined the outcomes of the CERCA project on the level of its design, implementation, monitoring and evaluation? Two main objectives for this evaluation were formulated. First, to study additional outcomes of the CERCA intervention not studied by the initial evaluation. Following Campbell et al. (2014), who did a similar qualitative study of an HIV prevention intervention in Zimbabwe, we “seek to contextualise the program more widely, focusing on the extent to which the intervention was able to generate social environments that supported the possibility of health-enhancing behaviour change”.

Second, to identify problems and facilitating factors in the design, implementation, monitoring and evaluation of the CERCA intervention that may have influenced its outcomes. A systematic review by Shepherd et al. (2014) indicated that such process evaluation are crucial in explaining the results of outcome and impact studies. While there has been a gradual increase, thorough process evaluations are rarely published in academic journals. In Shepard’s review, only three process evaluations were scored medium or high weight of evidence for both trustworthiness and usefulness. In assessing the strengths and weaknesses in the design, implementation, monitoring and evaluation processes, we aim to derive lessons for future community-based intervention research projects and contribute to the on-going discussion on evaluating complex community-based interventions using experimental versus alternative designs (Campbell et al., 2014; Ketting, Friele, & Michielsen, 2015; Laga, Rugg, Peersman, & Ainsworth, 2012).

2. Methodology

The post-hoc evaluation followed multiple steps. First, a range of documents relating to the CERCA Project was analyzed by the leading researchers in the coordinating institution. This included the project protocol, scientific publications, country reports and monitoring sheets. Based on these documents, we developed the theory of change of the CERCA intervention for the three settings and assessed the design of the intervention.

Second, data were collected through key informant interviews (KIIs) and focus group discussions (FGDs) with the key stakeholders of CERCA – adolescents, parents, teachers, community leaders, peer educators, health care providers, project leaders at the country level, implementers at the country level and the international consortium management team. The interview guides were designed by the coordinating institution and were validated by each partner. The interviews included similar topics for all respondents. In total, three guides were developed for the FGDs and two for KIIs. They were translated in Spanish, and translations were revised by each partner institution in order to adapt the guides to the specifics of the local context. The guides addressed the following topics: main achievements of CERCA, objectives and general set-up of the intervention, factors influencing implementation and participation. For the consortium management, questions on the effectiveness evaluation and on monitoring were included.

Data collection took place in October-November 2014 in Cochabamba (Bolivia), Cuenca (Ecuador), Managua (Nicaragua) and Ghent (Belgium). In the three Latin American settings, a combination of convenient and purposive sampling of the target community which had undergone the interventions (e.g. adolescents, health care providers) was used (Luborsky & Rubinstein, 1995), including both stakeholders that had been very much involved in the project, and those that weren’t. Participants were recruited by the local researchers in the communities, schools and health care centres of the three settings. Project leaders and implementers at the international and country level were subsequently contacted by e-mail. In total, 18 KIIs and 21 FGDs were performed, involving 153 respondents [Table 1]. KIIs and FGD lasted approximately one hour. All FGDs and KIIs were recorded with the consent of the respondents. The language used to conduct the FGDs was Spanish. KIIs were performed in English and Spanish. The interviews and FGDs were transcribed and translated (where needed).

Table 1.

Number of focus group discussions and key informant interviews.

	Belgium	Bolivia	Ecuador	Nicaragua
Focus group discussions
Adolescents	–	3	3	3
Parents	–	2	2	2
Health care providers	–	1	2	–
Friends of Youth/health promoters	–	–	1	2
Key informants interviews
Health care providers	–	1	2	–
Community leaders		2	2	2
Country implementers	3	–	–	–
Project leaders	3	–	–	–
Consortium management	3	–	–	–
Total number of KII and FGDs	9	9	12	9
Total number of participants	9	43	55	46

The study was approved by the Ethics Committee of the Ghent University Hospital. Everyone who agreed to participate in the study provided written informed consent. All respondents were informed that they could withdraw from the study at any time. Anonymity and confidentiality of participants were ensured throughout the study.

In addition, we included, in the analysis, the discussion notes taken during a three-day meeting of the researchers from the coordinating institution, the country project leaders and the study advisor from WHO in Belgium (Ghent, Belgium, 8–10 December 2014). In this meeting, all CERCA partners reflected on the Project, elaborated on the qualitative analysis and developed codes for the content analysis. This meeting was recorded, but not transcribed.

Data analysis was done in three steps. First, the initial data were analyzed by the leading investigators of the coordinating institution using directed content analysis based on the specific areas of interest to the project (Hsieh & Shannon, 2005). As more data were collected and reviewed, codes were grouped into concepts, and then into categories to create a code book. The broad set of codes was developed and discussed during the meeting in Ghent, and allowed for grounded coding. Further, data were analyzed by each partner institution separately; in each country two researchers independently coded the transcripts and cross-checked the obtained results. After finalizing the coding, the lead investigators in Ghent performed a quality check on the coded transcripts from all study settings, and requested clarifications were necessary. Finally, relevant quotations were selected based on frequency and richness to illustrate key points.

3. Results

The results section will subsequently present the strengths and weaknesses in the development, implementation, monitoring and evaluation of the CERCA intervention, and conclude with additional identified results.

3.1. Development of the CERCA intervention

3.1.1. Document analysis

The CERCA Project aimed to study the effect of a multicomponent intervention on pre-defined outcomes – preventing teenage pregnancies and outcomes contributing to that – improved adolescent communication on SRH with family, peers and partners, access to accurate SRH information, use of SRH services at the primary care level and use of modern contraceptives. Document analysis showed that the intervention was developed based on theories (socio-ecological models and cognitive behaviour theories), existing evidence found in literature and the intervention mapping technique¹ (Decat et al., 2013). A needs assessment, including a situational analysis, the baseline survey, evidence-based activities identified in the literature, and ongoing feedback of the target groups were used to define the intervention activities.

Since the Project was intended to be action research and changes were anticipated during its implementation, a theory of change was not explicitly developed before the start of the intervention. Based on the documents, we developed a post-hoc theory of change that illustrates how the CERCA intervention aimed to produce the desired outcomes (Fig. 1): a range of activities targeting different stakeholders aimed at influencing a variety of areas of action that in turn, aimed to influence outcomes including communication, contraception use, and health seeking behaviour. Based on the documents it was not possible to determine direct links and relations between all aspects of the flow chart, neither did it include process and context indicators.

Fig. 1.

Post-hoc theory of change for Bolivia.

3.1.2. Qualitative study

The results of the qualitative study are presented here according to the Project’s phases which include design, implementation, monitoring and evaluation. Furthermore, we included in this section the main achievements of the Project and results that were not captured during the original effectiveness evaluation.

3.2. Design of the intervention

During the interviews and FGDs three topics were frequently mentioned by the respondents – the importance of a shared vision on main study objectives, the lack of a community-level situational analysis, and the variety of cultural and professional backgrounds in the study team.

3.2.1. Shared understanding of what the Project was intended to achieve

While there was overall agreement among the consortium partners that the reduction of teenage pregnancies was the main Project goal, national and international project leaders agreed that it was not realistic to achieve within the timeframe of the intervention. It was more ‘a topic behind all the objectives that allowed sensitizing all target groups’ (Project Leader). However, they agreed that the Project had a potential to achieve its objectives. All Project partners remembered most of the specific objectives, although different emphases were placed in the different settings. This emphasis was linked to the background, experiences and expertise of the local partners (also see section Study team backgrounds and perspectives).

‘In Ecuador they put a lot of efforts to improve adolescents’ accessibility to health services. But I think it was related to the research team background, they came mostly from the medical field. And teams from Nicaragua and Ecuador they were from another area so they worked more on other issues.’ (Consortium Management)

When asked about the objectives of the intervention, it was clear that community members had good knowledge and awareness in all three countries. The most frequently mentioned objectives were reducing STIs and adolescent pregnancies, improving access to health care services, providing better sexuality education for adolescents, improving communication between parents and adolescents about sexuality, preventing sexual intercourse at an early age, and promoting the use of condoms.

‘The main objective of the CERCA Project was less STIs if they had sexual relations or less pregnancies, more communication in the family and between individuals, that is to say give them more confidence so they can talk about the topic and prevent pregnancies and social problems’. (Adolescent)

‘For me it's educating adolescents on sexuality so they are better informed on issues of abstinence, safe sex, STDs, and if they are well informed, the possibility of launching into having sex and getting pregnant or catch a disease is much lower’. (Parent)

3.2.2. Importance of in-depth community-level situational analysis

The qualitative post-hoc study confirmed that the CERCA Project was evidence- and theory-based and used community involvement and principles of action research. Although the overall goals for CERCA were the same in each setting, intervention activities were developed in a participatory manner with the target groups based on the available research evidence and programmatic experience on the existing interventions in each country. Furthermore CERCA planned to have a situational analysis in each country, as well as a qualitative study in each of the study settings to identify cultural specificities. Nevertheless respondents stressed that the design phase was carried out before a detailed community-level qualitative situational analysis was available. While it generated important insights, it was delivered too late to be used as input for the development of the activities.

‘I think it is important to involve the community, to know the problems from the inside and to make the community active. It is a part of the solution’ (Country Implementer)

‘…the qualitative analysis came late, when our interventions were already planned…’ (Project Leader)

This approach led to many changes during the course of the Project to meet the needs in each context, resulting in diverse activities in the three settings.

3.2.3. Study team backgrounds and perspectives

The activities were developed by the local study team and seemed to partly depend on their interests and capacities. Specifically, in Ecuador, where most members of the study team had a medical background, they focused more on access to health care services.

‘I think the activities were adapted to the local context but they were also adapted by local team group, so I think that, yes, all activities were adapted to local context and based probably on the capacities of the team.’ (Consortium Management)

It is noteworthy that several respondents emphasized difficulties in the development and start-up of the intervention, related to different perspectives in the study team. The consortium consisted of a wide variety of organizations and people with different backgrounds, which were identified as both an added value and a barrier.

‘It was very helpful to have very different expertise in our team, and communication with people with different opinions, they open new angles of approach to this problem.’ (Consortium Management)

‘I think that the problem was that we actually did not have enough time to work as a group, as a multi-disciplinary and multi-country group […]. I think that we needed more time to develop a very clear perspective, a common position. We needed even more time to discuss, to have different opinions and then consolidate our common vision.’ (Consortium Management)

3.3. Implementation of the CERCA intervention

As the intervention activities were developed and adapted throughout the Project, it is not possible to compare the planned intervention with the actually implemented activities, as a standard implementation evaluation would do. Instead, we will describe strengths and weaknesses of the implementation, as identified by the respondents.

The overarching observation was that the active involvement of policy makers, health leaders, community and religious leaders in Project activities was crucial for the successful implementation of the intervention. In addition, an enabling local policy environment played an important role in the successful implementation of the intervention and contributed to the creation of necessary alliances and networks, e.g. in Ecuador.

‘…from the beginning, since we started the project, we searched for mechanisms that allowed us to generate continuity of the project activities. To continue the activities of the intervention at the community level in Cuenca, we formed the SRH network together with 26 governmental and non-governmental institutions’ (Project Leader)

‘… alliances are essential to conduct all activities. We worked with political leaders at community level that allowed us to dialogue with health authorities and identify all the problems and sexual and reproductive health needs of adolescents in order (that) health care facilities can provide care to adolescents.’ (Country Implementer)

Other strengths that were regularly mentioned by the respondents were the creation of places for young people to get advice, active adolescent outreach activities (e.g. through mobile health consultations) and the use of young promoters or friends of youth.

‘The doctors and psychologists, who came here [school] in case someone had a question or an emergency and could advise you, help you or recommend you a real specialist’ (Adolescent)

‘People who came to deliver educational activities were young people. It was not a doctor or an adult, it was a young person educating another young person. This allowed us [adolescents] to have interactions with more trust’ (Adolescent)

The respondents identified a number of factors that hindered the process of implementation.

In line with the action research methodology it was aimed to adapt the interventions continuously in order to shape the interventions to the specific context. Project developers believed that this was the best way to achieve the intended impact. However, respondents indicated that the slowness of the adaptation process has been underestimated. It took a lot of time to detect hindering contextual factors, to find ways how to address them and to adapt the activities accordingly. The global intervention period was too short to complete this process for the many contextual barriers that showed up.

Another often mentioned barrier was resistance, personal attitudes and taboos of community members including parents, teachers and health care providers. The initial strong community resistance was underestimated, partly due to the delay in the situation analysis. Reports were received of health care providers refusing to talk about SRH with adolescents, and informants, including adolescents, underlined that the problem of taboo and cultural sensitivity around the topic of sexuality hindered the implementation of CERCA activities.

‘But, when we were advancing, we saw that the problem is really big; it involves a lot of cultural issues and strong taboos rooted in the population, that it is not easy to change.’ (Project Leader)

3.3.1. Preferred activities

Activities with personal interaction were seen by respondents as a key component of CERCA, and received very positive feedback. According to the respondents, workshops enabled people to actively exchange information, get immediate feedback, and stimulate openness in talking about sexuality. Workshops were the most popular activity in all countries, followed by mobile cinemas in Nicaragua and mobile health teams in Bolivia. The workshops for adolescents included not only information on SRH but also important topics such as bullying and gender norms, upon the request of the adolescents. Also teachers and parents highly appreciated the organization of workshops during which they learned to provide better information on SRH to adolescents and guide them in the related issues.

‘We each wanted to know more about the topic [SRH], we had more confidence with the workshops…’ (Adolescent)

The digital activities (such as Facebook, Twitter) were said to be complementary and helpful in specific situations. In addition, Facebook was found to be a cheap and easy way to approach adolescents. In Ecuador the CERCA Facebook page is still being used. Telephone help lines that adolescents could call with questions or concerns were also named as a very useful tool. Twitter was found practical to address health authorities and political leaders.

‘The moment when the young person has a question or doubt or emergency and cannot personally get in touch with project staff, he or she can ask through this media [Facebook]’ (Country Implementer)

‘They [policy makers] are not reading articles… we gave them all CERCA articles but they don’t read. It could be attributed to the difficulties to understand the terms. That’s why for this group of authorities the information which is delivered to them quicker is the condensed information through Twitter’ (Project Leader)

3.3.2. Participation

Overall, participation in the intervention was high and the CERCA Project succeeded in attracting a broad range of adolescents as well as other target groups. Nevertheless, the Project encountered difficulties with involving certain groups. Parents were seen as the most difficult group to include in any school-based activity as well as community based activities. Some parents mentioned that many of their peers are embarrassed to talk about sexuality with their children and that this possibly influenced their participation.

‘Parents are a group that has more difficulties, sometimes they are not very willing to work on these issues [SRH]…’ (Country Implementer)

The respondents also mentioned the under representation of fathers, and the fact that parents do not always want to be “taught” and to participate in discussions about SRH with adolescents and their children.

‘One of the weaknesses is a low participation of fathers, men… They started to participate but only, for example, two fathers were in the group. And it meant that men are reluctant to go. I had an opportunity to talk to the fathers and one man told me: “I am not going because they are all young people there… The topic is interesting and I would go when there are other men”. We definitely had some problems with fathers’. (Country Implementer)

Other respondents mentioned that they were not always aware of when and where activities were programmed. On the other hand, community activities such as health fairs or sport events allowed the groups rarely involved in educational activities to be part of the Project as well.

3.4. Monitoring

The monitoring process during the CERCA project could be divided into three levels: external, internal and community monitoring. For the external monitoring, monitoring sheets were used and sent to the consortium management by local project leaders every three months. They were based mostly on administrative and financial requirement but also partly served to identify target groups or topics that were not sufficiently addressed. At the country level (internal), for example in Nicaragua and Ecuador, monitoring was performed on a weekly basis by country implementers through meetings, reports and supervision of health promoters and FoYs which boosted the work that promoters were doing, stimulated them and helped to resolve doubts. Ongoing feedback was obtained from the communities and target groups through consultations and discussions performed by local researchers and volunteers alongside implementation which allowed adapting the activities and topics (community monitoring).

While the number, content and target population of most activities was monitored, there was no systematic monitoring of the intervention quality, and several respondents indicated that the Project would have benefited from closer monitoring.

‘We didn’t look sufficiently critically to the interventions and to the effect of interventions. There was a lack of continuous critical thinking on what was happening.’ (Consortium Management)

Furthermore, some activities were either partly documented or not documented at all (e.g. questions to the help line, use of mobile consultations). There was little exchange of experiences between the three countries, nor were there reports made of why and how activities were developed and changed during the course of the Project which means the fidelity check was not done.

3.5. Evaluation

As we mentioned in the introduction part, the original project evaluation demonstrated limited impact. Below we describe the main components of the original evaluation and after provide reflections of the main stakeholders on the challenges faced during the original evaluation (Table 2).

Table 2.

Original impact evaluation overview.

Target group and sample size	651, 1330 and 662 adolescents in respectively Bolivia, Ecuador and Nicaragua
Design and Method	Intervention and control group, two cross-sectional surveys (pre/post) that were later on matched
Data collection tools	Questionnaires with closed and open ended questions, largely based on previously validated questions and scales, and pilot tested. The questionnaire contained 59 questions on socio-demographic characteristics, relationships, communication skills, information-seeking behaviour, use of existing SRH services, reproductive history and sexual behaviour
Statistical methods	Descriptive, logistic multivariate regression, generalized estimating equations
Indicators	In order to measure the effect of the interventions on adolescents SRH two outcome indicators (communication on sex, use and knowledge of health services) and one impact indicator (condom use) were chosen
Main findings	In Ecuador, it found slight improvement in condom use and improved knowledge and use of health care services in the intervention group compared to the control group, while positive changes in communication on SRH topics were reported in Bolivia. No significant changes were found in Nicaragua
Limitations	High lost to follow up – 70% between waves, differences in baseline characteristics between cohorts, inconstant participation, the divergence in the applied interventions (consequence of the use of action research as methodology)

During the interviews and focus groups many concerns were raised on the original impact evaluation. First, critical comments were made concerning the content of the questionnaire. Because the three sites emphasized slightly different objectives and implemented different activities, the generic questionnaire may have been too broad. The questionnaire used for the evaluation was considered to be ‘rigid, closed’ in terms of measured outcomes and the questions’ formulations which did not allow for capturing more details.

‘Adolescents wanted to do it quickly to finish’ and ‘this tool minimized or did not take into account other processes’. (Country Implementer)

‘I was actually very surprised that this very rigid questionnaire was able to capture some small but still statistically significant changes’. (Consortium Management)

Furthermore, the questionnaire was only directed to adolescents, while the activities had different target groups, failing to measure changes in the communities. Additional evaluation could have been of additional value, for example surveys for all target groups, and a qualitative evaluation (FGDs and interviews with all main stakeholders).

‘It [the effect evaluation] should have been an assessment with all those who were constantly involved in the project, its beginning and its end, to see if the gains were good.’ (Project Leader)

As it was mentioned previously, the original evaluation was accompanied by high attrition rates. The informants during interviews suggested that high migration rates and graduation from school could lead to such results. Additionally, as it was noted by project leaders and implementers, the questionnaires were badly coded – and it was not possible to match the initials that participants used in the pre- and post-evaluations.

3.6. Main achievements and results of the CERCA Project

As the quantitative evaluation of the CERCA intervention demonstrated few results, we included a question on perceived results in the qualitative study. According to the majority of respondents, the most significant positive changes in all settings were in the following two objectives pursued by the CERCA intervention: improvements in the communication between adolescents and parents, and access to quality SRH information.

Furthermore, the stakeholders identified additional results of the CERCA Project that were not captured by the impact evaluation among adolescents. In their perception in all settings, the CERCA intervention sensitized the communities and authorities, and initiated a dialogue between different stakeholders including young people, parents, teachers and some health care workers. This helped to partly overcome the taboo existing in the intervention communities regarding adolescents and sexuality, and in doing so, created an enabling environment for additional interventions in this field.

‘I think CERCA opened the window, or prepared the soil for very good interventions. I think after CERCA many other projects can go and use this openness or preparedness of communities for interventions in this field.’(Consortium Management)

‘A lot of people were mobilized thanks to the intervention. Not only people directly involved in the intervention. People came on the street or, started working, thinking, talking, about sexuality.’ (Consortium Management)

‘Before, we didn’t talk about it [SRH] at home. Well we did talk about it but in a simple, light way. But to have a neutral person talking and informing them [adolescents], they obtained a better vision and more freedom to talk about this at home’. (Parent)

‘In the community CERCA initiated a dialogue between parents and children on sexual and reproductive health issues. Many parents said that previously it was difficult to address these issues because of shame and taboos about sexuality, but with various CERCA activities they became more sensitized.’ (Country Implementer)

4. Discussion and lessons learned

This post-hoc process evaluation set out to assess strengths and weakness in the CERCA Project’s design, implementation, monitoring and evaluation and identify additional results. It adds evidence to the limited number of process evaluation reports of large-scale sexual and reproductive health promotion interventions for adolescents.

While unprotected sexual intercourse is the basic reason for unintended pregnancies, this individual behaviour is strongly influenced by interpersonal, institutional, social and structural level factors. These determinants are context-specific and are not always linked in a direct causal pathway to the particular health behaviour, making adolescent pregnancy, increasingly referred to as, a complex issue (Nelson, Edmonds, Ballesteros, Encalada Soto, & Rodriguez, 2014). The CERCA Project took this complexity into account in its design and implementation; e.g. crucial stakeholders in the lives of young people were identified and included in the intervention, activities sought to address more than protected sexual intercourse and tackled aspects of equity in relationships and self-esteem. This was also highlighted as an essential element of successful implementation in a number of studies that have undergone process evaluation (Waqa, Moodie, Schultz, & Swinburn, 2013). While these were clear strengths, the CERCA Project underestimated the importance of cultural taboos and resistance to change, similar to Wight et al. (2012). When setting its objectives, the Project did not sufficiently take into account the stage of change of the target group. Behavior change is seen as a dynamic process that occurs in a cyclical order, involving the following stages: pre-contemplation, contemplation, preparation, action and maintenance (Prochaska & DiClemente, 1983). Despite strong taboos against adolescents’ sexual behaviors the Project aimed at substantially changing individual behaviors. However, at the beginning of the Project the different target populations could be considered to be situated mainly in the pre-contemplation phase where people do not intend to take action in the foreseeable future, or are even opposed to action. The CERCA project’s objectives were formulated at the action stage, in which people have already made specific overt modifications in their lifestyles. This leap of three stages (skipping contemplation and preparation) against the short implementation period (20 months) contributed to the hindering factors that prevented the Project to achieve substantial changes in young people’s sexual behaviors. A well-designed situational analysis and a model for change validated by the stakeholders could have forestalled some of these pitfalls (Aventin, Lohan, O’Halloran, & Henderson, 2015; Power et al., 2004). A large project like CERCA requires a preparatory phase and a thorough analysis of the situation at the country level as well as community-specific context information. This pilot or preparatory phase would have been useful to align all project partners, set realistic goals, determine the best ways to approach the target groups, optimize the intervention and make the major changes in the intervention activities before the start of the effectiveness study.

The complexity of the problem addressed should not only be reflected in the design and implementation of the intervention, but also in its monitoring and evaluation design. Complex interventions present various problems for evaluators, including restricted opportunities for randomization, inadequate effect attribution to specific interventions, and limited control over implementation (Leerlooijer et al., 2013). The CERCA impact evaluation was clearly confronted with these issues – by only measuring the individual effect based on pre-defined behavioral outcomes among adolescents, it missed out on other possible outcomes, including precursors of behaviour, on the perception of other stakeholders, on community changes, and on implementation challenges. And by only using quantitative methods, it did not draw out contextual information as it could have done. The evaluation design of the intervention was too narrow, leaving aside the problems in retention of respondents, adapting the surveys to the local contexts, and targeting only one group – adolescents. In short, there seems to have been a mismatch between the project’s community-led and action-based philosophy and the sole reliance on a quasi-randomized design to evaluate the Project.

Taking into account the community-based action research approach and a variety of target groups, CERCA would have benefited from a strong monitoring approach to detect problems early on. An essential part of community-based action research is to adapt the monitoring strategy if the interventions are changed and thoroughly document all activities. Monitoring’s primary purpose is to achieve the best possible project performance by providing feedback to project management and teams at all levels. This enables project teams to improve their operational plans and to take corrective action in the case of shortfalls and constraints (Horstman, Cleland, Douthwaite, Ambegaokar, & Salway, 2002). The monitoring process during the CERCA Project was constant and systematic; however, weaknesses in monitoring the quality of interventions and in careful documentation of the implemented activities and their content were identified To be useful, the monitoring process should have included a specific approach for each of the implemented activities (e.g. workshops, mobile consultancies, and helpline). Furthermore, additional data on the acceptability of the intervention, as well as factors helping and hindering implementation should have been collected using appropriate qualitative methods.

This post hoc evaluation indicated that some stakeholders felt CERCA prepared the soil for future interventions in ASRH. While Campbell et al. (2014) found that “externally-imposed programs that present new information without adequately engaging with local realities and constraints on action can be met by resistance to change” and Wight et al. (2012) identified lack of interventions at the cultural level as the main factors hindering individual behaviour change, CERCA countered this with high community-involvement (Campbell et al., 2014, Wight et al., 2012). Some of the stakeholders interviewed observed that following CERCA, people in the communities reported higher awareness of problems related to adolescent sexuality and willingness to communicate about it, policy makers are taking actions and different organizations are joining forces. They believed that CERCA generated a social environment that is supportive of the possibility of health-enhancing behaviour change, possibly resulting in individual behaviour changes in the long term.

4.1. Limitations

Several limitations should be noted. First, recall bias may affect the content of information reported by participants, as the CERCA Project ended about 6 months prior to this study. Given time and budget constraints, we could not involve all key actors in each setting (e.g. health care providers in Managua, Nicaragua). However many of the results were found to reinforce each other, suggesting we reached saturation. Finally, although this study was led by researchers not involved in CERCA, the CERCA partners participated in evaluation as respondents but also as researchers who collected and analyzed data. While a limitation, it also may be a benefit as the researchers could look at CERCA without the bias of having implemented it.

5. Conclusion

Based on this post-hoc evaluation, we identified the main strengths and weaknesses in the CERCA Project’s design, implementation, monitoring and evaluation as well as contextualized and better interpreted the quantitative results. Moreover, this post-hoc evaluation added evidence to the limited number of studies reporting on process evaluations of a large-scale sexual and reproductive health promotion intervention for adolescents and provided some key recommendations and lessons learned for the future projects.

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

OI and KM developed the first draft of the paper. All other authors revised it critically, provided their input and gave final approval to the submitted version.