Process evaluation of HIV prevention peer groups in Malawi: a look inside the black box

Abstract

This paper reports the process evaluation of a peer group intervention for human immunodeficiency virus (HIV) prevention which had positive outcomes for three target groups in Malawi: rural adults, adolescents and urban hospital workers. The six-session intervention was delivered to small groups of 10–12 participants by 85 trained volunteer peer leaders working in pairs. A descriptive, observational mixed methods design was used with a convenience sample of 294 intervention sessions. Using project records and a conceptually based observation guide, we examined five aspects of the implementation process. The context was favorable, but privacy to discuss sensitive issues was a concern for some groups. In study communities, program reach was 58% of rural adults, 70% of adolescents and nearly all hospital workers. Session records confirmed that all peer groups received the intended six sessions (dose delivered). The dose received was high, as evidenced by high participant engagement in peer group activities. Peer leaders were rated above the median for three indicators of peer group content and process fidelity: session management skills, interpersonal facilitation skills and whether more like a peer group than classroom. Documenting that this HIV prevention peer group intervention was delivered as intended by trained peer volunteers supports widespread dissemination of the intervention.

Introduction

Process evaluation is critical for understanding how an intervention works. If the intervention is effective, process evaluation provides important information about implementation to guide dissemination in other settings [1–6]. If the intervention is not effective, process evaluation can identify whether this is due to failure to implement the intervention as intended [3–8]. Despite a recent increase in inclusion of process evaluations [9], most intervention research still leaves the mechanisms that contribute to positive outcomes largely unexplored [10]. To scale up successful interventions, it is essential to understand what occurs inside the ‘black box’ during implementation.

This paper reports the process evaluation of a peer group intervention for human immunodeficiency virus (HIV) prevention which had positive outcomes for three different target groups in Malawi's Central Region [11–15]. Malawi, with a population of 13.9 million, is one of the poorest countries in the world, with 80% of Malawians living in rural communities characterized by subsistence agriculture, low education and high poverty [16]. The adult HIV prevalence is estimated at 11.8% [17]. Although HIV prevalence in rural areas is about two-thirds that of urban areas [18], the overall HIV burden is higher in rural areas because more people live there [17]. Also there is a lack of HIV prevention interventions in rural areas. Therefore, this intervention study was initiated in rural communities.

Heterosexual contact is the chief mode of transmission, responsible for ∼90% of cases [18]. Therefore, HIV prevention through behavior change remains of prime importance in the fight against HIV/acquired immunodeficiency syndrome (AIDS) in Malawi. The peer group intervention consisted of six sessions with defined content and skill-building activities to support sexual behavioral change for HIV prevention. Sessions were attended by a small group from the same community and facilitated by trained peer volunteers. This type of intervention differs from a social network-based peer-led intervention that usually does not have formal small group sessions for participants.

This process evaluation adapted the conceptual framework described by Linnan and Steckler [9]. Following an extensive review of process evaluation research, they developed a framework for systematically conducting process evaluations to capture key process components: context, reach, dose delivered, dose received and fidelity of intervention delivery [9]. ‘Context’ assesses aspects of the intervention's environment or setting that might affect its delivery or outcomes. ‘Reach’ is the proportion of the intended audience who actually took part in the intervention. ‘Dose delivered’ is the proportion of the intended intervention that was actually provided to the target audience. ‘Dose received’ is the extent to which the intended audience engaged with and used the intervention information and activities. ‘Fidelity’ is the quality of the intervention implementation, the extent to which it was delivered in the intended manner. Linnan and Steckler additionally discussed assessing recruitment and combining reach, dose delivered, dose received and fidelity to create a composite implementation score. We could not analyze recruitment as a key component of this process evaluation because we did not collect data on reasons for nonparticipation. We also did not create a composite implementation score because of the conceptual difficulty of determining the relative weight of each component when combined.

Peer groups have many features that make conducting process evaluation both difficult and important. Peer group interventions are complex because there are usually multiple groups and facilitators, and the social cognitive learning model requires attention to both process and content [3–5, 19]. Multiple groups and multiple facilitators increase the potential for variation in implementation and thus lack of fidelity. Additionally, it is challenging to ensure that volunteer peer leaders are well-prepared and that they implement the intervention as intended. To promote and sustain individual behavior change, peer group interventions emphasize both content and a group process that fosters participatory learning [20]. Therefore, the process evaluation of a peer group intervention must assess fidelity of both content and group process simultaneously. Evaluation of group process is particularly important because peer leaders and group members find it challenging to maintain the participatory group process without reverting to the more familiar ‘classroom’ model [21].

There are several published process evaluations of peer-led HIV prevention interventions [7, 10, 22–25] and teacher- or counselor-led HIV prevention programs in schools and community settings [3, 8, 26–28]. Peer-led interventions share the social-cognitive learning framework of peer group interventions, while school- and counselor-led interventions share the small group format of peer group interventions. These process evaluations have documented that these types of leaders can reach the target audience effectively. However, an extensive literature search has not identified any published process evaluation of an HIV-prevention peer group intervention.

We undertook this process evaluation to document whether trained volunteers can implement a peer group intervention with faithfulness to the peer group model. We examined context, reach, dose delivered, dose received and fidelity to the intervention content and participatory process.

Methods

Process evaluation design

A descriptive, observational mixed methods design was used. Qualitative findings corroborated and amplified quantitative results. A convenience sample of selected sessions was observed. The same procedures were followed for conducting the process evaluation in three samples: rural adults, rural adolescents and urban hospital workers.

The intervention being evaluated

The Mzake ndi Mzake (Friend-to-Friend) intervention, the subject of this process evaluation, integrated social-cognitive learning theory of Bandura [20], the World Health Organization's primary health care model [29] and contextual tailoring based on formative evaluation [30–33]. Session topics included the impact of the HIV and AIDS epidemic, stigma, human sexuality, sexually transmitted infections, HIV and AIDS transmission and prevention, partner communication for safer sex, condom use and community activities for HIV prevention. The intervention used trained volunteer peer leaders working in pairs to deliver the six-session intervention to small groups of 10–12 participants. The intervention was offered and evaluated using the same process evaluation procedures in two different settings, rural communities and an urban hospital.

In the rural communities, we began by providing the peer group intervention for rural health workers. During these rural health workers' sessions, the research team, comprising members from University of Malawi Kamuzu College of Nursing and University of Illinois at Chicago College of Nursing, developed the process evaluation and trained the observers. Because the procedures for process evaluation were being refined during the rural health workers' sessions, no process evaluation data are available from these groups.

Over 50 of these rural health workers volunteered to receive peer leader training to bring the peer group intervention to adults and adolescents in the communities served by their health facilities. Building on their participation in the intervention as a group member, all peer leaders were given 2 additional weeks of training in content presentation, group facilitation and the use of role plays and discussion to promote behavior change. Training used demonstrations and return demonstrations with corrective feedback.

In rural communities, after the first sessions were offered by the rural health worker volunteers, 35 community adults volunteered for training and co-facilitated peer groups in their own communities. In the rural communities, usually one peer leader was a health worker and the other was a community adult. We implemented the intervention with 2242 adults.

In the final stage, the project team, health workers and community members adapted the peer group intervention for adolescents and offered that program to 1500 adolescents. Based on our participatory research with the community, adolescents received the intervention in age-specific (10–12, 13–15 and 16–19 years) and gender-specific groups. At the community's request, content for adolescents was the same as for adults and health workers, except that the 10- to 12-year groups did not receive explicit sexual- or condom-related content. All adolescent groups were co-facilitated by trained volunteer health workers and community adults.

In the interval between the rural adults and adolescents receiving the intervention, we acquired supplemental funding to replicate the Mzake ndi Mzake intervention with 855 hospital workers in a large, urban referral hospital in Malawi's central region [11, 13]. The urban hospital was a large referral center serving an average of 700 inpatients and 560 outpatients daily, many of whom are living with HIV. In this setting, the peer leaders were two hospital worker volunteers or one hospital worker and a nurse researcher from the Kamuzu College of Nursing.

In the rural communities, potential participants were informed about the intervention by community leaders; interested persons attended a meeting where a research team member explained the project. After obtaining informed consent, research team members arranged them into single-gender groups of 10–12 people who lived within walking distance of each other. Peer leaders were not involved in recruiting group participants. At the urban hospital, potential participants were informed about the intervention by administrators and invited to a meeting where a research team member explained the project. Groups for those who consented to participate were organized with the help of unit administrators.

In almost all cases, a group was facilitated by the same pair of peer leaders and observed by the same observer(s) for all six sessions. The sessions varied in length for reasons such as the amount of content covered, amount of discussion the session topic generated and the level of interest and attention of group members. Adolescent groups were markedly shorter than either adults or hospital workers, likely due to shorter attention spans of the youth and their constraint when discussing sensitive topics such as sexuality and gender relations with their adult peer leaders in the presence of adult observers. For adults, adolescents and hospital workers, Session 4 on communicating with partners about HIV/AIDS prevention was significantly shorter than any of the other five sessions. This is likely to be due to the content, which involved role plays and communication on a culturally challenging topic.

Key outcomes for rural adults and adolescents included greater knowledge of HIV transmission, more favorable attitudes toward condoms, less stigmatization of those with HIV, greater self-efficacy for practicing safer sex, increased safer sex behaviors, more HIV testing and increased community HIV prevention activities in the intervention communities compared with control communities [34]. The same intervention was tested in an urban hospital with similar outcomes except that the urban health workers did not reduce their sexual risk behaviors [35].

Process evaluation setting and sample

In the rural communities, adult and adolescent groups occurred at sites with potential for privacy, such as schools and churches. The urban hospital worker groups occurred in small hospital meeting rooms which provided privacy for the group.

Table I summarizes the number of observations of each session, the number of distinct groups observed and brief demographic descriptions of group participants for all three samples. Data were analyzed from 115 adult sessions, 77 adolescent sessions and 102 hospital worker sessions. Most of the adult groups were single gender. Adolescents attended single-gender, age-specific groups. Hospital worker groups were mixed-gender and included all categories of workers, informally divided by level of education. We observed a total of 85 volunteer peer leaders working in pairs; 52 were rural health workers, 15 were rural community members, 10 were urban hospital workers and 8 were nurse researchers from Kamuzu College of Nursing who co-facilitated some urban hospital groups. Each pair of peer leaders was observed during multiple sessions.

Table I.

Observations and sample descriptions for groups observed in three study populations

Observations and participants	Rural adults	Rural adolescents	Urban hospital workers
Session topics and number of observations of the session	N = 115	N = 77	N = 102
Importance of HIV/AIDS prevention in your community	24	14	19
Understanding human sexuality	22	14	18
Understanding and preventing HIV/AIDS	15	15	17
Communicating with partner about HIV/AIDS prevention	15	13	17
Using condoms correctly	21	11	15
Community collaboration for HIV/AIDS prevention	18	10	16
Number of distinct groups observed	31	15	21
Number of groups in which all six sessions were observed	8	9	12
Length of session in minutes
Range	43–335	30–155	39–245
Mean (SD)	113.11 (47.01)	86.01 (27.20)	135.73 (50.07)
Number of participants per group
Range	6–19	4–16	7–11
Mean (SD)	11.37 (2.50)	9.30 (2.28)	9.47 (1.03)
Gender composition of peer groups	N = 115	N = 77	N = 102
Male only	28	63	0
Female only	58	14	0
Mixed gender group	29	0	102
Age of group participants (years)	N = 358	N = 148	N = 202
11 and under	0	26	0
12–14	0	49	0
15–17	2	51	0
18–19	3	22	0
20–29	111	0	26
30–39	96	0	84
≥40	146	0	92
Marital/partnered status of participants	N = 358	N = 148	N = 202
Married/partner away most of time	31	0	5
Married/living with partner	256	0	158
Regular partner/not living together	6	55	7
Single/no current partner	65	93	32

SD = standard deviation.

Protection of human subjects

The Malawi College of Medicine Research and Ethics Committee and the University of Illinois at Chicago Institutional Review Board reviewed and approved the research, including the process evaluation component. The informed consent was conducted in the native language. Signed informed consent included information that some group sessions would be observed.

Measures

Key process components (context, reach, dose delivered, dose received and fidelity of intervention delivery) were measured using both a standardized observation guide and project records. Guided by the study's theoretical framework, members of the research team collaboratively developed the Process Evaluation Guide to tap culturally grounded behavioral indicators of context, dose received (engagement) and fidelity (session management, interpersonal facilitation and session more like a peer group than classroom). We used direct observation of selected sessions by trained observers, as advised by Resnicow and by Israel et al. [2, 36].

The Process Evaluation Guide contained both qualitative comments and quantitative ratings, as recommended by Oakley et al. [3]. Qualitative data included systematically recorded comments from group members (what they liked best and least and topics on which they wanted more information or clarification) and peer leaders (how they felt about the session and their performance, what went well and what could be improved). Observers also noted group members' interactions, reactions to the content and any unusual occurrences or contextual issues.

For quantitative ratings, conceptual definitions of items and detailed procedural instructions were developed to help standardize observations and assist with observer training. The Process Evaluation Guide was pilot tested and refined during the observers' training, and ambiguous or redundant items were deleted. To reduce observer burden, the session-specific subtopic checklist was replaced by a single-item rating completeness and accuracy of content.

Project reach and dose delivered were evaluated using project records, rather than the Process Evaluation Guide. For the adults and adolescents, reach was evaluated by a question from the larger study's final survey asking if the respondent had attended the peer groups and, if so, how many sessions. The final survey used a random sample of the adults and adolescents in the participating communities, providing a good estimate of community member participation. For adults and adolescents, reach was calculated as the percentage of those surveyed who attended five or more intervention sessions. For health workers, reach was estimated as the proportion of the total workforce who participated. Dose delivered was obtained from project records of the peer group sessions.

The Process Evaluation Guide included ratings of four context variables, environmental factors that might affect intervention delivery (crowding, adequacy of lighting, uncomfortably hot or cold conditions and adequate privacy to discuss sensitive topics). The Guide measured dose received by rating participant engagement, calculated as the mean score of three culturally appropriate indicators of engagement: answers readily, comments in response to other members' comments and focuses attention on the presentation. We measured fidelity using three variables: peer leader session management behaviors (five items), peer leader interpersonal facilitation behaviors (nine items) and overall rating of the session as more like a peer group than a classroom (one item).

Table II lists the items and the internal consistency reliability coefficients for each construct assessing dose received and fidelity for each of the three samples. Alphas were >0.70 for most constructs. However, in the adolescent groups, engagement and interpersonal facilitation alphas were 0.62. For peer leader development purposes, interpersonal facilitation was rated for each peer leader separately, but the mean facilitation scores for each pair were used in data analysis. Except for engagement, all constructs were assessed for the session as a whole. Engagement indicators were rated in four 30-min periods per average 2-h session to identify changes during the course of each session. The summary measure of engagement used was the mean of the scores from the 30-min periods.

Table II.

Items and internal consistency reliability of measures for three study populations

Measures	Rural adults	Rural adolescents	Urban hospital workers
	Cronbach's α	Cronbach's α	Cronbach's α
Group members' engagement (three items, scored 0–1)	0.753	0.618	0.712
Group members answer peer leaders' (PLs') questions readily
Members comment in response to other group members' comments
Members focused on presentation
PLs' session management (five items, scored 1–5)	0.837	0.802	0.742
PLs work together as a team
PLs well prepared
PLs cover material accurately and completely
PLs present material rather than read aloud
PLs maintain good pace, not rushed or dragging
PLs' interpersonal facilitation (nine items, scored 1–3)	0.826	0.621	0.723
PLs encourage silent members to participate
PLs provide positive feedback for group members' comments
PLs track discussion, avoid unnecessary repetition of points already made by members
PLs refer to members' comments or ask for further discussion
PLs answer questions correctly or corrects misinformation politely
PLs enthusiastic
PLs confident
PLs respectful
PLs comfortable discussing sensitive topics like sexuality
Session rating of ‘class versus peer group’ (one item, scored 1–5, 1 = more like a classroom, 5 = more like a peer group)	N/Aa	N/Aa	N/Aa

^aSingle item (N/A = not applicable).

Procedures

To decrease bias, the process evaluation team comprised nurse researchers who were ‘external’ to the intervention delivery and outcome evaluation [6, 35]. They were trained in the observation procedures during the peer groups offered to the rural health workers at the beginning of the intervention; interrater agreement was ≥0.80. They attended selected sessions for the purpose of the process evaluation observation only and did not have an ongoing relationship with the peer leaders or group members.

Purposive sampling of groups and sessions to observe was used to ensure representation across different leaders and sessions. For some groups in each target population, all six sessions were observed to track group member engagement and peer leader session management and interpersonal facilitation behaviors across the six sessions. Process observations were also used to support peer leader skill development, so the first two sessions were observed most often to allow peer leader performance difficulties to be addressed early [5].

When conducting process observations, observers introduced themselves to the group and explained their purpose. They sat slightly apart so they could observe without distracting the group and did not actively participate. Using the Process Evaluation Guide, they rated each quantitative item and noted qualitative comments.

Data analysis

Process Evaluation Guide data were entered and verified line by line. Each sample was analyzed separately. Descriptive statistics were calculated for all quantitative measures. One-way analysis of variance (ANOVA) was used to identify significant differences across the six sessions and to examine changes in engagement within a session (across the 30-min periods) and the effects of each of the contextual variables on engagement, session management, interpersonal facilitation and class versus peer group. Linear regression was used to test the effects of engagement, session management and interpersonal facilitation on class versus peer group.

Content analysis was performed with qualitative data. Data were coded to identify observers', participants' and peer leaders' comments related to context, reach, dose delivered, dose received, fidelity and content factors including topic sensitivity, relevance and clarity.

Results

Context

The frequencies of the four contextual variables (seating, lighting, temperature/ventilation, privacy) are presented in Table III. The urban hospital worker sessions at the hospital received the optimal ratings for all context variables. The adult and adolescent groups took place in community settings, e.g., a church or school. The most common problem was inadequate lighting, but crowding, uncomfortable temperatures and lack of privacy occurred at a few community sessions.

Table III.

Frequencies of contextual variables in three study populations

Contextual variables	Rural adults' observations (N = 115)	Rural adolescents' observations (N = 77)	Urban hospital workers' observations (N = 102)
	%	%	%
Space/seating for group
Comfortable for size of group	83	95	100a
Crowded	10	5	0
Not comfortable	5	0	0
Missing data	2	0	0
Lighting
Good	78b	79	100a
Poor	22	21	0
Temperature/ventilation
Comfortable	87	91c	100a
Hot and stuffy	9	8	0
Cold and drafty	3	1	0
Dusty	1	0	0
Privacy
Yes	90d	91d	100a
No	9	9	0
Missing data	1	0	0

^aSessions were held at a hospital facility with uniformly adequate seating, lighting, temperature/ventilation and privacy for meetings.

^bResults of one-way ANOVAs. For adults only, lighting was related to session management: F(1,113) = 4.42, P < 0.05.

^cFor adolescents only, temperature/ventilation was related to engagement: F(2,74) = 11.69, P < 0.001, and peer group versus classroom: F(2, 74) = 4.44, P < 0.05.

^dFor adults, privacy was related to session management: F(1,112) = 5.52, P < 0.05, and peer group versus classroom F(1,108) = 11.61, P < 0.01. For adolescents, privacy was related only to peer group versus classroom: F(1,75) = 5.60, P < 0.05.

Lack of privacy was also noted in the qualitative comments (for representative examples, see Table IV). One privacy issue was the presence of young children, which was a distraction and hampered discussion of sensitive issues.

Table IV.

Examples of qualitative content analysis coding of key process components^a

Code	Positive quotations	Negative quotations
Context
Privacy	Observers limited their comments about context to problems that were noted.	There was no privacy—the house used … was close to other houses and people were walking by. A-52 Children were hiding to listen in [to the condom lesson]—no privacy. A-7
Dose received
Engagement	The group was very alert and very eager to understand HIV and AIDS; they asked a lot of questions. HW-114	The group was not very active; they played a listening role most of the time. Adol-66
Fidelity
Session management	Well-prepared Peer Leaders, confident; Peer Leader 2 was particularly well-prepared and used good examples and language common to the youth. Adol-19	Session started late and was rushed. A lot of sections were not tackled effectively and others were missed.A-108
Interpersonal facilitation	Peer Leaders were very confident and comfortable discussing the sexual organs. HW-115	Peer Leader 2 not active; failed to get the group going. A-136
More like a peer group than classroom	Participants were very open to share views. They came up with examples of traditional practices that spread HIV, e.g., widow inheritance and fisi practice. A-123	Started on a good note but later became a classroom when the writing of notes on the board started. Participants became absorbed in note taking. A-43
Content factors
Topic sensitivity	Members were free to discuss and mention names of private body parts and appreciated the importance of doing so. HW-95	Participants were very uncomfortable to mention male and female reproductive organs, covering their faces, looking down, and remaining quiet. HW-82
Relevance	One participant said this information on HIV prevention is long overdue. A-37	The group demonstrated knowledge on the subject matter; some expressed that the points were a repetition of what they already know. HW-81
Clarity	The youth showed understanding of the concepts with the role play, showed that they had grasped something. Adol-19	Group members demonstrated some knowledge on HIV prevention and transmission, but they looked confused during discussion of the pathophysiology of HIV infection and AIDS. Adol-62

^aA = adult group, Adol = adolescent group and HW = hospital worker group.

Reach and dose delivered

The intervention was designed to be offered repeatedly in a community until all interested, eligible community members had participated. In all, 2242 rural adults, 1500 adolescents and 855 urban hospital workers completed the intervention. Reach was assessed as the percentage of adults and adolescents surveyed who reported attending the peer groups. At the final evaluation, 58% of the adults surveyed reported participating in the peer groups, and 89% of these reported completing at least five of the six sessions. A higher proportion of adult females (67%) reported attending the intervention than males (49%), consistent with the greater involvement of men in economic activities outside the village. Among adolescents, 70% reported participating, of whom 77% had attended at least five sessions. There was no difference in participation for boys and girls (70% for each), but participation was higher among younger adolescents. Of adolescents ages 10–12, 78% attended; of ages 13–15, 76% attended and of ages 16–19, 61% attended. A higher proportion of older adolescents attended school or worked outside the community.

At the urban hospital, essentially all of the workers attended the intervention. The hospital staff is estimated at 860 workers, and 855 hospital workers participated in the intervention. For hospital workers, attending the on-site sessions was quite convenient, as workers attended before or after their shift. Thus, it is not surprising that the reach of the intervention was greatest for the health workers. Records of sessions held confirmed that in all three samples, all six sessions in their entirety were offered to each peer group.

Dose received

Table V displays the engagement scores at each 30-min period for each sample. The level of group member engagement increased from the first to the second 30-min period in all sessions for all three samples, but one-way ANOVA found no statistically significant change for any sample. Adolescent engagement started at a higher level and remained high. However, the number of groups that continued into the third and fourth periods declined sharply. Adult and hospital worker groups were more likely to continue for the full 2 hours planned for the intervention sessions.

Table V.

Changes in engagement across 30-min periods within a session for three samples

30-Minute segments	Rural adults		Rural adolescents		Urban hospital workers
	n	M (SD)	n	M (SD)	n	M (SD)
Period 1	115	0.64 (0.352)	77	0.84 (0.239)	102	0.78 (0.292)
Period 2	115	0.71 (0.303)	77	0.86 (0.250)	102	0.86 (0.232)
Period 3	106	0.71 (0.312)	52	0.86 (0.250)	90	0.85 (0.308)
Period 4	90	0.66 (0.379)	18	0.83 (0.261)	80	0.81 (0.351)
	NS		NS		NS

One-way ANOVA, all NS (not significant) with P < 0.05. SD = standard deviation.

Table VI provides the mean scores for engagement, broken down by session, and the overall mean as well as the results of the one-way ANOVA examining the changes in engagement across the six sessions for the three samples. The mean engagement scores are high for all three groups: with a possible range from 0 to 1, the overall mean engagement was 0.68 for adults, 0.85 for adolescents and 0.83 for health workers. Although rural adults were engaged in the peer groups most of the time, they showed less engagement than either rural adolescents or urban hospital workers. There were statistically significant increases across the six sessions for group member engagement for adult and hospital worker groups, but not for adolescents. For both adults and hospital workers, Session 6 had significantly higher engagement than Session 1 (one tailed, P < 0.01).

Table VI.

Process evaluation ratings for three study samples across six sessions

Variable	Rural adults (N = 115)		Rural adolescents (N = 77)		Urban hospital workers (N = 102)
	n	Mean (SD)	n	Mean (SD)	n	Mean (SD)
Group member engagement (three items, scored 0–1)
Session 1	24	0.58 (0.244)	14	0.86 (0.181)	19	0.74 (0.172)
Session 2	22	0.64 (0.266)	14	0.85 (0.178)	18	0.82 (0.186)
Session 3	15	0.72 (0.206)	14	0.89 (0.153)	17	0.89 (0.163)
Session 4	15	0.72 (0.206)	13	0.82 (0.205)	17	0.85 (0.194)
Session 5	21	0.75 (0.230)	11	0.87 (0.211)	15	0.81 (0.193)
Session 6	18	0.79 (0.197)a	10	0.79 (0.306)	16	0.90 (0.153)a
Overall	115	0.68 (0.249)	77	0.85 (0.200)	102	0.83 (0.182)
	F = 2.307 (5), P < 0.05a		NS		F = 2.102 (5), P < 0.05a
Peer leader session management (five items, scored 1–5)
Session 1	24	3.31 (0.654)	14	3.54 (0.672)	19	3.92 (0.593)
Session 2	22	3.56 (0.649)	14	4.20 (0.359)a	18	4.12 (0.456)
Session 3	15	3.71 (0.696)	15	4.21 (0.533)a	16	4.22 (0.406)
Session 4	15	3.56 (0.577)	13	4.04 (0.579)	17	4.20 (0.557)
Session 5	21	3.81 (0.542)a	11	4.33 (0.559)a	15	4.21 (0.381)
Session 6	18	3.93 (0.570)a	10	4.34 (0.667)a	16	4.20 (0.468)
Overall	115	3.63 (0.639)	77	4.09 (0.612)	101	4.14 (0.488)
	F = 2.623 (5), P < 0.05a		F = 3.663 (5), P < 0.05a		NS
Interpersonal facilitation (nine items, scored 1–3)
Session 1	24	2.15 (0.405)	14	2.40 (0.319)	19	2.62 (0.218)
Session 2	22	2.28 (0.366)	14	2.74 (0.119)a	18	2.64 (0.345)
Session 3	15	2.42 (0.285)	15	2.70 (0.203)a	17	2.75 (0.172)
Session 4	15	2.40 (0.332)	13	2.73 (0.211)a	17	2.78 (0.206)
Session 5	21	2.46 (0.310)a	11	2.78 (0.138)a	15	2.78 (0.119)
Session 6	18	2.47 (0.366)a	10	2.81 (0.178)a	16	2.75 (0.141)
Overall	115	2.35 (0.365)	77	2.68 (0.244)a	10	22.72 (0.222)
	F = 2.798 (5), P < 0.05a		F = 6.574 (5), P < 0.05a		NS
Peer group versus classroom (one item, scored 1–5)
Session 1	23	3.35 (0.885)	14	3.14 (0.864)	19	3.37 (0.684)
Session 2	20	3.75 (0.851)	14	3.64 (1.082)	18	3.72 (0.575)
Session 3	15	3.53 (0.834)	15	3.93 (1.100)	16	4.00 (0.632)
Session 4	15	3.53 (0.990)	13	3.77 (.927)	17	3.76 (0.752)
Session 5	21	3.81 (0.873)	11	4.00 (1.095)	15	4.13 (0.915)a
Session 6	17	4.12 (0.781)	10	3.70 (.823)	16	4.31 (0.793)a
Overall	111	3.68 (0.885)	77	3.69 (1.003)	101	3.86 (0.775)
	NS		NS		F = 3.704 (5), P < 0.05a

SD = standard deviation and NS = not significant.

^aSession significantly different from Session 1; one-way ANOVA with Tukey's post hoc analysis of significant F, P < 0.05.

Qualitative comments validated the quantitative measures of engagement, in that groups where engagement was rated as low also had comments about low engagement. Many comments also confirmed a warm-up period. Some groups had more difficulty engaging with specific activities, especially role plays and discussion of sexual topics (see Table IV).

Fidelity

Table VI displays the mean scores by session, overall mean and results of the one-way ANOVA examining changes in session management, interpersonal facilitation and peer group versus classroom in all three samples. Peer leaders for the adult, adolescent and health worker peer groups were rated above the median for all three indicators. For the three indicators of fidelity, peer leaders of the adult groups, which were implemented first, had the lowest overall ratings, and leaders of the health workers had the highest. For all three groups, peer leader session management increased from Session 1 to Session 6; for adults and adolescents, but not hospital workers, this increase was statistically significant. Hospital workers' ratings were consistently high. For adults, sessions 5 and 6 were significantly higher than Session 1, and for adolescents, sessions 2, 3, 5 and 6 were significantly higher than Session 1 (one tailed, P < 0.01).

A similar pattern was seen in peer leader interpersonal facilitation ratings. For adult groups, facilitation scores were significantly higher in sessions 5 and 6 than in Session 1. For adolescent groups, each of the other sessions was significantly higher than Session 1 (one tailed, P < 0.01). Again, the increases for hospital workers were not significant; they maintained initial high ratings. All three groups were rated increasingly more like a peer group versus a classroom over the six sessions, but the increase was only statistically significant for the hospital worker groups, for whom sessions 5 and 6 were significantly more like a peer group than Session 1.

Because content fidelity is often reported separately in process evaluations of other interventions, we also examined the single-item rating whether session content was delivered accurately and completely. On a scale of 1–5, where 5 = ‘fully’ accurate and complete, 70% of adult groups, 87% of adolescent groups and 93% of hospital worker groups received a rating of 4 or 5.

Qualitative comments also validated the quantitative measures of fidelity; groups where session management, interpersonal skills or ‘more like a peer group’ were rated as low also had comments about fidelity issues. Comments highlighted the importance of thorough preparation in allowing peer leaders to focus on group facilitation and noted that when only one leader was well prepared the leaders could not work as a team. Comments also highlighted aspects of the content that worked well or needed improvement. Sensitive topics were more challenging for both peer group leaders and members, but most groups were able to engage with sensitive materials. Comments also identified material that needed simplification.

Relationship between context, dose received and fidelity

We evaluated the effects of the environmental context on dose received (engagement) and the three indicators of fidelity (session management, interpersonal facilitation and peer group versus classroom) using one-way ANOVA. Hospital worker sessions were not included in this analysis because all groups had optimal scores on all context variables. As noted in Table III, of the four context variables, only privacy was significantly related to at least one indicator for both adults and adolescents. For adults, privacy was related to session management: F(1,112) = 5.52, P < 0.05, and peer group versus classroom: F(1,108) = 11.61, P < 0.001; for adolescents, privacy was related only to peer group versus classroom: F(1,75) = 5.60, P < 0.05.

Relationship between dose received and fidelity

For all samples, a regression analysis of the effects of engagement, session management and interpersonal facilitation significantly predicted class versus peer group (see Table VII). The effects were strongest for the adults. Adjusted R² for the adults was 0.515, indicating that 51.5% of the variance in class versus peer group was explained by the set of variables. For adolescents and hospital workers, 9.8 and 13.6%, respectively, of the variance were explained by these variables. For all three samples, β indicates that, of these variables, only session management contributed significantly to the effect on class versus peer group.

Table VII.

Regression: more like a peer group than classroom for three samples

Variable	B	Standard error B	β
Rural adults (N = 115)a
Engagement	0.214	0.260	0.060
Session management	1.022	0.133	0.741***
Interpersonal facilitation	−0.128	0.240	−0.052
Rural adolescents (N = 77)b
Engagement	−0.614	0.565	−0.122
Session management	0.649	0.228	0.397**
Interpersonal facilitation	−0.698	0.576	−0.170
Urban hospital workers (N = 102)c
Engagement	0.748	0.403	0.174
Session management	0.448	0.179	0.282*
Interpersonal facilitation	0.391	0.395	0.112

^aAdjusted R² = 0.515; ANOVA: F = 39.876(3), P < 0.001.

^bAdjusted R² = 0.098; ANOVA: F = 3.763(3), P < 0.05.

^cAdjusted R² = 0.136; ANOVA: F = 6.250(3), P < 0.001. *P > 0.05, **P > 0.01, ***P > 0.001.

Discussion

This process evaluation documented that a peer group intervention for HIV prevention facilitated by trained peer volunteers was delivered as intended for three different target groups in two very different settings: rural adults and adolescents in the same rural communities and urban hospital workers in central Malawi. The process evaluation also demonstrated that trained observers can capture the degree to which both content and process of a complex intervention adhere to the model, when the observation is based on the conceptual framework that guides the intervention.

Congruent with other studies [6, 8, 19, 27, 37], the process evaluation was useful in strengthening intervention implementation. Qualitative data on content sensitivity, relevance and clarity enhanced the quantitative ratings; both were useful in strengthening facilitator performance and refining content delivery. Early observations were used to help peer leaders develop their session management and group facilitation skills. The effectiveness of this strategy in strengthening intervention fidelity was demonstrated by the increase in peer leaders' performance and participants' engagement over the six sessions. The high ratings of accuracy and completeness of coverage documented that peer group leaders' attention to group process did not interfere with content fidelity. Overall, process evaluation records documented that the peer groups had broad appeal among adults and adolescents in rural communities, and even greater appeal for urban hospital workers.

This process evaluation identified components of implementation that contribute to success. For all samples, session management was the strongest predictor of the session being more like a peer group than a classroom. This has implications for developing effective peer group interventions because session management skills are amenable to strengthening through peer leader training and early feedback. Peer leaders need ample opportunities to process their own discomfort in discussing sensitive topics, especially with adolescents. This helps prepare them to lead these discussions and to communicate the appropriateness of discussing these topics. Ensuring privacy is essential for discussion of sensitive topics. Peer leaders should actively engage group members at the beginning of a session, to counter the ‘warm-up’ period. Interventions for adolescents either should be designed as 1-hour sessions or should incorporate a break halfway through a longer session. Adolescents' groups might be more effectively led by young people slightly older than group members, rather than by adults.

Methodological strengths of this process evaluation included the use of a conceptually based observation form, ‘external’ observers who were not part of the intervention or outcome evaluation to reduce bias and training to achieve high inter-observer reliability. A limitation of this process evaluation was the lack of detail about session-specific topics and activities such as role plays. The process evaluation was relatively costly, as external observers were dedicated solely to the process evaluation. Future research might test whether peer leaders' self-ratings are congruent with ratings of external observers, which could reduce the number of external observations.

This process evaluation builds the evidence base for peer group interventions by documenting that trained volunteer peer leaders can effectively deliver a complex peer group intervention as intended, with acceptable intervention context, reach, dose delivered and received and fidelity of learning modalities and content. Using well-trained volunteers who receive ongoing support can be efficacious as well as low cost, providing a model for the delivery of HIV preventions in low-resource settings.

Funding

National Institutes of Health, National Institute of Nursing Research (R01 NR08058 to K.F.N.); World AIDS Foundation (WAF 304 03-004 to K.F.N. and C.P.N.K.); Fulbright African Regional Research Fellowship and University of Illinois at Chicago to K.F.N.

Conflict of interest statement

None declared.