Participatory Evaluation of Community-Based HPV and Cervical Cancer Prevention and Control Efforts

Introduction

This paper describes efforts to evaluate community efforts to develop community-based interventions to reduce the incidence of cervical cancer and the mortality associated with cervical cancer in one South Carolina county. The overall incidence rate of cervical cancer in this community is 17.1 cases per 100,000 women and the mortality rate is 2.6 deaths per 100,000.1 However, the mortality rate is almost twice as high (4.2) among African American (AA) women.^{1, 2} Cervical cancer is largely preventable with regular screening (e.g., Pap test and humanpapilloma virus (HPV) DNA test) and adherence to recommended follow-up care. Unfortunately, many people remain unaware of the link between some types of HPV and cervical cancer. Studies in the United States and the United Kingdom have found low levels of HPV knowledge, including risk factors, sexual transmission, disease management, cancer risk, or even basic awareness among young adults, adolescents, and adult women.^3–16

In recognition of cervical cancer disparities and the opportunities for addressing precursors to behavior, the Community Partnership and Outreach Core (CPOC) of the University of South Carolina (USC) and Claflin University Center of Excellence in Cancer and HIV Research is a community-based participatory initiative to develop and implement culturally and linguistically appropriate public health programs, resources, and services to eliminate HIV and HPV related health disparities. The specific aims of the CPOC are:

1. To develop, assemble, and engage community leadership in the form of a Community Advisory Group (CAG) to identify strategies for meeting the preventive and treatment needs of the community specific to HIV/AIDS and HPV/cervical cancer;

2. To provide technical assistance to the community for planning, designing, adapting, implementing, and evaluating community programs, services, and resources;

3. To serve as a resource for the community;

4. To develop and implement programs, services, and resources to reduce HIV/AIDS and HPV and cervical cancer health disparities within the community;

5. To assist with community capacity development opportunities;

6. To evaluate the community-based participatory research (CBPR) approach impact on the community and health outcomes during the project period.

The CAG is a crucial component of the CPOC. It guides the efforts of the CPOC, engages the community in an assessment process, oversees a mini-grant process to establish programs in the community; and participates in training events to build HPV knowledge and capacity. CAG members are adult, minority (AA) residents of the intended county who have an expressed interest in HIV/AIDS and/or HPV and cervical cancer. Through the CAG, the CPOC uses a community-based participatory approach to address the specific aims. Thus, a participatory evaluation approach is used to evaluate the work of the CPOC. This approach was chosen because of its emphasis on involving stakeholders in designing and implementing the evaluation and in the use of evaluation findings.^17–24

The CPOC evaluation is a multistep, non-linear, process, adapted from Fuerstein (1988) and Springett in Minkler and Wallerstein (2003). Evaluation steps (Table 1) are revisited over the course of the project and reinitialized each time the project begins a new phase.^25–27 Having members of the project (CAG members) involved in data collection and analysis may be perceived as a potential risk to evaluation objectivity. However, given the participatory nature of this project and the importance placed on learning through the evaluative process, it was decided that steps (such as evaluation methods training, collecting, and analyzing data in teams) would be taken to ensure objectivity, while still including CAG members in these activities. Participatory evaluation recognizes the value of having those that are most familiar with the programs and services included in interpreting the data. It provides a context for the findings often missing in traditional evaluations. By having the CAG fully engaged in all evaluation activities, the process has also helped to build evaluation capacity within the community.

Table 1.

CPOC Evaluation Steps

Step	Activities
Step 1: Select an evaluation coordination team	The evaluation coordination team is com- prised of CPOC investigators, a consul- tant, CPOC staff, and CAG members. At least half of the team must be comprised of CAG members.
Step 2: Decide on evaluation objectives	The evaluation team develops evaluation objectives based on: assessment findings, CPOC specific aims, and CAG needs and capacities.
Step 3: Choose methods for attaining evaluation objectives	Qualitative and quantitative methods are used to assess CAG functioning (annual survey), CAG efforts (CAG member logs and CAG sponsored event surveys), community members’ knowledge of HPV, HPV services, and perceptions of HPV risk (focus groups and surveys).
Step 4: Report evaluation plan to CAG	The evaluation team shared the evaluation plan with the CAG, received feedback, made adjustments to the plan and had the plan endorsed by the CPOC and CAG.
Step 5: Test methods, collect and analyze evaluation data	Data is collected by the evaluation coordination team and analyzed by the team along with the investigators. It is recognized that having members of the project involved in data collection and analysis is perceived to potentially impact objectivity.
Step 6: Share findings and make decisions	Findings are shared in written, oral, vi- sual, and verbal formats to accommodate a wide variety of capabilities and prefer- ences for how information is learned.

Methods

The CPOC used a mixed-methods^{28, 29} evaluation design that employed several qualitative and quantitative methods. Data collection methods in the CPOC evaluation were influenced by the evaluation objectives and the abilities of the evaluation coordination team. The evaluation team, comprised of CAG members and investigators, focused on assessing: (1) CAG functioning and CAG efforts and (2) community impacts on knowledge of HPV, HPV services, and perceptions of HPV risk.

CAG Functioning and Efforts

The annual survey assessing CAG functioning was adapted from previous instruments measuring group partnership.^{21, 30} The central characteristics measured through this survey include: shared leadership, participation, and communication. It also assesses constructs of social capital (e.g., social trust, responsibility, and cooperation) and community capacity (e.g., leadership, community participation, community resources, advocacy, planning community involvement communication, and creativity).^{17, 31–34} This was a helpful tool in monitoring CAG functioning and engaging CAG members in discussions and problem solving, which is consistent with a participatory evaluation approach.

Prior to administering the survey each year the evaluation team reviewed the survey for appropriateness. The annual CAG functioning survey was completed by CAG members at the beginning of one meeting each year. Members that were not present during the meeting were mailed a copy of the survey and asked to complete it and bring it with them to the next meeting. After the data were collected they were entered into SPSS and mean scores were calculated for each survey item. Once compiled, data were shared with the evaluation team. The evaluation team discussed and collectively interpreted the findings. The results were then shared with the CAG. CAG member participation in HIV, HPV, or cervical cancer community activities were captured through a log that the members completed and returned to the evaluation team each month. The log assessed the activity type, topic and location; number of participants engaged in the activity, and the CAG members’ role in the activity. Data from the logs were entered into Excel each month. Evaluation team members reviewed the results each month and an activity report highlighting the frequency and type of activities was shared with the CAG during the monthly meetings.

Community Impacts

The CPOC and CAG conducted key informant interviews, focus groups, and surveys with community members (not included in the CAG) as part of an initial community-wide assessment to establish a baseline level of: awareness, knowledge, perceptions of services and programs, and views regarding stigmatization of those with HIV/AIDS and HPV/cervical cancer. This information was used to help guide the work of the CPOC and CAG. Key informants and focus group participants were asked to complete a short survey as part of the interview or group discussion. The evaluation coordination team included a small number of closed-ended questions regarding access to screening and other services pertaining to HIV/AIDS and HPV and cervical cancer.

Key informant and focus group participants were identified via a multi-step process. The first step involved dividing the county into 4 distinct regions. CAG members from each region identified local key informants (individuals) and key organizations in several different categories (e.g., government, business, education, faith-based, civic, social, and grass-roots groups). Once the regional lists of individual and organizations were compiled the individuals were contacted about participating in an interview and the organizations were contacted about hosting a focus group with their members. Key informants and organizations were also asked to identify additional key individuals and organizations to add to the list for each region. This process continued until a minimum of two individuals and two organizations were identified for each category.

CAG members interested in assisting with the key informant interviews and focus groups participated in a four hour (over two days) training on: qualitative data expectations; interviewer and moderator tasks, interviewer and moderator skills; attributes of effective interviewers and moderators; recognizing and limiting actions that may introduce bias; protecting confidentiality; the interview and focus group guide; and note taking skills. Interviews were conducted by the CPOC investigative team members with CAG members assisting as note takers. Focus groups were moderated by trained CPOC staff and CAG members. Interviews and focus groups were audiotaped then transcribed.

A five person team (two investigators, one CPOC staff member, one student and one consultant) reviewed a subset of transcripts to develop the initial set of analysis codes. Analysis of qualitative data (using the constant comparison method) began as soon as audiotaped interviews were transcribed verbatim and reviewed for quality control purposes.^{35, 36} In keeping with principles of qualitative analysis, the analysis team did not impose preconceived coding schemes onto the data. Coding was based on both a thematic and pattern analysis.³⁷ Themes were defined in terms of their meaning to the informant and/or focus group participant(s). Themes were identified, used to code transcripts, and then compared across interviewees and focus groups for presence or absence of specific themes. Pattern analysis, by contrast, was a more “etic” interpretation of text data that was grouped by specific codes. The meaning was identified by the analysis team based on behavioral and community theory, knowledge of HPV and cervical cancer, and geographic patterns in the community. Once analysis was complete findings were shared with CAG members for additional interpretation and meaning.³⁷ This was followed by presentations of the findings in various regions of the county followed group discussion to illicit local interpretation of the findings and spark local planning processes. Quantitative data from the survey conducted with interview and focus group participants were analyzed descriptively.

Results

CAG Functioning and Efforts

The results of the annual CAG survey (Table 2) revealed a highly functioning CAG. Most item scores averaged 3.0 or higher (on a scale of 1 to 4 with 4 being the value representing highest functioning or most positive impressions). Results revealed a CAG that struggled with a few issues in the second year of the project, but quickly regrouped and began functioning at a high level again in year 3. The highest average scores were found in measures assessing the CAG’s success in bringing people together with a shared interest in HIV/AIDS and HPV and cervical cancer and providing a forum to share ideas and information. Additionally, the survey revealed that CAG members almost doubled the amount of time that they committed to CAG related activities from 3.2 hours per month during the first year of the project to 6.2 hours per month in the third year. The CAG member logs revealed that CAG members reported participating in 69 HIV/AIDS or HPV and cervical cancer community events over a seventeen month period. Additionally, CAG members had a coordinating or organizing role in an overwhelming majority of these events.

Table 2.

CAG Functioning

	CAG Year 1 2006/2007 N= 21	CAG Year 2 2007/2008 N=18	CAG Year 3 2008/2009 N=19
Listen to ideas of all members	3.3	3.17	3.41
Give you a chance to share your ideas and views	3.29	3.06	3.44
Make people feel welcome and wanted as part of the CAG	3.57	3.11	3.28
Create a safe place to share views and ideas	3.33	3.31	3.28
Bring together people with an interest in HIV/AIDS and HPV/cervical cancer	3.62	3.28	3.61
Help new members understand the project	3.15	2.78	2.89
Use the storymaps to tell the story of disparities	2.95	3.13	3.29
Celebrate progress	3.30	3.06	3.35
Share information about CAG activities (for example, there is not a lot of “insider” information that only a few in the CAG know)	3.16	3.06	3.11
Share information about upcoming events and activities to all CAG members	3.48	3.00	3.33
Share information about HIV/AIDS and/or HPV/cervical cancer work in Orangeburg County	3.33	3.06	3.39
Accomplish what we should during the meeting	3.10	2.67	3.28
Accomplish what we should outside the meeting time	2.94	2.65	3.12
Have an agreed upon method of leadership	3.14	2.78	3.31
Have an agreed upon process for making decisions	3.10	3.85	3.06
Share responsibility and workload so that work is accomplished together	3.05	2.75	3.18
Build a clear mission of CAG/EXPORT	3.11	2.59	3.00
Resolve conflict within group	3.0	2.76	3.18
There is a shared vision for the CAG	3.15	3.00	3.11
The CAG has a set of goals and strategies for creating change in Orangeburg County.	3.05	2.88	3.06
I can see how my personal vision fits the CAG vision.	2.95	3.00	3.22
CAG works with a broad range of groups in Orangeburg County.	2.86	2.72	3.22
CAG members share responsibility and workload so that work is collaborative.	3.00	2.71	3.00
CAG decisions are made together with everyone on the CAG having the opportu- nity to participate.	3.00	3.00	3.06
As a group we are able to resolve conflict in order to reach a decision.	3.00	2.89	3.11
Decision making power is shared and not in the hands of a few.	3.05	2.94	2.94
Leadership in the CAG is shared.	3.10	2.71	3.11
All CAG members are heard and respected.	3.11	3.06	3.17
Capacities and skills of CAG members are recognized and used with the CAG process.	2.94	2.82	3.11
Information about upcoming CAG events and activities are shared with all mem bers.	3.25	2.88	3.44
The communication activities of the CAG enhance communication with the larger community.	2.94	2.76	3.06
CAG members have relationships to gain support of local organizations.	3.00	2.88	3.17
Successful activities of the CAG are incorporated into ongoing work of local organi- zations	2.90	3.00	3.11

Note: Table reflects means responses for how much each respondent agreed with each statement on a scale of one (strongly disagree) to four (strongly agree)

Community Impacts

Findings from the community surveys (Table 3, Table 4 and Table 5) as well as 31 key informant interviews and 11 focus groups conducted during the summer 2007 reflected several differences in the types of information that community residents perceived as most important to include in education and awareness programs focused on HPV and cervical cancer. Results also illuminated key differences and similarities in health issues, HPV needs, and resources. Finally, these findings prompted several key suggestions for methods to effectively reach community residents with prevention programs. Participants identified a number of health issues for rural communities. Interestingly, higher priority was assigned to social and economic concerns such as gangs, drug use, violence, poor education, and poverty as opposed to concentrating on specific illnesses or diseases. Also emphasized was the need for educational programs that keep children in schools (i.e., reducing drop-out rates). However, residents recognized and discussed how these issues are interconnected.

Table 3.

Key Informant Demographics

Characteristics	Frequency (%)
	Total = 31 (100)
Age (years)
Mean (S.D.) − 47.6 (12.1)
Gender
Male	13 (41.9)
Female	18 (58.1)
Race
Black	28 (90.3)
White	3 (9.7)
Education
≤ High School	1 (3.2)
Some college	8 (25.8)
College degree	22(71.0)
Marital Status
Married	18 (58.0)
Single	7 (22.6)
Other	6 (19.4)
Insurance
Private	22 (71.0)
Medicaid/Medicare	7 (22.6)
Other	2 (6.4)
Knowledge about the cause of cervical cancer
Correct	13 (41.9)
Incorrect	18 (58.1)
Knowledge about the route of HIV transmission
Correct	26 (83.9)
Incorrect	5 (16.1)
Heard about HPV
Yes	25 (80.7)
No	6 (19.3)
Heard about HPV vaccine
Yes	26 (83.9)
No	5 (16.1)
Known a woman who has had cervical cancer
Yes	18 (58.1)
No	13 (41.9)
Known anyone who has HIV
Yes	30 (96.8)
No	1 (3.2)

Table 4.

Urban Focus Group Participant Demographics

Characteristics	Frequency (%)
	Total = 37 (100)
Age (years)
Mean (S.D.) − 21.1 (2.2)
Gender
Male	10 (27.8)
Female	26 (72.2)
Race
Black	35 (94.6)
Other	2 (5.4)
Education
≤ High School	9 (24.3)
Some college	26 (70.3)
College degree	2 (5.4)
Marital Status
Married	4 (11.1)
Single	32 (88.9)
Insurance
Private	17 (47.2)
Medicaid/Medicare	7 (19.4)
Other	2 (5.6)
None	10 (27.8)
Knowledge about the cause of cervical cancer
Correct	24 (66.7)
Incorrect	12 (33.3)
Knowledge about the route of HIV transmission
Correct	36 (100.0)
Heard about HPV
Yes	28 (80.0)
No	7 (20.0)
Heard about HPV vaccine
Yes	15 (42.9)
No	20 (57.1)
Known a woman who has had cervical cancer
Yes	9 (25.0)
No	27 (75.0)
Known anyone who has HIV
Yes	18 (50.0)
No	18 (50.0)

Table 5.

Rural Focus Group Participant Demographics

Characteristics	Frequency (%)
	Total = 85 (100)
Age (years)
Mean (S.D.) − 21.1 (2.2)
Gender
Male	22 (25.9)
Female	63 (74.1)
Race
Black	83(97.6)
White	1 (1.2)
Other	1 (1.2)
Education
≤ High School	42 (49.4)
Some college	19 (22.4)
College degree	24 (28.2)
Marital Status
Married	33 (38.8)
Single	37 (43.5)
Other	15 (17.7)
Insurance
Private	47 (56.6)
Medicaid/Medicare	19 (22.9)
Other	3 (3.6)
None	14 (16.9)
Knowledge about the cause of cervical cancer
Correct	28 (33.7)
Incorrect	55 (66.3)
Knowledge about the route of HIV transmission
Correct	67 (79.8)
Incorrect	17 (20.2)
Heard about HPV
Yes	50 (58.8)
No	35 (41.2)
Heard about HPV vaccine
Yes	29 (34.9)
No	54 (65.1)
Known a woman who has had cervical cancer
Yes	34 (40.5)
No	50 (59.5)
Known anyone who has HIV
Yes	59 (69.4)
No	26 (30.6)

With reference to access to health care, residents identified several key factors associated with access. Issues included travel distance required to reach some forms of care and the need for transportation assistance and/or the need for more services offered within their local communities. They also talked about the cost of health care, especially for people without health insurance.

Participants had many questions about HPV and cervical cancer. Many of these questions confirmed a need to focus on increasing the level and accuracy of knowledge about HPV along with efforts to increase awareness of HPV. Example questions include:

1. Is HPV a new form of HIV?

2. How is HPV transmitted?

3. How are you tested to find out if you have HPV?

4. How effective is the HPV vaccine?

5. When are girls vaccinated for HPV?

6. What is (should be) the role of schools, families, friends in sex education?

7. What is the relationship between HPV and cervical cancer?

Participants had several ideas on how to help community members be healthier (in general). They discussed programs that addressed lifestyle behaviors such as exercise and eating. They also talked about the need for educational programs to raise awareness of a variety of health issues. They recognized the need for programs addressing youth needs related to youth risk behaviors and steps regarding how to prevent and/or treat these health conditions. Finally, participants discussed the broader need for more health care in the community. Within this conversation, the need for increased access to health screenings (locally, within community) came up as topic of particular interest.

Participants provided several suggestions in response to specific initiatives possible in their community regarding HPV and cervical cancer. Considerable discussion focused on the need for education/awareness programs and how these programs should occur in a wide variety of locations with a wide variety of people within the community. The participants also talked about the need for funding for educational programs and programs to assist people in need. Lastly, participants talked about the HPV vaccine and discussed their own uncertainties about the vaccine and the general lack of knowledge about the vaccine in terms of its availability, cost, effectiveness, etc.

Finally, participants suggested several methods that they thought would be effective for reaching the community regarding HPV and cervical cancer programs. Most of the methods that they identified represented interpersonal “grassroots” efforts such as working with local organizations, collaborating with the health ministries of churches and other faith-based institutions, and one-on-one conversations with community residents, as opposed to the use of print or electronic media.

Participants who were asked specific questions about HPV illustrated the need for more accurate information related to cervical cancer and HPV. Many incorrect responses (Table 3, Table 4 and Table 5) were offered to questions such as: (1) How is HPV transmitted? (2) How are you tested to find out if you find it? (3) What is the HPV vaccine? (4) What is cervical cancer? (5) What causes cervical cancer? (7) When are girls vaccinated for HPV?

Discussion

Decisions about methodologies for conducting an evaluation of a community program require consideration of several factors such as: how evaluation data will be used; the evaluation questions to be answered; and the need or desire to build community evaluation capacity³⁸ A participatory approach was used for this project because of the emphasis on continually sharing the evaluation findings to help guide and shape the community efforts and because of the desire to build evaluation capacity among the CAG. Findings from this evaluation were shared in written, oral, visual, and verbal formats to accommodate a wide variety of capabilities and preferences for how information is learned. All findings were first shared with the evaluation team for interpretation and decision-making regarding the best format for sharing with the CAG and the community. The goal was to create an open forum that generated discussion on how to improve community partnership and outreach efforts. This approach was based on evidence that suggests learning about your progress and critically reflecting on successes and challenges can be a difficult process for many. For example, in the second year of the CAG the evaluation data identified a decrease in functioning. This was difficult to hear, especially given the hard work that the members had put into the CAG. However, the CAG members used this information to help reorganize, develop by-laws and invest in the development of a strong member-leadership structure to guide the group. This type of reflection requires a safe environment, honesty, and skill on behalf of the facilitator to minimize defensive behavior.26 The CAG, through its program planning subcommittee, used the findings of the Center’s evaluation process to guide their actions and efforts to support education and awareness programs, link people to resources, and address gaps in current services and programs. For example, findings from the interviews and focus groups indicating concerns regarding youth risky behavior helped shape a focus on developing HPV and cervical cancer awareness and education programs for youth and young adults and helped shape the delivery methods for awareness building and educational information.

Conclusion

Cervical cancer disparities are influenced by complex social and behavioral factors. A community-based ecological approach to address individual (knowledge, attitude) interpersonal (family and peer influences), and community/policy (access to preventive service, access to care) is required to eliminate this disparity.³⁸ Through a participatory evaluation approach community members have the opportunity to use the evaluation findings to continually improve their efforts and build their evaluation skills. While this approach has many strengths (e.g., inclusive, findings raise awareness, shared control, builds skills), it has several inherent challenges. Participatory evaluations are at a great risk for bias in the finding.²⁴ It is crucial that the evaluation team takes measures (e.g., training evaluation team members; using mixed method designs) to minimize bias and ensure reliability and validity of the findings. Additionally, participatory evaluation approaches are typically more time-consuming and the role of an evaluation team comprised of professional evaluators and engaged community members is crucial.²³

Implications for Public Health

The participatory evaluation framework used in this study provided an example of functioning and impact data that can be used in evaluation of community-based participatory research approaches. It also shows how the evaluation can serve to create a climate of reflection and self and group assessment so that findings are used to establish a baseline, monitor efforts, and assess functioning. Through this process data are used to continually improve the group processes, and assess community impacts. This approach that used data systematically to inform HPV and cervical cancer practice and engage community members in HPV and cervical cancer discussions is in keeping with the key components of evidence based public health.³⁹