Evaluation Lessons Learned from Implementing CBPR in Native American Communities

The following articles of this issue of JCE report on “Native Navigators and the Cancer Continuum (NNACC),” a project that was funded by the National Center for Minority Health and Health Disparities (R24MD002811). NNACC was developed to address cancer health disparities among five populations of Native Americans in the northern and southern plains and Rocky Mountain regions of the USA. Efforts to address health disparities among Native Americans are a high priority as this population has a long history of poor survival from cancer and worse overall health status than their non-Native American counterparts. The poverty level of Native Americans is nearly twice the US rate, educational attainment is lower, and access to cancer prevention and control services is lower. Given this context, development and implementation of cancer education for Native Americans presents substantial challenge.

Awareness of cultural issues is critical to successful implementation of cancer education interventions for all populations. For Native Americans, however, gaining awareness of cultural issues brings special challenges because of the diversity of the population. While some cultural issues cut across American Indian organizations, many are specific to a group of tribes or even to a single tribe. As there are over 565 Native American federally recognized tribes, there is probably more cultural diversity among this population than of any racial/ethnic minority population in the USA. The diversity of cultural factors presents a substantial challenge to cancer educators. But as it turns out, community-based participatory research (CBPR) provided a very useful conceptual framework for ensuring that cultural factors are included in all phases of cancer education programs. CBPR engages local partners in program genesis, design, implementation, and evaluation, and in this process, culture is represented. NNACC incorporated CBPR with all five partners from the beginning of the project which led to clear inclusion of the needs, interests, and values of the partners in cancer education. CBPR empowered community partners in the NNACC project.

CBPR added breadth and depth to the education intervention. Each site had leadership for creating or modifying modules, and all partners shared information or activities that augmented the workshops. The subsequent quality of the workshops is why they were evaluated positively by participants regardless of education background, age, and gender (see the “NNACC findings” article in this issue).

However, the CBPR approach also brought a series of interesting challenges to the project, particularly to the evaluation component. The cancer education program was designed to include a series of modules to be used in community presentations. The modules were presented by Native Patient Navigators (NPNs) who were individuals from the partner communities who were trained and supported for their role on the project. Each presentation was to be evaluated using pretests and posttests, and the responses were to be collected using an electronic audience response system. The completed evaluations were to be submitted to the NNACC project office via a website and directly to the evaluator. The evaluation plan for the project was designed to provide a clear structure to assessment of the cancer education program, provide a consistent structure to data collection, and minimize burden on those delivering the cancer education program and those receiving the education.

The results from this CBPR project provided clear evidence that the partners remained fully involved in implementing the project. Their involvement resulted in the following observations:

1. Partners evaluated their communities, evaluated the modules, and selected those that best suited their communities.

2. The pre- and posttests were reviewed by community partners, and the Western IRB and decisions were made at the community level regarding which items to use and which not to use.

The audience response systems and website data submission plans worked well for the most part, but the online for submitting data to the project office was not intuitive and local administrative staff provided one-on-one supplemental training on how to use the program effectively. In addition, one tribal partner did not use the online website in a timely manner and from that site remains incomplete.

The evaluation and data analysis plans for NNACC were relatively simple and straightforward and included documenting and monitoring the implementation of the cancer education program by the five partners and analysis of change in the level of awareness and knowledge of attendees using data from pre- and post-presentation assessments. The effects of the decisions of the community partners relative to implementing the program listed above had a definite impact on how NNACC could be evaluated and how analyses could be carried out.

Analyses were carried out by module and began with a careful documentation of how the module was implemented across the five partners. As the analyses proceeded, it became clear that the number of partners contributing observations and the number of observations varied substantially from module to module. The analysis plan that started with a clear (and relatively simple) approach was modified to one that included extensive efforts to develop data sets for analysis. Rather than assessing pre- to posttest change for each module across the five partners, subsets of partners were used for the analyses.

The experience from evaluating the NNACC cancer education program reflects the larger issue of the difference between traditional approaches to evaluation community-based cancer education and evaluation of a CBPR program. When implemented with fidelity, CBPR is closely attuned to the needs, interests, and values of the community (tribal communities in this case). This approach has a powerful influence on the research process. Rather than emphasizing consistency in program implementation and evaluation, CBPR emphasizes partnerships and responsiveness to community needs. NNACC was CBPR and was highly responsive to the community (tribal) partners. As the manuscripts that follow will show, NNACC was an ambitious cancer education program that was successfully implemented in five tribal settings. While NNACC was designed with plans to use a traditional evaluation approach, CBPR necessitated that the evaluation approach be modified to respond to the partners’ decisions regarding the cancer education program. Data management became reactive because the community made the decisions regarding the cancer education program that directly affected evaluation.

The administrative team identified solutions to some of the evaluation issues. The partners would continue to have a template of workshops, and individual sites can select which workshops they choose to implement within the four components of the cancer care continuum (prevention, early detection, survivorship, and palliative care). However, within each component, at least two workshops needed to be included within each workshop series (e.g., “keep tobacco sacred” in the prevention component and the “Get on the Path to Health” modules for breast, cervix, colon, and prostate for the second component and so on. Such consistency would support evaluation. Another key recommendation to overcome CBPR inconsistencies within the evaluation plan was to urge the NPNs to upload to the online evaluation website and email ARS reports within a week of implementing workshops. There were also issues with local administrative staff associated with managing projects at a distance. This required one administrator to drive 5 hours to meet with the NPN in one of the settings to confirm the data were being sent and uploaded. The travel was repeated every 6 months, but in the future, at least initially, such efforts may be augmented by webinars and conducted monthly.

The articles that follow describe NNAC from several points of view. The first article presents an overview of study findings. The second describes subtle cultural foundations explaining why it is important to have native patient navigators rather than individuals from other racial or ethnic groups. The third is from the administrative team addressing partnership processes, and the fourth is from the NPNs’ perspective on lessons they learned about including community presentations/workshops within their scope of work as navigators.