Community-Based Participatory Research for Improved Mental Health

Abstract

Community-based participatory research (CBPR) focuses on specific community needs, and produces results that directly address those needs. Although conducting ethical CBPR is critical to its success, few academic programs include this training in their curricula. This paper describes the development and evaluation of an online training course designed to increase the use of CBPR in mental health disciplines. Developed using a participatory approach involving a community of experts, this course challenges traditional research by introducing a collaborative process meant to encourage increased participation by special populations, and narrow the parity gap in effective mental health treatment and services delivery.

Despite years of federal and state efforts devoted to improving the health of all Americans, large demographic groups across the nation remain “underserved,” and do not receive the medical or mental health care they need. As an example of persisting disparities in mental health care, higher rates of mental disorders and consequent disabilities among racial and ethnic minorities are not due to greater prevalence or severity of illnesses in these populations per se, but rather to the lack of care or inadequate care. Members of these populations remain underserved in their mental health services needs primarily because of various barriers to care such as poverty, language, homelessness, geographical and social isolation, and suffering from stigmatizing disorders (President’s New Freedom Commission on Mental Health, 2002; United States Public Health Service Office of the Surgeon General, 2001).

Barriers to mental health services delivery are also barriers to participation in research. Underrepresentation of special populations in research delays the advancement of effective treatment for these groups, and in forming national health policy that is largely based on data collected from research. For example, the U.S. Surgeon General’s 2001 Report, Mental Health: Culture, Race, and Ethnicity, describes in detail the limited or lacking information on certain racial and ethnic groups in randomized clinical trials evaluating the efficacy of treatments for bipolar disorder, major depression, schizophrenia, and attention-deficit/hyperactivity disorder. Such under-representation in mental health research is harmful to special populations, as without sufficient research data to inform national health policy, disparity in services delivery will remain unchanged.

The President’s 2002 New Freedom Commission on Mental Health Report identifies a pressing need for more research that involves, and will ultimately benefit members of underserved populations. Widespread implementation of the participatory research model in mental health and medical research can clearly play a key role in reducing health disparities. The community-based participatory model in which all stakeholders of a community being studied are invited and encouraged to take part in all phases of a research study is an ethical and effective way of conducting mental health research, especially with members of underserved communities (Minkler & Wallerstein, 2003; Sixsmith, Boneham, & Goldring, 2003; Macaulay et al., 1999; Agency for Healthcare Research and Quality [AHRQ], 2003).

Mental health researchers are recognizing the importance of asking communities to set the agenda and context for appropriate, respectful research. Research that starts “where the people are” has demonstrated that when the realities of community life become an integral part of the process, study results directly address the needs of the target community (Israel, Schulz, Parker, & Becker, 1998; Macauley et al., 1999; Minkler & Wallerstein, 2003; Sixsmith, Boneham, & Goldring, 2003). A community-based research approach increases participation of typically underserved groups, makes research outcomes more meaningful to communities, and fosters partnerships between community members and researchers. The ultimate goal of this approach is to achieve not only better mental health outcomes but to establish empirically validated services and interventions and to enhance the dissemination and implementation of research findings (Agency for Healthcare Research and Quality, 2002; House & Williams, 2000; Smedley, Stith, & Nelson, 2002).

Achieving broader participation through community-based participatory research (CBPR) requires wider community outreach, and mutual understanding and goal-setting between researchers and communities of people (Greenwood, Whyte, & Harkevy, 1993; Hatch, Moss, Saran, Presley-Cantrell, & Mallory, 1993; Minkler & Wallerstein, 2003). It requires researchers to consider community life on a personal level (i.e., the day-to-day realities of community members) and a systemic level (i.e., social, political, economic, cultural, and environmental) in developing appropriate context for their work. CBPR differs significantly from more traditional approaches to research in which researchers are more independent and less engaged directly with their target group. Further, given the unique challenges presented in community-researcher partnerships, additional ethical issues arise that often put the researcher in conflict with more traditional research ethics. For example, when community members share in all aspects of the study, there may be difficulties maintaining confidentiality, or a heightened burden for participants with stigmatizing illnesses.

Given that the goals of CBPR require researchers, institutions, and communities to work together in new and different ways, doing such research in a manner that is ethical and maintains respect for community needs is critical to its success. Yet, community-based approaches are rarely part of academic curricula, creating the risk of inappropriate or unethical applications of CBPR. To fill this training gap, we have developed an online training course and companion community engagement materials to help prepare researchers to engage in ethical community-based participatory research. With funding from the Small Business Innovation Research (SBIR) mechanism through the National Institutes of Health, we used a cyclical, participatory, data-gathering process for development and refinement of this course and all its ancillary materials. This paper describes the development and testing of the course designed to guide researchers in conducting ethical, collaborative, community-based research.

Community-Based Participatory Research Course

The CBPR course, entitled Community-Based Participatory Research for Improved Mental Health, was developed by Terra Nova Learning Systems in collaboration with Behavioral Health Research and Services (BHRS) at the University of Alaska, Anchorage (UAA). The course is designed to train researchers to recognize the differences between CBPR and traditional research strategies, and to understand the complexities of building and maintaining trusting relationships with study communities. This course intends to inform mental health research professionals about incorporating social, political, and cultural systems that exist in a given community into research protocol to conduct a study that will specifically address the needs of the very people who will contribute the study data. The course’s ancillary materials are meant to inform community members about the participatory research process, introduce the idea of community-researcher collaboration, and begin to dispel trust issues that have been formed due to wrongdoing by researchers in the past.

Course Description

This online course was designed as an educational intervention to enhance the ethical knowledge, sensitivities, skills, and behaviors of mental health researchers working with communities. The course contains five instructional modules designed to train researchers in the principles and practices of community-based mental health research (see Table 1 for details). The five modules are as follows: Background Preparation for the Participatory Research Process, Beginning Partnerships with Communities, Finding Common Ground within the Partnership, Sustaining the Partnership, and Evaluating the Partnership and Enhancing Future Successes. The modules focus extensively on educating researchers on the ethical issues of respect, promise-keeping, truthfulness, clarity, and integrity that must remain at the forefront, and underlie all decision-making at every level of their work. All modules contain insights, strategies, and personal reflections from those who have experience working with vulnerable and special populations. Through completing this course, users learn to clarify project goals, reflect on how personal assumptions and biases influence their work, and value the perspectives of those who possess practical, day-to-day experience with community life.

Table 1.

Community-Based Participatory Research for Improved Mental Health Outline

Module Key Questions Addressed	Module Content Focus
Module 1: Background Preparation for the Participatory Research Process
• What is important about community participation?	• Goals and importance of community    participation
• What are the goals of the research relationship with the community?
• How are communities chosen, scientifically, ethically, and collaboratively when    the research originates outside of the community?	• Defining the specific target community    (demographics, history, etc.)
• What is the relevant history, and where are the sources of trust and mistrust?	• Understanding the historic relationship    between research institution and    community
• How does the research staff prepare for the relationships and responsibilities of    community participatory research?
• What are the practical resource issues	• Planning for the time and personnel    needed; establishing inclusion values    within the project and staff
Module 2: Beginning Partnerships with Communities
• Who are the key leaders and natural wise persons with whom trustworthiness    and integrity must be demonstrated?	• Identifying key stakeholders and key    community informants (formal and    “natural” leaders)
• When and how do researchers approach a community?
• What information is appropriate to share about the scientific questions, the    methodology, and the anticipated risks and benefits to the community and to    individual participants?	• Deciding how and when to engage the    community; developing community    advisory boards (authority, structure,    function, duration)
• How do researchers figure out ways to minimize research risks to the    community?
• How are community advisory boards developed, and how will healthy, effective    functioning be facilitated?	• Timing the steps of community    involvement
• How should values conflicts and disagreements be addressed?
Module 3: Finding Common Ground Within the Partnership
• How do researchers determine the values and needs of the community?	• Understanding cultural appropriateness    and acceptability for the specific    community
• How are implicit concerns expressed?
• How do partners find common ground on differing goals and preferences?
• What are the processes for dealing with conflicts and disagreements on an    ongoing basis?	• Conducting community needs    assessments; developing mutual or    compatible goals – ethical issues (i.e.,    how does the community define them?)
• What if the partnership fails?
• What ethical safeguards does the community need or want?
• What are the jurisdiction issues?	• Data and safety monitoring
	• Issues from the community
Module 4: Sustaining the Partnership
• How are day-to-day issues addressed?	• Maintaining an active community    advisory board
• How is the equality of all partners maintained throughout the process?
• How are all partners kept informed of process and progress?	• Common problems/issues and how they    might be addressed
• How are community advisory boards maintained?
• How are major changes in the ‘ground rules’ addressed and communicated	• Dealing with significant changes in the    community or the research
	• Importance of ongoing communication    between community and research
Module 5: Evaluating the Partnership and Enhancing Future Successes
• How are study results used in ways that are acceptable to all?	• Determining “success” as defined by    various sectors of the community and by    research standards
• How is the relationship sustained when the work of the research study is    finished?
• How and why have reflection and debriefing become essential activities?	• Assessing strengths and weaknesses to    improve future collaboration
• How is the success of the process involved in project implementation evaluated    from the perspectives of all stakeholders?	• Building knowledge about community –    research collaboration
	• Sharing experiences

The learning objectives for this CBPR course are designed around the five core participatory research content components described in Table 1. It is expected that, at a minimum, individuals who complete the course will be able to:

1. Describe how CBPR differs from traditional research.

2. Summarize the primary issues associated with defining and identifying a community.

3. Explain the importance of joining a community in advance of conducting the work of a CBPR project.

4. Explain the activities necessary to initiate a CBPR project.

5. Recognize the importance of community input in effectively planning and carrying out an appropriate participatory research study.

6. Describe effective strategies for promoting and maintaining trust, communication, connectedness, respect, and meaningful work with a community.

7. Identify effective processes for resolving conflict and finding common ground within the research partnership

8. Apply strategies to build, sustain, and strengthen the community partnership.

9. Identify and synthesize strategies to monitor and maintain equality of power and influence within the partnership.

10. Recognize and explain the importance of community input in the interpretation of data and dissemination of study results.

11. Identify ways to successfully complete the study while sustaining the relationship with the community.

Community Engagement Materials Description

The CBPR course offers companion community engagement materials that highlight key questions, human participant rights, and processes for relationship- and trust-building with communities. These materials were designed to assist research teams in engaging and involving community members in research work, and to provide resource materials for course users (including an online community). The outreach materials include an informational brochure containing a thorough overview of community-based participatory research; a Microsoft PowerPoint slide show and Facilitator Guide for conducting informational meetings with communities; a customizable print ad for introducing the research study in local newspapers and other print periodicals; customizable meeting announcement posters; and a resource list for additional information about research participation and mental health issues (see Table 2 for details).

Table 2.

CBPR Course and Ancillary Materials Key Features

Key features included in each module

• Researcher Self-Reflections: delivered as quotes gleaned from real-world researchers reflecting on    experiences with community-based participatory research. These statements provide key insights into    changes in thinking, adjustments to prior assumptions about self or others, and other ways in which    these researchers have adjusted their process to respectfully join with community partners.

• Video Reflections: brief video vignettes from researchers and community members designed to    provide course users with comments, opinions, and experiences with CBPR.

• Case Studies: based on real-world examples of CBPR projects.

• Glossary: includes terms that are highlighted the first time they appear in each module and links to    glossary definitions. A full glossary is also included with the course.

• References: citations for specific content as well as additional citations describing published articles    on successful as well as unsuccessful CBPR projects.

Key features of Community Outreach Kit

• Community Brochure: designed to provide community members with basic information about    CBPR (e.g., can be used at the first community meeting, during discussions with community leaders,    as a guide for local research partners who will assist in a project.).

• CBPR Slide Presentation: corresponds to the Community Brochure and was designed for researcher    use during small and large group meetings, or for one-on-one discussions.

• Facilitator Guide: intended to assist researchers in conducting initial community meetings. The    Facilitator Guide contains general group facilitation tips, including slide-by-slide commentary that is    coordinated with the CBPR slide presentation, and background information on basic CBPR concepts.

• Posters: engaging, customizable posters for a variety of purposes (e.g., meeting announcements,    recruiting, community handouts, etc.).

• Newspaper ad template: for study recruitment or announcement purposes.

• Stationery: designed for multiple uses (e.g., letters, press releases, additional handouts).

• Community Resources List: related to mental health and research, for use by both community    members and researchers.

• Researcher Guide: handy, concise checklist of salient topics related to successful CBPR (e.g.,    relationship-building, conflict resolution, and self-reflection strategies).

Key features of Online Community

• Peer Communication: connects CBPR researchers for support and networking.

• Discussion Forums: provides CBPR researchers and students opportunity to discuss CBPR-related    topics

Course Development and Evaluation

Course Development

Development of the course and ancillary materials was iterative in nature and relied on a consistently strong emphasis on scientific principles, empirical validation of materials produced, cultural competency, and appreciation for the relationships that serve as the foundation for responsible engagement and collaboration between scientist and community. The course development strategy used was itself a participatory process involving a core team comprised of the staff at Terra Nova and BHRS; a consultant panel of mental health researchers and experts; and a group of community advisors, members, and advocates. All project consultants and advisors were selected as professionals whose cultural backgrounds and life experiences would directly inform the project development and evaluation. This team, functioning as a community of representative end-users, possessed unique expertise in research ethics, education, mental health services delivery, and community life, and contributed to the development of critical course features such as case examples of both successes and “pitfalls” of community-based participatory work.

Development of all materials resulted from both formal and informal data-gathering with advisors and consultants that centered on the following topics:

1. Identification of specific topics and content areas for in-depth attention in the course materials.

2. Development of illustrative case examples based on a literature review, public domain source review, and direct experience of the panelists to illustrate cardinal ethical concerns encountered in community-based mental health research.

3. Feedback on the proposed content of all modules from diverse perspectives.

4. Format preferences for presentation of course and community materials.

5. Strategies for development of companion resource materials involving community perspectives related to each topic/content area.

6. Development of sequential self-assessment exercises related to each module content area.

7. Preferences for optimal format and presentation of the overall course materials across diverse communities with differing resources.

8. Feasibility of developing the course in both Spanish and English.

9. Feasible approaches for updating the course as ethical guidelines and national practices and regulations evolve over time.

10. Possible strategies for augmenting the impact of the course through a support network of CBPR researchers in an online community.

11. Feasible and cost-effective approaches for evaluating the perceived value, effectiveness, and sustainability of the course and ancillary materials where utilized by researchers and community members.

Evaluation of Training Materials

The evaluation of the course, ancillary training materials, and Community Outreach Kit mirrored participatory principles by gathering both qualitative and quantitative data from research professionals and community representatives, analyzing and reflecting on that data, and making subsequent adjustments and edits to the project materials. The evaluation process involved three stages: 1) initial review of course content and format; 2) alpha-testing for content and format feasibility; and 3) beta-testing of learning outcomes. For each evaluation phase, two advisory groups were formed: 1) a professional group consisting of mental health investigators, researchers, program directors, and clinicians; and 2) a community group consisting of persons living with mental illness, their family members, and community consumer advocates. Achieving a racially and ethnically diverse membership for each group was a priority throughout project work.

In part, the evaluation assessed the degree to which the course met the original goals and objectives of the project. The overall goals and objectives were linked to a set of outcome measurements. Below is a list of the outcome measurement components identified as most integral to the project. These components formed the focus of the evaluation process.

• Knowledge of the components and goals of participatory mental health research

• Attitudes toward the importance of the components and goals of participatory mental health research

• Attitudes toward ethically important aspects of participatory mental health research

• Attitudes toward culturally important aspects of participatory mental health research

• Confidence in identifying ethically important aspects of participatory mental health research

• Confidence in identifying culturally important aspects of participatory mental health research

• Behavioral intent of learners in addressing key participatory research principles in their future work

• Measurable ability to use skills and knowledge acquired through the course by pre- and post-testing using real-life participatory research scenarios

• Preferences for future learning

• Retention of attitudinal changes and knowledge gains obtained from follow-up questionnaires at one month post-beta-testing

Stage 1: Initial Review of Content and Format

Through this initial review, professional and community consultants provided feedback regarding course content and organization; design, structural format, and depth and breadth of online course (both overall and per module); and design, format, visual appeal, and perceived value of community outreach materials. The seven professional consultants were doctoral level behavioral health professionals with particular experience working with underserved or marginalized communities, representing urban Milwaukee homeless, rural Alaska Native, and rural New Mexico indigenous populations. Four of the seven professional consultants have university affiliations and are regularly involved in teaching college-level research courses, and the others work in behavioral health services delivery. These consultants reviewed course materials twice over a 1-year period. During the first review, they evaluated drafts of materials (i.e., depth and organization of the course outline, appropriateness of learning objectives, accuracy and depth of content for five course modules, glossary, and references) and the Community Outreach Kit (i.e., appropriateness and scope of brochure content, and perceived value of the plan for additional outreach materials). During the second review, consultants evaluated the design of e-learning web platform pages for the course introduction, five instructional modules, glossary, resource materials, and plan for future course enhancements.

The first community consultant review was completed by a mental health consumer advocate for Native American communities. This individual evaluated for appropriateness and content, an early draft of the community brochure and an overview of items to be included in the Community Outreach Kit. The second community consultant review included this same consumer advocate as well as a psychologist who serves as a mental health consumer advocate within Alaska Native communities. In this review, the two consultants evaluated the scope, design, and content of the community outreach materials (i.e., brochure, slide presentation, recruitment ad, meeting announcement poster, and community mental health resources list).

Stage 2: Alpha-Testing for Content and Format Feasibility

This second evaluation stage involved professional and community advisors participating in a small feasibility and usability study of course materials by potential end-users. The professional consultants consisted of nine advisors: four university professors whose research focused on urban Milwaukee Latina/o, and rural Alaska Native populations; three behavioral health care program directors or coordinators who work with Midwestern rural, suburban older adult, and urban homeless populations; one mental health researcher working with Alaska Native groups; and one psychology doctoral student whose research focuses on substance abuse in homeless men. These consultants evaluated all aspects of the e-learning course (i.e., opening web pages, content pages, each of the five modules, course materials, outreach kit, references and resources, glossary, overall curriculum questions, site design, and future course enhancements) for design, organization, presentation, and depth and breadth of content. Also, professional advisors participated in one of two conference call discussions held to gather additional qualitative data about the course, outreach materials, and the e-learning center.

The community advisory group consisted of six community advisors: three consumers of mental health services seeking treatment for disability, depression, substance abuse, or homelessness; two advocates for indigenous peoples (one being a parent advocate whose work focused on FASD), and one rural community member and coordinator of rural health services. Community advisors evaluated the Community Outreach Kit by participating in a face-to-face semi-structured interview protocol that included general questions about the content and design of the brochure, slide presentation, recruitment ad, meeting announcement poster, and community resource list, as well as the overall perceived value of the outreach materials. The interview was audio-taped with participant permission and anonymously transcribed for data analysis purposes.

Stage 3: Beta-Testing of Outcome Measurements

This third and final evaluation stage involved professional and community advisors participating in a pilot study that addressed project outcomes and participant satisfaction with the course and community outreach materials. For the professional group, this evaluation focused on all aspects of the e-learning course, including knowledge acquisition and retention, course organization, perceived value of the glossary, case studies, and video reflections, overall design, and structural format of the web course, as well as behavior change intention as a result of taking the course. The beta test professional group consisted of 18 participants: five university professors with research foci on Alaska Native populations, behavioral health nursing, and rural mental health; eight behavioral health care program directors or coordinators working with Alaska Natives, children, older adults, rural, faith-based, urban homeless, and disabled populations; mental health student researchers working with Alaska Native communities, issues of substance abuse, and behavioral health nursing; and one research track medical resident from a rural Wisconsin community. These 18 advisors were randomly assigned to one of two groups of nine members each and were asked to: 1) complete a pre-test of general CBPR knowledge; 2) complete the online course including the five module assessments; 3) complete a general CBPR knowledge post-test and satisfaction survey; 4) attend and participate in a telephonic advisory panel; and 5) complete a two-month follow-up CBPR knowledge test and behavior change survey. For community consultants, two groups of beta test participants were formed. The first group evaluated the educational and presentation components of the Community Outreach Kit (brochure and slide presentation) regarding knowledge acquisition and retention, content, overall design, and perceived value and usefulness. This group consisted of two participants from Wisconsin and four participants from Alaska and included: mental health consumers of services for alcohol use, schizoaffective disorder, and bipolar disorder; family members of mental health consumers, and community advocates. During a face-to-face meeting, community advisors completed a CBPR general knowledge pre-test; reviewed the outreach kit materials; completed a CBPR knowledge post-test; and participated in a key informant interview. The interview was audiotaped with participant permission and anonymously transcribed for data analysis purposes. To evaluate the durability of the educational intervention and assess knowledge retention, CBPR knowledge follow-up testing was conducted one month after completion of the post-test.

The second community group consisted of five mental health consumers, family members of mental health services consumers, local mental health services directors, and community advocates from rural and urban areas of Wisconsin and Alaska. This group evaluated the online course and outreach kit in a manner similar to the professional advisors. During this stage, community advisors completed a CBPR knowledge pre-test; completed the online course including the five module assessments; completed a CBPR knowledge post-test and satisfaction survey; and completed a survey with general questions on the course, community materials, and behavior change intention.

Evaluation Findings

Stage 1: Initial Review of Content and Format

Professional Group – Course and Community Outreach Kit

Results from the first review of the paper course materials indicated that the module content was clear and organized, and that the course resources were valuable to the user. On a 9-point Likert scale, Module Four was rated highest at a mean of 8.8 (SD = 0.5) for organization, clarity, and depth of content. Module One was rated at 8.7 (SD = 0.6) for organization. Ratings of 8.5 were given to Module Two for relevance to the overall course (SD = 0.6) and Module Three for purpose, accuracy, depth of content (SD = 0.6). The lowest rating of 6.8 (SD = 1.3) was found for depth of content in Module One. Professional advisors were unanimous in their support (M = 9.0, SD = 0.0) of CBPR and recognized the overall value of the Community Outreach Kit (M = 8.3, SD = 1.2), as well as its ability to improve community members’ understanding of the community-based research approach (M = 8.3, SD = 1.2). Results from the second review by professional advisors revealed the highest mean rating of 8.8 (SD = 0.5) was for Module 4 clarity of purpose. Next highest ratings of 8.5 were given to: Module 1 for clarity of purpose and relevance to overall course (SD = 0.6 for both), Module 2 for clarity of purpose and addressing Learning Objective 2 (i.e., “Recognize the importance of community input in effectively planning and carrying out an appropriate participatory research study”) (SD = 0.6 for both), Module 3 for clear and concise language use (SD = 0.6), and Module 5 for purpose (SD = 0.6) and clarity of language (SD = 1.0). Lowest ratings were given to the efficiency of the e-learning center login process (M = 6.6, SD = 0.8) and benefit and technical quality of video reflections contained within each module (M = 5.8, SD = 4.0). The overall quality of the course was rated 8.3 (SD = 0.5). Regarding the outreach kit materials, professional advisors gave high ratings of 8.3 (SD = 0.5) to the Facilitator Guide, Community Resource List, and the accuracy of glossary definitions

Community Group – Community Outreach Kit

Results of the first review indicated that the Community Outreach Kit would be a useful tool for educating and engaging community members in the CBPR process. The community consultant rated all elements of the outreach kit at 8.0, except for a rating of 5.0 given to helpfulness of the stationery as a multipurpose tool for researchers. In the second review, the overall value of the Community Outreach Kit received a mean rating of 8.0 (SD = 1.4) from community advisors. The community brochure received ratings of 8.5 (SD = 1.4) for organization of content, and the ability to educate community members about community-based research, and the rights of research participants. Similar ratings were given to the community slide show which corresponds with the content of the brochure. The lowest rating of 7.0 (SD = 2.8) was given to the visual appeal of the recruitment ad.

Stage 2: Alpha-Testing for Content and Format Feasibility

Professional Group – Course and Community Outreach Kit

Table 3 provides means and standard deviations for a group of content-specific survey questions, as well as ratings for each module across all survey topics and specific topics across modules. Mean ratings across all topics for the five course modules ranged from 7.5 (Module 4) to 7.8 (Modules 2, 3, and 5). Highest ratings for specific topics across modules were 8.4 (SD = 0.6) for module topics’ relevance to the overall course, and 8.1 (SD = 0.8) for the relevance of questions addressed within the module content. Lowest rating of 7.0 (SD = 0.9) was found in depth of content across modules.

Table 3.

Professional Group Alpha Test: Evaluation of Course Modules by Topic (N=9)

Survey Topic	Module 1		Module 2		Module 3		Module 4		Module 5		Topic Ratings across Modules
	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD
This module is organized and logical1	7.0	0.8	7.7	0.7	7.7	1.1	7.7	0.7	7.9	0.6	7.6	0.8
The purpose of this module is clear1	8.1	0.6	8.1	0.6	8.2	0.7	7.7	1.0	8.1	0.6	8.0	0.7
This module’s topics are relevant to the overall course1	8.0	0.9	8.6	0.5	8.8	0.4	8.6	0.5	8.2	0.7	8.4	0.6
Rating of the module on2:
▪ Accuracy of content	7.3	0.7	7.8	0.4	7.6	0.7	7.7	0.7	7.8	0.4	7.6	0.6
▪ Clear and concise language	7.8	0.8	7.6	0.9	7.7	0.7	7.4	0.5	7.8	0.7	7.7	0.7
▪ Depth of content	7.2	1.0	7.3	1.1	7.2	0.8	6.4	1.0	7.1	0.8	7.0	0.9
▪ Appropriateness of test   complexity for audience	7.8	0.8	7.3	0.7	7.8	0.8	7.7	0.7	7.8	0.4	7.7	0.7
▪ Relevance of questions addressed	7.9	0.9	8.1	0.8	8.3	0.7	8.2	0.7	7.9	1.1	8.1	0.8
How well module content addressed learning objectives2
▪ Learning Objective 1	7.1	0.9	7.1	1.2	7.4	1.5	7.3	0.9	7.7	0.7	7.3	1.0
▪ Learning Objective 2	8.0	0.9	8.6	0.5	7.6	1.6	7.0	0.9	7.3	0.9	7.7	1.0
▪ Learning Objective 3	7.9	0.9	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a
Module ratings across topics	7.6	0.8	7.8	0.7	7.8	0.9	7.5	0.8	7.8	0.7

¹Rated on a 9-point scale from 1 = “strongly disagree” to 9 = “strongly agree”

²Rated on a 9-point scale from 1 = “very poor” to 9 = “excellent”

Community Group – Community Outreach Kit

Alpha test scores from community advisors showed highest mean ratings of 8.0 for the brochure organization (SD = 0.7) and its ability to educate research participants about their rights (SD = 1.2), the value of the meeting announcement poster (SD = 1.0), the value of the community resources list (SD = 1.0), and the perceived positive effect of the kit on improving community members’ understanding of the community-based research approach (SD = 0.7). The lowest ratings were given to the slide show regarding its visual appeal (M = 6.2, SD = 1.6), ability to communicate to a variety of learning levels (M = 6.4, SD = 1.5), and educating community members about the benefits of taking part in mental health research (M = 6.4, SD = 1.7).

Stage 3: Beta-Testing of Outcome Measurements

Professional Group – Course Knowledge Assessment

Table 4 provides average percentage correct for each of the five module assessments and the course overall, broken down by the two groups of professional advisors. To determine whether professional advisors showed a gain in knowledge about participatory mental health research, tests of binomial proportions were performed. For both groups, significant increases in knowledge were shown from pre-test to post-test for all modules and overall. From pre-test to follow-up, significant increases in knowledge were revealed for Modules 1, 2, 3, and 4, as well as for overall knowledge. As anticipated, the professional advisors showed significant gains in knowledge about participatory mental health research after completing the course and this gain was sustained over time. However, not unexpectedly there was a minor decrease in knowledge from post-test to one-month follow-up test, with 9 of 12 scores collected showing decreases in knowledge. Nonetheless, the results are encouraging as the follow-up level of knowledge was found to be significantly higher than the pre-test level of knowledge.

Table 4.

Professional Group Beta Test: Course Knowledge Assessment Results (N=18)

Module	Average Knowledge Scores			Difference Scores
	Pre- Test	Post- Test	Follow-Up	Pre to Post	Post to Follow- Up	Pre to Follow- Up
GROUP A (N=9)
Module 1: Background Preparation for the Participatory Research Process	76%	93%	93%	18%*	0.0%	18%*
Module 2: Beginning Partnerships with Communities	76%	96%	100.0%	20.0%**	4%	25%***
Module 3: Finding Common Ground Within the Partnership	58%	91%	73%	33%***	−18%	16%
Module 4: Sustaining the Partnership	60%	93%	78%	33%***	−16%	18%*
Module 5: Evaluating the Partnership and Enhancing Future Successes	69%	93%	82%	24%**	−11%	13%
Overall Knowledge Score	68%	93%	85%	26%***	−8%	18%**
GROUP B (N=9)
Module 1: Background Preparation for the Participatory Research Process	64%	93%	96%	29%***	2%	31%***
Module 2: Beginning Partnerships with Communities	53%	93%	82%	40%***	−11%	29%**
Module 3: Finding Common Ground Within the Partnership	62%	96%	89%	33%***	−7%	27%**
Module 4: Sustaining the Partnership	64%	96%	84%	31%***	−11%	20%*
Module 5: Evaluating the Partnership and Enhancing Future Successes	71%	91%	78%	20%*	−13%	7%
Overall Knowledge Score	63%	94%	86%	31%***	−8%	23%***

^*significant at p < .05

^**significant at p < .01

^***significant at p < .001

Professional Group – Course Satisfaction Survey

Table 5 provides means and standard deviations for responses to a nine-item online satisfaction survey administered to two groups of professional advisors upon completion of the e-learning course. Results showed that, overall 89% of the professional group would recommend the course to a colleague, with specific ratings of 100% from Group A and 78% from Group B. In each group, 78% of advisors agreed that the course succeeded in meeting stated learning objectives. On a 5-point scale, advisors in Groups A and B rated the course content at means of 4.3 (SD = 0.5) and 4.1 (SD = 0.6), and the course overall at 4.2 (SD = 0.4) and 4.1 (SD = 0.6) respectively.

Table 5.

Professional Group Beta Test: Course Satisfaction Survey Results (N=18)

Survey Topic	GROUP A (N=9)		GROUP B (N=9)
The Community-Based Participatory Research for Improved Mental Health course:	78% met stated learning objectives 55% is relevant to my practice of research 67% was free of commercial bias 55% will improve my ability to provide excellent research services .		78% met stated learning objectives 67% is relevant to my practice of research 44% was free of commercial bias 44% will improve my ability to provide excellent research services.
Overall, this e-learning course was _____ .	Mean1= 4.2	SD= 0.4	Mean1= 4.1	SD= 0.6
The course content was _____ .	Mean1= 4.3	SD= 0.5	Mean1= 4.1	SD= 0.6
The learning environment was _____ .	Mean1= 4.0	SD= 0.5	Mean1= 4.2	SD= 0.4
The audiovisual components were _____ .	Mean1= 3.9	SD= 1.0	Mean1= 3.9	SD= 1.0
How long did it take you to complete this course (content and assessments)?	Average = 3.9 hours		Average = 4.3 hours
How likely would you be to recommend this course to a colleague?	67% Would definitely recommend 33% Would probably recommend		45% Would definitely recommend 33% Would probably recommend

¹Rated on a 5-point scale from 1 = “poor” to 5 = “excellent

Professional Group – Behavior Change

Table 6 provides means and standard deviations for professional advisors’ responses to behavior change questions (four items for each group of advisors) on a follow-up survey administered two months after course completion. Results for the first group, using a 9-point Likert scale, showed highest mean ratings of 8.7 (SD = 0.5) for seeing the value of understanding a community’s perspective on mental health concerns, and 8.5 (SD = 0.8) for the likelihood of promoting a collaborative approach to research. For the second group, highest mean ratings of 7.4 (SD = 1.0) revealed that advisors felt better prepared to build collaborative relationships with different groups of people and communities. High ratings (M = 7.0, SD = 1.6) showed that evaluation participants had shared their knowledge of CBPR with colleagues since taking the online course.

Table 6.

Professional Group Beta Test: Two-Month Follow-Up Survey Results (N=13)

Survey Topic	Mean1	SD
GROUP A (N=6)
After taking this course, my overall beliefs about mental health research have changed.	6.7	1.0
After taking this course, I see the value of understanding a community perspective on concerns related to mental health.	8.7	0.5
After taking this course, I am more likely to promote a collaborative approach to research.	8.5	0.8
After taking this course, I am more likely to join a community-based research team.	7.2	1.5
GROUP B (N=7)
After taking this course, I am better prepared to build collaborative relationships with different groups of people and communities.	7.4	1.0
After taking this course, I am more likely to use a community-based participatory research approach in mental health research	6.9	1.2
After taking this course, I have incorporated my new knowledge of CBPR principles into my current work.	6.4	1.8
After taking this course, I have shared my knowledge of CBPR with colleagues.	7.0	1.6

¹Rated on a 9-point scale from 1 = “strongly disagree” to 9 = “strongly agree”

Community Group – Course Knowledge Assessment

Table 7 provides average percentage correct for each of the five module assessments and overall for the community advisors. To determine whether community advisors showed a gain in knowledge about participatory mental health research, tests of binomial proportions were performed. From pre-test to post-test, significant increases in knowledge were revealed for Modules 1, 2, 4 and 5. Overall, community advisors showed significant gains in knowledge about participatory mental health research as evidenced by an overall knowledge gain of 38%.

Table 7.

Community Group Beta Test: Course Knowledge Assessment Results (N=5)

	Average Knowledge Scores		Difference Score
Module	Pre-test	Post-test	Pre to Post
Module 1: Background Preparation for the Participatory Research Process	60%	96%	36%**
Module 2: Beginning Partnerships with Communities	52%	96%	44%**
Module 3: Finding Common Ground Within the Partnership	68%	92%	24%
Module 4: Sustaining the Partnership	46%	92%	46%***
Module 5: Evaluating the Partnership and Enhancing Future Successes	50%	96%	46%***
Overall Knowledge Score	55%	93%	38%***

^*significant at p < .05

^**significant at p < .01

^***significant at p < .001

Community Group – Course Satisfaction Survey

Upon completion of the course, community advisors were administered the same online nine-item satisfaction survey as the professional group. Results showed that of the five advisors who completed the survey, all would recommend the course to a colleague. On a 5-point scale, respondents rated the course overall at mean of 4.4 (SD = 0.9), the course content at 4.2 (SD = 0.8), and the learning environment at 4.8 (SD = 0.5).

Community Group – General Course Survey

Table 8 provides means and standard deviations for responses to six general questions presented to the community advisors after completing the course and assessments. On a 9-point scale, community advisors unanimously agreed on their support of the value of understanding a community perspective on mental health concerns (M = 9.0, SD = 0.0). Advisors also reported a high likelihood that they would join a community-based research study (M = 8.6, SD = 0.9).

Table 8.

Community Group Beta Test: General Course Survey Results (N=5)

Survey Topic	Mean1	SD
After taking this course, my overall beliefs about mental health research have changed.	8.2	0.8
After taking this course, I see the value of understanding a community perspective on concerns related to mental health.	9.0	0.0
After taking this course, I am better prepared to build collaborative relationships with different groups of people and communities.	8.0	1.4
After taking this course, I am more likely to join a community-based research study (if an appropriate one becomes available in my community).	8.6	0.9
After taking this course, I will incorporate my new knowledge of CBPR principles into my mental health advocacy efforts.	7.8	1.3
After taking this course, I will share my knowledge of CBPR with colleagues.	8.2	1.1

¹Rated on a 9-point scale from 1 = “strongly disagree” to 9 = “strongly agree”

Community Group – Community Outreach Kit Knowledge Assessment

The community advisors completed a 10-item multiple choice knowledge pre-test prior to beginning the Community Outreach Kit review, a post-test after reviewing the brochure and participating in the slide presentation, and a follow-up test one month after the post-test. Results revealed that participants’ knowledge at pre-test was already quite high (88%) and did not increase significantly to post-test (88%) or follow-up (90%).

Discussion

Community-based participatory research (CBPR) is a rapidly growing approach to addressing community needs through research. Although conducting such research in an ethical manner is critical to the long-term development of community-researcher partnerships, few academic programs include this information in their curricula. This article describes the development and refinement of an online training course and companion community engagement materials designed to meet the need of training researchers to implement CBPR in an ethical and productive manner. The online course with printable educational support and companion community materials was designed to instruct investigators, research team members, and students learning about mental health research on the essential principles and practices for successfully and ethically conducting participatory mental health research projects in real-world settings. The course includes an emphasis on forming respectful research partnerships with traditionally underserved communities and on practical ethical decision-making within a participatory mental health research model. Furthermore, the course is novel in its incorporation of ethical, cultural, and mental health issues within a participatory research framework. The human element in research, which is at the heart of the participatory model, is the essential cornerstone of the course as demonstrated by quotes and video reflections of real-world researchers available in each course module.

Through an evaluation process that mirrors CBPR, we used qualitative and quantitative strategies with key stakeholders to develop and refine the course and Community Outreach Kit to insure that they are presented in a manner that is met with high satisfaction, increases learners’ knowledge of CBPR, and positively affects learners’ attitudes about CBPR and its value. Evaluation findings indicated that the course and ancillary materials increased reviewers’ knowledge of, and positively affected attitudes about CBPR. Perhaps most importantly, after reviewing these materials, individuals were more willing to participate themselves in a community-based research project.

With increasing movement in the mental health field toward taking research initiatives directly into communities and having community members become equal partners in the research endeavor, this course and companion materials provide an important new avenue for the training of researchers. This e-learning course and accompanying materials are expected to be successful in instructing investigators, research team members, and students learning about mental health research on the essential principles and practices for conducting respectful and ethical participatory mental health research projects in real-world settings. This program has also been shown effective for many community members, particularly those in leadership and advocacy roles. Primary emphasis is placed on forming productive research partnerships with underserved populations and encouraging practical, ethical decision-making within a participatory mental health research model. The human element in research, which is at the heart of the participatory model, has come across as an essential cornerstone. This is best illustrated in this respondent’s summation of the course, “I was overjoyed to see the amount of effort put into (developing a training) that addressed the importance of creating a relationship with the community, and really, really respecting the social construction of the community and not bypassing any of the important people that make the community work or serve as informants.”

It is hoped that the use of this course will inspire more mental health researchers to consider taking their research out into the communities that they intend to help. With each new CBPR study, another community will benefit from the valuable information gathered within the unique, specific context of that community. As CBPR gains in acceptance and more researchers choose to work with communities, mental health services delivery will improve to all people, especially those who have been underserved in the past.

Future Directions

By developing and testing the Community-Based Participatory Research for Improved Mental Health course, we believe that both students and professionals who take the course will gain a deeper understanding of the unique nature of community-based research. Going forward, we will monitor the reactions and behavior change likeliness of all learners through the course’s online satisfaction survey, which is administered to all users. We will continue to improve and enhance the course as warranted based on this feedback and advancements in the field of CBPR.

Although this course provides a foundation in critical basic CBPR training, additional work and training needs to be done to more fully understand the unique complexities and ethical issues surrounding its application with unique and special populations (e.g., children, institutionalized adults, persons with significant developmental disabilities, prisoners, those with very stigmatizing conditions such as AIDS, or runaway youth). Further, this course was developed with a mental health focus, however some of our project advisors believed that CBPR training should not be limited to mental health issues and that community-researcher collaborations would enhance the quality of research outcomes in other health-related disciplines.

Conclusion

This course is meant to help foster and guide new forms of collaborative research relationships between researchers and participants by offering skills needed to view research as a cooperative, collaborative process in which influence and power are shared, and the context of the community guides the way. It suggests a methodology that prompts researchers to think about how, why, and for whom research is conceptualized and conducted, and how corresponding power might be allocated at each step of the research process. Our hope is that users of the course will begin to think about research in this new and different way. As more behavioral health researchers choose to work with communities, mental health services delivery will improve to all people, especially those who have been underserved in the past.