Increasing Community Research Capacity to Address Health Disparities: A Qualitative Program Evaluation of the Community Research Fellows Training Program

Introduction

Community-based participatory research (CBPR) is an approach to research that actively and equitably engages community stakeholders in all aspects of the research process (Israel, Schulz, Parker, & Becker, 1998; M. Minkler & Wallerstein, 2008). Engaging stakeholders across the research continuum (planning, implementation, evaluation, and dissemination) provides opportunities for community members to play an active role in improving the health of their community, thus promoting mutually beneficial academic-community partnerships for addressing health disparities (Jones & Wells, 2007; Ross et al., 2010; N. B. Wallerstein & Duran, 2006). The benefits of CBPR have been well documented (Leung, Yen, & Minkler, 2004; Meredith Minkler, 2005; O’Fallon & Dearry, 2002; Viswanathan et al., 2004; N. Wallerstein & Duran, 2010). In addition to creating novel partnerships, CBPR can be used as a vehicle for community members to increase local capacity and advocate for their communities (Johnson, Williams, & Gillis, 2015; Macaulay et al., 1999). In order to develop equitable and mutually beneficial partnerships, community stakeholders need research literacy (basic knowledge of research methods, study design and research terminology) to increase organizational capacity to participate in research. Programs designed to enhance the infrastructure for CBPR by increasing community stakeholder research literacy require evaluation for refinement and broader dissemination.

In 2013, the Division of Public Health Sciences at Washington University School of Medicine (WUSM) and the Program to Eliminate Cancer Disparities (PECaD) at the Siteman Cancer Center (SCC) began the Community Research Fellows Training (CRFT) program in the St. Louis metropolitan region. The CRFT program expanded the goals of a similar training program that had been implemented on Long Island, New York (M S Goodman et al., 2014; M S Goodman, Si, Stafford, Obasohan, & McHunguzi, 2012; Melody S. Goodman, Dias, & Stafford, 2010). The CRFT program was designed to (1) promote partnerships between community members and academic researchers, (2) enhance community members understanding of how to use research to improve health outcomes in their communities, and (3) train community members to become critical consumers of research (Coats, Stafford, Thompson Sanders, Johnson Javois, & Goodman, 2015; D’Agostino McGowan, Stafford, Thompson, Johnson, & Goodman, 2015). Using the core elements of this previous program curricula, CRFT was adapted to be culturally appropriate and region specific. In each geographic location, the examples used in the lectures and group activities were tailored to the racial ethnic diversity of the region and used region specific neighborhood names, supermarket chains, parks, universities and hospitals. Data presented on racial disparities and other health topics were based on local, regional and state statistics which were then compared to national averages to provide additional context.

The intended audience for the CRFT program were individuals who currently worked in community health and/or had a desire to improve public health in their community (for a full description of the recruitment and selection of CRFT fellows, see Coats et al., 2015). The CRFT program included a total of 15 weekly three-hour training sessions, and covered key public health topics (health disparities, community health, research methods, health literacy, cultural competency, epidemiology, quantitative methods, CBPR, research ethics, qualitative methods, clinical trials, health policy research, research with human subjects) (Coats et al., 2015). Through 2015, three cohorts of the CRFT program had been implemented¹. Classes were held on the WUSM campus and CRFT program faculty members included a multidisciplinary group of community leaders and academics from Washington University in St. Louis and Saint Louis University. There were 17 faculty members who taught in cohort I and 17 faculty members who taught in cohort II of the program; 14 (70%) faculty taught in both cohorts. Nearly half (47%) of the presenters in cohort I were African American, while 41% of cohort II presenters were African American. Prior to the first class, program participants (hereafter referred to as “fellows”) participated in an orientation session that presented an overview of the CRFT program: rationale, session topics, goals and expectations. To reduce barriers to participation, fellows were provided parking validation or public transportation vouchers and meals were served at the beginning of each session. At the end of the 15-week training, a certificate ceremony was held to celebrate the accomplishments of CRFT fellows. A total of 83 fellows from the St. Louis metropolitan region have graduated from the first two cohorts of the CRFT program; 45 fellows completed cohort I and 38 fellows completed cohort II.

The CRFT program aimed to implement culturally appropriate ways to increase research literacy among community members (D’Agostino McGowan et al., 2015; Melody S. Goodman et al., 2010). Culturally appropriate research training can provide fellows with the skills needed to critically examine and assess health disparities that may exist in their communities (Brach & Fraser, 2000; Gehlert & Coleman, 2010). Although numerous organizations have implemented research training programs for community members, a limited number of studies have comprehensively evaluated the effectiveness of community training programs in public health methods (D’Agostino McGowan et al., 2015; Kwon et al., 2012). Quantitative analyses of knowledge gained by CRFT program fellows indicated that there were significant increases in pre- versus post-test scores in several MPH curriculum topic areas (D’Agostino McGowan et al., 2015). In addition to increasing fellows’ understanding of public health research methods, the training program was beneficial to CRFT program fellows and CRFT program faculty as both groups rated training sessions very highly (D’Agostino McGowan et al., 2015). Comprised of a comprehensive mixed-method (quantitative and qualitative) evaluation of the CRFT program, we supplemented our earlier quantitative analyses of cohort I (D’Agostino McGowan et al., 2015) with a qualitative analysis of the first two cohorts of the CRFT program. Here we discuss findings from the qualitative component of the larger program evaluation.

Method

CRFT Fellows

A total of 50 fellows were accepted into cohort I, with 45 graduating (90% completion rate) from the program, while 39 fellows were accepted into cohort II, with 38 graduating (97% completion rate). In total, we conducted evaluation interviews with 44 cohort I fellows and 37 cohort II fellows; a majority (n=81; 91%) of the 89 cohort I and II fellows including 3 of the 6 fellows who did not complete the program. The majority of the CRFT program fellows were female (83%), with a mean age of 52 (±SD 11.5). Approximately 87% (n = 72) of fellows were African American and 12% were Caucasian (n = 10). Nearly half of the fellows had a graduate degree (46%), and 31% had a 2-year degree or some college coursework. Approximately 27% of the fellows were employees of a community-based organization and 20% self-identified as a community member with no organizational affiliation. More than half of the fellows reported previous research experience (48% conducting research, 53% participating in research). The fellows lived in 45 unique zip codes across the St. Louis metropolitan region. The training program was classified as program evaluation by the Human Research Protection Office at WUSM and thus not subject to IRB review.

Interview Procedures and Data Analysis

After the conclusion of the program, we recruited all fellows, regardless of whether they had completed the program, to participate in a semi-structured, in-person interview. Written informed consent was obtained prior to beginning the evaluation interview. Telephone interviews were conducted instead if fellows had moved out of the St. Louis area (n=1), scheduled appointments for in-person interviews were rescheduled by the fellow 2–3 times (n=4), or fellows requested a phone interview (n=1), and verbal consent was obtained. We did not note any differences between the in-person or telephone interviews in our results. The interview guide was developed to examine fellows’ experiences in the CRFT program, including overall perceptions and more specific questions on the following program elements: program length and structure, session topics, faculty, group activities, homework, class size, course material, utility of training, program barriers, and program logistics. All evaluation interviews were digitally recorded and transcribed verbatim.

We conducted a thematic analysis of interview data (Hsieh & Shannon, 2005). CRFT goals and quantitative data on the program informed the interview guide, which was used to develop the initial thematic domains and a preliminary codebook. The codebook was revised by the research team through an iterative process, where we added inductively derived codes and themes. The codes were entered into NVivo 10 (QSR International, 2012) by one PhD level member of the research team. We organized codes into overall themes, first, and then examined the most frequent codes within those overall themes.

Results

Results are presented from the qualitative exit interviews to demonstrate how the CRFT program was effective in achieving its main program goals. The results are organized in two sections. First, the various program components that contributed to fellows’ learning experiences are examined. In the second section, how the CRFT program overall led to an increased understanding of health research is demonstrated. Overall, these results highlight how the CRFT program provided a conducive learning environment and structure for community members to increase their research literacy and become more critical consumers of research.

Assessment of Learning Environment

To assess whether the program was effective in achieving program goals, we evaluated various program components specific to the CRFT training. There were five programmatic components that promoted a conducive learning environment for fellows: program faculty, educational and professional diversity of program fellows, small group activities, homework assignments, and logistical arrangements. Table 1 displays additional quotes that are illustrative of the program components that enhanced fellows learning in the program and support our previous quantitative analyses (D’Agostino McGowan et al., 2015).

Table 1.

Assessment of Learning Environment: Additional Quotes from Exit Interviews

Code	Representative quotes
Program Logistics
Location of training	“I liked the location. I liked that it was … held in the facility where you guys actually work and stuff. It made us feel more part of that than if you would held it somewhere neutral.” [Cohort I, Female, 58-years-old]
Provision of meals and transportation	“The food was good. There was an attempt to do healthy food. That’s always a plus because trying to get over here after work or after meeting or whatever, during that evening hour, we didn’t have to worry about trying to find dinner. You're eating on the run. So that was good to have that...the location is great. The parking is good. We didn’t have to walk very far from the parking lot to the building. It was a safe area.” [Cohort I, Female, 57-years-old]
Length/hours of training	“I think it was a good length of time for once a week. I think it was good. Any less and we wouldn’t have gotten what we needed to get to move forward outside of the class.” [Cohort I, Female, 47-years-old]
CRFT Faculty	“I was very impressed and happy that it was so many African American lecturers…it was very diverse I mean, and it was just excellent.” [Cohort II, Female, 55-years-old]
	“I just thought that the caliber of the instructors was just excellent…I had never seen such a talented group of African Americans in healthcare to take a lead on… it made me feel so good…I don’t have the words for it…because a lot of times you don’t see that, and you don’t see African Americans of that caliber taking the kind of interest that that group has taken in our community. And it was just wonderful.” [Cohort I, Female, 65-years-old]
Participant Diversity	“I think it was excellent because, again, there were a variety of people from different disciplines, educational levels, men and women, people who had no experience in taking college courses and people who had been to graduate school. I thought that represented the core of community based research, that you had people from different backgrounds. [Cohort I, Female, 64-years-old]
Break-Out Groups
Small group learning environment	“Well I was a lot more involved too in the breakout groups…I felt like I could say more and be more involved in the small groups. So that was good.” [Cohort II, Female, 25-years-old]
Interaction with other Fellows	“It gave me an opportunity to interact with more people, different people who didn’t have my beliefs. So things like that. So it gave me an opportunity to actually interact with a lot of different people from different backgrounds in the community.” [Cohort I, Female, 65-years-old]
Homework Assignments
Application of course material	“The homework assignments on the surface seemed easy, but then the lesson that we learned from them was more than that …it would seem like a little simple task, but just the impact of the lessons was awesome.” [Cohort II, female, 56-years-old]
Community focus	“No, I thought they were really good. It actually got us to think about what it was that we were going over and getting out into the community to do some great things.” [Cohort II, Male, 41-years-old]
	“One of the things that I really enjoyed was the PhotoVoice and the Windshield Survey. I was able to look at the community and the environment in a different lens so to speak. It’s just not driving through a community but being able to look at key factors and key things that help with the disparities and with social inequalities in regards to access to health.” [Cohort I, Female, 39-years-old]

Program faculty

We found the background of faculty who participated in the CRFT program to be an important factor in engaging community members in public health research topics for two reasons. The first is that faculty members were recruited who had experience working directly with community members and/or have participated in CBPR and were well-equipped to engage fellows during lectures and group activities. One fellow explained:

I liked the fact that the professors that came here were very engaging. It wasn't necessarily intimidating. They were very welcoming. Sometimes, you just don't know, especially academic folks, they're so versed and so knowledgeable in what they do, sometimes you're afraid to ask questions because you think it's a stupid question, or the wrong question. But they never gave me the impression that there was a stupid question.

[Cohort I, Female, 41-years-old]

Fellows remarked how the CRFT faculty created a “welcoming” and “learner friendly” environment that made fellows feel comfortable to ask questions and thus be more actively engaged in classroom discussions. This was particularly important for fellows who either did not have an advanced degree or who had not attended a university in many years. For example, another fellow said:

I was very impressed with the level of knowledge of the staff and the presenters and how they made things learner friendly. You didn’t have to have a PhD or a Master’s degree to understand and grasp their knowledge and the way they presented it.

[Cohort I, Female, 58-years old]

Fellows described how the CRFT faculty’s presentation style and ability to translate technical research terminology and concepts into laymen’s terms enhanced their learning experiences. One fellow [cohort II, female, 64-years-old] said of the presenters: “What I liked most about it was each instructor and how they took their time and imparted their information to us in a way that we could understand it as community people.” Fellows repeatedly remarked how the CRFT faculty engaged community members by explaining public health research terminology and theoretical concepts in terms that were understandable to them. Another fellow [cohort II, female, 59-years-old] explained: “I really thought the lectures were superb. They were excellent in presenting the information. I do think they took into consideration that a lot of us didn’t have exposure to certain terminology.”

The second background characteristic of CRFT faculty that had a positive impact on fellows’ experience in the program was the racial diversity of presenters, mainly that nearly half of CRFT faculty were African American. One fellow [cohort II, female, 55-years-old] commented: “I enjoyed the presenters…I really like that they were young African Americans, that just really was inspiration[al].” Another fellow remarked on the powerful experience of being taught by African American scholars:

It is always a joy to see African Americans who know what they’re talking about and can teach us what we need to know. You feel comfortable and relaxed knowing that these are people who you feel some akin to—even if it’s only by race—to teach you about research.

[Cohort I, Female, 65-years-old]

Given that the majority of fellows were African American, the presence of African American faculty in the program was something they found to be “inspirational” and cited in interviews as a motivational factor to participate in community health initiatives. The faculty-fellow interaction was mutually beneficial as CRFT faculty also reported they learned from the fellows, would teach in the program again, and would be willing to collaborate with fellows on CBPR pilot projects (Coats et al., 2015). Matching racial or other demographic variables of faculty and fellows is important to consider when trying to increase community capacity for CBPR in a specific locale.

Educational and professional diversity of program fellows

In addition to learning from CRFT presenters, fellows described gaining new perspectives on public health issues because of ideas or opinions presented by their peers. CRFT fellows’ professional affiliations included a range with some working in community-based organizations, health care, local government, faith-based organizations, academic institutions, as well as people who were retired or out of the workforce. Additionally, though the majority of fellows had attended at least some college, there was a wide array of health and non-health related disciplines represented. Fellows remarked how the educational and professional diversity among the group contributed to their learning. One fellow [cohort I, female, 45 years old] described how she learned from others in the program, “the interaction and the exchange that everybody was giving. And then to get the insight into other people’s ideas on what’s presented to them…I learned something new.” The dynamic communication between fellows was an important part of the CRFT program experience. Another fellow reflected on the importance to her that fellows had diverse backgrounds:

One of the biggest things that I liked was the cohesiveness of people from all types of different backgrounds and walks of life coming together discussing public health and health issues. That was the most exciting part to me; other people from different disciplines and backgrounds.

[Cohort I, Female, 39-years-old]

Fellows perceived the group’s diverse educational and professional backgrounds as an asset that contributed to their experience in the program. By admitting fellows from health and non-health related backgrounds, various educational levels, and different professions, fellows gained additional perspectives that enhanced the content taught in the program. Additionally, the social connections made during their participation in the CRFT program sometimes extended past the completion of the program where fellows in similar professional fields collaborated on projects that aimed to improve health outcomes. Perhaps the best example of how fellows bonded and become more active in improving health in their communities was the formation of a Patient Research Advisory Board (PRAB). Born out of their discussions in the CRFT sessions on Research Ethics and Human Subjects, cohort I fellows came together to establish the PRAB—a community research review board that advocates for community health concerns and projects with community benefit. The group continues to meet monthly and consists of a diverse mix of alumni across cohorts (discussed later in more detail in the Discussion section).

Small group activities

A majority of the sessions incorporated some form of break-out group activity; fellows were randomly assigned to groups of 4–7 people where they were asked to work together on an activity and report back to the larger group. Many of the fellows explained how effective these small group activities were because it gave them an opportunity to apply lecture topics and principles to real-world scenarios. For example, one fellow explains:

I loved the breakout groups because I like application and I like actually walk[ing] through an exercise, using what we were learning at the time. So, that’s very good for me because I am a hands-on kind of person.

[Cohort I, Female, 47-years-old]

The small group activities were described as useful exercises that provided “hands-on” application of concepts that were presented in the lecture. The break-out group activities were particularly beneficial for fellows who preferred small group learning environments to a larger lecture style setting. For example, one fellow explains:

I like the break out groups because, as I said, for me I’m not one to talk in large groups a lot. If I have a question I may ask it or if I have the answer to something I may or may not say it. But in the smaller groups it’s not as threatening for somebody who’s kind of quiet and shy in some regards, and that’s me. So I appreciated the break out groups because I talked more...

[Cohort II, Female, 39-years-old]

Depending on one’s learning style, the small group activities provided another format to absorb session materials that the majority of fellows found particularly useful. The small group activities allotted designated time where fellows could exchange ideas on how they might use research to improve health outcomes in their communities. A fellow who worked as a nurse explained the benefits of the break-out groups:

It was very refreshing, very honest. Participation from the people in the groups was just like invaluable. The breakout sessions were wonderful. Because usually when I go somewhere and do something, everybody that’s in the room is in the same field that I am in. They are either a nurse or in nursing school, so it was very valuable to have people from just the community to talk about things and they all came from different perspectives.

[Cohort II, Female, 62-years-old]

In the small group activities, fellows were able to interact with one another in ways that the large lecture format did not allow. Similar to the feedback given about the diversity of fellows’ educational and professional backgrounds, the small group activities provided an avenue for a nurse, for example, to discuss how to improve health outcomes in St. Louis communities with someone outside her professional sphere.

While most fellows viewed the break-out group activities as helpful, not everyone found the group work to be effective. Some fellows found it difficult to reach consensus with the people in their group: “Well, group work is hard. And sometimes it can be frustrating to get everybody to agree on one thing. And it takes a lot of work. That was a little frustrating” [cohort I, female, 65-years-old]. For others, it could be challenging to focus on the task at hand when working with others. One fellow stated:

The only thing I did not like and it might have been just even subjective because …I liked the work…but the group activities sometimes just gave me a headache [laughs] …Although we might have been given a task, everybody did not understand it sometimes, and they were just kind of doing what they wanted to do, which, unfortunately, because I am a task oriented person, it would just kind of drive me crazy, because people were just going off on thirty thousand tangents.

[Cohort II, Female, 39-years-old]

The challenges fellows discussed are common in group-based work and provide meaningful opportunities to build the skills necessary for collaboration on research teams (e.g., consensus building, time and task management). We took this into consideration and in cohort II reduced the number of sessions with group work (from 11 to 9 sessions), revised group activities that did not work well, and refined directions for clarity. Also, due to the smaller size of cohort II, we were able to adjust the size of the break out groups to 4–5 people per group, which was described as more productive.

Homework assignments

Fellows were assigned five main homework assignments across two cohorts: a windshield survey, grocery store audit², community park audit, a photovoice project examining social capital, and a family health history, which were all independently conducted. With the exception of completing a family health history form which involved interviewing family members (the assignment was not collected, due to HIPAA privacy concerns), all homework assignments involved going to different St. Louis neighborhoods to record and analyze observations related to identifying health disparities. In general, fellows remarked how completing the homework assignments was a compelling way of applying concepts presented in the classroom. One fellow [cohort I, male, 49-years-old] said of the homework assignments: “They actually offered an opportunity for participants to put in to practice some of the things that we were exposed to, or learned, in the sessions.”

Fellows described how the homework assignments were effective exercises in helping them to identify and understand health disparities first-hand especially because a few of the assignments (e.g. grocery store audit, and park audit) required travel to two distinct socio-demographic neighborhoods. Fellows were encouraged to explore St. Louis and select parks or grocery stores they were less familiar with. One fellow described the eye-opening experience of conducting an audit of a grocery store she had never been to before:

The homework assignment that we had for the grocery stores, that was very enlightening, because previously I didn’t really focus that much on the differences in different communities. I knew there were differences but I didn’t study it as comprehensively as I would have if I had not had the homework assignment. And it angered me, many of the things that I saw angered me to the extent that I actually wrote a letter to the grocery store.

[Cohort II, Female, 63-years-old]

Since nearly all of the homework assignments included observation of neighborhood characteristics, public parks, and grocery stores across St. Louis neighborhoods, fellows commented on how the assignments made them more conscious of features of their own communities. One fellow explains about the windshield survey homework assignment:

It made me take a different look at my neighborhood… it made me look at things that I had never thought about like actually looking at people's houses and trying to looking at the state of the neighborhood, the trash in the neighborhood, the signs in the neighborhood, what was offered in the neighborhood, loved that.

[Cohort I, Female, 59-years-old]

By exploring various neighborhoods, fellows were able to take note of characteristics in their own neighborhoods that may contribute to disparities in health (e.g. presence of fast food restaurants, access to parks, etc.).

Despite having work and family obligations, fellows completed assignments at high rates (Coats et al., 2015). Fellows expressed that the assignments facilitated learning by directing them to apply public health research concepts in community settings and were particularly effective in helping them to identify health disparities and think about how they can improve health outcomes in their own communities. Some of the homework assignments were so popular among fellows that they developed them to use with colleagues in their own professional settings.

Program logistics

Retention in the CRFT program was high (Coats et al., 2015; D’Agostino McGowan et al., 2015). Fellows discussed how logistical arrangements facilitated their participation and ability to complete the 15-week program. The program logistics discussed by fellows ranged from the central location, validated parking and public transportation vouchers, provision of meals, once a week program schedule, three-hour evening sessions, access to Washington University facilities. One fellow observed: “The location was great. The building was great...You had bus transportation and all that, I liked it.” Similarly, another fellow was appreciative of the CRFT program’s efforts to accommodate fellows, remarking:

It was convenient for me because it was…centrally located. I liked that…you all paid for the parking. That helped a lot... I really appreciate the free parking, the food. It’s like you took care of the things that would keep people from being able to come.

[Cohort II, Female, 65-years-old]

Also, fellows thought that the once a week training scheduled worked well, especially for people working full-time. One fellow [cohort, female, 64-years-old] noted: “The fact that it only met there one day a week made it good for me, too, because a lot of different days would’ve proved difficult.” Many of the fellows worked or had other responsibilities during the day, and the one-evening-a-week training schedule was described as feasible. Moreover, fellows were excited that the program took place at WUSM. One fellow remarked on the on-campus training, saying:

I think it is great…You feel more a part of the university itself…Even though I know I am an outsider, I feel more a part and I think that makes me want to be more a part and do more things, reach out. I feel more involved.

[Cohort I, Male, 64-years-old]

Hosting community members in an academic setting was an important criteria for community members to feel welcome on other parts of the WUSM campus. Holding the program on the university campus exposed fellows to speaker series or events that are open to the public. A fellow working in government commented:

A lot of times you see Wash[ington] U[niversity], and you think it's either for the physicians or the professors, it's like will I get anything from this, or can I even go? Is this for me? So I liked the fact that you would give us flyers to upcoming events at the Institute of Public Health, and knowing that we were able to attend, and then if we heard something in the presentation, you can be like I know what they're talking about because I heard that in class.

[Cohort I, Female, 42-years-old]

The CRFT program being on the WUSM campus for 15-weeks influenced fellows’ perceptions of their potential roles in academic-community partnerships. This was evidenced by an increase in their participation in university-sponsored events, community advisory boards, and even the creation of a Patient Research Advisory Board (which we discuss in further detail in the discussion). One fellow, a member of a faith-based organization, explained how she became involved in the PECaD colorectal cancer community partnership group after the training ended:

I'm working now with another group that I guess my participation in the fellows pro[gram] opened up an avenue for me to work with the colorectal cancer group, and met some more new people. This whole process has helped me to really understand the need to have the community involved in healthcare initiatives, and to help educate the community, who does not always know the answers or how to navigate through the process in making decisions.

[Cohort I, Female, 59-years-old]

Participating in the CRFT program led to increased involvement in community-academic partnerships and an awareness of the importance of community input in health initiatives. Paying attention to program logistics can improve retention and promote partnerships between community members and academic researchers.

Increased Knowledge of Public Health Research

The quantitative evaluation of the CRFT program has shown how the training was successful in increasing fellows’ knowledge in public health topics by demonstrating improvement of scores from baseline to final assessments and mean increases in pre and post-test scores (D’Agostino McGowen et al. 2015). Our qualitative analysis of exit interviews also support these previous findings by demonstrating how the CRFT program developed fellows to become “critical consumers of research.” Fellows described how the CRFT program helped them better understand the various steps of the research process:

The total scope of the program was very enlightening. It allowed me an opportunity to engage in the areas of research for healthcare disparities, to learn more about community health services, and how to identify what is a problem, what problems need to be researched, how to get fellows to get engaged in the research process.

[Cohort II, Female, 38-years-old]

Even for fellows who had health-related college degrees or worked in a health profession, the CRFT program provided a refresher on information they had learned years ago or presented new research methods they found useful in their jobs. A fellow who works in a community-based organization as a grant writer described how she has been able to apply the information she learned in the CRFT program at work:

I even use it on the job. A lot of the information and the research methods that we were doing… I was able to apply a lot of research methods that I was learning through the training and apply it to real life and share it with everyone else throughout my organization. So that was a positive.

[Cohort I, Female, 39-years-old]

Fellows described how the information from the training gave them additional tools to contribute to work-related projects and strategic planning. A fellow from a community-based organization said:

I guess in terms of planning, because now, my agency, we're finding ourselves rethinking what we're doing and our direction, and thinking about the future and where we want to go. So some of the information that I learned in the sessions about being able to assess your strengths and your weaknesses and identifying your opportunities. That has been helpful.

[Cohort I, Male, 49-years-old]

The CRFT fellows were able to employ the information they learned in CRFT in their careers and volunteer work. Since many fellows were already working in public health careers, the additional knowledge they gained from participating in the CRFT program better enabled them to implement solutions to address health disparities on projects they were already involved in.

In addition to discussing how the CRFT program contributed to their professional occupations, fellows also described how the CRFT training enriched their personal lives by raising their awareness of public health-related issues. Two fellows described how the CRFT training has increased their consciousness in positive ways:

I've learned to think more clearly about topics that I had not even realized were public health topics, things like food…and the scarcity of services in neighborhood. The lack of access to medical care in many neighborhoods. I've learned to evaluate that and think more about that, and as I read in the newspaper what is going on, I feel better able to understand and to reflect on what I'm reading, not just locally but nationally and internationally.

[Cohort I, Female, 67-years-old]

It has enlightened me somewhat. Every time I turn on the TV, they are talking about the surveys and I had never really paid any attention before about the different surveys. I am sitting and wondering [what] type of surveys. I wonder what do they do with research and when do they use it. I think about that, I would have never even known about that before, where they get their stats for and that goes to my mind, how many people are in research, what age group and that’s what I think about. Now that the things that I have learned, I can still apply it to everyday living.

[Cohort II, Female, 68-years-old]

As the fellows described above, their participation in CRFT increased their awareness of what constitutes public health as well as how research is conducted. After the training, they described how they applied what they learned in the training to their daily activities like watching television or reading the newspaper. These examples highlight how the CRFT program increased consciousness and developed community members to become critical consumers of research. The exit interviews demonstrate how the CRFT program improved fellows’ understanding of research and increased their awareness to apply what they had learned to their personal lives, professional occupations, and in their communities. As more time passes from the completion of the program, it will be useful to follow-up with fellows to see specifically how they used what they learned in the program to improve health outcomes in their community.

Discussion

In this qualitative study, we evaluated the first two cohorts of the CRFT program. Multiple CRFT program components provided a conducive learning environment for community members to learn about health research and how to use research to improve health in their communities. These key aspects included the once-a-week program schedule, faculty and fellow diversity, small group activities, and community-focused homework assignments.

When tailoring a program to a diverse group of community members, it can be challenging to strike a balance in accommodating different levels of education and learning styles. For example, while most fellows enjoyed group work, there were others who preferred a lecture format. A training program that uses multiple formats (e.g. lectures, small group activities, group exercises) offers something for everyone. In addition, the educational and professional diversity of the cohort and differences in perspectives are seen as an asset. Our results indicate that recruiting faculty with a similar racial background to fellows enhanced their experiences in the program. CRFT fellows were predominantly African American, and it was evident from their interviews that meeting and interacting with African American scholars was motivational and positively impacted their engagement in the program. CRFT faculty were similarly affected by their participation in the program, reporting that they too learned from the fellows and would be willing to collaborate with them on CBPR pilot projects (Coats et al, 2015). To date, the CRFT program has funded three pilot CBPR projects (two in cohort I, and one in cohort III). The research was led by teams of 3–4 fellows in community-based settings in collaboration with one CRFT faculty member, demonstrating how the CRFT program has enhanced the CBPR infrastructure in St. Louis (Coats et al., 2015; D’Agostino McGowen et al., 2015). In addition, CRFT alumni have collaborated on other projects funded through other mechanisms (Coats et al, 2015).

Previous iterations of this training program were implemented in community settings (M S Goodman et al., 2012). For the CRFT program, we decided to invite community members to the university campus. The benefits were two-fold: community members could come to campus and see that they were welcome; thus, making it more likely that they will participate in and contribute to future community-academic research initiatives. CRFT alumni remain connected to the program and the university, well after their graduation from the program. This is evidenced by their participation as small group facilitators in subsequent CRFT program cohorts, attendance at CRFT information sessions, orientations and certificate ceremonies, and attendance at university-sponsored community events and local conferences.

A prime example of how CRFT alumni have stayed involved in CBPR initiatives and worked to improve health outcomes in the St. Louis region is through the PRAB. The PRAB serves in an advisory role to academic researchers on issues of community engagement, building trust, and ethical considerations of research and study design. Currently, the PRAB planning committee has 11 members that meet monthly, while the general body of over 20 members spanning multiple cohorts meets quarterly. The planning committee has developed evaluation criteria to review grant proposals and provide academic researchers with input that addresses potential community concerns. They also serve as the community advisory board to programs and projects at the Saint Louis County Department of Public Health (e.g., Healthy Families America). The development of the PRAB and its continued involvement in academic research and public health practice is a key example of CBPR facilitated by the CRFT program. In addition, CRFT alumni are actively engaged in community-based research efforts and have participated as community reviewers for internal grants at Washington University in St. Louis and external grants through the Missouri Foundation for Health (Coats et al., 2015).

Feedback from the program evaluation has allowed the team to continually refine the program and adapt to current community needs. Given the mixed feedback on the break-out groups we tried some other formats for activities (e.g., debates) in the cohort III training. The August 2014 death of Michael Brown in the St. Louis suburb of Ferguson occurred as we were wrapping up the cohort II training. Several of the cohort II fellows discussed the protest and community needs in their follow-up interviews. One fellow [cohort II, female, 29-years-old] indicated having participated in the Ferguson protests and left the protests “wondering is there something now – we went through this training – that we can do and Wash[ington] U[niversity] can partner up and do something …to be part of that.” Another fellow [cohort II, female, 41-years-old] in making suggestions for program improvements said, “the whole issue of public policy and how to advocate…I think that would be very valuable to have some voice or some small amount of time that really lets people know how public policy change happens.” Fellows felt what they learned in the CRFT training could be used but wanted more information on how to do this. In response to this request, the cohort III training included a session on community organizing. Preliminary results from the cohort III evaluation suggest this session was a favorite among fellows.

Conclusions

Best Practices

The findings of this qualitative program evaluation confirm and complement prior quantitative studies of how the CRFT program has successfully increased community capacity to partner with academic researchers and address heath disparities in St. Louis (Coats et al., 2015; D’Agostino McGowen et al., 2015). Based on the findings of this qualitative study, we identified five best practices: (a) It is beneficial to conduct follow-up evaluation interviews with fellows shortly after the CRFT graduation. In cohort I of the program, interviews were conducted approximately 6 months after graduation whereas in cohort II interviews were conducted in the weeks following the certificate ceremony. While fellows in cohort I may have had more time to consider the impact of the CRFT program, we found that recall of program experiences was sharper when the interviews were conducted more closely following their participation in the program. (b) Reducing barriers to participation is central to achieving a high level of attendance and overall completion of the program. In CRFT, this was done by providing fellows with the supplies needed to participate, dinner at each of the weekly sessions and free metro passes/parking validations. (c) It is constructive to include both large group interactive lectures and smaller break-out discussions to accommodate multiple learning styles. We found the small group activities worked better when limited to five people per group. It also helps to experiment with different types of activities (e.g. structured assignments with worksheets, mock debates, etc.) and how often to use group work in sessions based on feedback from fellows. (d) Recruiting faculty with similar demographics to community fellows will foster a positive and more supportive learning environment. (e) Incorporating community-focused homework assignments enhances learning and encourages community members to venture out into their community and examine it in a different way.

Research Agenda

The CRFT program has been designed such that it can be tailored and implemented in different community settings. The CRFT program was implemented in an urban setting, while the CARES program was implemented in a suburban setting. In 2014, the Office of Health Disparities Elimination at the Mississippi State Department of Health adapted the program; to date, they have trained 76 residents across 3 cohorts, two in the Jackson, Mississippi metropolitan area and one in Hattiesburg, Mississippi. Similar to results found in St. Louis, preliminary evaluation results from CRFT-Mississippi suggest high levels of satisfaction among fellows and faculty. The program has now been implemented by academic and government entities in the northeast, midwest and south, and can be adapted in other communities. If possible, future training programs should coordinate evaluation efforts across multiple sites. It would be important to compare structure, format, content, and location to identify which strategies are most effective in particular settings.

Additionally, future trainings should also consider long-term outcomes. For example, do fellows participate in public health research activities one, two, or five-years after completing the CRFT program? And, if so, what are these activities and what is the individual and community-level impact of such activities? Examining long-term outcomes of the CRFT program will contribute to our understanding of the impact of CBPR activities on local communities.

Educational Implications

The CRFT program was designed to equip community members with the skills and knowledge to understand the essential components of the public health research process. The program consisted of culturally appropriate pedagogical practices that included community-focused homework assignments. The homework assignments and pilot CBPR projects played an important role in extending material beyond the classroom to community-based settings. By training community members on how to identify and advocate for community concerns, the CRFT program has fostered research partnerships that will be critical to successful community-engaged research in the St. Louis region. A health research training program like CRFT can be mutually beneficial to community members and academics interested in reducing health disparities and achieving health equity.