A case study of engaging hard-to-reach participants in the research process: Community Advisors on Research Design and Strategies (CARDS)^®

Abstract

Lack of diversity among study participants in clinical research limits progress in eliminating health disparities. The engagement of lay stakeholders, such as patient or community advisory boards (CABs), has the potential to increase recruitment and retention of underrepresented groups by providing a structure for gathering feedback on research plans and materials from this target population. However, many CABs intentionally recruit prominent stakeholders who are connected to or comfortable with research and academia and thus may not accurately represent the perspectives of underrepresented groups who have been labeled hard-to-reach, including racial minorities and low-income or low-literacy populations. We developed a partnership between the University of Wisconsin-Madison School of Nursing and two community centers to deliberately engage hard-to-reach people in two lay advisory groups, the Community Advisors on Research Design and Strategies (CARDS)^®. Community center staff recruited the CARDS^® from center programs, including parenting and childcare programs, women’s support groups, food pantries, and senior meal programs. The CARDS^® model differs from other CABs in its participants, processes, and outcomes. Since 2010, the CARDS^® have met monthly with nurses and other researchers, helping them understand how research processes and the language, tone, appearance, and organization of research materials can discourage people from enrolling in clinical studies. We have successfully used the CARDS^® model to bring hard-to-reach populations into the research process and have sustained their participation. The model represents a promising strategy for increasing the diversity of participants in clinical research.

The elimination of health disparities is a national priority (National Institute on Minority Health and Health Disparities, n.d.; U.S. Department of Health and Human Services, n.d.). Clinical research can play an important role in eliminating health disparities by making novel treatments available to underserved populations and identifying the effectiveness of particular treatments for particular populations. However, progress toward eliminating health disparities has been hindered by lack of participant diversity in clinical research studies (Kitterman, Cheng, Dilts, & Orwoll, 2011; Schroen et al., 2010). Researchers often do not reach recruitment goals for hard-to-reach participants such as racial and ethnic minorities and people with low income. Reasons for low recruitment include mistrust of research, perceived risks, and lack of culturally appropriate information about opportunities to participate in research (Ford et al., 2008), as well as burdens such as time commitment and lengthy questionnaires (Paskett et al., 2008).

A recommended strategy for overcoming barriers to recruitment is patient and community stakeholder involvement in the design and conduct of clinical trials (Institute of Medicine, 2011). One popular form of stakeholder involvement is community advisory boards (CABs). CABs serve several functions, depending on the project mission and requirements. They provide input into research agendas; serve as gatekeepers for researcher entry into communities; formally approve projects; and influence the design, conduct, and implementation of studies (Fernandez-Pena et al., 2008; Silvestre, Quinn, & Rinaldo, 2010; Strauss et al., 2001).

Recommendations for CAB membership often emphasize inclusion of influential stakeholders and opinion leaders from the community (D'Alonzo, 2010). Typical CAB members include service providers, community leaders, and representatives of local agencies and organizations (Newman et al., 2011); in this paper, high-level stakeholders are referred to as “prominent community representatives”. Prominent community representatives bring valuable skills, perspectives, and resources to CABs due to their experience with group and meeting processes, influence and connections within the community, and decision-making power within their organizations. CABs also sometimes include lay stakeholders who are not prominent community representatives, but can bring an important viewpoint to a research project, such as residents of a particular neighborhood or patients or caregivers with experience related to a particular health issue (James et al., 2011; Pinto, Spector, Rahman, & Gastolomendo, 2013). Lay stakeholders can help researchers improve the cultural sensitivity and appropriateness of recruitment materials and methods; recommend study implementation strategies (Joosten et al., 2015); and design relevant, meaningful interventions (GreenMills, Davison, Gordon, Kaigang, & Jurkowski, 2013). The perspectives of lay stakeholders who are disconnected from academia and local power structures may offer clinical researchers unique insights and perspectives on recruitment barriers and strategies for improving recruitment and retention of hard-to-reach populations. This paper describes the origins, participants, processes, and outcomes of a unique CAB comprised solely of lay stakeholders that brings voices of rarely heard groups into the research enterprise.

Project Origins and Community-Academic Partnership

Our team at the University of Wisconsin-Madison (UW-Madison) School of Nursing partnered with two local community centers to develop lay advisory boards of community members who are not prominent community representatives. Throughout this paper, “we” refers to the authors. The Lussier Community Education Center and Goodman Community Center have served their neighborhoods for over 30 years, offering programs and services that reflect the diversity of their communities. In 2010, we worked with staff at the community centers to submit a proposal for a program grant to the National Institutes of Health (NIH). We received a three-year grant to establish a sustainable infrastructure to link community members with researchers to improve the quality of health sciences research. The UW-Madison Health Sciences Institutional Review Board (IRB) designated our grant activities as exempt from review. For one of our grant activities, we worked with the community centers to develop and pilot two lay advisory boards, the Community Advisors on Research Design and Strategies (CARDS)^®. We established a CARDS^® group at each partnering community center. For the past five years, each group has met monthly with guest researchers to provide feedback on the appeal, clarity, and accessibility of materials and processes used in clinical research. Our NIH grant ended in 2013, and since then we have sustained the CARDS^® as a program within the Wisconsin Network for Research Support (WINRS), a fee-for-service community and patient engagement center supported by the School of Nursing (http://winrs.son.wisc.edu/). The UW-Madison Institute for Clinical and Translational Research (ICTR), a Clinical and Translational Sciences Award site, provides additional support for the program.

The CARDS^® Program

The CARDS^® are based in the community, so researchers often assume that CARDS^® input is relevant only for researchers interested in community-based participatory research (CBPR) or community-engaged research (CER). In fact, since the program’s inception, the CARDS^® have worked extensively with researchers conducting clinic or hospital-based studies. The CARDS^® provide lay, patient-centered feedback on materials for any type of human subjects research, regardless of the specific research topic, setting, methodology, or study population. Researchers and their project staff typically attend one or two meetings for each project that they bring to the CARDS^®, although we place no limit on the number of CARDS^® meetings that a researcher may attend. The CARDS^® program makes it feasible for researchers who are not conducting CBPR or CER to get timely, meaningful lay input on their research materials, without substantial investment of resources.

Program Participants

Members

For the CARDS^®, we wanted not prominent community representatives, but people who have been under-represented in healthcare research (Hasnain-Wynia & Beal, 2014). We intentionally sought to recruit people who are not connected with academia, research, healthcare, or local power structures. Unlike recruitment for many CABs, we did not establish specific inclusion criteria, such as residency in a certain neighborhood or diagnosis of a particular health condition (Stewart et al., 2015). We asked community center staff (CARDS^® liaisons) to recruit typical users of services at their centers. Liaisons shared an informational flyer with community members who used center services such as parenting and childcare programs, women’s support groups, job clinics, food pantries, and senior meal programs. People who were interested followed up with the CARDS^® liaison to complete a membership application that asked applicants to briefly explain why they wanted to be part of the CARDS^®. Sometimes, people who worked or volunteered at the centers heard about the program and completed applications. Currently, 15 CARDS^® participate in the program. Nine participants (60%) are women and 10 (67%) are African-American. Ages range from early-20s to mid-70s, and educational attainment ranges from incomplete high school preparation to post-high school coursework.

Many CABs have expansive roles for their members that may encompass defining research agendas, providing entrée into a community (gatekeeping), engaging in the conduct of research, and providing feedback on scientific papers (Israel et al., 2005). The CARDS^® sole function is to meet monthly with researchers to review, discuss, and provide feedback on research plans and materials, including recruitment materials, data collection procedures and instruments, web-based materials, and smartphone apps. They are never required to complete work between meetings. For each 90-minute meeting, the CARDS^® receive a cash stipend of $35. Although it is convenient for institutions to pay board members with checks, many people in hard-to-reach populations do not have checking accounts. As one of the CARDS^® explained, “$35 isn’t $35 if I have to pay a fee to cash the check.”

WINRS and community center staff

WINRS staff provides overall management of the CARDS^® program. The School of Nursing employs two WINRS staff members at a total of 1.7 full-time equivalents. WINRS staff work on a variety of projects, but together devote approximately 0.5 full-time equivalents to management of the CARDS^®. Both employees have advanced degrees (MA; PhD) and had previous experience with community engagement. One employee has primary responsibility for the program, with the other providing support. The School of Nursing and ICTR provide salary support for the full-time employee, and revenues from WINRS support the salary of the part-time employee. In addition, each community center provides a staff member who serves as the center’s CARDS^® liaison. Table 1 lists the respective responsibilities of WINRS, community center staff, and the CARDS^®. We pay the centers a monthly facility fee to compensate them for liaison time, room rental, childcare during meetings, and transportation for members as needed. Monthly facility fees range from $205 – $290, depending on whether childcare is provided.

Table 1.

Responsibilities of Community-Academic Partners for CARDS^® Program

Community Center Staff	WINRS Staff	CARDS®
Recruit people who use center services to participate in CARDS® program	Design, deliver orientation program for new CARDS®	Complete orientation to develop skills for giving feedback to researchers
Schedule community center facilities for meetings, orientations	Conduct outreach to identify guest researchers for CARDS® meetings	NA
Arrange child care, transportation for monthly meetings; serve as contact person for members to confirm attendance	Meet with researchers to prepare agenda, materials for meetings; send agenda, meeting reminder to CARDS®	RSVP for monthly meetings; read meeting announcement with description of research topic and materials
Participate in CARDS® meetings	Facilitate CARDS® meetings; take notes; disburse member stipends	Provide feedback to guest researchers at meetings
Invoice UW-Madison School of Nursing for facility fee	Write summary reports, revised materials for researchers; complete post-meeting evaluation surveys with guest researchers	NA
Problem-solve with members to address barriers to their participation	Work with liaisons, members to support participation and sustain program	As needed, meet privately with program staff to address issues or problems related to participation
Participate in program evaluation	Design and lead program evaluation	Participate in individual interviews and group discussions to evaluate program

Guest researchers

Since the CARDS^® program began in 2010, we have conducted meetings with 40 research teams representing 21 disciplines, including Cardiology, Family Medicine and Community Health, Kinesiology, Nursing, Pharmacy, Rheumatology, and Surgery. We have also worked with researchers from other institutions, sometimes conducting CARDS^® meetings via teleconference. Nurses constitute the largest proportion of researcher teams (58%, n = 23) who have used the CARDS^® service, and our guests have included nursing faculty, doctoral students, and nurses working in community settings. Nurse researchers have brought a variety of materials and plans to the CARDS^® for their review and feedback, including recruitment materials (brochures, flyers, letters, scripts); study processes (recruitment plans, interview processes, website development); focus group protocols; interview and survey questions; study information sheets; and consent forms.

We sometimes field inquiries from researchers who are interested in lay review of their materials but are skeptical that lay advisors who are not members of their particular target population can provide useful feedback. Materials used in human subjects research—recruitment notices, information sheets, questionnaires and other instruments, consent forms—share a common set of challenges for members of the public due to their technical language, dense presentation of information, embedded assumptions, counter-intuitive organization, and academic tone. As members of the general public with specific training on how to give feedback to researchers, the CARDS^® can offer a uniquely fresh perspective on research materials, no matter the specific content of the documents. The CARDS^® have provided useful feedback on materials for a wide range of health studies, despite having limited or no personal experience with many of the topics that researchers present. At one of our most memorable meetings, no women in the CARDS^® were able to attend, so five male CARDS^® provided feedback on a doctoral student’s survey about dysmenorrhea. We introduced the survey by reminding the CARDS^® that we all have experience with pain and discomfort, and we asked them to think about the survey questions in the context of their personal pain experiences. Based on CARDS^® feedback, the student made substantial revisions to her survey.

Program Processes

Member orientation

One of the key features of the CARDS^® is our orientation program. Many CABs provide minimal orientation or training for board members (Albert Einstein College of Medicine, The Bronx Health Link, & Community-Campus Partnerships for Health, 2012). In contrast, the focus of the CARDS^® orientation program is hands-on practice with research materials. We designed an interactive group orientation to help the CARDS^® develop and practice the skills needed to participate effectively at meetings. Our orientation program emphasizes content that is directly relevant to the work that board members will do and gives members experience reviewing and providing feedback on recruitment flyers and letters, consent forms, and focus group questions. Completion of the 2.5-hour orientation is a requirement for membership in the CARDS^® and provides meaningful preparation for respectful, productive interactions with researchers.

Preparation of guest researchers

Each CARDS^® group meets monthly at the community centers with guest researchers. Several weeks before the meeting, WINRS staff sends the scheduled researcher a timeline of steps to prepare for the meeting, along with a link to a short web-based survey requesting key information about the researcher’s desired outcomes from the meeting. WINRS staff then engages in a series of email messages and a telephone call or face-to-face meeting with the researcher to plan and prepare for the meeting.

During the planning meeting, WINRS staff and guest researchers review answers to the web-based survey and discuss possible materials for the CARDS^® meeting. Researchers often bring their staff members and students to planning meetings and CARDS^® meetings. We select the materials that can be thoroughly discussed during a 90-minute CARDS^® meeting and develop a detailed working agenda. Without this preparation, researchers may have unrealistic expectations of how many documents can be reviewed during a meeting. During the planning meeting, WINRS staff share advice on working with lay advisors. For example, we remind researchers to use plain language, and we help them prepare brief, straightforward explanations of their research interests. We also encourage researchers to gracefully acknowledge all suggestions from the CARDS^®. Realistically, researchers may not want to or be able to follow certain suggestions, but thanking the CARDS^® for all suggestions helps them feel that their voices are heard and welcomed. Decisions about how to use CARDS^® feedback can be made by researchers after they attend meetings, when they can decide whether to incorporate specific suggestions fully, partially, or not at all.

As part of meeting preparation, we discuss fees and payment arrangements with the guest researcher. We bundle our costs for WINRS and community center staff time, CARDS^® stipends, and other costs for the CARDS^® service into a fee of $1964 for a 90-minute meeting, preparatory work with the guest, and post-meeting summary documents. As required by the UW-Madison, our fee structure is based on a cost-recovery model; the fee for the CARDS^® service covers our costs for providing the service, but does not generate profit. Researchers typically pay the fee with grant funds. For doctoral students or junior faculty without funding, we offer a reduced fee. After we meet with guest researchers, we send a brief meeting announcement to the CARDS^® to provide information on the guest researchers, including their focus areas, materials they will present at the meeting, and type of help they would like from the CARDS^®. The announcement also includes a personal statement from the guest researcher about why she or he cares about the research topic. We work closely with researchers to help them craft personal statements. The CARDS^® have told us that learning about a researcher’s personal motivations for doing research has helped them overcome their stereotypes of researchers as cold academics who use research participants to serve their own ends.

Meeting structure and practices

We follow a structured sequence of activities at each meeting. We start each meeting with an opening question. Everyone at the meeting participates in a round-robin sharing of names and brief responses to the opening questions. Our opening questions give everyone at the meeting a chance to share something about themselves. The activity helps to break stereotypes that researchers and lay community members may hold about each other. Over time, the sharing that occurs with the opening question helps group members establish personal connections and build a sense of community. We often craft opening questions that are related to the research topic that we will discuss at the meeting. For example, our opening question for a meeting with a researcher studying nasal irrigation was “Please say your name, and tell us one favorite home remedy for dealing with a stuffy nose or other sinus problems.”

After the opening question, guest researchers introduce themselves briefly, using plain language to explain the goal of their research and why it is important. We use the remaining 75 minutes of the 90-minute meeting to discuss the researcher’s materials. The structure of the discussion is standardized and uses 3 steps to elicit feedback from the CARDS^®:

1. The researcher describes how and where a prospective study participant would encounter the research materials, e.g. mailed letter to a home address; flyer posted in a primary care clinic; consent form presented in a community setting; website that the participant will access from a home computer.

2. We ask guest researchers to read their materials aloud, several lines at a time, to facilitate full participation of everyone present, regardless of literacy level.

3. WINRS staff facilitate a section-by-section review of the materials. After a section is read, the CARDS^® offer comments. We take detailed notes and write CARDS^® feedback on a flip chart, a practice that helps to affirm the value of everyone’s contribution to the discussion. After finishing review of one section, we move to the next section and repeat the process.

Post-meeting products and survey

During the first two years of CARDS^® meetings, we provided a summary report to researchers who attended meetings. The 1–3 page report summarized the overall feedback from the CARDS^® on the research materials reviewed at the meeting and highlighted specific issues related to the content, language, organization, and format of the materials. However, when researchers later shared their revised materials with us, we saw that they often did not translate CARDS^® feedback into concrete changes to their materials, even though they rated the value of the meetings very highly. Consequently, we now deliver revised materials to all guest researchers, to make it easy for them to implement recommendations from the CARDS^®. Researchers consider and balance numerous factors when designing their study materials, and we encourage researchers to use the CARDS^®-revised materials in whatever way makes the most sense for them, based on their experience and expertise. To complete our consultation, we send a post-meeting online survey to all researchers who attend CARDS^® meetings to ask them to evaluate their experience.

Program Outcomes

CARDS^® Insights and Recommendations

The value of the CARDS^® lies in their ability to provide researchers with fresh insights and feedback that academic or professional colleagues immersed in research may not be able to offer. As the concept of health literacy has permeated health sciences research, researchers have developed more awareness of how using technical jargon can undermine subject recruitment. Use of readability tools such as SMOG (National Cancer Institute, 1989) or the Flesch-Kincaid tool in word processing programs can be helpful for improving readability. Researchers also sometimes rely on colleagues, graduate students, high-level stakeholders, or IRBs to identify glaring problems in research materials. While these resources can be helpful, they may not be sufficient for ensuring that materials are acceptable and inviting to the general public. Over five years of monthly meetings, the CARDS^® have highlighted several key characteristics of research materials that may turn people away from participating in research.

Passive language

Plain language guidelines highlight the importance of using active language instead of passive language (“staff on my research team will ask you several questions” instead of “you will be asked several questions”). Passive sentences lack the clarity of active sentences because they do not clearly identify the actor. The CARDS^® highlight other problems with passive language, describing it as confusing, impersonal, and evasive. For them, passive language in a research document raises questions and reservations about who is behind the research project, the true motives for the project, and what researchers will do with participants and their information. Passive language pervades consent forms, and it particularly troubles the CARDS^® when it appears, as it does routinely, in material related to privacy and confidentiality. Vague assurances that “your information will be stored in a secure location” arouse their suspicions; they want to know exactly who will take responsibility for protecting their information.

Specialized use of everyday language

The CARDS^® have provided feedback of a nuanced nature that highlights issues not measured by readability tools. The CARDS^® repeatedly point out things that are confusing, patronizing, or even offensive or frightening to potential research participants. For example, even though researchers may scrupulously avoid using technical jargon, they often use everyday language in a highly specialized manner. The CARDS say that words such as “data”, “procedures”, “health outcomes”, “technique”, and “investigate” sound ominous or threatening. Even the word “study”, universally used by researchers in recruitment materials, can evoke images of being subjected to dangerous tests or used like a guinea pig. The CARDS^® strongly prefer “project” as a non-threatening alternative. When referring to research subjects, the CARDS^® strongly prefer the phrase “project participants”, which sounds inviting, inclusive, and less scary than the off-putting, objectifying “subjects”. Table 2 lists additional examples of how the CARDS^® interpret common research language.

Table 2.

What Researchers Say, What CARDS^® Hear, and What CARDS^® Recommend

What Researchers Say	What CARDS® Hear	What CARDS® Recommend
My colleague is the PI on this study.	I work with a private investigator who will poke into your personal business.	I work with Xxx Xxxx. (S)he is the lead researcher on this project.
Data will be collected about your lifestyle.	We will violate your privacy and make judgements about your personal life.	My research team will ask you some questions about a typical day for you.
Participants will take part in several procedures.	Participants will have scary, invasive medical acts performed on them.	Participants will be involved in several activities, including…
We will monitor your progress during this study.	We will track you with something like an electronic ankle bracelet during this study.	We will keep in touch with you during the project.
This study will test an experimental technique for treating sinus infections.	You will be a guinea pig for something dangerous, untried, and invasive.	This project will test a new way of treating sinus infections.
We would like to include your name in our study registry.	We want to put your name on a registry, like a sex offender registry!	We would like to add your name to a list of people who might like to be part of future projects.
Your input will contribute to the design of an intervention to support couples in parenting their babies.	You will help the research team confront couples who are having problems with parenting.	Your input will help us develop a new program to support parents as they care for their babies.
This study aims to improve how we provide care to people with type 2 diabetes.	Being in this study will improve your diabetes care.	We’d like to hear about your experiences to help us figure out how to improve care for people with type 2 diabetes.
We will consider your individual home environment and search for assessable risks for falls.	We will judge you and your home and snoop into your personal belongings.	We will use a checklist to look for risks that you can change to prevent falls.
To manage your blood pressure, be more active and drink less alcohol.	We assume that you’re not active and that you drink too much.	To manage your blood pressure, be active and limit your intake of alcohol.

When we meet with researchers to prepare for CARDS^® meetings, we explain that the CARDS^® may occasionally suggest alternative language that an IRB will reject. To our knowledge, these instances are infrequent and have not limited the value of the service. Currently, we are collaborating with an IRB workgroup that is charged with improving templates for informed consent. The working group has attended three CARDS^® meetings to learn what kinds of information about research are important to the non-academic community and to help craft clearer language for consent documents. In the coming year, the working group expects to pilot new templates that incorporate CARDS^® feedback.

Perceived tone of documents

The CARDS^® have noted several other key considerations for making documents accessible and inviting. The overall tone of a document can serve to engage or disengage potential participants. For example, although best practices in health education often recommend repeating key messages in documents, the CARDS^® say that redundant language feels insulting, by implying that they are not smart enough to understand something the first time that it appears in a document. In addition, materials that sound non-judgmental may be more likely to engage potential participants, especially for studies that address health behaviors. The CARDS^® often use the term “gentler” when recommending alternative language. For example, when researchers ask about tobacco use, the CARDS^® note that the simple question “Do you want to quit smoking?” is a complex, sensitive question and that asking a “yes” or “no” question may not be appropriate. The CARDS^® suggest “Are you thinking about cutting back or quitting?” as a gentler alternative that raises the smoking issue without provoking defensiveness by implying that the person is doing something wrong.

Requests for demographic information

Demographic questions are standard items in many research instruments, and members of the public also frequently encounter them in the context of consumer research or participation in various programs. Despite the ubiquity of demographic questions, they may provoke negative reactions due to their highly personal nature. The CARDS^® have told us repeatedly that they expect researchers to use demographic information to stereotype people and present them in a bad light. The CARDS^® have shared personal stories about skipping demographic questions, providing false answers, or declining to complete an instrument altogether if it includes demographic questions. They recommend that researchers ask only those questions that are critical for answering the study questions. In addition, the CARDS^® recommend that researchers provide a short, straightforward introduction to demographic questions to explain why the researcher needs the requested information. Explanations that simply invoke grant requirements for demographic information may be insufficient and potentially offensive; rather, a thoughtful explanation for asking demographic questions can explicitly acknowledge that demographic questions are personal and offer a straightforward rationale for asking them.

Lessons Learned on Engaging Hard-to-reach Groups

When we initiated the CARDS^® program, we deliberately sought participants outside of the academic environment and not situated in positions of power. Our community center liaisons intentionally recruited people not typical of advisory boards. Many of them have had challenging life experiences, including poverty, homelessness, long-term underemployment, and chronic health problems. We have structured the CARDS^® program to minimize barriers to participation and to demonstrate our respect and appreciation for all members. Our use of conscious practices that engender trust and help people feel valued have helped us to achieve outstanding retention of our lay advisors. Six of the current members (40%) have participated since 2010, and the remaining nine members have participated since 2012. Half of the former members who dropped out of the groups from 2010 – 2012 (n = 6) left because they moved out of the area, changed their work schedules, or experienced re-incarceration. Attendance has been very consistent. Since 2013, when we standardized many of our meeting processes, aggregated attendance for the program is 81% (486 instances of member present for meeting/601 total opportunities to attend). Attendance for individual CARDS^® over this period has ranged from 58% of meetings attended to 97%.

Consistent staffing at both WINRS and the community centers has been a critical component of developing trusting relationships with our members and retaining them in the group over time. The same WINRS staff have facilitated every CARDS^® meeting since 2010, and the CARDS^® liaisons have typically stayed in their positions for at least two years. Consistent staffing has helped us to develop trusting, long-term relationships with our members, which in turn support honest dialogue and problem-solving with members when challenges arise. On several occasions, we have met with individual CARDS^® to address problems related to inappropriate behavior toward a program participant or insufficient participation in meeting discussions. The orientation that all CARDS^® complete serves as a useful point of reference for constructive discussions about performance and expectations.

During our first few years, we often wrote short individualized notes to send to the CARDS^® after meetings, to thank them for their contributions. We call these notes “affirmations”; because they affirm something unique and positive that the person brings to the group. We also hold annual celebrations with the CARDS^®. Our celebrations have included meals, holiday treats, and small tokens such as gift cards.

For some of our members, applying for jobs is a frequent and difficult task. We provide tangible support with letters of reference and assistance with resume-writing. We write letters of reference for members based on their work with the CARDS^® and provide resume templates that highlight the unique skills and tasks of the CARDS^®. With the advent of web-based systems for providing references, we have also completed on-line surveys to support job applications for some of the CARDS^®.

Program Evaluation by the CARDS^®

In 2014, we completed a program evaluation with the CARDS^®. We conducted separate focus groups at each community center to explore members’ opinions and attitudes in four areas: their orientation to the program; their work at CARDS^® meetings; factors that contribute to their retention in the program; and changes in how they view research and researchers. The CARDS^® valued the orientation and particularly emphasized the impact of group training activities on communicating respectfully and giving effective feedback. They also appreciated the orientation as a chance to experience what they would do at meetings, and several members credited the orientation with giving them confidence that they could do the job. The CARDS^® described multiple characteristics of the monthly meetings that make the work meaningful and enjoyable. These included learning about research topics that were relevant to them and their community and having a chance to give advice directly to researchers. The most prominent theme throughout the focus groups was the sense of community that motivates the CARDS^® to stay in the program. They used words such as respect, good will, support, camaraderie, and fellowship to explain why they have stayed in the program. They also described how our standard meeting practices such as the opening question, use of the flip chart, meeting snacks, and non-hierarchical facilitation contribute to a sense of community. Another major factor related to retention was the sense of service to others. As one member said, “I like being able to help others. We may be helping folks who will never even know we helped them.” Finally, the CARDS^® discussed how their attitudes toward research and researchers have shifted. They said that the opportunity to work regularly with researchers has helped them respect researchers, rather than mistrust their motives. Some CARDS^® also indicated that they are more interested in participating in research themselves: “Now I feel it’s important to participate in research. Researchers are doing it for the community; it’s bigger than just one person.”

Researchers’ Evaluation of CARDS^® Service

Since 2012, all guests who attend CARDS^® meetings receive an email one week after the meeting inviting them to complete a brief web-based evaluation survey. To minimize the response burden on researchers, the survey includes just a few key questions to assess guests’ satisfaction with the overall CARDS^® service, including pre-meeting preparation and post-meeting reports. Guests at CARDS^® meetings have rated the service very highly. Ninety-seven percent (n = 70) of survey respondents have indicated that they felt sufficiently prepared by WINRS to meet with the CARDS^®. Guests typically brought one or two items for review, and they most frequently brought data collection materials such as survey or focus group questions (51% of researchers, n = 37) and recruitment materials (37%, n = 27). A large majority of guests (90%, n = 65) planned to change their materials based on CARDS^® feedback. (Several respondents self-identified as project staff that did not have authority to make changes to research materials.) Ninety-nine percent of survey respondents (n = 71; 1 missing response) have indicated that they would recommend the service to colleagues. Table 3 displays findings related to guests’ perceptions of the utility of the CARDS^® program.

Table 3.

Guests’ Perception of Utility of CARDS^® Services, 2012 – 2016

	Total N = 72
	n	%
Which services of the CARDS® program were particularly useful?
Completing web-based survey to initiate planning	28	39
Meeting with program staffa	44	61
Presenting ideas and materials to CARDS®	56	78
Getting feedback from CARDS®	70	97
Receiving summary of CARDS® feedback	65	90
Receiving revised study materials	60	83

Note. Percentages do not sum to 100% because respondents could select multiple options.

^aNew question added to survey in 2103.

When we initiated the program, we wondered whether regular contact with researchers would acclimate the CARDS^® to research language and over time diminish their value to researchers. To preserve the lay perspective of the CARDS^®, we have maintained a schedule of meeting only once monthly at each center. Researchers’ evaluations of the CARDS^® service have been favorable across the lifespan of the program, suggesting that the CARDS^® have continued to offer valuable feedback. Additional findings related to the effectiveness of the CARDS^® are published elsewhere (Bowers, Krupp, & Jacobson, 2016).

Conclusion

Health sciences research should serve everyone. Everyone should have opportunities to participate in research and benefit from research. When researchers have study participants who reflect the rich diversity in our communities, our society has a real chance to develop policies and practices to eliminate health disparities. But current research suggests that equitable representation of diverse groups in research has yet to be achieved (Bonevski et al., 2014).

Social justice is a central concern of the nursing profession, and nursing research literature offers numerous examples of studies using approaches such as CBPR to engage disenfranchised or hard-to-reach stakeholders in the research process (e.g. Perry & Hoffman, 2010; Stacciarini et al., 2011). In addition, many nurse researchers have designed or implemented interventions with input from community health workers or promotores (e.g. Nies, Troutman-Jordan, Branche, Moore-Harrison, & Hohensee, 2013; Whittemore, Rosenberg, Gilmore, Withey, & Breault, 2014). The CARDS^® represent a new, unique model for engagement of people who are not prominent community representatives in the design of research, and the model offers distinct advantages over other forms of public engagement. Although lay stakeholder engagement can take diverse forms, sometimes it can seem as if the options are all or nothing—either researchers conduct CBPR or they do nothing to seek community input. The CARDS^® model makes reaching hard-to-reach stakeholders feasible and convenient for researchers.

The CARDS^® are a truly innovative stakeholder engagement model, one that brings hidden voices into the research enterprise and makes these voices easily accessible to researchers. With our partners at the community centers, we have built a program that provides a valuable service to researchers and a meaningful experience for lay stakeholders. The CARDS^® represent an innovative model for nurse researchers to support sustainable, meaningful engagement of hard-to-reach populations in research planning and activities; increase the diversity of groups represented in research studies; and ultimately help to reduce health disparities.