Women’s Experiences in a Community-Based Participatory Research Randomized Controlled Trial

Abstract

Integrating community-based participatory research (CBPR) into traditional study designs can enhance outcomes in studies with disadvantaged groups. Little is known, however, about study participants’ experiences with these approaches, the underlying processes involved in creating more positive outcomes, and whether undesirable effects on study outcomes occur simultaneously. We conducted focus group interviews with 31 disadvantaged women who participated in a CBPR-driven randomized controlled trial (RCT) both to explore their study experiences and to obtain their interpretations of select study findings. Using dimensional analysis, we found the tailored health questionnaire, treatment by study staff members, and RCT participants’ understanding of and responses to randomization were salient to what women described as transformative experiences that occurred over the course of the RCT. These findings have implications for understanding how CBPR and non-CBPR aspects of interventions and study designs have the potential to affect both process and endpoint study outcomes.

Over the past decade, eliminating health disparities and, most recently, establishing health equity have emerged as national health priorities in the United States (U.S. Department of Health and Human Services, 2010). Early attempts to achieve these goals using standard approaches to intervention development, however, yielded little in the way of improving health outcomes for lower socioeconomic and/or racial/ethnic minority groups. In other words, interventions constructed for disadvantaged populations that relied solely or heavily on the views of scientists—who most often resided outside the target group—were largely ineffective. The community of scientists and public health/health care practitioners working with this population soon realized that any meaningful progress toward reducing health disparities would require a new approach to research (Flaskerud & Nyamathi, 2002). In response, researchers began applying community-based participatory research (CBPR) methodology in partnership with disadvantaged groups to develop interventions (Cook, 2008; Minkler, 2005; Viswanathan et al., 2004). CBPR is defined as

a collaborative approach to research that equitably involves all partners in the research process and recognizes the unique strengths that each brings. CBPR begins with a research topic of importance to the community with the aim of combining knowledge and action for social change to improve community health and eliminate health disparities (Israel, Eng, Schultz, & Parker, 2005, p. 4).

Findings from comprehensive literature reviews indicate that, when applied in combination with a rigorous study design, using a CBPR approach to develop and test an intervention is more effective in improving health outcomes. In addition, applying a CBPR approach can minimize some of the threats to internal validity (e.g., attrition) that are common in randomized clinical trials (RCTs) conducted with disadvantaged populations (Smith, Rodriguez, & Bernal, 2011; Viswanathan et al., 2004). These improved study outcomes are attributed to embedding structural components of CBPR into the intervention delivery and study design as recommended by community partners (Viswanathan et al., 2004). Such components often take the form of culturally tailoring the intervention, study design, or study materials to the target community and/or hiring members from the community onto the research team. Findings suggest that enhancing personal relevance for study participants, reducing communication barriers with participants, and increasing the thoughtfulness with which participants review study-related materials likely contribute to positive study outcomes (Kreuter & Wray, 2003; Ochocka, Janzen, & Nelson, 2002; Reeve, Cornell, D’Costa, Janzen, & Ochocka, 2002).

Despite the growing approval of CBPR as a useful methodology in the scientific community, little is known about the experiences of participants in studies designed with a CBPR approach. To date, investigators have focused process evaluations of CBPR studies on developing and maintaining community partnerships (Kobeissi, Nakkash, Ghantous, Saad, & Yassin, 2011; Mitchell & Baker, 2005; Parker et al., 2003; Pivak & Goelman, 2011) and on the perceptions of community partners regarding institutional review board challenges (Brown et al., 2010; Malone, Yerger, McGruder, & Froelicher, 2006) rather than on the experiences of participants in the study. Moreover, whether or how integrating CBPR components into study designs negatively affects study outcomes remains virtually unexplored. This void is somewhat surprising, given that some aspects of CBPR (such as the frequent use of tailoring components to the target population and including members of the target population as research staff) have the potential to amplify other threats to validity (e.g., maintaining an adequate control condition or exaggerating instrumentation effects; French & Sutton, 2010).

Researchers have raised similar concerns in the nursing literature, although not specific to CBPR, about the difficulty of parsing active intervention from interpersonal relationships between study staff and research participants, or among study participants (in the context of group interventions; Beal, Stuifbergen, Volker, & Becker, 2009; Fogg & Gross, 2000). These tensions, observed in the process of conducting studies, are rooted in philosophical paradigms of science that challenge the applicability of positivist lines of inquiry for the purpose of understanding human experiences (Harding, 1991; Shalin, 1986). The general claim from these paradigms—that “the research act inevitably leaves its mark on the object” (Shalin, 1986, p. 22)—aptly reflects the increasing interest in and concern about unanticipated, and potentially problematic, effects of the research design on study findings.

In this article, we present findings from a process evaluation of a recent RCT in which the participants’ experiences with CBPR study components were explored. These findings inform our understanding of the interplay between study design, study processes, and endpoint outcomes. More specifically, we describe how women’s embedded experiences of carrying the label welfare recipient, and the women’s subsequent treatment within existing social and health systems shaped their perceptions of, and responses to, the role of study participant. Finally, we discuss the relationships among study design aspects, participant perceptions and actions, and depressive symptom outcomes by group, as well as the implications of our findings for future research. To put the rationale for conducting this post-RCT study into context, we provide a brief overview of key components of the primary study—a CBPR-grounded RCT—and group differences on select outcomes. We then present the post-RCT process evaluation methods, analysis, and findings.

Brief Description of Primary Study and Specific Aims

The primary study comprises two major components (for a more detailed description of the supporting background and other aspects of the study, see Kneipp et al., 2011, and Lutz, Kneipp, & Means, 2009). In the first component we redesigned a health screening questionnaire from the Welfare Transition Program (WTP) to be culturally relevant, sensitive, and acceptable to the target population. The women in the study indicated they wanted the questionnaire to include health-related dimensions they deemed relevant to their lives (e.g., stress and coping questions), and they emphasized the importance of conveying respect in the format, language, and underlying assumptions of the questionnaire. In addition, the women guided us in constructing a questionnaire using language that engenders trust and facilitates the disclosure of personal information concerning their health.

In response to the women’s concerns, we made several changes to the questionnaire. We included sections on stress and coping, because the women viewed those areas as highly relevant to their health. Standard screening tools or health-history questionnaires do not routinely address these issues. We provided a prequestion narrative to explain why select questions were being asked. Finally, we offered atypical response options for answering more sensitive health-related questions (e.g., options “yes,” “no,” and “maybe, but later” to questions such as “Would you like more information about [health-related topic]?” or “Would you like to discuss anything about [health-related topic] with the nurse?”).

We asked 111 women in WTPs to review the tailored questionnaire and rate it on aspects of relevance, sensitivity, and respectfulness. During this process, the majority wanted to complete it (although they were asked only to review and rate it). Given this reponse, we believed we had constructed a questionnaire that would be acceptable to the target group and would allow us to more fully engage them in health screening, data collection, and study enrollment. We then used the questionnaire to identify whether potential participants met the enrollment criteria for having a chronic health condition and to gather health-related data during the RCT.

In the second component of the study and using CBPR methodology, we tested the efficacy of a public health nursing (PHN) case management intervention in improving health and employment outcomes among women in WTPs who had chronic health conditions. To add more CBPR components to the primary study, we included a 17-member advisory group and a full-time research staff. The advisory group advised the first author and had decision-making authority on select aspects of the study. We also hired three women from the WTP for the staff. One served as the study coordinator, and two were community-based research assistants (RAs) who recruited participants and collected follow-up data.

During the RCT, all participants first completed the health screening questionnaire. They were then briefly assessed for meeting study enrollment chronic health condition criteria by the PHN. If women met enrollment criteria, they were subsequently randomized to either the control or intervention group. Intervention participants received a comprehensive PHN health assessment, referrals to address health needs, and case management services at baseline, 3, 6, and 9 months. During the intervention, PHNs adjusted the intensity of case management to meet the needs of individual participants. Based on recommendations from the target population during our intervention development work, we incorporated a wait condition into the control group, whereby women in the control group had access to an attenuated version of the PHN intervention at the end of their 9-month observation period. Control group participants received no intervention during the active study period; however, the community-based RAs collected data from women assigned to the control group at the 3-, 6-, and 9-month follow-up visits. Research staff members were assigned to study participants to maintain continuity over the 9-month study, such that intervention group participants maintained the same PHN during the RCT, and control group participants maintained the same community-based RAs.

Methods

We designed the study as a descriptive, exploratory process evaluation of the RCT described above. Following the recommendations by Morgan and Krueger (1997), we used a focus group format for data collection. As researchers integrate CBPR principles into more traditional study designs, process evaluations are becoming increasingly important for understanding the multiple, overlapping, and often subtle processes that co-occur within studies and potentially influence study outcomes (Steckler & Linnan, 2002). We also designed the study to involve the target population in interpreting the RCT study findings, a fundamental component of CBPR (Minkler & Wallerstein, 2003). Specific to the RCT outcomes of interest, although depressive symptoms improved significantly in the intervention group as compared to those in the control group (Treatment x Time p = .016), depressive symptoms improved considerably in the control group over time as well. Given the unexpected, relatively steep decline in depression in the control group, we wanted to understand the participants’ perceptions as to what factors outside the PHN intervention might have affected depression outcomes in each group.

Members of the research team with prior experience in moderating focus groups used a semistructured guide to elicit focus group discussions. Focus group moderators asked participants to review a graph depicting changes in depression over time in the intervention and control groups, and they gave a brief explanation of the decline in depressive symptoms found across the intervention and control groups. We organized questions within the guide around our primary process evaluation aims, which were to (a) explore how women who had participated in the RCT interpreted findings of group differences in health-related outcomes (with an emphasis on depression outcomes), (b) identify what aspects of the study were considered most valuable and/or least desirable by RCT participants, and (c) identify any additional intervention components RCT participants felt they needed to further enhance their health and well-being. Examples of questions include the following:

• [Following a brief explanation of a graph showing changes in depression over time, by group] “I need your help to understand these findings. Why do you think women in the control group had such a steep decrease in depression, when they did not receive the intervention during that time?”

• “Tell me about your experiences related to being in the CBPR study in general—what was good, what was not good? Describe anything that comes to mind.”

Sample

We obtained a purposive sample of prior RCT participants, which included women who were randomized to the intervention and control groups within the two counties the RCT was conducted. To be included in the study, women had to have completed their final follow-up from the RCT. Enrollment and all follow-up in the RCT took place between February 2007 and April 2010, with recruitment for focus groups beginning in early June 2010. As such, the time from RCT completion to being contacted for possible focus group participation ranged from a few months to approximately 2 years. We initially attempted to contact eligible prior RCT participants by telephone. Given the housing and telephone access instability in this population, we contacted participants who had completed the study within the prior 6 months first. We sent a letter to each of the RCT participants who could not be reached by telephone explaining the focus group study and how to contact the research team if interested in participating.

Participants in the focus groups ranged in age from 20 to 53 years with a mean age of 38 years. More than half of the participants (81%) self-identified as Black. Close to half (52%) were single, and slightly more than half (61%) were unemployed. Approximately 84% were from the urban county the RCT was conducted in, whereas the remaining 16% were from the more rural area.

Data Collection

We received approval from the affiliated university’s institutional review board prior to beginning the study. Participants received a $25 gift card to a local retail store and an additional $5 gift or gift card for their participation, and a light lunch was provided as the discussion took place. Five focus groups were conducted with 31 participants from the RCT: 20 from the control group, and 11 from the intervention group. Greater representation by the control group was not sought a priori through our sampling strategy. We believe this difference in representation occurred because fewer women in the control group were employed, and more women were therefore available to attend the focus groups. Each focus group had between 3 and 12 participants. Focus group moderators included the first author (and study principal investigator), the third author (a faculty member who had been an intervention nurse on the RCT while a doctoral student), the fourth author (a doctoral student in nursing), and a community member RA who had also worked on the RCT. Each focus group was held at a convenient time and location for the greatest number of participants as possible.

The first two focus groups were composed of all control participants, the third included all intervention participants, and the last two focus groups included both intervention and control participants. Each focus group discussion lasted approximately 80 minutes. Discussions were digitally audio recorded, transcribed verbatim by a professional transcription service, and verified for accuracy by members of the research team. A member of the research team collected field notes during the sessions, and researchers constructed reflexivity statements prior to beginning data analysis. By the end of the fourth focus group, data saturation had been reached. We held one additional focus group (Focus Group 5) in the rural county where we conducted the RCT to identify any new or different information from that found in the urban site.

Data Analysis

We used dimensional analysis and a constant comparison approach to analyze data. All authors participated in the data analysis process under the guidance of the second author, who has expertise in qualitative methods. Developed by Schatzman (1991), the goal of dimensional analysis is to construct or reconstruct the salient dimensions of a complex social phenomenon—in this case, whether and how CBPR and/or tailored intervention components of the RCT impacted perceptions of the study and/or study outcomes. Dimensional analysis has its origins in grounded theory; however, as a distinct method, it emphasizes understanding the complexity of phenomenon rather than developing a unified theory. Moreover, dimensional analysis is characterized by an explicit analytic process, whereby the researcher delays engaging in constant comparisons until the full complexity of the phenomenon is reflected in the data (Bowers & Schatzman, 2009). Using this approach, we moved progressively from identifying dimensions in the raw data, to developing conceptual categories, to identifying relationships among the categories, and finally to a more dynamic, comprehensive conceptualization of how participants interpreted their experiences in the study and responded to those interpretations.

We included data from transcripts, field notes, reflexivity statements, and audio recordings from each focus group in the analysis. The fourth author coded the transcripts after they were entered into NVivo 9.0 (QSR International, 2010), and the third author hand coded the data. Members of the research team independently read through transcripts, conducted line-by-line open coding, and met to discuss coding, raise questions, and identify salient dimensions. We then took the preliminary coding to an interdisciplinary qualitative data analysis group for peer debriefing and to start categorizing the dimensions using a card sort method (Lincoln & Guba, 1985). This process enabled us to verify salient dimensions and the relationships among them with the help of other researchers who were not associated with the study. Once the codes were categorized, we went back to the transcripts to perform more axial-focused coding.

We continued to meet weekly to discuss and refine the codes and their relationships. During these meetings we developed, discussed, and presented several conceptual models to the qualitative data analysis group for feedback and discussion. Finally, after the study team had analyzed and generated an initial set of findings, we asked three consultants to review the transcripts and provide their interpretations of the data relative to the findings by the primary analysis group. Given that the backgrounds of women in the sample differed from those on the primary analysis team by race (81% of the sample was Black, and all primary analysis team members, White) and socioeconomic status, we wanted to ensure that the analysis represented the perspectives of women who shared these characteristics with the study sample. Two of the consultants were Black women: one was an established qualitative researcher with expertise in sociocultural influences on health behaviors among Blacks; the other was a former WTP client and had worked with the first author as a community-based RA on other studies. The third consultant was an established qualitative researcher who helped revise the conceptual model to maximize understanding and interpretability.

Findings

Overall, our findings suggest that women in the RCT had a number of experiences best described as transformative. Given the longitudinal design of the study, they had repeated encounters with key aspects of the study (i.e., the tailored health questionnaire and study staff). As such, their experiences were iterative, or cyclical, in nature. The responses of RCT participants to their experiences, particularly in the control group, likely influenced study endpoint outcomes by minimizing group differences over time.

We present the details of our findings using an explanatory matrix, which, within dimensional analysis, provides a structure for understanding how each dimension plays a role in constructing the phenomenon of interest. Explanatory matrices allow researchers to categorize salient dimensions within these areas: context, perspective, conditions, processes, and consequences (Schatzman, 1991). “Context” includes factors in the broader environment that influence the phenomenon of interest. Shaped by prior experience, findings organized within the “perspective” dimension explain the lens, or filter, through which the phenomena of interest are observed, interpreted, and experienced. “Conditions” represent dimensions that alter how processes occurring within the phenomenon operate. Finally, “consequences” represent the outcomes that result from interactions within and between the context, perspective, and conditions dimensions (Trotta, 2007). We provide narrative accounts as supporting data and/or exemplars throughout the description of the findings.

All three aims had some degree of conceptual overlap. As the focus group discussion unfolded, however, it became clear to the research team that some aims were more similar than others. Specifically, the first and second aims (i.e., to obtain interpretations of key study findings, and to better understand what aspects of the study were considered most valuable and/or least desirable) were more similar to one another than to the third aim (i.e., to identify any additional intervention components RCT participants felt were needed). Consequently, we focus on dimensions related to the first and second aims in both the discussion that follows and the explanatory matrix of study findings (see Figure 1).

Figure 1.

Participants’ Transformative Experiences over the 9-Month Study

CBPR = community based participatory research; RCT = randomized controlled trial.

Context: Welfare and Medicaid Policy, Health Disparities, and Science

In the United States, the structures for funding social services and medical care that people at the poverty level receive often shape how individuals experience care within these systems. The goal of welfare reform is to promote self-sufficiency and to create an opportunity to build human capital. However, social service and medical care agencies face a loss of federal funds if caseload-reduction requirements are not met. This threat and other work-related requirements set the tone for how welfare agency staff members often view, and subsequently treat, welfare clients (Fagnoni, 2001). Similarly, on the health care front, there are systematic differences in the type and quality of care individuals receive in the United States based on race/ethnicity and socioeconomic status (Smedley, Stith, & Nelson, 2003).

Research practices also influence how disadvantaged groups are treated. Eliminating health disparities has become a national goal, but the scientific standards for how to study interventions to meet this goal have changed little at the core. Although the increasing application of CBPR and other approaches by researchers has modified how studies are conducted among disadvantaged populations, many of the scientific design-related tenets to establish intervention efficacy have remained intact, with the gold standard continuing to be the RCT. The majority of women in both the RCT and the focus groups described here had not participated in any prior research studies; therefore, they did not bring perspectives into our RCT that had been shaped by personal experiences in other research studies. In addition, the women in our sample had few experiences with being randomly assigned to intervention and/or control groups, which is critical for understanding how this shaped their perspectives.

Taken together, these contextual factors were fundamental to the perspectives women brought into the RCT as both novice study participants and women labeled as welfare recipients. Their social experiences—their treatment at the hands of social service and health care agents as welfare recipients, and the absence of mistreatment in prior studies—informed their experiences coming into the RCT. Consequently, their experiences in relation to these contextual factors also influenced the conditions and processes within the RCT that could have affected study outcomes (most notably, the marked reduction in depressive symptoms in the control group).

Perspective: Living the Label of Welfare Recipient

We asked focus group participants to describe their thoughts about participating in the RCT, including what they liked and did not like. In the discussion that followed this open-ended question, participants compared their generally positive experiences of treatment in the study with previous experiences with personnel from agencies that share common features with the focus of the RCT (i.e., social service agencies and the health care system). Participants framed the discussion around their experiences as welfare/Medicaid recipients and how they had been treated as a result. Most of these experiences involved interpersonal and written forms of communication that characterized welfare or Medicaid recipients as stereotypically lazy, unmotivated, and/or undeserving. The following statement gained support from other focus group participants and was reflected across group discussions: “Oh, they think that most of the people that’s getting the money is people that are in the ‘hood—the ghetto—that don’t want to work, don’t want to do nothing, that’s why they treat us like that.”

When participants described their study experiences, they repeatedly did so in relation to their prior experiences within these agencies, where they were treated by others with disrespect, disregard, and disinterest. As described by one participant, “It’s like when you first started [the WTP], they [WTP staff] want to take your respect from you.” Another participant added:

Case managers [WTP staff] seem to have, like, a perception that they’re better than you or they’re giving you something out of their pockets, so they have a way of making you feel really worse or lower than you already are [feeling].

Recalling her interactions with WTP staff, another participant described this particularly emotional encounter:

I want to tell you just from what I dealt with a little over two years ago, they [WTP staff] are hateful. I just know that what I went through personally, I have walked out of there [social service office] crying. They degraded me… . It was just unbelievable how they treated me.

Women in the intervention group extended their comparisons of study-related and other experiences to their treatment in other health care settings. These comparisons included negative experiences with nurses in the past. One participant gave more detail, indicating she knew what was expected of her during interactions with health professionals. In essence, she had come to expect being treated with indifference in these settings, as in the following:

You know when you go to the doctor’s office and you are feeling bad and you got to answer all those questions sometime doctors belittle you and make you feel low, the nurse I had [referring to the PHN study intervention nurse] didn’t do that. I’ve been in and out of the hospital a lot so I’ve had a lot of dealings with nurses and the rudeness that you get from them [it’s] as if you know they don’t want you to say [anything]—you’re not supposed to say anything. The only thing you’re supposed to be [is] seen and not heard.

In contrast, participants described their interactions with study personnel as positive and directly compared them to their more negative encounters with others. This finding was consistent across RCT group assignment and study staff roles, whereby intervention group participants interacted far more with the PHN (with essentially unlimited access), and control participants interacted with the community-based RAs only on the occasions of baseline, 3-, 6-, and 9-month data collection. As such, interactions in the RCT were generally dependent on group assignment; however, each participant was recruited by one of two community-based RAs (i.e., women who had previously been through a WTP, and hired onto the study), and all study participants had gone through a study eligibility screening process with the PHN to ensure a chronic health condition was present prior to being randomized.

Participant descriptions of their experiences with the study RAs were overwhelmingly positive and conveyed the importance of communication. One participant stated, “She [the RA] is a communicatin’ person. She will communicate with you very well, and if you don’t understand it she will explain it to you.” Participants also described positive accounts of interactions based on what study staff members did not do, such as “[The study RA] never [gave] you that feeling like you’re imposing or none of that,” and “She [the RA] wouldn’t never, say, interrupt me or say anything disrespectful or nothing.”

When discussing their experiences with the PHN, intervention group participants provided numerous examples of the PHN communicating with concern, respect, caring, and without making preconceived judgments about them based on their receipt of welfare. To optimally manage symptoms from chronic health conditions, PHNs often addressed the need for participants to obtain follow-up care and make difficult health-related behavioral changes. When asked by the focus group moderator whether PHNs were perceived as irritating, unsupportive, or pesky at times when working with participants to accomplish these goals, one intervention group participant said,

None of the nurses came off like that. They came off like they really were concerned and they’re very caring. Their voice—I mean the way they talk to you—it’s like ‘I’m here for you regardless of what you’re going through I’m here for you.’

Echoing the importance of being nonjudgmental, during the same discussion, one intervention group participant stated, “They [the PHNs] were caring when they talked to ya, even though you expressed to them you’re not doing the things you was supposed to do.” Also emphasizing a nonjudgmental approach, a second intervention group participant added,

It was helpful what they [the PHN] was saying, we knew it was helpful, we knew what we should be doing and they found a way to tell us where, you know, I’m telling you what you should do but at the same sense I’m not putting you down or fussing at you.

Across group assignment, study participants described their interpersonal interactions with research personnel as a significant determinant of their overall positive experience. This dimension shaped how participants experienced the RCT and how those experiences might have affected outcomes. In addition, focus group participants described this dimension of “living the label” as having a direct effect on one of the outcomes of interest: the rapid decline in depressive symptoms over the first 3 months of the study, particularly in the control group. Participants explained that as they found jobs (approximately 55% began working within the first 3 months of RCT enrollment), they were less dependent on the welfare system and had less and less contact with WTP personnel. This decline in contact meant they experienced the psychological distress that often occurred during their interactions with WTP staff less frequently. When asked why moving into employment might have improved mental health, many responded with descriptions similar to the following: “because you wasn’t depending on them [WTP/welfare caseworkers], you wasn’t seeing them every day.” This type of response also meant they were able to distance themselves from living with the active label of welfare recipient.

As presented in the next section, women also provided more nuanced descriptions of factors specific to the CBPR components of the primary study design that were likely to have influenced outcomes in the control group. Collectively, these nuanced findings inform our understanding of how the approaches we use as nurse investigators might have unintended—and difficult to detect—consequences for outcomes. We describe these additional, study-specific findings as conditions and processes in the following section.

Conditions and Processes: CBPR, Tailoring, and Randomization Influences

Condition-related dimensions in an explanatory matrix represent factors that affect the way central processes within the phenomenon of interest operate. Given the difficulty in representing these dimensions as discrete entities, we present the conditions together with the descriptions of processes to convey the tacit, but highly interdependent, relationships among them. Salient conditions that affected participants most were of two types: those that were more static and unchanging over the course of the study, and those that tended to shift. Conditions considered more static included the tailored health screening questionnaire, study staff characteristics, and group assignment. In contrast, participant understanding of randomization and expectations for help were conditions that tended to shift during the course of the study. Process-related dimensions included being engaged with the aspects of the study (with staff and the questionnaire), increasing self-awareness/reflection, and taking action on one’s own behalf. Taken together, participants described these dimensions as collectively influencing RCT outcomes.

In addition to being treated with care and respect by study staff members, participants also readily identified the health screening questionnaire as prominent in their experience. As mentioned previously, this questionnaire was tailored by women in WTPs with the express intent of being completed by the same population during the RCT (see Lutz et al., 2009, for a detailed description of the tailoring process). Constructing the questionnaire to be sensitive to the unique circumstances of women in WTPs, to be relevant to the target population, and to convey respect for the women answering it was our primary goal in the study. As such, study participants felt a greater sense of respect and sensitivity from this questionnaire than from the materials they had routinely received during encounters with social service and/or health care agents. The questionnaire contained both valid and reliable clinical screening and other research measures, questions that were typical for a routine health assessment in a primary care office, and additional questions to reflect areas of health viewed as important by women in WTPs (i.e., current stressors and approaches to handling stress). It was completed for screening purposes at baseline and at the 3-, 6-, and 9-month follow-up visits both to collect data (in both the control and intervention group) and for the PHNs to monitor health status and guide their case management interventions with the intervention group.

Engagement and self-reflection.

Across focus groups, the health screening questions were repeatedly described as eliciting intense self-reflection as women engaged the questionnaire to complete it prior to their follow-up visits, although the intensity of these processes varied across participants. Engaging with the questionnaire took the form of directing their full attention to the questions being asked, opening up to the questions, and interacting with them. As one participant succinctly put it, “you had to sure enough dig deep.” One participant stated, “It made me think … about things that you probably would have pushed aside. It helped me, like, get focused on some things that were important that I was neglecting in my life.” Another gave a more detailed account of how she carefully considered the response options across items during this process, stating, “It just really made me, like, think more about questions that I sometimes would not put myself in that category [referring to an item response option], but did I really belong there?”

In some cases, the degree of engagement that occurred when completing the questionnaire and the cognitive and emotional responses that it provoked were significant enough that participants said they psychologically prepared themselves to complete it. One participant described how she arranged to be alone to put herself “in that thought frame of mind from the questions and … be able to work on it, deal with it.” Other participants described their interactions with the questionnaire in therapeutic terms. One participant stated, “[I opened] my feelings up on how I feel and I don’t normally do that a whole lot. I put every little problem that I had down. Every little meaning of how I felt.” Others directly referred to the process of completing the questionnaire as “therapeutic,” stating “after I completed it [the questionnaire], I went over it and I kind of rediscovered myself… . I found the packets to be therapeutic.”

Randomization influences.

Whether the questionnaire was considered helpful or therapeutic was, for some women, conditional on their group assignments, their understanding of randomization, and their expectations for receiving help during the study. RCT participants in the intervention group, who knew they could process the feelings that were elicited and the problems or questions that surfaced while completing the questionnaire with the PHN during the intervention, framed this experience more positively. However, one participant who understood that being in the control group did not offer immediate intervention described feeling angry when completing the questionnaire:

I know one time I even got angry because one of the questionnaires made me angry and I went in there [the study office] angry like [the community-based RA] had did it but, you know, but it really wasn’t her but it was [the questions].

Several participants in the control group affirmed this experience. Participants whose understanding of randomization was consistent with the intent (i.e., that they were not receiving the intervention, or help, from their perspective) described their responses to completing the questionnaire not with anger but removed indifference, stating,

I guess I was always expecting to get some kind of feedback, and that never happened. I guess what I was looking for was for more—and I didn’t know whether it would come from either of the groups… . We’re [control group participants] all kind of on our own and just waiting to see what the outcome is or to see if someone contacts us or whatever.

During the same focus group discussion, an intervention group participant responded by directly contrasting her experience to that of the control group member, stating, “I never was left with that open, void space—not knowing what to expect from whatever was going on.”

Taking action.

More frequently, control group participants described how they began taking action on their own behalf to constructively deal with the feelings and issues that surfaced over time when completing the health questionnaire. One participant described this process as “[Saying to myself] I’m getting this [the control assignment and the questionnaire], I have to make it ‘cause it’s all I got. So, that helped too.” Another framed how she dealt with being in the control group in survival-like terms, stating, “Another way of saying it is, we all learned to adapt. It’s like a natural mechanism when we—that fight or flight thing—when we realize we don’t have anywhere else to go, we fight for [ourselves].”

Many women in the control group reported they took it upon themselves to seek out the help they needed and/or make changes in their lives after they had completed the health questionnaire and reflected on their current situations. These changes occurred over time as they completed the questionnaires at baseline, 3, 6, and 9 months during the course of the study. One participant described how she changed her approach to situations as a result of this self-reflective process:

It was like, okay, well, have you thought about this today, and how are you feeling, and when you’re mad is it because of everybody else or is it because of you, so it just made me think about stuff even in the future. Like before I would yell at my kids, be like, okay, well, think about how they’re feeling and why they’re mad.

The strategies women in the control group used to take action also seemed to be conditional on study staff characteristics and the women’s expectations of help available from others. Control group participants consistently met with an assigned community-based RA who had recently been in the study participants’ situation (i.e., recently in the WTP) and, as described previously, treated participants with care and respect. In many cases, the control group women leveraged whatever interpersonal help they could get from the community-based RAs. One participant stated,

And even though [I] was [in the] control group you still got to figure out how do I move forward? … [The community-based RA] gave me a lot of insight to feel like, okay, I really don’t need this intervention part, I really can work this right here.

Often, the control group participants voiced frustrations or worries, or reflected on their current or past situations at the data-collection visit. One control group participant noted the community-based RA was “always ready to listen, no matter what.” Another more specifically described how the interactions with the community-based RA were helpful:

Every time I went to go see her [the community-based RA] I was in more of a cheerful mood. I knew what to expect. I knew she was going to sit there and be honest with me and try to make me feel better because most of the time, when I went there, she always knew that my nerves was the worst.

The community-based RAs received the women’s cues that they needed to talk with an openness to listen. One participant recalled a situation in which she confronted the community-based RA over being in the control group and not being able to process the emotions that arose from answering the study questionnaire:

Miss [name of community-based RA], ya’ll making people do things like this right here [completing the questionnaire] and there’s nobody to talk to. How you going to do people like that? And she … fought my attitude with niceness, you know what I’m saying, and I was like, Miss [name of community-based RA], this is bull. How you have people answer questions like this and nobody to talk to? And she was like … let’s talk.

In other cases, control group participants took action on their own behalf by reaching out to friends and/or family members in new ways. One participant clearly described how, through self-reflection after completing the health questionnaire, she sought help from others and received it:

It [the health questionnaire] made me go to other people, like family members. I remember one of the questions being if you needed a ride or if you needed someone to talk, do you have someone that you could go to. In the past, like I was marking no, no. I’m just thinking like nobody cares. Everybody’s got their own stuff going on, and then once I realized I was like, well, maybe I should just talk to my aunt or ask somebody to see what would happen. Then she was like oh, yeah, you can come over and talk to me or oh, yeah, I’ll give you a ride and stuff that I would have never done. It made me, like, step out and do it.

Time and repeated exposure.

The experiences women described during the RCT were iterative and took place over 9 months from study enrollment to the final follow-up visit. As participants completed the health questionnaire, existing needs surfaced and new needs for self-care were identified. Coming into the study, women brought a particular perspective that informed their expectations for receiving help, which are best described as guarded hope given their prior (oftentimes negative) experiences of treatment by others. Once enrolled, their expectations for help depended on their understanding of randomization and interactions with study personnel. As their understanding of what it meant to be in their assigned group (intervention or control) was clarified over time and as they interacted with both the questionnaire and study staff members on different occasions, their expectations, and perceived potential, for help aligned with their situational (i.e., group assignment and other life context) realities.

As a whole, women in the focus groups described a complex process that was transforming for them in different ways. The first author was aware from anecdotal evidence gathered in prior studies with women in WTPs that they did not gloss over health-related research questionnaires but instead took them seriously. At the time, the first author thought the women were trying to answer questions as carefully and accurately as possible, but did not fully appreciate the extent to which the experience of answering questions would prompt RCT participants to action or potentially influence study outcomes. In conjunction with the other dimensions described here, the level of increasing self-reflection in response to the health questionnaire and the strategies women used to take action on their own behalf began to explain how CBPR study-related factors might have affected select group RCT outcomes we had observed over time—most notably, the steep decline in depressive symptoms early in the study in the control group. In the context of these and other findings, we revisit the study process and endpoint outcomes in the next section as consequences.

Consequences: Study Process and Endpoint Outcomes

One of our aims in this focus group study was to explore participants’ experiences in the RCT to better understand what helped us achieve positive outcomes in the study processes (e.g., study recruitment/enrollment/retention), and what we might need to alter in future studies. A more specific aim was to better understand how RCT participants explained the rapid decline in depressive symptoms among women in the control group, including what study-related and other factors were believed to have contributed most. The condition- and process-related dimensions identified by women in the focus groups clearly indicate that instrument effects likely precipitated increased self-reflection and awareness over the course of the study, as well as behavioral change, such as taking action to improve health. Women in the RCT maximized the resources available to them to improve their health by strengthening social support, obtaining information, and/or making needed, and previously deferred, health care appointments. Many or all of these process-related dimensions were expected as a result of the PHN intervention. The degree to which the control participants took action to improve their health on their own, however, was unanticipated, and might have plausibly contributed to the unexpected steep drop in depressive symptoms over time in that group.

Discussion

Our primary finding from this study was that many women in the RCT had transformative experiences, albeit in different ways. Second, both CBPR and non-CBPR aspects of the study design likely influenced not only how women experienced being in the study, but also how they responded to those experiences. The positive experiences of participating in the study ran counter to the women’s oppressive experiences of being labeled a welfare recipient. The CBPR approach to this research and the interpersonal communication styles exhibited by the study staff members facilitated the sensitive and respectful treatment and positive experiences women had in the RCT. The negative experiences voiced in this study related to the RCT design (primarily to their control-group experience). Whether and/or how a CBPR approach and RCT design can and/or should coexist are areas in need of additional philosophical, conceptual, and methodological consideration. To better understand the nature and scope of the tensions inherent in socially transformative methodologies, such as CBPR and the study processes prescribed by designs representing the scientific canon, additional process evaluations such as the one presented here are needed.

Much of what we learned contributes to what is currently understood about instrumentation effects (French & Sutton, 2010). By describing the instrument and nonintervention-focused social treatment mechanisms involved in producing cognitive, emotional, and behavioral change among women in our control group, we are able to better understand what unanticipated effects might occur in our research, how they occur, and why. Instrument reactivity has been documented across a broad range of studies, has led to behavioral change in many instances, and in some cases follows a dose–response relationship (French & Sutton, 2010; Hughes, Keely, Fagerstrom, & Callas, 2005; Rubin, Boals, & Klein, 2010). In the nursing intervention literature, previous psychosocial questionnaire administration and the timing of the administration have demonstrated independent, therapeutic effects among women recovering from miscarriage (Swanson, 1999). Although it is not surprising that we found women reacting to the items on the health questionnaire, we did not expect the intensity of reflection and the transforming experiences that were initiated by completing the questionnaire. The extent to which instrumentation effects might have been amplified as a result of the CBPR-related tailoring of the health questionnaire, however, was not well represented in our data and remains unclear.

Our findings generally parallel those from other studies in which researchers have examined women’s experiences in RCTs. Others have reported that the interpersonal communication skills of study staff members are critical for enrollment, retention, and intervention efficacy (Beal et al., 2009; Jackson et al., 2010; Slade et al., 2010). In a study by Beal and colleagues (2009), women in a fibromyalgia symptom management RCT attributed their positive outcomes at least in part to viewing that the interventionists had credibility, given they themselves had fibromyalgia. The authors noted that a role-model effect might have produced this outcome. This same role-model effect seemingly occurred in our study, in that several women mentioned not only that they appreciated how respectfully they were treated by the community-based RAs (who also had the shared history of being in a WTP) but that they looked up to them, and they framed them as examples of what they aspired to achieve. Also similar to our findings, studies indicate that a key factor among women considering study enrollment is the possibility of personal benefit; however, the understanding of randomization, as well as the purpose of a control group, varies widely (Jackson et al., 2010; Slade et al., 2010; Smyth, Jacoby, & Elbourne, 2011).

Because the majority of our focus group participants were Black women, their experiences of racially driven oppression might have contributed to our findings in unique but subtle ways. Black women have a long history of drawing on their inner resources and self-determination to overcome whatever obstacles necessary to result in a positive self-image (Ladner, 1995; Poussaint & Alexander, 2000; Taylor, 1999). As such, this tendency might have played a role in the extent to which women engaged with the health questionnaire to use it as a guide, or tool, for self-care. Moreover, for women in the control group, the ability to express their situations, concerns, and worries with the community-based RAs reflects some of the features of storytelling. Studies have shown that storytelling has had both therapeutic and emancipatory outcomes among diverse groups of women (Banks-Wallace, 1998; Grassley & Nelms, 2009). Moreover, for the Black women in our study, sharing stories with the community-based RAs (who were also Black) might have had a more novel effect or significance, because storytelling in the Black community is rooted in the oral traditions that date back to African slaves in early U.S. history (Banks-Wallace, 2002). Although perceived as a positive experience among study participants, this interaction might have introduced an unintended intervention into the control condition. Because researchers conducting CBPR studies often hire community members for specific research roles and to address the needs of disadvantaged, often minority, groups, cultural nuances might benefit or adversely affect the outcomes of these studies.

We should point out potential methodological weaknesses of our study. First, research staff members, rather than a more neutral (and/or unfamiliar) third party, conducted the focus groups. Although this might have led participants to place more emphasis on the positive experiences in the study, participants generally provided very positive feedback about the study at their final (9-month) follow-up throughout the RCT. In addition, during focus group discussions, control group participants did voice their dislike of not having an opportunity to talk about the issues raised in the questionnaire. Second, we were unable to take our findings back to focus group participants or other RCT participants for member-checking purposes, although whether this is considered a study limitation remains highly controversial (Angen, 2000; Rolfe, 2006; Sandelowski, 1993). There were, however, women involved in the data analysis phase of the research who shared characteristics with the study sample in terms of race, socioeconomic status, and welfare receipt.

In our RCT, tailoring the health screening questionnaire and involving members of the target population in the research as hired study personnel were driven by the CBPR approach to our research. As others have found, the use of CBPR in our RCT appeared to clearly benefit certain aspects of internal validity (i.e., recruitment, retention); however, women in our focus groups described less desirable influences on other aspects of internal validity (i.e., potentially exaggerated measurement effects because of tailoring). The process of completing the health questionnaire clearly served as an intervention in its own right, because it influenced the cognitions, emotions, behavior—and possibly study outcomes—of control participants. Although there is very little in the literature describing socioeconomic differences in measurement effects, findings from a study with predominantly White women by Clayton and colleagues (1999) suggest that disadvantaged women interact to a greater degree with health-related questionnaires when completing them than do their more advantaged counterparts. As CBPR is increasingly applied to studies with underserved, disadvantaged, or otherwise vulnerable populations, studies on whether specific aspects of CBPR-related designs impact study outcomes, how they operate, and the magnitude of the effects are needed. Without such studies, we risk not only generating a body of evidence that attenuates potentially promising intervention effects but also minimizing the social value of implementing the interventions needed to reduce health disparities.

Bios

Shawn M. Kneipp, PhD, ARNP, is an associate professor at the University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA.

Barbara J. Lutz, PhD, RN, APHN-BC, is an associate professor at the University of Florida, Gainesville, Florida, USA.

Catherine Levonian, PhD, MPH, RN, is a professional development specialist at Thomas Jefferson University Hospital, Philadelphia, Pennsylvania, USA, and a recent graduate of the University of Florida Doctor of Philosophy in Nursing Science program.

Christa Cook, MSN, RN, APHN-BC, is a candidate in the Doctor of Philosophy in Nursing Science program at the University of Florida, Gainesville, Florida, USA.

Jill B. Hamilton, PhD, RN, is an assistant professor at the University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA.

Dawne Roberson is a research assistant and community-based participatory research advisor to the first author at the University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA.