Using Integrated Mixed Methods to Develop Behavioral Measures of Factors Associated With Microbicide Acceptability

Abstract

Our current understanding of factors associated with microbicide acceptability and consistent use typically has been derived from separate and distinct qualitative or quantitative studies. Specifically, rarely have investigators used mixed methods to both develop and validate behavioral measures. We utilized an integrated mixed methods design, including qualitative metasyntheses, cognitive interviews and expert reviews, psychometric evaluation, and confirmatory qualitative analyses of the correspondence between quantitative items and original qualitative data to develop and validate measures of factors associated with microbicide acceptability and use. We describe this methodology and use the development of the Relationship Context Scale to illustrate it. As a result of independent confirmatory analyses of qualitative passages corresponding to survey items, we demonstrated that items from the same subscales are frequently double coded within a particular textual passage, and thematically related, suggesting associations that resulted in a unique factor structure within the subscale. This integrated mixed method design was critical to the development of this psychometrically validated behavioral measure, and could serve as a model for future measure development.

Because women represent nearly half of all new human immunodeficiency virus (HIV) infections (Joint United Nations Programme on HIV/AIDS [UNAIDS], 2008), an HIV prevention method that can be initiated and potentially controlled by women is a global health necessity. Microbicides are products being developed to reduce the risk of sexual transmission of HIV. Even though several topical vaginal microbicides have reached effectiveness trials (AIDS Vaccine Advocacy Coalition, n.d.; Global Campaign for Microbicides, n.d.), only one has demonstrated promising results (Abdool Karim et al., 2010), and some trials have been marred by less-than-adequate adherence to study product use. As a result, incorporating behavioral research into clinical trials that can target adherence to and acceptability of products has been identified as “an utterly critical priority” for microbicide development research (Microbicide Development Strategy [MDS] Working Groups, 2006, p.10).

Because successful uptake and consistent use of these prevention products will be critical to their effectiveness and, ultimately, their impact on the HIV pandemic, behavioral and social scientists have explored and identified psychosocial factors hypothesized to be associated with user acceptability, willingness to use products, and other factors essential to consistent and correct microbicide use (AIDS Vaccine Advocacy Coalition, 2007; Morrow, Fava, Rosen, Christensen et al., 2007; Morrow, Fava, Rosen, Vargas et al., 2007; Morrow & Ruiz, 2008; Severy, Tolley, Woodsong, & Guest, 2005; Wang et al., 2008). More recently, given the lessons learned in first-generation microbicide efficacy trials, those factors are being reconsidered with respect to their impact on adherence to product use, both in the context of and, eventually, subsequent to, clinical trials (Morrow & Ruiz; Tolley et al., 2010). It is essential to understand both who will use microbicides and what factors in their lives and relationships will motivate, support, and maintain that use.

To date, microbicide researchers have assessed acceptability with users of candidate microbicides in clinical safety trials (e.g., Bentley et al., 2004; El-Sadr et al., 2006; Jones et al., 2009; Morrow et al., 2003), by trial participants’ partners (e.g., Carballo-Dieguez et al., 2007), through surveys of hypothetical product attributes among potential users (e.g., Coggins et al., 2000; Holt et al., 2006; Morrow et al., 2006), and by using surrogate products or placebo gels similar in basic characteristics to a microbicide (e.g., Coggins et al., 1998; Hammett et al., 2000). Often, investigators gather both quantitative and qualitative data. Nevertheless, even when both methods have been used, the methods have remained effectively distinct: one primary (usually the quantitative), and the other (often the qualitative) secondary, collected to provide context or explanation for the primary method. The data in such trials are designed to be complementary. However, the methodologies and data analyses often largely remain distinct from each other. We employed a model to deliberately integrate the qualitative and quantitative elements, which are equally important to the measure development process.

Four classic designs for blending qualitative and quantitative methods, emphasizing the complementarities of the approaches, have been articulated. In each design, either the qualitative or quantitative approach serves as the principal methodology, whereas the other is used as a complementary, either preliminary or follow-up, method. The four designs are (a) preliminary qualitative methods in a quantitative study (e.g., formative participant observation, focus groups, or in-depth interviews prior to developing a behavioral intervention); (b) preliminary quantitative methods in a qualitative study (e.g., a preliminary survey to define the sample and/or identify research questions prior to ethnographic research); (c) follow-up qualitative methods in a quantitative study (e.g., exit interviews or focus groups with intervention participants after a trial is completed to assess acceptability or feasibility); and (d) follow-up quantitative methods in a qualitative study (e.g., subpopulation surveys to determine if the qualitative findings can be generalized to other populations; Morgan, 1998; Ulin, Robinson, & Tolley, 2005).

In mixed methodologies, the strengths of quantitative research, including measurement, generalizable samples, experimental control, and effective statistical tools, can be combined with the strengths of qualitative research, such as the in-depth examination of the context and narrative of complex behavioral and/or social repertoires, or what Miles and Huberman (1994) called “the up-close, deep credible understanding of complex real-world contexts” (p. 42). Innovative strategies for the effective integration of methods and analyses, as opposed to the more complementary approaches suggested by Morgan (1998) and others, often have not been as well articulated, and are only recently being explored in more pedagogical discussions. In this article, we describe an integrated mixed methodology that began with a metasynthesis of qualitative data that then was used to generate draft quantitative survey items. Those items, as well as their context and placement within the complete survey, were refined by cognitive interviews (CIs) and expert review, resulting in a survey that was administered to 531 participants. Our psychometric analysis of the survey data led us to develop several scales. Survey items from one of those scales became coding categories used in a thematic reanalysis of the original qualitative data designed to provide confidence in, and explanatory evidence for, the resulting scales.

In this methodology, we thoroughly integrated qualitative and quantitative steps: We used qualitative analysis to create quantitative survey items. We vetted those survey items via CIs in which participants provided responses to the quantitative items and then explained their responses in semistructured qualitative discussion that helped to further refine the survey items, as well as their framing and ordering. In some cases, those discussions also led us to include additional constructs or items in the overall survey. As a result, additional and novel qualitative data collected by research staff in the course of CIs were integrated into the process. After quantitative analysis of the resulting survey items was complete, the qualitative data were reexamined using the content of the quantitative questions from the scales as coding categories. This allowed us to explore the correspondence between the psychometrically validated subscales and the qualitative data. We refer to this methodology as “integrated mixed methods,” and view it as a “full-circle approach.” It allows us to use qualitative data analysis methods to generate and clarify quantitative survey item generation, then further verify those quantitative concepts (i.e., subscale constructs) with the original qualitative data (see Figure 1).

Figure 1.

The full-circle integrated mixed methodology used in the Phoenix Project

*Meta synthesis process also included accommodating existing scales and literature regarding similar constructs into our thinking.

**Scale k represents any additional scales after Scale 3.

Although many social and behavioral scientists would relish the chance to qualitatively examine the patterns of product use and complexities of circumstance that lead women to use or not use biomedical HIV prevention products in the context of clinical trials (and beyond), the thousands of participants required in efficacy trials obviate comprehensive qualitative acceptability studies as prohibitively expensive and time consuming. Even significantly smaller subpopulation studies are often prohibited by funding and timeline constraints in clinical trials, as well as participant burden concerns. Therefore, acceptability assessment in clinical trials is likely to rely more on quantitative measurement. Many of the measures currently used have not been psychometrically validated for use in microbicide trials (or with the specific populations enrolled in those trials), and minimize examination of the context of use. To meet the clear need for validated quantitative measures of microbicide acceptability, we investigated context-based models of willingness to use microbicides, developed instruments measuring factors hypothesized to be related to microbicide acceptability, and explored associations between person-, product-, and context-related factors related to acceptability (Morrow, Fava, Rosen, Christensen et al., 2007; Morrow, Fava, Rosen, Vargas, & Barroso, 2008; Morrow, Fava, Rosen, Vargas et al., 2007). This investigation became known as the Phoenix Project, and is referred to as such throughout this article.

Morse noted (2009) that two different qualitative methods can together constitute “mixed methods,” and Sawyer, Deatrick, Kuna, and Weaver (2010) provided an example of a longitudinal mixed method design that nested quantitative methods within a more comprehensive qualitative method. In the case of the Phoenix Project, we utilized a “full-circle” approach, incorporating qualitative metasynthesis, mixed method CIs, psychometric evaluation (i.e., quantitative methods), and qualitative reanalyses. In this article, we present the integrated mixed methodology we employed in the Phoenix Project to develop and validate behavioral measures, and illustrate that methodology using one of the resulting scales (the Relationship Context Scale).

Study Design and Methods

Vaginal microbicides do not currently exist, yet they must be designed to be both effective and acceptable to develop products women and their partners will consistently and willingly use. To develop an acceptability measure (or set of measures) for use in clinical trials, we integrated four distinct methods of research into a measurement development and evaluation process. First, we conducted a formative qualitative metasynthesis on existing data from previous studies examining microbicide acceptability (Bentley et al., 2000; Mason et al., 2003; Morrow et al., 2003). We used those results to identify and conceptualize salient themes and constructs of interest in the measurement development process. We also reviewed then-available measures of similar or overlapping constructs and domains of interest. Second, we generated potential survey items from the metasynthesis results—phrased as agree/disagree statements in Likert format—and refined those items via CIs. A panel of experts in biological, psychosocial, anthropological, and psychometric aspects of microbicide use and measure development also reviewed those items. Based on the analyses of the CI data, we finalized the survey items, their order, and the framing of questions or sections (i.e., sets of items), and administered the survey to a cross-sectional sample of potential microbicide users. Our analyses led us to develop several psychometrically validated scales: the Willingness to Use Microbicides Scale (Morrow, Fava, Rosen, Christensen et al., 2007), the Important Microbicide Characteristics Scale (Morrow, Fava, Rosen, Vargas et al., 2007), the Microbicide Confidence Scale (Fava et al., 2010), and the Multi Dimensional Risk Evaluation Scale (Morrow et al., 2010), as well as the Relationship Context Scale (Morrow et al., 2008). Finally, we conducted a confirmatory qualitative reanalysis of the original transcript data used in the qualitative metasynthesis to provide explanatory evidence for, and evaluate credibility of, the resulting scales, using subscale items as coding categories. This integrated mixed method strategy is illustrated in Figure 1.

Formative Qualitative Metasynthesis

We began the Phoenix Project by conducting a metasynthesis of qualitative data previously collected in microbicide acceptability studies for the purposes of identifying critical constructs and generating survey items that accurately reflected the perspective of microbicide users. Because we could not interview current microbicide users (there are none, because no product has been approved), we instead assessed the needs of potential users by reviewing the comments of clinical trial participants and a category of women known to be at high risk for HIV transmission from their sexual partners. We obtained permission to use deidentified data from the principal investigators of two Phase I microbicide clinical trials, as well as a study of acceptability of vaginal products among high-risk women (i.e., potential microbicide users; Bentley et al., 2000; Mason et al., 2003; Morrow et al., 2003). HIVNET 009 (HIV Network for Prevention Trials 009) was a safety and acceptability trial of BufferGel (Reprotect, LLC) completed by 27 low-risk participants in the United States, 14 of whom participated in qualitative in-depth interviews (IDI) or focus groups (FG). HPTN 020 (HIV Prevention Trials Network 020) was a Phase I clinical trial of Pro2000/5 (Indevus) in which a total of 30 U.S. participants completed semistructured IDIs or FGs. The final study assessed the acceptability of several “surrogate” vaginal formulations among a cohort of high-risk, drug-involved women who used either heroin or cocaine, or who were the sexual partners of injection drug-using (IDU) men. Sixteen heroin users, 15 cocaine users, and 14 sexual partners of IDU men participated in six U.S. focus groups. In all, we reanalyzed 32 group discussions and interviews representing the comments of 89 women as part of the Phoenix Project’s qualitative metasynthesis.

As Thorne, Jensen, Kearney, Noblit, and Sandelowski noted (2004), “qualitative metasynthesis” is a comprehensive term applied to a variety of methodologies which are designed to develop relevant new knowledge from a careful reanalysis of previously collected qualitative data. In this project, we utilized qualitative metasynthesis to reexamine the comments of women participating in vaginal microbicide research to identify product use characteristics and person-in-context factors that we hypothesized would be related to acceptability. We designed the analytical approach for this qualitative metasynthesis for the specific purpose of generating an initial quantitative survey item set. We based the coding scheme for the metasynthesis on health behavior and health-promotion models particularly applicable to the unique issues facing at-risk women and microbicide use. These included the Social Ecological Model (Bronfenbrenner, 1979; McLeroy, Bibeau, Steckler, & Glanz, 1988) and the Theory of Reasoned Action (Ajzen & Fishbein, 1980). In addition, we considered historical models within reproductive health and contraception (Severy, 1999; Woodsong, Shedlin, & Koo, 2004), as well as then-current conceptualizations within the literature and emerging models of microbicide acceptability (e.g., Woodsong & Koo, 2002). Thus, we designed the coding scheme in an effort to capture such constructs as attitudes about the couple in a larger social context (and beliefs about what qualities exist in a “good couple”), attitudes and beliefs about referent others and social norms (about sex, for instance), and concepts of covert use and whether various microbicides could (or should) be used covertly.

Because two of the Phoenix Project investigators had participated in each of the original studies, we were able to access the original raw data transcripts, as well as initial analyses and reports. Although our intimate knowledge of these data facilitated the metasynthesis, the new coding structure differed entirely from the initial work. We sought, by reanalyzing the data across these 89 women, to use our understanding gained about risk behavior, descriptions of relationships, and even the language used in describing these risks and relationships, to develop quantitative survey items that were grounded in the actual experiences of potential microbicide users, i.e., women at risk for HIV.

The analytical codes included assessment of the participants’ risk behaviors (e.g., protected or unprotected sex, number of sexual partners, drug-related risk); relationship factors (e.g., “relationship demographics” [that is, characteristics of the relationship itself as opposed to the individuals, e.g., age discrepancy between partners, length of sexual relationship, desire for childbearing in the index relationship] and type of partnership); attitudes, beliefs, and perceived norms, which included the opinions of the participant, her family members, friends, and community members (e.g., about HIV prevention and typical sexual behavior); use-associated factors (e.g., leakage and lubrication effects on sexual pleasure and the possibility of covert use); self-efficacy; and willingness to use a product. At least two different researchers independently coded each original transcript. We rotated pairs of coders to reduce potential coding drift and enhance the equivalence of coding across researchers. We compared the two coded transcripts to ensure comprehensiveness of coding and identified and discussed discrepancies until we reached consensus. Phoenix Project staff entered coded transcripts into NVivo qualitative software (QSR International, 2002) to organize transcripts, as well as individual constructs and themes. Investigators completed summaries focusing on major themes and associations with other codes. We reviewed every summary and completed a detailed report representing each topic.

Item Development and Expert Review

We drafted potential quantitative items based on the results of the metasynthesis, representing all of the identified domains deemed relevant to our theoretical conceptualizations of microbicide acceptability. We also reviewed then-available measures of similar or overlapping constructs and domains. For instance, factors present in available condom use self-efficacy measures (e.g., Cecil & Pinkerton, 1998; Forsyth, Carey, & Fuqua, 1997; Smith, McGraw, Costa, & McKinlay, 1996) that we deemed equally critical in microbicide self-efficacy were modified or accommodated into the larger pool of items hypothesized to capture salient constructs in microbicide use self-efficacy. With respect to sexual relationship quality, we paid particular attention to constructs consistent with the long-standing contraceptive use literature (for example, the reviews provided by Severy, 1999, and Severy & Spieler, 2000), hypothesizing that the dynamics impacting contraceptive use might impact microbicide use in similar ways, especially if a microbicide were also shown to be contraceptive. Here again, we sought to ground the items about microbicide acceptability in relevant research and the available qualitative data. Thus, items consistent with some of those suggested by Severy (1999) were incorporated into the larger item pool (see Figure 2, all items revised per CI outcomes).

Figure 2.

Relationship context scale items

In each of the items, “[partner]” refers to the male sexual partner (of those reported), selected at random. A Likert scale was utilized as the response format: 1 = do not agree at all; 2 = agree a little; 3 = agree somewhat; 4 = agree a lot; 5 = agree completely; refuse to answer.

The total pool of potential items across domains and constructs was reviewed by a panel of five expert subject-matter reviewers, including two medical doctors (board certified in infectious diseases and obstetrics and gynecology); a quantitative psychologist; and two anthropologists with expertise in HIV prevention and microbicide research. Willis (2005) noted that expert appraisal or review “should always be carried out before cognitive testing” (p. 231). Based on the expert reviews, we completed several levels of revisions: the inclusion or exclusion of specific items or constructs; wording of items for comprehension, clarity, and fidelity to the construct; relevance of response formats; ordering and framing of items; and clarity of written instructions across subsections of the developing survey.

Cognitive Interviews

A cognitive interview (CI) is a set of specific inquiries into how participants understand assessment questions. Commonly used in measure development, we specifically designed these CIs to identify (a) if the questions were consistently understood across respondents, (b) whether answer options accurately described respondent experiences (i.e., relevancy), (c) whether respondents understood answer options, and (d) if answers were valid measures of what questions are designed to measure (Fowler, 2002; Willis, 2005).

We divided the resulting pool of items into four CIs that could each be administered in less than 2 hours to minimize participant burden. Each of the 43 participants completed one of the four CI versions. This method is truly a “mixed” method; in each case, interviewers presented individual items using a traditional interviewer-administered face-to-face protocol, and obtained a response from the participant. This was followed by semistructured questions we designed to clarify the items (i.e., their intention, wording, or response formats—for comprehension), evaluate the appropriate item ordering, and elicit the participants’ cognitive processes used in responding to the items (i.e., to assess relevancy). We audiotaped CIs and entered summaries of relevant items into a response matrix. Thus, across all participants who were administered a particular item, we reviewed responses to, first, determine what the participants thought the item meant, and second, revise the item as necessary to be understood as intended. We revised some items to simpler statements as a function of their complexity; that is, avoiding inclusion of more than one concept in a single item. Other items included terms that were poorly understood; when this occurred, we either revised items to include clearer, more direct language, or revised items to include examples. We employed the latter approach with demographic items or vaginal product history items more often than with items found in the Relationship Scale.

In general, our goal was to create items and response options that allowed for the greatest degree of understanding and comfort for the greatest number of potential participants. Therefore, we also conducted literature reviews of other scales, as well as epidemiological data, to generate the most relevant ranges or formats for behavioral frequency items. For instance, how often a participant had sexual intercourse with her partner proved to be the best response format for adult women, for both those who had sex relatively infrequently (less than once per week) and those who had sex relatively frequently (every day or nearly every day).

We also evaluated response formats during the CIs. In the case of the Relationship Context scale, we utilized a Likert scale response format to capture agreement/disagreement. We initially evaluated traditional approaches to Likert agree/disagree scales (e.g., strongly agree = 1, strongly disagree = 5, no other benchmarks; 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree), but as a result of CIs, we diverged from tradition to accommodate how the participants thought through their responses. We evaluated both the number of benchmarks to be used and the labels for the benchmarks, as well as how many of the benchmarks should have labels. As women responded, we asked them to describe what the difference was between, for instance, a “3” and a “4.” It was often difficult for them to describe scenarios that made sense and/or were relatively equidistant between benchmarks. Having participants consider labeling the benchmarks themselves proved useful. Given all that transpired, we finalized a Likert scale that did not have an overwhelming number of benchmarks (5-point scale), with each benchmark labeled. Of note, participants were not comfortable with a “neutral” center to the scale (e.g., “neither agree nor disagree”), believing that it was a “cop-out” and expressing the opinion that we (the investigators) just needed to ask what we needed to ask, and be clear. As one women said, “Ask me what you want to know.”

Ultimately, CIs allowed us to ascertain whether data captured via specific survey items represented constructs and/or concepts as intended. Again, unrelated to the Relationship Context scale, one of the variables we assessed was history of condom use. The original question was, “Have you ever used a male condom?” As participants responded (yes or no), interviewers would verify their response, by either getting a brief history of their condom use, to validate a yes response, or checking to see if they had ever even tried a male condom, to validate a no response. Some women responded “No” to ever having tried a male condom, which was initially taken as a validity check. However, one of the participants remarked that she had not, but that her male partner had. This was a critical point of understanding for us, especially in the context of a microbicide trial: Who has agency? We ultimately determined that we needed to be clearer and, despite the length of the item, chose: “Have you ever had a male partner who used a condom when you had sex with him?” With this question, we believed that we would obtain the response we were actually seeking; subsequent CIs, using the revised item, demonstrated this.

Changes Made to Items and Survey

Analyses of qualitative data, expert review data, and CI data resulted in iterative changes to the items and overall survey. For example, expert reviews and CIs raised questions about the length of the survey and provided feedback on whether item subsets resulted in an appropriate flow of information from one topic to another. As a result, we deleted or reordered items and, for instance, restricted detailed sexual behavior information to one randomly chosen partner, rather than repetitiously iterating the same sets of questions across multiple partners. In doing so, we both shortened the time it took to complete the survey, and helped participants focus clearly on one specific partner and the related risk and relationship concerns of that specific sexual partnership. Women who reported having one sexual partner in the last 12 months were asked to label him as either a “main,” “casual,” or “other” partner. We defined a main partner as “someone you feel an emotional commitment to, like a boyfriend, fiancée, husband, or your man.” We defined a casual partner as “someone you know, such as a friend, but aren’t in any sort of committed relationship with,” and an other partner as “someone you don’t know much about, for example, a client, or maybe someone you have never met before you have sex.” We asked women who reported having two or more sexual partners in the last 12 months to identify their least frequent, most frequent, most important, and most recent sexual partners, and to label them as main, casual, or other partners, as described above. For multipartnered women, we designed the computer program to randomly choose one of her identified partners; for women who reported only one partner, that person was used. Ultimately, the computer program facilitated our ability to stratify the sample by main (56%) and non-main (i.e., casual or other; 44%) partners.

Interviewers noted that it was common for CI participants to want to anecdotally tell the interviewer about their partners, particularly when participants believed that the questions asked did not adequately describe the relationship itself, or when they believed a description would help the researchers better understand the context of certain responses. As a result, we added a series of items to the survey to assess aspects of relationship quality, including its “demographics” (e.g., the length of the relationship, whether any age discrepancy existed between the partners, and so forth), context (e.g., partner-specific norms, partner-specific risk perceptions, and cooccurrent drug and/or alcohol use with sex) and, as can be seen in Figure 2, communication within the relationship, and the role of sex, sexual pleasure, and companionship within the relationship.

Contrary to many psychosocial and behavioral assessment packages that are often comprised of a somewhat disconnected set of items or constructs to which participants or patients are asked to respond, an important decision we made was to order both the items and sections of questions such that movement from question to question and section to section allowed for progression of thought as if in a logical conversational flow. For instance, we ordered items in the sexual risk sections to progress from who the partner was, to the demographics of the relationship, to the participant’s perception of sexual attitudes within the relationship and the quality of the relationship, to what exactly happened the last time they had sex, and finally to the participant’s evaluation of whether, if the situation were the same, she would have used a microbicide that last time and, if so, what she would have needed the microbicide to be like (i.e., product characteristics, use instructions, protection from pregnancy, sexually transmitted infections [STIs], HIV, or all of these functions, and so forth).

In addition to item wording, ordering of items, framing, and instructions, we also changed the administration format for the survey as a function of CIs. Interviewers noted relatively frequent signs of unease and discomfort among CI participants, and identified potential issues in the validity of responses to potentially stigmatizing questions. These questions included the number of male sexual partners a woman had, and whether or not she engaged in certain sexual (e.g., anal sex) or drug-related (e.g., illegal drug use) behaviors. Concerned that the face-to-face administration might have been contributing to this effect, we made the decision to administer the survey via audio computer-assisted self-interview (A-CASI) format.

Survey Evaluation

Once we completed development, we administered a cross-sectional survey to 531 participants. The nonproportional quota sampling procedure we employed resulted in a sample of 166 (31.3%) Latinas, 193 (36.3%) Black women, and 172 (32.4%) White women; the mean age of the sample was 33.8 years (SD = 9.6). We have reported a detailed description of the sample, along with survey administration procedures, elsewhere (Morrow, Fava, Rosen, Christensen et al., 2007). The full survey consisted of an initial eligibility screener and items assessing previous history with vaginal product use, sexual attitudes and beliefs, and a variety of sexual risk variables and contexts. After identification of the participant’s sexual partner(s), the computer-generated randomization process noted earlier randomly selected one partner. Subsequent items focused on this particular relationship context and whether the participant would have used a microbicide with the selected partner during the last episode of sex with him. This sexual partnership was the subject of the Relationship Context Scale, which we have provided as the example in this article. We have detailed additional information about the measure development portion of the study, including eligibility criteria and recruitment procedures, elsewhere (Morrow, Fava, Rosen, Christensen et al., 2007; Morrow, Fava, Rosen, Vargas et al., 2007). Across each stage of the Phoenix Project, we sought and received human subjects protection board approval by the sponsoring institution’s institutional review board.

Quantitative Analyses

In the Phoenix Project, we developed and psychometrically evaluated several measures, including the Relationship Context Scale, and tested each scale’s association with a woman’s willingness to use microbicides. We conducted psychometric analyses of the relationship-oriented items, resulting in three factors representing three subscales, each measuring a particular aspect of relationship quality. Internal consistency, as measured by Cronbach’s Coefficient Alpha (Cronbach, 1951), was very good (DeVellis, 2003) for two longer subscales (Positive Aspects of Relationship/Companionship, 8 items, α = .90; and Sexual Compatibility, 5 items, α =.86), and adequate for a short (3-item) scale (Negative Aspects of Relationship, α = .60). Subsequent concurrent validity analyses revealed multiple person-, relationship-, social-, and context-related variables to be predictive of Relationship Context Scale scores, as hypothesized. Partner type (i.e., main versus casual or other), communication about sexual risk history, the length of the sexual relationship and the frequency of sex within the relationship, and the participant’s perception of her partner’s STI/HIV risk were all significantly associated with the Positive Aspects of Relationship/Companionship subscale and the Sexual Compatibility subscale. The number of sexual partners the participant had in the previous 12 months, her STI history, and the congruence (or lack thereof) between her desire to have children with this partner and her perception of her partner’s desire to have children with her were all significantly associated with the Negative Aspects of Relationship subscale. Furthermore, in structural equation modeling analyses (Morrow et al., 2006), the Sexual Compatibility subscale and the Negative Aspects of Relationship subscale were each inversely associated with the participant’s reported history of protecting herself sexually (i.e., using condoms). Additionally, women’s history of protecting themselves with condoms and their Negative Aspects of Relationship subscale scores were directly associated with their Willingness to Use Microbicides scale score.

Confirmatory Qualitative Analyses

We returned to the original qualitative data to provide explanatory evidence for the resulting factor structure (i.e., subscales), and further integrate our qualitative and quantitative methodologies, a process we refer to as “confirmatory qualitative analyses.” We used items from the Relationship Context Scale in a final qualitative analysis, in which the items themselves (see Figure 2) became thematic codes utilized in a confirmatory review of the correspondence between the validated scale items and the original qualitative data. This resulted in an entirely new analysis of the same transcripts used for the original Phoenix metasynthesis. An independent coder (one who had not participated in the original metasynthesis) completed the relationship scale item coding. Notably, those women whose qualitative data were used to generate the pool of quantitative items (N = 89) were not the women (N = 531) who completed the survey.

The Relationship Context Scale: An Illustration

Scale items were derived from both singular statements, as well as passages that illustrated complexities within relationships. For example, item H1, “Sex brings me and [partner] closer” (see Figure 2), is an item derived from a singular response from a participant when asked to reflect on the intimacy in her relationship:

This interaction [engaging in intercourse while using a candidate microbicide] satisfied me. Yes, brought us closer. Yeah, I think it brought us closer because he could see that, like I said, this interaction was different, adding to the relationship instead.

Item H11, “[Partner] thinks about my happiness,” was derived from a more complex set of thoughts and feelings:

Yeah, I was really excited to take part in the study. … Because I have done a lot of work on … women’s lack of autonomy and decision making, this for me was like a way to put my money where my mouth is, and do something. So I was nervous asking my boyfriend. … I was a little bit scared. Because … I would like to see him more, even though we are equally busy … he doesn’t need to see me as much. So I had a feeling that he’d be like, “No! If I have to have sex with you two times a week that means that I have to come down to [town] and like that’s a pain.” But he was like, “Hey, having sex with you twice a week and get paid for it?! Sure!” So [participating in the trial that required participants to have sex at least twice per week], that was kind of fun, it made me feel good, that he was into it, and he was like, “If it’s important to you, of course I’ll do it.”

Explanatory Evidence of Subscale Validity

Confirmatory qualitative analysis of the Relationship Context Scale’s subscales revealed that specific textual passages elucidating relationship quality and its contexts were commonly multiply coded with items (used as coding categories) from the same subscale. Double coding of a passage as both items H1 and H13 (see Figure 2) suggests an association between the concepts “sex brings me and [partner] closer” (H1) and “I make an effort to sexually satisfy [partner]” (H13), lending credibility to their membership together in the “Sexual Compatibility” sub-scale. Similarly, a passage coded as related to item H3 was also coded with item H14, indicating an association between the concepts “[partner] feels I should get sexual pleasure” (H3) and “I feel happy when I have sex with [partner]” (H14), also validating their membership on the same subscale. For example:

Because we interacted [about using condoms during the microbicides trial], I wasn’t doing for him and he wasn’t just doing for me, we were doing for each other. So that was the feeling. I think that he knows that excited me, which excited him. Anything that’s going to make me happy, and make me enjoy it and make me excited, he’ll go for.

Thematic review of the passages suggested that sexual pleasure for both partners, communication between partners, and a sense of “mutuality”—that is, that the encounter satisfied both partners—were relevant to the Sexual Compatibility subscale.

The Positive Aspects of Relationship/Companionship subscale contained the largest number of relationship items/codes. Three of the codes, items H6, H7, and H11, had no double coding, suggesting that the concepts they captured were unique but related to the other items in the subscale. Three others, items H9, H10, and H16, were often double coded, both with each other and with other items in the relationship scale item codes. Notably H9 (i.e., “[partner] wouldn’t cheat on me”) and H16 (i.e., “[partner] wouldn’t put me at risk”) were frequently double coded with each other, suggesting a clear association between these concepts. Often these were comments made by high-risk participants indicating that their partners cheated on them, and that, as a result, they perceived that they were at risk for HIV infection. Thematic review of these codes indicated that respect and trust, or a lack of either, were interrelated relationship components. The presence of trust was cited as a reason to not use condoms. In particular, the construct of trust seemed to be used to indicate that participants believed their partners were not cheating on them, did not put them at risk by using injection drugs, and were being honest about their HIV status; conversely, a lack of trust equated to being cheated on, having a partner who was using injection drugs, or who was perceived to be secretly HIV positive:

Participant (P): My boyfriend, he’s a [drug] user. … I know that he’s not messing around with me. And I know from different people that know that he’s a user. They always said that he had been a clean user. … So I don’t have no cause for [any] condoms ….

Facilitator (F): So you’re saying that you don’t use condoms with your boyfriend?

P: We never.

F: Even though he’s a drug user, but because you trust him?

P: Yes. Because as a matter of fact, from—you know, mostly what everybody tells me, they tell me he do not see no needles. So I know he don’t mess around on me, or anything like that.

Additionally, our analyses suggested that companionate relationships, identified by items from the Positive Aspects of Relationship/Companionship subscale, can be characterized by a sense that a partner would protect a participant, for instance, by not cheating, not using drugs (or being a “clean user” if he does), that he would provide for her, and/or that there is a sense of mutual caring between the partners. For example: “I’m with my fiancée for … almost two years … we are trustful, so we don’t use condoms.”

There was no double coding within the passages most frequently coded by the three items in the Negative Aspects of Relationship subscale, suggesting these items were capturing three distinct latent constructs that formed negative aspects of a relationship; however, there was double coding of these items with items from the Positive Aspects of Relationship/Companionship subscale. Of particular relevance, the passages so double coded were instances in which partners did something to put the participant at risk for HIV and/or cheated on her, suggesting an inverse association between these two subscales. The Negative Aspects of Relationship subscale item codes contain passages about partners who cannot be trusted to tell the truth, and who lie, either about having STIs, having other sexual partners, or about their drug use. Participants spoke about trusting partners whose behavior ultimately revealed that they were not to be trusted:

If the person’s looking at you straight in the face and telling you, “I don’t have it and I don’t use, I don’t share needles, or whatever,” okay, and then you and that person have sex and then you come to find out that that person shares needles, he’s HIV positive, and then what do you do? Then it’s like you want to kick yourself in the ass.

Discussion of Scale Illustration

Other researchers have clearly shown that partner communication and the negotiation of product use will be essential to microbicide use for some women (Orner et al., 2006; Short, Perfect, Auslander, DeVellis, & Rosenthal, 2007). In various studies, both female participants and their male partners have also suggested that partner involvement is important and necessary for microbicide use (Auslander, Perfect, Breitkopf, Succop, & Rosenthal, 2007; Carballo-Dieguez et al., 2007; Mantell et al., 2005; Montgomery et al., 2008; Woodsong & Alleman, 2008). Additionally, women make choices about how and when to protect themselves against pregnancy and STIs based, in part, on their sexual partners and their perceptions of the quality of the relationship(s) they are in. Consequently, there will likely be tremendous variation in microbicide use among women and between partners, based in part on the context and type of relationship and the communication (and negotiation) possible within the relationship. Clearly, relationship factors, including partner choice and whether or not women have the confidence and power to negotiate microbicide use in the relationship, are critical to microbicide acceptability.

The goal of the Phoenix Project was to develop valid quantitative scales to assess the person-in-context elements of microbicide acceptability. Historically, these elements have been evaluated primarily via costly and time-consuming, qualitative in-depth interviews or focus groups, as well as study-specific surveys that used hypothetical or surrogate vaginal formulations, but have not been psychometrically validated. The integrated mixed methodology we used in the Phoenix Project represents an essential process in effectively capturing the contextual elements related to microbicide acceptability, allowing the development of innovative measures, like the Relationship Context Scale, which allow researchers to quantitatively capture the dyadic contexts in which microbicides will be used. That said, researchers will likely require a combination of measures, including the Relationship Context Scale, to gather information regarding women’s decisions to use a microbicide. Indeed, it will be relationship characteristics, in a larger informational, behavioral, economic, and social context, that will facilitate our understanding of microbicide acceptability. How a relationship “scores” on these measures, for instance, will contribute to a user’s decision-making process, but her knowledge of the product itself, for instance, and whether or not the product has a tendency to lubricate her vagina during sexual intercourse, will likely also be determining factors. Thus, relationship context is only a piece of the overall process of making a decision to use or not use a microbicide. An integrated mixed methods design (a) ensured that survey items were grounded in the experiences of low- and high-risk women, representing a range of potential microbicide users; (b) allowed analysis of potential users’ experiences via a particular theoretical framework; (c) clarified participants’ understanding of potential survey items, including their response format, ordering, and framing; (d) resulted in the inclusion of relationship-focused items that contextualized microbicide use; and (e) provided credibility of, and explanation for, the resulting factor structures in the scales developed (illustrated here in the three Relationship Context subscales).

A key goal of qualitative work, both generally and with regard to mixed methods specifically, is to establish the credibility of data (Ulin et al., 2005). An integrated mixed methods design serves this purpose. Grounding the quantitative measures in the experiences (and qualitative data) of microbicide research participants allowed us to develop a survey for measuring factors related to microbicide acceptability that was closely tied to users’ actual experiences. Additionally, because the metasynthesis data were derived from participants different from those who ultimately completed the Phoenix Project’s quantitative survey, any concordance we observed between scale questions and the original data lends credence to the results. The Phoenix Project methodology demonstrates both the credibility and validity of integrating qualitative and quantitative methods to effectively develop psychometrically sound behavioral measures.

Conclusions

There are limitations to the Relationship Context Scale itself. As the microbicide field moves toward an approved and marketed microbicide, and as clinical trials of next-generation candidates begin, the Relationship Context Scale will need to be studied within those contexts. Generalization needs to be established, and the scale’s test-retest reliability and validity need to be evaluated in other contexts. It is also important to note that this scale does not capture the entire range of relationship characteristics and variables that are likely important to microbicide acceptability and use, nor did we intend it to do so. The scale should be used within a package of appropriate measures, where such issues as relationship violence; unwanted, forced or pressured sex; and paid sex can be established and understood in context.

The use of integrated mixed methods offers several advantages over more traditional “top-down” approaches to measure development and evaluation. Most important, it allows the design of a quantitative assessment tool grounded in the experiences of the target population. By using qualitative data gathered across diverse contexts, and by being intent on capturing the range of variables associated with a particular phenomenon, researchers employ a more comprehensive approach to assessing a particular construct or constructs of interest. Researchers’ utilization of the input of the target population at multiple time points in the development process allows an iterative progression and a sense of confidence as the measure development progresses. Employing this methodology also allows researchers to discover new information, which might garner attention in ways that might have been lost—or never considered—in earlier stages (e.g., the need to administer the survey via A-CASI, in the case of the Phoenix Project). Finally, using the methods described here affords researchers progressive conceptualization across stages, as more is learned about the phenomenon of interest.

That said, challenges to engaging this methodology also exist. The specific challenges of each method integrated here remain: a heavy reliance on skilled staff and researchers, both qualitative and quantitative; a need to devote substantial time to each element in the method as well as the integration of elements; and, at least in the case of HIV biomedical prevention approaches, the critical role of a multidisciplinary team. Overall, the Phoenix Project took several years to complete. The results, however, have been important, and make substantive contributions to the microbicide field’s measurement strategies. We do not intend to replace integrated qualitative methods in research with brief quantitative instruments (indeed, we strongly believe in the continued need for both integrated and parallel qualitative studies to continue as microbicide development progresses). That said, we believe these scales can fulfill a specific role within heavily burdened clinical trials where qualitative studies might not be supported, as well as in future behavioral intervention studies designed to increase microbicide uptake. Developing even more efficient progressions in mixed methodologies should remain a priority for behavioral and social scientists engaged in health promotion and disease-prevention research.

Biographies

Kathleen M. Morrow, PhD, is an associate professor of psychiatry and human behavior at the Centers for Behavioral and Preventive Medicine at The Miriam Hospital and The Warren Alpert Medical School of Brown University in Providence, Rhode Island, USA.

Rochelle K. Rosen, PhD, is an assistant professor at the Centers for Behavioral and Preventive Medicine at The Miriam Hospital and The Warren Alpert Medical School of Brown University in Providence, Rhode Island, USA.

Liz Salomon, EdM, is assistant director of the education unit of the Massachusetts General Hospital Clinical Research Program in Boston, Massachusetts, USA.

Cynthia Woodsong, PhD, is director for behavioral and social science at the International Partnership for Microbicides in Paarl, South Africa.

Lawrence Severy, PhD, is a professor emeritus at the University of Florida in Gainesville, Florida, USA.

Joseph L. Fava, PhD, is research associate at the Centers for Behavioral and Preventive Medicine at The Miriam Hospital in Providence, Rhode Island, USA.

Sara Vargas, BA, is a graduate student in clinical health psychology at the University of Miami in Miami, Florida, USA.

Candelaria Barroso is a senior research specialist at The Miriam Hospital in Providence, Rhode Island, USA.