Reconciling incongruous qualitative and quantitative findings in mixed methods research: exemplars from research with drug using populations

Abstract

Mixed methods research is increasingly being promoted in the health sciences as a way to gain more comprehensive understandings of how social processes and individual behaviours shape human health. Mixed methods research most commonly combines qualitative and quantitative data collection and analysis strategies. Often, integrating findings from multiple methods is assumed to confirm or validate the findings from one method with the findings from another, seeking convergence or agreement between methods. Cases in which findings from different methods are congruous are generally thought of as ideal, while conflicting findings may, at first glance, appear problematic. However, the latter situation provides the opportunity for a process through which apparently discordant results are reconciled, potentially leading to new emergent understandings of complex social phenomena. This paper presents three case studies drawn from the authors’ research on HIV risk among injection drug users in which mixed methods studies yielded apparently discrepant results. We use these case studies (involving injection drug users [IDUs] using a needle/syringe exchange program in Los Angeles, California, USA; IDUs seeking to purchase needle/syringes at pharmacies in Tijuana, Mexico; and young street-based IDUs in San Francisco, CA, USA) to identify challenges associated with integrating findings from mixed methods projects, summarize lessons learned, and make recommendations for how to more successfully anticipate and manage the integration of findings. Despite the challenges inherent in reconciling apparently conflicting findings from qualitative and quantitative approaches, in keeping with others who have argued in favour of integrating mixed methods findings, we contend that such an undertaking has the potential to yield benefits that emerge only through the struggle to reconcile discrepant results and may provide a sum that is greater than the individual qualitative and quantitative parts.

Introduction

Mixed methods research is a re-emerging area of interest in the field of drug use research, and has received attention across multiple disciplines, including anthropology, epidemiology and sociology (Bourgois, 2002; Moss, 2003; Rhodes, 2009). While there are increasing calls for and advocates of mixed methods research designs, combining qualitative and quantitative strategies can present a number of distinct philosophical and logistical challenges, including whether and how to integrate findings that may initially appear to conflict. Methodological scholars have engaged with the process of how and when to combine findings from qualitative and quantitative research (for reviews see Denzin, 2010; Kelle, 2001). In its most recent iteration, termed the “third methodological moment” (Denzin, 2010, p. 422), researchers conducting mixed methods research have revisited the discussion, and some note a resurgence of interest in integration in the design, analysis, and interpretation of mixed methods studies (e.g. Bazeley, 2009). Some have noted that the process of integration, particularly integration of apparently discrepant data, can itself be a productive enterprise (Fielding, 2009); though the success of such integration efforts may depend on the perspective with which researchers approach the discrepancies.

At first glance, integration is often thought of in terms of seeking consistency between qualitative and qualitative findings. In one commonly used form, integration through the “triangulation” of findings from multiple methods implies a search for increased validity; the use of multiple methods producing similar results suggests that the phenomenon of interest has been more accurately measured than it could have been using a single method (Moran-Ellis et al., 2006). In the social sciences, triangulation has its roots in the classic multitrait-multimethod matrices proposed by Campbell and Fiske (1959) as a means of establishing convergent validity of psychological scales. Though ultimately favouring the classic conceptualization of triangulation in the trigonometrical sense, Kelle (2001) also offers a third interpretation of triangulation: triangulation as a means of integrating multiple perspectives (complementarity). From a more postmodern perspective, the use of multiple methods can been seen to extend the “scope and depth of understanding” (Fielding, 2009, p. 429), and can provide a richer understanding of complex social issues.

In the fields of drug use and HIV prevention research, scholars have increasingly employed mixed methods designs to investigate the complex environmental, social, and individual factors that are associated with negative health outcomes (Bourgois et al., 2006; Clatts, Welle, Goldsamt, & Lankenau, 2002; Pollini et al., 2010; Torrone et al., 2010), though the complementary use of epidemiology and ethnography in the study of drug use is not new (Agar, 1997; Clatts, Southeran, Heimer, & Goldsamt, 1999). In the process, some have noted difficulties in integrating findings from mixed methods research. Critical to this discourse is attention to the assumptions that underlie the methods that researchers are attempting to combine. While differences between qualitative and quantitative approaches to complex social situations can often be reduced to differences in data collection processes and analytic methodologies, these superficial differences can mask more fundamental epistemological differences – quantitative methods emphasize deductive approaches that test a priori hypotheses; qualitative methodologies emphasize inductive approaches that “stress how social experience is created and given meaning” (Denzin & Lincoln, 2000, p. 8, emphasis in the original). For example, in complementary articles Bourgois (2002) and Moss (2003) articulate some of the challenges experienced in the context of a large mixed methods study of disease transmission among young injection drug users, which employed participant observation, ethnographic interviews, and epidemiological surveys. These included reconciling differences between clinical and statistical significance, discipline-specific constructions of scientific validity and reliability, and different levels of disciplinary focus (i.e., ethnographic focus on macro-level social structures vs. epidemiological focus on individual-level behaviour).

In this paper, we extend this discussion and argue that the process of engaging with divergent results can yield a more comprehensive and nuanced understanding of the situation under study, offering multiple viewpoints and encouraging a “dialogue between differently committed minds” (Fielding, 2009, p. 436). In an effort to engage critically and reflexively with the process of doing mixed methods research, we present three case studies from our own research focused on the association between injection drug use and HIV infection as exemplars (for more details we refer the reader to the original publications: Davidson, 2009; Pollini et al., 2010; Wagner et al., 2010; Wagner et al., 2011). The projects differ in their specific research questions, research designs and in their methods of attempting to reconcile conflicting findings. The common theme, however, is that the work of seeking to integrate initially conflicting findings yielded rich insights into both the nature of the phenomena and of the methods used to investigate them. We provide a descriptive account of each case study, briefly summarizing the study design and implementation, analysis and results; and discuss of the efforts to reconcile apparently incongruous results. We conclude with a discussion of the theoretical and methodological lessons learned from implementing these mixed methods projects, and with recommendations for how investigators engaged in research with drug using populations may more successfully anticipate and manage the integration of findings from multiple methods.

Case Study 1: The role of “dopesickness” in influencing risky injection behaviour among IDUs

Background

Current public health recommendations hold that using a new needle/syringe for every injection event is necessary to avoid the spread of HIV and other blood-borne viruses among IDUs (Gayle, O’Neill, Gust, & Mata, 1997). Considerable research has examined the role of the perceived risks associated with sharing injection equipment in influencing needle/syringe sharing behaviour. However, less research has been dedicated to the converse – understanding the influence of the perceived risk of refusing to share injection equipment. That is, the consequences that IDUs associate with engaging in “safer” behaviour than can reduce the risk for HIV infection. Epidemiological and ethnographic studies have identified some factors that might be identified as potential consequences of refusing to share needle/syringes, such as the threat of becoming “dopesick” (i.e., experiencing withdrawal symptoms associated with the delayed use of opioids), risking citation or arrest for possession of drug injection paraphernalia or violating social norms associated with sharing. This study was designed to qualitatively identify the consequences that IDUs associate with refusing to share injection equipment, and to examine the quantitative associations amongst those perceived consequences, other psychosocial correlates of injection risk behaviour and needle/syringe and paraphernalia sharing.

Method

This study was designed as a two-phase, sequential mixed methods study, in which the exploratory qualitative phase provided data that were used to inform the development of items for a quantitative questionnaire. In Phase One, semi-structured interviews were conducted with 26 IDUs recruited from a single Needle/Syringe Exchange Program (SEP) in Los Angeles, California, USA. The interviews focused on the most recent injection event in which the participant shared injection equipment, and participants were asked to describe the potential consequences to them if they had refused to share equipment at that time. Interviews were digitally recorded and transcribed in their entirety. A theoretical framework that provided some a priori thematic codes derived from the ethnographic and epidemiological literature guided the qualitative analysis, and emergent codes were added to the codebook throughout the analysis.

For the second phase of the study, 187 IDUs were recruited from the same SEP to participate in a structured interview using Audio Computer Assisted Self Interview (ACASI). Qualitative data were distilled into 11 possible consequences of refusing to share, and in the quantitative survey participants were asked how frequently each of the 11 consequences “influenced their decision whether or not to share injection equipment in the past 30 days”. The consequences were weighted by their importance (e.g. “how much of a problem would [that consequence] be for you?”), since more severe consequences may be more salient in their influence (Fishbein, 1967). The perceived consequence and importance questions were included in the quantitative survey, along with measures of other psychosocial constructs commonly used to explain injection risk behaviour (e.g., perceived risk of HIV or HCV, self-efficacy for safer injection, response efficacy, perceived severity of HIV or HCV, knowledge and peer norms for safer injection).

The aim of the quantitative analysis was to determine whether the perceived consequences of refusing to share injection equipment were associated with injection risk behaviour when controlling for other theoretically-relevant psychosocial correlates. To investigate this question, we first created two sub-scales created from the perceived consequences items using exploratory factor analysis, which largely conformed to the theory that emerged from the qualitative interviews: social/internal consequences and structural/external consequences. We then employed logistic regression analysis in which two dependent variables (receptive needle/syringe sharing and receptive paraphernalia sharing) were regressed on the psychosocial correlates and the perceived consequences sub-scales.

Results

In the qualitative narratives, almost all participants described risky injection events in which they were already experiencing, or were trying to avoid, the intense drug withdrawal symptoms (i.e. “dopesickness”) that results from delayed use of heroin among heroin-dependent injectors. They described events in which, if they did not use the available injection equipment, they would most certainly become dopesick, and therefore said they injected with shared equipment in an attempt to avoid it. However, this key finding from the qualitative analysis –that risky injection events were overwhelmingly contextualized by an omnipresent risk of experiencing dopesickness – was less clearly borne out by the quantitative data. In the quantitative analyses, the perceived consequence of becoming “dopesick” yielded slightly inconsistent results in the exploratory factor analysis, showing moderate double loadings on both sub-scales. Due to the salience of this particular consequence in the qualitative narratives, this finding inspired additional quantitative analyses in which we removed the item from the perceived consequences scales and evaluated its significance as an independent covariate. Results from these supplemental analyses showed that, on its own, “dopesickness” was not associated with either receptive or distributive sharing, and removing it from the sub-scales did not substantially change the findings regarding the main effects of the perceived consequences on injection risk behaviour.

Integration Challenges and Lessons Learned

Rather than use this discrepancy between the quantitative and qualitative findings as a rationale for discrediting the results of either analysis, we attempted to understand the circumstances under which such a discrepancy could arise, and its implications for future research of this type, in which qualitative data are collected in order to inform the design of a quantitative survey. We suggest two potential frameworks for understanding this discrepancy: the first rooted in an understanding of the politics of representation and social roles and biases inherent in the conduct of public health research among drug users, and the second grounded in a study design perspective that emphasizes a reciprocal, rather than linear, relationship between quantitative and qualitative data.

This particular case study highlights issues related to the politics of representation that result from the interaction between public health researchers and IDUs. Though the interviewer/author had an ongoing collaborative relationship with the SEP where the data collection took place and had interacted with many of the interview participants for over two years at the time that the interviews took place, she was not a member of the target population. Also, since the interviews occurred at the SEP, the researcher was often identified by participants as part of the SEP staff (even though she emphasized that she was not employed by the SEP). The findings from the qualitative portion of the project, then, must be understood within this context. It is possible that the emphasis on the role of dopesickness was offered as a way to avert anticipated blame from the researcher for engaging in behaviour that both parties understood to confer a disease transmission risk. Attributing their risky behaviour to the threat of drug withdrawal, then, could represent an attempt by interview participants to avoid violating the shared expectations of the participant and the SEP regarding acceptable behaviour. In the quantitative interviews, on the other hand, participants had the freedom to enter their responses to questions about the importance of dopesickness in influencing their behaviour directly into the computer without having to explain or justify their actions the research team. Thus, one explanation for the observed discrepancy between the relative importance of dopesickness in the two phases of the study may have to do with the social context of describing risky injection events to a researcher. In short, this finding could have been influenced by social desirability bias (a risk faced in all behavioural research relying on respondent self-report). While the absence of statistical correlation in the quantitative phase neither confirms nor refutes the presence of social desirability bias, it does highlight the need for sensitivity to and discussion of its potential effects. One contribution of the mixed methods approach, in this case, is its ability to highlight the need for a reflexive examination of identity politics and the influence of the social-spatial context in which research occurs (Bourgois, 2002).

The discrepancies highlighted in this case study also point towards another contribution of the mixed methods approach – the ability of a mixed methods design to create a reciprocal dialogue between quantitative and qualitative data, such that findings from one method inform the development and analysis of the next, which can then be used to refine subsequent phases of the investigation. This study was designed in two phases, in which the qualitative results were used to inform the development of a quantitative scale. Unexpectedly, the initial lack of agreement between the qualitative and quantitative results on the importance of the influence of “dopesickness” inspired the investigators to pursue additional quantitative analyses that would have remained unexplored in the absence of the compelling qualitative results. Though not feasible under the funding mechanism and timeline for the current study, a more comprehensive use of a mixed methods design could capitalise on the apparently discrepant results to refine the data collection instruments based on lessons learned through the attempt to reconcile the results. For example, it is possible that the quantitative questions did not accurately measure the same construct that was discussed in the qualitative interviews, or that the difference in referent timeframes in the two phases influenced the results (i.e. the most recent risky injection event described in the qualitative interviews versus quantitative questions that asked participants to think more generally about behaviour over the past 30 days). Simply piloting the quantitative interviews (which was also done in this study) would not have identified this issue. Additional data collection in the form of cognitive interviews to understand how participants interpret the quantitative questions or “member-checking” interviews or focus groups to discuss with participants whether the findings from the research resonate with their experience, could shed some light onto the validity of the findings. In the absence of the mixed methods design that explicitly attempted to integrate the findings from the two phases, however, the discrepancies may not have been discovered and the lessons for future study designs, data collection instruments and analysis strategies would have been lost.

Case Study 2: Barriers to Needle/syringe Purchase at Retail Pharmacies in Tijuana, Mexico

Background

In Mexico, it is legal to purchase and possess sterile needle/syringes without a prescription. However, findings from qualitative research with IDUs in the north-western border city of Tijuana, Baja California, Mexico, suggested that some pharmacies refuse to sell or charge higher prices to persons suspected of being IDUs (Strathdee et al., 2005). In prior qualitative interviews, IDUs directly linked these pharmacy-based barriers to needle/syringe access to the practice of needle/syringe sharing, thus increasing their risk for HIV infection. To understand more completely the context of pharmacy sales in Tijuana, we undertook a mixed methods study designed to 1) quantify the frequency of refusals and overcharges reported by IDUs attempting to purchase needle/syringes from pharmacies in Tijuana, 2) identify correlates of experiencing these barriers to needle/syringe access and 3) provide a comprehensive understanding of these correlates within the context of IDUs’ experiences.

Method

A detailed account of the study methods and results has been published elsewhere (Pollini et al., 2010). Briefly, the parent study (Proyecto El Cuete Phase III) recruited 1,056 IDUs via respondent driven sampling (RDS; Heckathorn, 1997) between 2006 and 2007 into a longitudinal study of behavioural and contextual risk factors for HIV, syphilis and TB infection (Strathdee et al., 2008). Participants completed an interviewer-administered survey at baseline and every six months thereafter that included questions on demographics, drug use behaviours, needle/syringe acquisition and use and experiences with police. At the second study visit, participants were asked whether they had attempted to purchase needle/syringes at a pharmacy in the past six months and, if so, to provide information on the outcome of the most recent purchase attempt. IDUs who reported being overcharged and/or refused were defined as having experienced barriers to pharmacy-based purchase and were compared to those who did not experience barriers using univariable and multiple logistic regression methods with RDS adjustment.

Between October 2008 and March 2009 we recruited a subset of 47 IDUs from the parent study to participate in focus groups designed to provide detailed accounts of experiences related to pharmacy-based needle/syringe purchase. We conducted seven focus groups, of which five consisted entirely of males and two entirely of females. The focus groups were audio recorded, transcribed verbatim, translated into English by a certified translator and then double-coded using a descriptive and thematic approach rooted in Grounded Theory (Glaser & Strauss, 1967; Willms et al., 1990). An explanatory design was then used to explain or build upon the quantitative results (Creswell & Plano Clark, 2006); that is, we used themes from the focus groups to elucidate why certain characteristics were associated with experiencing barriers to needle/syringe purchase.

Results

Overall, 649 IDUs completed the Visit 2 survey and 627 (96.6%) responded to the question on pharmacy needle/syringe purchase. Most (80.7%) reported purchasing a needle/syringe in a pharmacy in the past 6 months and 71.6% of all participants cited pharmacies as their primary source of needle/syringes. Only 16.0% experienced barriers to pharmacy-based needle/syringe purchase, including being refused purchase, being overcharged or both. Factors independently associated with experiencing barriers to needle/syringe purchase were being homeless, engaging in receptive needle/syringe sharing, reporting an average of more than five uses per needle/syringe and having a greater number of lifetime abscesses.

These quantitative findings directly contradicted earlier qualitative interviews by our research team in which pharmacy-based barriers to needle/syringe access emerged as a key theme (Strathdee et al., 2005). We investigated this discrepancy in the seven focus groups by inviting participating IDUs to help us explain these discordant findings. Participants described a scenario in which pharmacies are reluctant to sell needle/syringes to persons suspected of illicit drug use, reportedly because of fear of police (despite the fact that needle/syringes can be legally sold and possessed without a prescription), fear of IDU theft and violence and overall concerns about the impact IDU customers would have on the “image” of their business. They explained that IDUs adapt to this restrictive purchasing environment by working to develop a personal relationship with staff at a single pharmacy; by building this relationship the IDU becomes “known” to the staff as someone who will not cause trouble for them, thus increasing their willingness to sell needle/syringes. This pharmacy becomes the primary source of needle/syringes for that IDU, with purchases attempted elsewhere only when that pharmacy is not available (e.g., after hours, immediately after release from jail when withdrawal requires immediate injection). It is in these latter cases that IDUs are most likely to encounter refusals or overcharging.

Integration Challenges and Lessons Learned

Lessons learned from this case study again point to in part to a methodological issue that would not have been identified in a single-method study - our discrepant findings were likely due in part to how we worded the questions in Phases I and III of the parent study. In the Phase I qualitative study we asked “Will most pharmacies sell syringes to drug users in this city?” followed by probes designed to ascertain the nature of any barriers cited (e.g. requiring a prescription). In doing so, we asked IDUs to describe needle/syringe access at the community level. In contrast, the Phase III quantitative study asked: “In the last 6 months, did you purchase a syringe from a pharmacy in Tijuana?” thus focusing on access at the individual level. While the wording of these questions may not have been optimal, it is often the case with multi-year studies that question wording and emphasis changes. In this case, using a mixed methods approach that included a third type of data collection – focus groups in which participants offered their opinions of and explanations for the data from the contradictory qualitative and quantitative data – allowed us to provide more another perspective that enhanced our understanding of the IDU risk environment. Specifically, the “member checking” focus groups revealed the adaptive process of becoming “known” to pharmacies, which would have remained unobserved in the absence of the discussions about the apparently contradictory results. Relying on the qualitative or quantitative interviews alone would have resulted in a highly distorted view of the needle/syringe access situation in Tijuana.

This case study also points to the value of conducting mixed methods studies that employ a sequential design that explicitly allowed for an ongoing dialogue between the quantitative and qualitative phases. The subsequent focus groups afforded us an opportunity to examine the reasons for our discrepant findings from El Cuete using input from the participants themselves. We specifically asked them to identify reasons for the discrepant findings, and what emerged was a very richly detailed account of a dynamic adaptive process designed to negotiate needle/syringe access in a highly restrictive and complex environment. This level of detail would not have been obtained had the previously identified discrepancy not been used as the starting point for our focus group discussions.

Case Study 3: Young injectors, policing and SEP access in San Francisco, California, USA

Background

Currently San Francisco, California, USA has fifteen legal SEP sites, eleven run by a city agency and the remainder by non-profit organizations receiving most of their funding from the city (NASEN, 2010; SAFAF, 2010). In 2005, the city took advantage of a change in California State law to formally legalise the possession of needle/syringes without a prescription. However, a considerable body of literature describes social and logistical barriers to accessing sterile injecting equipment, even in jurisdictions such as San Francisco that are well-served by SEPs and where the legal consequences of needle/syringe possession have been minimised. These barriers include fear of police, fear of being identified as a drug user and limited opening hours and/or service location (Bluthenthal, Kral, Lorvick, & Watters, 1997; Case, Meehan, & Jones, 1998; Gostin, 1998; Heimer, Bluthenthal, Singer, & Khoshnood, 1996; Rich, Strong, Towe, & McKenzie, 1999). While “fear of police” is frequently cited in studies of barriers to utilisation of SEPs, little work has been done to explore this finding in more detail in order to examine which aspects of interaction with the judicial system are particularly likely to influence access to SEPs and injecting equipment. One factor that has not been sufficiently examined is how mobility patterns among young, homeless IDUs may mediate the association between contact with law enforcement and access to services. We hypothesized that young IDUs’ experiences with law enforcement would influence their comfort travellingtraveling to various parts of the city, particularly those neighbourhoods where HIV prevention services are located (e.g. SEPs), and that would in turn influence their access to injecting equipment via SEPs.

Method

Simultaneous cross-sectional quantitative (N=455) and qualitative interviews (N=17 interviews with 16 individuals) were conducted as a sub-study within a larger epidemiologic study of HCV transmission among young (<30 year old) IDUs in San Francisco. Quantitative interviews assessed two sets of measures: current judicial status and injection risk behaviour. Variables assessing current judicial status included: whether someone had warrants out for their arrest, was on probation or parole, had been stopped by police for any purpose in the past three months or had been incarcerated overnight or longer within the past three months. Lifetime history of incarceration (ever incarcerated, and if so for how long in total; ever been on probation or parole) was also recorded. To assess injection risk behaviour, we created a Safer Injection Indicator (SII), which measured how closely each individual approximated the U.S. Public Health Service standard of using one new needle/syringe for every injection (Gayle et al., 1997) in the last 30 days. The SII was constructed by taking the number of needle/syringes acquired in the last 30 days from any source and subtracting the number of self-reported injection events in the last 30 days.

Qualitative interviews were semi-structured and carried out at the field sites of the parent study with a sub-sample of individuals who were or had been in the parent study and who were present at the parent study field site. Qualitative interviews addressed the possible relationships between engagement with the judicial system and access to SEPs, by more broadly exploring how and where respondents moved through the city; what made them go to or avoid certain areas; what role their interactions with the judicial system and police had in influencing these movements and through exploration of how these patterns of movement through the city and their relationships with the judicial system affected their ability to visit SEP sites.

Qualitative transcripts were coded and thematically analysed based on Grounded Theory/Situational Analysis approaches (Clarke, 2003; Glaser & Strauss, 1967; Strauss & Corbin, 1998). For quantitative data, the Mann-Whitney U test was used to compare differences between the median SII values grouped by judicial status measures. For non-parametric continuous variables, variables were categorised and compared using a chi-square test.

Results

In the quantitative analysis, those who had been stopped by police within the past three months were significantly less likely than those who had not been stopped to report obtaining enough needle/syringes to meet their injecting needs. No other measure of judicial status had a statistically significant association with SII in either direction. These data suggest a direct negative association between frequent contact with police and access to sufficient needle/syringes to meet one’s needs.

However, in the qualitative interviews, it appeared that there was very little association between being stopped by police and IDUs’ ability to access needle/syringes; participants did not identify contact with police as a barrier to needle/syringe access. Instead, participants in the qualitative interviews described a situation in which contact with police influenced where and how they moved around the city (i.e. an association between police contact and mobility), an association that itself was largely influenced by participants’ strategies for income generation. Those who used techniques such as panhandling (i.e. begging), for example, were used to endemic but “trivial” contact with police, and expressed relatively little concern about police contact. In contrast, those who shoplifted or sold drugs usually had less contact with police but the contact that did occur was of more substantial legal significance and hence of higher concern. The combination of economic necessity and relative attitudes towards contact with police shaped where people spent time and how comfortable they were in those areas – a panhandler would, out of necessity, spend a lot of time in parts of the city with high levels of foot traffic; would be very “visible” in those places; would be relatively unconcerned that she was “known” to police in the area and would subsequently have relatively few barriers to accessing services such as needle/syringe exchange in those parts of the city. By contrast, a shoplifter would spend relatively brief periods of time in parts of the city with specific types of retail stores; would attempt to be “invisible” in those places; would be highly concerned about becoming “known” to police anywhere in the city and would have relatively strong incentives to avoid going to commercial districts other than when shoplifting due to fear of recognition by merchants or police.

Most SEPs in San Francisco are deliberately located in parts of the city with a visible population of IDUs – i.e. in commercial areas with high levels of foot traffic where panhandling is common. As such, our qualitative data described a situation where frequency of contact with police did not in itself appear to affect people’s willingness and ability to visit SEPs. Rather, willingness and ability to visit parts of the city where SEPs were located was shaped by how someone earned money, and the consequences of that method of income generation on the nature of their relations with police. Through integrating the findings from the two methods – one that found an association between police contact and needle/syringe access and the other that found an association between police contact and mobility/income generation – we were able to arrive at a more complete understand of the factors and underlying processes associated with needle/syringe access.

Integration Challenges and Lessons Learned

The apparent discrepancies between quantitative and qualitative results (that frequent contact with police was statistically associated with less access to needle/syringes, but qualitatively the association was less apparent and was mediated by income generation strategies and consequent physical movement around the city) are a reminder of the basic truism that correlation is not causation. Unmeasured in the quantitative data, identification of this third variable through the qualitative interviews helped add complexity and nuance to an apparently simple direct association. In doing so, the qualitative approach suggested both future directions for research (such as systematically attempting to measure associations between income generation strategies, criminal justice consequences and access to needle/syringes), and different directions for possible interventions designed to improve needle/syringe access (such as delivering needle/syringes in more dispersed locations, such as community health clinics, or conducting designing satellite exchange programs specifically targeting groups who under-utilize SEPs). This case study also highlights differences in the level of focus that can be obtained from quantitative and qualitative data collection. Quantitative assessments are adept at measuring discrete events (i.e. being stopped by police) and analyses focuses on determining whether statistically significant associations exist between those events and pre-defined outcomes. In contrast, qualitative interviews can reveal larger cultural and social processes and dynamics (e.g. relations between IDUs and police) that are more difficult to distil into discrete items to be measured by a quantitative survey (Bourgois, Prince, & Moss, 2004).

Discussion

We have described three case studies from our own research with IDUs in which mixed methods designs were employed to investigate barriers to safer injection. Our selection of these three case studies is not meant to be globally representative of all mixed methods research with drug users. Instead, we present them as examples of different designs and approaches to mixed methods research that share a common theme of working towards a reconciliation of apparently discordant findings from their mixed methods designs. By working to integrate initially conflicting qualitative and quantitative findings, we achieved a more nuanced understanding of the issues than may have been achieved through the use of a single method. It should be noted, however, that our interpretations of the mechanisms behind the observed differences are both speculative and partial. Similar to any other analysis, it is possible that other researchers could identify different explanations. Nonetheless, all three projects served to set a new expectation regarding agreement between quantitative and qualitative data and highlighted the unique challenges and contributions of the mixed methods approach when faced with apparently discrepant data. In our case studies, these came in two primary forms: In Case Studies 1 and 2, our conflicting findings highlighted a need to be attuned to the methodological differences that may emerge between quantitative and qualitative data collection phases, and the implications of those differences for the integration and interpretation of findings. In Case Studies 2 and 3, the use of multiple methods allowed for different levels of focus, contrasting the measurement of associations between discrete factors with the assessment of underlying social, behavioural or cultural processes that are less easily measured and were not initially identified as a primary variable or issue under study.

These contributions underscore some of the primary epistemological and methodological differences between quantitative and qualitative methods. The power to define the issue under study and the acceptable response options a priori is situated with the investigator in quantitative research, while in qualitative interviews the respondent has more freedom to introduce new topics or define unexpected response options. Qualitative methods are particularly useful in highlighting the participant’s “voice” and allowing him/her to explain responses or provide conditions or caveats that could be lost in a strictly quantitative assessment. They also allow the investigator to develop insights that might be difficult to infer quantitatively, working from the “ground up” to identify variability in behaviour and environmental influences (Clatts et al., 2002). For example, in Case Study 2, focus group interviews allowed participants to explain the unexpected quantitative findings and contextualize them in their descriptions of an adaptive process that was unanticipated and unmeasured in the quantitative survey. However, some qualitative methods may be more vulnerable to issues of personal representation or social biases that influence how information is exchanged between respondent and interviewer, encouraging a more reflexive examination of the process that can be incorporated into the analysis. Qualitative methods such as ethnography are able to uncover cultural and social meanings and processes that can help explain and contextualize quantitative associations, while quantitative methods such as epidemiological surveys may reveal associations between discrete, measured variables that participants are unaware of or unable to explain (Agar, 1997). For example, in Case Study 3, respondents in the qualitative component spontaneously introduced income generation techniques as an important factor and made associations between those techniques and their relationships with police. Through integration of the qualitative results with the quantitative findings, the investigators were able to provide a more holistic view of the complex system influencing respondents’ risk for HIV. These underlying issues should be considered in advance when designing a mixed methods study, and the different methods can be used to capitalize on their unique strengths.

While we acknowledge that not all mixed methods research projects will produce conflicting findings, anticipating that possibility may serve to benefit future projects that employ similar designs. Based on the lessons learned from these case studies we conclude with a brief discussion of some logistical challenges that face investigators implementing mixed methods studies, including challenges to study design, grant writing and manuscript publication.

In terms of the design and implementation of mixed methods studies, several issues can be considered in advance. These include project timelines, sampling schemes, and the use of additional methods. Planning for the work of integrating findings during the study design process and developing project timelines in advance that allow time for integration and/or subsequent data collection phases is a critical factor in ensuring the success of mixed methods studies. Study designs that are flexible enough to incorporate emerging findings during the research process and to allow one phase to inform the next (rather than holding all integration activities for the end) may have a greater chance of success. However, funding mechanisms are often oriented around linear rather than iterative or reciprocal study designs. More complex iterative designs may require longer timelines than are allowable under most common funding mechanisms. Successful integration of discrepant findings will be more likely if project timelines are sufficient, or supplementary funding is specifically available for mixed methods projects.

Another design issue to be addressed in mixed methods studies is that of sampling. Priorities for both sample size and sampling strategy may differ between qualitative and quantitative methods – while quantitative methods are concerned primarily with demonstrating sufficient statistical power, qualitative methods are more concerned with achieving conceptual or theoretical saturation. Planning a mixed methods study that allows for the collection of qualitative data in a way that is sufficiently flexible to achieve saturation can be challenging, particularly given the requirements of the grant development and ethical review process that sample sizes be determined a priori (Mason, 2010). Long standing qualitative approaches such as Grounded Theory (Charmaz, 2006; Glaser & Strauss, 1967; Strauss & Corbin, 1997), in which sampling is guided by a search for theoretical saturation and can evolve to address emergent issues in the data, may be particularly informative here. The use of participant observation or ethnographic methods to inform quantitative sampling strategies may also be recommended (Clatts, Davis, & Atillasoy, 1995).

Planning for subsequent phases of data collection that are designed to enlist the respondents in the reconciliation process (e.g. “member checking” in Case Study 2) can aid in the reconciliation of findings from multiple methods, and can provide an explicit forum for one phase to inform the other. Other designs may include the use of additional methods of data collection. While the projects described here made use of particular forms of qualitative and quantitative methods, including other data collection methods such as GIS mapping, participant observation, visual ethnography or social network analysis could provide yet another perspective on the phenomenon under study. Importantly, none of these approaches negate the need for thorough piloting of interviews (both qualitative and quantitative). Including community members in this process may further enhance validity.

Finally, compared to single method studies, manuscripts that report integrated findings from mixed methods studies are often more challenging to publish. They are often longer, frequently include more complicated methods sections, and are likely to include a range of results from qualitative, quantitative or other methods. They can also be challenging for both readers and peer-reviewers, who may have greater expertise or confidence in one method but not others (Bryman, 2007). Epistemological differences that underpin journal publishing preferences may further compound these difficulties (Rhodes, Stimson, Moore, & Bourgois, 2010). Perhaps as a result of some of these challenges, there are still relatively few published papers available to serve as exemplars of an integrative mixed methods approach (Bryman, 2007; Creswell & Tashakkori, 2007).

Conclusion

Though methodologists and scholars continue to debate the merits of various theoretical approaches to the integration of mixed methods research, the use of mixed methods designs is increasingly being used in public health research, including research with drug using populations. Whether these approaches are employed from a purely pragmatic perspective or from a more theoretical one, they have the potential to present challenges to integration. Whether one subscribes to a positivist view of a single, empirically measurable “truth” that can be derived, or to a more postmodern view of reality in which findings are viewed as socially-situated and partial, researchers will face challenges in the integration of findings from mixed methods studies. Despite the challenges inherent in reconciling apparently conflicting findings from qualitative and quantitative approaches, in keeping with others who have argued in favour of integrating mixed methods findings, we contend that such an undertaking has the potential to yield benefits that emerge only through the struggle to reconcile discrepant results. That is, the integration of mixed methods results (even if they initially seem to disagree) may provide a sum that is greater that the individual qualitative and quantitative parts (Bryman, 2007), and the reflexive process of engaging with the discrepant findings can itself be a useful undertaking (Fielding, 2009). On a practical level, we suggest that when planning a mixed methods research project there is considerable value in preparing for the possibility that different methodological approaches may produce different and often conflicting results, that processes for engaging with these differences should be planned in advance, and that engaging with the process of integrating results has the potential to produce richer, more nuanced, and ultimately more useful results than either method does alone.