Qualitative Methods in Mental Health Services Research

Abstract

Qualitative and mixed methods play a prominent role in mental health services research. However, the standards for their use are not always evident, especially for those not trained in such methods. This paper reviews the rationale and common approaches to using qualitative and mixed methods in mental health services and implementation research based on a review of the papers included in this special series along with representative examples from the literature. Qualitative methods are used to provide a “thick description” or depth of understanding to complement breadth of understanding afforded by quantitative methods, elicit the perspective of those being studied, explore issues that have not been well studied, develop conceptual theories or test hypotheses, or evaluate the process of a phenomenon or intervention. Qualitative methods adhere to many of the same principles of scientific rigor as quantitative methods, but often differ with respect to study design, data collection and data analysis strategies. For instance, participants for qualitative studies are usually sampled purposefully rather than at random and the design usually reflects an iterative process alternating between data collection and analysis. The most common techniques for data collection are individual semi-structured interviews, focus groups, document reviews, and participant observation. Strategies for analysis are usually inductive, based on principles of grounded theory or phenomenology. Qualitative methods are also used in combination with quantitative methods in mixed method designs for convergence, complementarity, expansion, development, and sampling. Rigorously applied qualitative methods offer great potential in contributing to the scientific foundation of mental health services research.

There is a rich tradition of using qualitative methods in mental health services research, most notably represented in the ethnographies of populations with mental health problems (e.g., Estroff, 1981; Hopper, 1988), and the institutions that serve them (e.g., Caudill, 1958; Goffman, 1961; Rhodes, 1991). Nevertheless, as in other areas of scientific research (Kuhn, 1970; Patton, 2002), qualitative methods in mental health services research have long been regarded as being “unscientific”, largely due to a lack of understanding of and experience with such methods (Robins, Ware, dosReis, Willging, Chung, & Lewis-Fernández, 2008; Hopper, 2008,Scarpinati Rosso & Bäärnhielm, 2012). This perspective began to change in the last two decades with calls for more of an interdisciplinary perspective and a recognition that qualitative methods could offer more in terms of an understanding of the need for and delivery of health services in general (Berwick, 2008) and mental health services in particular (Hohmann, 1999; Slade & Priebe, 2001) than was available from the use of quantitative methods alone. Since that time, qualitative methods have increasingly been used in mental health services research, both as the primary or exclusive method of data collection and analysis (e.g., Brunette et al., 2008; Proctor et al., 2008; Ware et al., 1999), and increasingly when combined with quantitative methods in mixed method designs (Robins et al., 2008; Palinkas, Horwitz, Chamberlain, Hurlburt, & Landsverk, 2011). In both instances, there have been concerted efforts to demonstrate the rigor applied to the collection and analysis of qualitative data as well as the scientific basis for qualitative methods, characteristics that are also valued in the use of quantitative methods. In addition, the unique value of qualitative methods to scientific inquiry and understanding of mental health services has become more evident.

The aim of this paper is to provide an overview of the use of qualitative and mixed methods in mental health services and implementation research by drawing from the examples of their use embodied in the other papers in this special series, as well as from the larger mental health services literature, and to offer some guidelines on how such methods can and should be used to maximize their potential and insure rigor in their application to addressing important questions related to the need for and delivery of mental health services.

Rationale for using qualitative methods

Qualitative methods represent an approach to understanding that does not require, or does not lend itself to, enumeration (Bernard, 2002). They can be viewed as both an art and a science. As in other fields of inquiry, they have often been used in mental health services research to provide a “thick description” (Geertz, 1970) of phenomena by providing a depth of understanding to complement the breadth of understanding afforded by quantitative methods, aiding in the interpretation of results obtained from quantitative methods, and contextualizing phenomena of interest. Examples of the use of qualitative methods in mental health services research for this purpose include Rhodes' (1991) ethnography of an emergency psychiatric unit; a descriptive account of the way in which clinicians reported making treatment decisions, their beliefs about how decisions should be made, and barriers to making treatment decisions (Simmons, Hetrick & Jorm, 2013); use of qualitative data to contextualize the outcomes evaluation of a quality improvement approach for implementing evidence-based employment services in specialty mental health clinics (Hamilton et al., 2013); and an examination of the context and intervening mechanisms of an RCT evaluating an intervention for shared care in mental health (Byng, Norman, Redfern & Jones, 2008). In the papers included in this special series, Rodriguez, Southam-Gerow and O'Connor (in press) use qualitative methods to “localize” evidence-based practices by providing the necessary insight into the local context in which practices that have been evaluated for their “global” generalizability must be applied.

Another major reason for the use of qualitative methods is that they are ideal for eliciting the perspectives of those being studied. Qualitative methods “allow people to speak in their own voice, rather than conforming to categories and terms imposed on them by others” (Sofaer, 1999, p. 1105). By eliciting participant perspectives, qualitative methods serve to enhance the validity of data being collected because it enables the investigator to compare their own perception of reality with the perception of those who are being studied. For instance, Lee, Munson, Ware, Ollie, Scott, and McMillen (2006) gave youths in foster care an opportunity to voice their experiences with mental health services and specific providers. Turner, Sharp, Folkes and Chew-Graham (2008) conducted in-depth interviews to explore women's views and experiences of antidepressants as a treatment for postnatal depression. In this special series, Rodriguez and colleagues (in press) used qualitative methods explicitly to better understand the perspectives of three groups of stakeholders on children's mental health services (parents, clinicians, and clinical directors) with the intention of seeing how these perspectives compared with one another. Similarly, Lyon, Ludwig, Romano, Koltracht, Vander Stoep, and McCauley (2013) used qualitative methods to elicit provider perspectives on the appropriateness of implementing an evidence-based, modular therapeutic approach within a school-based health clinic. Dorsey, Conover and Cox (in press) sampled foster parents to elicit their perspectives on engagement that could be incorporated into the adaptation of an existing engagement intervention. Murray and colleagues (2013) assessed counselors, children and caregivers perspectives on the use of Trauma-Focused Cognitive Behavioral Therapy (TF-CBT) in Zambia.

Qualitative methods are often found to be especially useful during initial stages of research because they enable investigators to acquire some understanding of issues, to obtain “pilot data”, or when there is too little previous research or absence of theory to allow for identification of hypotheses to be tested. Examples of such use include an exploration of the experiences of providers who have encountered immigrant patients in their services on a daily basis and to reflect on areas where difficulties may arise and how these are managed in mental health services (Sandu et al, 2012) and an exploration of wraparound services for youth with serious emotional disturbance (Mendenhall et al., 2013). Chew-Graham, Slade, Montana, Stewart, and Gask (2007) used qualitative methods to explore the function of community health teams in managing referral decisions at the primary-specialist interface from the perspectives of both referrers (primary care providers) and referred to (mental health specialists). In this special series, Lyon and colleagues (2013) employed qualitative methods “because the study of modular psychotherapy is in its infancy.” Dorsey and colleagues (in press) used qualitative methods as part of a pilot project effort to tailor an existing engagement intervention for use in implementing TF-CBT with a small group of foster parents to identify any population-specific areas of adaptation to the engagement intervention. Such exploration can be used to develop new theories or conceptual frameworks or to expand upon existing ones and to generate new hypotheses that may be tested using quantitative methods or to develop valid and reliable quantitative methods by identifying the content and form of questions to be asked and by identifying the target population through observation and interviews. For instance, Byng and colleagues (2008) used qualitative methods to generate provisional hypotheses, ready to be tested using the analytic induction process, that were derived both from themes emerging during initial coding of interview transcripts and by examining the matrix of obvious patterns. Beehler, Funderburk, Possemato and Vair (2013) used qualitative methods to develop a self-report measure of behavioral health provider adherence to co-located, collaborative care.

Finally, qualitative methods have been used in mental health services research for an evaluation of process. Such methods are frequently used in evaluation research to explain how a program or intervention operates. Harris and colleagues (2012) used qualitative methods to help understand why Early Intervention in Psychosis (EIP) services are effective by exploring the personal experiences of a small sample of services users. Byng et al. (2008) used qualitative methods to conduct a process evaluation of a complex intervention for shared care in mental health. In the papers in this series, Aarons and colleagues (in press) use qualitative methods to examine the process of scaling up of an EBP across an entire service system using an Interagency Collaborative Team (ICT) approach. Such methods can provide insight as to why program does not work as intended; it may also provide insight into unanticipated benefits or outcomes.

Characteristics of Qualitative Methods

Qualitative and quantitative methods are similar in that they both adhere to certain principles of scientific inquiry and rigor. The principles of validity, reliability, generalizability and objectivity that govern sound quantitative research have their counterparts in the principles of credibility, dependability, transferability, and reflexivity that govern sound qualitative research (Patton, 2002; Bernard, 2002). One of the techniques used to insure validity in qualitative methods is the concept of “saturation, the point at which no additional data collection in needed, no new codes are developed, and themes and subthemes have been fully fleshed out” (Padgett, 2008, p. 171). Saturation refers to completeness or fullness necessary to insure that everything related to the phenomenon of inquiry that can be collected and analyzed has been done so within the limits of the forms of collection and analysis chosen. In a study of women's attitudes toward using antidepressants to treat postnatal depression, Turner and colleagues (2008) reached saturation of key themes after interviews with 27 women, while Chew-Graham et al. (2007) achieved saturation with 35 general practitioners. Another technique used to insure validity is the identification of deviant or nonconfirmatory cases, the exceptions to the rule. This technique was employed by Turner and colleagues (2008) in their study of women's views of antidepressants as a treatment for postnatal depression and by Chew-Graham and colleagues (2007) in their study of community mental health teams. Validity has also been supported by means of “member checking” whereby study participants or others who share similar characteristics review study findings to confirm and potential elaborate on them. In their study, Dorsey and colleagues (in press) presented the findings of their interviews with foster parents to two boards, one comprised of foster parents and one comprised of caseworkers. They further “triangulated” the data collected from both boards.

Reliability or dependability of qualitative data analysis is usually achieved by establishing a specified level of agreement in identification of topics or themes through both qualitative and quantitative means. The inductive approach typically includes a process of “coding by consensus”, which includes one or both of two activities, 1) regular meetings among coders to discuss procedures for assigning codes to segments of data and resolve differences in coding procedures, and 2) comparison of codes assigned on selected transcripts to calculate a percent agreement or kappa measure of inter-rater reliability. Most studies in the mental health services literature report the first method (e.g., Gilburt, Slade, Bird, Oduola & Craig, 2013; Mittal, Drummond, Blevins, Curran, Corrigan, & Sullivan, 2013; Turner et al., 2008), while others report measures of inter-rater reliability of coding of qualitative transcripts (e.g., Bradley et al., 2003; Lee et al., 2007; Palinkas et al., 2008). Rodriguez and colleagues (in press) appear to have used both methods to insure the reliability of coding assigned; however, they also acknowledge that they relied primarily on a priori codes rather than identify new or emergent codes. (in press) and colleagues, Lyon and colleagues (2013) and Dorsey and colleagues (in press) rely only on the first method.

Qualitative methods also acknowledge the importance of generalizability, often referred to in the literature as the transferability of findings from one context or population to another. For the most part, this acknowledgment is usually described as a limitation. For instance, Rodriguez and colleagues (in press) note that all the parents recruited in their study were women. Lyon and colleagues (2013) urge caution when generalizing findings to other models or settings because their study included a sample of clinicians working in one particular type of school-based service delivery system. Murray and colleagues (2013) note their study was conducted only with children and caregivers of children who completed treatment and thus did not include the perspectives of children who did not complete treatment. It was also limited to local lay workers in an urban setting. However, as noted earlier, as qualitative methods are designed for depth and not breadth of understanding, the generalizability of findings is of less importance at this stage of the process of scientific inquiry than the attainment of maximum insight from the data that are collected.

Another characteristic of some qualitative studies is the explicit reflexivity employed by investigators as a means of identifying and addressing potential biases in the collection and interpretation of data. Such bias may be associated with the investigator's preconceived beliefs, assumptions and theoretical orientations; demographic characteristics; experience with the methods used and familiarity with the phenomenon under investigation. Gianakis and Carey (2011), for instance, make explicit their background and initial expectations in their study of adults who experience improvement in psychological functioning without the benefit of psychotherapy.

Other methods used in qualitative studies to enhance rigor of analysis include triangulation of viewpoints by purposefully interviewing people in various roles within an organization, peer debriefing and support meetings among research team members, and providing a detailed audit trail during analysis (Harris, Collinson, & das Nair, 2012; Miles & Huberman, 1994; Mendenhall, Kapp, Rand, Robbins, & Stipp, 2013).

Design Strategies

While adhering to the same scientific standards, qualitative methods are nevertheless distinguished from quantitative methods by certain other characteristics, including design strategies, data collection and analysis strategies. One of the most obvious distinctions in the design of qualitative studies is the reliance upon generally smaller samples for investigation. Although there are qualitative mental health services studies involving large samples, they generally do not require them since their aim is to achieve a depth of understanding rather than a breadth (generalizability) of understanding. Consequently, the number of participants is often based on availability of participants and feasibility of data collection rather than quantitative power calculations. Nevertheless, there are certain conventions in identifying how many participants to include in a qualitative study, including precedent and saturation (Guest, Bunce & Johnson, 2006; Miles & Huberman, 1994; Strauss & Corbin, 1998).

Another familiar feature of qualitative study designs is the use of purposeful sampling to identify and recruit study participants. Unlike quantitative studies that rely on random samples to minimize bias and confounding, qualitative studies rely principally on “purposeful sampling” designed to maximize the information gained from what is typically a much smaller group of participants than found in most quantitative studies (Palinkas, Horwitz, Green, Wisdom, Duan, & Hoagwood, 2013). Among the most common forms of purposeful sampling are extreme or deviant case sampling, criterion sampling and maximum variation sampling. Sampling extreme or deviant cases is designed to reduce variation and highlight the most prominent features of a phenomenon under investigation. For instance, Sandu and colleagues (2012) sampled services providers in 16 European countries after first sampling mental health services in areas with high concentration of immigrants in consultation with a research center for that country and then asking for an interview with a practitioner with the most extensive experience of providing mental health care to immigrants in the service. Criterion sampling is also intended to reduce the range of variation and limit the possibility of collecting information not directly related to the phenomenon of interest. For instance, Byng et al (2008) limited their sample to participants of an RCT of a complex health services intervention for shared care for people with long-term mental illness. As its name suggests, maximum variation sampling is intended to expand the range of variation and thereby select participants who are representative of a larger population and can maximize the opportunity for a comprehensive view of the phenomenon of interest. In a study of implementation issues related to several evidence-based practices for adults with serious mental illness that were included in a national demonstration project, Isett and colleagues (2007) asked the mental health commissioner's office in each participating state to identify potential participants who were knowledgeable about evidence-based practice (itself a criterion) and came from various backgrounds to capture a broad range of perspectives. Chew-Graham et al (2007) sampled general practitioners to insure variation in gender, ethnicity, experience, and practice size. Other forms of purposeful sampling have been used in mental health services research, including random sampling (Mendenhall et al., 2013) and convenience sampling (Mittal et al., 2013) or both (Stergiopoulos et al., 2012), but these two in particular are usually considered less likely to obtain information rich participants (Patton, 2002).

In this special series, Rodriguez and colleagues (in press) wanted to obtain a sample that was representative of the clinics that were the focus of their investigation using different sampling methods. They did so by sampling the universe of directors and clinic-affiliated therapists who worked in the study clinics and a subset of parents through informational flyers and invitations from providers. The other studies also appear to have used criterion sampling to identify potential participants, including full-time mental health provider in their respective schools (Lyon et al., 2013), foster parents based on referrals from caseworkers and youth exposure to at least one traumatic event and youth PTS symptoms (Dorsey et al., in press), all “outer” and “inner” context stakeholders involved in the implementation of an evidence-based practice in one county (Aarons et al., in press) and participants in a larger feasibility study of TF-CBT in Zambia (Murray et al., 2013). While these strategies are commonly employed in mental health services researchers, investigators should be mindful of the fact that representativeness is only one criterion for purposeful sampling. Other important criteria include familiarity with the topic under investigation based on experience, a willingness to share that information, and an ability to share that information based on the informant's own insight and communication skills (Bernard, 2002).

Another feature of qualitative research design observed in mental health services research is its emphasis on naturalistic inquiry, “a ‘discovery-oriented’ approach that minimizes investigator manipulation of the study setting and places no prior constraints on what the outcomes of the research will be (Patton, 2002, p. 39). Qualitative designs are, for the most part, observational in nature. Data are collected in situ, usually as events happen. These qualities are more often employed in long-duration ethnographic studies that make use of participant observation than in focus groups and semi-structured interviews to collect information guided by a priori conceptual frameworks (Padgett, 2008). The reason for this focus is to avoid or eliminate any potential bias, either on the part of the observer or those being observed or interviewed (Guba, 1978).

A final characteristic of qualitative studies of mental health services is the emergent and iterative nature of qualitative research. The emergent design is based on the principle that circumstances often dictate changes in focus or means of data collection and that the researcher should be prepared to accommodate to those changes rather than adhere to a plan to use potentially inappropriate or inadequate methods (Padgett, 2008). Qualitative mental health services research are often iterative in nature in which there is a constant back and forth between data collection and analysis (Bernard, 2002). In contrast, quantitative studies generally initiate the analysis phase only after data collection is complete or near completion. For instance, Isett and colleagues (2007) developed a protocol for conducting follow-up interviews based on three dominant themes identified in their analysis of the first set of interviews. Chew-Graham et al (2007) modified their interview schedule in light of emerging data.

Data Collection and Fieldwork Strategies

There exist several different forms of data collection strategies in qualitative studies of mental health services. The most common strategies are extended interviews and focus groups, followed by ethnographic fieldwork, document reviews and more structured approaches that involve both qualitative and quantitative methods. The extended interview is the most frequently used method for data collection in mental health services research and is intended to elicit information on the participant's experience, opinions, and perceptions of mental health services. This form of data collection can range from brief responses to open-ended questions on more structured interviews or surveys (e.g., Marcus, Westra, Angus, Kertes , 2011) to a series of extended interviews with “key informants,” individuals especially knowledgeable about the topic under examination (e.g., Sandu et al., 2012). In this special series, Aarons and colleagues (in press) conducted interviews with executive staff from a state child welfare system, CBOs providing home visitation services, and a local foundation, using an interview guide consisting of open-ended questions tailored to each stakeholder group to assess roles and responsibilities and perceptions of the implementation of the SafeCare intervention. Lyon et al. (2013) conducted one-hour semi-structured interviews with 17 school-based mental health providers. Dorsey and colleagues (in press) conducted interviews of shorter duration (13 to 27 minutes) with 7 foster parents to collect information on the initial telephone call to facilitate engagement and experience with the first TF-CBT treatment session. Rodriguez et al. (in press) interviewed three parents using a semi-structured guide to obtain perceptions of causes of anxiety, depression and conduct-related problems in children; ideal treatments for these problems; barriers to making these treatments available; and additional comments. Murray et al (2013) trained counselors to ask a series of six open-ended questions to children in Zambia undergoing treatment for trauma and their caregivers, although they acknowledge potential problems with counselors interviewing children and caregivers, leading to hesitation to report negative feedback. This was followed by a second interview for further clarification and probing on the participants' initial responses.

A particular form of extended interviewing is the structured narrative, in which the participant describes the experience of having an illness and seeking services for that illness. Scarpinatti Rosso and Bäärnhielm (2012) collected narratives from 23 newly referred immigrants seeking help at a psychiatric outpatient clinic in Stockholm, Sweden, using a Cultural Formulation (CF) interview protocol (Bäärnhielm, Scarpinatti Rosso, & Pattyi, 2009). Marcus et al (2011) used narratives to understand client experiences of using motivational interviewing for treatment of generalized anxiety disorder. Rappaport, Jerzembek, Doel, Jones, Cella and Lloyd (2010) used narrative content analysis (Coffey & Atkinson, 1996) to construct narratives of uncertainties about treatment of mental health from free text responses to a questionnaire.

Another form of data collection that has been used extensively in mental health services research is the focus group. Focus groups are interviews that are designed to use group interaction to generate data and insights less accessible in individual interviews (Krueger, 1988; Morgan, 1988). Although this cannot always be achieved in service settings, the ideal composition of a focus group is between 6 and 10 “homogeneous strangers,” individuals who are similar by virtue of their experience with or familiarity with the topic but who otherwise are not closely linked to one another. In the articles in this special series, Aarons and colleagues (in press) conducted 9 focus groups with case manager supervisors, trainers, the seed team, and service provider team trained by the seed team. Rodriguez and colleagues (in press) conducted two focus groups with 11 providers using a guide similar in structure to the one used with individual interviews with parents; however, they acknowledge that while perhaps logistically feasible, the focus group format “created an uncomfortable environment for staff, making it difficult to disclose in the presence of fellow colleagues”.

Ethnographic fieldwork often consists of several different modes of data collections, but perhaps the most distinctive feature is the technique of participant observation. Participant observation consists of spending time and talking with people in their own settings (Ware et al., 1999). Estroff conducted ethnographic fieldwork with a group of discharged mental hospital patients living in Madison, Wisconsin. Her participation included taking antipsychotic medication to better understand the challenges of living with their side effects. Ware et al (1999) engaged in participant observation at two public community mental health centers (CMHCs) and one emergency psychiatric evaluation unit in Boston to identify the interpersonal processes of giving and receiving day-to-day services through which individual providers create experiences of continuity for consumers. Palinkas and colleagues (2008) participated in training workshops in three different evidence-based treatments whose effectiveness in their standard use and in a modular fashion in a randomized controlled trial. They also conducted site visits of each of the clinics participating in the study.

Finally, some qualitative mental health services studies have relied upon more quasi-statistical techniques for data collection. These techniques often represent the iterative nature of qualitative methods in that the investigators alternate between qualitative data collection, transformation of qualitative data into quantitative data, and validation or elaboration using another round of qualitative data collection. An illustration of this process is the technique of concept mapping (Trochim, 1989). Aarons, Wells, Zagursky, Fettes, and Palinkas (2009), solicited information on factors likely to impact implementation of EBPs in public sector mental health settings from 31 services providers and consumers organized into 6 focus groups. Each participant then sorted a series of 105 statements into piles and rated each statement according to importance and changeability. Data were then entered in a software program that uses multidimensional scaling and hierarchical cluster analysis to generate a visual display of how statements clustered across all participants. Finally, 22 of the original 31 participants assigned meaning to and identified an appropriate name for each of the clusters identified (Aarons et al, 2009). Another technique for data collection that relies on the iterative collection and analysis of data is the Delphi approach where opinions from content experts are collected and summarized with the primary goal of consensus building, thereby helping to insure content validity. Beehler et al. (2013) used this technique to develop a list of self-report measures of behavioral health provider adherence to co-located, collaborative care, beginning with the development of a 56-item measure of collaborative care, obtaining qualitative feedback from content experts while quantitatively rating each item's relevance for behavioral health provider practice through three rounds of emailed surveys. Items with consensus ratings of 80 percent or greater were included in the final adherence measure.

Data Analysis Strategies

For the most part, qualitative studies rely on a variety of methods for inductive analysis and creative synthesis. For instance, Byng and colleagues analyzed their data using Realistic Evaluation, “a framework for a context sensitive process evaluation accompanying an RCT, designed to unpack the complexity of the intervention by examining interactions between intervention components and context and then further refining its core functions” (2008, pp. 3-4). This modified form of analytic induction was used to examine the empirical data from case studies and iteratively build “middle range theories.” However, there are instances of qualitative mental health services research that have also employed deductive approaches. For instance, in a study of stigma associated with PTSD among treatment seeking combat veterans, Mittal and colleagues (2013) began with an inductive approach based on grounded theory methods (Strauss and Corbin, 1998), followed by a deductive analysis with the use of an a priori model of the participants reaction to the stigmatizing labels they perceived. Hamilton and colleagues (2013) used a hybrid deductive/inductive thematic analysis approach in their study of implementation of employment services in specialty mental health.

Many of the coding strategies employed for analyzing qualitative data in mental health services research fall under the general rubric of “content” or “thematic” analysis. Such analysis often involves a rigorous process of reviewing transcripts and other documents line by line and assigning codes based on a priori and/or emergent topics or themes, and the construction of themes (Miles & Huberman, 1994; Strauss & Corbin, 1998). Coding also occurs in stages in which initial preliminary codes are followed by secondary or focused coding (e.g., Green et al., 2008; Simmons et al., 2013), or in which open codes are followed by axial codes (e.g., Hamilton et al., 2013). The papers by Rodriguez and colleagues (in press), Lyon and colleagues (2013), and Dorsey and colleagues (in press) provide an illustration of the inductive process or content or thematic analysis. They refer to the process of “unitizing” (i.e., construction of units) the data by creation of codes based on an a priori classification system (the MHSE model), construction of a codebook with a list of these units, and then the identification of themes through the use of text analysis software such as NVivo (Dorsey et al., in press; Rodriguez et al., in press) or Atlas.ti (Lyon et al., 2013). The papers by Murray et al (2013) and Aarons et al. (in press) employ another commonly used analytic process found in MHSR qualitative studies. Similar in many ways to the content analysis based on a priori topics described above, this process adheres more to a grounded theory (Glaser & Strauss, 1967; Strauss & Corbin, 1998) in which both a priori and emergent topics are coded to construct a conceptual framework or theory. Stergiopoulos and colleagues (2012) analyzed interview and focus group transcripts of a Housing First model for homeless individuals with mental illness using a grounded theory methodology. Isett et al (2007) utilized grounded theory case study methods developed by Yin (2003). In addition to systematizing the process of coding the data, qualitative analysts using this approach engage in an iterative process of “constant comparison” (e.g., Chew-Graham et al., 2007; Turner et al., 2008).

Although not as common as the grounded theory approach, another approach to qualitative data analysis used in mental health services is based in a phenomenological tradition. Drawing from the work of Husserl (1962), Schutz (1970), and others, phenomenology aims at gaining a deeper understanding of the nature or meaning of our everyday experiences. As equally concerned with rigor as is the grounded theory approach described above, phenomenology gives more attention to understanding the lived experience of individuals using or in need of mental health services while controlling for preconceptions and potential biases on the investigator. Gianakis and Carey (2011) utilized Interpretative Phenomenological Analysis (IPA: Smith, 1996), a cyclical iterative process with a constant revisiting of transcripts to insure the superordinate themes generated directly relate to the shared experience of the participants, in their investigation of the phenomenological experience of psychological change following distress from a range of problems in individuals who have not used psychotherapy to resolve those problems. Harris and colleagues (2012) used IPA to explore the experiences of being in contact with Early Intervention in Psychosis (EIP) services in a small sample of service users.

While the detail underlying the coding of data and generation of themes is critical to demonstrating the rigor applied to qualitative analysis, a holistic perspective is equally important. This perspective requires that the investigator goes beyond the enumeration of themes to provide “the big picture”, for instance, by explaining how the themes are linked together to provide a more comprehensive understanding of their meaning, operation, and relationships, and by paying particular attention to context. For instance, in the study by Lyon and colleagues (2013), key elements of the fit between the modular approach and the school context at the client and clinician levels are summarized, elaborated and integrated using the identified themes. Aarons and colleagues (in press) relied on the principle of constant comparison to condense coding categories into broader themes using a format that placed the phenomenon under investigation within a broader framework of understanding collaborations, negotiations and resolutions while considering inner and outer contextual characteristics.

One analytical strategy used in mental health services research to provide such a holistic perspective is the case study approach. Often relying on multiple forms of qualitative data (interviews, focus groups, participant observation) rather than a single form, case studies are less concerned with representativeness or generalizability and more concerned with richness in detail of individuals, groups, organizations, systems or experiences and their context (Yin, 2003). Examples include a multiple case study of implementation as usual in children's social service organizations (Powell et al., 2013), a phenomenological case study of communication between clinicians about attention-deficit/hyperactivity disorder assessment (Lynch, Cho, Ogle, Sellman & dosReis, 2014), and an evaluation of the state policy context of implementation of several evidence-based practices for adults with serious mental illness (Isett et al., 2007).

Mixed Methods

Qualitative methods are increasingly represented in mental health services research in the form of mixed method designs that focus on collecting, analyzing and merging both quantitative and qualitative data into one or more studies. The central premise of these designs is that the use of quantitative and qualitative approaches in combination provides a better understanding of research issues than either approach alone (Cresswell & Plano Clark, 2011). In such designs, qualitative methods are used to explore and obtain depth of understanding while quantitative methods are used to test and confirm hypotheses and obtain breadth of understanding of the phenomenon of interest (Teddlie & Tashakkori, 2003).

Mixed method designs in mental health services research can be categorized in terms of their structure, function, and operation (Palinkas et al., 2011). Quantitative and qualitative methods may be used simultaneously or sequentially, with one method viewed as dominant or primary and the other as secondary, although equal weight can be given to both methods. The function of mixed method designs are usually based on whether the two methods are used to answer the same question or to answer related questions and whether they were used to achieve convergence (using both types of methods to answer the same question, either through comparison of results to see if they reach the same conclusion or by converting a data set from one type into another, e.g., quantifying qualitative data or qualifying quantitative data); complementarity (using each set of methods to answer a related question or series of questions for purposes of evaluation or elaboration, e.g., using qualitative data to examine treatment process and quantitative methods to examine treatment outcome); expansion (using one type of method to answer questions raised by the other type of method, e.g., using qualitative methods to explain findings from an analysis of quantitative data); development (using one type of method to answer questions that will enable use of the other method to answer other questions, e.g., using qualitative methods to construct a questionnaire or a theoretical model that can be tested using qualitative methods); or sampling (using one type of method to define or identify the participant sample for collection and analysis of data representing the other type of method, e.g., purposefully selecting participants for individual interviews based on their responses to a survey). Finally, the use of mixed methods in mental health services research involves three distinct processes or strategies for combining qualitative and quantitative data: merging or converging the two datasets by actually bringing them together, connecting the two datasets by having one build upon the other, or embedding one dataset within the other, so that one type of data provides a supportive role for the other dataset.

Some of the articles in this special series offer illustrations of the combining of qualitative data. For instance, Rodriquez and colleagues enumerated the number of units within each of the identified themes and compared these units using nonparametric statistics. Lyon et al. (2013) identified the percentage of participants who mentioned a particular topic or theme during their interview. The salience of the topics was indicated by the percentage of clinicians who discussed them during the interviews. Murray and colleagues (2013) also use frequency counts to indicate the salience or importance of identified themes. The study by Rodriguez et al (in press) was an early phase of a mixed-method university community partnership endeavor designed to adapt and test evidence-based practices for anxiety and depression. The study by Lyon et al (2013) occurred subsequent to an intervention study, while the study by Dorsey et al. (in press) occurred in the first phase of a two-phase feasibility trial of TF-CBT with youth in foster care.

The technique of concept mapping used by Aarons et al. (2009), where qualitative data elicited from focus groups are “quantitized” using multidimensional scaling and hierarchical cluster analysis, is an example of convergence. In a study of the implementation of evidence-based psychotherapies for PTSD in VA specialty clinics, Watts and colleagues (2014) conducted semi-structured interviews with staff at participating clinics using the PARiHS framework to develop overarching questions. Transcripts of these interviews were then coded by domain and element of the PARiHS framework and then a scoring rubric was used to transform each element of the framework into a numeric value. They then conducted a Poisson linear regression that used element scores for each facility as independent variables and percentage of patients at each sites receiving any evidence-based therapy as the dependent variable. Gilburt et al. (2013) used mixed methods to achieve complementarity in their evaluation of implementation of a recovery-oriented practice through training across a system of mental health services, using a quantitative audit of care plans in a random sample of 700 patients to assess change in core plan topics and in responsibility of action and semi-structured interviews with team leaders to explore understanding of recovery, implementation within the service and the wider system, and perceived impact of the training on individual practice and that of the team. The Delphi approach used by Beehler and colleagues (2013) is an example of mixed methods to achieve development and elaboration. Woltmann et al (2008) used qualitative data obtained through interviews with staff, clinic directors and consultant trainers to create categories of staff turnover and designations of positive, negative and mixed influence of turnover on implementation outcomes. These categories were then quantitatively compared with implementation outcomes via simple tabulations of fidelity and penetration means for each category.

Conclusions

Whether used in combination with quantitative methods in a mixed method design or alone, qualitative methods offer enormous potential to contribute to the field of mental health services research. Although they are distinguished from quantitative methods by features of design (reliance on small samples, purposeful sampling, emphasis on naturalistic inquiry, and iteration between data collection and analysis), data collection (interviews, focus groups, participant observation), and data analysis (grounded theory, phenemonology, holistic perspective), they share with quantitative methods a regard for rigor, validity and reliability. It must be kept in mind, however, that while there are certainly areas of overlap, qualitative methods are not a substitute for quantitative methods, but rather represent a specific set of tools that can be used with greater effectiveness in some phases of the process of scientific inquiry and with less effectiveness in others. Interview or focus group data may indeed be of little value when analyzing the outcomes of a randomized controlled trial; however, such data can enable an investigator to achieve a deeper understanding of process and context of an RCT, develop better instruments for measuring process and outcome, more efficiently target potential study participants, enhance the external validity of the findings, and account for unexplained findings of an analysis of quantitative data.