A Methodology for Conducting Integrative Mixed Methods Research and Data Analyses

Abstract

Mixed methods research has gained visibility within the last few years, although limitations persist regarding the scientific caliber of certain mixed methods research designs and methods. The need exists for rigorous mixed methods designs that integrate various data analytic procedures for a seamless transfer of evidence across qualitative and quantitative modalities. Such designs can offer the strength of confirmatory results drawn from quantitative multivariate analyses, along with “deep structure” explanatory descriptions as drawn from qualitative analyses. This article presents evidence generated from over a decade of pilot research in developing an integrative mixed methods methodology. It presents a conceptual framework and methodological and data analytic procedures for conducting mixed methods research studies, and it also presents illustrative examples from the authors' ongoing integrative mixed methods research studies.

Overview on Mixed Methods Approaches

Emergence of Mixed Methods Approaches

Contrasting strengths of qualitative and quantitative methods

Within the social and behavioral sciences a schism has existed for decades that separates the qualitative and quantitative research traditions (Tashakkori & Teddlie, 2003; Teddlie & Tashakkori, 2003). Recently, mixed methods approaches have emerged that offer the promise of bridging across both traditions (Haverkamp, Morrow, & Ponterotto, 2005). The strengths of quantitative approaches include the following: (a) accurate operationalization and measurement of a specific construct, (b) the capacity to conduct group comparisons, (c) the capacity to examine the strength of association between variables of interest, and (d) the capacity for model specification and the testing of research hypotheses. One major limitation of the quantitative approach is that measurement typically detaches information from its original ecological “real-world” context (Moghaddam, Walker, & Harre, 2003), a phenomenon referred to as decontextualization (Viruel-Fuentes, 2007).

In contrast, the qualitative approach examines the “whole person” holistically within that person's natural environment—a fully contextualized approach (Gelo, Braakman, Gerhard, & Benetka, 2008). The strengths of the qualitative approach include the following: (a) the capacity for generating rich detailed accounts of human experiences (emotions, beliefs, and behaviors) and (b) narrative accounts that are examined within the original context in which observations occur (Guba & Lincoln, 1994). Moreover, the qualitative approach affords an in-depth analysis of complex human, family systems, and cultural experiences in a manner that cannot be fully captured with measurement scales and multivariate models (Plano Clark, Huddleston-Casas, Churchill, Green, & Garrett, 2008). Limitations of the qualitative approach include difficulties in the reliable integration of information across observations or cases (Kirk & Miller, 1986) and difficulties in assessing links and associations that occur between observations, cases, or constructs. Furthermore, qualitative research methods often lack well-defined prescriptive procedures (Morse, 1994), thus limiting the capacity for drawing definitive conclusions (confirmatory results), an important aspect of scientific research. In addition, purely qualitative studies have been challenged for their small or unrepresentative samples, and thus their limited capacity to produce generalizable findings, although some qualitative analysts have argued that the cannons of scientific research—generalizability, replication, reliability, and validity—are not relevant for qualitative research (Denzin & Lincoln, 1994). Whereas this alternative perspective has raised important epistemological issues, nonetheless, purely qualitative studies have often been regarded as methodologically weak when applied to the conduct of scientific research (Dreher, 1994).

Issues of sample size and approach

Qualitative studies are idiographic in approach, typically focusing on depth of analysis in small samples of participants. One pervasive qualitative practice in sample selection is the goal of “reaching saturation.” Once the investigator concludes that response saturation has been attained, sampling ceases. However, criteria for defining “saturation” are often intuitive or inexact. Unfortunately, saturation promotes the collection of smaller, “just enough” sized samples, for example, samples sizes of 8 to 20, which from a quantitative perspective is antithetical to attaining sufficiently large-sized samples for conducting stable multivariate data analyses (Dreher, 1994) that can generate credible research results. In contrast, under an integrative mixed methods (IMM) study, the determination of an appropriate sample size requires a broader integrative perspective: (a) that balances qualitative considerations favoring small manageable samples for conducting in-depth qualitative analyses (n = 20–40), against (b) quantitative considerations favoring larger sample sizes (n = 40–200) for conducting reliable multivariate statistical analyses (Gelo et al., 2008; Yoshikawa, Weisner, Kalil, & Way, 2008).

Limitations in qualitative data analytic methods

The field of qualitative research has been rich in strategies for “entering the field” and for engaging special or hidden populations (Denzin & Lincoln, 1994), although by contrast qualitative approaches have often been methodologically weak in procedures for “mixing” qualitative and quantitative methods and data and for processing their inductively derived information (verbal evidence; Dreher, 1994; Gelo et al., 2008; Plano Clark et al., 2008). These limitations include weaknesses in precisely describing interrelationships that exist among two or more of inductively generated constructs or categories. Although such associations can be explored using visual case-ordered and predictor-outcome matrix methods that allow a cross-tabulation of categorical information (Miles & Huberman, 1994), nonetheless, these methods have lacked the capacity to reliably assess the strength of association among key categories or constructs, as can be accomplished with quantitative methods such as correlational analyses.

Even among mixed methods studies, a common limitation has been the use of qualitative and quantitative approaches in a sequential temporal order, thus limiting the integration of both data forms under a unified process of data analysis (Bryman, 2007). Typically, focus group information has been obtained during Stage 1 (e.g., a pilot study) to develop or refine instruments and procedures, followed by Stage 2 (e.g., the “core study”) in which survey or other quantitative data are then collected (Creswell, 1994). Unfortunately, few studies have effectively integrated qualitative and quantitative approaches under a unified and fully integrative research design and data analytic plan (Bryman, 2007; Dreher, 1994; Hanson, Creswell, Clark, Petska, & Creswell, 2005). Based on a decade of our pilot research, the IMM approach, as presented here, has been designed for a concurrent, integrative, and unified analysis of qualitative and quantitative data. It aims to incorporate the strengths of qualitative and quantitative approaches for conducting rigorous data analyses that meet scientific standards of reliable and valid measurement and analysis.

Mixed Methods Design Approaches

Sequential mixed methods designs

Creswell, Plano Clark, Gutmann, and Hanson (2003) classified mixed methods designs into two major categories: sequential and concurrent. In sequential designs, either the qualitative or quantitative data are collected in an initial stage, followed by the collection of the other data type during a second stage. In contrast, concurrent designs are characterized by the collection of both types of data during the same stage. Within each of these two categories, there can be three specific designs based on (a) the level of emphasis given to the qualitative and quantitative data (equal or unequal), (b) the process used to analyze and integrate the data, and (c) whether or not the theoretical basis underlying the study methodology is to bring about social change or advocacy (Creswell et al., 2003). In accord with this typology, the three types of sequential mixed methods designs are (a) sequential exploratory, (b) sequential explanatory, and (c) sequential transformative.

Concurrent mixed methods designs

The three concurrent mixed methods designs identified by Creswell et al. (2003) are the following: (a) concurrent triangulation, (b) concurrent nested, and (c) concurrent transformative designs. In each of these designs, the quantitative and qualitative data are collected during the same stage, although priority may be given to one form of data over the other. The purpose of concurrent triangulation designs is to use both qualitative and quantitative data to more accurately define relationships among variables of interest. In concurrent nested designs, both qualitative and quantitative data are collected during the same stage, although one form of data is given more weight over the other (Creswell et al., 2003). Similar to sequential nested designs, concurrent transformative designs are theoretically driven to initiate social change or advocacy, and these designs may be used to provide support for various perspectives.

Integrative mixed methods designs

Within the context of these design approaches, the need persists for a methodology that affords a rigorous and integrative analysis of qualitative textual evidence and quantitative numeric data (Schwandt, 1994). Given the noted strengths and weaknesses of the qualitative and quantitative approaches, it would be advantageous to have a truly integrative methodology for the concurrent use of both methods in a manner that offers the descriptive richness of text narratives and the precision in measurement and hypothesis testing afforded by quantitative approaches (Carey, 1993; Hanson et al., 2005). Regarding such integrative designs, Creswell et al. (2003) have indicated that, “there is still limited guidance for how to conduct and analyze such transformations [the qualitative–quantitative exchange of data] in practice” (p. 229).

Integrative mixed methods paradigm

Figure 1 presents a paradigm for an IMM research approach. A core feature of this approach is parallelism in study design, where integration begins with a unified conceptualization of information as “research evidence,” which can take the form of verbal text narrative evidence (qualitative) or numeric data evidence (quantitative). This IMM design is closest in form to a “concurrent triangulation” design as described by Creswell et al. (2003), Hanson et al. (2005), and Plano Clark et al. (2008).

Figure 1.

Paradigm for the integrative mixed methods research approach

Note: This figure was adapted from an earlier version that was published in the journal, Cultural Diversity and Ethnic Minority Psychology, (2007) 13 (4), 269–284.

Based on a specified theory or conceptual framework, a core category or construct, such as machismo, can be featured as a study's core construct. The basic IMM design proceeds in six stages: (a) parallelism in study development, (b) evidence gathering, (c) processing/conversion, (d) data analyses, (e) interpretation, and (f) integration. In principle, a well-crafted study with this design would allow “seamless” data conversions, for example, the conversion of qualitative thematic categories into numeric thematic variables (Castro & Coe, 2007). Then, via recontextualization this conversion would relate statistically derived results back to their original qualitative context (Morse, 1994), thus allowing a rich interpretation of the quantitatively derived results. Generally, the greater the qualitative–quantitative parallelism that is designed a priori into a study, the easier to transform, transfer, and interpret textual and numeric data forms across modalities (Plano Clark et al., 2008). Under a full integrative perspective, the principal aim is to examine research evidence gathered using both data forms, to generate “deep structure” conclusions (Castro & Nieri, 2008) that offer enhanced explanatory power above and beyond the sole use of a qualitative or quantitative approach.

Advancing Integrative Mixed Methods Research

A case for the integrative mixed methods approach

This IMM approach builds on fundamental concepts drawn from Grounded Theory, as described by Strauss and Corbin (1990), although these investigators did not speak of mixed methods research per se. One core feature under the IMM approach is the equal emphasis given to qualitative and quantitative data forms (QUAL + QUANT; Hanson et al., 2005) to facilitate rich, “deep structure,” data analyses (Resnicow, Soler, Braithwait, Ahluwalia, & Butler, 2000) and interpretations.

Constructing and deconstructing factorially complex constructs

The IMM approach offers procedures to study factorially complex constructs, such as the Latino gender-role construct of machismo (Torres, 1998). Recently, the structure of machismo has been described as consisting of distinct positive and negative factors (Arciniega, Anderson, Tovar-Blank, & Tracey, 2008; Rollins, 2003). Social science research features many such factorially complex constructs. These constructs include the following: acculturation (Lara, Gamboa, Kahramaninan, Morales, & Hayes Bautista, 2005), ethnic identity (Phinney, 1990), biculturalism (LaFromboise, Coleman, & Gerton, 1993), resilience (Masten, 2001), well-being (Jones & Sumner, 2009), leadership (Hogan & Kaiser, 2005), self-regulation (Gross & John, 2003), and various emotions such as guilt and regret (Zeelenberg & Bruegelmans, 2008) and anticipated regret (Sheeran & Orbell, 1999).

Describing the nuances and complexities of emotions

Research in health psychology has long examined and tested various cognitive models of health-related behaviors, such as the health belief model (Champion & Skinner, 2008). Recently, these models have been criticized for their overemphasis on cognitive–rational decision making, limiting attention to other important factors, such as emotions, which can also influence health-related behaviors (Moser, 2010).¹ The assessment of emotions as motivational factors in models of health behavior has been difficult partly because the self-report measurement of emotions using scales has typically been unidimensional and because it often assesses cognitive aspects of emotion, for example, cognitions about anxiety. The IMM approach may aid in a more complete assessment of emotions as motivators of health-related behaviors by capturing the affective verbal responses of complex emotions within their situational context. The reliable encoding of complex emotions, such as ambivalence, could provide new insights into the influences of such emotions as motivational determinants of health-related behaviors.

Temporal process analysis

Based on our prior research, the IMM approach can also be used to conduct a temporal analysis of events. An interview protocol can be developed that consists of a temporally ordered series of open-ended focus questions that examine the natural sequence of “unfolding of events” that has occurred before, during, and after a significant life event. Thus, temporal process analysis uses interview-assisted retrospective recall of relevant thoughts, feeling, and behaviors that have occurred at each of several specified “windows of time,” or milestones. For example, in a sample of drug-dependent males, we examined the temporal sequence of events (situations, thoughts, feelings, behaviors) that preceded and that followed the occurrence of a first drug relapse (or a “close call”; Denne, Castro, & Harris, 2001).

Goals of This Methodological Description

A major goal of the present IMM methodological description is to present issues and methods for the design and implementation of an IMM study (Castro & Nieri, 2008). A second goal is to describe methodological adaptations of our original IMM approach (Castro & Coe, 2007), which was originally developed using an earlier-generation text analysis software program, TextSmart 1.1 (Statistical Package for the Social Sciences, 1997).² We have adapted this IMM approach for use with a later-generation qualitative text analysis program, Atlas.ti (Muhr, 2004). Using selected cases from our ongoing studies, we will illustrate specific aspects of this IMM approach for conducting scientifically rigorous and culturally sensitive data analyses that integrate qualitative and quantitative data.

A Methodology for Integrative Mixed Methods Studies

Overview

The IMM approach, as we have developed it, is implemented in six steps: (a) creating focus questions and conducting focus question interviews, (b) extracting response codes, (c) creating thematic categories (a “family” within Atlas.ti), (d) dimensionalizing the thematic category via scale coding, (e) qualitative–quantitative data analysis, and (f) creating story lines (Castro & Coe, 2007). As indicated, in Figure 2, the process of generating qualitative evidence (text data) involves the following: (a) eliciting verbal responses (R_i) to a specific focus question, (b) identifying response codes (C_j), (c) creating thematic categories (families; F_k), and (d) converting these categories into thematic variables (V_m; see Figure 2).

Figure 2.

A flow chart of the process of thematic text analysis

Step 1: The Focus Question and Eliciting Responses

A first aim in the content analysis of open-ended text narratives is to identify relevant responses (and their response codes) that answer a specific focus question. This methodology, as we have developed it, is a variation of a content analysis approach—an open-ended “topic category” interview that was developed by Flannigan, McGrath, Meyer, and Garcia (1995). Also, an interview protocol that is similar to our focus question interview protocol (our Platica) is the Adult Interview Guide used in the International Resilience Project (Ungar, 2010). From our prior research, we found that the identification of relevant responses (response codes) is facilitated by framing a focus question narrowly, sometimes in the form of a sentence completion, for example, “What does being resilient mean to you?; being resilient means: _______.” Furthermore, regarding the study of a personal attribute such as machismo, we found that two focus questions are needed: (a) asking about the general concept of machismo (machismo beliefs) and (b) asking about the respondent's own identification with these beliefs (machismo self-identification).

Specifically, our Platica interview protocol consists of a series of specific focus questions. For example, to the focus question, “Please tell me what a real `macho' man is like,” one response was that being macho is, “Someone who probably does not respect the opinions of females … but that to me it is almost a stereotype,” with the relevant response code being, “[ID536] does not respect the opinions of females.” Here, the response, “… but that to me it is almost a stereotype,” is solely a comment, and this would not be coded as a relevant response.

As collected via independent in-depth audio-recorded interviews, each participant serves as a “case,” and the “case” (not the response codes) serves as the “unit of analysis.” In response to a given focus question, each participant or case will contribute zero, one, or more verbal responses (R_i), which are then used to create response codes (C_j). As we have developed this methodology, response codes that have functionally equivalent meaning are combined into a thematic category (F_k; see Figure 2). This procedure of identifying and labeling thematic categories is similar to open coding as described by Strauss and Corbin (1990), a procedure that includes discovering categories and category naming (Strauss & Corbin, 1990).

As an example, from our IMM studies of resilience among Hispanic leaders, community residents, and users of illegal drugs, within our audio-recorded qualitative Platica protocol, we examined machismo beliefs and machismo self-identification. First, we presented an introductory statement, “Within the Latino/Hispanic cultures and in other parts of the world, men are often described as being `manly' or `macho' and women are often described as being `feminine' or `motherly.'” Subsequently, the first focus question was, “Please tell me what a real `macho' man is like (their traits or characteristics): _________.” An example of a pattern of responses and subsequent interviewer probes is the following:

Participant: “Machismo is the foundation of family, [a macho] is the heart of the family.”

Interviewer probe: “When you say, `heart of the family,' what does that mean?”

Participant: “He provides for the family, you know, [he is] the provider and protector. [He takes] care of the family.”

Step 2: Identifying Response Codes

In vivo coding

Response codes encode relevant answers to the focus question, and using Atlas. ti we have identified these by highlighting a relevant quote that answers the focus question. Within the Atlas.ti text analysis window, we also tag each response code at the beginning with the participant's case ID number to link each response code to other quantitative data gathered in the structural interview, such as demographic variables and also outcome measures, for example, a Life Satisfaction Scale. Within Atlas.ti, our preferred coding modality is “In vivo coding,”³ which allows interactive identification and labeling (Strauss & Corbin, 1990) in creating response codes and then in creating thematic categories.

Step 3: Creating Thematic Categories

Next, the aim of creating thematic categories consists of assigning several response codes (C_j) that have functionally equivalent meaning to a higher order (superordinate) thematic category or family (F_k).⁴ Family is the term used within Atlas.ti to refer to these thematic categories. Identifying thematic categories is similar to the process of “discovering categories” and naming them as described within Grounded Theory (Strauss & Corbin, 1990). Within the IMM approach, a response code can be assigned to one or more thematic categories.

Approach for creating thematic categories

Based on our prior research, a heuristic goal in creating thematic categories is to “create the smallest number of `strong' thematic categories,” where strong categories contain at least 20% of the total number of response codes, thus accounting for a remarkable percentage of the explanatory variance.⁵ A major challenge in thematic category construction involves creating thematic categories that will replicate across independent coders, thus obtaining high intercoder concordances or reliabilities. This concordance is important, since in practice a series of thematic category solutions can be generated from the same set of (C_j) response codes. The principal aim in creating thematic categories is to attain an optimal solution, which is one (a) that consists of the same or similar thematic categories identified across independent coders and (b) that captures all relevant themes as expressed by the participants. The latter aim relates to content validity, “the extent to which a specific set of items [or thematic categories] fully reflects a content domain” (DeVellis, 1991, p. 43).

For example, regarding machismo beliefs, we asked, “What is a real macho like,” and Coder 1 identified 10 thematic categories, with their long descriptive labels being the following: (a) “non-womanizer, respectful to women, equal opportunity”; (b) “physical control—abusive to women and others”; (c) “emotional control—arrogant and bossy to women and others”; (d) a “man who drinks”; (e) “protects, supports, and provides for the family”; (f) “emotionally unavailable”; (g) “being responsible”; (h) “emotionally available”; (i) “emotional—showing honor”; and (j) “physical representation of strength, outer appearance.” Coder 2 identified eight thematic categories: (a) “avoids being negative—self-control”; (b) “expresses emotions”; (c) “high-ego, superiority, controlling, and arrogant”; (d) “disrespectful to women, womanizer, verbally and physically abusive, drinks”; (e) “family oriented—caring, respectful, provider, and protective”; (f) “honorable—related to cultural manliness”; (g) “non-womanizer, shows respect to women, believes in equal rights”; and (h) “physical attributes—strong and tough.”

Matching thematic categories produced by the independent raters

As we have developed this methodology, in a concordance analysis, we examine both independent coder solutions to reconcile them into an “optimal solution,” as defined above. During “Round Table 1,” a “thematic category round table review,” this optimal solution is attained using a constant comparison of the independently derived thematic categories and the response codes which they contain.

Table 1.

Optimal Solution Table for Machismo Beliefs and for Machismo Self-Identification

Rank	Thematic Variables: Machismo Beliefs	Percent Cases Mentioning	Rank	Thematic Variables: Machismo Self-Identification	Percent Cases Mentioning
1	Honor, respect, and self-control	63.7	1	Denies negative traits	51.9
2	Family oriented	56.8	2	Identifies as a man, endorses the male role	48.1
3	Abusive, disrespectful toward women	55.1	2	Respectful and egalitarian toward women	48.1
4	Respectful and egalitarian toward women	29.3	4	Controlling and domineering	46.2
5	Physically controlling and abusive	27.5	5	Family oriented	38.5
6	Emotionally controlling and domineering	24.1	6	Tries to control negative traits	36.5
7	Culturally constructed stereotypes	24.1	7	Situational aggression, aggressiveness, and control	34.6
6	Other physical traits	15.5	8	Both positive and negative traits	28.8
9	Emotional suppression	13.7	8	Developmental: Once was negative and now positive	28.8
10	Emotional availability	13.7	10	Denies positive traits	19.2
11	Alcohol based	12.0

Note: These results are based on 58 cases for machismo beliefs and 52 of these same cases for machismo self-identification.

In the present example, the replicated thematic categories formed across independent coders were (a) “non-womanizer, respectful to women, equal opportunity” (from Coder 1) and (b) “nonwomanizer, shows respect to women, believes in equal rights” (from Coder 2), where these matching thematic categories clearly contributed to the optimal solution. These two thematic categories were almost identical, and we relabeled these as the common category, “respectful and egalitarian toward women” (see Table 1). Further inspection of the thematic categories across both coders revealed an approximate matching of response codes for the following categories: “protects, supports, and provides for the family” from Coder 1, and “family oriented—caring, respectful, provider, and protective” from Coder 2. We then relabeled that common thematic category as “family oriented” (see Table 1). Under this concordance analysis, this reconciling process yielded six thematic categories that had sufficient interrater agreement to contribute common thematic categories to the optimal solution (see Table 1).

For two other categories there remained unmatched codes that did not explicitly mention “women.” Coder 1 contributed the categories of (a) “physical control—abusive to … others,” (b) “emotional control—abusive to … others,” (c) “emotional—showing honor,” and (d) “being responsible.” And Coder 2 contributed to (a) “avoids being negative—self-control”; (b) high-ego, superiority, controlling, and arrogant”; and (c) “honorable—related to cultural manliness.” An inspection of these categories and their response codes (looking beyond the labels) revealed that the two categories from Coder 1, “emotional—showing honor” and “being responsible,” corresponded well with two categories from Coder 2, “avoids being negative—self-control” and “honorable—related to cultural manliness.” Accordingly, the optimal solution involved creating the common thematic category of “honor, respect, and self-control” (see Table 1).

Moreover, the review team agreed that extraction of certain culturally framed aspects/codes of machismo were important and should be developed into a category. Thus, in the optimal solution, this category was labeled culturally constructed stereotypes (as these related to the Mexican or American cultures). A similar rationale was applied to the remaining categories of physical control—abusive to … others and emotional controlߞabusive to … others,” as contributed by Coder 1, and their match to the broader, “high-ego, superiority, controlling, and arrogant” category contributed by Coder 2. This led to the creation of the thematic categories of “physically controlling and abusive” and “emotionally controlling and domineering” (see Table 1).

In summary, this concordance analysis used initial and revised solutions to generate an “optimal solution,” while also working to create “strong thematic categories.” In the present illustration, merging two smaller thematic categories into larger and broader thematic categories, “physically controlling and abusive” and “emotionally controlling and dominating,” contributed toward satisfying this heuristic criterion of creating “strong thematic categories.”⁶

Issues involving “weak thematic categories.”

As noted, thematic categories that contain less than 20% of the total number of response codes are regarded as “weak” because they contain few response codes and thus few cases that mention the noted theme, that is, a high percentage of null (0 = no mention) codes. In the extreme case, a thematic category consisting of one or zero response codes is by definition an “empty thematic category.” From our prior research, “weak thematic categories” later produce “skewed thematic variables,” which are problematic for quantitative data analyses. As an important note, a low-response weak thematic category can be strategically retained if the investigator believes that it is of interest to identify and later examine certain cases that are unique or different, to give voice to these cases in later analyses.

Step 4: Dimensionalization or Scale Coding

Rationale for dimensionalization

Dimensionalization via scale coding⁷ adds the dimension of frequency of response or intensity of emphasis to a thematic category. Recall that for the construct of machismo, the first focus question asked, “Please tell me what a real `macho' man is like (their traits or characteristics: _________).” From the response codes, one inductively produced thematic category was “aggressive/controlling.” Some representative response codes that were used to identify this thematic category were “likes to get into fights,” “his word is the law,” and “he bosses everyone around and makes [others] do what he wants.” As we have developed this procedure, in IMM, scale coding allows the encoding of nuances or “shades of emphasis,” an approach described as dimensionalization by Strauss and Corbin (1990). Scale coding also converts a dichotomous category (0 = no mention, 1= mention) into an ordinal or interval-level variable, that is, 0 = no mention to 3 = highest mention.

The task of scale coding

For each thematic category, a team of two or more independent coders conducts scale coding. Given that the thematic analysis of a single focus question typically generates 3 to 12 thematic categories, each member of a two- or three-person team of coders independently rates all response codes within each thematic category. In this process, we have used a coordinating supervisor who oversees the work of the two or three independent coders. The supervisor also participates in “Round Table 2,” a “scale coding round table review,” in which team members compare their independent response code ratings for matches (concordances) and nonmatches (discordances) across coders. Via a constant comparison review and discussion, the goal is to agree on the most accurate scale code ratings that capture with fidelity the tenor of participant's responses.

Table 2.

Rotated Component Matrix for the Machismo Self-Identification Thematic Categories (Families; N = 52)

Thematic Variable Items	Factor I: Control and Dominance (Negative Machismo)	Factor II: Caballerismo and Family Oriented (Positive Machismo)
Controlling and domineering	.794	−.248
Both positive and negative traits	.737	.060
Tries to control negative traits	.684	.063
Situational aggression, assertiveness, and control	.643	−.054
Developmental: Once was negative and now positive	.482	− .374
Denies negative traits	− .441	.256
Respectful and egalitarian toward women	−.049	.750
Denies positive traits	.190	− .650
Identifies as a man, endorses the male role	−.244	.391
Family oriented	.072	.373

Note: Numbers in boldface indicate items loading above .30 on the noted factor.

From our prior research, we have identified two ways to conduct scale coding: (a) frequency scale coding and (b) intensity scale coding. For intensity scale coding, a 4-point scale coding scale has worked best (Denne et al., 2001). The levels of “intensity scale coding” are the following: (0) = no mention—the theme is not expressed in any way; (1) = suggested mention—suggestive of the theme; (2) = basic mention—a clear mention of that theme; and (3) = emphatic mention—a mention with strong emphasis or great intensity.

Frequency scale coding

Frequency scale coding is the more rudimentary of these two forms of scale coding, although it also yields higher levels of interrater agreement. For a given case, frequency scale coding simply involves counting the number of mentions of a given theme, that is, the number of response codes appearing within a given case ID number.⁸ For example, under the thematic category of “physically controlling and abusive,” and for case [ID133], the thematic statement about “controlling” responses appeared three times: (a) “He rules, his home as his castle”; (b) “[He] runs his house like a king, or worse like a slave driver”; and (c) “[His] wife and kids are there to serve him.” These three functionally equivalent responses involving dominant control were assigned to the “physically controlling and abusive” thematic category as mentioned by case [ID133]. Thus, for participant/case [ID133], the frequency-of-mention scale code value for the “physically controlling and abusive” thematic category is a value of “3.”

Frequency scale codes are tabulated in a Case-Theme Scale Coding Matrix for ease of inspection. This matrix simply lists all cases in rows and all thematic categories (families) in columns. For thematic categories, this matrix consists of vectors of 0s and 1s, representing “not mentioned,” and “mentioned,” respectively. However, after frequency-of-mention scale coding (or intensity scale coding), this matrix consists of discrete numerical values that represent the scale code values of 0, 1, 2, or 3, for the number of mentions contributed by each case and as listed within a given thematic category.

Intensity scale coding

In contrast, for these same narrative statements from case ID133, under intensity scale coding we can code the second statement, “runs his house like a king, or worse like a slave driver,” as 3 = emphatic mention, because of the adjective “worse” and the addition of “slave driver,” which are regarded as more cruel than just being “a ruler or king.”⁹ Finally, the first and third statements, “He rules, his home as his castle” and “Wife and kids are there to serve him,” are instances of 2 = a solid mention, as these depict a medium level of emphasis.

As contrasted with frequency scale coding, for intensity scale coding we have three scale code values for case ID133. However, given the convention that, “the case is the unit of analysis,” each case should contribute only one scale code value to a given thematic category, so what to do? Under intensity scale coding, we have established the heuristic rule that “regardless of the number of codes that emerge, assign the highest intensity-level code from among all values” and that value will constitute the optimal intensity scale code value that is contributed by a particular case. In this case, with observed scale code levels of 3, 2, and 2; clearly the single scale code value contributed by case ID133 for the “physically controlling and abusive” thematic category would be a “3.”¹⁰

Creating exemplar anchor codes

In preparation for conducting intensity scale coding, we have found that for a given thematic category it is necessary to examine the response codes generated for all cases and then to identify a set of exemplar anchor codes that best describe each of the three levels of intensity scale coding. Within a given thematic category, these exemplar anchor codes provide category-specific benchmarks to help the coders rate the response codes across all cases. For example, for the “physically controlling and abusive” family, one would identify the following exemplar anchor codes to define the three levels of intensity scale coding: (3) = “He is very aggressive and intimidates the entire family”; (2) = “He is aggressive with his family”; and (1) = “He tries to tell his family what to do.”

As noted, using a Case-Theme Scale Coding Matrix aids in summarizing the respective machismo code values resident within each machismo thematic category. Under this process, the team supervisor compares and summarizes the newly created thematic variable scale values generated independently by the two independent raters as encoded onto their Independent Solution Table (not shown). Then in the “Round Table 2,” the “scale coding round table review,” the coordinating supervisor and the independent raters compare and discuss these independently generated scale codes to research consensus in generating an Optimal Solution Table (see Table 1).

Intercoder reliabilities

Within each thematic category, for initial frequency or intensity ratings, one can calculate an interrater reliability coefficient or a Cohen's kappa coefficient that assesses intercoder agreement beyond chance (Fleiss, 1981), as an index of initial levels of intercoder agreements. However, the final scale code ratings move beyond these preliminary ratings based on the results of the round table discussions to generate an optimal solution. In summary, scale coding dimensionalizes a thematic category, converting it from a thematic category, which has the codes, 1 = mentioned and 0 = not mentioned, into a thematic variable (which has code values of 0, 1, 2, or 3).

Once dimensionalized, each newly created thematic variable has distributional properties similar to those of a measured variable, which is defined by scores describable as Likert-type scaling.¹¹ When dimensionalized, and if treating coded values as a Likert-type scale, a thematic variable can then be used as a conventional measured variable and incorporated into conventional correlation, regression, or other multivariate data analyses. A thematic variable may also be used as a moderator variable, one that encodes conditional effects. A moderator variable is “a variable that modifies the form or strength of the relation between an independent and a dependent variable” (MacKinnon, 2008, p. 275). A moderator variable that is derived from qualitative text analyses may operate as a “discovered” conditional effect, one that was not previously anticipated during the design stages of a given research study (Yoshikawa et al., 2008) but one that as a discovered variable can aid in describing new and important conditional and interactive effects.

Step 5: Data Analytic Approaches

Overview of data analytic approaches

Descriptive and correlation analyses may now be conducted to examine associations among the qualitatively constructed thematic and the quantitatively based measured variables (Castro & Coe, 2007). The newly constructed thematic variables as well as the measured variables (scales and individual items) can both be used as predictor variables of any outcome variable of interest, for example, of a Life Satisfaction Scale. Within a hierarchical regression analysis, the predictive effects of the inductively derived thematic variables can also be examined (a) as a unified block consisting of a set of thematic variable predictors along with a set of measured variable predictors or (b) as thematic variable predictors of an effect above and beyond (in sequentially introduced blocks) the effects of a previously entered block of measured variable predictors (Cohen, Cohen, West, & Aiken, 2003). In this latter case, the inductively generated “discovered” information encoded by thematic variables can introduce additional explanatory variance that otherwise would have remained undetected if solely incorporating the measured variables into the regression model.

Types of data analyses

Preliminary data analyses can include descriptive frequency analyses to examine the distributional properties of the thematic variables. Thematic variables can first be examined for remarkable skew (values of 2.0 or greater) and kurtosis. Ideally, all thematic variables, especially those developed as “strong thematic variables,” will exhibit distributional properties that are devoid of excessive skew.

Subsequently, correlational analyses allow the examination of a matrix that examines the strength of association among all thematic variables. Other correlation matrices can be generated that examine associations between a set of thematic variables, as correlated with a set of quantitative measured variables (see Castro & Coe, 2007, Table 5). Similarly, one can also examine predicted or hypothesized associations using a multitrait–multimethod matrix (Campbell & Fiske, 1959), thus conducting statistical triangulation, to examine the convergent associations (convergent validity) among the thematic and measured variables, as related to one or more core constructs, for example, positive machismo or negative machismo. For example, from our prior research, in a sample of 58 males, we observed that the measurement scale of Responsible Family Protector Attitudes (positive machismo; α = .85) (Rollins, 2003), was positively correlated with the macho self-identification “situational aggression, assertive control” thematic variable (r = .28, p < .05), suggesting that positive macho attitudes are associated with a greater endorsement of assertive or aggressive actions in situations where urgent action is needed (Kellison, 2009). Also, the measurement scale of Aggressive and Self-Centered Attitudes (negative machismo; α = .82) (Rollins, 2003) was negatively correlated with the macho self-identification “denies negative traits” thematic variable (r = −.35, p < .01), suggesting that having high levels of negative macho attitudes is associated with low denial of negative macho traits, that is, admitting to these traits (Kellison, 2009).

Exploratory factor analyses

As examined in our prior studies, one can conduct an exploratory factor analysis with a set of thematic variables that measure a factorially complex construct such as machismo to examine its factor structure. Subsequently, one can then use results from this factor analysis to compute factor scores that can then be used as predictor variables within a hierarchical regression analysis of an outcome variable of interest, for example, Life Satisfaction Scale scores (Kellison, 2009).

For example, we created factor scores for machismo self-identification, as generated from relevant thematic variables (see Table 1), which were entered into a principal components analysis with oblimin rotation (Kellison, 2009). In contemporary Latino research, machismo has been conceptualized as a complex construct defined by two principal components: negative machismo and positive machismo (Arciniega et al., 2008). In our exploratory factor analysis, we also hypothesized and attained a two-factor solution. Table 2 presents the results of this principal components analysis (Kellison, 2009). A scree plot analysis revealed the viability of a two-factor solution, and as expected, these thematic variable factor loadings aptly identified two principal components: (a) negative machismo, which we labeled “control and dominance,” and (b) positive machismo, which we labeled “caballerismo and family oriented.”

The results of this exploratory factor analysis provided initial confirmatory evidence in support of the content validity of the constructed machismo thematic variables, as these thematic variables aptly captured the expected two-factor structure for this construct of machismo self-identification. Subsequently, these machismo factor scores were used as predictor variables in hypothesis-driven multiple regression model analyses, in which the conventional measured (scaled) variables were entered blockwise in the regression model Step 1, with the thematic variable factor scores entered blockwise in Step 2 (as an example, see results from a prior study, Castro & Coe, 2007, Table 7). Thus, in these integrative data analyses, both data forms were used as predictors of a dependent variable of interest, that is, life satisfaction.

Step 6: Coming Full Circle: Creating “Story Lines” and Recontextualization

A recontextualization of the data

In qualitative data interpretation, contextualization is used to “give a meaning of the obtained results with reference to the specific and particular context of the study” (Gelo et al., 2008, p. 277). Furthermore, recontextualization has been described as the real power of qualitative research, as it involves “the development of emerging theory so that the theory is applicable to other settings and to other populations to whom the research is applied” (Morse, 1994, p. 34). Within IMM, recontextualization involves a return to the original context in which the observations were made by relating statistically derived outcomes back to select indicated quotes to generate stories that “give voice” to the very people who stated them. Examining selected text narratives identified by the results of a regression model analysis allows the creation of Story Lines that can contribute to a deep-structure analysis that moves “beyond description to conceptualization” (Strauss & Corbin, 1990, p. 120). The IMM Story Line analysis is similar to the Grounded Theory Story Line analysis, which is used to generate “a descriptive story about the central phenomenon of the study” (Strauss & Corbin, 1990, p. 119).

Contrasting story lines by levels of life satisfaction

Table 3 presents the macho self-identification responses for a set of contrasting groups analysis. Narrative responses are presented in a stratified analysis for five cases having the highest Life Satisfaction Scale scores as contrasted with the five lowest scoring cases (Kellison, 2009). This is a form of purposive sampling that examines quotes involving machismo self-identification, based on the finding that this variable was significantly associated with the outcome variable of Life Satisfaction. This then allows us to “learn as much as possible about the outliers” (Gelo et al., 2008, p. 275). In this particular contrasting groups analysis, Story Line 1 for members of the highest-scoring strata of cases on Life Satisfaction voices positive machismo self-identification themes that involve caballerismo (chivalry; Arciniega et al., 2008) and responsibility to family: “For me it's acting like a gentleman” and “I do my best to take care of my family” (see Table 3). In contrast, Story Line 2 from the lowest-scoring strata of cases voices negative machismo themes that involve selfishness, irresponsibility, and antisocial conduct: “I don't identify with working hard or taking care of my family”; “I'm lazy, I'm selfish, I have a short fuse”; and “I have low self-esteem.” These contrasting Story Lines reveal the presence of high life satisfaction among family-oriented responsible males, as contrasted with low life satisfaction among males who lack family involvement and who are irresponsible.

Table 3.

Contrasting Groups Story Line Statements for the Five Highest and Lowest Cases on Life Satisfaction

Case Number	Life Satisfaction Score	Quoted Statement About Machismo Self-Identification	Story Lines
Highest on Life Satisfaction
ID133	2.17	“I care about my family”; “For me it's acting like a gentleman”	Story Line 1: Men who value and engage in family caretaking exhibit high levels of caballerismo (positive machismo) in their male gender role identity, are giving and responsible, and they also experience high levels of life satisfaction
ID147	1.57	“I'm respectful of women”; “I never bring shame to the family”
ID164	1.50	“I do my best to take care of my family”
ID343	1.48	“I treat women with respect and don't beat them”
ID371	1.42	“I bring home money and make sure there is food on the table”
Lowest on Life Satisfaction
ID160	−1.15	“I have my flaws, I'm selfish”; “I hold a grudge forever”; “I'm not afraid to cry in front of others even strangers”	Story Line 2: Men who do not value or engage in family caretaking exhibit low levels of caballerismo (positive machismo) in their male gender role identity, are selfish and irresponsible, and they also experience low levels of life satisfaction
ID162	−1.21	“I don't identify with working hard or taking care of my family”
ID149	−1.67	“I never had aspirations to have any children or family responsibility”
ID399	−2.58	“In prison I acted in ways I didn't want to, and even today I still do”
ID370	−2.63	“I'm lazy, I'm selfish, I have a short fuse”; “I have low self-esteem.”

Status and Areas for Refinement

Some Challenges and Limitations

Adequate data gathering

Despite the stated advantages offered by the IMM approach, several challenges exist. One challenge involves the need for effective interview data collection that requires adequate probing after an initial focus question response. Insufficient probing will produce limited verbal responses and subsequently will yield shallow and uninformative thematic categories.¹² A focus question should include one or more probes, for example, “please tell me more about that,” which will generate a more complete response from which to construct sound thematic categories.

Skewness in thematic variables

Thematic categories and their derived thematic variables that exhibit remarkable skew, that is, a skewness value of 2.0 of higher, can be regarded as “weak thematic variables,” when skew is driven by many zero values that indicate a large proportion of null responses involving “no-mentions.” From the quantitative perspective, weak thematic variables will likely violate basic assumptions of normality necessary to test linear regression models (Cohen et al., 2003), and often these variables yield nonsignificant results in correlational and multiple regression analyses.

Cross-sample stability, validity, and replicability of thematic variables

Whereas some measured variables in the form of established scales have been tested and validated in prior studies, thus establishing their psychometric properties including reliability (internal consistency) and validity (construct, concurrent, predictive), inductively constructed thematic variables do not have this history. Accordingly, issues may be raised regarding various forms of validity involving these single-item thematic variables and for which reliability (scale internal consistency) cannot be ascertained. Such thematic variables can exhibit face validity or perhaps instrumental validity (Kirk & Miller, 1986) when it is shown that “observations match those generated by an alternative procedure that is itself accepted as valid” (Kirk & Miller, 1986, p. 22). In general, further analyses are typically needed to ascertain the overall validity of newly constructed thematic variables and whether these thematic variables would be replicated in a second sample that is drawn from the same subcultural group or population. Investigators should examine evidence, for example, via statistical triangulation, that substantiates the identity of their newly developed thematic variables, such as by using a multitrait–multimethod matrix and also via exploratory factor analyses, to support or refute (Dreher, 1994) the identity of their newly constructed thematic variables and the meaning that they convey. Future IMM research can provide additional evidence regarding the properties involving the stability, validity, and utility of these inductively generated thematic variables.