Using Qualitative Comparative Analysis (QCA) of Key Informant Interviews in Health Services Research: Enhancing a Study of Adjuvant Therapy Use in Breast Cancer Care

Abstract

Background

Qualitative Comparative Analysis (QCA) is a methodology created to address causal complexity in social sciences research by preserving the objectivity of quantitative data analysis without losing detail inherent in qualitative research. However, its use in health services research (HSR) is limited, and questions remain about its application in this context.

Objective

To explore the strengths and weaknesses of using QCA for HSR.

Research Design

Using data from semi-structured interviews conducted as part of a multiple case study about adjuvant treatment underuse among underserved breast cancer patients, findings were compared using qualitative approaches with and without QCA to identify strengths, challenges, and opportunities presented by QCA.

Subjects

Ninety administrative and clinical key informants interviewed across ten NYC area safety net hospitals.

Measures

Transcribed interviews were coded by three investigators using an iterative and interactive approach. Codes were calibrated for QCA, as well as examined using qualitative analysis without QCA.

Results

Relative to traditional qualitative analysis, QCA strengths include: (1) addressing causal complexity, (2) results presentation as pathways as opposed to a list, (3) identification of necessary conditions, (4) the option of fuzzy-set calibrations, and (5) QCA-specific parameters of fit that allow researchers to compare outcome pathways. Weaknesses include: (1) few guidelines and examples exist for calibrating interview data, (2) not designed to create predictive models, and (3) unidirectionality.

Conclusions

Through its presentation of results as pathways, QCA can highlight factors most important for production of an outcome. This strength can yield unique benefits for HSR not available through other methods.

INTRODUCTION

Innovative research designs, often borrowed from other disciplines, can help surmount challenges to conducting health services research (HSR) studies (1). Healthcare providers exhibit great variation in organizational structures, care processes, and outcomes (2), necessitating a methodological approach flexible enough to capture high levels of detail, yet rigid enough for validity, reliability, and generalizability. Furthermore, HSR approaches may need to be adapted to accommodate smaller sample sizes, as gathering site-specific data can require significant time and limit the number of sites that can feasibly be studied. While quantitative approaches offer many benefits, they may be inappropriate to deal with small sample sizes, and may not take into account the different causal pathways that can produce an outcome. Alternatively, traditional qualitative approaches may not provide the level of objectivity and reproducibility desired to understand the relationship between site characteristics and low-quality care in complex delivery systems.

Qualitative Comparative Analysis (QCA) provides an approach that captures the objectivity of quantitative designs, yet can take advantage of the detail from qualitative interviews (3-5). QCA is considered ideal for small- to medium-N case-oriented research studies, thus making it a good fit for many investigations. Originally developed as a research tool for social science, QCA has been used sparingly in HSR (6,7), typically to uncover patterns to achieve a given outcome in large data sets (8). Examples of QCA based on key informant interviews, particularly in HSR, are even more rare.

This paper evaluates the use of QCA in HSR using key informant interview data, and compares this approach to more traditional qualitative methods. This analysis focuses on the strengths and weaknesses of QCA methodology, using a case to help investigators evaluate the applicability of QCA within the HSR context.

METHODS

Research Design

Interview data were collected as part of a larger intervention study designed to reduce underuse of adjuvant therapies for breast cancer patients by closing referral loops between surgeons and oncologists in safety net hospitals (9). The aim of the ongoing study is to uncover factors associated with high-quality (i.e., low underuse) cancer care in resource-limited safety net hospitals. Ten hospitals serving high proportions of minority breast cancer patients in the New York metropolitan area were recruited to participate in the intervention project.

Data Collection

Prior to intervention implementation, principal investigators at each hospital site identified key informants who could discuss how cancer care and quality improvement occur at their institutions. Key informants included hospital leaders in both cancer care and quality, oncologists, nurses and nurse managers, clerks, and patient navigators. In all, 90 in-depth, semi-structured interviews were conducted with: 59 clinicians; 16 administrators; 12 clerical staff; and 3 others.

Interviewees were asked about the structure and processes of breast cancer care delivery and perceived facilitators of and barriers to patients obtaining their prescribed adjuvant treatments. Interviews, lasting 30-60 minutes, were recorded and transcribed verbatim.

Data Coding

An integrated approach was used to code interview data that included both deductive and inductive methods (10). The three-member coding team, consisting of senior investigators (authors) and an experienced research assistant (author), held regular meetings to discuss decisions about codes and themes as they emerged.

A preliminary coding dictionary was developed based on broad topics from the key informant interview guide. This coding dictionary defined categories a priori based on the question domains, such as “Current Process for Coordinating Breast Cancer Care,” or “Barriers to Handoffs and Tracking,” or “Readiness for Innovation Implementation.” Each coder applied this preliminary structure to three common interview transcripts, then met to reach consensus about codes and definitions.

Each coder subsequently coded additional interviews independently, using the constant comparative method of qualitative data analysis to develop and refine codes in vivo (11,12). Frequent meetings helped to ensure consistency and accuracy of coding, as well as clarification about emergent codes. This approach followed the standards of code development for rigorous qualitative analysis (10, 13). ATLAS.ti 7.0 software was used to facilitate the coding process (14).

Qualitative Comparative Analysis

QCA involves listing and counting different combinations of variables in a given data set, and then applies logical inference rules (i.e., Boolean algebra) to evaluate whether alternative inferences are supported by the data (15). This approach identifies one or more solution pathways that produce an outcome of interest (15), thereby providing information about alternative approaches to achieve a given outcome. The QCA methodology contains five steps: 1) identify an outcome of interest and a list of conditions that may be associated with that outcome; 2) develop calibration metrics using a crisp, dichotomous or a fuzzy, ordinal set; 3) calibrate the data; 4) develop a truth table showing all possible combinations of conditions, identify necessary conditions, and use logic to minimize the truth table and arrive at pathways to the outcome; and, 5) assess these pathways using two parameters of fit called consistency and coverage (16). A summary of these QCA-specific terms and accompanying definitions are provided in Table 1.

Table 1.

Terms and Definitions of Qualitative Comparative Analysis (QCA)

Term	Definition
Case	A site under study.
Outcome	The result of one or more conditions.
Condition	A factor that is associated with an outcome of interest.
Crisp set	A type of QCA in which conditions are either “present” or “not present” for each case; conditions are coded as 1 or 0, respectively, for each case.
Fuzzy set	A type of QCA in which cases are assigned a value between 0 and 1, based on their degree of membership in the condition. Cases with membership over .5 are considered to be more in than out of a condition, and cases with membership below .5 are considered to be more out than in.
Necessity	A condition is necessary if the condition is present every time the outcome is present.
Sufficiency	A condition (or combination of conditions) is considered sufficient if the outcome is present each time the condition (or combination of conditions) is present.
Calibration	A number that is assigned to a condition to represent a case's degree of membership in that condition.
Calibration structure	A framework devised by researchers that breaks each condition down into tangible measures, and breaks each measure down into fuzzy set values. Clear definitions are created for each fuzzy set value.
Solution pathway	A combination of two or more conditions that is associated with an outcome.
Consistency	A measure of the degree to which cases that follow the same pathway also share the same outcome.
Coverage	A measure of the frequency of a given pathway to an outcome.

In this case example, underuse of adjuvant treatments is the outcome of interest. Similar codes and themes were grouped into conditions that could be associated with the outcome. For example, the theme of “no shows” was combined with the emergent theme of “tracking across specialties” to specify “follow-up” as a condition.

Fuzzy set calibration structure was then created by assigning a value between 0 and 1, based on degree of membership in the condition. To continue the example above, the “follow-up” condition was calibrated through a multi-step process that involved defining measures for the condition, creating anchor points (1, .5, 0) for each measure, and then calibrating the fuzzy set (1, .8, .6, .4, .2, 0). The first measure defined for this condition focused on whether “no shows” received phone calls and letters. An anchor point value of 1 was assigned to those sites that always made phone calls and/or sent letters; .5 was assigned if the site “sometimes” did this; and 0 was assigned to sites where this was never done. Once these anchors were established, more specific scores were assigned.

Finally, relative weights were assigned to each measure within a condition according to their importance to the outcome. For example, because follow-up is important for adjuvant treatment rates, the basic approach of phone calls and letters was not as important as additional steps some hospitals took to reaching patients (e.g., telegrams, urgent patient home visits, missed appointment notification). As a result, the “basic” no show follow-up was weighted lower (e.g., .3) than the “extra steps” measure (e.g., .7). The output from this calibration process was represented as the weighted levels shown in a data matrix, as described above.

Next, we developed a truth table showing all possible combinations of conditions, and assigned cases (hospitals) to truth table rows based on the presence/absence of identified conditions (e.g., follow-up). Using Boolean logic, the truth table was minimized to arrive at pathways to the outcome (e.g., (8)). Three solution pathways were found to achieve the outcome of low underuse of adjuvant therapies. Finally, the fifth step required assessing these pathways using the consistency and coverage parameters of fit (16, 19).

RESULTS

Strengths of Using QCA in Health Services Research

Five strengths of QCA, relative to a traditional qualitative approach, were identified: 1) the methodology addresses equifinality and causal complexity, 2) findings can be presented as a “pathway” as opposed to a list, 3) necessary conditions are helpful to characterize actionable results, 4) QCA offers the option of fuzzy-set calibrations, and 5) QCA-specific parameters of fit allow researchers to compare pathways to an outcome. These strengths are each discussed in more detail next.

1. The methodology addresses equifinality and causal complexity

Equifinality is defined as two or more pathways that lead to the same outcome. QCA allows for multiple pathways to be identified as part of a solution, which is especially useful when the cases under study have different strategies to achieve a desired outcome. As an example, the three solution pathways showed alternative approaches to achieve low rates of underuse for breast cancer adjuvant therapies, rather than only one best way. Additionally, QCA hinges on the principle of causal complexity--the idea that more than one condition can be combined to produce an outcome. This case study included as many as six different factors, reflecting the complexity inherent in the care coordination process. The value of a methodology that accommodates causal complexity is particularly clear in healthcare organizations where outcomes are often the result of specific combinations of environmental, procedural, and social conditions.

2. Findings can be presented as a “pathway” as opposed to a list

In a traditional thematic analysis of qualitative interview data, such as grounded theory, multiple factors that are associated with an outcome are typically identified. QCA allows researchers to take this analysis a step further because it enables them to a) identify relevant factors as well as b) determine the ways in which those factors combine to produce the outcome. The ability to identify pathways to an outcome as opposed to a list of factors that might impact an outcome helps to present more detailed recommendations for practice.

3. Necessary conditions are especially helpful for actionable results

A necessary condition is a condition that must be present for the outcome to occur. For a favorable outcome, making sure all necessary conditions are in place is imperative to achieve that outcome, yet often necessity varies. For example, while follow-up was consistently necessary, flexibility was necessary in the absence of system support, but unnecessary when system support was present. When necessary conditions are identified, implementing or removing these conditions is an important first practical step towards achieving or preventing an outcome.

4. QCA offers the option of fuzzy-set calibrations

A fuzzy set analytic approach enables researchers to classify degrees of a condition's presence for a given case and allows them to retain much of the detail present in in-depth qualitative interviews. In this study, degrees of follow-up were measured as “always,” “usually,” “occasionally,” “rarely,” and “never.” Fuzzy set calibrations thus enable researchers to assess the degree to which a condition is present at each site, and allow them to see how closely each case follows a given pathway to the outcome.

5. QCA-specific parameters of fit allow researchers to compare pathways to an outcome

The two QCA-specific parameters of fit are consistency and coverage. Consistency is a measure of the degree to which cases that share a combination of conditions (pathway) also share the same outcome, and coverage measures the prevalence of a given pathway. For this study, the level of consistency for the three solution pathways ranged from .82 to 1.0, consistent with acceptable levels in social science research, and providing confidence that the presence of these conditions in these hospital sites was consistent with low underuse of adjuvant therapies. Coverage, on the other hand, ranged from .19 to .49, reflecting variability in these different pathways, as well as the complexity of impacting the outcome of low underuse. When all identified pathways are equally grounded in theory, consistency and coverage can help researchers assess the accuracy and empirical relevance of each individual pathway. These parameters of fit can help researchers decide whether to recommend one pathway over another.

Weaknesses of Using QCA in Health Services Research

Three challenges of using QCA in a HSR study with interviews as the primary data source were identified: 1) very few guidelines and examples exist for calibrating interview data, and calibration content is likely to vary based on the outcome of interest, 2) QCA was not designed to create predictive models, and 3) QCA is unidirectional. These drawbacks are each discussed further below.

1. Very few guidelines and examples exist for calibrating interview data and calibration content is likely to vary based on the outcome of interest

Ragin and other QCA scholars have developed multiple methods for converting numerical data into crisp and fuzzy sets (16, 20), but they have not proposed any guidelines for calibrating interview data. Basurto and Speer (21) have published suggested guidelines, but they note that most QCA studies which might otherwise be used as examples do not disclose their calibration methods, and also that their guidelines are just one of many possible ways to calibrate interview data. Furthermore, these guidelines center around more general social science research, and are not specific to HSR projects. Thus, in keeping with the general principles of qualitative analytic rigor, designing a project-specific calibration structure takes careful attention and several revisions, and often requires an iterative process to ensure that final calibrations are justified. In this case example, the calibration of the “follow-up” condition involved multiple iterations, with members of the research team holding extensive discussions about how to characterize conditions (e.g., what themes were appropriate to include), how to define measures (e.g., “basic” follow-up vs. “extra”), and how to weight measures (e.g., what measures would have the most impact on the outcome). In the absence of information about the different HSR-related conditions in the literature, an iterative process is necessary to ensure that the final calibrations are not arbitrary.

2. QCA was not designed to create predictive models

QCA is very useful in identifying pathways to an outcome. However, the pathways developed by a qualitative comparative analysis are not designed to determine how outcomes would change if specific sites moved to a different calibration bracket. Although this is possible to some extent on a case-by-case basis, it would be very time consuming to make predictions on a large scale. This challenge is only a limitation in the methodology if predictions are necessary to develop convincing results.

3. Specificity of QCA is unidirectional

Finally, the pathways produced by QCA are only associated with the outcome in question; the opposite of a pathway is not necessarily associated with the opposite of the outcome. If researchers are interested in pathways both to an outcome and to its opposite, two separate analyses must be conducted, and different conditions may need to be used for each analysis (5). This second analysis follows the same procedure as the original analysis (identifying conditions, summarizing the data, creating calibration structures, reducing the truth table, etc.), and takes a comparable amount of time to conduct as the first analysis. As a result, investigating an outcome as well as its opposite is a very time-consuming analytical procedure.

Opportunities to Enhance Health Services Research Using QCA

Using QCA to further analyze interview data that had been coded thematically provided new insights that were not achievable with traditional qualitative methods. Some of the opportunities provided by QCA are highlighted in Table 2, and described in greater detail below.

Table 2.

Opportunities Presented to Enhance Health Services Research Through Qualitative Comparative Analysis (QCA)

	Qualitative Comparative Analysis (QCA)	Qualitative Analysis without QCA
Study Design
Steps required	Site visits, coding, identify themes, summarize themes, create and apply calibration structure, perform analysis.	Site visits, coding, identify themes.
Number of cases	About 10 (minimum)	1 or more
Results
Level of detail	Each condition is defined and ordered. A fuzzy set calibration can be used to express the level of presence of each condition at each site. Factors are represented numerically and can be quickly compared across sites.	A great level of detail may be expressed in prose, but may be cumbersome to compare across sites.
Type of results	Identifies different solution pathways associated with quality care. Raises possibility of testing solutions in different settings by providing detailed conditions on which organizations can focus.	Lists factors associated with low underuse with strength of association of proposed causal factors. Lacks condition details embedded within calibration structure and in associations with outcome. Lacking these details, organizations may focus on all factors equally.
Number of findings	One outcome per analysis; takes longer to identify combinations of factors associated with success or failure (e.g., low screening vs. low treatment rates).	Factors can be identified and grouped with different outcomes (e.g., facilitators of and barriers to screening/treatment completion) during the same analysis.
Presentation of findings	Focus on specific pathways hospitals use to achieve high quality; less focus on barriers to achieve low underuse.	General discussion of facilitators of and barriers to achieving low underuse.
Level of bias	Bias inherent in coding process and specification of conditions, but process of creating specific and objective measures of conditions forces greater level of proof/objectivity of findings.	Bias inherent in coding process

First, with respect to the level of detail that could be presented in the results, QCA permitted comparison across sites by focusing on the level that a condition was present at each site, represented numerically; the corresponding level of detail might have been cumbersome to express in prose comparisons of multiple sites. For instance, the theme of “no shows” involved how the hospitals tracked patients who did not show up for their appointments. The sites varied in their ability to address “no shows,” and describing this variability entails different examples from different sites. Using traditional qualitative approaches, this description includes addressing both similarities and differences across sites, and involves a considerable amount of textual description. In contrast, QCA enables the assignment of a level for the condition at each site, providing the ability to parsimoniously compare each hospital's level in a data matrix. For the condition that included “no shows,” levels ranged from .10 to .96, clearly reflecting considerable variability across sites. QCA also requires calibration, and therefore specifications of the key elements that contributed to a theme, including forced ranking of the included elements. This enabled the collapse of themes into overarching domains, allowing the emergence of different patterns of associations that had remained more diffuse in the absence of QCA.

Another important opportunity presented by QCA is the type of results that can be produced. QCA permitted the identification of three distinct solution pathways associated with low underuse and forced classification of necessary conditions for these solution pathways. The different solution pathways included different conditions, suggesting that there may be more than one way to reduce underuse of adjuvant therapies. For example, two hospitals had good follow-up, a patient-centered culture, shared information, and good system support. However, another solution pathway was shared by two different hospitals and involved good follow-up, a patient-centered culture, and shared information, but did not require good system support. This level of specificity enabled us to move beyond merely developing a list of factors associated with low underuse (e.g., follow-up, culture) but with no sense of strength of association among the proposed causal factors. Further, QCA enabled us to clarify how trade-offs might exist among the different factors (e.g., flexibility vs. system support).

A third opportunity presented by QCA is the generation of testable hypotheses. The development of the solution pathways provides an opportunity to test the veracity of these assumptions in different settings. In the context of quality improvement, determining conditions that can lead to high quality provides managers with a testable basis for practice change. For research, testable hypotheses generated from QCA provide the basis for larger scale and more generalizable studies. For example, shared information was a necessary part of both solution pathways described above. This finding could be used to support a hypothesis about the role of information systems in providing high-quality care. Finally, while bias is inherent in any coding process, the process of creating specific and objective measures and conditions in QCA forces a greater level of proof and objectivity of findings in the analysis process.

DISCUSSION

QCA has both value and limitations as a methodology for HSR that uses interviews as the primary data source. Its potential to present pathways to outcomes of interest in a causally complex, equifinal manner and its ability to identify actionable “necessary conditions” enable researchers to synthesize their results in a more detailed and pragmatic manner than is typically possible. However, limitations such as its inability to evaluate multiple outcomes or to create predictive models are clearly considerations.

An important limitation of QCA highlighted in this study is the paucity of examples available to help advise the process of calibration of qualitative data in HSR. Most recent studies in HSR that have used QCA (6-8) have used surveys (occasionally combined with interviews) to collect their data. When surveys are used as the QCA data source, it is much easier to match respondents’ answers with conditions that can be both identified and calibrated a priori. However, for studies involving interviews, the process typically allows for reflection on the part of the interviewee, and provides the interviewer with an opportunity to adjust or amend the interview questions. The tradeoff between structure and flexibility is especially evident when considering the amount of time required to use QCA in a study where interviews are the primary data source. Nonetheless, using surveys in a study such as this one would have made it extremely difficult to uncover emergent issues (22), and would have limited the ability to characterize some conditions in the solution pathways. Researchers therefore must be prepared to create calibration structures suited specifically to their studies, including spending sufficient time calibrating interview data.

While prior studies have compared QCA to conventional statistical methods in the context of quantitative data analysis (e.g., (18)), the use of QCA reported here suggests that there is an important opportunity to use QCA in HSR when interviews are the primary data source. While decision modeling can yield similarly actionable results (23), QCA is especially relevant to HSR because its presentation of results as a pathway to an outcome can allow healthcare organizations to make informed decisions about how to allocate their resources. The benefits that accompany presenting results in this manner are unique to QCA and should be considered by researchers as they select among methodological approaches.

Limitations

Similar to other qualitative methods, the time and energy required for in-depth analyses of key informant interview data in QCA is considerable and can create challenges for large studies. Although QCA was designed for studies with as few as 10 cases, studies with more cases typically make it easier to create and analyze the truth table – particularly in highly complex contexts, such as HSR. This study of 10 hospitals including 90 key informant interviews would be considered quite large; thus a lower limit of 10 cases implies a considerable amount of work. Further, as described above, QCA-specific limitations include the inability to be used for predictive modeling, as well as the potential problem of bias inherent in any coding process. Finally, while the specificity of the conditions enables greater objectivity, it is also possible that this specificity threatens generalizability to other conditions or scenarios.

Conclusions

QCA can afford researchers unique insights and actionable results, making QCA an appropriate and valuable methodology for certain HSR endeavors. It is particularly applicable to studies with clear, specific outcomes in question, and especially for those studies in which there is likely more than one way to reach the outcome and each way of reaching that outcome is likely to depend on multiple factors. In these instances, QCA can bring increased clarity and detail to the analysis of interview data, and provide specific guidance given evidence about how different factors impact an outcome.