CACE-MM: using mixed methods to strengthen causal inference in medicine and public health

Abstract

Background

Causal inference in medicine and public health almost always depends on untestable assumptions. Estimating valid causal effects thus often requires substantive knowledge about the study context that quantitative methods alone cannot provide. This paper introduces CACE-MM, a mixed methods framework that integrates qualitative approaches with complier average causal effect (CACE) estimation to strengthen causal decision-making and assess the plausibility of key underlying assumptions. CACE-MM is the first framework to systematically integrate qualitative inquiry with causal effect estimation in the presence of noncompliance.

Methods

We present a proof-of-concept application of CACE-MM using data from The Youth Empowerment Study (YES), a randomized trial of a trauma-informed intervention for youth involved in the juvenile legal system (n = 630). Following CACE analyses using an instrumental variable approach (invoking the exclusion restriction) and principal score approach (invoking the assumption of principal ignorability), we conducted 10 semi-structured interviews with key informants. Qualitative data were analyzed using inductive open coding followed by deductive mapping to the Capability, Opportunity, Motivation-Behavior (COM-B) model and the Theoretical Domains Framework (TDF). The study team assessed the plausibility of the key assumptions underlying each quantitative approach before and after the qualitative inquiry to generate integrated metainference about assumptions.

Results

Qualitative findings identified predictors of participation and outcomes that were not captured in baseline quantitative measures, raising concerns about the plausibility of principal ignorability. Interviews also clarified how meaningful exposure to intervention components was understood by implementers, informing the defensibility of participation thresholds used to invoke the exclusion restriction. More broadly, the findings demonstrate how qualitative inquiry can inform key analytic decisions that shape causal estimates, including how participation is defined, which covariates should be prioritized for measurement, and whether particular identification strategies are appropriate for specific outcomes.

Conclusions

Building on these insights, we propose the full CACE-MM framework, incorporating both exploratory and explanatory phases, and outline decision points to guide application in applied health research. CACE-MM offers a rigorous and systematic approach for integrating qualitative evidence into causal analyses and ultimately strengthening the transparency and interpretability of the CACE in applied health research.

Trial registration

ClinicalTrials.gov NCT03242447.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-026-02908-y.

Introduction

Estimating causal effects is central to decision-making in medicine and public health [1, 2]. Researchers, practitioners, and policymakers rely on causal evidence to determine whether interventions improve health, reduce harm, or should be implemented at scale [2]. In public health practice, causal reasoning informs choices about interventions and service delivery, while in policy contexts it underpins regulatory decisions and processes [3, 4]. In medicine, causal inference is embedded in the principles of evidence-based practice, where assessments of benefits and harms depend on the strength of determined causal relationships [2, 5].

A causal effect refers to the difference in outcomes that would be observed for the same unit (e.g., an individual, school, or community) under different, well-defined treatment conditions [6]. Because a unit cannot experience multiple treatment conditions at the same time, directly observing a causal effect for a single unit is impossible [6, 7]. This challenge is commonly referred to as the Fundamental Problem of Causal Inference [8]. As a result, outside of randomized trials with minimal missing data – conditions that are often difficult to achieve due to ethical, logistical, financial, and time constraints – estimating causal effects necessarily depends on untestable assumptions [7]. Although statistical tools such as sensitivity analyses can probe the robustness of estimates, the assumptions themselves cannot be assessed using observed data: causal assumptions are claims about the unobserved potential outcomes, whose plausibility depends on the specific study context [7]. In other words, credible causal conclusions require substantive and contextual knowledge that quantitative methods alone cannot provide [7, 9–11]. Yet the causal inference toolkit has largely underutilized the methods particularly well suited for generating this type of knowledge: qualitative approaches [9].

Mixed methods to enhance causal inference

Mixed methods, the deliberate integration of qualitative and quantitative approaches within a single study [12–14], offer a promising way to rigorously incorporate substantive knowledge into causal analyses [15]. Quantitative causal methods are well suited for estimating effects of well-defined exposures under stated assumptions, but they rely on strong conditions that are difficult to assess, particularly in the context of real-world evaluations of complex, multisession interventions. These include assumptions about how treatments are defined and experienced, which factors influence treatment receipt, which variables confound treatment-outcome relationships, and whether hypothesized mechanisms are operating as intended [16]. As causal methods are increasingly applied to complex interventions in the health sciences [2, 17], the need for systematic ways to rigorously identify and bring domain knowledge into causal analyses becomes more pronounced [9]. Although the importance of substantive expertise is recognized [9, 18], there is limited guidance on how such information should be generated, documented, and integrated.

Qualitative methods have often been used alongside quantitative analyses to explain findings, understand implementation, or provide contextual insight. We propose an additional role for qualitative inquiry: generating and incorporating the substantive knowledge causal assumptions require, both to strengthen the conditions under which they are likely to hold and to assess their plausibility. This extends mixed methods integration to the assumptions and analytic decisions that shape causal estimates and their interpretation.

CACE estimation as a motivating example for mixed methods

To illustrate how mixed methods can inform causal analyses in practice, we focus on one common challenge in health research; namely, problems with program participation, commonly referred to as “noncompliance” in the statistical literature. In such settings, researchers may be interested in estimating not only the intent-to-treat (ITT) effect (essentially, the effect of being assigned to one treatment versus another), but also in estimating the effect among individuals who would actually participate in the intervention as intended, known as the complier average causal effect (the CACE) [19, 20].

In trials where individuals assigned to the control group do not have access to the treatment, estimating the CACE requires distinguishing two groups: compliers, who would participate in the intervention if assigned to it, and noncompliers, who would not [19–21]. The challenge is that compliance status is observed only for individuals assigned to the treatment condition. For individuals assigned to control, we cannot know whether they would have participated if offered the intervention. Therefore, estimating the CACE relies on strong and untestable assumptions about unobserved potential outcomes. Two commonly used approaches to estimating the CACE – the principal score approach and the instrumental variable approach – each rely on a key untestable assumption.

The principal score approach aims to identify individuals in the control group who are likely to be compliers [20, 22]. This approach relies on an assumption known as principal ignorability, which states that the potential outcome under control is independent of principal stratum membership, conditional on a set of pre-treatment covariates [20]. In practice, this means that the measured covariates must capture important factors that both predict participation in the intervention and the outcome under control [18–20, 23]. If important factors are left unadjusted for, the assumption is violated and the result may be biased. For example, in a trial of a smoking cessation program delivered through primary care settings, patients with stronger social support networks may be more likely to attend counseling sessions if they are assigned to the intervention. At the same time, even in the absence of the intervention, individuals with stronger social support may be more likely to quit smoking. If social support is not measured at baseline, then analysis cannot account for this underlying difference. As a result, individuals in the control group who would have been compliers may appear to have better outcomes than those who would not have participated, not because of the intervention, but because they differ in their level of social support. When the principal score model compares outcomes for predicted compliers, it cannot distinguish the effects of the intervention from these underlying differences. In this case, the assumption of principal ignorability is violated, and the estimated effect among compliers may be biased.

The second approach uses treatment assignment as an instrumental variable and relies on the exclusion restriction. Under this assumption, assignment has no direct effect on outcomes for noncompliers, and thus, the entire effect of assignment is attributed to its impact on compliers [24, 25]. The exclusion restriction may be difficult to defend in trials of complex, multisession interventions where compliance is defined by a binary participation threshold [24, 26, 27]. For example, in a trial of a multicomponent diabetes self-management program, researchers might define the participation threshold as completing all seven modules. Under this definition, a patient who completes five modules, covering diet, exercise, glucose monitoring, and medication adherence, would be classified as a noncomplier. However, if those five modules lead to meaningful changes in self-management behavior, then the assumption that noncompliers experience zero effect does not hold. Another way the exclusion restriction may be violated is if assignment itself induces compensatory behaviors or psychological responses, even among individuals who do not participate in the intervention [20, 28]. For instance, a patient who learns they were selected for a diabetes intervention might independently begin improving their diet or monitoring their glucose levels. In both cases, assignment affects outcomes through pathways other than the defined treatment exposure, violating the exclusion restriction and potentially biasing the estimated CACE. These examples illustrate that defending the exclusion restriction in a multisession intervention requires knowledge about what constitutes meaningful participation so that the defined threshold is substantively defensible. The key assumptions underlying each approach are summarized in Table 1.

Table 1.

Key Assumptions Underlying CACE Estimation

Principal Ignorability

• Principal ignorability assumes that, after accounting for measured baseline covariates, compliers and noncompliers in the control group have similar expected outcomes. • The assumption may be violated if important factors that predict both participation and the outcome are not adjusted for. If such unmeasured confounders exist, the estimated effect for compliers may be biased. • The plausibility of principal ignorability depends on how well the measured covariates capture the factors that jointly drive participation in the intervention and outcomes of interest under control.

Exclusion Restriction

• The exclusion restriction assumes that treatment assignment affects the outcome only through its effect on treatment received. Under the exclusion restriction, the entire effect of assignment is attributed to compliers, and noncompliers are assumed to experience zero effect. • One potential violation of the exclusion restriction is if participation is defined by a binary cutoff (e.g., attending a minimum number of sessions), but individuals classified as noncompliers still receive meaningful exposure in earlier sessions. Additional threats to the exclusion restriction include psychological effects of assignment itself, which could influence outcomes independently of treatment. • The plausibility of the exclusion restriction depends on whether we can reasonably assume zero effects for noncompliers.

These examples also highlight that assessing the assumptions is connected to broader quantitative analytic decisions, including which baseline covariates are measured and included in the model, how participation is defined, and which identification strategy is most appropriate for the study context. Previous authors have called for increased collaboration between statisticians and applied researchers to assess these assumptions [21] and for designing experiments that collect covariates predictive of both compliance and outcomes [18, 22, 29]. Which covariates those are, and what constitutes meaningful participation, are questions that require substantive knowledge of the specific intervention, population, and context. Structured qualitative inquiry can provide this knowledge, helping researchers determine which assumptions are most defensible and which analytic approach is most appropriate for a given study context, ultimately strengthening both the transparency and credibility of CACE estimates.

Overview

This paper demonstrates how mixed methods can strengthen causal inference in applied health research through a proof-of-concept application of what we term the “CACE-MM” method: a structured integration of qualitative inquiry with quantitative CACE estimation. To our knowledge, CACE-MM is the first systematic application of mixed methods to probe the assumptions underlying estimation of causal effects.

The application draws on a randomized trial of a trauma-informed intervention for youth involved in the juvenile legal system and used qualitative inquiry to inform CACE analyses and assess the plausibility of two key assumptions: principal ignorability and the exclusion restriction. The qualitative findings elicited rich substantive information, including unmeasured threats to identification and important components of the intervention’s mechanisms that were previously not understood by the CACE-MM study team. These observations motivate a broader vision of CACE-MM as a method that encompasses both post-trial assessment of assumptions, as well as early exploratory qualitative work to inform the range of design, measurement, and analytic decisions that shape causal CACE estimates.

We organize the paper as follows. In Section Methods: the CACE-MM application, we describe the trial, the study design, and the procedures used in the proof of concept. In Section Results: CACE-MM application findings, we present the findings: first, the metainference, demonstrating how qualitative evidence informed the assessment of principal ignorability and the exclusion restriction, followed by procedural lessons about how to conduct this type of work. In Section Discussion: From proof of concept to framework: CACE-MM, we introduce the full CACE-MM framework, consider the broader implications for strengthening quantitative decisions in CACE analyses, and offer guidance for researchers who wish to integrate mixed methods from the design stage.

Methods: the CACE-MM application

The YES trial

The CACE-MM application draws on the YES trial, a randomized controlled trial (RCT) of electronic Practice Self-Regulation (e-PS-R), a 12-session trauma-informed intervention for youth involved in the juvenile legal systems of West Virginia and New Mexico. e-PS-R’s theory of change posits that by strengthening youths’ intentions, commitment, and self-efficacy to practice emotional regulation, youth will be supported in practicing those skills with the ultimate goal of reducing risky sexual behaviors [30]. As is common for multi-session or complex interventions, program participation was a pronounced challenge for the YES trial. Only 30% of youth assigned to treatment completed all sessions; median exposure was 5 (of 12) program components. See Qaragholi et al. [31] for more details on the e-PS-R intervention and on YES implementation and study results. The substantial participation challenges motivated interest in estimating the CACE for two outcomes, self-regulation and recent vaginal sex, to better understand the potential for intervention effect among those who experience the full intervention.

Using data from the YES trial, the CACE-MM study team – which included members of the original YES trial team as well as additional biostatisticians and mixed methodologists – estimated the CACE via an instrumental variable approach (relying on the exclusion restriction) and a principal score weighting approach (relying on principal ignorability). The two approaches yielded broadly similar findings for both outcomes. For self-regulation, measured approximately three months post-baseline, full participation was defined as completing at least 9 of the 12 program components, the point at which the intervention introduces concrete self-regulation strategies. The analytic models adjusted for age, race, ethnicity, sex, living with a parent or guardian, state, baseline affect regulation, and baseline self-regulation. The estimated CACE was 0.08 (SE 0.14) using principal score weighting and 0.10 (SE 0.27) using an instrumental variable approach. For recent vaginal sex, measured at long-term follow-up, full participation was defined as completing at least 6 of 12 components. The models adjusted for the same covariates as the self-regulation analysis, as well as baseline recent vaginal sex, intentions to have sex, and importance of having sex. The estimated CACE was − 0.07 (SE 0.06) using principal score weighting and − 0.09 (SE 0.08) using the instrumental variable approach.

However, each approach rested on strong identifying assumptions. To probe these assumptions and thus the plausibility of the resulting effect estimates, the CACE-MM study team designed and conducted the qualitative study described below. The quantitative findings are more fully described in a separate manuscript currently under review [31].

Study design

The CACE-MM application was designed to explore how qualitative inquiry could inform CACE analyses for the YES trial, including the plausibility of the identifying assumptions. Quantitative CACE analyses were conducted first, estimating the effect of participating in e-PS-R on participants’ self-regulation at post-program and recent vaginal sex at long-term follow-up. The qualitative component was subsequently conducted to assess the plausibility of the exclusion restriction and principal ignorability and inform interpretation of the CACE estimates. This sequencing aligns with established purposes for post-trial qualitative inquiry, including explaining quantitative findings and clarifying mechanisms within a theoretical model [14]. Figure 1 presents the study flow for the CACE-MM application, visually depicting how the quantitative CACE analyses and the qualitative inquiry were sequenced and integrated.

Fig. 1.

Overall design of the CACE-MM application

Participants and procedures

The parent YES trial was implemented at 19 sites in rural West Virginia and New Mexico between August 2016 and October 2022. A total of 630 youth enrolled in the trial; these data were used in the quantitative CACE analyses.

Participants for the qualitative study were purposively selected based on depth of experience with the e-PS-R program model, knowledge working with youth in the juvenile legal system, and familiarity with implementation across study sites. The final sample included ten key informants: two program developers, three program implementers, and five program trainers. Sample size was guided by the concept of information power [32].

Interview guide development

We developed two semi-structured interview guides: one for intervention developers, program trainers, and implementers, and a second for youth participants. The guides were reviewed by a biostatistician for alignment with the causal assumptions and discussed with a study advisory committee; the youth guide was additionally reviewed by a panel of youth aged 19–24.

The guides consisted of open-ended questions designed to elicit information relevant to predictors of participation, predictors of outcomes independent of intervention exposure, and the program’s theory of change. Questions addressing principal ignorability focused on challenges to participation (e.g., “What kinds of barriers or challenges make it harder for youth to participate in the intervention?”). Questions addressing the exclusion restriction tapped mechanisms of action (e.g., “Which session or sessions are the most important for helping youth strengthen their self-regulation skills?”).

Because the YES trial had closed (see Section Lessons: sampling), we were unable to recruit youth participants for the CACE-MM proof-of-concept application. Consequently, only the key informant guide was used. However, both guides are included in Supplementary File 1 as resources for researchers applying CACE-MM.

Analytic approach and integration

The quantitative analyses, described in Section The YES trial, estimated the CACE for each outcome using instrumental variable and principal score approaches. The qualitative data were analyzed using a two-stage coding process [33–35]. First, open coding identified categories of phenomena emerging from the data [36]. In a second stage, themes were mapped deductively to the COM-B model and the Theoretical Domains Framework (TDF) [37, 38]. This step enabled systematic connection of emergent themes to established behavioral constructs and standardizing of terminology across participants.

COM-B and TDF were selected for this application because evaluating principal ignorability and the exclusion restriction requires understanding both the factors that influence participation and outcomes and the pathways through which intervention exposure is expected to produce change. COM-B and TDF are frameworks that were developed with the explicit purpose to connect intervention components with the mechanisms through which behavior change is expected to occur; they provide theoretically informed domains spanning individual, social, and contextual determinants of behavior [37, 38]. As such, they provide a useful framework for examining the behavioral and contextual factors relevant to these assumptions, especially for a behavioral intervention such as e-PS-R.

Within CACE-MM, COM-B and TDF served as the integrative frameworks connecting qualitative findings to the causal assumptions. This structured process of mapping qualitative themes to behavioral domains and linking those domains to the causal assumptions constitutes the analytic integration step of the CACE-MM framework application. Themes were organized within behavioral domains to identify factors that plausibly influence program participation and factors that plausibly influence outcomes in the absence of the intervention. Domains that appeared relevant to both participation and outcomes were considered potential threats to principal ignorability if left unmeasured during the YES trial. The qualitative findings were also used to assess how key informants conceptualized meaningful exposure to intervention components and whether the participation thresholds used in the CACE analyses were defensible for invoking the exclusion restriction.

To structure the integration, the study team assessed the plausibility of each assumption for each outcome prior to the qualitative inquiry, based on program theory and previous experience on the trial. Following the qualitative analysis, the team reassessed each assumption in light of the new evidence. The metainference was produced by comparing prior and posterior assessments, identifying where qualitative findings corroborated, extended, or complicated the team’s initial judgments.

Results: CACE-MM application findings

The central aim of the CACE-MM application was to assess whether qualitative inquiry could inform the plausibility of the identifying assumptions underlying CACE analyses. We present the metainferences below. We then describe procedural lessons about how to conduct this type of work.

Metainference

The qualitative findings provided substantive evidence that informed interpretation of the CACE analyses for both outcomes. Prior to the qualitative study, the CACE-MM study team assessed the plausibility of each assumption for each outcome based on program theory, internal discussions, and consultation with program developers. The YES trial represented favorable conditions for this kind of assessment: it was grounded in an articulated theory of change, collected an extensive set of baseline covariates, and was conducted by a team with deep knowledge of the program and population. Even so, the qualitative inquiry produced information that was not available from the quantitative data or the team’s prior knowledge. Below, we present the prior assessments for each assumption and describe how the qualitative findings extended or complicated them.

Principal ignorability

The central question for principal ignorability is whether important joint predictors of both participation in the intervention under treatment and the outcome of interest under control remain unmeasured. Although the YES trial collected extensive baseline covariates aligned with the intervention’s theory of change, the CACE-MM study team identified several plausible unmeasured factors that likely influenced both participation and the outcomes under control. Quantitatively, differences between compliers and noncompliers were small, and there was substantial overlap in predicted compliance between treatment compliers and control participants, suggesting that the measured covariates provided limited information for distinguishing likely compliers from noncompliers. That result alone does not determine whether principal ignorability is plausible; however, it motivated further inquiry into whether important joint predictors of participation and each outcome remained unmeasured.

The qualitative interviews helped identify specific predictors that may represent unmeasured threats to principal ignorability. The principal score models included baseline covariates such as self-regulation, intentions, and values related to sexual behavior; in the post-trial interviews, key informants also identified these constructs as relevant to both participation in the intervention and outcomes under control. However, informants identified several additional constructs as predictors of both participation and at least one of the two outcomes under control. These included baseline levels of trauma exposure, memory and decision processes, social influences, environmental context and resources (including housing stability, transportation, and digital access), behavioral health, and cultural norms and stigma; these constructs were not measured at baseline in the YES trial and therefore could not be included in the models. Capability- and opportunity-related factors were emphasized most frequently across interviews. Because these constructs were not adjusted for, they represent potential threats to principal ignorability.

The qualitative work also surfaced candidate predictors unlikely to emerge from empirical literature alone. For example, the CACE-MM study team had considered digital access as a predictor of participation (but not outcomes) prior to the interviews. However, informants suggested that digital access may also predict sexual behavior, given the evolving nature of online relationships for contemporary adolescents. This illustrates how qualitative insight can expand understanding of which factors might threaten principal ignorability and increase transparency about where residual bias may remain.

Even with qualitative data, principal ignorability remains untestable. However, the qualitative findings identified several plausible joint predictors of participation and outcomes that were not measured at baseline, making principal ignorability more difficult to defend under the available covariate set for either outcome in the YES trial. These findings point to cautious interpretation of the principal score CACE estimates and motivate targeted sensitivity analyses.

Exclusion restriction

The plausibility of the exclusion restriction depended on whether noncompliers could reasonably be expected to experience no effect of assignment. Prior to the qualitative interviews, the CACE-MM study team defined the minimum participation threshold for self-regulation as completion of at least 9 sessions, a threshold aligned with the intervention’s theory of change: earlier sessions focus on understanding trauma’s effects, whereas session 9 introduces concrete self-regulation strategies. Without exposure to this content, the CACE-MM study team deemed it unlikely that participants would acquire the skills necessary to practice self-regulation. Potential violations were also considered via psychological pathways; however, the CACE-MM study team judged these unlikely to produce substantial changes in self-regulation given the broader stressors faced by youth in the juvenile legal system. For recent vaginal sex, the CACE-MM study team had difficulty identifying a clear exposure threshold below which effects would plausibly be zero – the team could not point to a specific cutoff in the intervention as the minimum necessary for producing change. This raised concerns about the exclusion restriction for this outcome.

For self-regulation, the qualitative interviews confirmed that exposure to concrete regulation strategies was central to change, corroborating the CACE-MM study team’s prior assessment. However, qualitative analysis also revealed that co-regulation and facilitator modeling emerged as meaningful components linked to self-regulation. These pathways were not documented in formal program materials and had not been previously identified by either the YES trial team or the CACE-MM study team. This finding suggests that defining compliance solely in terms of the number of sessions completed may incompletely capture the intervention’s mechanism for producing change in self-regulation.

For recent vaginal sex, no clear minimum threshold emerged from the qualitative data either. Informants struggled to identify a specific point at which the intervention should have an effect. This convergence between the CACE-MM study team’s prior difficulty and the informants’ difficulty strengthens the conclusion that imposing a binary cutoff and invoking the exclusion restriction is difficult to justify for this outcome. In this case, the question shifts from which threshold to use to whether a binary compliance is appropriate for this outcome at all.

Taken together, the findings suggest that the CACE estimates from the YES trial should be interpreted with caution. For the principal score estimates, the omission of plausible joint predictors raises concerns about bias in the estimated effects for both outcomes. For the instrumental variable estimates, the co-regulation finding complicates the exclusion restriction for self-regulation, while the absence of a defensible threshold undermines the exclusion restriction for recent vaginal sex. In some cases, the qualitative inquiry confirmed concerns the team had already identified. In others, such as the co-regulation component and the role of digital access as a predictor of outcomes, it revealed information that meaningfully changed the team’s assessment and that would not have emerged from the quantitative data or the team’s prior expertise alone.

Lessons from the CACE-MM application

In addition to the metainference, the CACE-MM proof-of-concept application yielded methodological lessons about how mixed methods approaches can be used to support causal inference. We organize these by key components of the qualitative study: design, sampling, interview guide development, and analytic strategy.

Lessons: design

Conducting qualitative work after the trial enabled the team to assess the plausibility of the underlying assumptions and inform interpretation of the CACE estimates. However, the design limited the extent to which qualitative findings could strengthen the quantitative strand. Because interviews were conducted after the trial, newly identified predictors of participation and outcomes could not be incorporated into quantitative baseline questionnaires or captured through administrative proxies and thus could not be included in the principal score models. Similarly, the co-regulation mechanism could not be used to refine how participation was defined, because it had not been measured during the trial. Had qualitative inquiry been conducted before the trial, these findings could have directly informed survey design and participation definitions; identified post-trial, they are limited to informing interpretation and informing future studies.

Lessons: sampling

Evaluating the exclusion restriction and principal ignorability requires individuals with deep knowledge of the intervention model, the target population, and the implementation context. These considerations guided our inclusion of program developers, trainers, and implementers, who were well positioned to describe how the intervention operates in practice and which factors shape youth participation.

However, our sampling strategy in the application was constrained by the fact that the YES trial was a closed study, which prevented us from contacting former participants. Youth perspectives on barriers to participation and experiences of intervention exposure would likely have strengthened the analysis. In particular, participants assigned to treatment could provide insight into the factors shaping their own participation and whether the exposure they received produced meaningful change, while those assigned to control could describe influences on outcomes independent of the intervention.

In the absence of participant input, our findings reflect professional interpretations of youth behavior, which may differ from lived experience. When possible, future studies should include participants in the qualitative sample. Stratified purposeful sampling that links observed participation levels to qualitative inquiry may yield valuable data on why some individuals participate more than others and whether defined thresholds reflect meaningful differences in exposure [16, 39]. Additional key informants with contextual knowledge (e.g., probation officers and counselors in the context of the YES trial) may also contribute valuable insights about factors that shape participation and outcomes for specific populations.

Lessons: interview guides

Translating formally defined assumptions into qualitative research questions requires close interdisciplinary collaboration. Within the CACE-MM study team, biostatisticians clarified what types of substantive information would be relevant to evaluating the assumptions, while qualitative and mixed methods team members translated these needs into focused research questions that could be meaningfully addressed through interviews. Effective integration required developing a shared analytic language and establishing agreement on how qualitative findings would inform causal modeling decisions.

Stakeholder review played an important role in refining the interview guides. Feedback from a biostatistician was particularly helpful in identifying a key conceptual issue: early drafts of the guide did not clearly distinguish between factors influencing participation and factors influencing outcomes under control. This distinction is central to assessing principal ignorability. Revisions clarified this separation.

The study advisory committee also helped translate causal concepts into accessible language. Questions were reframed to focus on concrete program features and implementation experiences rather than abstract causal terminology. In addition, the youth advisory panel emphasized that interviewer characteristics, including age proximity, gender, and prior rapport, may influence participants’ comfort discussing sensitive topics. These considerations underscore how interviewer positionality can affect the quality and depth of qualitative data.

Lessons: analytic strategy

The two-stage inductive-deductive approach yielded a methodological finding: the qualitative approach and analytic strategy should be matched to the specific assumption under examination. Assessing the exclusion restriction requires understanding how the intervention produces change and whether the participation threshold captures meaningful exposure. In the application, open coding and thematic analysis allowed informants’ descriptions of core components and mechanisms to emerge without predefined constraints, which is how the co-regulation finding was identified.

Assessing principal ignorability involves a more bounded task: identifying constructs that predict both participation and outcomes. Because the goal is to surface candidate predictors rather than to understand how a process unfolds, structured approaches, such as deductive coding to a behavioral framework or rapid qualitative approaches may provide more efficient pathways for generating candidate constructs [40, 41].

Discussion: from proof of concept to framework: CACE-MM

Estimating causal effects in applied settings depends on a series of quantitative decisions and on the choice of assumptions about unobserved – and unobservable – outcomes. These decisions shape the credibility of the resulting causal estimate. As the proof of concept demonstrates, qualitative inquiry can meaningfully inform these quantitative decisions. At the same time, this exercise revealed the limits of using qualitative work only after the trial has been completed: the CACE-MM study team’s ability to strengthen the quantitative analyses was constrained by decisions that had already been made in earlier phases of the parent trial.

Taken together, these findings motivate a broader framework that integrates qualitative inquiry earlier in the research process and treats it as a tool for strengthening the quantitative decisions on which CACE analyses depend. This perspective also aligns with prior recommendations that issues related to noncompliance and causal assumptions should be considered during study design [18, 22, 29]. CACE-MM builds on these ideas by providing a structured mixed methods framework for doing so.

The CACE-MM framework

The CACE-MM framework integrates qualitative inquiry with CACE analyses across two complementary phases (Fig. 2): an exploratory phase, conducted prior to the trial, and an explanatory phase, conducted during or after the quantitative analysis.

Fig. 2.

CACE-MM: Integrating qualitative and quantitative strands for CACE analysis. The shaded portion of the figure indicates the component implemented in the proof-of-concept application

In the exploratory phase, qualitative inquiry is used to generate the substantive knowledge needed to design CACE analyses that rest on defensible ground. This phase focuses on informing key quantitative design decisions, including which constructs should be measured at baseline, how participation should be defined, and which identification strategies are most plausible given the intervention and study context.

In the explanatory phase, qualitative inquiry is used to assess whether the assumptions invoked in the quantitative analysis are plausible and to contextualize the interpretation of the resulting causal estimates. This phase involves examining the mechanisms through which the intervention is believed to operate and the factors shaping participation and outcomes.

Figure 2 illustrates the overall structure of the framework, showing the progression from exploratory qualitative inquiry to quantitative CACE analyses and then to explanatory qualitative assessment. The shaded portion of the figure indicates the component implemented in the application. Table 2 summarizes the key decision points that structure the framework, spanning decisions made before the trial through interpretation of the resulting causal estimates. Section CACE-MM exploratory phase: informing study design discusses considerations for implementing the exploratory phase, and Section CACE-MM explanatory phase: assessing assumptions and interpreting estimates discusses the explanatory phase.

Table 2.

CACE-MM: decision points, contribution, and recommendations

Decision Point	Analytic Concern and CACE-MM Contribution	Outputs for CACE Analysis	Recommendations
Exploratory phase (before the trial)
Which baseline covariates should be measured?	Principal ignorability requires adjustment for covariates that predict both participation and outcomes under control Omitting joint predictors may bias estimates CACE-MM contribution: Uses qualitative inquiry to surface context-specific constructs that influence both participation and outcomes	• Candidate baseline covariates to measure • Constructs to include in principal score models • Documentation of which constructs were considered and how they were identified (e.g., semi-structured interviews with key informants)	• Include participants in qualitative interviews when possible; stratified purposeful sampling can reveal why some individuals participate more than others • Prioritize measurement of identified constructs based on feasibility, available measures, alignment with the theory of change, and prior empirical literature • If feasible, pilot measurement of qualitatively identified constructs to assess their predictive value for participation and outcomes
How should participation be defined?	Participation definitions affect both identification strategies. For instrumental variable approaches, the exclusion restriction requires a threshold below which the intervention plausibly has zero effect. Mis-specified thresholds bias instrumental variable estimates For principal score approaches, the participation definition determines who is classified as a complier and therefore which population the CACE estimate represents CACE-MM contribution: Probes theory of change for how the intervention produces change and what constitutes meaningful exposure from multiple expert perspectives	• Candidate participation thresholds grounded in substantive knowledge • Alternative compliance definitions for sensitivity analyses • Assessment of whether a binary compliance framework is appropriate for each outcome	• Use candidate definitions for the primary analysis and for sensitivity analyses under alternative definitions • If no defensible threshold can be identified for a given outcome, reconsider the binary compliance framework
Explanatory phase (during or after the analysis)
Which identification strategy should be used?	Different CACE approaches rely on distinct assumptions Plausibility may vary across outcomes within the same study CACE-MM contribution: Identifies threats to each assumption to clarify which approach is most defensible	• Selection of analytic approach (instrumental variable vs. principal score) informed by qualitative evidence • Identification of assumption vulnerabilities, by outcome	• Assess the defensibility of each assumption at the level of analysis • Use qualitative evidence to inform whether principal score, instrumental variable, or both approaches are appropriate • If neither assumption is clearly defensible, consider running multiple approaches as sensitivity analyses
Is principal ignorability plausible?	If unmeasured factors predict both participation and outcomes, principal score estimates may be biased CACE-MM contribution: Identifies constructs whose omission is most consequential	• Identification of plausible unmeasured confounders with qualitative evidence for why they matter • Inputs for sensitivity analyses or bounding approaches	• Transparently document omitted constructs • Use qualitatively identified constructs to parameterize bounding approaches or sensitivity analyses • If qualitative evidence suggests principal ignorability is unlikely to hold, consider alternative approaches and interpret principal score estimates with caution
Is the exclusion restriction plausible?	If noncompliers experience nonzero effects, instrumental variable estimates may be biased CACE-MM contribution: Assesses whether participation definitions capture meaningful exposure	• Assessment of defensibility of participation thresholds for each outcome • Candidate alternative thresholds for sensitivity analyses • Identification of mechanisms not captured in program documentation	• Assess the exclusion restriction separately for each outcome • Evaluate whether the threshold reflects meaningful exposure or omits relevant components • If mechanisms beyond the compliance definition are identified, discuss implications for instrumental variable estimates • Conduct sensitivity analyses across alternative thresholds informed by qualitative findings
How should the estimates be interpreted?	CACE estimates are conditional on assumptions. Readers need to understand who the estimated effects apply to and where uncertainty remains CACE-MM contribution: Contextualize who compliers represent, what participation entails, and where uncertainty lies	• Interpretation of effect estimates grounded in substantive evidence • Transparent discussion of where assumptions are most vulnerable	• Discuss the plausibility of each assumption for each outcome • Organize the metainference for each assumption with transparent documentation • Clearly communicate where assumptions are most vulnerable and how that affects conclusions
How can findings inform future studies?	Vulnerabilities to assumptions vary across interventions and contexts; building cumulative knowledge improves future CACE design CACE-MM contribution: Documents which constructs function as joint predictors and which thresholds are defensible to build cumulative knowledge	• Documented constructs and their relevance to assumptions for future researchers • Substantive evidence base for where CACE approaches and their key assumptions may be most vulnerable	• Report qualitatively identified constructs even if they could not be incorporated into the current analysis • Document which assumptions were most vulnerable and why • Use qualitative evidence to inform covariate selection and measurement, participation definitions, and sensitivity analyses in future studies
Considerations across phases
Cross-cutting implementation considerations	The quality and relevance of qualitative findings depend on who is sampled, what questions are asked, how data are analyzed, and whether the qualitative work is aligned with the causal aims of the study CACE-MM contribution: Provides a structured approach for connecting qualitative methods to the specific assumptions and decisions that CACE analyses require	• Shared understanding of causal questions and assumptions • Structured linkage between qualitative findings and causal assumptions • Interview guides aligned with the causal assumptions • A sampling plan matched to the decision points the study is intended to inform	• Involve biostatisticians and mixed methods researchers early in study design • Use purposive sampling to recruit individuals with knowledge of the intervention, study population, and implementation context • Use a theoretical framework (e.g., COM-B, TDF) when useful to organize findings and connect them to the causal assumptions • Match qualitative methods to the analytic task (e.g., inductive approaches for exploring mechanisms and meaningful exposure; structured or rapid approaches for identifying predictors relevant to principal ignorability)

CACE-MM exploratory phase: informing study design

The goal of the exploratory phase is to generate the substantive knowledge needed to strengthen CACE analyses. For studies where participation challenges are anticipated and CACE analyses are planned, integrating qualitative inquiry before baseline questionnaires are finalized allows researchers to incorporate this knowledge directly into study design.

Exploratory qualitative work may take several forms, including a qualitative pilot phase, exploratory qualitative work early in the study (prior to finalization of baseline questionnaires), or structured consultation with stakeholders who have deep knowledge of the intervention and the target population. One important contribution of qualitative inquiry at this stage is identifying candidate baseline covariates; qualitative approaches may be especially valuable when context-specific predictors of participation and outcomes, as well as the intervention’s mechanisms of action, are not well understood.

Qualitative inquiry can also inform how participation should be defined. Participation definitions play different roles depending on the identification strategy used in the CACE analysis. For instrumental variable approaches, the participation threshold determines the point below which the intervention is assumed to have zero effect, and the central question becomes whether that threshold is substantively defensible. Qualitative inquiry conducted prior to the trial could therefore inform more defensible participation definitions.

Notably, for principal score approaches, different participation thresholds correspond to potentially different complier populations of interest and different causal estimands. For example, different stakeholders may identify different thresholds for what constitutes meaningful exposure to the intervention. Specifically, participants may emphasize the components that influenced their behavior, implementers may focus on what level of participation is realistic in practice, and intervention developers may emphasize the level of exposure required by the program’s theory of change. Each perspective reflects a different understanding of how the intervention operates and for whom it is expected to work. Eliciting these perspectives prior to the trial enables researchers to define participation in terms that are grounded in domain knowledge and, when appropriate, to estimate effects across multiple substantively meaningful participation definitions.

A natural concern is whether integrating qualitative inquiry at this stage introduces excessive measurement burden. In practice, the application findings identified a relatively focused set of candidate predictors, several of which were relevant across outcomes. Further, qualitative findings do not require that all identified constructs be measured; rather, they generate a set of candidates that researchers can prioritize based on considerations such as feasibility, existing measurement tools, and alignment with the intervention’s theory of change.

One strategy for prioritization may be to pilot the measurement of qualitatively identified constructs in a preliminary study and evaluate their associations with participation and outcomes before incorporating them into the main trial. Rosenbaum and Silber [42] provide an example of how qualitative methods during a pilot phase can guide covariate selection for causal analyses. The broader point is that if researchers anticipate using CACE analyses, or other causal methods that rely on strong assumptions, designing studies with those analyses in mind is a necessary step for producing credible estimates. Mixed methods offer a principled way to inform that design.

CACE-MM explanatory phase: assessing assumptions and interpreting estimates

For studies in which the trial has already been conducted, qualitative inquiry can still play an important role in evaluating the plausibility of the assumptions invoked in the analysis and in guiding interpretation of the resulting causal estimates.

Qualitative evidence about intervention mechanisms and the factors shaping participation can clarify which assumptions are most vulnerable for each outcome and why. This information can help researchers evaluate whether the chosen identification strategy is defensible in the study context and identify potential threats to the underlying assumptions.

For example, qualitative findings may reveal constructs that plausibly influence both participation and outcomes but were not measured at baseline, thereby identifying potential threats to principal ignorability. In such cases, the qualitative findings can inform sensitivity analyses or bounding approaches by identifying specific constructs that may represent unmeasured confounders.

Similarly, qualitative evidence about how the intervention produces change can inform the defensibility of the compliance definition used in instrumental variable analyses. If the participation threshold does not clearly distinguish meaningful exposure to the intervention, the exclusion restriction may be difficult to defend. Qualitative findings can therefore help researchers evaluate whether the chosen threshold is substantively justified and whether alternative thresholds should be examined.

Practical considerations

Implementing CACE-MM requires deliberate planning across several dimensions. The lessons from the proof of concept (Section Lessons from the CACE-MM Application) address several aspects of this process, including sampling strategies, interview guide development, interdisciplinary collaboration, and analytic approaches. Here we highlight additional considerations for applying the framework in practice.

First, the scope of the qualitative inquiry should be guided by the specific causal decisions it is intended to inform. The application focused narrowly on two identifying assumptions, which produced targeted and actionable findings. In some contexts, broader qualitative inquiry – for example, exploring how different stakeholders understand meaningful participation – may generate additional insights that inform participation definitions and sensitivity analyses. However, expanding the scope of the inquiry also increases analytic complexity. Without clear prioritization, researchers risk generating qualitative data that are difficult to connect directly to causal modeling decisions.

Second, the choice of theoretical framework can facilitate the integration of qualitative findings with causal reasoning. In the application, the COM-B model and TDF provided a structured vocabulary for organizing themes related to participation and behavioral outcomes. Using such frameworks can help researchers systematically link qualitative findings to the factors relevant to principal ignorability and the exclusion restriction. Alternative theoretical frameworks may be appropriate for different interventions or assumptions under investigation.

Finally, integrating qualitative inquiry into a trial requires additional investment in study design, data collection, and interdisciplinary collaboration. Several features of the CACE-MM approach are intended to make this investment manageable. Focusing the qualitative inquiry on specific decision points helps limit the volume of data that must be collected and analyzed. Matching the qualitative analytic strategy to the assumption being examined allows researchers to use more efficient approaches when appropriate. And organizing findings assumption-by-assumption provides a transparent structure for documenting how qualitative evidence informs causal interpretation.

Limitations

The proof-of-concept application demonstrates how qualitative inquiry can be integrated with CACE analyses and how such integration can inform key causal decisions. A notable limitation concerns the scope of the application. We implemented only the explanatory phase; the exploratory phase, which we argue offers the greatest potential for strengthening CACE analyses, has not yet been tested. Further, the present application illustrates CACE-MM within a randomized trial using a binary participation definition and focuses on two assumptions underlying CACE estimation. Although the broader logic of CACE-MM – using qualitative inquiry to inform and assess the assumptions underlying a causal estimate – is not specific to this setting, additional work is needed to adapt the framework for other settings, such as partial or continuous compliance, time-varying exposures, more complex principal strata, alternative causal estimands, or nonrandomized designs. Applications across these settings, and implementation of the exploratory phase, will be the natural next steps in developing the framework.

Another limitation concerns the composition of the qualitative sample. Qualitative inquiry represents one source of evidence for identifying constructs relevant to causal assumptions; because informants draw on their own professional and lived experience and vantage point, it does not necessarily generate an exhaustive set of predictors. Informants may emphasize factors that are most visible within their professional roles or most salient within the context of program implementation. In the present study, for example, informants identified behavioral health as a predictor of participation but did not raise it as a predictor of sexual health outcomes, despite a substantial body of adolescent health research documenting associations between mental health and sexual risk behavior [43–46]. These considerations highlight an important implication of the CACE-MM approach: qualitative findings should be interpreted alongside existing empirical evidence and theory when identifying candidate predictors for causal analyses. The purpose of qualitative inquiry in this framework is therefore not to replace prior knowledge, but to complement it by surfacing context-specific factors that may otherwise remain unobserved in quantitative data.

Additionally, not all constructs identified through qualitative inquiry are straightforward to operationalize at baseline. Some, such as trauma exposure or digital access, may be captured with existing validated tools or brief self-report items. Others, such as the quality of social influences or the role of cultural norms and stigma, may require more targeted measurement strategies that are not always available or feasible within the constraints of a trial. Identifying a construct as a plausible threat to principal ignorability is a necessary first step, but translating that insight into an adequate baseline measure requires additional methodological decisions. This is a practical boundary of the CACE-MM approach: it can surface what matters, but it cannot guarantee that what matters can be measured.

Finally, an important area for future methodological development concerns the use of qualitative findings to inform sensitivity analyses or bounding approaches (Table 2). In the present study, qualitative inquiry was used to identify and characterize potential threats to the identifying assumptions. A natural next step is to explore whether qualitative or expert knowledge can be incorporated more directly into sensitivity analyses by helping quantify the likely strength of qualitatively identified threats to the assumptions. Related work in Bayesian prior elicitation highlights both the potential value and methodological challenges of formally incorporating expert knowledge into quantitative analyses and may provide a useful starting point for future work in this area [47]. Nevertheless, substantial methodological development is needed before such approaches can be routinely applied in the context of CACE analyses.

Conclusion

Mixed methods designs are especially well positioned to support trials where participation challenges are anticipated and CACE analyses are planned. This recommendation rests on the premise that the yield of mixed methods integration, in terms of improved measurement, more defensible assumptions, and more transparent interpretation, justifies the additional investment in study design and interdisciplinary collaboration. The findings from this application support this premise. In the YES trial, both the instrumental variable and principal score approaches yielded broadly consistent CACE estimates for recent vaginal sex, yet careful examination of the assumptions revealed that neither estimate rested on defensible ground for that outcome. CACE-MM provides a structured approach for both strengthening and assessing the credibility of the resulting causal estimates.

The implications of this work extend beyond methodological audiences. Practitioners and policymakers rely on causal estimates to guide decisions about interventions and resource allocation; by strengthening the credibility and transparency of those estimates, mixed methods integration can help ensure that the evidence guiding those decisions reflects how interventions actually operate and for whom they work. More broadly, systematic documentation of which constructs function as joint predictors, which participation thresholds are defensible, and which assumptions are most vulnerable in specific settings can build cumulative knowledge that improves both the design of future CACE analyses and the development of the interventions they evaluate. The CACE-MM framework (Fig. 2) and accompanying decision points (Table 2) are intended to support researchers in this work. Ultimately, strengthening causal inference in medicine and public health requires bringing statistical methods into closer dialogue with substantive knowledge about how interventions are delivered and experienced in practice. The CACE-MM framework offers one approach for doing so.

Abbreviations

CACE

Complier average causal effect

CACE-MM

Complier average causal effect mixed methods framework

COM-B

Capability, Opportunity, Motivation-Behavior model

e-PS-R

Electronic Practice Self-Regulation

RCT

Randomized controlled trial

TDF

Theoretical Domains Framework

YES

Youth Empowerment Study

Authors’ contributions

NQ conceptualized the study, developed the CACE-MM framework, designed the qualitative study, conducted the qualitative data collection and analysis, performed the integration of qualitative and quantitative findings, and wrote the original draft of the manuscript. JJG contributed to the conceptualization of the mixed methods design, provided methodological guidance on qualitative approaches, and reviewed and edited the manuscript. LKB contributed to the conceptualization of the mixed methods design, provided methodological guidance on qualitative approaches, and reviewed and edited the manuscript. TQN contributed to the quantitative CACE analyses, provided methodological guidance on causal inference, and contributed to the interpretation of qualitative findings through a causal inference lens. SW reviewed and edited the manuscript. EAS contributed to the conceptualization of the study, provided methodological guidance on causal inference and CACE estimation, provided feedback on the qualitative interview guides and interpretation of qualitative findings through a causal inference lens, and reviewed and edited the manuscript. All authors read and approved the final manuscript.

Funding

The quantitative analyses reported in this publication were prepared under Grant Number 90AP2679-01-00 from the Family and Youth Services Bureau within the Administration for Children and Families (ACF), U.S. Department of Health & Human Services (HHS). The views expressed are those of the authors and do not necessarily represent the policies of HHS, ACF, or the Family and Youth Services Bureau. The qualitative component and doctoral training of the lead author were supported by the Bloomberg American Health Initiative at the Johns Hopkins Bloomberg School of Public Health.

Data availability

The quantitative data used in this study are from the Youth Empowerment Study (YES) trial (ClinicalTrials.gov: NCT03242447) and are not publicly available due to participant confidentiality protections. Requests for access to the data may be directed to the corresponding author.

Declarations

Ethics approval and consent to participate

The quantitative component of this study used data from the YES trial, which was conducted under Sterling IRB approval (IRB ID: 5868) with informed consent obtained from all participants. The qualitative component received a non-human subjects determination from the Johns Hopkins Bloomberg School of Public Health Institutional Review Board.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.