Leveraging machine learning approach to identify relationships between practice facilitation strategies and practice characteristics based on the implementation research logic model

Abstract

Background

Machine learning (ML)—a field of study dedicated to the principled extraction of knowledge from complex data—can benefit implementation science, quality improvement (QI), and primary care research. Given the general complexity of implementation research and the need to develop strategies for understanding relationships among practice characteristics and practice facilitation strategies, we chose the Implementation Research Logic Model (IRLM) as an underlying structure for the data and to identify relationships that might be associated with outcomes.

This study illustrates this novel method involving ML and an IRLM in the context of a practice facilitation-supported QI program in primary care.

Methods

We applied advanced statistical methods within a machine learning framework to data from the Healthy Hearts in the Heartland (H3) study, including practice facilitation data and practice and staff participation survey, to assess the relationship between practice attributes and practice facilitator strategies and their impact on successful implementation of QI interventions. We used PCA for feature selection, incorporated practice facilitators' knowledge for contextual factor validation, and employed Structural Equation Modeling (SEM) to analyze relationships among contextual factors, latent variables, practice facilitation strategies, and outcomes.

Results

We selected 20 contextual factors and identified practice facilitation strategies and mapped them to the IRLM. Cronbach’s alphas of contextual factors in the five domains (Intervention characteristics, outer setting, inner setting, characteristics of individuals, and implementation process) are 0.71, 0.82, 0.72, 0.89, 0.86, respectively. We used structural equation modeling to analyze the relationships among contextual factors, latent variables, practice facilitation strategies (Doing Tasks, Project Management, Consulting, Teaching, and Coaching), and outcomes (number of implemented QI interventions and Change Process Capability Questionnaire (CPCQ) score). All five facilitation strategies had statistically significant associations with the implementation of QI interventions (all P < 0.05).

Conclusions

The combination of ML and the theory behind the IRLM can be used to identify relationships between inner and outer context determinants and implementation strategies and study outcomes in pragmatic research study datasets. All the proposed strategies in H3 were statistically associated with completed QI interventions; and the strategies had more impact on the implementation of interventions than CPCQ change. By understanding the relationship between outcomes, practice determinants and coaching strategies, practice facilitators can better help primary care practices adapt and implement interventions and build capacity to adapt to change.

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-025-00850-6.

Contributions to the literature.

• This study aimed to assess the relationships between practice context, practice facilitation strategies and study outcomes by developing a model using the Implementation Research Logic Model (IRLM) to structure and group the data.

• This study demonstrates that it is possible to use machine learning in the context of the IRLM to develop interpretable models from implementation research data.

• This study also extends application of the IRLM by adding important computational process to address the problem of how to combine existing knowledge and machine learning to facilitate the intersection of implementation science and informatics research.

Introduction

Implementation research is the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice, and, hence, to improve the quality and effectiveness of health services and care [1]. This relatively new field includes the study of influences on health care professionals and organizational behaviors [2]. Implementation science in primary care involves investigation at multiple levels, and the targets of investigation include patients, clinical team members, primary care clinics, health care organizations, communities, society, and the health system as a whole [3].

The Implementation Research Logic Model (IRLM) is a semi-structured, principle-guided tool based on existing implementation and evaluation frameworks for implementation science designed to improve the specification, rigor, reproducibility, and testable pathways involved in implementation research projects [4]. The IRLM provides a conceptual structure based on a generalized theory of implementation research by specifying the relationships between five key elements, including (1) Determinants: contextual factors that influence implementation, (2) Implementation Strategies: methods used to facilitate adoption (e.g., practice facilitation activities), (3) Mechanisms: processes through which strategies produce effects, (4) Implementation Outcomes: proximal results of implementation efforts, and (5) Clinical/Service Outcomes: ultimate impacts on care quality [4]. The IRLM's strength lies in making explicit the hypothesized relationships among these elements, improving specification, rigor, and reproducibility of implementation research [5].

Machine learning (ML) exists at the intersection of statistics, which seeks to learn relationships from data, and computer science, with its emphasis on efficient computing algorithms. ML has gained significant attention in the past decades, particularly in the context of improving health and well-being [6]. ML works by combining large amounts of data with fast and iterative processing using intelligent algorithms, allowing the system to identify and learn patterns or features from the data [7]. In implementation science, features can be referred to contextual factors or determinants of implementation. Although there are an increasing number of studies and publications of ML applications in health care [8, 9], only a few have demonstrated the value of ML in implementation and primary care research [10, 11] This study aims to leverage ML and the IRLM to evaluate the relationship between practice contextual factors and practice facilitation strategies in a large-scale cardiovascular quality improvement research project implemented in primary care practices.

Case example

To illustrate the use of these novel methods for implementation research, we present a case example drawing data from a completed quality improvement (QI) project that used practice facilitation to support implementation of multiple practice-level interventions to improve blood pressure in primary care.

Study population

Healthy Hearts in the Heartland (H3) was the Midwestern collaborative within a larger, nationwide program—EvidenceNOW, Advancing Heart Health [12] which was funded by the Agency for Healthcare Research and Quality, to improve cardiovascular care quality in small- and medium-size primary care practices (N = 226) based on the Million Hearts Measures [13]. Each practice was assigned a practice facilitator who deployed the H3 intervention and worked with practice staff to design and implement their QI plan. The intervention period included a 12-month intervention implementation phase, followed by a 6-month sustainability phase. The H3 study examined the role of practice facilitation on improvement of the 4 Million Hearts cardiovascular clinical quality measures (CQM) in small primary care practices in Illinois, Indiana, and Wisconsin. The 4 clinical quality measures include appropriate Aspirin therapy, Blood pressure control, Cholesterol management, and Smoking cessation [14] and there were 35 intervention components (see Supplemental Table 2) to support the implementation of the 4 measures. Full study details have been described in Ciolino et al. [14] and overall study outcomes were described in Persell, et al. [15].

Practice characteristics

We collected the data utilizing multiple platforms: practice facilitation activity data and intervention completion data was collected with the Facilitation ACtivity and Intervention Tracking System (FACITS), implemented in Quickbase (Quick Base, Inc.; Cambridge, MA USA) and survey data was collected using the Research Electronic Data Capture (REDCap) platform [16]. The main data sources for this study include practice survey, practice staff survey, and practice facilitation activities. Practice characteristics, including the aforementioned variables for ML modeling and additional variables uniform to the EvidenceNOW cooperatives were recorded for each enrolled practice at baseline, and a subset of these variables were collected at 12- and 18-month follow-up [17].

Practice facilitation

Practice facilitation is a supportive service (implementation strategy) that assists practices with implementing changes and developing capacity for sustained QI and to address intervention implementation gaps [18]. A growing body of evidence suggests that QI programs that use practice facilitation can produce meaningful positive change in primary care practices [14, 19], including improvements in the chronic disease process and outcome measures in primary care for diabetes, asthma [20], cardiovascular disease, and cancer [21]. However, much work remains to be done to determine the most effective practice facilitation strategies for different practice types and configurations to support sustained practice change.

Practice facilitators are individuals who are specially trained to provide QI coaching and help practices engage in QI projects and develop capacity for continuous quality improvement [18]. Practice facilitators had varied expertise in QI, EHR technical experience and clinical backgrounds. In H3, the practice facilitation activities were mapped to 5 standardized categories (Doing Tasks, Project Management, Consulting, Teaching, and Coaching (Tables 2 and 3)) [11] to support understanding of the types of facilitation and their relationship to practice success in implementing the intervention. Practice facilitation encounters occurred 1–2 h each month for twelve months. Practice facilitators documented their activities for each practice encounter during H3 in FACITS, a web-based data collection platform developed specifically for rapid collection of practice facilitation focused study data. All practice facilitators received standardized training on activity documentation, and the research team conducted periodic quality checks; these efforts reduced recall bias and variations in the interpretation and documentation of activities. Details of the FACITS were described in prior publications [11, 22].

Table 2.

Standardized coefficients of the associations between latent factors and implementation strategies for completed interventions

Relationship	Standardized coefficient estimate	Standard error	p-value
Doing Tasks <—Intervention Characteristics	0.81	0.11	< 0.001
Doing Tasks <—Inner Setting	0.12	0.06	0.24
Doing Tasks <—Outer Setting	0.65	0.12	0.04
Doing Tasks <—Characteristics of Individuals	0.22	0.08	0.08
Doing Tasks <—Process	0.45	0.34	0.11
Project Management <—Intervention Characteristics	0.72	0.08	0.32
Project Management <—Inner Setting	0.49	0.13	0.03
Project Management <—Outer Setting	0.23	0.09	0.09
Project Management <—Characteristics of Individuals	0.48	0.11	0.02
Project Management <—Process	0.68	0.12	0.02
Consulting <—Intervention Characteristics	0.54	0.17	0.08
Consulting <—Inner Setting	0.25	0.21	0.07
Consulting <—Outer Setting	0.12	0.05	0.33
Consulting <—Characteristics of Individuals	0.56	0.25	< 0.001
Consulting <—Process	0.22	0.20	0.60
Training <—Intervention Characteristics	0.48	0.14	0.26
Training <—Inner Setting	0.62	0.10	0.04
Training <—Outer Setting	0.28	0.08	0.03
Training <—Characteristics of Individuals	0.11	0.02	0.09
Training <—Process	0.28	0.11	0.03
Coaching <—Intervention Characteristics	0.69	0.07	0.01
Coaching <—Inner Setting	0.24	0.11	0.01
Coaching <—Outer Setting	0.36	0.09	0.06
Coaching <—Characteristics of Individuals	0.68	0.10	< 0.001
Coaching <—Process	0.32	0.11	0.02

Table 3.

Standardized coefficients of the associations between latent factors and implementation strategies for CPCQ change

Relationship	Standardized coefficient estimate	Standard error	p-value
Doing Tasks <—Intervention Characteristics	0.65	0.14	< 0.001
Doing Tasks <—Inner Setting	0.05	0.04	0.32
Doing Tasks <—Outer Setting	0.17	0.05	0.06
Doing Tasks <—Characteristics of Individuals	0.04	0.03	0.12
Doing Tasks <—Process	0.07	0.09	0.21
Project Management <—Intervention Characteristics	0.58	0.10	0.33
Project Management <—Inner Setting	0.20	0.08	0.01
Project Management <—Outer Setting	0.06	0.04	0.18
Project Management <—Characteristics of Individuals	0.10	0.03	< 0.001
Project Management <—Process	0.11	0.03	0.04
Consulting <—Intervention Characteristics	0.43	0.21	0.16
Consulting <—Inner Setting	0.10	0.13	0.01
Consulting <—Outer Setting	0.03	0.02	0.68
Consulting <—Characteristics of Individuals	0.11	0.08	< 0.001
Consulting <—Process	0.04	0.05	0.18
Training <—Intervention Characteristics	0.38	0.18	0.28
Training <—Inner Setting	0.25	0.06	0.01
Training <—Outer Setting	0.07	0.03	0.42
Training <—Characteristics of Individuals	0.02	0.01	0.08
Training <—Process	0.04	0.03	0.24
Coaching <—Intervention Characteristics	0.55	0.09	< 0.001
Coaching <—Inner Setting	0.10	0.07	0.01
Coaching <—Outer Setting	0.10	0.04	0.12
Coaching <—Characteristics of Individuals	0.14	0.03	< 0.001
Coaching <—Process	0.05	0.03	0.06

Study outcomes

Two outcomes are used for this study: (1) the number of completed interventions at 12 months and (2) the change in CPCQ score between the baseline and at 12 months. R version 4.0.5 (R Foundation, Vienna, Austria) was used for statistical analyses. A two-sided p-value < 0.05 was used to define statistical significance.

Completed intervention

Practice facilitators worked with practice primary contacts to complete a 35-item survey of different QI interventions to determine if the practice was currently active or planned to implement. Each item response employed a Likert scale coding procedure (incomplete through complete), and primary contacts completed the survey with assistance from the practice facilitator at baseline and 12 months. The practice facilitators reviewed progress with the primary contacts on their selected QI interventions regularly. Practice facilitators and practice staff convened discuss progress or practices completed in an email form, which was then reviewed by the facilitator. Practice facilitators were responsible for documenting interventions as “Complete” in FACITS if the practice had successfully implemented the QI intervention at the time of review.

Change Process Capability Questionnaire (CPCQ)

In H3, all practices were asked to respond to the CPCQ at the start of the intervention period, and at 12 and 18 months following. The CPCQ includes 14 items assessing the extent to which the practice has used specific QI strategies to improve cardiovascular preventive care and evaluating the practice’s resiliency/capacity for change [23]. This scale was developed by expert clinic implementers in an iterative modified Delphi process [24]. The scale has been previously validated in small practices and it is reliable in measuring practice use of QI strategies, and correlates well with change in practice and care quality outcomes [25]. The CPCQ “score” was computed as a sum of items rated from − 2 (strongly disagree) to 2 (strongly agree): the overall score of the 14 items ranged from − 28 to 28 [26]. Higher scores indicate greater use of QI strategies. Because CPCQ was treated as an outcome for this study, we excluded it from the feature list.

Methods

Overview of quantitative methods

This study employed a multi-stage quantitative analysis framework to investigate the complex relationship between practice contextual factors and practice facilitation strategies. The methodological approach consisted of several key stages: data collection and cleaning, feature engineering, feature selection, feature set optimization, and statistical modeling using Structural Equation Modeling (SEM). Each stage was carefully designed to extract meaningful insights from a complex, high-dimensional dataset, ensuring that the findings are both statistically robust and practically relevant. The quantitative methods were chosen to address the study’s primary objective of identifying the most effective practice facilitation strategies and understanding their influence on two outcomes.

Machine learning

The application of statistical learning and ML techniques in this study was motivated by the need to manage and analyze a dataset characterized by high dimensionality and complex interactions among variables. Traditional statistical methods, while useful, often struggle with such complexity, particularly when the goal is to uncover nuanced patterns and interactions that may not be immediately apparent. We employed principal component analysis (PCA), a statistical dimensionality reduction technique commonly used in machine learning pipelines for feature selection; followed by feature set optimization and Structural Equation Modeling (SEM), which is a powerful statistical technique used to explore and quantify causal relationships among variables. SEM was chosen for its ability to model hypothesized structural relationships involving multiple dependent and independent variables simultaneously, which was essential given the multifaceted nature of practice facilitation and its associations with the outcomes. These classical statistical methods were implemented within an ML framework characterized by: (1) data-driven feature discovery rather than a priori variable selection, (2) algorithmic optimization of model components, (3) integration of both data-driven and knowledge-driven insights, and (4) iterative model refinement.

Figure 1 demonstrates the pipeline of combining ML and IRLM in the application of health care. It begins with the data collection and cleaning, then moves to feature engineering, building models, deploying and implementation, and collecting feedback from stakeholders that can be used to improve the system. After generating the results, we worked with stakeholders in H3 program to interpret them and discuss any discrepancies. The following sections described the sequences of the applied quantitative methods and process.

Fig. 1.

The pipeline of combining machine learning and IRLM in the application of health care. Data cleaning is the process by which data are cleaned (e.g. deciding how to handle missing values) and labeled (e.g. using human annotators to provide ground truth). Feature engineering is the process of splitting, combining, and/or transforming large amount of data to identify meaningful features; in this study, we include both data- and knowledge- driven approaches to ensure the selected features are robust and meaningful. Model selection is the process of choosing the appropriate model to uncover the pattern from the data (e.g. leveraging structural equation modeling to perform multivariate statistical analysis and analyze structural relationships)

Data collection and cleaning

The methods employed in this study began with a comprehensive data collection and cleaning process, essential for ensuring the accuracy and reliability of subsequent analyses. Data were sourced from multiple platforms, including the Facilitation ACtivity and Intervention Tracking System (FACITS) and REDCap, which provided a rich dataset comprising practice characteristics, facilitation activities, and intervention outcomes. In H3, the practice facilitators collected practice facilitation activity and survey data; the research team conducted interviews that were used to identify contextual factors and build a ML model. The research team regularly check missing data, outliers, and data inconsistencies. We reached out to practices to handle the data relevant issues. For the survey data, we performed tests of inter-rater reliability, internal consistency, and test–retest reliability to ensure that the survey data is reliable and valid for the intended analysis.

Feature selection

Choosing informative, discriminating and independent features is a crucial element of effective ML algorithms. In H3, we collected a large volume high-dimensional dataset and we obtained a subset of variables from the original feature set by prioritization of criteria for the research question [27], and removing redundant and irrelevant features [28]. Good feature selection can improve the accuracy ML models, reduce computing time, and improve interpretation of results [29]. For this study, we selected features to help us identify and understand the most effective practice facilitation strategies for different practice types and configurations to support sustained practice change using the contextual factors from the study IRLM (see Fig. 2). Continuous variables (e.g., practice size, number of staff) were retained in their original form after assessing distributions. Categorical variables (e.g., FQHC status, urban/rural location) were coded as binary indicators. Practice facilitation activities documented in FACITS were aggregated by practice over the 12-month intervention period and categorized into five strategy types (Doing Tasks, Project Management, Consulting, Teaching, and Coaching) based on previously established mapping criteria. QI capabilities were assessed through the CPCQ scale. Intervention completion was quantified as the count of fully implemented interventions from the 35-item survey.

Fig. 2.

Implementation Research Logic Model for H3 Program. The specific Implementation Strategies are shown in Supplemental Table 1

We then used PCA, which consists of an orthogonal transformation to convert samples belonging to correlated variables into samples of linearly uncorrelated features, for feature selection [30]. PCA is a linear transformation of data to minimize the redundancy (measured through covariance) and maximize information (measured through variance). Each resulting principal component (PC) is a new variable that is a linear combination of the original variables and uncorrelated with any other PC. The rationale for this step was to enhance the interpretability and predictive power of the models by ensuring that the included features were both meaningful and aligned with the study’s conceptual framework.

Feature set optimization

To determine the optimal number of features for modeling, we performed parallel analysis (PA) [31] and Velicer’s Minimum Average Partial (MAP) test [32]. Since PA tends to underestimate the number of components and Velicer’s MAP test tends to overestimate, [33] we performed both tests. We implemented the following process pipeline: 1) PA and Velicer’s MAP to determine the number of components; 2) Maximum Likelihood robust extraction method (also called Satorra-Bentler method), which is recommended for non-normal distributed data [34]; 3) orthogonal (varimax) and oblique rotations (promax) to assess stability of the factor solution across rotation types; and 4) item reductions based upon item loadings (0.4 or higher on two or more factors or less than half the difference of factor loading with other factors) [35] that affect the Cronbach’s alpha reliabilities. Following feature reduction, we repeated the pipeline until the final solution was reached, in which scree plot examination showed an elbow at 5 components (Supplemental Fig. 2).

Knowledge-driven feature review

Because the features that were selected by ML may not be meaningful to the real world, we incorporated expert opinion of the feature list. To confirm the important features, we first conducted a focus group discussion with experienced practice facilitators from H3. Following the Participatory Impact Pathways Analysis format [36, 37], the participants agreed on a shared vision—in this case, improving practice facilitation. Then, the participants collectively discussed the results from the ML model and factors that they perceived as actively playing a role in improving practice facilitation. Using a visual representation on a large board, the attendees grouped these factors. Next, participants mapped how they perceived each factor to work within the H3 program. For example, we excluded irrelevant features such as patient demographics, physical location of data, and Completely Electronic EHR. Through continued dialogue during the focus group discussion on appropriateness, acceptability, and feasibility, participants identified key factors that impact practice facilitation. We finalized the feature lists and mapped the 5 components onto the domains of the Consolidated Framework for Implementation Research (CFIR) [38]. During the process, we observed that the features clustering together in each PC corresponded conceptually to CFIR domains. For example, PC 1 included features related to staffing levels (number of clinical staff, number of office staff), which aligned with “Characteristics of Individuals”. PC 2 included EHR reporting capabilities and QI infrastructure, corresponding to “Intervention Characteristics”. This convergence of data-driven patterns with established theoretical frameworks strengthened our confidence in the results and provided construct validity for our approach.

Statistical modeling

Structural equation modeling (SEM) is a powerful, multivariate technique found increasingly in scientific investigations to test and evaluate hypothesized multivariate structural relationships. SEM was used to investigate the relationships among the latent variables and encompasses two components: (a) a measurement model (essentially the confirmatory factor analysis (CFA)) and (b) a structural model with the results of path analysis [39]. Path analysis aimed to find the relationship among variables by creating a path diagram. The measurement model of SEM is the CFA and depicts the pattern of observed variables for those latent constructs in the hypothesized model. A major component of a CFA is the reliability test of the observed variables [40]. The structural model displays the interrelations among latent constructs and observable variables in the proposed model as a succession of structural equations—akin to running several regression equations. Path analysis was used to calculate the standardized coefficients between the latent factors and practice facilitation strategies, which were mapped into the SEM [41]. The root mean square error of approximation (RMSEA) was used to evaluate the model fit. RMSEA is adequately sensitive to model misspecification and has been commonly used as guidelines for interpretation that seem to yield appropriate conclusions about model quality [42]. The RMSEA also takes the model complexity into account as it also reflects the degree of freedom. If the RMSEA is < 0.05, which indicates a convergence fit to the analyzed data of the model. A range of 0.05 to 0.08 indicates a fit close to good [43].

Results

We analyzed the data, including practice survey, practice assessment, and data in the FACITS [11], from the primary care practices in H3 program. Of the 226 practices initially enrolled in H3, 126 practices (55.8%) were included in the final analysis. Practices were excluded for the following reasons: (1) 42 practices (18.6%) withdrew from the program before the 12-month assessment, (2) 31 practices (13.7%) did not complete the CPCQ at 12 months, (3) 18 practices (8.0%) did not provide complete intervention completion data, and (4) 9 practices (4.0%) had > 20% missing data on key contextual variables needed for the analysis. Excluded practices differed from included practices on several baseline characteristics. Excluded practices were more likely to be smaller (mean 4.2 vs. 6.8 clinicians, p = 0.03), less likely to be FQHC (12% vs. 23%, p = 0.04), and had lower baseline CPCQ scores (mean 3.2 vs. 3.7, p = 0.02). Our sample-to-variable ratio of is acceptable for SEM given our model characteristics, including strong communalities (factor loadings > 0.4) and moderate-to-large effect sizes [44, 45].

Figure 2 illustrates the IRLM for the H3 program. We identified determinants, implementation strategies, mechanisms, and outcomes for the H3 project according to the process described in Smith, et al. [4]. Our analysis operationalized the IRLM structure as follows: We used PCA to identify latent contextual factors from the original features collected in H3. These determinants were mapped onto CFIR domains, which constitute the “Determinants” component of the IRLM. Practice facilitation activities documented in FACITS represented “Implementation Strategies” in the IRLM framework. We then employed SEM to explicitly model and quantify the relationships between determinants and strategies (the “linking” function in IRLM terminology) and between strategies and our two outcomes (completed interventions and CPCQ change). This approach allowed us to move beyond simply cataloging determinants and strategies to understanding the pathways through which contextual factors influence strategy selection and effectiveness. Regarding the specific determinates of the 5 domains of CFIR, the Intervention Characteristics include a clinician or a staff person in the practice who can create CQM reports; using guidelines for CVD prevention; using guidelines for patient management; and using at least one registry (IVD, hypertension, high cholesterol, diabetes, prevention services, and high risk patients); Inner Setting includes practice size, FQHC, practice experienced major changes, location (urban/rural), and hospital/health system owned; Outer Setting include part of an accountable care organization (ACO), practice was in a network, and practice received addition payment on performance based on measurement; Characteristics of Individuals include number of clinical staff and number of office staff; Process include aspirin report, blood pressure report, smoking report, reporting QI measures at practice-level, using EHR to extract data, and routinely discussing clinical quality data. The practice facilitation strategies are designed for H3 program have been discussed with more details in a prior study [11], briefly, facilitation strategies are grouped based on the degree to which the facilitator is performing specific tasks for the practice or coaching the practice on QI work. Supplemental Table 1 shows the specific implementation strategies and the practice facilitation activities within each domain. The mechanisms include educational, implementation, and clinical or patient-level approaches to facilitate the strategies to improve both implementation and clinical outcomes.

When the two outcomes were evaluated, practices implemented a median (IQR) of 10 (0–17) interventions, ranging from 0 to 35 completed interventions. The mean CPCQ score for baseline was 3.70 (SD = 0.88), 12 months was 4.06 (SD = 0.06), and the difference between 12 months and baseline is 0.36 (95% CI: 0.17, 0.54; P value < 0.01). Supplemental Table 3 and Supplemental Fig. 1 presents the responses to the 14 CPCQ items. The CPCQ results demonstrated good sustainability of improvement and capacity for making changes in H3.

Table 1 illustrates the finalized features after the whole selection process and their factor loadings. We performed PCA on our dataset and identified 5 principal components that aligned with the 5 domains on the CFIR. We mapped the 5 principal components onto the 5 domains of these determinant domains: Intervention Characteristics, Inner Setting, Outer Setting, Characteristics of Individuals, and Process. Cronbach’s alphas of the five domains are 0.71, 0.82, 0.72, 0.89, 0.86, respectively.

Table 1.

Selected feature and factor loadings. Factor 1: Characteristics of Individuals; Factor 2: Intervention Characteristics; Factor 3: Outer Setting; Factor 4: Process; Factor 5: Inner Setting

Feature	Factor1	Factor2	Factor3	Factor4	Factor5
Practice Size					0.626
FQHC					0.692
Major change					0.411
Location (urban/rural)					0.726
Hospital/health system owned					0.820
Aspirin report		0.791
BP report		0.932
Smoking report		0.824
Using EHR to extract data		0.496
Report QI measures at practice-level		0.436
Routinely discuss clinical quality data		0.502
Addition payment on performance based on measurement			0.612
Part of an accountable care organization (ACO)			0.489
Network			0.408
QI reports by clinician				0.798
Use guidelines for CVD prevention				0.702
Use guidelines for patient management				0.698
Use of at least one registry				0.501
Number of clinical staff	0.981
Number of office staff	0.985

To understand the relationship between the determinants and the practice facilitation strategies with regards the number of completed intervention outcomes, we performed structural equation model (SEM) analysis. Figure 3 shows the SEM results for the completed intervention outcome. The association between the 5 implementation strategies and completed intervention were 0.72, 0.42, 0.68, 0.76, 0.79, respectively. All 5 implementation strategies are associated with the completed intervention with statistical significance. The RMSEA for the SEM is 0.032, which means the model has a good fit [46].

Fig. 3.

Results of the structural equation model for the completed interventions. Cronbach's α is a measure of internal consistency, which illustrates how closely related a set of items are as a group. The values of 0.7 or higher indicate acceptable internal consistency. ^*p < 0.05

We also performed the similar SEM analysis to understand the relationship between the determinants and the practice facilitation strategies with regards the CPCQ. Figure 4 shows the SEM results for the CPCQ outcome. The RMSEA for the SEM is 0.054, which means a moderate fit [46]. Only Consulting (β = 0.36), Training (β = 0.48), and Coaching (β = 0.45) are associated with the CPCQ change with statistical significance.

Fig. 4.

Results of the structural equation model for CPCQ change. ^*p < 0.05

In the SEM, each of the 5 principal components represent a practice context determinant (latent factor) in the model. The SEM generates the standardized coefficient, which represents the association between the variables; the coefficient estimates are significant at p < 0.05. Table 2 shows the association between the practice context determinants and implementation strategies for completed intervention. Practice context factors like Intervention Characteristics (β = 0.81) and Outer Setting (β = 0.65) have statistically significant associations with Doing Tasks. Inner Setting (β = 0.49), Characteristics of Individuals (β = 0.48), and Process (β = 0.68) have statistically significant associations with Project Management. Characteristics of Individuals (β = 0.48) has statistically significant associations with Consulting. Inner Setting (β = 0.62) and Process (β = 0.28) have statistically significant associations with Training. Intervention Characteristics (β = 0.69), Inner Setting (β = 0.24), Characteristics of Individuals (β = 0.68), and Process (β = 0.32) have statistically significant associations with Coaching. Table 3 shows the association between the practice context determinants and implementation strategies for CPCQ change. The results show similar pattern to the completed intervention outcome, which means practices that completed more intervention were more likely to improve the CPCQ scores. Practice context factors like Intervention Characteristics (β = 0.65) has statistically significant associations with Doing Tasks. Inner Setting (β = 0.20), Characteristics of Individuals (β = 0.10), and Process (β = 0.11) have statistically significant associations with Project Management. Inner Setting (β = 0.10) and Characteristics of Individuals (β = 0.11) have statistically significant associations with Consulting. Inner Setting (β = 0.25) has statistically significant associations with Training. Intervention Characteristics (β = 0.55), Inner Setting (β = 0.10), Characteristics of Individuals (β = 0.14) have statistically significant associations with Coaching.

Discussion

The study illustrates the novel use of intersecting implementation science (IRLM), informatics (creating the data structures to compare data across projects), and data science (ML technologies) methods to interpret the relationships between practice context, practice facilitation strategies and study outcomes. We used ML to manage the high-dimensionality and diverse sources of data in the H3 study, uncover patterns in the contextual factors, and discover the relationships that can help us better understand the process of practice facilitation. Coupled with human knowledge, we selected and refined key features as practice contextual determinants.

The integration of ML with the IRLM proved to be a robust approach for analyzing the complex and multi-dimensional data collected in H3. This methodology allowed us to uncover nuanced relationships between practice context determinants, implementation strategies, and outcomes—insights that traditional statistical methods might have overlooked due to their reliance on linear assumptions and limited capacity to manage high-dimensional data [47]. Integration of implementation science methodologies into primary care practice change initiatives can help improve generalizability of evidence-based interventions when coupled with frameworks that support alignment of complex variable sets [48]. As an example of quality improvement and implementation research, the H3 study provides a rich source and large volume of data from over 200 practices and 15 practice facilitators, which provided the opportunity to test the methodology paradigm. The core purpose of the IRLM is to specify the relationships between determinants, strategies, and outcomes in an implementation project to create an explicit model of the relationship of practice and study characteristics to mechanisms of impactful strategies and outcomes. Understanding this chain of relationships will help support the selection of implementation strategies for future studies and identify opportunities to ask new questions within implementation research. However, the IRLM alone may not provide sufficient guidance on understanding the relationship between determinants, and the relationship between determinants and implementation strategies (a process often referred to as linking). It is essential to develop modeling strategies that can leverage the IRLM’s underlying generalized theory of implementation research [4] and other theoretical models such as theory of planned behavior change [49] and systems change theories [50] to help better understand complex implementation contexts that impact project outcomes [51].

ML leverages well understood statistical modeling methodologies and presents an important strategy for elucidating key patterns and relationships in large, complex, multi-dimensional datasets such as those created through health care quality improvement implementation studies [52]. Combining ML and the IRLM supports the creation of informatics-driven pipelines based on systems thinking that supports a better understanding of how each element in the system interacts and impacts each other, thus helping to identify successful intervention strategies and for specific practice environments. Last, we also used ML to assess the association among determinants, implementation strategies, and outcomes. This study demonstrates that it is possible to use ML in the context of the IRLM to develop interpretable models from implementation research data. This study also extends the use of the IRLM for modeling by adding important computational process to address the problem of how to combine existing knowledge and ML to facilitate the intersection of implementation science and informatics research. That is, a research team could use the IRLM to plan based on theory and prior data and then apply the computational methods with the real data from the study to check the hypothesized relationships.

Our methodological approach which illuminated significant findings regarding practice facilitation, also underscores the need for ongoing refinement in how we communicate these insights within implementation science, ensuring that our methods not only yield robust data but also translate effectively into practical applications. We found that practice facilitation strategies had more impact on the implementation of interventions than CPCQ change. The results of the SEM showed that all the proposed strategies were statistically associated with completed intervention; CPCQ showed a similar pattern but Doing Tasks and Project Management strategies did not have a statistically significant association. We think these two strategies in H3 were more directly related to intervention implementation rather than helping the practice to improve organization capacity. When examining the association between the practice context determinants and implementation strategies, we found a similar pattern for the two outcomes, which means practices that completed more interventions were more likely to improve CPCQ scores. For both outcomes, Intervention Characteristics have statistically significant associations with Doing Tasks; Inner Setting, Characteristics of Individuals, and Process have statistically significant associations with Project Management; Characteristics of Individuals has statistically significant associations with Consulting; Inner Setting has statistically significant associations with Training; Intervention Characteristics, Inner Setting, Characteristics of Individuals have statistically significant associations with Coaching. These results align with prior EvidenceNOW findings that practices with the ability to extract and use EHR clinical quality data had higher mean CPCQ scores [26]. In our study, extracting and using EHR clinical quality data were features under Intervention Characteristics, which had statistically significant associations with Doing Tasks and Coaching; these two strategies were statistically associated with the CPCQ outcome. Even accounting for the expected confounders of the complex primary care practices, our data suggest that coaching has been an effective strategy for implementing targeted interventions and achieving positively meaningful change. Our results demonstrate the relationship between practice determinants and facilitation strategies in the H3 study and may be a useful example to understand the relationship between the determinants and strategies and outcomes in primary care QI programs. This informatics-driven implementation approach may help QI implementation teams adapt QI work more effectively to their specific primary care settings.

Prior studies have shown that coaching strategies encourage: (1) expansive, multi-directional, attentive styles of communication, (2) practical problem solving, (3) facilitative leadership, and (4) an expanded vision of care [53]. The results of this study suggest that coaching can be an important strategy to achieve transformative change at both intervention implementation and organizational improvement. Coaching appears crucial as it goes beyond connecting practices to resources and implementing new technological components [54]. In the H3 study, practice facilitators provided coaching activities including holding frequent practice- wide, team-based meetings through intensive, hands-on style coaching activities to build rapport and encourage practice staff’s self-belief in their own ability to manage challenges through brainstorming, teamwork, and resourcefulness [55]. We found that Doing Tasks and Coaching showed an almost opposite pattern of relationships with contextual factors and was associated with the Outer Setting of the practice and Intervention Characteristics. Doing Tasks, which is primarily focused on “just helping out”, is important for intervention implementation but had little impact on change capacity. Consultation, defined as the transfer of specific information (such as assisting in the incorporation of target components and providing resources or data), Project Management and intervention-focused Training are directed at the practice as a whole, while Coaching is directed at individuals or groups of individuals within it and focuses on personal skill development that goes beyond connecting practices to resources and implementing new technological components. Effective coaching strategies provide expansive, multi-directional, and attentive styles of communication in the context of shared problem-solving activities between practice facilitators and practice staff while implementing the study interventions. A prior case study showed that the coached practice staff identified their practice facilitator as the key figure who helped their practices make the transition to a new style of care while overcoming the inevitable roadblocks that emerged [55]. This suggests that an external practice facilitator utilizing a coaching approach can be an effective strategy for helping primary care practices adapt to change.

We also found that practice facilitation strategies had more impact on the implementation of interventions than CPCQ change. It is possible that the 12-month exposure to practice facilitation, for which most practices averaged about 6 h of direct time with a practice facilitator during the H3 study intervention period was too short to show substantial organizational change [11]. Consulting, Training, and Coaching are associated with the CPCQ change with statistical significance. We found that these three strategies were likely driven by Inner Setting, Characteristics of Individuals, and Intervention Characteristics. Providing support or optimizing these contextual factors may be beneficial to these practice facilitation strategies, thus improving CPCQ.

Limitations

This study has several limitations. First, while SEM provides a framework for modeling theoretical relationships among variables, our cross-sectional observational design limits our ability to make definitive causal claims. The associations we identified are consistent with our theoretical model but could also reflect unmeasured confounding or reverse causation. Second, the patterns of association we observed emerged from small and medium-sized primary care practices in the Midwestern United States participating in a quality improvement initiative. These practices operated within particular healthcare delivery and payment systems, including varying levels of participation in ACOs and performance-based payment programs. Excluded practices differed from included practices on baseline characteristics including practice size and FQHC status, which may limit generalizability to other practice settings, regions, or healthcare contexts. Third, ML algorithms can sometimes uncover spurious correlations due to chance, which raises questions about the robustness of the identified patterns. To address this, we plan to apply our methods to other implementation research studies to validate our results. Despite these limitations, the proposed methods and results may provide valuable insights into effective practice facilitation strategies for different practice contexts and configurations to support sustained practice change.

Conclusion

The promise of implementation science lies in the ability to conduct rigorous and reproducible research, to clearly understand the findings, and synthesize findings from which generalizable conclusions can be drawn and actionable recommendations for practice change emerge. ML can be used to understand the complex interrelationships among the determinants of implementation, strategies, and outcomes in the context of the IRLM. Our data suggest that coaching was an effective strategy for implementing targeted interventions and achieving meaningful change in the H3 study, and that effective coaching strategies were associated with inner context practice variables. The results demonstrate the relationship between practice determinants and facilitation strategies and may be a useful example to understand the relationship between determinants, strategies, and outcomes in primary care QI programs. Our results also present a modeling strategy and functional example for undertaking this complex but critical task for the field.

Authors’ contributions

JY and TW conceived of the idea for the study. JY performed all the analyses and wrote the draft of the manuscript. JY, JB, AK, JS, and TW contributed to the interpretation of the results and revised the article for important intellectual content. All authors approved the final version of the manuscript.

Funding

This research was funded by grant 1R18HS023921 from the Agency for Healthcare Research and Quality. Drs. Walunas, Kho, and Smith received support from grant 5UG3HL154297 from the National Heart, Lung, and Blood Institute of the U.S. National Institutes of Health. Dr. Smith also received support from the National Center for Advancing Translational Science of the National Institutes of Health under Award Number UM1TR004409.

Data availability

All data generated or analyzed during this study are included in this published article and its supplementary information files.

Declarations

Ethics approval and consent to participate

The study protocol was reviewed and approved by Northwestern University’s Institutional Review Board.

Consent for publication

Not applicable.

Competing interests

Dr. Walunas receives research funding from Gilead Sciences. Dr. Kho is a strategic advisor for Datavant. Dr. Smith is co-developer of the IRLM and is paid as a consultant in its use by academic research teams and community-based implementers. Dr. Smith is also Associate Editor of Implementation Science. Other authors do not have conflicts of interest.