A Framework and Method for Measuring the Implementation of Data Science in Critical Care

Abstract

BACKGROUND:

The implementation of data science concepts, skills, and tools in critical care research and practice faces multiple, complex barriers.

METHODS:

We developed an implementation science-based framework and method for measuring the adoption, implementation, and sustainment of data science concepts, skills, and tools in critical care—the Society of Critical Care Medicine (SCCM) Discovery Data Science Campaign (DSC) Implementation Research Logic Model (IRLM). Our IRLM specifies constructs for: 1) key determinants (i.e., barriers and facilitators) influencing the implementation of data science concepts, skills, and tools in critical care; 2) implementation strategies deployed by the SCCM Discovery DSC to address these determinants; 3) theorized mechanisms of action by which these strategies affect outcomes; and 4) upstream and downstream implementation outcomes influenced by implementation strategies.

RESULTS AND CONCLUSIONS:

We believe that our model can facilitate more rigorous measurement of theoretically grounded, empirically assessable factors driving implementation of data science concepts, skills, and tools in critical care.

KEY POINTS

Question: How can we measure implementation of data science concepts, skills, and tools in critical care?

Findings: An Implementation Research Logic Model is a useful framework and method for facilitating measurement of implementation determinants, strategies, mechanisms, and outcomes.

Meaning: Better measurement of processes driving the adoption, implementation, and sustainment of data science concepts, skills, and tools in critical care can improve outcomes for researchers, administrators, clinicians, and ultimately patients and families.

A researcher aims to examine the relationships between diagnostic and procedure codes and clinical outcomes across diverse ICU settings during a rapidly evolving pandemic. A healthcare administrator seeks to enhance the accuracy of sepsis diagnosis and treatment within a hospital system by implementing innovative, disease-specific predictive models. Meanwhile, a frontline clinician plans to leverage a cutting-edge, machine learning-enabled clinical decision support system to inform patient care.

For such researchers, administrators, and clinicians, data science concepts, skills, and tools provide powerful analytic resources and capabilities. Examples of the paradigmatic research and practice shift afforded by data science’s advent include: the extraction of real-world data from electronic medical records as part of embedded, pragmatic clinical trials; aggregation of patient data in quality improvement initiatives to create real-time learning healthcare systems; and, integration and analysis of patient data across multiple “omics” (e.g., genomics, environmental exposomics) profiles to enable revolutionary precision health (1–4). Given that the ICU produces a larger volume, velocity, and variety of data—that is, the “3 Vs” of Big Data (5)—than any other healthcare setting, critical care is arguably positioned in the vanguard of this monumental health care advancement.

Yet despite these advances, the adoption, implementation, and sustainment of data science in critical care research and practice face multiple, complex barriers. These barriers are wide in scope, ranging from high variability in local data vocabularies, to incompatible information technology infrastructures, to an overall lack of expertise in the application of data science methods in intensive care clinical practice. Recognizing these challenges, Society of Critical Care Medicine (SCCM) Discovery, the Critical Care Research Network, launched its Data Science Campaign (DSC) in 2022. The mission of the DSC is to improve the care of critically ill patients by leveraging the use of Big Data for research capabilities, with the ultimate goal of data science application in clinical environments. The DSC’s implementation strategies comprise three main domains: 1) Establish guidance for large-scale data harmonization and data sharing (6–8); 2) Develop a data hub at SCCM; and 3) Conduct SCCM-sponsored datathons.

In this article, we present on our development of an Implementation Research Logic Model (IRLM) (9) that specifies: 1) key determinants influencing the implementation of data science concepts, skills, and tools in critical care; 2) specific implementation strategies deployed by the DSC to address these determinants; 3) theorized mechanisms of action by which these strategies affect implementation outcomes; and 4) upstream and downstream outcomes influenced by DSC implementation strategies. Our goal is to enable a more rigorous, theory-driven approach to measuring the factors that drive successful implementation of data science concepts, skills, and tools in critical care (10). The previously described researcher, healthcare administrator, and clinician could, in principle, use this model to systematically evaluate the key barriers, strategies, mechanisms, and outcomes involved in applying data science concepts, skills, and tools to critical care research and practice.

METHODS

Our methods leverage the IRLM to synthesize the multiple complex factors inherent in innovation implementation (9). The IRLM, a frequently used tool in implementation science, frames a four-part logic model comprised of implementation determinants, strategies, mechanisms, and outcomes. Importantly, the model is temporally agnostic, which means it can be developed and applied at any stage of the implementation process (i.e., pre-, during, and/or post), for planning, evaluation, and/or reporting. We iteratively developed a logic model throughout the course of the campaign using a combination of academic literature review, analysis of program documents (e.g., DSC business plan), and workshops and meetings among this article’s co-authors. Our project was reviewed and determined to be exempt from institutional review board (IRB) review because it did not meet the definition of research (WCG IRB, No. 1-1670047-1, June 14, 2023, “Discovery Data Science Campaign Implementation Science Logic Model”).

To begin, we specified “The Thing” being implemented, which is typically defined as an intervention, practice, or innovation (11). In the case of the DSC, we define “The Thing” as: the use of data science concepts, skills, and tools in critical care for research capabilities, with the ultimate goal of application in clinical environments. For short, we describe “The Thing” being implemented as the “use of data science in critical care.”

Determinants

Next, we populated the logic model’s first column, identifying the determinants influencing the use of data science in critical care. Determinants, often conceptualized as “barriers and facilitators,” are contextual factors that function as either independent variables or moderators to influence implementation processes and outcomes (12, 13). Several systematic literature reviews and empirical studies have identified implementation determinants for the use of data science across social sectors and within healthcare and critical care specifically (14–18). We derived the determinants in our logic model from Reddy’s “synthesized factors associated with ‘Data Science Strategy’ in organizations,” which in our appraisal achieves the highest level of comprehensiveness, relevance, applicability, simplicity, logic, clarity, usability, suitability, and usefulness of existing frameworks (19). We extracted each of this model’s determinants, mapped them to constructs from the (Updated) Consolidated Framework for Implementation Research (CFIR)—the most frequently cited implementation determinants framework in implementation science (20)—and tabulated all constructs (20). Guided by Reddy’s synthesized factors and the corresponding CFIR constructs, we developed a list of relevant areas for determinant measurement in critical care (Table 1).

TABLE 1.

Determinants of the Use of Data Science in Critical Care

Reddy’s Synthesized Factors	(Updated) Consolidated Framework for Implementation Research Constructs	Relevant Areas for Determinant Measurement in Critical Care
Data characteristics: volume, velocity, and variety of data; data quality	Innovation complexity	Nature and number of connections and steps involved in real-time analysis of granular data to inform changing clinical conditions in life-or-death treatment scenarios
		Quality and level of integration of disparate data sources, especially: lifespan data (e.g., pedigree); systems data (e.g., environmental exposome)
Technology: Infrastructure (hardware and software)	IT infrastructure	Ability to port, access, merge, and analyze critical care clinical (and other) data sources, such as from ICU monitoring systems
Organizational structure/model; organizational agility: business process, realignment of work practices, intensity of learning; ambidexterity	Work infrastructure	Organizational architecture and staffing to optimize dynamic, agile learning processes, characterized in particular by:
		Flexibility to (re)create roles, teams, and administrative procedures to catalyze digital innovation
		Standardized training to drive organizational learning
Data-driven decisionsInnovativeSustainable	Culture	Support for data-driven decision-making, and especially for open-source common data models (such as OMOP and FHIR)
		Willingness to enact and sustain IT infrastructure necessary for data science implementation
Alignment with core strategy: distinct data strategy, (coordination of) stakeholder interests	Mission alignment	Presence of measurable goals and objectives related to the integration of data science into critical care research and practice
		Resources to support novel data science initiatives amid multiple institutional priorities, in particular through strategic integration with existing priority initiatives (e.g., grants, quality registries, industry collaborations, etc)
Policies and regulations	Policies and laws	Data security and privacy policies to protect patient and organizational data
Governance: control (access), risk management, compliance, privacy, security, sharing, ownership		Data governance processes with respect to the context of data reuse, accuracy, archival, curation, platforms, architecture, and effective sharing within and across ICUs and organizations
Business environment: competitive dynamics, industry structure, partner readiness, consumers (public)	External pressure	Urgency to operationalize real-world data to produce clinical knowledge (e.g., on COVID-19)
		Market pressures influencing availability of resources for integration of critical care/data science
		Incentive to align public and private strategic partnerships (e.g., electronic medical record/electronic health record industry partners) to facilitate alignment of proprietary data
Managerial willingness: top management support, trust and acceptance, breaking silos	High-/mid-level leaders	Buy-in from key administrative decision-makers about the acceptability, appropriateness, and feasibility of data science
		Human-machine teaming trust
Talent (knowledge and skillset)	Implementation facilitators (SMEs)	Synergistic clinical and informatics expertise related to collecting, analyzing, and operationalizing large datasets to improve clinical practice and facilitate clinical trials

FHIR = Fast Healthcare Interoperability Resources; IT = information technology; OMOP = Observational Medical Outcomes Partnership; SMEs = subject matter experts.

Implementation Strategies

The second column of the logic model was then completed, detailing the DSC’s three core implementation strategies. Broadly speaking, implementation strategies may be defined as methods for increasing adoption, implementation, and sustainment of an intervention, innovation, or practice (21). In other words, they are the “stuff we do to try to help people/places do The Thing” (11). As with data science implementation determinants, previous literature reviews and empirical studies have named and defined implementation strategies related to data harmonization and sharing, use of data registries/warehouses/platforms, and datathons, within the health care sector and critical care specifically (2, 22–32). We mapped the DSC’s three domains onto the Expert Recommendations for Implementing Change (ERIC) compilation of implementation strategies (21)—the most frequently cited implementation strategy taxonomy—and tabulated implementation strategy names and definitions, excerpted from the ERIC compilation and subsequent refinements (21, 33) (Table 2). For each domain, we further specified implementation strategies’ actors, actions, temporalities, doses, and justifications (34).

TABLE 2.

Implementation Strategies for the Use of Data Science in Critical Care

Data Science Campaign Domain	(Refined) ERIC Implementation Strategies	(Refined) ERIC Implementation Strategy Definitions	Actors, Actions, Temporalities, Doses, and Justifications
Data harmonization and sharing	Develop resource-sharing agreements	“Develop partnerships with organizations that have resources needed to implement the innovation”	Actors: Clinicians, researchers, administrators, data scientists
	Stage implementation scale-up	“Phase implementation efforts by starting with small pilots or demonstration projects and gradually move to a system wide rollout”
			Actions: Establish principles for data standardization and sharing, develop CDEs for critical care, and perform a pilot project utilizing standardized CDEs
			Temporality: CDEs modified and expanded regularly
			Dose: Pilot: 2 yr
			Justification: Fragmentation of data elements (especially in United States) substantially impedes research and quality improvement efforts
Discovery Critical Care Datahub	Use data warehousing techniques	“Integrate clinical records across facilities and organizations to facilitate implementation across systems”	Actors: Data scientists
	Centralize technical assistance	“Develop and use a centralized system to deliver technical assistance focused on implementation issues”	Actions: Aggregate electronic medical record data on ICU admissions from across sites to develop a cloud-based multipurpose platform for analysis of both clinical and nonclinical information. Enable federated data approaches (i.e., noncentralized data locations usable as a unified whole)
			Temporality: Once developed, always available
			Dose: Continuously maintained datahub, with multiple pathways for further data ingestion over time
			Justification: Current process of manual data extraction and transformation across sites requires unsustainable personnel effort requirements
Society of Critical Care Medicine-sponsored datathons	Use data experts	“Involve, hire, and/or consult experts to acquire, structure, manage, report, and use data generated by implementation efforts”	Actors: Clinicians, researchers, data scientists
	Increase demand	“Attempt to influence the market for the clinical innovation to increase competition intensity and to increase the maturity of the market for the clinical innovation”
			Actions: Host recurring competitions focused on using data science techniques to solve real-world issues
			Temporality: Annually
			Dose: Two-d workshop
			Justification: Pressing need for expedited application of data science in critical care research and practice, especially in emergency situations (e.g., natural disasters, emerging epidemics/pandemics)

CDE = common data element, ERIC = Expert Recommendations for Implementing Change.

Mechanisms

The third column of the logic model outlines the theorized mechanisms through which our implementation strategies influence outcomes. Implementation mechanisms are processes or events that operate as mediators between implementation strategies and outcomes (10, 35–38). They are often conceptualized via theory-driven, “realist” causal explanations for how and why implementation strategies influence (or fail to influence) context-specific outcomes (10, 35–38). Multiple literature reviews and empirical studies have named and defined mechanisms related to data science implementation across sectors and within health care and critical care (39–48). The mechanisms column of our logic model derives from Galetsi’s “path to value Big Data Analytics in Healthcare” (49, 50). In brief, Galetsi’s pathway may be summarized as: the assemblage of Big Data Analytics Resources (BDAR) to build Big Data Analytics Capabilities (BDAC) that create health care value. Galetsi’s pathway satisfies multiple criteria for establishment of mechanisms, including strong association, specificity, consistency, experimental manipulation, timeline, gradient, and plausibility or coherence (35, 51). We tabulated all constructs in Galetsi’s framework and developed relevant examples pertaining to data science in critical care (Table 3).

TABLE 3.

Mechanisms of Implementation for the Use of Data Science in Critical Care

Mechanisms	Relevant Examples in Critical Care
Big Data Analytics Resources (per Groves et al [52], Waller and Fawcett [53], Chen and Zhang [54] as cited in Galetsi et al [49, 50])
Data types	Clinical data in the electronic medical record for hospitalization or pre-/post-ICU care; clinical device data (e.g., waveforms, IV pumps, ventilators, dialysis machines); questionnaires (e.g., family satisfaction, patient-reported outcomes); diagnostic and procedure codes; drug utilization and wastage data
Analytical resources	Society for Simulation in Healthcare; PhysioNet (e.g., Medical Information Mart for Intensive Care IV, eICU); National Clinical Cohort Collaborative
	Phillips eICU Research Collaborative Database; Amsterdam University Medical Centers Database
Big Data Analytics Capabilities, per Groves et al (52) as cited in Galetsi et al (49, 50)
Monitoring, prediction/simulation, data mining, evaluation, and reporting	Monitoring of high-fidelity waveforms and processed data from bedside monitors
	Prediction models: Disease-based (e.g., sepsis) (55); intervention-based (e.g., hemodynamic monitoring); prediction of changes in stability and acuity (56, 57)
	Mining of ICU data lakes (i.e., centralized repositories for storage of large amounts of raw data)
	Integrated evaluation and reporting of complex data streams and thresholds
Created values
Better diagnosis for provision of more personalized health care	Rapid diagnosis, predictive and prognostic enrichment (e.g., Covid-19) (58, 59)
Supporting/replacing professionals’ decision-making with automated algorithms	Machine learning-enabled clinical decision support (60)
New business models, products, and services	Prediction models
Enabling experimentation, expose variability, and improve performance	Synthetic datasets modeling and experimentation where ethical considerations would not allow randomized trials
Healthcare information sharing and coordination	Mapping of clinical data to common data models (25)
Creating data transparency	Data cleaning and post-processing systems
Identifying patient care risk	ICU scoring systems (e.g., SOFA, pediatric SOFA) (61–63)
	Early warning systems (64)
	Prediction of ICU readmissions (65)
Offering customized actions by segmenting populations	Population enrichment for interventional trials
Reducing expenditure while maintaining quality	Improved efficiency in treatment of sepsis
Protecting privacy	Cyber security and privacy systems in the ICU

eICU = electronic ICU, SOFA = Sequential Organ Failure Assessment.

Outcomes

Finally, we completed the fourth column of the logic model, detailing the outcomes related to using data science in critical care. The overarching framework for these outcomes is the CFIR Outcomes Addendum, which is designed to be directly compatible with use of the CFIR’s categorization of determinants and conceptualizes outcomes in terms of three principle components (66). First, “antecedent assessments” are contextual phenomena that reliably predict implementation and innovation impact. Second, “implementation outcomes” are dimensions of implementation success or failure, whether anticipated (expected in future) or actual (occurring in present or occurred in past). Third, “innovation outcomes” are the equitable impacts of implementation on key decision-makers, deliverers, and recipients. We tabulated implementation outcome constructs and construct dimensions and selected measures applicable to data science in critical care (Table 4).

TABLE 4.

Implementation Outcomes for the Use of Data Science in Critical Care

CFIR Outcomes Addendum Implementation Outcome Domains	CFIR Outcomes Addendum Implementation Outcome Constructs	Suggested Measures
Antecedent assessments	Acceptability	Acceptability of intervention measure (67)
	Appropriateness	Intervention appropriateness measure (67)
	Feasibility	Feasibility of intervention measure (67)
	Implementation readiness	Various measures of implementation readiness (e.g., Texas Christian University Organizational Readiness for Change) (68) and implementation climate (e.g., adaptations of Klein and Sorra’s construct developed within the context of information technology implementation) (69)
	Implementation climate
Implementation outcomes	Anticipated and actual	Penetration (70) of Critical Care Data Dictionary common data elements
	Adoptability and adoption	Penetration (70) and maintenance of Discovery Critical Care Datahub
	Implementability and implementation	Maintenance of annual Society of Critical Care Medicine datathons
	Sustainability and sustainment	Publication of peer-reviewed articles related to data science in critical care
Innovation outcomes	Equitable population impact on:
	Decision-makers (e.g., administrators)	Institute of Medicine service outcomes (70):
		Efficiency
		Safety
		Effectiveness
		Equity
		Patient-centeredness
		Timeliness
	Deliverers (e.g., researchers, data scientists, clinicians)	“Quadruple Aim” outcomes (71) pertaining to health care deliverers, including:
		Reduced burnout
		Improved work experience
	Recipients (e.g., patients)	Barr’s health/effectiveness outcomes (72):
		Family and surrogate well-being/quality of life
		Physical functioning
		Post-ICU trajectory
		Psychologic well-being
		Satisfaction

CFIR = Consolidated Framework for Implementation Research.

RESULTS AND CONCLUSIONS

Figure 1 presents a consolidated SCCM Discovery DSC IRLM for the use of data science concepts, skills, and tools in critical care research and practice. The model uses horizontal arrows to denote the left-to-right “flow” of the implementation “pipeline”—from determinants to strategies to mechanisms to outcomes—as well as vertical arrows to indicate progression within each of these “piping” components. We believe that our proposed SCCM Discovery DSC IRLM will facilitate more rigorous measurement of theoretically grounded, empirically assessable factors driving the adoption, implementation, and sustainment of data science in critical care (10).

Figure 1.

Society of Critical Care Medicine (SCCM) Discovery Data Science Campaign Implementation Research Logic Model for the use of data science concepts, skills, and tools in critical care research and practice. The model uses horizontal arrows to denote the left-to-right “flow” of the implementation “pipeline”—from determinants to strategies to mechanisms to outcomes—as well as vertical arrows to indicate progression within each of these “piping” components. BDA = Big Data Analytics, BDAC = Big Data Analytics Capabilities, SMEs = subject matter experts.

For example, the previously imagined researcher might use the logic model to comprehensively assess barriers and facilitators to extraction of diagnostic and procedure codes and clinical outcomes data across multiple differing ICUs. The healthcare administrator might use the model to select validated implementation outcomes—such as measures of acceptability, appropriateness, and feasibility—as part of planning integration of sepsis prediction models into quality improvement efforts. The frontline clinician intending to use machine learning-enabled clinical decision support can examine the data science implementation strategies and mechanisms driving this innovation. For researchers, healthcare administrators, and frontline clinicians interested in making further use of the model’s approach, the toolkit appended in this article’s supplemental digital content includes additional guidance on how to create a custom IRLM for a new investigation or new adoption/application in a clinical setting. With this framework and method in hand, we believe our model will enable the critical care data science community to better unite and synthesize knowledge about how to accelerate implementation of data science concepts, skills, and tools in the ICU.

We acknowledge that our framework and method has limitations. First, while we aimed to narrowly define “The Thing” being innovated as the use of data science concepts, skills, and tools, we recognize that the literature often conceptualizes this “Thing” ambiguously—for example, as “Big Data,” data science, BDAR, and/or BDAC. We nevertheless maintain that our operationalization of the use of data science concepts, skills, and tools as a concept in the model coheres intelligibly. Second, the logic model does not specify whether a given determinant operates as a barrier or facilitator, given the high variability of barrier and facilitator valences across contexts. We note that Reddy’s model does formulate each factor as either a barrier or enabler, and that later research by Reddy et al (73) analyzes empirical evidence from across multiple organizations to identify key barriers for data science implementation and the hierarchical relationships between them. Future research might synthesize barriers and facilitators from across multiple literature reviews and frameworks, including Reddy’s and others cited in this article. Third, although comprehensive, the model inevitably does not list all possible specifications of determinants, strategies, mechanisms, and outcomes, nor does it map potential causal pathways between them. We believe the parsimonious selection of constructs in our model to be an indicator of our comprehensive thinking through and prioritization of relevant concepts rooted in the literature, and we encourage future mapping of causal pathway diagrams between specific components as part of “realist” implementation research. Fourth, it is important to note that our model may conflict with ascendant conceptualizations of causal pathway diagrams that place determinants “to the right” of implementation mechanisms, under the premise that determinants are factors that either constrain or enable implementation strategies’ effects (35, 36). We believe that our placement of determinants “to the left” of implementation strategies aligns more closely with current standard conceptualizations of the relationship between these concepts in implementation science as well as our own intuitive understandings. We invite future revisions to the model that conceive of the order of determinants, strategies, and mechanisms differently.

Future research should revise and expand on our model to account for theories and frameworks pertaining to large language models and agentic systems that have arisen since the conception of the SCCM Discovery DSC and initial creation of its implementation strategies. In particular, we highlight the theories and frameworks synthesized in Reddy’s implementation science-informed translational path on application, integration, and governance of generative artificial intelligence (AI), which in our estimation satisfy the stringent criteria with which our data science-specific theories and frameworks were evaluated for inclusion (74). Reddy’s components of AI implementation could be directly mapped onto our logic model: “acceptance and adoption” in the antecedent assessments and implementation outcomes; “data and resources” in the BDAR and BDAC mechanisms; and “technical integration” and “governance” in the implementation strategies. The extensibility of the IRLM in terms of its ability to continually incorporate updated theories and frameworks such as those related to AI is ultimately one of its principal strengths, given the complexity of implementation as new innovations arise over time.

Ultimately, we share with other critical care researchers, data scientists, and clinicians the goal of successfully leveraging data science in critical care to improve outcomes for the critically ill and injured. We present the SCCM Discovery DSC IRLM in service of that aim.