Use of the consolidated framework for implementation research to guide dissemination and implementation of new technologies in surgery

Abstract

Background

Improving surgical outcomes is important to patients, providers, and healthcare systems. Understanding best methods to ensure evidence based practices are successfully implemented and sustained in clinical practices leads to improved care. Dissemination and implementation (D&I) science facilitates the successful pathway from clinical trials to sustained implementation.

Methods

We describe D&I science, introduce the consolidated framework for implementation research (CFIR), a D&I framework, and provide an example of how CFIR was utilized to facilitate the translational process from design adaptations to implementation, broad utilization by clinicians, and sustainability of the SUrgical Risk Preoperative Assessment System (SURPAS) tool into regular clinical practice. SURPAS creates data-driven individualized risk assessments of common adverse postoperative outcomes to enhance the informed consent process, shared decision making, and consequently improved surgical outcomes. The CFIR provided a structured systematic way to identify constructs influencing the D&I of SURPAS, including adaptations for the process and tool.

Results

We identified three domains, each with specific constructs, that participants believed would strongly influence effectiveness of SURPAS implementation efforts: the importance of patients’ perspectives (outer setting); the quality of SURPAS (intervention characteristic); and integration of SURPAS into the electronic health record (inner setting). Additionally, providers’ positive attitudes toward and support of SURPAS (characteristics of individuals); and the ease of integration of SURPAS into the workflow (process), were also identified. Tension emerged between patients’ preference of the provision of risk information and providers’ concern about additional clinic time required for formal risk discussion with low-risk patients.

Conclusions

Systematically identifying constructs from the beginning of the design through the implementation process can guide design of a multi-component strategy for future large-scale implementation by assessing the relative impact of factors on implementation using the CFIR framework. In the example studied, this allows key stakeholders to ensure success of D&I of SURPAS at multiple levels and times, continuously optimizing the process.

Introduction

Improving surgical outcomes is important for patients, providers, and healthcare systems. To facilitate high quality outcomes, the utilization of evidence based interventions, practices, procedures, and policies (EBP) that are successfully implemented into various clinical environments is imperative. Yet, barriers can occur at various points in the implementation process, leading to less effective interventions and gaps in care. Multiple factors known to prevent successful adoption and implementation of an EBP include: resistance of providers; lack of belief or knowledge of the EBP; lack of skills to implement the EBP, and lack of organizational management support and resources (1). Therefore, understanding specific theories and processes to promote the uptake of EBPs into surgical care is imperative. Dissemination and implementation (D&I) science address these needs.

Implementation research consists of “scientific investigations that support movement of evidence-based, effective health care approaches from the clinical knowledge base into routine use” (2). Dissemination research, as defined by the National Institutes of Health, is the “targeted distribution of information and intervention materials to a specific public health or clinical practice audience” (2). Together, D&I is used to guide the process from initial design to implementation and dissemination of EBP, utilizing specific theories, models, frameworks, and processes to promote the uptake of EBP into surgical care.

Conceptual frameworks

Conceptual frameworks are utilized in D&I to increase the “generalizability and interpretability of research findings” (3). They are used to better understand and explain how and why the D&I of evidence based innovations succeed or fail, may guide the assessment of the process, and identify factors that might influence the implementation and effectiveness (3-5). Consolidated framework for implementation research (CFIR) is a pragmatic meta-theoretical framework that encompasses a repository of standardized D&I constructs (3,5). Damschroder et al. integrated the previously published implementation science theories to identify constructs that either conceptually or empirically influenced implementation and compiled a single, common language, consolidated framework with defined domains and constructs (3,5). CFIR has five major domains and 39 individual constructs within the domains.

Intervention characteristics address the characteristics of the intervention/innovation being implemented and their potential influence on implementation. The eight constructs within the domain address specific factors such as: source of the innovation; evidence strength and quality; relative advantage; adaptability of the innovation; trialability; complexity; design quality and packaging; and finally its costs (3,4).

Inner and outer settings include the social, political, and economic contexts though which the implementation process occurs. These domains are dependent on the context of the implementation process. For example, an EBP may be implemented within a hospital unit (inner setting) but implementation efforts may be affected by factors outside the hospital (outer setting). The outer setting has four constructs: patient needs and resources; cosmopolitanism—networked with other organizations; peer pressure—mimetic/competitive; external policy; and incentives. The inner setting has five broad constructs, and two of the broad constructs have sub-constructs. The constructs include: structural characteristics; networks and communications; culture; implementation climate (includes six sub-constructs); and readiness for implementation (includes three sub-constructs) (3,4).

Characteristics of individuals contain five constructs: knowledge and beliefs about the intervention; self-efficacy to execute the innovation; individual stage of change; individual identification with organization; and other personal attributes. Finally, the process domain has four broad constructs and one has sub-constructs: planning; executing; reflecting and evaluating; and engaging (includes five sub-constructs) (3,4).

CFIR was designed to be used across all phases of implementation to systematically identify factors that may influence D&I efforts (3,4). CFIR can be used to help design a roadmap or implementation protocols, by guiding data collection and analyses. In the pre-implementation phase, CFIR can be used to assess contextual capacity and needs, by providing defined constructs that identify potential barriers and facilitators to implementing an EBP in specific environments (5). During the implementation process CFIR can be used to monitor various processes and progress, leading into post-implementation where it can be used to determine which constructs influenced outcomes and effectiveness (5).

An example using CFIR to assess D&I of the SUrgical Risk Preoperative Assessment System (SURPAS)

We utilized CFIR to facilitate the translational process from design to implementation, broad utilization, and sustainability of the SURPAS tool. SURPAS creates data-driven individualized risk assessments of common adverse postoperative outcomes that may enhance the informed consent process, improve shared decision making, and lead to improved surgical outcomes.

Risk assessment in surgery is essential to guide patient-centered treatment decisions but is variable in practice (6,7). Presently, risk assessment of perioperative complications is based on accepted or previously reported values and influenced by subjective provider assessment of individual patient comorbidities (8,9). Formal risk assessment tools exist, such as the Veterans Health Administration’s Veterans Affairs Surgical Quality Improvement Program (VASQIP) risk calculator and the American College of Surgeons’ (ACS) National Surgical Quality Improvement Program (NSQIP) risk calculator. However, these, and other surgical risk assessment tools, are time intensive to use, may not provide clinically useful information, and are not regularly integrated into clinical workflow (8,9).

SURPAS aims to address these issues. It was developed from the ACS NSQIP database and the detailed design and statistical methodologies of SURPAS have been previously described (8,10,11). The SURPAS tool is user-friendly and is based on a more parsimonious set of eight easily accessible preoperative risk variables. It calculates accurate risk predictions of 11 meaningful adverse surgical outcomes across nine adult surgical specialties and is integrated into the electronic health record (EHR) (Figure 1) (8-11). SURPAS is utilized at the preoperative patient visit and is designed to improve clinical surgical outcomes, patient communication, and health literacy, while better informing the provider of patient-specific needs (8-11). A pictorial and numeric display of the results is printed out and provided to the patient (Figure 2).

Figure 1.

SURPAS screen for data input and risk display for the surgical team. This figure shows the EHR screen: on the left providers enter patients’ surgical procedure, age, functional health status, ASA class, planned hospital status following procedure, surgical specialty, and if it was an emergency procedure. On the right side of the screen are presented the results of the individual risk for the 11 clinically-meaningful adverse surgical outcomes. Additionally, the results are compared to the national risks in the ACS NSQIP database, shown by the blue line. SURPAS, SUrgical Risk Preoperative Assessment System; EHR, electronic health record; ACS, American College of Surgeons; NSQIP, National Surgical Quality Improvement Program.

Figure 2.

SURPAS’ individualized pictograph of the output for postoperative risk. The SURPAS individualized pictograph provides patients with their results for 30-day postoperative adverse outcomes: mortality; overall morbidity; infectious; urinary tract infection; transfusion; cardiac; renal; pulmonary; venous thromboembolic; and neurological complications. The document contains two pictographs for each outcome: (I) the first (on the left) represents their individualized risk for the postoperative outcome; and (II) the second (on the right) represents the ACS NSQIP national average for the outcome. SURPAS, SUrgical Risk Preoperative Assessment System; ACS, American College of Surgeons; NSQIP, National Surgical Quality Improvement Program.

In preparation for the broad D&I of SURPAS to all University of Colorado (UC) Hospital surgical clinics and to prepare for further spread of SURPAS to all six UCHealth System hospitals, we conducted a formative evaluation, “designed to identify potential and actual influences on the progress and effectiveness of implementation efforts. Formative Evaluation enables researchers to explicitly study the complexity of implementation projects and suggests ways to answer questions about context, adaptations, and response to change.” (12). Findings from the formative evaluation are guiding design of a multi-component strategy for future scale-up. In this formative evaluation, we used CFIR to guide the assessment of the process and identify factors that might influence SURPAS implementation and thus, its effectiveness (3-5).

The objective of this paper is to review the need for D&I, introduce one D&I framework, and provide an example of how it is used to facilitate the implementation and sustainability of an intervention into multiple clinical environments. In this article, we describe how we used CFIR to identify D&I factors that emerged from the formative evaluation. We also describe the identified CFIR domains and constructs to design and guide a pragmatic D&I protocol for further implementation and scaling-up of SURPAS.

Methods

The formative evaluation included two phases: contextual baseline assessment and trial implementation. The contextual baseline assessment was performed at the UC Hospital and utilized a qualitative methodology. Stakeholders were engaged through focus groups with surgical patients and providers, and individual interviews with senior administrators, which provided suggestions about improving SURPAS’ use and utility as a tool and provided guidance to facilitate the implementation such as: seeking early wins; identification of local champions; and performing iterative improvements. Detailed findings of the Contextual Baseline Assessment are published (13).

We then utilized these findings to guide the trial implementation to study the feasibility of the uptake of SURPAS and the contextual baseline assessment of five UC Hospital surgical clinics. These included thoracic, endocrine and general, orthopedic, vascular surgery, and urology. Specifically, we obtained opinions and personal experiences of using SURPAS, identified adaptations to SURPAS for improvement, assessed the context of the clinical environment for SURPAS, and were provided suggestions to improve the implementation process.

Data collection

Contextual baseline assessment consisted of focus groups and individual interviews with key stakeholders: surgical patients; surgical providers; and hospital administrators. These participants were shown a presentation of the SURPAS tool. Opinions, suggestions for adaptations to improve the SURPAS tool, and suggestions to facilitate the eventual implementation of SURPAS were elicited.

Patient recruitment included surgical patients who underwent an operation within the previous 12 months of the study start date at UC Hospital, were enrolled in NSQIP data collection, and lived within 30 miles of UC Hospital. Recruitment letters describing the study were mailed to 200 patients randomly selected from the above cohort, inviting them to contact us. Those who contacted us sharing interest in participation were scheduled for participation in a focus group. All patients who attended one of the original focus groups were invited to attend a follow-up focus group approximately one year later. All patient participants received $75 for their participation in each focus group.

A convenience sample of surgical provider and clinical researcher participants was recruited from the University of Colorado School of Medicine Department of Surgery’s Surgical Outcomes and Applied Research program monthly meeting. A follow-up focus group was held approximately one year later. Individual interviews were performed with administrative officials. Individual postcard informed consent was provided at the time of all focus groups and interviews.

During the Trial Implementation phase, surgical providers from the involved clinics, and their respective clinic administrators were recruited to participate in the study. Providers were asked to use the SURPAS tool with consented patients, complete a quantitative survey on each patient, and agree to be subsequently interviewed. The clinic administrators were individually interviewed on their opinions about SURPAS and its impact on work flow and patient clinical care.

Surgical patients of the participating providers, who attended a pre-operative clinic appointment, were invited to participate in the study. Recruited patients were consented and provided a copy of the signed informed consent. Participation included utilization of the SURPAS tool in their risk discussion during the preoperative clinical visit, completion of a quantitative survey after the visit, and possible observation during the clinic visit and interview afterward to obtain their opinions and experience with SURPAS.

The qualitative inquiry elicited opinions about SURPAS, suggestions for improvement, opinions for optimizing the implementation process based on experiences of implementation within the contextual climate, culture, and its proclivity for change. The 30–45 minute interviews and focus groups were conducted by Master- and PhD-educated members of the team trained in qualitative research, were audiotaped and transcribed verbatim. This study was approved by the Colorado Multiple Institutional Review Board.

Data analysis

CFIR coding

A matrix analysis qualitative approach was used to code, analyze, and organize the qualitative data (14,15). We used all of the 39 CFIR constructs and their definitions (4) as our a priori codebook to code qualitative segments that influenced effectiveness of the SURPAS tool, its processes and process adaptations, and barriers and facilitators to the D&I of SURPAS. Important domains and constructs were identified based on the frequency they were mentioned, the degree of importance articulated by the participants or researchers, or both. This iterative process involved three qualified analysts (AC Lambert-Kerzner, DM Aasen, DM Overbey). Each coded the transcribed text segments from interviews of individuals to specific CFIR constructs. We then compared coding and used a consensus process when disagreements occurred. Upon thematic saturation (not eliciting any new themes), the team concluded the analysis and identified the final domains, placing them into the matrix for comparative analysis.

The matrix allowed simultaneous comparison of a large volume of data and contrasting findings between the different groups: patients, providers, and administrators (14,15). These findings identified factors related to SURPAS characteristics, assessed the readiness of the clinical environments to implement SURPAS, assessed the culture of the environment, and identified barriers and facilitators to the implementation processes. All analyses and findings were integrated and documented with an audit trail (14-17). Illustrative quotes were selected by consensus of all members of the analytic team for each CFIR construct.

Results

The contextual baseline assessment consisted of interviews with three focus groups of 18 patients (Table 1); two focus groups with 13 surgical providers, one researcher, two medical students, two clinic administrators; and five individual interviews with administrative officials (Table 2). During the trial implementation, nine surgical providers and five clinic administrators agreed to participate after reviewing post-card consents. This resulted in recruitment of 199 patients enrolled through the participating surgical clinics.

Table 1. Demographics of patient participants.

Variable	Number [%]
Contextual baseline assessment
Patients	18
Age (years)
30–39	2 [11.1]
40–49	6 [33.3]
50–59	4 [22.2]
60–69	4 [22.2]
70+	2 [11.1]
Gender (N=18)
Female	13 [72.2]
Trial implementation
Patients interviewed	27
Age (years)
20–29	3 [11]
30–39	1 [3]
40–49	5 [19]
50–59	6 [22]
60–69	7 [26]
70+	5 [19]
Gender (N=27)
Female	14 [52]

Table 2. Demographics of providers.

Variable	Number of females [%]	Educational attainment (N)
Contextual baseline assessment
Clinic administrators (N=2)	N=1 [50]	MHA (N=1); BS (N=1)
Providers (N=13)
Surgeon (N=10)	N=3 [30]	MD (N=7); MD, PhD (N=1); MD, MHS (N=1); MD, MBA (N=1)
Anesthesiologist (N=2)	N=1 [50]	MD (N=2)
Internist (N=1)	N=1 [100]	MD, PhD (N=1)
Clinical researcher (N=1)	N=0 [0]	MS (N=1)
Medical student (N=2)	N=0 [0]	BS (N=2)
Administrative official (N=5)	N=1 [20]	MD (N=4); MBA (N=1)
Trial implementation
Clinic administrators (N=5)	N=4 [80]	BS (N=3); PT, MBA (N=1); MSN (N=1)
Providers (N=9)
Surgeon (N=7)	N=3 [43]	MD (N=5); MD, MPA (N=1); MD, MBA (N=1)
Nurse practitioners (N=2)	N=2 [100]	MSN (N=2)

We interviewed the seven surgeons and two nurse practitioners who implemented SURPAS, five clinic administrators (Table 2), as well as a convenience sample of 27 of the 199 patients (Table 1) who used SURPAS during their pre-surgical risk assessment.

Of the 31 constructs within the CFIR, factors affecting D&I of SURPAS were spread across three of five CFIR domains. Four CFIR constructs strongly influenced the effectiveness of the implementation of SURPAS based on the degree of importance articulated by the participants or researchers, or both. These domains and constructs were affirmed by data analyzed from the trial implementation. Five additional constructs, identified as being strongly influential across the remaining two CFIR domains, emerged through the course of the trial implementation. Specific quotes for each CFIR domain and construct from each phase of the study are presented in Table S1. The following sections describe each construct.

Outer setting domain

Patients’ needs and resources

This construct emerged as a central focus of the formative evaluation. Providers and administrators identified SURPAS as a mechanism to provide individualized, patient-centered care to patients. SURPAS was thought to engage and empower patients with personalized information to participate in a conversation that supports a shared decision regarding their treatment plan.

“Any type of personalized care is gonna be better than generalization for people. Having that kind of information on paper is extremely helpful because you could imagine you get bombarded with a lot of information leading up to the surgery.” (Patient #3).

Optimization of pre-surgical care was identified by the administrators and providers. Surgeons may change their preoperative workup, medical optimization, or potentially select different surgical or therapeutic treatment options. Recommendation for patients to change some of their lifestyle habits, in light of their increased surgical risks, could be made.

Patients perceived that SURPAS provided desired information, increased engagement with providers, and enhanced patients’ surgical experience. When specifically asked about the SURPAS tool, all but one patient was impressed with SURPAS.

“Pretty informative”; “Simple to understand”; “It did a great job of delineating likely risk to unlikely risk”; “More realistic”; and “Better understanding”. (Multiple patients).

Intervention characteristic domain

Evidence strength and quality

The empiric evidence that SURPAS was grounded in the ACS NSQIP database and statistical methodologies supported participants’ strong view of its ability to provide accurate data to providers and patients. The participants were confident the data would improve the risk assessment process and could ultimately improve outcomes of morbidity and mortality. Providers found SURPAS to be helpful in accurately presenting the surgical risk to patient’s families, especially in the more complex cases. Most providers indicated it enabled a more in-depth discussion of risk than the current standard of care and ensured that patients were educated about their operation.

“I think, [SURPAS] is informing our providers and our faculty based on science and the robust database that we have, then one could say that, “This is evidence-based, and we have all the data back here to back this, and we should listen to it,” because if we can predict with X percent accuracy, why wouldn’t you use this? Then, why wouldn’t it improve outcomes?” (Administrator #7).

Adaptability

During the contextual baseline assessment phase, the adaptability construct emerged as a facilitator of tool usage by the local providers. Suggestions to improve the SURPAS tool included providing definitions for the American Society of Anesthesiology physical status classification (ASA class), emergent/elective operation status, and functional health status of the patient on the input screen, provide drop-down menus for the input of the SURPAS predictor variables, current procedural terminology (CPT) codes, and names for operations frequently performed by the provider. In addition, due to the variability in each clinical environment, multiple suggestions were provided to guide the implementation process in these various contexts. Setting up protocols for implementing SURPAS in addition to protocols or pathways for bundles of complications were suggested. Patients were shown multiple variations of data display and the highest rated was chosen.

In the Trial Implementation, Adaptability emerged as both a facilitator and a barrier. Specifically, the constraints of the ACS NSQIP database-limited operation code choices, which frustrated providers at times. The flexibility in utilization of SURPAS allowed each clinic to implement the tool to fit their local needs, such as the option of having someone other than the surgeon input the data and then provide the results to the surgeon or another provider, such as an advanced practice provider (APP), to engage in the risk discussion.

Complexity

In the trial implementation, complexity was identified as an important aspect to the implementation of SURPAS. Specifically, its accessibility in the EHR system, ease of data input, direct EHR documentation, and the printable patient handout facilitated implementation in individual clinics. Overall, providers found it did not disrupt their clinic workflow.

“That’s one of the best things I like about it. I’ve been part of lots of studies and it’s real easy. It doesn’t mess up my clinic flow, it’s quick and easy and nice to work with, which is great.” (Provider #7).

Design quality and packaging

The design quality and packaging construct emerged as administrators, providers, and patients believed SURPAS provided a functional, quality, and efficient tool. All nine providers in the trial implementation found SURPAS to be easy to use. All nine found that SURPAS provided a concise mechanism to estimate and document the risk of surgery and that it was useful for sharing the anticipated surgical risk with patients.

Patients shared that they appreciated the data being presented both numerically and with a pictograph (Figure 2). Patients shared that their perception of the tool and their opinions of utilizing SURPAS in their pre-surgical appointment “...lowers my anxiety about the procedure”; is “reassuring”; “made me feel more comfortable”; and “clarified a lot of things” (Multiple patients).

Inner setting domain

Implementation climate

Throughout this study, the organizational commitment to implement SURPAS was identified as a critical component of the potential success of SURPAS in the UC Hospital setting. High-level administrators and surgical care providers described the contextual components that existed and provided suggestions to leverage these for the successful implementation and dissemination of SURPAS. This included the importance of integrating SURPAS into the EHR, broad communication to ensure clinic staff were aware that this system is supported by the administration, and collaboration between departments.

Tension did emerge between low-risk patients’ preference for SURPAS and providers’ concern about additional clinic time required for risk assessment in obviously low risk patients. Therefore, a few providers shared that they would most likely continue using SURPAS, but only with their high-risk patients, after the Trial Implementation ended.

Compatibility, a sub-construct, addresses the importance of integrating SURPAS into existing clinic workflows. Providers agreed that SURPAS improved clinic workflows in a couple of ways: SURPAS enabled a more in-depth discussion of risk than the current standard of care; and provided an opportunity to ensure patients were more educated about their operative risks. Providers said patients were very receptive to SURPAS, which helped patients see the surgical risks visually, improving their understanding. Providers of low-risk surgeries shared that many patients said it was beneficial to them and reassured them that the decision to undergo surgery was the correct one.

Providers identified barriers to the implementation of SURPAS. One shared that medicine is traditionally conservative and lags in information technology (IT) implementation. Unavailability of computers or the inability to print the patient document in the clinic may cause providers to be resistant to adoption of SURPAS. Providers shared that it is customary to load work on doctors who, thus, may be reticent about adding new tasks. One surgeon was concerned that providers could be held legally accountable if the risks were low but higher than national averages and nothing was done to mitigate the “elevated risk”.

Characteristics of individual domain

Knowledge and beliefs about the intervention

Administrators and providers believed SURPAS had the potential to support their clinic to attain various goals such as improving patient care and satisfaction, and reducing length-of-stay and re-admissions. They were interested in utilizing SURPAS to optimize patients’ outcomes by preventing infections, decreasing pain, improving functional abilities, and increasing awareness of social determinants of health. Administrators believed that a multi-faceted partnership with case management, social work, and other clinical disciplines, would improve pre-operative care to prepare for surgery. Additionally, individual providers’ motivation to improve patient selection, patient care, and patient engagement was a reason they supported using SURPAS. Nonetheless, one provider believed his complication rates were better than those reported by SURPAS and opted not to use it in his clinic.

“I found it helpful, primarily, with more of the more complicated patients presenting for more complicated surgery. It did provide a good reference as to their potential risks with the more complicated procedures or surgery with the less than healthy patients.” (Provider #9).

Process domain

Planning

Clinical providers and administrators consistently identified planning factors that they believed would be vital to the successful implementation of SURPAS. Senior administrators suggested planning ahead with identified teams responsible for the implementation of the tool and technical support. The next steps included “marketing” the value to the end user to “generate buzz” and the use of rapid feedback to improve the tool and its implementation. Administrators also suggested planning for successful implementation by tying SURPAS to strategic goals of the healthcare system, including innovation, safety, access, growth, and patient-centeredness.

Clinic administrators identified specific activities such as clinic staff being properly educated about SURPAS; defining specifically who in the clinic is responsible for SURPAS’ accessibility on the computers, availability of printers for the SURPAS patient handouts, and continued technical support; who will access the information and input the data into SURPAS; and who will have the risk assessment conversations with the patients. These participants identified that planning needed to be in alignment with the senior management group in both bundled care and care management continuums.

Engaging

The engaging construct identified factors that administrators and providers believed would encourage adoption of SURPAS. Attracting and involving key individuals in the implementation and use of SURPAS needs multifaceted strategies including marketing, education, role modeling, and training. Additional engagement strategies included: testimonials of patients and providers regarding their perceived value of SURPAS; data to support the accuracy of SURPAS; using multi-level communications, emails, one-on-one demonstrations, presentations at clinic meetings and Grand Rounds; and iterative trainings for the various levels of providers, such as surgical residents in training, as they rotate through the different surgical services.

Additional findings

Although penetration (the integration of a practice within a clinical setting), and sustainability (the maintaining or institutionalizing of an intervention within a clinical environment’s culture through policies and practices) are not CFIR constructs, they are key implementation outcomes (18). Surgical providers shared how they will use SURPAS outside of the study setting, indicating that ease of use was the key to facilitate its implementation. All participating groups thought that SURPAS supported risk assessment and shared decision making and facilitated the documentation of operational risks.

One unique environment that implemented SURPAS was the pre-procedure services clinic, where APPs conducted in-depth pre-operative patient evaluation. The APPs used SURPAS for their evaluation and provided the results back to the surgeon. However, since the patient had already seen their surgeon, and potentially discussed operative risk, the APPs in the pre-procedure services clinic voiced that they were not comfortable reviewing risk because of potential discrepancies with the surgeon’s assessment and discussion, and because they were not versed in further expansion of specific patient risks associated with each type of operation. This process prevented any discrepancies between the APP’s and the surgeon’s interpretation of SURPAS results and their effect on the decision to proceed with surgery.

Publicizing feedback about the progress and quality of the tool and its implementation emerged as the most frequent suggestion for the successful implementation and sustainability of SURPAS. Suggestions included: keep SURPAS updated as new data from the ACS NSQIP becomes available; provide recommendations to mitigate the risk; and provide feedback to the participating surgeons comparing them to their peers. Process suggestions included: touch base with clinic staff on a regular basis; ask “Do we need to tweak it? What’s working well? What’s not working well?” Public display of clinic goals and providing data on implementation progress was viewed as very important by providers.

Discussion

In this article, we describe the importance of using D&I science in surgery and the successful utilization of CFIR to identify and categorize barriers and facilitators during the design and implementation of a novel surgical clinical decision support tool, SURPAS. The identified constructs that strongly affected the broad implementation of SURPAS at specific UC Hospital surgical clinics are likely to affect scaling-up further spreading SURPAS to all six UC Health System hospitals.

We designed this study to guide the D&I of SURPAS with the objective of utilizing CFIR’s domains and constructs to organize the data collection and evaluation for the two phases in the formative evaluation: the contextual baseline assessment and the trial implementation (4,5,13). Using CFIR to systematically assess the qualitative data, we were able to integrate the data and identify strategies to enhance the D&I process. Strategies, defined as “methods or techniques used to enhance the adoption, implementation, and sustainability of a clinical program or practice” (19,20) are aimed at supporting the integration of SURPAS into various clinical environments across the UC Health System. We used intervention mapping (20-23) to map the CFIR constructs to: (I) identify barriers to and facilitators for scaling up; (II) describe how these manifest in context; (III) identify specific strategies to leverage the facilitators or mitigate the barriers; and (IV) identify the desired outcomes. Specific examples are shown in Table 3.

Table 3. Intervention mapping.

CFIR construct	Facilitator/barrier	Manifestation	Strategy to mitigate barrier leverage facilitator	Desired outcome
Outer setting domain—patients’ needs and resources construct	SURPAS provides individualized, patient-centered care to patients	SURPAS provides patients with information, increased engagement with providers, and enhances patients’ surgical experience	Conduct local meetings with key stakeholders (i.e., patients) to build collaboration and awareness of SURPAS	Increased patient satisfaction and improved clinical outcomes
Intervention characteristic domain—evidence strength and quality construct	Empirical evidence that SURPAS was grounded in the ACS NSQIP database and the statistical methodologies utilized	No one questioned the empirical evidence	Ongoing educational meetings with stakeholders, individual meeting with clinical champions and implementation facilitators	Providers and patient confident with data provided by SURPAS
Complexity construct	SURPAS’ ease of use	Simplicity of understanding SURPAS’ message	Develop an implementation manual to ensure appropriate implementation and use of tool	Successful implementation of SURPAS
Design quality and packaging construct	SURPAS’ data display and patient handouts	Provided a concise mechanism to estimate and document the risk of surgery, supportive to populate clinical notes, and useful for sharing the anticipated surgical risk with patients	Create a learning collaborative	Increased patient’s knowledge of risk assessment of surgical complications
Adaptability construct	Limited CPT codes in SURPAS and lack of specific comorbidities	Provider frustration that specific surgical CPT codes were not available	Work with technical experts to continually update SURPAS and sharing surrogate codes between providers and promote adaptability	Provider utilize new specific CPT code or a similar (surrogate) one
Inner setting domain—implementation climate construct	Highly supported by administrators and surgical care key stakeholders and providers	Established integration of SURPAS in the EHR	Identification and preparation of local clinical champions	Fully implemented in all surgical clinics
Implementation climate construct	Tension between low-risk patients’ preference of SURPAS and providers’ concern about additional clinic time required to discuss risks	Providers only use SURPAS with high-risk patients	Utilization of patient and family feedback	Providers use SURPAS with all patients
Compatibility sub-construct	Contextual and technological variations affecting implementation	Clinic room environment does not provide computers for assessment of SURPAS	Utilization of facilitation and technical experts—internal and external	Experts work with clinic to provide access to computers or change of process to perform SURPAS before entering clinic room
Characteristics of individuals domain—knowledge and beliefs about the intervention construct	Provider believes his/her complication rates better than those reported by SURPAS	Provider chooses not to use SURPAS	Utilization of outcomes rates audit and feedback	Provider sees that their complication rates are similar to SURPAS
Process domain—planning construct	Planning ahead with identified teams responsible for the implementation of the tool and technical support	Providers unable to access SURPAS from the local computer	Utilization of facilitation and technical experts - internal and external	Fully implemented in all surgical clinics
Engaging construct	Factors to encourage adoption of SURPAS	Provider chooses not to use SURPAS	Conduct ongoing education, training, evaluation, and consultation	Integrated into all pre-surgical patient-provider interactions

SURPAS, SUrgical Risk Preoperative Assessment System; ACS, American College of Surgeons; NSQIP, National Surgical Quality Improvement Program; CPT, Current Procedural Terminology.

Data from the contextual baseline assessment allowed us to improve the functionality of the SURPAS tool and enhanced our understanding of potential barriers and facilitators in the trial implementation. The findings from the trial implementation identified specific barriers and facilitators to implementing SURPAS across the six hospitals in the system, allowing the creation of specific strategies to address each one.

For example, although most participants were supportive of SURPAS, tension emerged between low-risk patients’ preference to use SURPAS and providers’ concern about additional clinic time required for formal risk discussion. Therefore, if this barrier is also identified at other hospitals, a strategy to address this issue is to use patient and family feedback and share the results with the providers and administrators through a rapid feedback loop (23). Additionally, providers highlighted barriers such as the medical profession being slow to adopt new technologies, which may indicate resistance to any new tool such as SURPAS. Strategies to address these issues involve incentivizing using SURPAS and de-incentivizing use of the traditional method of risk assessment with policies and procedures created through cooperation with administration.

A few issues arose due to the constraints of SURPAS’ background database having limited CPT choices. Therefore, strategies will need to be employed to work with surgical providers and technical experts to continually update SURPAS’ inclusion of CPT codes and sharing codes other providers used in place of missing codes. Dissemination of the latest information can be provided through educational meetings with stakeholders, individual meetings with clinical champions and implementation facilitators.

Most providers believed in the accuracy of the individual risk assessments provided by SURPAS; yet, one provider believed the complication rates provided by SURPAS were higher than his actual rates, and therefore did not use SURPAS. This barrier is categorized in the knowledge and beliefs construct. Audit and feedback strategies can address this barrier by collecting clinical performance data and providing it to clinicians and administrators to confirm or dispute such beliefs. The UC Health System presently collects these data, which could be reported back to each provider (23).

These strategies ensure interactive problem solving and support through individuals who are familiar with both the local clinical environment and SURPAS. Local IT professionals will be supported by an implementation expert and a technology team, which are available throughout the implementation process.

The literature supports the need to identify specific D&I strategies and to apply systematic methods for selecting and tailoring these strategies to specific contexts and innovations (19,20,24,25). The literature supports the variability in using CFIR coding yet it also supports the recommendation of Damschroder et al. to report the rationale for selection of each construct (3,13). Kirk et al. identified gaps in the use of CFIR in all of the phases of an individual implementation (5). This study contributes to the D&I literature through the successful utilization of CFIR in the translational process from the design of SURPAS to the proposal of the broad implementation. Additionally, this study provides data supporting the use of SURPAS, thus adding our findings to those from other studies, building the evidence for what, how, and when implementation factors and strategies improve implementation processes (26).

Strengths of this study include the initial integration of a broad range of opinions to improve SURPAS and identify barriers and facilitators of the implementation process. Our inclusion of all possible CFIR constructs will promote the integration of our findings with findings from other studies using CFIR. Potential limitations may include social desirability bias—i.e., participants responding in a certain way to please the interviewer—and that the qualitative data were elicited only from people directly involved in this study. Other surgeons, administrators, and patients who would be affected by a broad implementation were not included.

Conclusions

Using CFIR to guide and code the formative evaluation of introduction of new technology within surgery, such as SURPAS, has provided essential information by identifying constructs and processes to inform design of future larger-scale D&I of this technology (19,27). This prospective process using CFIR has allowed us to critically improve SURPAS and design the implementation process, improving the likelihood of a successful D&I of SURPAS in the surgical clinics of UC Hospital and further to the UC Health System.

Table S1. Specific quotes for each CFIR domain and construct from each phase of the study.

CFIR domains	CFIR constructs	Contextual baseline assessment example	Trial implementation example
Outer setting domain	Patients’ needs and resources construct	“You’re always told risks. It’s verbalized. All this stuff is being thrown at you, the surgery, recovery … You don’t necessarily always remember. When you go home, it’s just like, “Well, what did they say?” I’m like, “Eh, it must not be too bad ‘cause he said go ahead and do the surgery.” Having this as something to fall back on and look at would’ve been great.” (Patient #5)	“I like that I got a printed form that I could leave with. I think that’s very, very helpful. I think that it was explained in terms that I could understand.” (Patient T47–180)
		“In a patient that is a higher risk, you have objective data saying they’re a higher risk, and it gives your ideas of certain areas that you could address, potentially, to lower that risk before you operate on them, unless it’s an absolutely emergent case.” (Provider #13)	“I think that’s very beneficial, because it engages the patient, it empowers them with additional health literacy, it informs them, and ideally relates to them that they are not only a consumer of healthcare, but they have power to change their own outcomes based on their own behavior and their own health.” (Provider #2)
		“...focused on the patients and being patient-focused, to have a way that really individualizes some information that is incredibly complex, and individualizes it to that person. What I really like about it is having involved the patients themselves, and the design, and the evaluations, so that the information is actually really helpful, seen as helpful, by patients.” “From a patients’ safety perspective, to have a way that is evidenced-based, to really be—to be focusing our patient-safety initiatives on the patients who are most likely to benefit.” (Administrator #8)	“I think it’d be positive. I would think length of stay would be shorter. I think patients’ peace of mind would be higher. It might even with some patients point out that maybe I need to change some of my life habits.” (Administrator #3)
Intervention characteristic domain	Evidence strength and quality construct	“I think, [SURPAS] is informing our providers and our faculty based on science and the robust database that we have, then one could say that, “This is evidence-based, and we have all the data back here to back this, and we should listen to it,” because if we can predict with X percent accuracy, why wouldn’t you use this? Then, why wouldn’t it improve outcomes?” (Administrator #7)	“I think what I really like about this tool is it does start to get to some more of the comorbid conditions inherently, ‘cause it’s built through ACS data points. What I like about this tool is there is a social component in this tool.—the functional health status and readmissions.” (Administrator # 6)
		“This is very important. What we ought to be doing with our databases and to streamline the calculators, have them at the bedside or in the clinic. Use ‘em for shared decision-making with the patient and the family, individualizing healthcare.” (Provider #13)	“I would say, in general, surgeons always are trying to estimate the risk of surgery versus not proceeding with an operation, so I think it would be helpful to most surgeons. I’m not sure who wouldn’t wanna use the tool.” (Provider #10)
		“I think it is an excellent tool. Any type of personalized care is gonna be better than generalization for people. Having that kind of information on paper is extremely helpful because you could imagine you get bombarded with a lot of information leading up to the surgery.” (Patient #3)	“I would have expected someone to say, “Okay, these are the potential risks,” but not, “There’s a one point five percent chance of me having A, and a zero point six percent chance of having B.” That kind of, I think, reassures me in a way that it wouldn’t have otherwise.” (Patient E3–15)
	Adaptability construct	Multiple participants offered suggestions to improve SURPAS –theses suggestions were applied to improve the utility of SURPAS: i.e., “I would have a dropdown menu, so each one of those has a couple of choices. I think it’s pretty straightforward to choose something from a dropdown menu, and very quick. It’s much easier to decide between three choices than to come up, de novo, with whatever you’re looking for.” (Provider #9)	Lack of ability to assess all CPT codes is a barrier to physicians: “It’s also unable to account for the wide breadth of complexities in vascular procedures. Some procedures are borderline experimental endovascular procedures.” (Provider #6)
	Adaptability construct/complexity construct	“You have someone in clinic would/could do it ahead of time for you.” Flexibility in the data input process (Clinic Administrator #3)	“The way our flow is, the Medical Assistant checks in the patient, goes through their meds, takes their vitals, and then the physician sees the patient. It could be potentially—SURPAS could be one of the MA tasks, if they know what operation the patient needs.” (Provider #10)
		Not emergent in this phase	“That’s one of the best things I like about it. I’ve been part of lots of studies and it’s real easy. It doesn’t mess up my clinic flow, it’s quick and easy and nice to work with, which is great.” (Provider #7)
	Complexity construct/design quality and packaging construct	Not emergent in this phase	“We’ve been looking for a more simplified tool that we could put in information into and then provide to the patients, as well” (Administrator #4)
	Complexity construct/design quality and packaging construct	Not emergent in this phase	“The documentation is automatically generated. It helps you populate your note. That actually may save some time. It improves your documentation.” (Provider #4)
			“Seeing is believing, and it’s difficult for patients to comprehend. We, as healthcare providers, talk in a language they don’t understand or comprehend. The pictures are helpful. Seeing the relationship between my life and the lives of all other people who’ve undergone this, and, because of my risk factors—I think that can speak volumes to patients and their families.” (Administrator #5)
	Design quality and packaging construct/implementation climate construct	“I see administrative support in terms of working with IT and our chief information officer to make sure we have the groundwork laid to say that this is important, and this is going to impact better outcomes for patient care, particularly in the surgical realm. This needs to be prioritized in terms of something we adopt and implement after we get through the initial research phase to say, “Yes, this is the thing to do.” Also, I think it’s the support of saying if we’re on to something that’s unique and innovative, and is gonna be tied into Epic, then how we partner with Epic to say that this should be the standard of care for the product that you’re delivering to other clients, so that it’s not just something great that we’re doing here” (Administrator #7)	“Good. I've never seen it put in this type of format before. You know, in the military we understand things based on—we rate people exactly like this. How many people out of how many people. To me, it speaks to what I'm used to … We look at if there’s 5.5 people on here out of 100 who had the second—it’s visually pleasing to see that, instead of hearing about the numbers.” (Patient # T4-4)
	Design quality and packaging construct/implementation climate construct	“I see administrative support in terms of working with IT and our chief information officer to make sure we have the groundwork laid to say that this is important, and this is going to impact better outcomes for patient care, particularly in the surgical realm. This needs to be prioritized in terms of something we adopt and implement after we get through the initial research phase to say, “Yes, this is the thing to do.” Also, I think it’s the support of saying if we’re on to something that’s unique and innovative, and is gonna be tied into Epic, then how we partner with Epic to say that this should be the standard of care for the product that you’re delivering to other clients, so that it’s not just something great that we’re doing here” (Administrator #7)	“I think that opens our eyes a little more to the risk, and especially seeing it on a graph. Gives you something’ to compare to, and I like that.” (Patient #V1–28)
			“Knowing that it provides a documented risk assessment is an incentive. The idea that it can aid in me identifying patients at a higher risk of a complication and guide my perioperative management to decrease those risks, and therefore those patients have better outcomes, and me by proxy have better outcomes of my care delivery, is an incentive itself.” (Provider #2)
			“Well, I guess being at an academic center, you tend to hear about things more so. You read a lot of research that’s going on. I think there’s a higher acceptance of new technology, new ways of doing things that we want to trial and see whether they work, so that it can improve care” (Administrator #4)
			“I think [SURPAS] will fit into our total joint package in trying to make sure that we can reduce those readmissions. I think it’ll fit in quite easily” (Administrator # 2)
Inner setting domain	Knowledge and beliefs about the intervention construct	Not emergent in this phase	“I probably won’t use it in my elective clinic unless I’m seeing a patient who has a uniquely high-risk procedure or has serious health issues, like severe pulmonary hypertension or something where I think it would be worth maybe relying on a more objective means for calculating these risks, rather than my eyeball test.” (Provider #3)
			“I found it helpful, primarily, with more of the more complicated patients presenting for more complicated surgery. It did provide a good reference as to their potential risks with the more complicated procedures or surgery with the less than healthy patients.” (Provider #9)
			“Ultimately, I think getting alignment with the surgeons to understand the value, to me is probably one of the first steps. I think that there is value in this, but my initial thought is how does this align with the overall organizational priorities.” (Administrator #6)
Characteristics of individuals domain	Planning construct	Not emergent in this phase	Usually, when we roll out things that involve the physicians, there’s tip sheets. There’s expectations. There’s support, especially as it relates to electronic support. At-the-elbow support to make sure people fill it out correctly or know how to get to it or do whatever. (Administrator #5)
			I would probably say that a good way to implement it is to go back through the Department of Surgery Grand Rounds and show the tool to the residents, and how it actually is implemented and what the experience in the clinics that have trialed it. Then make the case for broadening it out into more than just the trial clinics that you piloted it in. (Provider #4)
			“Staff is properly educated on what this would do and then truly defining who in the clinic is gonna own the responsibility & the benefit of why that would help to obtain the staff buy-in.” (Administrator #2)
Process domain	Engaging construct	Not emergent in this phase	“Saying how accurate it is and maybe a couple of testimonials of a patient and provider. You really thought it was, you know, great. I think that that's really what it's going to take it going to different faculty meetings and resident educational things to let people know about it. You have to just keep doing that, because the other thing is the residents rotate. They're often gonna be the ones to do this.” (Provider #3)
			“With training sessions? With presentations? Probably a variety. I think once the workflow is determined, that I would share in writing as well as verbal communication with the staff, at a staff meeting. I would be in communication with the physicians to see how to work out the workflow, I think you would do more than one way to share it with the staff, and it would be ongoing. This is the plan. This is how it’s gonna work, Send it out then in an e-mail, because, of course, you can’t give them too much information. Then probably follow up periodically, either in my daily huddles—probably daily huddles follow up and then at a staff meeting.” (Administrator #3)
			“Then make sure that we roll it out in a fashion so that not just the providers but all of the staff—the MA’s (medical assistants), the clerical staff, the nurses, understand what it is, why we’re doing it, and what it entails. How it might change their work flow.” (Administrator #3)
			“I think that if you just engrain it in the residents and you get enough faculty members on—it's sort of like reaching a critical mass. Once there's enough people on different services that are touting it and champing it and saying I want you to do this when you see my patients. Tell me what the numbers are. Then, they start actually doing that for other faculty members.” (Provider #3)

CFIR, consolidated framework for implementation research.

Ethical Statement: The Colorado Multiple Institutional Review Board approved this study (#15-1044).