The Price of Practice Change

Abstract

Background:

Clinicians, health care administrators, and implementation scientists know that it takes intentional effort, resources, and implementation strategies to integrate research findings into routine clinical practice. An oft-cited concern for those considering whether and how to implement an evidence-based program is how much it will cost to implement the change. Yet information about the cost of implementation is not often available to health care decision-makers. Teams that received Implementation Award funding from PCORI are conducting implementation projects to promote the uptake of evidence-based practices in health care settings. As part of their implementation efforts, a number of teams have examined the costs of implementation. In this Topical Collection, 5 teams will report their findings on implementation costs and discuss their methods for data collection and analysis.

Discussion:

The teams’ costing efforts provide specific information about the costs sites can expect to incur in promoting the uptake of specific evidence-based programs. In addition, the papers illuminate 3 key features of the teams’ approaches to measuring the cost of implementation: (1) the use of specific micro-costing methods with time-driven activity-based costing serving as the most popular method; (2) different ways to categorize and organize costs, including a site-based and non-site-based framework; and (3) cost collection challenges experienced by the teams.

Conclusion:

The cost of implementation is a critical consideration for organizations seeking to improve practice in accordance with research findings. This Topical Collection describes detailed approaches to providing this type of cost information and highlights insights to be gained from a rigorous focus on implementation cost.

BACKGROUND

It is widely recognized that integrating research findings into clinical practice takes concerted and intentional effort.¹ Successfully adopting an evidence-based practice often requires undertaking a number of different implementation strategies, such as training clinical staff, identifying and working with clinical champions, updating electronic record systems, or preparing audit and feedback reports. Implementing an evidence-based program may also involve acquiring new equipment, reconfiguring physical space, or creating new clinical teams.² The cost implications of implementation–that is, the costs to support the time and resources required–are among the many factors health care decision-makers may consider when contemplating whether and how to implement an evidence-based intervention.

In fact, implementation costs are often cited as a major consideration for health care decision-makers.^3,4 Having cost information available during the decision-making process—before implementation occurs—has been found to be particularly important so that leaders can consider financial implications and weigh trade-offs before investing in implementing an evidence-based practice change.⁵ However, little real-world information on the costs of implementation is available. A recent systematic review found that only 6 out of 30 implementation or service improvement studies included an explicit assessment of the costs of implementation strategies used.⁶ Another systematic review published in 2007 found that fewer than 10% of implementation studies include information about the cost of implementation.⁷

Consequently, the field of implementation science has called for including more economic evaluation in implementation studies.⁸ Implementation science offers valuable structured approaches and frameworks to systematically understand how implementation occurs and to evaluate the success or failure of implementation efforts. Several frameworks include cost assessment—for example, the RE-AIM framework. However, the literature suggests that cost is still one of the least-reported elements of RE-AIM, and studies that do include cost use inconsistent approaches to capture this information.⁹

Producing Useful Information for Decision-makers

Leadership support for evidence-based practice uptake—including leadership involvement in decisions regarding whether and how to implement, in planning implementation, and in dedicating or rearranging resources—is an important organizational factor to ensure successful implementation.^10,11 To support leaders’ decision-making in operational health care settings, it has been noted that cost analyses that elucidate specific details on the resources and activities associated with implementation can be particularly useful and relevant.^12,13 To provide these specific details, micro-costing methods such as time-driven activity-based costing (TDABC) and the cost of implementing new strategies (COINS) allow for precise assessment and measurement of costs (Table 1; definitions).

TABLE 1.

Definition of Terms

Term	Description
Micro-costing	A cost estimation strategy that examines detailed resource utilization and unit cost data to generate precise cost estimates14
TDABC	A micro-costing method that provides detailed cost information through the use of process mapping15–17
SIC	A tool that defines 8 stages of implementation processes and milestones: engagement, consideration of feasibility, readiness planning, staff hired and trained, fidelity monitoring processes in place, services and consultation begin, ongoing services and fidelity monitoring, and competency18
COINS	A cost-mapping tool that is similar to TDABC and maps costs associated with implementation as defined by the SIC19

COINS indcates cost of implementing new strategies; SIC, stages of implementation completion; TDABC, time-driven activity-based costing.

These methods call for observing and measuring time and resources spent over the course of conducting an implementation effort.^13–17,19 Although direct observation and measurement techniques result in the most accurate and precise estimates of costs, undertaking these micro-costing efforts can be expensive and time-consuming.²⁰ Yet, policymakers have called for these accurate costing measures to enable health care leaders to better understand costs and identify new opportunities in organizing care delivery.²¹ Illuminating detailed implementation costs is also beneficial for other stakeholders with an interest in the uptake of evidence-based practices (eg, patients and payers) because implementation costs—which are typically nonreimbursable as they fall outside of care delivery—can present barriers to implementation.

Teams Implementing Evidence-based Practices

Project teams that received Implementation Award funding from PCORI are conducting active implementation projects to promote the uptake of evidence-based practices into health care settings, taking evidence from the research context into real-world practice. These implementation projects are intended to demonstrate the feasibility of implementation approaches and document their impact, to lay the groundwork for future evidence uptake in health care settings beyond the project. Project evaluation includes the measurement of health and health care outcomes as well as implementation outcomes.

Under a targeted PCORI supplemental funding opportunity, a number of teams conducting implementation projects are also collecting data to measure the costs associated with implementing the evidence-based practice or program that is the focus of their project. This cost information is intended to be useful to health care leaders and decision-makers at future sites, who may weigh a decision on whether to implement these practices or programs and could use specific details on the resources and activities that may be required to implement.

As noted, measuring and reporting detailed cost information related to implementation is challenging and rarely done. There is a need not only for the cost information itself but also for demonstrations of approaches to measuring and reporting out detailed implementation costs. By undertaking this work, the project teams assessing the costs of implementation are providing instructive examples, demonstrating how this cost data collection, measurement, and analysis occurred within the context of their projects. These models can serve as potential roadmaps for other implementation teams seeking to capture and calculate their implementation costs.

In this Topical Collection, 5 teams describe their approaches and methods for determining the costs of implementation, as well as report their cost findings. The teams’ implementation projects encompass a diverse set of evidence-based practices and programs. The first implementation project team incorporated results from a PCORI-funded study that examined 3 common treatments for prostate cancer into an existing decision aid, then worked with 3 different health care sites to put the decision aid into use.²² The second team put a shared decision-making program into place at 5 gynecology clinics to help women with uterine fibroids and their doctors make decisions about treatment.²³ The third team integrated a set of context factor assessment surveys, called Healing Encounters and Attitudes Lists, or HEAL, into regular patient care through a patient-reported outcomes electronic platform at 7 pain clinics at the University of Pittsburgh Medical Center, to support patient-provider conversations about pain management, including nonopioid pain treatments.²⁴ The fourth team worked with 20 primary care clinics in North Carolina to de-implement daily self-testing for patients with type 2 diabetes who do not use insulin.²⁵ Finally, the fifth team implemented a pediatric weight management program at 3 health systems to support children with overweight or obesity.²⁶

Teams’ Costing Efforts

For their costing efforts, all teams included in this Topical Collection used micro-costing methods to measure the costs of implementation. Their efforts began with mapping the specific workflows and resources associated with implementing these evidence-based programs and then measuring the associated costs. Their cost assessments are from the perspective of the implementing site—that is, the health system, hospital, clinic, or other settings, that is implementing the evidence-based practice within their routine workflow. If relevant, certain teams also captured costs that were borne by an entity outside of the implementing site (eg, a center that provided specific support and assistance across sites throughout implementation).

The focus of all 5 analyses is specifically on implementation costs—the costs associated with the resources required to put the evidence-based program into place, deliver the program, and sustain the program.¹³ The analyses do not include measurement of costs associated with the consequences of implementation (eg, patient or caregiver time spent for the intervention), or downstream costs related to changes in health care utilization or productivity.²⁷ These analyses also do not include other forms of economic evaluation that involve examining implementation costs in a broader context, for example, return on investment analyses or cost-utility analyses.

Table 2 presents the papers included in this Topical Collection. In the first paper, Ho and colleagues discuss their use of a TDABC approach to capture all steps involved with delivering and using the prostate cancer treatment decision aid. These steps included identifying new patients with prostate cancer, sending the decision aid in advance of each appointment for the patient to complete, and reviewing the patient’s results. Each implementing health care site put in place a workflow that best suited their context and available resources. The team describes how they produced process maps to reflect these workflow differences, then measured the time and resources associated with each process step to calculate the cost. Their cost activity revealed that electronic health record (EHR) integration is an important factor in improving efficiency and reducing cost.

TABLE 2.

Cost of Implementation Papers Presented in the Topical Collection

Paper Title	Authors
Health System Perspective on the Cost of Delivering a Decision Aid for Prostate Cancer Using Time-driven Activity-based Costing	David R. Ho, Robert Kaplan, Jonathan Bergman, David F. Penson, Benjamin Waterman, Kristen C. Williams, Jefersson Villatoro, Lorna Kwan, and Christopher S. Saigal
The Costs of Implementing a Conversation Aid for Uterine Fibroids in Multiple Health Care Settings	Stephanie C. Acquilano, Rachel C. Forcino, Danielle Schubbe, Jaclyn Engel, Marisa Tomaino, Lisa C. Johnson, Marie-Anne Durand, and Glyn Elwyn
Costs of Implementing Electronic Context Factor Assessments and Patient-reported Outcomes in Pain Clinic Settings	A. Rani Elwy, Simone Taubenberger, Nathan Dodds, Rebecca DeSensi, Andrea Gillman, Ajay Wasan, and Carol M. Greco
A Cost Analysis of Rethink the Strip: De-implementing a Low-value Practice in Primary Care	Lisa P. Spees, Laura A. Young, Jennifer Rees, Kathleen Mottus, Jennifer Leeman, Marcella Boynton, Erica Richman, Maihan B. Vu, and Katrina E. Donahue
Costs to Implement a Pediatric Weight Management Program Across 3 Distinct Contexts	Natalie Riva Smith, Meg Simione, Haley Farrar-Muir, Jazmin Granadeno, Jennifer W. Moreland, Jessica Wallace, Holly M. Frost, Jackie Young, Cassie Craddock, Kerry Sease, Simon J. Hambidge, Elsie M. Taveras, and Douglas E. Levy

In the second paper, Acquilano and colleagues illustrate their use of the COINS with a TDABC approach to measure the costs associated with planning and set-up, implementation, and sustainability for implementing 2 uterine fibroid treatment conversation aids. The team describes their use of different data sources, including internal documentation and interviews, to capture details on the activities conducted at the clinics to put the conversation aids into use. The time spent on different activities and associated costs were found to be highly variable across the health systems. The team describes the factors that drove the variability, such as whether clinicians at the clinics were already using some type of decision aid.

In the third paper, Elwy, Greco, and colleagues describe using the COINS tool to measure and map the costs of implementing the HEAL surveys organized by the 3 overarching stages of implementation: pre-implementation, implementation, and sustainment. Within each stage, the team presents details on all implementation activities, including the personnel who conducted the activity and the associated cost. For example, a major activity during the implementation stage was programming the HEAL survey measures into the patient information registry system, which took an estimated 330 hours of programmers’ time.

In the fourth paper, Spees and colleagues describe their cost analysis for the “Re-Think the Strip” program, which included activity-based costing for personnel time spent on 5 implementation strategies: educational meetings, practice facilitation, provider champions, audit and feedback reports, and educational materials. The team found that EHR programming—in this case for the audit and feedback performance reports—was a major cost for the health system. Due to Re-Think the Strip being implemented in 20 primary care clinics, the team also found variability in costs per clinic for the same set of implementation strategies. Notably, the team explored potential strategies to minimize implementation costs, such as holding virtual educational meetings instead of in-person meetings.

Lastly, in the fifth paper, Smith and colleagues present their TDABC methods to assess the costs associated with activities to put the “Connect for Health” pediatric weight management program into place across 3 different health systems, which in turn each implemented the program in multiple clinics. The costs for each health system were found to vary depending on available resources and how each health system tailored the program to fit its existing care models. The team highlights that gathering and incorporating stakeholder input (eg, on how to integrate the program’s clinical tools into the EHR and how to integrate the program within workflows) takes time but is an important up-front activity to ensure successful and sustainable implementation. In addition, the site that organized and facilitated the implementation efforts incurred more costs than the other participating sites.

DISCUSSION

The papers in this Topical Collection contribute to the field of implementation science in 2 ways. First, these papers provide more information to health care decision-makers on the actual observed costs associated with implementing evidence-based practices. Each team was able to capture the specific types of personnel, as well as the detailed tasks and activities, involved in implementation, essentially laying out clear pathways for future sites considering whether and how to put these evidence-based practices into place. These cost estimates reflect the specific experiences of the sites included in the teams’ implementation projects; therefore, the results may have limited generalizability. However, the findings can still inform future sites’ understanding of the types of costs and resources needed for implementation.

Second, these papers add significantly to the otherwise scant literature on demonstrated approaches for measuring the costs for implementation. Each team showcases the use of micro-costing to collect implementation costs, serving as examples for future implementers and researchers conducting cost-of-implementation analyses. Further, we discuss 3 main features of the approaches the teams used: (1) their specific micro-costing methods; (2) ways in which teams categorized and organized costs; and (3) the challenges encountered by the teams and how they addressed them.

Micro-costing Methods

The choice of micro-costing to capture the costs of implementation reflects the use of a more precise and accurate way to measure costs. TDABC was the most popular micro-costing method, used explicitly by 3 teams (Ho and colleagues, Acquilano and colleagues, and Smith and colleagues). Spees and colleagues used a mix of data sources but relied on a similar activity-based costing approach to map and measure an important component of costs, personnel costs. Acquilano and colleagues paired their TDABC approach with the COINS method to organize their costs by implementation phase. Elwy and colleagues similarly used a costing approach organized by implementation phase, using Stages of Implementation Completion to define the stages of implementation, then COINS—which resembles the TDABC approach—to cost the activities within those stages.

The micro-costing methods used by the teams enabled them to identify and analyze the factors driving variation, identifying elements of potential importance to decision-makers interpreting the results. Sources of variability included different clinical workflows, IT infrastructure, and training and meetings already in place. For example, Ho and colleagues report that the full EHR integration undertaken at one site enabled a more efficient workflow, and thus resulted in lower costs compared with other sites. Similarly, in Smith and colleagues, one site undertook a larger portion of implementation support activities, such as the creation of audit and feedback reports. These costs were not reflected in other sites’ costs. These findings underscore the value of micro-costing in enabling future decision-makers to clearly see the distinct and detailed activities, resources, and costs required for implementation and determine those that are most relevant to their own institutions. Health care leaders can use this detailed information to identify resources already in place at their institution versus new resources they will require, or to understand whether resources developed elsewhere could be integrated into their implementation efforts. Further, this information can support resource allocation decisions, such as a decision to invest in an EHR build versus relying on manual delivery of an intervention, on the basis of the impact on overall costs for ongoing program delivery.

Organization of Costs

In all papers, the cost information is organized into categories designed to convey useful and relevant information to decision-makers. Three teams organize costs using the overarching framework of site-based costs versus non-site-based costs. This framework is novel, introduced by PCORI for this targeted Cost of Implementation supplemental funding opportunity. To our knowledge, this type of categorization is not previously described in the implementation science literature. Site-based costs are the costs implementing sites can expect to incur to support internal resources (eg, staff salaries, purchased materials, space, and equipment). Site-based costs are intended to be directly salient for health care leaders, clinicians, and other stakeholders making decisions about implementation in health settings. Non-site-based costs are the costs to support requirements external to the site. Non-site-based costs may be for maintaining components of an intervention—for example, updating the evidence in a decision aid or in existing training modules. Alternatively, they may support elements of the implementation strategy, such as the cost of operating a Learning Collaborative or providing external practice facilitation. Sites may pay a cost to access these non-site-based resources or other entities, such as a health professional society or a university, may support non-site-based costs for sites, essentially providing them as a “public good.”

Not all implementation efforts rely on no-nsite-based support. In some cases, all resources are acquired, and costs are borne, by the implementing sites. The first 2 papers, Ho and colleagues and Acquilano and colleagues, only report costs to sites; they do not mention non-site-based costs. However, in both cases, program implementation did require accessing materials only available from outside the implementing site—the patient decision support tools (ie, the decision aid or conversation aid). In the future, the cost to access these tools will be borne by the implementing health system or clinic. The absorption of all costs by the implementing site (ie, the absence of nonsite-based costs) may pose an advantage for future implementation, as sites may be able to implement these interventions with more independence and control.

Examining non-site-based costs, where they exist, suggests that the future sustainability of implementing the evidence-based program may depend on developing models to ensure the costs are borne by someone—for example, the implementing sites. Three papers discuss non-site-based costs (Elwy and colleagues, Spees and colleagues, and Smith and colleagues). Each has a different model for how non-site-based supports will be provided in the future (ie, after the implementation project is complete), and the degree to which implementing sites will take on these costs. Two of these papers indicate that a “center” will provide non-site-based support moving forward. For instance, Elwy and colleagues discuss how the Center for Innovation in Pain Care at the University of Pittsburgh will provide consultation services to support HEAL implementation at future sites. Smith and colleagues also indicate that non-site-based supports for Connect for Health, such as training for clinicians and staff, will likely be provided by a future central coordinating center. Elwy and colleagues have already determined that sites will pay consultation fees to access support from the Center for Innovation in Pain Care; this team reports on those costs in their paper. Smith and colleagues do not indicate whether or how future sites will pay to access the Connect for Health coordinating center. Separately, Spees and colleagues take a different approach to defining non-site-based costs as the costs borne by an integrated health system to support implementation across primary care clinics. During the implementation project, the health system provided support, such as practice facilitation, for all participating clinics. Future integrated health systems would need to take on these non-site-based costs to provide similar support across their clinics.

Aside from the site-based and non-site-based framework, the papers illustrate additional approaches to organize and categorize costs in ways that are practical and useful for decision-makers. As mentioned, several teams organize costs by the phases of implementation (eg, pre-implementation, implementation, and sustainability). Ho and colleagues use domains from TDABC to organize costs (space, personnel, devices, and materials). Acquilano and colleagues separate one-time costs from recurring costs. Two teams (Spees and colleagues and Smith and colleagues) organize costs by implementation strategy. Of note, organizing costs by strategy helped to illuminate that creating audit and feedback reports is an expensive strategy, driven by both IT costs (eg, to enable data pulls from the EHR or data warehouse) and personnel costs to create and deliver the reports.

Challenges of Cost Collection

Across the papers, there are insights into the challenges of collecting cost information. For instance, the papers demonstrate the challenge posed by the evolving stages of implementation. Costs associated with the early stages of implementation—such as the pre-implementation stage—are difficult to capture once the project has progressed beyond this stage. Across the 5 teams’ cost analyses, a common limitation was that it was not possible to collect cost information concurrently with all implementation activities. All teams conducted some retrospective analysis. This approach required staff recall of time spent on various implementation activities, or the use of time records where available (eg, calendars and emails). One team, Ho and colleagues, reports they were unable to capture the cost of EHR integration, which occurred early on during the implementation project. To note, because these cost analyses were conducted in the context of structured implementation projects, all teams have records of time and activities they could draw upon. These were supplemented with other data sources, including observations and interviews with site-based staff and project personnel. But a clear message from the experience of these projects is that it is important to initiate the collection of costs of implementation as early as is feasible.

Two papers highlight another cost collection challenge that is specific to measuring decision support tool costs. Two teams, Ho and colleagues and Acquilano and colleagues, observe that the costs to access decision support tools are important to consider, but difficult to capture. Decision support tools are typically provided to health care organizations by vendors through subscriptions or fees. These costs are negotiated directly with the health care organization and are not made public. Ho and colleagues note this as a limitation of their analysis. Acquilano and colleagues offer estimates, rather than exact costs, for the cost of subscriptions to the conversation aid.

CONCLUSION

The cost of implementation is a critical consideration for organizations seeking to improve practice in accordance with research findings. Efforts to capture and report costs using precise, detailed methods do not regularly occur, although the information from micro-costing methods can be especially valuable to health care decision-makers in real-world settings. This Topical Collection describes detailed approaches to providing this type of cost information and highlights insights to be gained from a rigorous focus on implementation cost. These analyses elucidate approaches to conceptualizing and assessing costs of implementation, as well as providing specific information about the costs sites can expect to incur in promoting the uptake of specific evidence-based programs.