States Encouraging Value‐Based Payment: Lessons From CMS's State Innovation Models Initiative

Abstract

Policy Points.

• Six states received $250 million under the federal State Innovation Models (SIM) Initiative Round 1 to increase the proportion of care delivered under value‐based payment (VBP) models aligned across multiple payers.

• Multipayer alignment around a common VBP model occurred within the context of state regulatory and purchasing policies and in states with few commercial payers, not through engaging many stakeholders to act voluntarily.

• States that made targeted infrastructure investments in performance data and electronic hospital event notifications, and offered grants and technical assistance to providers, produced delivery system changes to enhance care coordination even where VBP models were not multipayer.

Context

In 2013, six states (Arkansas, Massachusetts, Maine, Minnesota, Oregon, and Vermont) received $250 million in Round 1 State Innovation Models (SIM) awards to test how regulatory, policy, purchasing, and other levers available to state governments could transform their health care system by implementing value‐based payment (VBP) models that shift away from fee‐for‐service toward payment based on quality and cost.

Methods

We gathered and analyzed qualitative data on states’ implementation of their SIM Initiatives between 2014 and 2018, including interviews with state officials and other stakeholders; consumer and provider focus groups; and review of relevant state‐produced documents.

Findings

State policymakers leveraged existing state law, new policy development, and federal SIM Initiative funds to implement new VBP models in Medicaid. States’ investments promoted electronic health information going from hospitals to primary care providers and collaboration across care team members within practices to enhance care coordination. Multipayer alignment occurred where there were few commercial insurers in a state, or where a state law or state contracting compelled commercial insurer participation. Challenges to health system change included commercial payer reluctance to coordinate on VBP models, cost and policy barriers to establishing bidirectional data exchange among all providers, preexisting quality measurement requirements across payers that impede total alignment of measures, providers’ perception of their limited ability to influence patients’ behavior that puts them at financial risk, and consumer concerns with changes in care delivery.

Conclusions

The SIM Initiative's test of the power of state governments to shape health care policy demonstrated that strong state regulatory and purchasing policy levers make a difference in multipayer alignment around VBP models. In contrast, targeted financial investments in health information technology, data analytics, technical assistance, and workforce development are more effective than policy alone in encouraging care delivery change beyond that which VBP model participation might manifest.

States have long been recognized as the locus of public health authority and responsibility.1 ^{(pp 9‐11, 41)} States regulate their health care systems by providing oversight of health insurers, health insurance products, and health care professionals. States also set health policy through their roles as major purchasers of health care for state employees and retirees; as conveners that “create a local political moment and set an agenda for reform,” especially among business leaders, patient advocates, and providers; and more recently, as governing bodies to address emerging legal issues posed by new advances in health care data management (eg, electronic health information exchange platforms and all‐payer claims databases).2 ^{(pp 349‐350)}

In 2012, the Center for Medicare and Medicaid Innovation (Innovation Center) at the Centers for Medicare and Medicaid Services (CMS) established the State Innovation Models (SIM) Initiative to leverage a federalist approach to supporting health care transformation. The premise of the SIM Initiative is that state government leadership is necessary to harness the policy levers available to advance a transformative agenda within the context of local health needs and resources while aligning with federal priorities.

The SIM Initiative Funding Opportunity Announcement (FOA)3 required states to propose:

• New payment and service delivery models (accountable care, medical or health homes, bundled payments/payments for episodes of care) for publicly insured individuals that will support “better health, better care, and lower cost.”3 ^(p11)

• Use of policy, regulatory, or legislative authorities “to deliver broad‐based accountability for high value outcomes and include multi‐payer alignment.”3 ^(p14)

• Transition a preponderance of providers into a “value‐based clinical and business model,” therefore enhancing care coordination for a larger amount of the state's population and increasing provider accountability for patient outcomes.3 ^(p14)

The Innovation Center intentionally gave state policymakers wide latitude to determine the most effective strategies to achieve “sustainable delivery system transformation that significantly improves health system performance”3 ^(p1) with the expectation that they would know what would work best in each state's context. One review criterion for selecting Round 1 Model Test states was that they offered a credible environment in which to expand value‐based payment (VBP) models because of their prior experience and supportive legislation.

Arkansas, Massachusetts, Maine, Minnesota, Oregon, and Vermont received awards ranging from $33 million to $45 million, representing a total investment of more than $250 million. In April 2013, they began their 42‐month SIM Initiative Model Test cooperative agreements. These states leveraged in‐kind contributions of provider, payer, and state agency stakeholders in their initiatives’ design and governance. Although these six states were similar in their readiness to capitalize on this funding opportunity, they were geographically diverse, had a heterogeneous mix of pre–SIM Initiative health care delivery reform activities, and advanced a variety of SIM‐funded strategies.

As part of the SIM Initiative, we conducted an independent evaluation that addressed the following questions: How did states combine financial investment and policy development to spread new payment and service delivery models to publicly insured individuals? What distinguishing factors were associated with states’ progress toward multipayer alignment around VBP models? What targeted strategies, inclusive of VBP models, were effective in encouraging providers’ changes in care delivery to enhance care coordination? How did states increase state and provider accountability for patient outcomes by spreading VBP models to more providers? This article provides answers to each of these questions.

Methods

This summary of methods for qualitative data collection and analysis during the SIM Initiative evaluation is described in more detail elsewhere.4, 5, 6

Data Collection

Data collection consisted of document review, monthly discussions with state officials leading the SIM Initiative in each state, key informant interviews, and focus groups with providers and consumers (with composition of focus groups varying by state and year). Document review to extract information on states’ plans, activities, and self‐reported successes and challenges began shortly after the evaluation contract was awarded in 2013, and it included states’ annual and quarterly reports to the Innovation Center, operational plans, and relevant news articles. Quarterly reports to the Innovation Center also included states’ self‐reported reach of VBP models among different payer populations (Medicare, Medicaid, commercially insured populations, and self‐insured populations, varying by state) and providers serving those populations (again, data available varied by state). Beginning in April 2014, we supplemented document review with monthly calls, usually an hour long, among the evaluation team, the state's SIM Initiative leadership, and the state's Innovation Center project officer for the SIM award to discuss state implementation and self‐evaluation updates and gather more in‐depth information on select topics of interest for the evaluation.

On‐site data collection took place during three sets of site visits to each state over the course of the SIM Initiative, in 2014, 2015, and once close to the end of each state's SIM Initiative (which occurred between September 2016 and January 2018, with variation by state occurring due to no‐cost extensions to the period of performance). Each year and in each state, the state evaluation teams conducted 20–30 interviews and conducted 4–8 focus groups with consumers and health care providers expected to have experienced some part of the SIM Initiative, either through providers’ participation in a VBP model (eg, Medicaid beneficiaries attributed to a Medicaid accountable care organization [ACO] or providers affiliated with a Medicaid ACO), or through another delivery system change supported with SIM funds. More information on data collection is available in the Appendix.

Data Analysis

Each year, state evaluation teams triangulated themes from different data sources (key informant interviews, focus groups, state evaluation calls, and document review that took place that year). We did not use qualitative coding software; instead we documented all inputs by each evaluation domain of interest and synthesized within the context of each perspective (eg, state official, payer, provider, consumer). The domains were (1) strategies and policy levers to support health care delivery system and payment models; (2) stakeholder engagement; (3) behavioral health integration with primary care; (4) quality measurement and reporting; (5) health information technology (IT) and data analytics infrastructure; (6) practice transformation and workforce development; (7) population health; and (8) sustainability. The cross‐state analysis relied on state‐specific case studies. We compared themes across states within each of these domains and determined patterns that emerged from differences between state context and themes. Multiple staff responsible for state‐specific analysis reviewed the original results from this cross‐state examination for accuracy and consensus, and at least one member of each state‐specific evaluation team reviewed this study for accuracy.

What Policies and Investments Resulted from the SIM Initiative in Round 1 States?

The SIM Initiative required states to implement new delivery and payment models to address outcomes for publicly insured individuals (under Medicare, Medicaid, or the Children's Health Insurance Program). To comply with that requirement, all SIM Round 1 states either broadened an existing Medicaid VBP model, launched a new Medicaid VBP model (alone or in coordination with other payers), or in the case of Oregon, continued support for a statewide delivery reform model (the Patient‐Centered Primary Care Home [PCPCH]) and Medicaid care delivery model (the Coordinated Care Model [CCM]), and included the CCM in state employee health plan contracts. States also sought to optimize providers’ performance and willingness to participate in VBP models. States pursued this goal by financially supporting health IT, data analytics, technical assistance, and workforce development. Additionally, states supported administrative tasks, such as convening stakeholder discussions and program evaluation, which contributed to new policy development codified in state law in Arkansas and Oregon (see Table 1 and more description later in the paper).

Table 1.

SIM‐Supported Activities Across States, by the 2 Most Prominent Funding Domains, 2013–2018

	Arkansas	Maine	Massachusetts	Minnesota	Oregon	Vermont
Award amount	$42 million	$33 million	$44 million	$45 million	$45 million	$45 million
Period of performance	April 2013 to September 2016	April 2013 to September 2017	April 2013 to April 2018	April 2013 to December 2017	April 2013 to May 2017	April 2013 to June 2017
Delivery system reform and payment models	65% of funds Multipayer PCMH and 14 EOCs Individualized workforce training to practices State law authorizing LTSS payment model Planned Medicaid LTSS payment model (PASSE)	58% of funds Medicaid HH and BHH Medicaid ACO CHW pilot Individualized assistance to BHHs to improve care coordination Learning collaboratives	82% of funds PCPRI, a Medicaid PCMH with BH integration, shared savings (2014‐2016) Pilot Medicaid ACOs (2016‐2017) Full Medicaid ACO launched 2018, with 3 ACO models and BH and LTSS Community Partners	34% of funds Medicaid ACO Medicaid BHH Practice transformation support to HCHs Grants to ACHs CHW pilot Learning collaboratives Toolkits on integrating new workforce types	50% of funds Multipayer participation in coordinated care model Multipayer PCMH Established Transformation Center to provide TA on APMs Regulations on insurers to require primary care investments	42% of funds Multipayer ACO and PCMH Planned All‐Payer ACO model Learning collaboratives
Data analytics and health IT	20% of funds Performance feedback reports via multipayer portal Policies to encourage providers' use of real‐time alerts of patients' hospital and ED use	28% of funds Performance feedback reports from Medicaid and assistance to practices to help interpret them Reports from Medicaid to ACO providers for use in managing attributed patients Connected BH organizations to state HIE TA to providers on how to use HIE connections System to offer real‐time alerts of patients’ ED and inpatient use to Medicaid care managers	16% of funds Performance feedback reports from Medicaid to providers participating in PCPRI Reports from Medicaid to ACO providers for use in managing attributed patients Policies and regulations to increase use of HIE E‐Referral to facilitate referrals from primary care to community resources who then send feedback to primary care	42% of funds Reports from Medicaid to ACO providers for use in managing attributed patients Grant program for providers to use data analytics Grant program for providers to exchange health information Policy development via the E‐Health Roadmapa	26% of funds Performance feedback reports to providers contracted by Coordinated Care Organizations System to offer real‐time alerts of ED and inpatient use to nonhospital providers	24% of funds Grant program for providers to develop data analytics System to offer real‐time alerts of patients’ ED and inpatient use to nonhospital providers Data repository for BH data Analysis of gaps in health IT infrastructure

Abbreviations: ACH, Accountable Communities for Health; ACO, accountable care organization; APM, alternative payment model; BH, behavioral health; BHH, behavioral health home; CCO, Coordinated Care Organization; ED, emergency department; EOC, episode of care; HCH, health care home; health IT, health information technology; HH, health home; HIE, health information exchange; LTSS, long‐term services and supports; PASSE, Provider‐led Arkansas Shared Savings Entities; PCMH, patient‐centered medical home; PCPRI, Primary Care Payment Reform Initiative; SIM, State Innovation Models; SSP, Shared Savings Program; TA, technical assistance.

^aThe e‐Health Roadmap provides use cases and recommendations for the adoption and use of health information technology in Minnesota's four “priority settings.” The Minnesota e‐Health Roadmap for Behavioral Health, Local Public Health, Long‐Term and Post‐Acute Care, and Social Services (August, 2016). The Minnesota Department of Health website: https://www.health.state.mn.us/facilities/ehealth/roadmap/index.html. Accessed February 22, 2019.

The percentage of SIM funding allocated to various domains was self‐reported by states. The remaining percentage of SIM funds beyond these 2 funding domains was largely allocated to funding other operations needed to administer activities related to the award. Source for award amounts from State Innovation Models Initiative: Model Test Awards Round One. Centers for Medicare and Medicaid website. https://innovation.cms.gov/initiatives/State-Innovations-Model-Testing/index.html. Accessed February 9, 2019.

The specific design of VBP models varied considerably, with some states implementing multiple approaches simultaneously to address varied policy issues and goals. States used SIM funds to design or enhance ACO‐type models with one‐sided risk (potential for shared savings only, no risk of loss) in Maine, Minnesota, and Vermont (in its Medicaid Shared Savings Program [SSP] operational 2014–2016). ACO‐type models with two‐sided risk were developed in Maine, Massachusetts (fully implemented in 2018), Minnesota, and Vermont (in its Medicaid Next Generation model piloted in 2017 and fully implemented in 2018). Alternatively, three states implemented or built on primary care–based models. These were the Primary Care Payment Reform Initiative (PCPRI) in Massachusetts (2014‐2016), with two‐sided risk for some costs, and a Medicaid patient‐centered medical home (PCMH) model in Arkansas with optional one‐sided risk; the PCPCH model enlarged in Oregon had no financial risk component. Health home models were also employed for Medicaid beneficiaries focused on people with chronic illness or behavioral health conditions in Maine and Minnesota (no financial risk). A model of payments for 14 episodes of care (EOCs) was implemented in Arkansas, with retrospective calculation of shared savings or losses (similar to bundled payments). Payers aligned with Medicaid in the Arkansas PCMH and EOC models, the Oregon CCM (via state employee health plan purchasing contracts), and the Vermont ACO SSPs.

Although states emphasized implementation and refinement of VBP models, they also focused on health IT and data analytics, which they viewed as a key infrastructure support necessary to operate VBP successfully. SIM funds often were used to strengthen Medicaid agencies’ ability to conduct health care claims‐based data analytics. In the case of Minnesota, SIM funds were distributed as grants to ACOs participating in its Medicaid ACO model, with the goal of improving their own data analytic capabilities. Most data analytics were used to give performance feedback and person‐level data to providers on patients whose quality of care and expenditures contributed to the calculation of incentives under VBP models. A state official in Arkansas explained the importance of these types of data: “Physicians [are] actually looking at things. Especially the older physicians, who have been doing the same thing for 30 years or more. They see there's another way to do it and are actually seeing their results as well…. Enlightenment is the one word I would use to describe it.”

Additionally, most states used SIM funds to enhance electronic health information exchange (HIE) across providers. Maine invested in connecting 20 behavioral health organizations, 18 of which participated in the new behavioral health home (BHH) model, to the statewide HIE. Maine also offered technical assistance on how to use the HIE; as one state official explained, “You can have a great new tool and nothing happens with it because there aren't resources to figure out how to integrate it within the current workflow.” Oregon and Vermont built new systems, or contracted with developers of existing systems, to deliver real‐time alerts to primary care providers of emergency department (ED) and inpatient use. Maine built similar alerts for its Medicaid care managers. Minnesota offered grants to groups of providers to plan and develop exchange of data locally. Two states (Arkansas and Massachusetts) changed their policies to facilitate greater connectivity to state HIEs.

States also used SIM funds for workforce development and training opportunities. Maine and Minnesota implemented community health worker (CHW) pilots, and Maine developed and implemented training to help define CHWs’ core competencies and role on care teams. These and other states offered individualized technical assistance and peer‐to‐peer learning sessions in the form of learning collaboratives.

Finally, states used SIM funds to (1) convene stakeholders for the purposes of encouraging voluntary collaboration, participating in the initiatives’ governance, or soliciting advisory guidance, and (2) self‐evaluate delivery and payment models supported by or implemented under the SIM Initiative for the purpose of rapid‐cycle learning and continuous improvement. Stakeholders cited these activities as important contributions to health policy in their states:

“While SIM funding ends, this work will continue to be ongoing, largely because of infrastructure and relationships.” —Oregon state official

“The SIM project, by having work groups, even though it was a complex project and resource intensive, created mechanisms for bring[ing] representatives of diverse groups together to talk about significant issues and recommendations to take.” —Vermont stakeholder

Two states illustrate how their work under SIM related to new state legislation. Arkansas spent SIM Initiative funds to convene providers of long‐term services and supports (LTSS), behavioral health care, and care for people with developmental disabilities (DD) to design a health home model for Medicaid beneficiaries. Although the state ultimately decided not to adopt the health home payment model, those discussions laid the groundwork for 2017 state legislation authorizing the Provider‐led Arkansas Shared Savings Entity (PASSE) model of care, under which specialty managed care plans coordinate physical health care with behavioral health and DD community services.7 State officials in Oregon credited the findings from their SIM‐funded self‐evaluation, which found a reduction of expenditures associated with receiving services from a PCPCH (Gelmon et al., 2016),8 as a contributing factor to passage of state legislation (S.B. 934).9 This law required Medicaid CCOs to financially support PCPCHs in their networks if they offer payments to other practices in their networks participating in other PCMH models, authorized the state's insurance regulator to place similar requirements on commercial insurers, set a minimum of 12% of medical expenditures to be spent on primary care for all health insurance carriers—including CCOs, state employee health plans, and others—by January 1, 2023, and mandated annual reporting of these entities’ percent spent on primary care.9

The combination of trying to address many aspects of the health care delivery system in the midst of turnover in state leadership contributed to states requesting more time to spend their SIM funds. Five out of the six Round 1 states (all but Arkansas) requested and received no‐cost extensions that allowed them to spend their SIM Initiative award beyond the original 42‐month period of performance.

What Factors Contributed to States’ Progress in Multipayer Alignment on Value‐Based Payment Models?

The six Model Test states selected for the first round of SIM Initiative awards were chosen in part because they had previous experience and capacity from preexisting delivery or payment models supported by multiple payers. However, this experience did not necessarily lead to multipayer alignment on new VBP models designed or implemented with support from the SIM Initiative; only Arkansas, Oregon, and Vermont achieved in having both Medicaid and commercial insurers align around VBP models in their contracts with providers. Medicare did not participate as a payer in any SIM Round 1 models.

As shown in Table 2, several factors distinguished these three states with multipayer alignment. Arkansas and Vermont shared characteristics that may have facilitated commercial health plans’ alignment around a common VBP model (EOC and PCMH in Arkansas, ACO SSPs in Vermont). First, both states have a single dominant commercial insurer in the state (Blue Cross Blue Shield). Second, Medicaid and commercial insurer participants agreed on some flexibility in model design. In Arkansas, commercial payers participating voluntarily in the EOC model could implement a subset of the 14 EOCs developed in collaboration with Medicaid. In Vermont, although both the Medicaid and commercial SSPs were modeled after the Medicare Shared Savings Program, these payers selected quality measures that were more suitable for non‐Medicare populations; additionally, these payers were aligned in the selection of their quality measures with only one difference, developmental screening within the first three years of life, a part of the Medicaid Child Core Set.

Table 2.

Characteristics and Strategies That Distinguished Arkansas, Oregon, and Vermont in Achieving Alignment of Value‐Based Payment Models Across Payers

	States That Achieved Multipayer Alignment			States That Did Not Achieve Multipayer Alignment
Characteristic or Strategy	Arkansas	Oregon	Vermont	Maine	Massachusetts	Minnesota
State legislation compelling qualified health plans to participate in SIM‐supported payment models	X
Few commercial payers in state	X		X
Collaborative payment model design across commercial payers and Medicaid offered some degree of flexibility in model implementation	X		X
Use of state health plan employee contracting to impose new model on commercial payers		X
Preexisting delivery or payment model supported by multiple payers in state	X	X	X	X	X	X
State legislation prior to 2013 setting goals for adopting new payment models			X		X	X
Majority of SIM funds spent on supporting new payment models	X			X	X
SIM funds helped convene payers as stakeholders in voluntary payment model alignment efforts	X	X	X	X	X	X

Other policy levers also made a difference. In Arkansas, payers were compelled by state law to participate in the state's PCMH model for a portion of their population. Rule 108, which expanded Medicaid eligibility under the Private Option, required qualified health plans to pay PCMHs recognized by Medicaid a per member per month (PMPM) payment, the same as Arkansas Medicaid did, in addition to the existing payment mechanism. Oregon successfully incorporated elements of the CCM in commercial health plan offerings through its procurement of health plans for state employees, thereby aligning operating principles for Medicaid's CCOs and health plans offered to state employees.

Despite the states’ best efforts, not all attempts at multipayer alignment were successful. For example, state laws that set a health policy agenda to bring Medicaid into better alignment with other existing initiatives (ie, in Medicare or among commercial payers) could signal the state's commitment to aligning across payers and garner reciprocal commitments from other payers to align with Medicaid voluntarily. Indeed, three states (Minnesota, Vermont, Massachusetts) passed such laws prior to the SIM Initiative's start in 2013. In Minnesota, the First Special Session Article 16 Section 19 in 2010 amended the 2008 Health Reform Act to require the Department of Health to test new delivery models in Medicaid. In Vermont, also in 2010, Act 128 established goals for health reform and expanded the Blueprint for Health model statewide. Perhaps most boldly in terms of setting specific goals for the reach of VBP models—prescient even before the SIM Initiative set similar goals—was Massachusetts's Chapter 224, passed in 2012, which required 80% of Medicaid beneficiaries receive care under alternative payment contracts by July 1, 2015 (goal not met). The law also required implementation of alternative payment methods in the state's health insurance marketplace and state employee health plans, and it addressed changes in many other sectors of Massachusetts's health care infrastructure such as health IT and workforce.10 However, these state laws did not produce voluntary payer participation.

Preexisting delivery or payment models, financial resources devoted to model development, and convening of stakeholders to align efforts also may be necessary, but they alone did not result in voluntary multipayer alignment. All states possessed each of these factors—and Arkansas, Maine, and Massachusetts even spent most of their SIM funds on model development—yet only Arkansas, Oregon, and Vermont succeeded in achieving voluntary multipayer alignment.

What Targeted Strategies Did States Adopt to Encourage Enhanced Care Coordination?

A key premise of the SIM Initiative was that greater reach of VBP models would spur care delivery change to enhance care coordination. States also sought to accomplish this goal by investing in additional infrastructure. The combination of these strategies to help providers deliver more coordinated care resulted in providers in some states mentioning more use of electronic health information and greater use of care teams within primary care practices over time during the SIM Initiative.

Strategies Related to Health IT

First, Maine, Minnesota, Oregon, and Vermont—the four states where providers noted increased access to and use of electronic health information—invested more than 20% of the state's total SIM award in health IT infrastructure (Table 3). These funds helped providers in Maine and Oregon achieve connections to the statewide HIE. In Maine, new connectivity to the state's HIE, which was prohibitively expensive for behavioral health providers without funding support from the SIM Initiative, was viewed positively by providers, state officials, and advocates; as one behavioral health provider said, “I just love HealthInfoNet. I can tell if my client has been to the emergency room or admitted.” Behavioral health providers also noted how access to a patient's medical records in the HIE helped them develop and modify care plans and improve care coordination with primary care. In Oregon, a provider reacted to their care manager's use of the Emergency Department Information Exchange, which SIM funds helped to develop, implement, and spread, by saying that “it's been great for helping with transitions of care,” since they use it for patient follow‐up after discharge.

Table 3.

Targeted Strategies That Distinguished Maine, Minnesota, Oregon, and Vermont in Encouraging Providers’ Use of Electronic Health Information for Care Coordination

	Providers Noted More Use of Electronic Health Information at End of SIM Initiative				Providers Did Not Note More Use
Strategy	Maine	Minnesota	Oregon	Vermont	Arkansas	Massachusetts
Targeted
More than 20% of total SIM funds dedicated to health IT	X	X	X	X
SIM funds supported ACO‐like payment models, which offer financial incentive for within‐ACO provider organizations to coordinate care	X	X		X		X
SIM funds invested in third‐party service to communicate electronic hospital admission, discharge, transfer notifications	X		X	X
SIM funds for technical assistance to connect providers to health information exchange	X		X	X		X
SIM‐funded grants to provider organizations to help plan electronic exchange of health information		X
General
Policies to encourage primary care physicians to access real‐time alerts of hospital and ED use					X	X
SIM funds used to develop policies aimed at improving data quality in health information exchange				X
SIM funds used to develop health IT privacy and security policy		X
Preexisting statewide health information exchange platform	X	X	X	X	X	X

Abbreviations: ACO, accountable care organization; ED, emergency department; health IT, health information technology; SIM, State Innovation Models.

SIM funds in Maine, Oregon, and Vermont also facilitated third‐party services to communicate electronic hospital admission, discharge, transfer (ADT) notifications. A consistent theme in interviews and focus groups with primary care providers was the increased use of electronic event notifications—ie, ADT and ED use alerts—to focus their patient outreach and post‐hospital‐visit follow‐ups. In Vermont, which subsidized access to this service for providers who did not receive these alerts in their electronic health records already, the ACO comprising mostly federally qualified health centers noted, “It was a tool they found so valuable they wanted to make sure they have it in their toolbox for all patients, not just ones [the] ACO flagged.”

States also offered technical assistance to providers to connect to the HIE: assistance to behavioral health organizations in Maine; assistance to behavioral health and LTSS providers in Massachusetts; and in Minnesota, toolkits for behavioral health, public health, social service, and LTSS providers, as well as grants to ACO‐affiliated providers to help plan local exchange of electronic health information with one another. Additionally, the presence of ACO‐like payment models, which offer financial incentives for within‐ACO provider organizations to coordinate care, may have made a difference in these states, but likely in combination with other factors.

In contrast, states with fewer supports to providers did not achieve the same consistency in providers’ response with regard to use of electronic health information. Even though these states created policies meant to encourage primary care physicians to access real‐time alerts of hospital and ED use connections to the HIE (ie, Arkansas through PCMH requirements and Massachusetts through streamlining consent policies), these policies alone did not increase providers’ use of the electronic information systems.

Strategies Related to Broadening Care Teams

Greater use of care teams within primary care practices was a consistent theme in Maine, Massachusetts, Minnesota, and Oregon (Table 4). These states used specific strategies to encourage incorporation of nontraditional health care providers in the primary care workforce. Maine and Minnesota invested SIM Initiative funds into efforts targeted at increasing use of CHWs. Both states piloted CHW integration in primary care, Maine developed and implemented a curriculum for clinics on how to define the CHW role and how to integrate them in primary care practice, and Minnesota produced toolkits for primary care practices on integrating CHWs and other new types of health professionals into practice. Minnesota also certifies CHWs, who have been able to bill Medicaid since 2009, and providers that incorporated them into their care teams found them invaluable:

“CHWs is the glue that holds our model together. This population is inclined to use everything but primary care. But having someone like a CHW who looks like them and understands their life experiences are very important.” —Minnesota provider

Table 4.

Targeted Strategies That Distinguished Maine, Massachusetts, Minnesota, and Oregon in Encouraging Providers’ Use of Care Teams in Health Care Delivery

	Providers Noted More Use of Care Teams at End of SIM Initiative				Providers Did Not Note More Use
Strategy	Maine	Massachusetts	Minnesota	Oregon	Arkansas	Vermont
Targeted
SIM funds for pilot tests of integrating community health workers into care teams	X		X
SIM funds for training to clinics on integrating the community health worker role in primary care practice	X			a
SIM funds for toolkits to show how new health care workers can be integrated into care teams			X	a
SIM‐supported payment model required colocation of behavioral health and primary care providers		X
SIM‐supported payment model offered care management fee		X
General
Peer‐to‐peer learning collaboratives on delivery models or care coordination	X		X	X		X
Individualized technical assistance from vendors		X	X	X	X

X = Strategy present in the state.

^a= Strategy present but not SIM‐funded.

In a contrasting strategy, Massachusetts leveraged the payment model developed under the SIM Initiative to enhance the use of care teams. Under the PCPRI, adopted by 28 provider organizations covering 62 sites during the program's operations March 2014 through December 2016, clinics received a risk‐adjusted care coordination payment that some providers used to hire patient navigators and other care coordination staff. Providers in focus groups noted how these staff communicated with high‐risk patients with multiple health care needs:

“Coaches, navigators, CHWs [community health workers], all these different words we use to describe the same thing. When those people are co‐located with us at the center in the community, I find so much more value rather than some nurse sitting in some office somewhere in Seaport [off‐site] calling my patient occasionally.” —Massachusetts provider

Primary care providers in Oregon similarly praised the flexibility that supportive ancillary staff, including Traditional Health Workers (THWs), had to address patients’ needs for services other than physical health care, although efforts to train and certify THWs were separate from SIM‐funded activities.

Behavioral health providers also became part of a broader care team in some states. Under the PCPRI, providers that opted for a higher PMPM payment were required to have at least one master's‐ or doctoral‐level behavioral health provider on‐site for 40 hours per week and a psychiatrist eight hours a week, as well as demonstrate the ability to accommodate behavioral health provider appointments within 14 days of a request. Although the state ultimately ended the PCPRI, providers in Massachusetts consistently cited the benefit they received from changing their on‐site care team to include behavioral health. Similarly, Medicaid BHH providers in Maine praised the payment model they received as a result of adopting this model of care. Citing the monthly capitated rate they received for each BHH enrollee, one BHH provider said, “[T]he BHH can be more of a wellness model.… [Y]ou are not chasing a productivity model so you can do a lot more programming and communication and coordination of services.” Additionally, health homes in Maine—incentivized to coordinate with BHHs—also found value in this model: “It [team‐based care] decreases the no‐show rate for mental health patients but also decreases the stress of the medical physician.”

Despite the flexible payment structures offered under VBP models, financial incentives under payment models alone did not cover the cost of hiring additional staff dedicated to care coordination, according to providers from most states. This may have been a barrier to expanded care teams if not overcome by improved efficiency, other infrastructure, or payment model features.

Four states offered peer‐to‐peer learning collaboratives on implementing new delivery models or enhancing care coordination. Although providers who participated in learning opportunities around these topics all valued peer‐to‐peer sharing to learn best practices and network with other providers and community partners, this effort didn't make the difference in reporting greater use of care teams. Still, learning collaboratives may have affected other types of care coordination across providers, as they offered yet one more opportunity to foster relationships using SIM Initiative dollars. As one Vermont provider said, the learning collaborative “created a slightly more advantageous way of thinking about shared community engagement around complex issues.… I would like to think that translated into, not only how do we work more collaboratively with our community partners, but also how we communicate within our organization.”

How Did the SIM Initiative Help States Increase State and Provider Accountability for Patient Outcomes?

The SIM Initiative helped states increase state and provider accountability for patient outcomes first by expanding VBP model participation to more providers, and then, toward the end of the initiative, by growing the degree of risk offered to providers under those models.

Expanding VBP Model Participation to More Providers by Offering Flexibility and Data

The percentage of Medicaid beneficiaries receiving care under Medicaid ACO–participating providers increased to 20%, 46%, 56%, and 58% in Maine, Vermont, Massachusetts, and Minnesota, respectively, with only Minnesota having any Medicaid ACO program prior to the SIM Initiative. All states committed to the VBP models they launched or grew during the initiative, through continued use of Medicaid state plan amendments and waivers. Each of these states used flexibility in payment model design and new data availability funded under the SIM Initiative to attract new providers to participate.

Specifically, some states built flexibility into their VBP models by giving providers a choice in the degree of financial risk they could assume. Maine and Vermont gave their Medicaid ACOs the choice of whether to have one‐sided or two‐sided financial risk models, and all ACOs in both states chose one‐sided models. Minnesota recruited additional providers to participate in its model in each year, offering some flexibility in whether shared financial risk was one‐sided or two‐sided for participating providers, and in the quality measures for which they were held accountable, so that rural, smaller, or specialty providers could participate as equally as the large vertically integrated health systems in the state.

Making new data available to providers participating in VBP models was another approach used to make participation more attractive. Maine offered its ACO providers new performance feedback reports and technical assistance to help interpret them, and Vermont improved its data in the statewide HIE for its participating ACOs. Minnesota's SIM Initiative supported activities that made more beneficiary‐level data available to providers who contracted with the state as Medicaid ACOs, both in the form of grants to help Minnesota Medicaid ACOs enhance their capability to run their own data analytics on their Medicaid patients, and by compiling data on Medicaid beneficiaries regardless of Medicaid MCO enrollment, a resource for managing their Medicaid patient panel that providers hadn't had previously. As one provider organization in Minnesota said, “[D]ata through the [Medicaid ACO] gives us something to work with.… It's helping us develop the skills of data analysis and how that's driving patient care.”

Expanding Degree of Risk Offered to Providers

States refined their specific VBP models over time, with the result of increasing provider accountability for financial and quality outcomes. The incremental changes states made were generally aimed at moving more providers to two‐sided financial risk models for their Medicaid populations, as in Massachusetts's ACO model, Minnesota's “2.0” version of the Medicaid ACO, and Vermont's All‐Payer ACO model.

After limited uptake of the PCPRI model, Massachusetts switched gears toward the end of its SIM award and spent its remaining SIM funds on developing a new Medicaid ACO with significant levels of provider accountability. The state worked in collaboration with many provider and payer stakeholders to ensure the model would have substantial participation. One ACO leader said of the stakeholder engagement process: “The state works with us in a way that shows they are not just giving us lip service. I don't get everything I want on the provider side, but the engagement and willingness to discuss has been terrific.” Ultimately, Massachusetts offered three different types of ACO arrangements to providers to facilitate broad provider participation, and it worked out numerous issues during the pilot ACO phase, paving the way for ACO providers to receive accurate data on their patient panel. Although the Medicaid ACO model officially launched toward the end of the SIM award, it is expected to reach more than half of Medicaid beneficiaries in the state, with the majority of participating ACOs electing prospective capitated payment arrangements with two‐sided risk.

What Challenges Did States Face in Implementing Their Statewide Health Care Transformation Plans?

States did not achieve all of the Innovation Center's goals for the SIM Initiative, and during the course of the initiative, several states reassessed their original innovation plan and reallocated resources as a result. In general, states fell short of meeting the SIM Initiative's expectations for garnering multipayer alignment around VBP models and, regardless of payment model, alignment on quality measure reporting required of providers. Stakeholders also noted ongoing concerns about restrictions on the ability to share patient data across all providers who care for patients, especially related to exchanging mental health and substance abuse treatment data that have specific protections under federal law. Finally, providers expressed clear concerns about the unintended consequences from financial accountability, and consumers reported mixed reactions to changes instigated by VBP models.

Multipayer Alignment Around VBP Models

As one state official in Minnesota said, “We will have them around the table, they will come and listen and speak in generalities—you know, ‘We are committed to payment reform’—but in terms of what does that specifically look like in your organization, or what are your goals, or how can we think about alignment and what does alignment even mean … we don't get very far.” Commercial payers in Arkansas, Maine, and Minnesota identified several reasons they were not interested in adopting VBP‐oriented contracting with providers that aligned with Medicaid's VBP models. Commercial payers that had lines of business in more than one state found it impractical to tailor a VBP model to align with just one state. Also, with local businesses as their primary client, these payers wanted flexibility to design VBP models that could respond to those clients’ needs, rather than commit to adhering to a VBP model shared by other payers in the state. Additionally, commercial payers considered some elements of payment design as proprietary. Finally, some commercial payers were concerned that if they invested in a payment model that succeeded in reducing overall utilization and expenditures for a practice's entire patient panel, they would end up subsidizing the care of patients covered by free‐riding payers who were not making similar investments in a new payment model. Even where states did achieve some degree of alignment with commercial payers, it was often for narrow slices of their covered population (eg, the commercial SSP in Vermont covered only people enrolled in the individual and small‐group market; Arkansas commercial payers selected the EOCs they adopted).

Although the Innovation Center provided states the opportunity to submit proposals for CMS to consider for Medicare's alignment with proposed multipayer models, lack of Medicare participation also contributed to challenges in multipayer alignment. Maine officials devised and submitted a plan for Medicare participation, but after receiving feedback from CMS, the state chose to discontinue these plans because it could not meet at that time the needed parameters to engage with Medicare. Still, CMS and Vermont officials worked closely together toward the end of the SIM award to negotiate a separate model from the SIM Initiative that included Medicare as a participating payer (Vermont All‐Payer ACO). Although other state officials were disappointed not to have Medicare at the table to participate in SIM‐related models, ultimately CMS and states saw the value in providers participating in other multipayer Innovation Center models, such as the Comprehensive Primary Care Plus Model (Arkansas, Oregon).

Health IT Implementation

Although efforts to improve health IT to support delivery system and payment models made up the largest category of investment that states made with SIM Initiative funds after delivery system and payment model development, states’ experiences in this area demonstrate how events rarely go as planned. All states pivoted when their original expectations for achieving bidirectional data exchange on time and on budget were met with challenges, such as too‐high costs for connecting key providers to HIEs (Arkansas) or concern over privacy and security laws that regulated sharing of certain types of health information (Massachusetts and Vermont). In most instances, states found an eventual solution but were at times frustrated by delays and shifts in strategy.

Provider Concerns

The lack of multipayer alignment in VBP models meant that providers continued to express concern over the burden of reporting different quality measures in different ways using different reporting mechanisms for different payers. For example, Medicare had a separate set of metrics and reporting requirements (eg, Physician Quality Reporting System, the precursor to Merit‐based Incentive Payment System), and each commercial payer may have its own set of metrics and reporting system. Payers in Maine were unwilling to adopt a common measure set developed by the state under the SIM Initiative, in part because they had already invested in their own measures for monitoring performance. Providers in Oregon were frustrated by having multiple sets of metrics from different plans. In addition to the burden of multiple submissions required and increased number of measure constructs, providers noted concern with having to produce slightly different versions of essentially the same measure based on the requirements of individual payers.

Providers reported several unintended consequences or no improved care delivery from payment model participation. Providers in Massachusetts, Minnesota, and Vermont reported that their time with patients was restricted because of increased reporting burdens of a model, including quality measure reporting. In providers’ own words:

“I have to document it in three different ways in order for it to be compliant. And, I'm doing my notes at home at the end of the day at 9:30 at night.… The only way to get money is to jump through these hoops and to check these boxes, but I see fewer patients in any day so I'm providing less care to needy patients and I'm burning out.” —Vermont provider

“[T]the biggest challenges are time constraints, and having to do more and more of that busy work—making sure you're checking all those boxes all the time. It's less and less time engaged with the patient, and more and more time making sure you're checking all the right boxes and doing all the right things.” —Minnesota provider

Some EOC‐participating providers in Arkansas, who had the most direct financial risk for outlier patient expenditures, were wary about taking on new Medicaid patients with unknown prior health service utilization but who might trigger payment under an EOC. As one Arkansas provider said, “My concern… with some of these programs is that I will be financially penalized for this mother's overuse of emergency services and what I fear that my only response is going to ultimately be … well, I'm not going to be able to provide care for this patient.” Providers in other states expressed concern that they couldn't sufficiently influence consumer behavior in a manner that would improve outcomes for which they were accountable. At the same time, some other providers—especially in states with Medicaid ACO models—were unaware of their participation in an ACO or of any financial incentives for changing their care patterns, and they felt they were still being evaluated on the volume of patients seen. Although health care utilization and expenditure consequences of care delivery may not be top‐of‐mind for busy providers, lack of awareness of payment model participation impedes the expected trajectory for payment model implementation to affect care delivery.

Consumer Concerns

Consumers also reported dissatisfaction with some changes in provider care related to VBP model implementation and states’ SIM Initiatives. Although consumers appreciated same‐day access to primary care providers, they also reported seeing a wider variety of practitioners as a result—an outcome they viewed negatively. In the words of one Massachusetts Medicaid beneficiary, echoed by others in that and other states: “In the four years that I've been going to [name of provider], I've only seen her a handful of times. I usually see a PA; you know, a physician's assistant or a nurse practitioner. I really would like to see my own doctor.” Moreover, where providers had incentive to adhere to evidence‐based care (eg, for back pain or viral common cold treatments), consumers viewed the care they received with some degree of dissatisfaction, because it conflicted with their understanding of the kind of care they should receive (eg, surgery, antibiotics).

What Lessons Can States Take From the SIM Initiative?

States differed in how they combined policies, payment models, and health system infrastructure development to shape their SIM Initiatives, but their collective experience offers the following lessons learned for other states to consider when making their own policy decisions about payment model designs and future investments in improving care delivery and care coordination.

First, in the five states that implemented more than one SIM‐supported VBP model, it was imperative for state leadership to demonstrate how these models were complementary, not competing, especially in light of the perceived burdens of model participation, such as increased quality measurement and reporting. States accomplished this coordination between models in several ways. The Arkansas Medicaid PCMH model set care management standards in the primary care setting, whereas the state's EOC model was viewed as a payment for specialty care (and also some primary care conditions). In Maine, Medicaid health homes received an extra PMPM payment for coordinating with a BHH for the same patients. In response to provider concerns about alignment between the preexisting multipayer Blueprint for Health model and the newly implemented ACO SSPs, Vermont leadership focused on aligning the two models’ goals, definitions, and regulatory requirements, including quality measures used in the pay‐for‐performance aspect of Blueprint for Health. In contrast, Vermont decided to drop its initial plans to use SIM funds to develop an EOC model, given uncertainty on how the EOCs would affect providers participating in the other two existing models.

Second, VBP models can be supported by investments in data analytics resources, such as feedback reports to providers and grants or technical assistance to help providers use those data. As one ACO recipient of a data analytics grant in Minnesota said, “I wouldn't have the data staff I have without the SIM dollars that allowed us to build and make that area more robust. Those are the pieces that allowed us to move from throwing random, raw data … to actually usable, actionable data.… [G]etting that data can help us accomplish a lot of the care delivery reforms that we're trying to do.” Other providers participating in Massachusetts's PCPRI felt that seeing the data feedback prepared them to be successful under future pay‐for‐performance metrics and helped them identify previously unrecognized shortcomings. At the same time, concerns about the accuracy of the data in the feedback reports represented a barrier to making changes based on the data; as one Massachusetts provider said, “Until you have an accurate static panel, it really is hard to really trust the quality measures that are coming our way.”

Third, state Medicaid programs can structure opportunities to help more providers gain experience with VBP models. VBP models in Medicaid implemented in Massachusetts, Minnesota, and Vermont during the SIM Initiative later evolved, offering providers the ability to transition to payment models in which they assumed greater financial risk after gaining experience managing a smaller financial risk. By the time Vermont's Medicaid Next Generation ACO was implemented in 2017 and Massachusetts implemented the Accountable Care Partnership Plan (one of their three Medicaid ACO types) in 2018, the states offered providers prospective, capitated payments. In other states, participation in state VBP models helped situate providers for later federal VBP models; for example, providers in Arkansas's Medicaid PCMH program later participated in CPC+, which started in Arkansas in 2017.

Finally, participation in the SIM Initiative enabled state government leadership to foster partnerships across providers, payers, and state agencies in working toward shared health care system transformation goals. Stakeholders reported that the partnerships developed during the initiative were some of the most valuable outcomes from the SIM Initiative, especially with regard to identifying how relationships can affect key health outcomes like social determinants of health:

“We have many stakeholders that say to us every week this has never been done before; the degree to which you have engaged with us, listened to us, reached out to us has really never been done at Medicaid before. And they feel very involved, knowledgeable, respected, heard, and it was all because of SIM.” —Massachusetts state official

“I would have never guessed at the beginning of SIM that it would be something we say we did, but the narrative has changed in Minnesota about ‘What is health?’ There has been a big shift in the awareness of social determinants, the kinds of relationships that need to be in place; it's not fully due to [the] SIM [Initiative]—those conversations were happening in many other places—but SIM provided a venue and some funding opportunities to accelerate those conversations and to put them into practice.” —Minnesota state official

“The relationships built through [the] SIM [Initiative] have led to alignment and interaction across organizations.” —Vermont state official

Discussion

The final evaluation of the SIM Initiative demonstrates that some components of health system change require strong state regulatory policy and benefit less from financial investment (eg, multipayer coordination on VBP models). Other changes require financial investments to support policy (health IT, data analytics, technical assistance, workforce development). State policies may be necessary to compel commercial payer participation in multipayer alignment around VBP models. Examples from the SIM Initiative states demonstrate the ability of state law to encourage commercial payer participation, such as Arkansas's mandated alignment of qualified health plans with the Medicaid PCMH program and Oregon's rules to prompt greater investment in primary care in Oregon. Financial resources help to build a supportive infrastructure for VBP models, in terms of connecting providers to HIEs and access to electronic event notification services; offering more comprehensive performance feedback reports and peer‐to‐peer learning opportunities; technical assistance on use of health IT and data; and funding pilot tests of integrating new workforce members, including CHWs, who can support broadened team‐based care for increased prevention, medication management, access, and patient follow‐up after hospital visits. These infrastructure changes are helpful in achieving federal goals of increased VBP participation and changing provider behavior to enhance care coordination.

These final SIM Initiative evaluation findings also highlight how policy levers and financial support can help states overcome headwinds that often slow policy progress. As early as September 2013, a review of SIM Initiative states’ experiences hinted at potential barriers to achieving the high expectations placed on SIM states. Commercial payers were reluctant to participate in state‐led conversations about VBP models. State officials looking for multipayer alignment across payment models struggled with the “right balance between standardization and flexibility.”11 ^(p4) Providers expressed skepticism in their readiness to manage financial risk or felt they lacked the tools, such as electronic health records and quality measure reporting, needed to change care delivery. All stakeholders were concerned about the availability of data needed to measure providers on their performance on cost and quality.11 By the end of the SIM Initiative, we learned states could overcome these barriers with a combination of policies and resources.

Although prior delivery and payment reform experience, stakeholder alignment, and general investments in supportive training and infrastructure may lay a critical groundwork for states and providers to engage in health care transformation, it is still evident that these factors alone were not sufficient to drive change, at least within the short term. Instead, targeted strategies helped to distinguish the ability of some states to encourage more providers to adopt use of health IT and care teams to support care coordination. Additionally, the relationships established through the SIM Initiative across providers, payers, and state officials may yet yield lasting changes in the future.

Limitations

The evaluation of the SIM Initiative gathered data from the most heavily involved stakeholders at state agency, payer, and provider organizations, and from volunteer focus group participants among providers and consumers who were identified as participating in SIM‐related VBP models. Although we hope our synthesis across data sources achieves the true balance of perspectives on actual implementation results, our analysis is limited by the fact that participants in our data collection may not be representative, especially given the significant scope of each state's SIM‐related activities.

Conclusion

The scope of the SIM Initiative as stated in the FOA was ambitious: a test of state governments’ ability to drive health policy toward alignment of VBP models, with the goal of giving providers clear and consistent incentives from multiple payers. Leaders in each state undertook this challenge in the midst of changes in policymakers and health reform champions within state government; market‐based forces affecting provider consolidation and competition; evolving federal health policy; and emerging health policy issues such as the opioid epidemic, rising prescription drug costs, and growth in state employee retirements. Despite the challenges inherent in evaluating the effects of an intervention when so many other changes are happening, we believe the experiences of the six Round 1 Model Test states participating in the SIM Initiative offer insight into promising strategies for designing state policy levers that reach a significant number of providers and patients. These strategies are leveraging state law supportive of multipayer alignment around VBP models; connecting providers to electronic health information; testing integration of new care roles; offering providers technical assistance in these areas; and developing partnerships across providers, payers, and community organizations.

Although it may have been too ambitious to accomplish all these goals in the short window of time allotted to the periods of performance, the SIM Initiative may have helped states and providers move farther along the continuum from fee‐for‐service to VBP models. As a result, these states are well positioned to participate in newer models (including the Vermont All‐Payer ACO model, developed during the SIM Initiative) that hold states accountable for population health and health care system goals. More evaluation of these delivery changes and others implemented in the SIM Initiative Round 2 Model Test states and other state‐led models will prove useful in moving from volume‐based to value‐based care nationwide.

List of Acronyms

• ACO: accountable care organization

• BHH: behavioral health home

• CCM: Coordinated Care Model

• CCO: Coordinated Care Organizations

• CHW: community health worker

• CMS: Centers for Medicare & Medicaid Services

• DD: developmental disabilities

• EOC: episode of care

• ED: emergency department

• HIE: health information exchange

• LTSS: long‐term services and supports

• PCMH: patient‐centered medical home

• PCPCH: patient‐centered primary care home

• PCPRI: Primary Care Payment Reform Initiative

• PMPM: per member per month

• SIM: State Innovation Models

• SSP: Shared Savings Program

• THW: traditional health worker

• VBP: value‐based payment

Methods

We conducted three site visits to each state, except Massachusetts. Because the state changed the VBP model it implemented over the course of the SIM Initiative, Massachusetts had four site visits, with the last two occurring in November 2016 and January 2018. During each visit, state‐specific evaluation teams traveled to more than one region of a state—usually at least one urban and one rural area—to conduct a total of 20–30 key informant interviews with the state's SIM Initiative leadership, other state officials, commercial payers, providers and provider associations, consumer representatives, and organizations contributing to some part of the health system infrastructure (eg, entities governing electronic HIEs, workforce development initiatives). Different discussion guides were developed for each major type of stakeholder and tailored for each state, but they generally focused on the initiative's implementation successes, challenges, lessons learned, and potential for sustainability, in addition to specific topics: policy impacts; stakeholder participation; changes to health care delivery; implementation of new payment models; quality measurement and reporting activities; population health efforts; health information technology (health IT) and other infrastructure investments; and workforce development and technical assistance or learning opportunities offered to providers. After each interview, the lead interviewer and the note taker reviewed each set of notes for accuracy, comparing to an audio recording if necessary.

In most states, four consumer and four provider focus groups occurred during each on‐site data collection period, for a total of more than 130 focus groups involving more than 1,000 participants over the course of the evaluation. The purpose of the focus groups was to understand consumers’ and providers’ current experience and reflections of care delivery during the SIM Initiative and changes they have observed over time. To capture this, the moderator's guide addressed consumer and provider perspectives on quality of care, care coordination, use of health IT, and, in provider groups, reaction to opportunities for participation in new delivery systems, payment models, or other infrastructure supports (eg, training and technical assistance) related to the state's SIM Initiative. Members of the evaluation team that observed each focus group reviewed, cleaned, and analyzed transcripts from that focus group.