From patient outcomes to system change: Evaluating the impact of VHA's implementation of the Whole Health System of Care

Abstract

Objective

To describe how a partnered evaluation of the Whole Health (WH) system of care—comprised of the WH pathway, clinical care, and well‐being programs—produced patient outcomes findings, which informed Veterans Health Administration (VA) policy and system change.

Data Sources

Electronic health records (EHR)‐based cohort of 1,368,413 patients and a longitudinal survey of Veterans receiving care at 18 WH pilot medical centers.

Study Design

In partnership with VA operations, we focused the evaluation on the impact of WH services utilization on Veterans' (1) use of opioids and (2) care experiences, care engagement, and well‐being. Outcomes were compared between Veterans who did and did not use WH services identified from the EHR.

Data Collection

Pharmacy records and WH service data were obtained from the VA EHR, including WH coaching, peer‐led groups, personal health planning, and complementary, integrative health therapies. We surveyed veterans at baseline and 6 months to measure patient‐reported outcomes.

Principal Findings

Opioid use decreased 23% (31.5–6.5) to 38% (60.3–14.4) among WH users depending on level of WH use compared to a secular 11% (12.0–9.9) decrease among Veterans using Conventional Care. Compared to Conventional Care users, WH users reported greater improvements in perceptions of care (SMD = 0.138), engagement in health care (SMD = 0.118) and self‐care (SMD = 0.1), life meaning and purpose (SMD = 0.152), pain (SMD = 0.025), and perceived stress (SMD = 0.191).

Conclusions

Evidence developed through this partnership yielded key VA policy changes to increase Veteran access to WH services. Findings formed the foundation of a congressionally mandated report in response to the Comprehensive Addiction and Recovery Act, highlighting the value of WH and complementary, integrative health and well‐being programs for Veterans with pain. Findings subsequently informed issuance of an Executive Decision Memo mandating the integration of WH into mental health and primary care across VA, now one lane of modernization for VA.

What is known on this topic

• Policy decisions regarding large‐scale system changes often cannot wait for research results to inform policy.

• The Veterans Health Administration (VA) has been working on the implementation of the Whole Health System (WHS) of Care, an approach to health care that focuses on what matters most to patients to empower and equip them to take charge of their health and well‐being.

• Little is known about the impact of VA's implementation of the WHS on important Veteran outcomes.

What this study adds

• Collaboration between researchers and operational partners informed the structure and relevant outcomes for timely evaluation.

• Use of WHS services resulted in improvements in patient‐reported outcomes and reductions in opioid use for Veterans with chronic pain.

• Findings presented to key stakeholders led to policy changes to expand the implementation of the WHS.

1. INTRODUCTION

The United States Veterans Health Administration (VA) is the nation's second largest integrated health care system, serving approximately 8.9 million military Veterans. Since the landmark 2001 Institute of Medicine Report outlined recommendations for improving the nation's health care systems, VA has been a leader in quality improvement, including research on and implementation of patient‐centered care and evidence‐based approaches to care. ¹ , ² Recently, it has also become leader as a learning health care organization, committed to the continuous quality improvement made possible through intentional collaborations between researchers, health care providers, and policy makers. Embedded within the health care system, VA health services researchers play an important role in this learning health care organization, ³ working collaboratively across all levels of the organization to align “the science with clinical priority goals, conducting more rapid and efficient studies, and leveraging existing data to deploy and evaluate innovations and best practices”. ⁴

Learning health care organizations, however, confront challenges bridging differences in perspective, experience, demands, and pace of researchers, policy makers, and practitioners. ⁵ Researchers are driven to develop studies with replicable, protocolized designs (e.g., randomized control trials), which are expensive and take years to generate knowledge. Policy makers respond to windows of opportunity to make a change in complex systems and often cannot wait for the completion of rigorous scientific studies. They also must consider a breadth of information to develop policies that affect an entire health care system, including economic and human resources, political values, and scientific research.

Despite these differences, collaboration among researchers and policy makers within health care settings can lead to improvements in the effectiveness of health care, more efficient use of resources, consistency in care practices and policies, and greater health equity. ⁶ , ⁷ , ⁸ This paper showcases an exemplary collaboration, ⁴ focusing on a long‐term partnership to develop and study a large‐scale implementation of the Whole Health System (WHS) of Care. VA took advantage of a window of opportunity to generate evidence about the implementation and effectiveness of the model in 18 VA medical centers—evidence needed to catalyze further spread into clinical practice across the entire VA system. ⁹ , ¹⁰

1.1. The Whole Health System of Care

Whole health (WH) is defined as an “approach to healthcare that empowers and equips people to take charge of their health and well‐being and live their life to the fullest.” ² The goal of VA's WHS is to transform the organization and culture of care to start with understanding the Veteran's life mission, aspiration, and purpose (i.e., what matters most to the Veteran) and provide care to improve Veterans' overall health and well‐being. The WHS comprises three components: (1) WH Pathway—in which Veterans are introduced, often by peers, to WH concepts, explore their mission, aspiration and purpose, and develop a personal health plan; (2) WH Clinical Care—in which providers align allopathic and complementary integrative health (CIH) care, with Veterans' personal health plan, goals, and mission, aspiration, and purpose; and (3) Well‐being programs—in which Veterans participate in CIH services, health coaching, and self‐care and skill‐building groups to equip Veterans to manage their health.

1.2. Policy “window of opportunity”

In 2013, the VA's Office of Patient‐Centered Care and Cultural Transformation (OPCC&CT) began partnering with VA health services researchers to develop, evaluate, and refine a patient‐centered approach to care that included CIH. The Center for Evaluating Patient Centered Care (EPCC), funded by the VA's Quality Enhancement Research Initiative, identified implementation strategies used by facilities to implement a patient‐centered, integrative approach to care. ¹ , ¹¹ This early work laid a strong foundation for OPCC&CT to take advantage of a policy window that opened with Congress' passage of the 2016 congressional bill, the Comprehensive Addiction and Recovery Act (CARA), to address the nation's opioid epidemic. ¹²

CARA specifically directed the VA to develop an approach to address Veterans' opioid addiction and chronic pain. Moreover, it required an evaluation of the effect of CIH and other approaches on the health and well‐being of Veterans. Prior research in VA demonstrated the positive impact of CIH therapies such as yoga, Tai Chi, acupuncture, and mindfulness practices for Veterans, ¹³ , ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ yet the implementation of these services had up to that point been unsystematic and poorly coordinated overall. ¹⁹ , ²⁰ , ²¹ In response, OPCC&CT formalized the WHS, and in October 2017, each Veterans Integrated Service Network (VISN ‐ regional networks) identified and funded a WH flagship VA medical center for a 3‐year WHS pilot. The evaluation was to culminate in April 2020 in a congressional report, including key findings on the pilot's impact on Veterans.

VA viewed CARA as a mandate to provide CIH services to Veterans and an opportunity to do so in the context of the WHS; the intention from the start was that the demonstration projects and the evaluation would inform VA policy more broadly. Thus, in collaboration with OPCC&CT, EPCC set forth to conduct a timely evaluation of the implementation of the WHS, focused on addressing key stakeholder questions about the effectiveness of the WHS on Veterans with chronic pain. The goal was to both address the requirements of the CARA legislation and to gather evidence about the effect of the WHS to inform future VA policy about implementing the WHS throughout VA. The evaluation focused on the impact of engagement with WHS services on two areas: (1) opioid use and (2) key patient‐reported outcomes.

2. METHODS

The evaluation was conducted at 18 geographically distributed flagship medical centers, one in each VISN, that were piloting the WHS. These included large complex urban medical centers, as well as smaller rural medical centers. This work was conducted as a quality improvement activity in accordance with VA Handbook 1058.05 and Program Guide 1200.21.

2.1. Designing the evaluation in partnership with stakeholders

Designing an evaluation of large system level interventions requires a strategy to engage key stakeholders; in this case, OPCC&CT (co‐author BK). This involved a series of meetings in which OPCC&CT leadership first described in general terms what evidence they were seeking from this evaluation. As a result, the evaluation was extensive, examining progress toward the implementation, patient, and employee outcomes.

OPCC&T identified two important evaluation questions: first, to evaluate the use of WHS on opioid use (due to CARA's focus on opioid use); second, was to assess the impact on patient‐reported outcomes. We subsequently conducted a collaborative logic model process with OPCC&CT, ²² , ²³ , ²⁴ to outline WHS services, potential outputs of these services, and expected outcomes for Veterans. It became clear that the primary expected outcomes were not disease outcomes; they were improved patient experiences, engagement with health care and self‐care, quality of life, and overall well‐being. The process led to the identification of patient‐reported outcome measures to use in a longitudinal survey of Veterans with chronic pain (Table 1).

TABLE 1.

Patient‐reported outcomes measures used in the Veterans' health and life survey

Construct	Measure	Description
Experiences of care
Quality of provider interactions	CARE: Consultation and relational empathy 25 , 26	Perceptions of the patient‐provider relationship
Patient‐centered communication	collaboRATE 27	Perceptions of shared decision making
Veteran satisfaction with care	Satisfaction with care, adapted from Veterans Health Administration (VA) survey of health experiences of patients survey	Satisfaction with their VA primary care provider in the past 6 months
Help with goals	Process questions developed internally	Two items assessing patient goal progress Q1: How often have you discussed your goals with your provider? Q2: How helpful were your providers in helping you with your goals.
Engagement in Care
Engagement‐ health behaviors	ACE‐C: Altarum consumer engagement‐ commitment sub‐scale 28	Engagement in health care decisions and confidence and ability to participate in treatment decisions
Engagement‐ health care decisions	ACE‐N: Altarum consumer engagement‐navigation subscale 28
Meaning and purpose
Meaning and purpose 1	LET: Life engagement test 29	Meaning and purpose in life
Meaning and purpose 2	Institute for Health care Improvement's (IHI) 100 million healthier lives 30 , 31	Agreement with statement—“I lead a purposeful and meaningful life.”
Well‐being
Physical health	PROMIS‐10 (patient‐reported outcomes measurement information system) physical health subscale 32	Self‐reported global health. Assessment of symptoms and function. Two dimensions: Physical health and mental health
Mental health	PROMIS‐10 (patient‐reported outcomes measurement information system) mental health subscale 32
Stress	PSS: perceived stress scale 33 , 34	Perceptions of ability to manage stress
Pain
Pain	PEG 35 , 36 , 37	Assesses pain intensity (P), interference with enjoyment of life (E), and interference with general activity (G). This measure was designed and validated for use among Veterans.
Pain ‐ current	DVPRS (Defense & Veterans Pain Rating Scale) 38	Assesses pain intensity over past 24 hours
Length of pain	Question developed internally	Assesses length of time that pain has been a problem for Veterans

2.2. Data collection methods

We used data extracted from the VA electronic health record (EHR) to assess WHS use for both aims. We examined pharmacy data to assess changes in the use of opioids among Veterans with chronic pain. We conducted a longitudinal patient survey to assess the effect of WHS use on patient‐reported outcomes. As the CARA report was due to Congress in early 2020, we conducted interim analyses of the data collected through October 2019, and analytical approaches were adjusted accordingly.

2.2.1. Whole Health System service use

Sample

Veterans who used VA health care during the study period were identified in the EHR with no exclusion criteria other than being a Veteran. The exposure of interest was the time (fiscal year), and the descriptive study outcome was the utilization of WHS services delivered in VA and in the community (but paid for by the VA). We identified 10 types of care in the EHR: (1) Core WH, including WH pathway, coaching, and education; (2) chiropractic care, and eight CIH services encompassed in the standard VA medical benefits package, including acupuncture, therapeutic massage, biofeedback, guided imagery, clinical hypnosis, meditation, yoga, and Tai Chi/Qi gong. We used CPT codes (for chiropractic care, acupuncture, and massage), clinic stop codes for chiropractic care, clinic location names, CHAR4 codes, clinic note titles, health factors, and community care billing information. We developed search terms based on guidance from OPCC&CT and feedback from subject matter experts. To avoid double counting services identified by different coding strategies on the same day, we collapsed concurrent episodes of care among the 10 care types. For example, if we found one or more CPT codes for acupuncture, a note title for acupuncture, and a CHAR4 code for acupuncture on the same day, this was coded only as a single acupuncture encounter.

To assess the overall use of WHS during the evaluation period, we examined the use of each of the 10 types of WHS services and a summary measure of any WHS utilization. In addition to describing overall WHS utilization, we developed exposure groupings of WHS in collaboration with OPCC&CT leadership, as a single WHS encounter was deemed unlikely to have a meaningful effect on patient outcomes. These exposure groupings were utilized for the opioid analysis among new WHS users and the analysis of patient‐reported outcomes surveys based on WHS use at the time of survey participation. The purpose of these groupings was to provide insight for stakeholders about the experience of Veterans regarding the amount of use of the 10 types of WHS and how Veterans combined CIH with Core WH services. Stakeholders indicated the categories should be overlapping with Veterans included in the Intensive and Comprehensive Use categories also included in categories of lower use levels. The survey analysis also considered Veterans' self‐reported utilization of WHS services, including utilization not paid for by VA. Exposure categories were defined as follows: Comprehensive WHS Use: >=8 total WH services using a combination of >=2 Core WH encounters +>=2 CIH encounters. Core WHS Intensive Use: >=4 Core WH, 0 or more CIH; CIH Intensive Use: >=4 CIH, 0 or more Core WH.

Veterans with no WHS utilization were included in both the opioid analysis and the survey analysis as comparison groups (called Conventional Care). For the survey analysis, the comparison group came from the initial survey wave of a random sample of Veterans with chronic musculoskeletal pain. Because WHS use was low during this period, most of these survey participants were considered to have received Conventional Care and served as the comparison group to Veterans utilizing WHS.

2.2.2. Opioid use

Sample

We assessed opioid use among the subset of Veterans with chronic musculoskeletal pain at the 18 flagship medical centers who consistently used VA health care between October 1, 2017 and March 31, 2019. Veterans had to have at least three visits to be included to ensure they were consistent VA health care users and fully capture their VA opioid prescription changes. This included a health care visit in the 6‐month period between October 2017 and March 2018, another visit in the 6‐month period between April 2018 and September 2018, and at least a third visit in the 6‐month period from October 2018 to March 2019. Veterans who newly initiated WHS services were categorized as to the exposure of the WHS described above. The comparison group was Veterans with chronic musculoskeletal pain using VA health care during the same study period as the Veterans newly initiating WHS services, but these Conventional Care users had no WHS during the entire study period.

Outcome

Opioid prescriptions were extracted from VA's Managerial Cost Accounting Pharmacy data and converted to mg morphine equivalent (MME) using the Centers for Disease Control and Prevention Opioid MME tables based on reported drug name, dosage, and quantity. ³⁹ Opioid prescriptions were identified using the VA drug class code CN101—buprenorphine and nontabular forms of methadone were excluded because these are most often used to treat opioid dependence and not pain, which was considered best practice at the time of the evaluation, but future studies may consider, including these formulations. ⁴⁰

2.2.3. Patient‐reported outcomes

Sample

Veterans were sampled across the 18 flagship medical centers who had a recent primary care, mental health, or pain clinic visit. Exclusion criteria included age 90 or older, serious mental illness, and having an inpatient visit within 30 days of their recent visit. An initial phase randomly sampled Veterans who also had an existing diagnosis of chronic musculoskeletal pain identified in the EHR using a combination of ICD10 codes and numeric rating scale pain severity assessments. A second phase sampled Veterans with recent utilization coded with stop code 139, an early coding approach for WHS. The objective of this phase was to identify a higher proportion of Veterans using WHS. The comparison group constituted Veterans with no WHS use prior to or up until the 6‐month survey.

Outcomes

We conducted a longitudinal survey (baseline, 6 months, and 12 months) of eligible Veterans using 12 measures of patient‐reported outcomes identified during the logic model process assessing experiences of care, engagement in care, meaning and purpose, pain, and well‐being (Table 1). We also asked patients about their interest in using WHS services. We used a tailored survey administration method. ⁴¹ At each time point, Veterans were sent an introductory invitation letter, then sent an initial survey (including a $5 gift card), and if they did not reply, were sent a reminder postcard and a second copy of the survey. Veterans were informed that participation was voluntary and given the option to opt out by phoning the project manager.

2.3. Analysis

2.3.1. Overall WHS utilization

The cumulative use of any of the 10 types of WHS services among Veterans using VA health care was assessed beginning with Veterans receiving care (one or more visits) at the 18 flagship medical centers in Q1FY17 (October 1, 2016–December 31, 2016) through Q3FY19 (April 1, 2019–June 30, 2019). Historical utilization was assessed beginning October 1, 2015. To calculate the reach of the WHS pilot, we assessed the proportion of Veterans using VA health care in a quarter who used any WHS service in that quarter or cumulatively in prior quarters.

2.3.2. WHS and opioid use

We summed the total MME filled per patient per quarter (MME/patient‐quarter) and reported the change in average quarterly MME between baseline and follow‐up for each WHS use group. We compared the change in opioid dose levels between each of the four WHS use groups with those in the Conventional Care group. The focus of the analysis was the change in the 6‐month period prior to initiating WHS compared to the 6‐month period after WHS utilization. This approach was selected to reduce regression to the mean and confounding by indication biases potentially associated with reasons patients may have for initiating WHS or being referred to WHS.

2.3.3. WHS and patient‐reported outcomes

We examined differences in patient‐reported outcomes measures between baseline and 6‐month follow‐up. All measures were continuous variables on Likert‐type or continuous scales. Raw change between baseline and 6‐month survey time points was calculated for each of the four WHS use groups and those in the Conventional Care group. To provide summary estimates of meaningful differences of interest to key stakeholders between each of the levels of WHS use compared to Conventional Care, we calculated standardized mean differences as the primary estimate of effect size associated with the use of WHS. ⁴² , ⁴³ This allowed stakeholders to assess the magnitude of the effect, whether the effect was greater than zero, and compare effect sizes across patient‐reported outcomes using a common metric regardless of the scale of the measure. Notably, because the survey was ongoing and multiple outcomes were being examined, formal tests for statistical significance were not performed.

3. RESULTS

3.1. Overall WHS utilization

A total of 1,368,413 Veterans utilized health care at the 18 flagship medical centers during the evaluation period. Use of WHS services increased from 4.4% of these Veterans using VA health care in Q1FY17 to 15.9% among Veterans using VA health care in Q3FY19, an increase of 259%. Consistent with the objectives of CARA, the 18 flagships reached patients with chronic musculoskeletal pain. Among Veterans with chronic pain utilizing VA health care, 31% used WHS services by Q3FY17, up to 55% at one flagship.

3.2. WHS and opioid use

We identified a subset of 114,397 Veterans at the 18 flagship medical centers with chronic musculoskeletal pain who were regular VA health care users with a visit in each 6‐month period between October 1, 2017 to March 31, 2019. This included 6594 Veterans who initiated in WHS for the first time between April 1, 2018 and September 30, 2018, with no prior utilization of WHS and a Conventional Care group of 107,763 Veterans who did not utilize WHS during the study period (Table 2).

TABLE 2.

Change in opioid dose over 18 months associated with new use of Whole Health compared to conventional care

	VA users with chronic pain identified in EHR	MME dose period before using WH	MME dose period started WH	MME dose period after WH use	Change MME (before‐after)	Change % (before‐after)
Conventional care	105,608	634	593	563	−72	−11%
Any WH use (2+)	6594	759	683	583	−176	−23%
CIH intensive	4198	710	626	529	−181	−26%
Core WH intensive	961	557	453	346	−211	−38%
Comprehensive	601	658	496	410	−248	−38%

Abbreviations: CIH, complementary integrative health; EHR, electronic health record; MME, milligram morphine equivalent; VA, Veterans Health Administration; WH, Whole Health.

During the analysis period, opioid prescription fills decreased among Veterans with chronic pain in all WHS use groups and among Veterans who used only Conventional Care, consistent with national VA efforts to reduce opioid prescribing and use. ⁴⁴ Larger average decreases in opioid levels among VA prescription fills were observed among the Core WH Intensive and Comprehensive WH users (−38%; 95% CI: −60.3 to −14.4) and CIH Intensive users (−26%; 95% CI: −30.9 to −18.4), and the full group of Veterans who used any two or more WHS services (−23%; 95% CI: −31.9 to −6.5), compared to those who used Conventional Care (−11%; 95% CI: −12.0 to −9.9).

3.3. WHS and patient‐reported outcomes

A total of 3266 Veterans completed baseline and 6‐month surveys across the 18 flagship medical centers at the time of the interim analysis for the Congressionally mandated report, with a 50% response rate for the baseline survey and 74% of these respondents completing the 6‐month follow‐up survey. The majority of patients in both Conventional Care (95.6%) and WH (98.7%) groups reported interest in at least one WH service. Survey participants who used WHS services were more likely to be female, younger, have higher levels of pain intensity and longer duration of chronic pain compared to those who did not use WHS services. Over 91% of Veterans participating in the survey reported pain lasting more than 6 months, with 80% reporting moderate or severe pain intensity, 18% reporting mild pain intensity, and only 2% indicating no pain or not reporting pain intensity information (Table 3).

TABLE 3.

Characteristics of Veterans participating in patient‐reported outcomes survey during the flagship evaluation

Variable	Overall	Conventional care	Any 2+ WH use	CIH intensive	Core WH intensive	Comprehensive
N	3266	1712	1554	696	273	145
Female: N (%)	298 (9.1)	115 (6.7)	183 (11.8)	110 (15.8)	32 (11.7)	34 (23.4)
Age: N (%)
18–39	97 (3.0)	28 (1.6)	69 (4.4)	37 (5.3)	6 (2.2)	6 (4.1)
40–54	398 (12.2)	176 (10.3)	222 (14.3)	122 (17.5)	32 (11.7)	27 (18.6)
55–64	759 (23.2)	370 (21.6)	389 (25.0)	196 (28.2)	82 (30.0)	51 (35.2)
65–74	1418 (43.4)	790 (46.1)	628 (40.4)	261 (37.5)	111 (40.7)	51 (35.2)
75–90	594 (18.2)	348 (20.3)	246 (15.8)	80 (11.5)	42 (15.4)	10 (6.9)
Race: N (%)
White	2656 (81.3)	1412 (82.5)	1244 (80.1)	561 (80.6)	202 (74.0)	112 (77.2)
Black	353 (10.8)	185 (10.8)	168 (10.8)	75 (10.8)	48 (17.6)	20 (13.8)
Asian	11 (0.3)	4 (0.2)	7 (0.5)	3 (0.4)	0 (0.0)	0 (0.0)
Multiple	123 (3.8)	56 (3.3)	67 (4.3)	28 (4.0)	10 (3.7)	8 (5.5)
Other	40 (1.2)	16 (0.9)	24 (1.5)	12 (1.7)	5 (1.8)	3 (2.1)
Not reported	83 (2.5)	39 (2.3)	44 (2.8)	17 (2.4)	8 (2.9)	2 (1.4)
Hispanic: N (%)	183 (5.6)	95 (5.5)	88 (5.7)	41 (5.9)	11 (4.0)	8 (5.5)
In a relationship: N (%) a	2223 (68.1)	1173 (68.5)	1050 (67.6)	439 (63.1)	183 (67.0)	92 (63.4)
Unstable housing: N (%)	18 (0.6)	5 (0.3)	13 (0.8)	3 (0.4)	2 (0.7)	1 (0.7)
Currently employed: N (%) b	770 (23.6)	398 (23.2)	372 (23.9)	167 (24.0)	47 (17.2)	35 (24.1)
Education: N (%)
High school diploma	1221 (37.4)	690 (40.3)	531 (34.2)	202 (29.0)	110 (40.3)	32 (22.1)
2 or 4‐year degree	1702 (52.1)	877 (51.2)	825 (53.1)	393 (56.5)	125 (45.8)	89 (61.4)
Grad school	329 (10.1)	140 (8.2)	189 (12.2)	97 (13.9)	36 (13.2)	24 (16.6)
Not reported	14 (0.4)	5 (0.3)	9 (0.6)	4 (0.6)	2 (0.7)	0 (0.0)
Served in combat: N (%)	1648 (50.5)	845 (49.4)	803 (51.7)	357 (51.3)	134 (49.1)	66 (45.5)
DVPRS: N (%)
None	29 (0.9)	22 (1.3)	7 (0.5)	2 (0.3)	1 (0.4)	1 (0.7)
Mild	580 (17.8)	341 (19.9)	239 (15.4)	93 (13.4)	40 (14.7)	14 (9.7)
Moderate	1375 (42.1)	732 (42.8)	643 (41.4)	282 (40.5)	97 (35.5)	51 (35.2)
Severe	1242 (38.0)	596 (34.8)	646 (41.6)	311 (44.7)	129 (47.3)	78 (53.8)
NA	40 (1.2)	21 (1.2)	19 (1.2)	8 (1.1)	6 (2.2)	1 (0.7)
Length of pain: N (%)
Not a problem	34 (1.0)	20 (1.2)	14 (0.9)	3 (0.4)	3 (1.1)	1 (0.7)
<3 months	89 (2.7)	58 (3.4)	31 (2.0)	10 (1.4)	5 (1.8)	2 (1.4)
3–6 months	70 (2.1)	42 (2.5)	28 (1.8)	8 (1.1)	3 (1.1)	1 (0.7)
6+ months	2988 (91.5)	1550 (90.5)	1438 (92.5)	662 (95.1)	250 (91.6)	138 (95.2)

Abbreviations: CIH, complementary integrative health; DVPRS, Defense & Veterans Pain Rating Scale; WH, Whole Health.

^a Relationship = married, civil union, engaged, or in a relationship.

^b Currently employed = working full‐ or part‐time or a home maker.

Veterans with WHS use reported greater improvements in the experience of care, engagement with care, and overall well‐being (Table 4 and Figure 1). Six measures were associated with effect sizes larger than 0.10 associated with WHS utilization.

TABLE 4.

Change in patient‐reported outcomes associated with participating in Whole Health compared to conventional care

Outcomes	Conventional care	Any 2+ WH use	CIH intensive	Core WH intensive	Comprehensive
Patient experience
Quality of provider interactions (CARE)
Baseline	37.7	37.7	37.3	38.6	39.2
6 months	37.8	38.4	38.2	40.2	40.4
Change	0.1	0.6	0.9	1.4	1.5
SMD		0.04	0.08	0.14	0.14
Patient‐centered communication (CollaboRATE)
Baseline	7.0	6.9	6.8	7.1	7.2
6 months	7.1	7.1	7.0	7.3	7.4
Change	0.1	0.2	0.2	0.2	0.2
SMD		0.07	0.07	0.05	0.05
Veteran satisfaction
Baseline	7.5	7.4	7.3	7.9	7.7
6 months	7.5	7.6	7.5	8.0	7.8
Change	0.0	0.2	0.2	0.1	0.2
SMD		0.08	0.09	0.04	0.09
Help with goals
Baseline	5.6	5.8	5.8	6.5	6.3
6 months	5.4	5.9	5.9	6.9	6.8
Change	−0.1	0.1	0.1	0.4	0.5
SMD		0.12	0.11	0.30	0.32
Patient engagement and life meaning and purpose
Engagement‐health behaviors (ACE‐C)
Baseline	2.4	2.4	2.4	2.4	2.3
6 months	2.4	2.4	2.3	2.4	2.3
Change	−0.0	−0.0	−0.0	0.0	0.1
SMD		−0.02	−0.04	0.06	0.10
Engagement‐health care decisions (ACE‐N)
Baseline	2.6	2.7	2.7	2.7	2.8
6 months	2.6	2.7	2.7	2.8	2.8
Change	0.0	0.0	0.0	0.0	0.1
SMD		0.02	0.04	0.07	0.12
Meaning and purpose (LET)
Baseline	22.6	22.0	21.7	22.0	21.2
6 months	22.4	22.0	21.7	21.8	21.5
Change	−0.2	−0.0	−0.1	−0.2	0.1
SMD		0.04	0.03	0.01	0.09
Spiritual well‐being (IHI)
Baseline	5.0	4.8	4.7	4.8	4.4
6 months	5.0	4.8	4.6	4.8	4.5
Change	0.0	0.0	0.0	0.0	0.2
SMD		0.01	0.0	0.00	0.15
Functional status, well‐being, and pain
Mental health (PROMIS10)
Baseline	43.2	41.4	40.2	40.5	38.6
6 months	43.0	41.4	40.2	40.4	38.9
Change	−0.2	0.0	0.0	−0.1	0.3
SMD		0.04	0.03	0.02	0.09
Physical health (PROMIS10)
Baseline	38.0	37.1	36.3	36.3	35.8
6 months	38.2	37.2	36.4	36.5	36.0
Change	0.3	10.	0.1	0.1	0.2
SMD		−0.05	−0.04	−0.06	−0.02
Stress (PSS)
Baseline	5.7	6.3	6.6	6.7	7.1
6 months	5.8	6.3	6.4	6.6	6.6
Change	0.1	−0.1	−0.2	−0.2	−0.4
SMD		0.06	0.10	0.11	0.19
Pain (PEG)
Baseline	6.4	6.8	6.9	7.0	7.0
6 months	6.3	6.6	6.8	6.7	6.8
Change	−0.1	−0.2	−0.1	−0.3	−0.2
SMD		0.00	−0.03	0.09	0.03

Note: The outcomes for PEG and PSS relative to the “No Use” group have been reversed so that positive values indicate more improvement.

Abbreviations: ACE‐C, Altarum consumer engagement‐commitment; ACE‐N, Altarum consumer engagement‐navigation; CARE, consultation and relational empathy; CIH, complementary integrative health; IHI, Institute for Health care Improvement; LET, life engagement test; PEG, Pain intensity (P), interference with enjoyment of life (E), and interference with general activity (G); PROMIS10, patient‐reported outcomes measurement information system; PSS, perceived stress scale; SMD, standardized mean difference in change relative to conventional care; WH, Whole Health.

FIGURE 1.

Survey patient‐reported outcomes. ACE‐C, Altarum consumer engagement‐commitment; ACE‐N, Altarum consumer engagement‐navigation; CARE, consultation and relational empathy; CIH, complementary integrative health; IHI, Institute for Health care Improvement; LET, life engagement test; PEG, pain intensity (P), interference with enjoyment of life (E), and interference with general activity (G); PROMIS10, patient‐reported outcomes measurement information system; PSS, perceived stress scale; WH, Whole Health; WHS, Whole Health System [Color figure can be viewed at wileyonlinelibrary.com]

3.3.1. Patient experience of care

Veterans who used WHS services reported greater improvements in quality of health care interactions with VA providers and improved satisfaction with VA care compared to those receiving Conventional Care. The largest improvements were observed in Veterans reporting discussions of personal health goals with their providers, indicating that Veterans with more WHS services discuss and get help with personal health goals more than those who received Conventional Care. The effect size associated with this measure was greater than 0.1 for all Veterans who used any level of WHS and greater than 0.3 for Veterans who used Core WH services.

3.3.2. Engagement in care and life meaning and purpose

Veterans who used WHS services reported higher levels of engagement in healthy behaviors and participation in health care decisions compared to Veterans who received Conventional Care. These patterns were strongest among Veterans who used Core WH services (effect sizes of 0.05–0.1). Additionally, there were small improvements in overall meaning and purpose in life, especially among those who utilized Comprehensive WH services. Notably, the Institute for Health care Improvement's question about leading a purposeful and meaningful life had an effect size of 0.15.

3.3.3. Quality of life and well‐being

Veterans who used WHS services reported slight improvements at 6 months in quality of life and well‐being measures compared to Veterans who received Conventional Care. The WHS users improved the most on the Perceived Stress Scale (effect size 0.11–0.19), an important measure of one's ability to manage the challenges associated with chronic illness. There were small improvements in mental health scores for all WHS users relative to those receiving Conventional Care, with Comprehensive WHS users experiencing slightly better improvements. Although physical health scores were slightly better at 6 months for all groups, because the Conventional Care group also improved and improved by more than the WHS users, the relative effects show small negative trends among the WHS user groups. Pain scores improved slightly for all groups at 6 months, including the Conventional Care group, with the Core WH group experiencing slightly greater improvement compared to the Conventional Care group, although the changes were not clinically meaningful for any group.

3.4. Policy changes resulting from evaluation findings

In October 2019, OPCC&CT leadership had approached the VA Governance Board proposing an Executive Decision Memo be signed to spread the integration and implementation of WH throughout the system of care, focused on mental health and primary care services. Such memos help define VA policy and program priorities. The Governance Board replied with a request for greater evidence that the WHS improved Veteran experience and outcomes before they would endorse WHS expansion. Although OPCC&CT leadership had been spreading WHS through engaging additional medical centers in a WHS learning collaborative, further evaluation data was needed to foster a change in policy.

In January 2020, we completed a white paper describing the interim findings of our evaluation. ⁴⁵ This formed the foundation for VA's report to Congress in response to the CARA legislation. OPCC&CT leadership (co‐author BK) brought the findings, using the graphs in Figure 1, to the Governance Board; this was the evidence they needed to endorse the further expansion of WHS. The Executive Decision Memo ⁴⁶ was signed in March 2020, specifically citing our findings as to the foundation for the decision to proceed with the expansion of WH, and this became one of VA's lanes of modernization.

At the request of OPCC&CT leadership, we gave presentations of the evaluation findings to national VA audiences working in WH. The findings soon became the cornerstone of OPCC&CT presentations to VA leadership, including top leadership throughout the nation. OPCC&CT also presented the findings at the House Veterans' Affairs Committee Health Subcommittee hearing entitled “Resilience and Coping: Mental Health of Women Veterans,” March 2020. WH leaders throughout the nation continue to cite the findings when making the case for investment in WHS at the regional and local levels.

In June 2021, OPCC&CT, the VA Office of Mental Health and Suicide Prevention, and the Office of Patient Care Services held a virtual national conference with over 3000 attendees to launch the further integration of WH in mental health and primary care services throughout VA. Our findings were cited by presenters throughout the conference, and over 600 individuals attended our evaluation findings presentation. Notably, attendees stated they would use the slides to garner local medical center leadership support for the implementation of WH.

4. DISCUSSION

Policy makers are often faced with different proposals for improving the health care system. In large systems such as the VA with many competing priorities, such decisions are optimally made based on good supporting evidence. Yet, evidence that is scaled to a broad health care system is rarely available. Instead, policy makers rely on a combination of extant scientific knowledge, variability across the systems in human and economic resources, and social and political values. The goals of VA's OPCC&CT to implement WHS, based on available small‐scale evidence, were heightened by the opportunity presented by CARA to pilot this new model of care to address the challenges of the opioid crisis. In the absence of clear evidence to support this substantive system‐level change, the demonstration project provided an opportunity to collect data to guide VA policy decisions regarding the expansion of the WHS. Under normal circumstances, researchers would conduct randomized or pragmatic trials of interventions to establish an evidence base for implementing new policies. The timeline presented by CARA, however, required immediate, large‐scale action and a report to Congress within 3 years, prohibiting such formal research studies. This reflects the inherent tension between the timing of policy decisions and traditional research practices that focus on targeted populations and require long timelines. Large health care system innovations that are propelled forward in the absence of solid evidence have the potential to have both benefits and harms. Engaging in a partnership between a dedicated team of researchers and policy makers demonstrates that it is feasible to strategically leverage legislation to fast‐track high‐quality research to inform system change.

Our health services research team was able to conduct a simultaneous evaluation of the effect of WHS implementation on Veterans as it was being rolled out by identifying Veterans with WHS service use, using administrative data from the EHR, and conducting Veteran surveys. Our findings supported the implementation of the system of care; the WHS was having a positive impact on Veterans' experiences of care, engagement in care and well‐being, and a reduction of opioid use.

Prior studies have demonstrated the positive effects of both CIH therapies and patient‐centered care on patient adherence, engagement, and clinical outcomes, including perceptions of pain. ¹³ , ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ , ⁴⁷ , ⁴⁸ , ⁴⁹ , ⁵⁰ , ⁵¹ , ⁵² , ⁵³ , ⁵⁴ These studies, however, focus on discrete interventions and do not fully capture the impact of a system‐level holistic approach to care. Policy makers wanted to see the impact of this systemic implementation on important outcomes for Veterans.

Thus, a key component of our evaluation was determining what level of evidence was most relevant to policy makers. It became clear through our collaboration with OPCC&CT that identifying the positive effects of WH on opioid use reduction and measures of well‐being were critical. Subsequently, the relative larger reduction in opioid use and the patient‐reported outcomes effect sizes we found constituted sufficient evidence for policy makers to move forward on adopting the approach more broadly, particularly in the context of excessive levels of opioid addiction, overdose, and suicide among Veterans.

4.1. Limitations

The use of VA pharmacy data may not have captured opioid use by Veterans who received opioids from outside of the VA. This is most likely to impact the Conventional Care users who may be more likely to be less reliant on VA than Veterans who participated in WH. We may have underestimated the use of WHS services, as services being delivered may not have been fully captured in the EHR early on. Although this analysis includes a Conventional Care comparison group of Veterans who did not use WHS services, Veterans were not randomly assigned. WHS service use or nonuse may be associated with several factors, including attributes of those who choose to use WH, combining or substituting WHS services for other types of available care, and other factors associated with accessibility or availability of WHS services. Therefore, the effects attributed to WHS may be related to other confounding variables not captured by this evaluation.

5. CONCLUSION

Health care systems seek to develop new models of care to address the ever‐changing needs of populations, improve outcomes, and reduce costs. Yet, despite the aspirations of researchers, systems infrequently use research‐based evidence linearly to guide care improvement. In contrast, policy makers are often called upon to respond to an urgent concern. The opioid crisis and the burden of opioid use led Congress to pass CARA. This presented policy makers with a challenge and an opportunity to mitigate the harms of addiction in the absence of high‐quality evidence about optimal interventions. Generating evidence through collaborative planning and the evaluation of the WHS pilot guided important policy decisions to spread and sustain the implementation of the WHS in the VA. Pairing large‐scale demonstration projects such as the WHS flagship pilot project with rigorous health services research can inform the learning health care organization.

Bokhour BG, Hyde J, Kligler B, et al. From patient outcomes to system change: Evaluating the impact of VHA's implementation of the Whole Health System of Care. Health Serv Res. 2022;57(Suppl. 1):53‐65. doi: 10.1111/1475-6773.13938