Impact of a Remote Primary Care Telehealth Staffing Model on Primary Care Access in the Veterans Health Administration

Abstract

Background

The Veterans Health Administration (VHA) implemented the Clinical Resource Hub (CRH) program to fill staffing gaps in primary care (PC) clinics via telemedicine and maintain veterans’ healthcare access.

Objective

To evaluate PC wait times before and after CRH implementation.

Design

Comparative interrupted time series analysis among a retrospective observational cohort of PC clinics who did and did not use CRH during pre-implementation (October 2018–September 2019) and post-implementation (October 2019–February 2020) periods.

Participants

Clinics completing ≥10 CRH visits per month for 2 consecutive months and propensity matched control clinics.

Main Measures

Two measures of patient access (i.e., established, and new patient wait times) and one measure of clinic capacity (i.e., third next available appointment) were assessed. Clinics using CRH were 1:1 propensity score matched across clinical and demographic characteristics. Comparative interrupted time series models used linear mixed effects regression with random clinic-level intercepts and triple interaction (i.e., CRH use, pre- vs. post-implementation, and time) for trend and point estimations.

Key Results

PC clinics using CRH (N = 79) were matched to clinics not using CRH (N = 79). In the 12-month pre-implementation, third next available time increased in CRH clinics (0.16 days/month; 95% CI = [0.07, 0.25]), and decreased in the 5 months post-implementation (−0.58 days/month; 95% CI = [−0.90, −0.27]). Post-implementation third next available time also decreased in control clinics (−0.48 days/month; 95% CI = [−0.81, −0.17]). Comparative differences remained non-significant. There were no statistical differences in established or new patient wait times by CRH user status, CRH implementation, or over time.

Conclusions

In a national VHA telemedicine program developed to provide gap coverage for PC clinics, no wait time differences were observed between clinics using and not using CRH services. This hub-and-spoke telemedicine service is an effective model to provide gap coverage while maintaining access. Further investigation of quality and long-term access remains necessary.

BACKGROUND

The Veterans Health Administration (VHA) is the largest integrated healthcare system in the United States serving over 9 million enrolled veterans.¹ Primary care (PC) is delivered through a patient-centered, team-based model lead by a primary care provider (PCP) (i.e., medical doctor, nurse practitioner, or physician assistant). PCPs are an essential component of effective team-based care and hold responsibility for overall comprehensive care, especially for complex patients. PC access is associated with lower mortality rates, improved health outcomes, and increased access to specialty care physicians.^2–4 Despite rising PC demand, United States PCP vacancies are projected to increase.^5–8 Innovative solutions that provide care during PCP staffing shortages are necessary to meet the growing demand of the healthcare system.

When PCP staffing gaps occur, patient wait times can lengthen, hurting the ability of patients to receive care. Rural VHA clinics are more vulnerable to staffing gaps due to being smaller and having fewer PCPs.^9,10 They also serve a population with greater healthcare needs,^11,12 with less access to routine care,¹³ and longer travel distances. Geographical distance has been identified as a substantial barrier to PC follow-up after an acute care episode and is associated with 90-day readmission.¹⁴ Moreover, disparities in the population-based density of PCPs in rural compared to urban settings have widened,¹⁵ potentially exacerbating rural staffing gaps and limiting PC access for those already at increased risk of limited care options.

In October of 2019, the VHA Office of Primary Care implemented the national Clinical Resource Hub (CRH) program. The CRH program is primarily a telemedicine model meant to provide contingency staffing from 18 regionally based “hubs” to “spoke” sites at VA outpatient locations.^16–18 CRH visits are typically provided using clinical video telehealth, where the provider is engaged remotely at the hub, but staff and the patient are at the spoke location in-person or by VA Video Connect. Use of these services is elective whereby clinical leadership must request to engage the CRH. The goal of the CRH was to address provider contingency planning across an array of specialties, including PC, mental health, and pharmacy services. Though CRH-delivered PC services have doubled from September 2019 to October 2020, the effect of this care delivery model on access has not been studied.

The objective of this study was to evaluate the effect of CRH implementation upon PC wait times. We hypothesized clinics using CRH services would have longer wait times prior to CRH implementation. Further, because CRH can rapidly supply gap coverage for PCPs, we hypothesized wait times would comparatively improve among clinics using CRH services following CRH implementation.

METHODS

Study Design and Setting

We performed a comparative interrupted time series analysis using a retrospective national observational cohort of clinics using (CRH user) and not using CRH (control clinics) from October 1, 2018, to February 28, 2020. This 17-month observational period included a 12-month pre-implementation (October 1, 2018–September 30, 2019) and 5-month post-implementation period (October 1, 2019–February 28, 2020). The analysis was restricted to prior to the COVID-19 pandemic because the restrictions placed on in-person visits, infection concerns, and the variability nationwide in community spread of COVID-19 dramatically affected waited times and clinic capacity.

We followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.¹⁹ The study was approved by the University of Iowa Institutional Review Board and the Iowa City VA Healthcare System Research and Development Committee. We conducted this work without direct patient contact using routinely collected data deemed of minimal risk. A waiver of informed consent was obtained. The authors had full access to and take full responsibility for the integrity of the data.

Data Sources

Patient- and clinic-level data used the Veterans Informatics and Computing Infrastructure (VINCI) to access and abstract data from the VHA’s Corporate Data Warehouse (CDW), an integrated system including VHA electronic health records and administrative data. Patient-level data, including demographics (i.e., VHA Nosos risk adjustment score, gender, rural residency), the date, location, and delivery method of outpatient PC visits, and a measure of the third next available appointment and community care referral rates were obtained from the CDW. Additionally, the area deprivation index (ADI), a measure of neighborhood socioeconomic disadvantage,²⁰ was spatially merged to the fiscal year–specific latitude and longitude of each veteran’s home address using 2020 Census Bureau census block TIGER/Line shapefiles. The fiscal year–specific latitude and longitude of each veteran’s home address was obtained from the Planning Systems Support Group (PSSG). Clinic-level data, including clinic type (i.e., VHA medical center, community-based outpatient clinic, Health Care Center (HCC), and other outpatient services), was obtained from the VHA Site Tracking system. Other outpatient service sites are those in which “veterans receive services that do not meet the criteria of a community-based outpatient clinic or Health Care Center, including social services, homelessness outreach, and other services.”²¹ The observed to expected PC panel size ratio, the number of PCP full-time equivalents (FTE), and established and new patient wait times aggregated by clinic-month were derived from the VHA Support Services Center. Clinic size was obtained from the Primary Care Management Module.

CRH User and Implementation Status

Our primary exposure was the CRH delivery of PC. We categorized all CRH PC visits using claims codes across hub-and-spoke sites with de-duplication across the same patient and day. All PC visits were aggregated to the clinic-month level. For a clinic to be considered a CRH user, the clinic was required to have two or more consecutive months with at least 10 PC CRH visits. This cutoff was chosen following a series of team discussions where we used statistical measures and graphical approaches to inform a standard cutoff that captures meaningful CRH engagement without being overly onerous for smaller clinics. Each facility meeting this criterion by February 28, 2020, was considered a CRH user. Control clinics were those that did not use PC CRH services throughout the study period. Facilities using CRH services, but which did not meet the minimum threshold to be defined a CRH user, were excluded.

Outcomes

We assessed three measures of access to PC: (1) established and (2) new patient wait time and (3) third next available. Established patient wait time (EPWT) was measured from the patient indicated date to the date the appointment was completed. New patient wait time (NPWT) was measured from the date the appointment was created to the date the appointment was completed. Third next available (TNA), a measure of clinic capacity, was defined as the average time to the third open appointment in a provider’s clinical schedule.^22,23

Covariates

We adjusted for patient and clinic variables influencing clinical capacity and demand. Patient variables included demographics: clinical risk defined by Nosos score,²⁴ residential ADI, gender, and residential rurality. The geocoded location of the patient’s home via rural-urban commuting area codes was dichotomized into urban and rural.²⁵ All patient characteristics were aggregated at the clinic-level and described as monthly averages or percentages. Clinic variables included the observed to expected PC panel size ratio, the number of unique PC patients (i.e., clinic size), the number of PCP FTE, the PC community referral rate, and clinic type. Each clinic belongs to one of 18 decentralized regional Veterans Integrated Service Networks (VISNs), a system in which facilities work together to deliver care, that implemented CRH according to their needs; this was also included as a clinic variable. Panel fullness was defined as the number of patients greater than expected based upon support staffing, provider type, room availability, and gender. Unique PC patients were determined based on their assignment to the clinic by quarter. Clinics unable to meet access standards of seeing patients within 20 days are required to refer to the community for VHA-paid PC and serve as a mechanism to maintain access. We identified the rate of PC community referrals per 10,000 unique patients.

Statistical Analysis

We implemented a propensity score match among clinics using and not using CRH by estimating the probability a clinic would be a CRH user via logistic regression. Key clinic and patient variables averaged over the 3 months prior to the pre-implementation period (July 2018–September 2018) were included. These variables included the observed to expected panel size ratio, clinic size, proportion of patients living in rural locations, and PC community referral rate. In addition, we included all three outcomes (EPWT, NPWT, and TNA) averaged over the 12 months prior to implementation. An exact match on VISN and clinic type was required. We used a 1:1 optimal matching technique to minimize the total absolute pair-wise distances thus maximizing correct matching and reducing bias. The absolute standardized mean difference for the propensity score matches was <0.1 suggesting an appropriate balance between groups (Fig. 1). All matching procedures were performed in R using the MatchIt package.²⁶ In both matched and unmatched cohorts, we evaluated differences between groups using Welch’s t-test for continuous measures.

Figure 1.

Standardized mean differences for the propensity score models developed to match CRH user and control clinics.

To assess the CRH model, we report the number of visits, unique patients served, and visit rate per unique patient among CRH clinics, all eligible and matched control clinics, respectively, in our study period. We describe the distribution of visits within these clinic groups based on clinic type, categorical clinic size, and the rurality of the clinic based on geographic location.

Among matched clinics, we conducted a comparative interrupted time series linear mixed effects analysis with random intercepts at the clinic-level. There are three primary covariates of interest in our model: treated (i.e., whether the clinic met the CRH user inclusion criteria), intervention status (i.e., pre- vs. post-implementation), and time (i.e., months). The model included two-way and three-way interactions between these covariates and was adjusted for average Nosos risk adjustment score, average ADI, percentage of the patient population that was male, percentage of the patient population that was rural, and the observed to expected panel size ratio. Using an interrupted time series model, we calculated the overall model slopes and the instantaneous “jump” post-implementation in EPWT, NPWT, and TNA outcomes across the pre- and post-implementation periods among CRH user clinics and matched controls. Similar comparisons across groups within the pre- and post-implementation periods were also made. Sensitivity analyses among rural clinics were also conducted. Confidence intervals were calculated using bootstrap methods with a type I error of 0.05. All model analyses were conducted in R using the lme4 package.²⁷ Plots of predicted values derived from these models over time and stratified by CRH user status are provided.

RESULTS

From October 1, 2019, to February 28, 2020, 80 clinics met our inclusion criteria as a CRH user (Table 1). After propensity score matching, the mean standardized difference illustrated a balance of key clinical characteristics across CRH user and control clinics, except for clinic size (Fig. 1).

Table 1.

Characteristics of CRH Users and Control Clinics Before and after the Propensity Score Matching

	All eligible clinics			Propensity score matched clinics
	CRH user* N = 80	Control† N = 802	p-value	CRH user N = 79	Control N = 79	p-value
Site type, N (%)
Community-based outpatient clinic	52 (65.0)	549 (68.5)	N/A‡	52 (65.8)	52 (65.8)	N/A
VHA medical center/ Health care center	21 (26.2)	127 (15.8)		20 (25.3)	20 (25.3)
Other outpatient services	7 (8.7)	126 (15.7)		7 (8.9)	7 (8.9)
Clinic characteristics§, mean (SD)
Panel fullness	89.3% (14.3)	91.2% (22.0)	0.07	89.3% (14.4)	90.6% (19.6)	0.41
Primary care provider FTEs	8.9 (7.8)	6.0 (6.6)	< 0.001	8.7 (7.6)	7.9 (7.5)	0.27
Clinic size, mean (SD)	7571 (6581)	5204 (5394)	< 0.001	7406 (6456)	6781 (6781)	0.30
Community primary care referrals‖	12.3 (34.6)	5.0 (40.1)	0.01	10.5 (31.9)	10.3 (49.5)	0.97
Rural residing, % (SD)	54.2 (34.1)	50.9 (35.5)	0.15	54.7 (34.0)	57.8 (33.5)	0.31
Outcomes¶, mean days (SD)
Established patient wait time	5.6 (6.0)	4.1 (3.3)	< 0.001	5.5 (4.9)	4.4 (3.3)	0.01
New patient wait time	21.8 (13.6)	18.8 (12.1)	< 0.001	20.9 (13.8)	19.7 (10.7)	0.28
Third next available	9.8 (6.4)	8.6 (7.6)	< 0.001	10.6 (6.6)	9.6 (6.8)	0.13

*To be defined as a CRH user clinic, a minimum of 2 consecutive months with ≥10 primary care CRH visits between October 1, 2018, and February 1, 2020, was required

^†Control clinics did not use the CRH within the study period

^‡An exact match based on site type was implemented in the propensity score model

^§Reported within the 3 months prior to the pre-implementation period (e.g., July 2018–September 2018) in line with the propensity score model

^‖Community referrals per 10,000 unique VHA patients at the clinic

^¶Reported within the prior fiscal year (e.g., October 2017–September 2018) in line with the propensity score model

Among CRH users, 46,995 unique veterans participated in 115,062 CRH PC visits, yielding 2.5 visits per patient, on average (Table 2). Compared to control clinics, CRH-based PC visits were more likely to occur at community-based outpatient clinics (66.4% vs. 42.1%, p < 0.001), medium-sized clinics (60.8% vs. 31.9%, p < 0.001), and rural clinics (50.8% vs. 14.9%, p < 0.001).

Table 2.

Clinical Characteristics of Primary Care (PC) Delivery Across Clinical Resource Hub (CRH) and Control Clinics

	CRH PC Visits among “CRH User” clinics	Non-CRH PC visits among “CRH User” clinics	p-value*	PC visits in matched control clinics	p-value†
Number of visits	115,062	1,481,047		1,359,551
Number of unique patients served	46,995	1,400,671		1,273,391
Visits per patient	2.45	1.06		1.07
Site type, N (%)			<0.001		<0.001
Community-based outpatient clinic	76,390 (66.4%)	725,212 (49.0%)		572,077 (42.1%)
VHA medical center/HCC‡	30,743 (26.7%)	731,027 (49.4%)		762,259 (56.1%)
Other outpatient services	7929 (6.9%)	24,808 (1.7%)		25,215 (1.9%)
Categorized clinic size§, N (%)			<0.001		<0.001
Small, 450–2399	24,600 (21.4%)	71,101 (4.8%)		158,385 (11.6%)
Medium, 2400–9999	69,941 (60.8%)	542,315 (36.6%)		433,467 (31.9%)
Large, 10,000+	20,521 (17.8%)	867,631 (58.6%)		767,699 (56.5%)
Rurality of clinic, N (%)			<0.001		<0.001
Rural	58,506 (50.8%)	275,015 (18.6%)		202,938 (14.9%)
Urban	56,556 (49.2%)	1,206,032 (81.4%)		1,156,613 (85.1%)

*p-values determined by ${\chi }^2$ test comparing CRH and non-CRH visits delivered at “CRH User” clinics

†p-values determined by ${\chi }^2$ test comparing CRH PC Visits at “CRH User” clinics and PC visits in matched control clinics

^‡Health Care Center (HCC)

^§Clinic size based upon unique patients receiving care at the given clinic

Established Patient Wait Time

During the pre-implementation period, the baseline difference in EPWT between CRH user and control clinics was not significantly different (−0.07 days/month, 95% CI = [−0.16, 0.01] days/month) (Table 3, Fig. 2). During the post-implementation period, both CRH user (0.11 days/month; 95% CI = [−0.04, 0.26]) and control (0.18 days/month; 95% CI = [−0.01, 0.37]) clinics demonstrated a non-significant increase in EPWT. However, the within group and comparative differences in EPWT across pre- and post-implementation periods were not statistically different.

Table 3.

Comparative Change in Wait Times with Bootstrapped 95% Confidence Intervals Among CRH user and Control Clinics in the Pre- and Post-implementation Phases

Group	Change in wait time (days/month)*		Difference†	Pre-post jump‡
	Pre-implementation	Post-implementation
Established patient wait time
Control	0.06 (−0.00, 0.14)	0.11 (−0.04, 0.26)	0.05 (−0.12, 0.21)	−1.02 (−1.79, −0.38)
CRH user§	−0.01 (−0.05, 0.03)	0.18 (−0.01, 0.37)	0.19 (−0.01, 0.40)	−0.44 (−0.98, 0.09)
Difference	−0.07 (−0.16, 0.01)	0.07 (−0.16, 0.32)	0.14 (−0.13, 0.44)	0.59 (−0.29, 1.51)
New patient wait time
Control	−0.02 (−0.18, 0.14)	−0.21 (−0.81, 0.40)	−0.18 (−0.81, 0.44)	−1.64 (−3.91,0.61)
CRH user§	−0.07 (−0.28, 0.14)	−0.03 (−0.74, 0.68)	0.04 (−0.75, 0.83)	0.26 (−2.07, 2.68)
Difference	−0.05 (−0.31, 0.22)	0.17 (−0.74, 1.11)	0.22 (−0.80, 1.25)	1.90 (−1.35, 5.13)
Third next available
Control	0.07 (−0.03, 0.16)	−0.48 (−0.81, −0.17)	−0.55 (−0.89, −0.22)	0.83 (−0.35, 2.12)
CRH user§	0.16 (0.07, 0.25)	−0.58 (−0.90, −0.27)	−0.73 (−1.09, −0.39)	0.60 (−0.55, 1.77)
Difference	0.09 (−0.04, 0.22)	−0.09 (−0.55, 0.36)	−0.18 (−0.70, 0.31)	−0.22 (−1.88, 1.46)

*The change in wait time corresponds to trends over time within the pre- and post-implementation periods, and the difference is the difference in these trends across control and CRH user clinics within period

^†The reported difference in this column is the difference post- to pre-implementation

^‡The pre-post jump is the immediate effect of CRH implementation

^§To be defined as a CRH user clinic, a minimum of 2 consecutive months with ≥10 primary care CRH visits between October 1, 2018, and February 1, 2020, was required

Figure 2.

Comparative interrupted time series plots for established (EPWT) and new patient wait time (NPWT), and third next available (TNA) outcomes using predicted models.

New Patient Wait Times

The comparative difference in NPWT for CRH user and matched control clinics was not significantly different during pre-implementation (−0.05 days/month, 95% CI = [−0.31, 0.22]) and post-implementation (0.17 days/month, 95% CI = [−0.74, 1.11]) periods (Table 3, Fig. 2). Further, the comparative difference in NPWT among CRH and control clinics from the pre-implementation period to post-implementation period was not statistically different (0.22 days/month, 95% CI = [−0.80, 1.25]).

Third Next Available

During the pre-implementation period, TNA significantly increased among CRH clinics by 0.16 days/month (95% CI = [0.07, 0.25]) but was not statistically different compared to matched control clinics (0.09 days/month, 95% CI = [−0.04, 0.22]) (Table 3, Fig. 2). During the post-implementation period, TNA decreased in control clinics (−0.48 days/month; 95% CI = [−0.81, −0.17]) and CRH user clinics (−0.58 days/month; 95% = [−0.90, −0.27]), but no evidence of a difference between CRH user and control clinics during the post-implementation period was detected (−0.09 days/month, 95% CI = [−0.55, 0.36]). When comparing pre- to post-implementation periods, a comparative decrease in TNA was noted within matched control (−0.55 days/month, 95% CI = [−0.89, −0.22]) and CRH User clinics (−0.73 days/month; 95% CI = [−1.09, −0.39]). The comparative change in TNA across CRH user and control clinics was not statistically significant (−0.18 days/month, 95% CI = [−0.70, 0.31]).

The full model output from which these calculations were derived is available in the Appendix. Sensitivity analyses using only rural clinics revealed very similar results.

DISCUSSION

Within VHA, PCPs are often the first to address a patient’s healthcare needs.²⁸ When PCP staffing gaps occur in PC clinics, delays in care can lengthen wait times, especially in rural locations where provider recruitment can be challenging.⁷ To address staffing deficits, the VHA implemented the CRH program to provide PC through video telemedicine using a hub-and-spoke model. Overall, we observed early implementation of CRH services did not significantly impact wait times. However, appointment availability as denoted by TNA improved post-implementation both among CRH user and matched control clinics. While our results do not support our hypotheses that CRH would decrease PC wait times, these results suggest, compared to similar control clinics, early implementation of CRH did not worsen access and likely maintained access to care. There are a few points to consider when interpreting these early results.

First, during this early period of the program, hubs may not have been agile enough to rapidly implement and impact access over the 5-month period. In a mixed methods investigation of early implementation,¹⁶ only 11% of clinics met high levels of progress within the first year. This is demonstrated by the relatively low number of CRH PC visits which following the current study period accelerated as the number of clinics using the CRH program expanded. These difficulties in early adoption likely explain the decreasing trend in TNA (or appointment availability) with no appreciable difference in EPWT or NPWT but do not imply the CRH program was not successful. In this telemedicine model, appointment availability needs to be followed by staff competency of new clinical processes and equipment and acceptance of patients to participate in telemedicine. Importantly, telemedicine use has proliferated since the COVID-19 pandemic expanding patient and provider experiences with virtual care.^29–34 Future work should consider how this expansion may have impacted the long-term effect of the CRH program on access to care, provider burn-out, quality of care, and patient satisfaction with CRH services.

Second, the ability to maintain access may be particularly beneficial for rural clinics, which have less new patient demand and risk significant discontinuity. This is especially salient, as approximately 50% of CRH encounters compared to 20% of standard PC visits occurred at a geographically rural clinic. Importantly rural communities, both within and outside VHA, have faced PC shortages despite policy efforts to incentivize medical residents to accept rural PC positions.^35,36 With PCP losses at rural clinics, access worsens and wait times prolong which can result in transferring care outside VHA. This discontinuity of care may put patients at risk. Rapidly filling these positions through the CRH model may improve access while also maintaining care in VHA. However, consideration is needed to ensure any existing disparities in access based on rurality do not widen during staff shortages. For example, it is well known broadband coverage limitations are more prominent in rural locations.^37–39 Though the CRH program offers patients the ability to receive PC, this care is predominantly delivered via video telemedicine which may not be achievable in some rural settings. Offering care at a clinic with reliable broadband coverage to sufficiently access the CRH program is a key component of this model. Combined with VHA’s digital divide program, which addresses infrastructural issues such as internet or device needs and connects veterans with resources for affordable internet access,^40,41 these potential disparities could be mitigated.

This research has limitations. First, these results among VHA clinics may not be generalizable to other healthcare systems using similar contingency staffing methods. However, the size of this nationwide study across geographic regions and a diverse patient population remains a significant strength. Second, because the CRH program is a VISN-level initiative, there is heterogeneity in implementation and processes. This could make it more difficult to detect differences in access measurement, as measured by wait times and appointment availability. We did not determine quality of care or if a CRH visit successfully addressed the patient’s concern or resulted in subsequent outpatient PC visits. This may be particularly relevant as some care needs may not be well suited to video telemedicine. However, access to the CRH program may facilitate timely follow-up appointments, improving quality of care in the presence of provider gaps. Third, this study was restricted to the first 5 months of CRH implementation, which may not be reflective of its full impact. This was required as PC wait times rapidly declined as the COVID-19 pandemic led many patients to cancel scheduled visits or delay care and did not return to historic levels until late 2020. Nevertheless, the CRH program has demonstrated an ability to provide care when it is needed and for patients served represents an important improvement over care delays. Finally, we were not able to account for residual confounders that could affect a clinic’s ability to be an early adopter of the CRH (e.g., strong leadership support, administrative bandwidth, access to support staff). Despite these limitations, this work is the first we are aware of that addresses access to PC before and after implementation of a novel telemedicine gap staffing program.

Early implementation of the VHA’s CRH program maintained wait times to PC when staffing gaps were present. This study adds to the growing literature about staffing contingency methods and the impact of such programs on patient care.

Appendix

Mixed effect comparative interrupted time series models with a three-way interaction effect for CRH use status, intervention status, and time for established and new patient wait times, and third next available outcomes

Predictors	Established patient Wait time (days/month)		New patient Wait time (days/month)		Third next available (days/month)
	Estimate (95% CI)	p-value	Estimate (95% CI)	p-value	Estimate (95% CI)	p-value
Intercept	5.63 (−0.96, 12.21)	0.094	58.56 (38.09, 79.03)	<0.001	5.58 (−7.41, 18.57)	0.400
Clinic: average Nosos risk score, 1/10th-unit increase	0.11 (−0.09, 0.31)	0.289	−0.22 (−0.9, 0.46)	0.529	0.54 (0.15, 0.93)	0.006
Clinic: average ADI national ranking, 10-unit increase	−0.68 (−1.01, −0.36)	<0.001	−1.36 (−2.34, −0.37)	0.007	−0.53 (−1.17, 0.11)	0.107
Clinic: percent male	0.02 (−0.04, 0.09)	0.526	−0.30 (−0.50, −0.11)	0.003	0.05 (−0.08, 0.18)	0.422
Clinic: percent rural	0.00 (−0.02, 0.02)	0.970	−0.05 (−0.10, 0.00)	0.065	−0.05 (−0.08, −0.02)	0.003
Observed to expected Panel ratio, 1/10th-unit increase	0.09 (0.003, 0.17)	0.041	0.38 (0.069, 0.71)	0.021	0.28 (0.13, 0.43)	<0.001
CRH clinic user (Ref: control clinic)	−0.15 (−1.18, 0.88)	0.779	−0.10 (−3.36, 3.16)	0.952	0.21 (−1.81, 2.23)	0.837
Intervention status (Ref: pre-intervention)	−1.02 (−1.69, −0.35)	0.003	−1.62 (−4.27, 1.03)	0.232	0.84 (−0.37, 2.05)	0.173
Time	0.06 (0.01, 0.10)	0.020	−0.02 (−0.21, 0.17)	0.832	0.07 (−0.02, 0.15)	0.131
CRH clinic user x intervention status	0.58 (−0.38, 1.55)	0.234	1.89 (−1.91, 5.70)	0.329	−0.24 (−1.98, 1.51)	0.791
CRH clinic user x time	−0.07 (−0.14, 0.00)	0.047	−0.04 (−0.31, 0.22)	0.745	0.09 (−0.03, 0.21)	0.147
Intervention status x time	0.05 (−0.13, 0.24)	0.593	−0.19 (−0.92, 0.55)	0.618	−0.55 (−0.89, −0.22)	0.001
CRH clinic user x intervention status x time	0.07 (−0.20, 0.34)	0.608	0.17 (−0.89, 1.22)	0.753	−0.09 (−0.57, 0.40)	0.723

Funding

This material is based upon work supported (or supported in part) by the Department of Veterans Affairs, Veterans Health Administration, Office of Rural Health, Veterans Rural Health Resource Center – Iowa City, and the Office of Research and Development, Health Services Research and Development (HSR&D) Service through the Comprehensive Access and Delivery Research and Evaluation (CADRE) Center (CIN 13-412), as well as the VHA Primary Care Analytics Team (PCAT), funded by the VHA Office of Primary Care. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Data Availability:

The datasets generated and analyzed are not publicly available due to VHA privacy and confidentiality requirements.

Declarations:

Ethics Approval and Consent to Participate:

This work was designated non-research since it was carried out as a quality improvement evaluation under the terms of a signed attestation of non-research from the VHA Office of Primary Care (OPC). This VHA documentation ensures that work not carried out under a human subjects protocol is part of institutionally sanctioned quality improvement activities. Data for this report were developed by PCAT.

Conflict of Interest:

This manuscript is not under review elsewhere and there is no prior publication of manuscript contents. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs or the United States Government. The authors report no conflict of interest regarding this study.