Improving Quality Improvement Capacity and Clinical Performance in Small Primary Care Practices

Abstract

PURPOSE

We undertook a study to assess whether implementing 7 evidence-based strategies to build improvement capacity within smaller primary care practices was associated with changes in performance on clinical quality measures (CQMs) for cardiovascular disease.

METHODS

A total of 209 practices across Washington, Oregon, and Idaho participated in a pragmatic clinical trial that focused on building quality improvement capacity as measured by a validated questionnaire, the 12-point Quality Improvement Capacity Assessment (QICA). Clinics reported performance on 3 cardiovascular CQMs—appropriate aspirin use, blood pressure (BP) control (<140/90 mm Hg), and smoking screening/cessation counseling—at baseline (2015) and follow-up (2017). Regression analyses with change in CQM as the dependent variable allowed for clustering by practice facilitator and adjusted for baseline CQM performance.

RESULTS

Practices improved QICA scores by 1.44 points (95% CI, 1.20-1.68; P <.001) from an average baseline of 6.45. All 3 CQMs also improved: aspirin use by 3.98% (average baseline = 66.8%; 95% CI for change, 1.17%-6.79%; P = .006); BP control by 3.36% (average baseline = 61.5%; 95% CI for change, 1.44%-5.27%; P = .001); and tobacco screening/cessation counseling by 7.49% (average baseline = 73.8%; 95% CI for change, 4.21%-10.77%; P <.001). Each 1-point increase in QICA score was associated with a 1.25% (95% CI, 0.41%-2.09%, P = .003) improvement in BP control; the estimated likelihood of reaching a 70% BP control performance goal was 1.24 times higher (95% CI, 1.09-1.40; P <.001) for each 1-point increase in QICA.

CONCLUSION

Improvements in clinic-level performance on BP control may be attributed to implementation of 7 evidence-based strategies to build quality improvement capacity. These strategies were feasible to implement in small practices over 15 months.

INTRODUCTION

Primary care practices in the United States are in a period of transition, with considerable pressure from payers and policy makers to improve quality of care and reduce costs.^1-4 Value-based pay contracts, quality reporting requirements, and market consolidation are accelerating the pace of change in many markets.⁵ Larger, well-organized primary care practices participate in state and national initiatives to improve measurement, transform care, and stay abreast of these rapid changes. These large-scale quality improvement (QI) efforts, however, require investments of time and resources that make engagement in and findings from these efforts impractical or irrelevant for smaller practices.^6-13

Smaller practices often lack key infrastructure and personnel capabilities—including multidisciplinary care teams, health information technology, and other resources¹⁴—necessary to be successful at implementing changes to quality or business systems. To survive in this new marketplace, smaller practices must build their capacity to adapt and improve. Building this capacity has been described as “…the planned development of knowledge, skills and other capabilities of a system or an organization to improve quality.”^15,16 Efforts to build QI capacity in primary care must involve appropriate strategies that can work in smaller practices,¹⁷ where almost two-thirds of US office-based physicians work.¹⁸

The Agency for Healthcare Research and Quality launched the EvidenceNOW initiative to better understand optimal approaches to building quality improvement capacity in smaller practices—defined as practices with fewer than 10 clinicians—that may have been unable or unwilling to participate in previous large-scale QI efforts.¹⁹ The EvidenceNOW initiative encouraged researchers and practice-change experts to test practice support strategies explicitly adapted to meet the needs of smaller practices across 7 regional cooperatives within the United States.²⁰ The overall purpose was to improve measures of cardiovascular disease risk known as the ABCS of heart health—appropriate aspirin prescribing, blood pressure (BP) control, cholesterol control, and tobacco screening/cessation counseling—by building QI capacity. To achieve these aims, 1 of the 7 cooperatives, Healthy Hearts Northwest (H2N), provided practice facilitation as a unifying strategy for all practices, with varied combinations of external support, academic detailing, and shared learning opportunities, to smaller practices in Washington, Oregon, and Idaho.²¹ Results from the randomized trial of different combinations of external support on observed changes in cardiovascular risk factor performance have been previously reported.²² Briefly, compared with facilitation only, practices randomized to all 3 support strategies had a small but significant improvement in performance on BP control.

The analysis reported here extends our understanding of the previously reported changes in clinical performance measures by testing a hypothesized mechanism whereby practices achieved improvements in clinical performance. A prespecified aim of the study was to examine whether small practices were able to improve their QI capacity in response to the external support provided and, if so, to assess whether that improvement was associated with gains in performance on clinical quality measures (CQMs) related to cardiovascular risk factors. To our knowledge, this is the first article to report on a validated measure of implementation of quality improvement capacity strategies at 2 points in time that assesses change in response to external practice support. It is also the first to evaluate whether those improvements in quality improvement capacity were associated with a change in clinical performance.

METHODS

Study Setting and Participants

The Healthy Hearts Northwest cooperative enrolled 209 smaller primary care practices across Washington, Oregon, and Idaho. To be eligible, practices had to have fewer than 10 full-time clinicians in a single location and participate in stage 1 meaningful use federal certification for their electronic health record.²³ The analysis reported here was determined to be exempt (category 2) by the Kaiser Permanente Washington Health Research Institute’s Institutional Review Board, waiving the requirement for informed consent but not ethics review.

Interventions

The original study used a 2-by-2 factorial design to compare the effectiveness of adding shared learning and educational outreach to practice facilitation to improve cardiovascular care and outcomes using an intention-to-treat analysis.²² A detailed description of the external support interventions has been previously published.²² All practices received 15 months of practice facilitation. Facilitation support was provided by 2 organizations: Qualis Health (now Comagine Health) for practices in Washington and Idaho, and the Oregon Rural Practice-based Research Network for practices in Oregon. The facilitation protocol included at least 5 planned quarterly face-to-face visits, with monthly check-ins (in-person visits, telephone calls, or e-mails) in-between. Facilitators met with a team within each practice to assist them in extracting their cardiovascular CQMs and developing and testing Plan-Do-Study-Act cycles of improvement. Sixteen facilitators were to provide 15 months of active support to all practices guided by 7 change strategies described below.

Data Measures and Collection

Practice and Patient Characteristics

A baseline practice questionnaire was completed by the office/practice manager in each enrolled practice to collect information about practice characteristics including size, ownership, staffing, and patient characteristics such as insurance status and age groups.

Quality Improvement Capacity Assessment

The Quality Improvement Capacity Assessment (QICA) is a validated measure of the QI capacity of primary care practices and is organized around 7 strategies, termed high-leverage changes (HLCs), that guide efforts to expand clinical QI capacity.²⁴ Within improvement science, an HLC is defined as an intervention point within a system that has a high likelihood of causing a transformational change that improves outcomes.²⁵ These HLCs acted as the curriculum for the Healthy Hearts Northwest practice facilitators and practice teams to guide their practice change activities.²⁴ The 7 hgh-leverage changes were as follows:

1. Embed clinical evidence into daily work to guide how care is delivered to patients

2. Utilize data to understand and improve clinical performance measures

3. Establish a regular QI process involving cross-functional teams

4. Identify at-risk patients through proactive population management and outreach

5. Define roles and responsibilities across the team to improve care

6. Deepen patient self-management support to improve clinical outcomes

7. Link patients to resources outside of the clinic to support patients

Clinical teams were trained on and encouraged to use rapid-cycle tests of change (Plan-Do-Study-Act cycles) to iteratively test and then implement the HLCs. Most commonly, the tests were tied to improving measurement of BP and identification of hypertension, for example, “Medical assistant tried taking BP at the end of rooming process.” Other examples of rapid-cycle tests included trying daily “huddles” and “counting how many patients will receive a follow-up appointment for 2 weeks and how many actually come in over the next month.”²⁶ This local testing and adaptation enabled sites to adjust the sequence, timing, and pace of implementation of these HLCs to fit their context.

The total QICA is composed of 20 individual items, each scored between 1 and 12. A score of 1, 2, or 3 is noted as a Level D, which describes the element (eg, use of a registry to conduct outreach to high-risk patients) as not present in the practice. A score of 4, 5, or 6 is noted as Level C and describes the element as available or present, but often without routine application. A score of 7, 8, or 9 is noted as Level B, with the element present and frequently but not consistently applied, and a score of 10, 11, or 12 is noted as Level A, indicating consistent use of or application of best practice care. A change in score from a 5 to a 7 would therefore indicate that a practice has moved from an element of QI capacity that is available but not used, to one that is frequently but not consistently used during routine patient care.

A copy of the QICA questionnaire is available as a supplement in a previously published article.²⁴ Each HLC is assessed by 1 to 4 items on the questionnaire, and the score for each HLC is the average score given across all items in that category. The QICA score is the average score across all 20 individual items; hence, HLCs are weighted unevenly in the total score. Scores for each HLC and for the QICA overall range between 1 and 12. The QICA was to be completed twice during the study: once at the first in-person facilitation meeting and again at the fourth in-person quarterly facilitation meeting.

Cardiovascular CQMs

We collected 3 CQMs assessing cardiovascular risk factor management among practice patients: appropriate aspirin use, BP control (<140/90 mm Hg), and smoking screening and cessation counseling. For more detail on the CQM specifications, see the Supplemental Appendix (available at https://www.AnnFamMed.org/lookup/suppl/doi:10.1370/afm.2733/-/DC1). All measure sets and definitions were endorsed by the Centers for Medicare & Medicaid Services.^27-30 We intended to collect the lipid/statin therapy measure (Centers for Medicare & Medicaid Services measure 347); however, this measure was under revision at the start of the study because of recent changes in evidence-based clinical guidelines, and it was therefore not systematically collected and is excluded from analyses.

Each CQM is reported as the proportion of the eligible patient population meeting the quality standard. Practices were asked to report CQMs each quarter, using a 12-month lookback period. This analysis uses data corresponding to clinical care provided for calendar year 2015 (the year before the 15-month study intervention) and calendar year 2017. External practice support interventions started in some practices in the last 2 months of 2015 and concluded by the end of calendar year 2017 (Figure 1).

Figure 1.

Timing of QICAs relative to CQM lookback periods 1 and 2.

CQM = clinical quality measure ( appropriate aspirin prescribing, blood pressure control, cholesterol control, and tobacco screening and cessation counseling [ABCS]) ; Jan = January; QICA = Quality Improvement Capacity Assessment.

Note: Figure shows the timing of QICA completions relative to the period of time included in 2015 (lookback period 1) and 2017 (lookback period 2) CQM summary measures. The vertical lines denote the first and last observed dates of QICA 1 and QICA 2 completion. The CQMs assess outcomes over an entire calendar year; lookback period 1 covered 2015, while lookback period 2 covered 2017.

Analysis

We report the mean and 95% CI for changes from baseline in total QICA score and in individual HLC subscores within the QICA. Baseline and follow-up values, and changes in CQMs (2017 measures minus 2015 measures) are shown for clinics that completed 2 QICA evaluations and both years of CQM measurement, with t tests conducted to assess whether average differences from baseline were equal to zero.

We fit a separate linear regression model for each clinical outcome, with change in the CQM as the dependent variable and change in QICA as the primary independent variable of interest. These models estimate the mean change in the CQM (in percent) with each 1-point increase in QICA score.

Given that each practice aimed to achieve a performance of 70% or higher on each CQM, we also fit Poisson regression models to assess the likelihood of the 2017 CQM being above the 70% threshold with each 1-point increase in QICA. We report both unadjusted analyses and analyses adjusted for baseline CQMs, because of the suspected causal relationship between baseline clinic performance and ability to improve over the course of study. Generalized estimating equations with independent working correlations and robust standard errors were implemented, allowing for clustering by practice facilitator to account for correlation among clinics facilitated by the same individual. The facilitator was defined as the individual who spent the longest time coaching each clinic between the 2 QICA evaluations. We did not account for study arms in these analyses because trial results showed no significant association between the intervention arm practices were assigned to and the change in any continuously measured CQM. In addition, adjustment for study arm in our analyses did not change inference and minimally altered point estimates of association (results not shown).

We additionally report P values from 2-sided Wald tests for the existence of an association between changes in QICA and each measure of change in each CQM. Tests yielding P values less than .05 were deemed statistically significant. We emphasize the exploratory nature of these analyses, however, and recommend their consideration as hypothesis generating rather than definitive statements on association. Analyses were performed using Stata statistical software, version 15.0 (StataCorp LLC)³¹ and R version 4.0.2 (R Project for Statistical Computing).³²

RESULTS

Of the 209 enrolled practices, 165 completed QICA surveys at both baseline and follow-up and were included in this analysis (Figure 2). Practices included in the analysis did not differ significantly from those not included in terms of organization characteristics (urban-rural location, size, organizational ownership type), patient characteristics (payer mix, and age and sex distribution), or baseline BP CQM performance (Supplemental Table 1, available at https://www.AnnFamMed.org/lookup/suppl/doi:10.1370/afm.2733/-/DC1). Practices included in the analysis, however, had lower QICA scores at baseline (P = .03), and disproportionately represented those in Washington and Idaho over Oregon because the former more often returned both baseline and follow-up QICA questionnaires.

Figure 2.

Flowchart of practice inclusion in various analyses.

BP = blood pressure; CQM = Clinical Quality Measure; QICA = Quality Improvement Capacity Assessment.

The mean baseline QICA score was 6.45 (SD 1.39), which increased on average by 1.44 points (95% CI, 1.20-1.68) (Table 1). The mean changes in both overall QICA score and individual HLC subscores differed significantly from zero (P <.001). The largest improvements were noted in using data to improve performance and establishing a regular QI process.

Table 1.

Changes in QICA Score and Clinical Quality Measures Between Baseline and Follow-up

Measure	Baseline, Mean (SD)	Follow-Up, Mean (SD)	Change, Mean (95% CI)	P Valuea
QICA score b
Total score	6.45 (1.39)	7.88 (1.4)	1.44 (1.20-1.68)	<.001
HLC subscore				<.001
1. Embed clinical evidence	6.95 (2.29)	8.32 (1.74)	1.37 (1.01-1.73)	<.001
2. Utilize data to improve Performance	4.98 (2.43)	6.97 (2.40)	1.98 (1.58-2.39)	<.001
3. Establish regular QI processes	5.14 (2.20)	7.39 (2.16)	2.25 (1.85-2.65)	<.001
4. Identify at-risk patients	5.46 (1.86)	7.06 (1.85)	1.60 (1.27-1.93)	<.001
5. Define roles and responsibilities	6.96 (1.80)	8.15 (1.80)	1.20 (0.90-1-.51)	<.001
6. Improve patient self-management	7.44 (1.82)	8.51 (1.78)	1.08 (0.78-1.39)	<.001
7. Link patients to outside resources	8.20 (1.64)	8.99 (1.40)	0.79 (0.52-1.07)	<.001
Clinical Quality Measure c
Aspirin use, %	66.81 (16.61)	70.79 (13.20)	3.98 (1.17-6.79)	.006
Blood pressure control, %	61.48 (12.00)	64.84 (11.48)	3.36 (1.44-5.27)	.001
Smoking screening/cessation counseling, %	73.78 (22.88)	81.27 (21.26)	7.49 (4.21-10.77)	<.001

HLC = high-leverage change; QI = quality improvement; QICA = Quality Improvement Capacity Assessment.

^aFrom a t test that tested for differences of the mean change from zero.

^bLimited to clinics that completed both QICA surveys (N = 165). Possible range of total score and of each HLC subscore is 1 to 12 points; higher scores denote greater QI capacity.

^cPercent of the eligible patient population achieving the measure. Limited to clinics that completed both QICA surveys and reported clinical quality measures in both 2015 and 2017 (N = 130 for aspirin use, N = 161 for blood pressure control, and N = 130 for smoking screening/cessation counseling).

Average improvements for all 3 CQMs—the percent of eligible patients using aspirin (from 66.8% to 70.8%), having controlled BP (from 61.5% to 64.8%), and receiving smoking screening and cessation counseling (from 73.8% to 81.3%)—were statistically significant as well.

There was also strong evidence in favor of an association between change in QICA and change in the continuous BP CQM in the adjusted analysis: for every 1-point increase in QICA score, we would expect an additional 1.25% of patients with hypertension to have well-controlled BP defined as a measurement of less than 140/90 mm Hg (95% CI, 0.41%-2.09%; P = .003) (Table 2). There was little evidence to support associations between change in QICA score and changes in either aspirin use or smoking intervention CQMs as continuous measures, given that neither observed association was statistically significant.

Table 2.

Mean Change in Clinical Quality Measure With Each 1-Point Increase in QICA Score

Clinical Quality Measure	Unadjusted Mean Change (95% CI)	Adjusteda Mean Change (95% CI) [P Valueb]
Aspirin use (N = 130)	‒0.58 (‒3.44 to 2.28)	‒0.20 (‒2.61 to 2.21) [.87]
Blood pressure control (N = 161)	1.74 (0.74 to 2.74)	1.25 (0.41 to 2.09) [.003]
Smoking screening/cessation counseling (N = 130)	0.16 (‒1.88 to 2.20)	0.52 (‒1.20 to 2.24) [.55]

QICA = Quality Improvement Capacity Assessment.

^aAdjusted for baseline Clinical Quality Measure.

^bFrom Wald test for difference of coefficient from 0; see Methods for assumptions made to estimate the standard error.

We conducted analyses to determine whether the adjusted association between change in QICA and change in the BP CQM were driven by results from outlier practices by calculating DFBETA statistics, which show the effect that removing each observation has on the estimates for regression coefficients.³³ Removing clinics that caused the biggest changes in regression estimates did not result in any changes to inference. We conducted the same analysis for aspirin and smoking CQMs to determine whether outlying clinics attenuated the estimated association but found no evidence of such influence.

Table 3 shows the association between change in QICA score and likelihood of meeting the 70% threshold in 2017 for each of the clinical quality outcomes. With each 1-point increase in QICA score, a clinic was estimated to be 1.24 times more likely to reach the 70% threshold for BP control (95% CI for relative risk, 1.09-1.40; P <.001); however, evidence of associations for the aspirin and smoking outcomes was weak.

Table 3.

Likelihood of 2017 Clinical Quality Measure Being Greater Than 70% With Each 1-Point Increase in QICA Score

Clinical Quality Measure	Unadjusted RR (95% CI)	Adjusteda RR (95% CI) [P Valueb]
Aspirin use (N = 130)	1.05 (0.87-1.26)	1.03 (0.87-1.24) [.72]
Blood pressure control (N = 161)	1.21 (1.04-1.41)	1.24 (1.09-1.40) [<.001]
Smoking screening/cessation counseling (N = 130	1.00 (0.95-1.05)	1.01 (0.96-1.06) [.78]

QICA = Quality Improvement Capacity Assessment; RR = relative risk.

^aAdjusted for Clinical Quality Measure value at baseline.

^bWald test for difference of RR from 1; see Methods for assumptions made to estimate the standard error.

DISCUSSION

With 15 months of external practice support, we observed significant improvements in both QI capacity and all 3 cardiovascular risk factor performance measures in these smaller primary care practices. In addition, the observed improvement in QI capacity was associated with improved performance on the BP control CQM, but not on the appropriate aspirin use or smoking cession measures. For each 1-point increase in the QICA score, practices were 24% more likely to reach the Million Hearts campaign goal of 70% of patient with well-controlled BP (<140/90 mm Hg).

Why was improved QI capacity observed to be associated with improvements in BP control, but not in appropriate aspirin use or smoking screening and cessation counseling? One possible explanation is that improving BP control is a much more complex organizational process than the other 2 measures that can show rapid improvement through more thorough documentation in the electronic health record.³⁴ Existing literature suggests that team-based multicomponent efforts are required to improve BP control clinical performance measures and include engaging staff in proper measurement, intensifying medication therapy by clinicians, and supporting patients around medication adherence and self-management.^35,36 The measure of QI capacity used in this study accounted for many if not most of these BP improvement activities, including support for patients in self-management activities. Another possible explanation is that practices built their QI capacity by focusing on BP control. In a separate analysis of notes kept by practice facilitators after each encounter with a practice, improving BP measurement was the most common focus of the Plan-Do-Study-Act cycles of improvement that practices engaged in.²⁶

Several limitations of this analysis deserve mention. First, the QICA is a self-reported tool that was completed by clinical teams whose composition changed over the course of the project. As such, the change in score may not reflect change in QI capacity. Facilitators reviewed baseline practice QICA scores with the teams, however, and used these scores to guide improvement efforts. This assistance is reflected in the finding that the baseline QICA score was predictive of both the number and the types of topics discussed.³⁷ For follow-up scores, facilitators were asked if they agreed with the practice’s self-scores, specifically, that the scores accurately reflected externally observable changes in care processes; most agreed. In addition, changes in the HLC subscores were greatest for the HLCs where the facilitators focused most of their time and effort, which increases confidence that the tool is measuring real improvements.³⁸ Another study limitation is that the second QICA evaluation occurred during the observation period for the follow-up CQMs. This overlap may have biased results toward the null hypothesis of no association between change in QICA score and change in CQM, because the changes made in the practice to improve QI capacity may have not had adequate time to impact improvements in quality measures. Finally, there was no control group of practices that did not receive the practice facilitation intervention against which to compare these findings.

By prospectively defining a package of evidence-based strategies, then assessing their implementation, this study provides evidence for a plausible pathway from external practice support to improvements in patient cardiovascular risk factors via improvements in QI capacity. It also provides a key opportunity to close the gap between known best practices in the literature for addressing cardiovascular risk and common practice by defining a set of 7 effective strategies that can be spread to others.^39,40 Practices were able to make improvements in all areas, with most improvements occurring in the domains related to QI where facilitators focused their efforts. Measuring implementation progress on those HLCs using the 20-item QICA was feasible for small practices and may provide an intermediate measure that is sensitive to change in the short term. In addition to measuring progress on improving QI capacity, change in the QICA score was associated with changes in a clinical quality measure. The research field continues to grapple with how best to implement and disseminate evidence-based strategies, especially to smaller practices that lack robust QI infrastructure, but where many Americans continue to receive care.^41-43 Using a prospectively designed set of evidence-based strategies with a concordant tool for assessing implementation may be a strategy for accelerating our translation of evidence into practice as it can serve as a tested guide for future spread and sustainment efforts.⁴⁴

This study makes a unique contribution to the literature by demonstrating that the 7 HLCs used may provide a reasonable set of activities for small practices to undertake over a relatively short time period to build their QI capacity for the purpose of improving clinical outcomes. Of interest to policy makers, it appears the relatively light support provided by a facilitator was sufficient in fostering change within the complex system of a primary care practice,^45,46 although variation in context is important and more research is needed to understand how practice characteristics influence their ability to implement and sustain change.