Qualitative evaluation of a cardiovascular quality improvement programmereveals sizable data inaccuracies in small primary care practices

Introduction

Among the most promising quality improvement (QI) interventions for small primary care practices are those led by practice facilitators (PFs), specially trained individuals who help practices develop capacity for continuous QI.^{1 2} They provide coaching on best practices for QI implementation, including using technology to improve care.³ PF-led QI initiatives are positively associated with guideline adoption,^{4 5} and may be cost-neutral if they reduce even a small number of high cost events (eg, admissions).⁶ As part of Healthy Hearts in the Heartland (H3), a programme from the Agency for Healthcare Research and Quality’s EvidenceNow initiative, PFs worked with small and medium-sized primary care practices to implement QI strategies for cardiovascular disease prevention.⁷ To identify lessons learnt from the programme, we interviewed practice leaders and PFs from practices that experienced the largest and smallest gains in quality scores to understand their experiences.

Methods

All participating practices were assigned a primary PF for 12 months who met with practices on demand, typically once a month. PFs offered practices QI interventions related to the ABCS of heart health (Aspirin therapy, Blood pressure control, Cholesterol management, and Smoking screening and cessation) with the goal of improving four ABCS measures that are used in national quality incentive programmes, such as the Merit-based Incentive Payment System.^{8 9} Information about the H3 intervention, outcome measures and study design can be found elsewhere.^{10 11}

Practice leaders from 16 practices with large improvement on the ABCS measures after 12 months, and 15 practices with minimal improvement after 12 months received up to 6 contact attempts asking them to complete a 30 min telephone interview. Practice leaders were individuals at the practice who were most familiar with the intervention, generally physicians and QI managers. Following commitment from the practice leader, we invited the corresponding PF to complete a separate interview. Interviews were conducted between March and April 2018, ~8 months after the 12-month intervention period.

Semi-structured interview protocols were constructed based on the Consolidated Framework for Implementation Research.¹² Interviews were digitally recorded and analysed iteratively and inductively for emergent themes and patterns using the constant comparison approach.

Results

We completed interviews with practice leaders from 14 of 31 eligible practices (45%), and all 7 PFs assigned to those practices (table 1). On average, practices implemented 5.7 electronic health record (EHR)-based QI strategies (eg, clinical decision support prompts) and 7.4 non-EHR strategies (eg, workflow changes).

Table 1.

Characteristics of participating practices

	Characteristics of respondents from practices with the greatestimprovement in ABCS scores (n=5)	Characteristics of respondents from practices with the leastimprovement in ABCS scores(n=9)
No of providers in the practice
1	2	0
2–5	1	5
6–10	1	0
11–20	0	4
Part of larger health system, % yes	80	33
State
IN	2	2
IL	2	2
WI	1	5
Median number of H3 QI encounters over 12 months (IQR)	9 (7to 12)	10 (6to 11)
Median percentage improvement on aspirin scores (IQR)	11%(4%to 30%)	3%(−20%to 9%)
Median percentage improvement on blood pressure scores (IQR)	7%(0%to 16%)	−3%(−20% to 8%)
Median percentage Improvement on cholesterol scores (IQR)	12%(7%to 32%)	−10%(−15% to −7%)
Median percentage improvement on smoking scores (IQR)	2%(0%to 18%)	0% (−27% to 6%)

ABCS, ABCS scores—A=ischaemic vascular disease: use of aspirin or other antithrombotic (CMS164v4); B=controlling high blood pressure (CMS165v4); C=statin therapy for the prevention and treatment of cardiovascular disease (CMS PREV-13); and S=preventive care and screening: tobacco use: screening and cessation intervention (CMS138v4).

H3, Healthy Hearts in the Heartland; QI, quality improvement.

The practices experienced sizeable changes in ABCS performance measures—both positive and negative—over the 12-month assessment period (table 2). Although most practice leaders and PFs described H3 positively, and could offer examples of how H3 improved care in the practices, respondents typically noted that the largest changes in ABCS scores likely reflected improvements in documentation due to coaching or fixes to EHR data ‘glitches’ rather than changes in care delivery (eg,table 2, practice E). In other cases, respondents were puzzled by observed changes in measured performance, but could not attribute large improvements (or declines) in performance to the H3 interventions (eg,table 2, practice B).

Table 2.

Examples of large changes in ABCS scores, and perceptions of changes by practice facilitators

Example practice	Changes in ABCS scores	Quote
Practice A	A:+11% B:+7% C:+3% S:+34%	‘Once the provider realized [documentation]had to be in the screening section, that’s when we saw improvement [on the smoking score]. She was doing the counseling, but it wasn’t picking up in the report’.
Practice B	A:+13% B: −2% C:+11% S: +2%	‘It surprises me that they had such jumps in aspirin and cholesterol, because we didn't really cover those topics(under H3)’.
Practice C	A: +9% B: −26% C: −10% S:+1%	‘[The scores are]not what I would have expected…For BP, I would have expected to see improvement after H3. This [practice]reached out to all patients not diagnosed with hypertension but who had a high BP reading in the past 6 months—40 people. They were invited back in to have BP tested again. Some were put on BP medication due to that second visit, others were back to normal. Three people were sent directly to the emergency room. This was a great moment for the [practice]—they made a big impact’.
Practice D	A: −25% B:+9% C:−10% S:+9%	‘BP and smoking were the two that were focused on(under H3). Others were not a high priority. So, I was glad that BP and smoking improved. They report aspirin through Epic, and there were some concerns about those numbers at 12 months. There might have been a glitch’.
Practice E	A:+46% B:+20% C:+48% S:+2%	‘This practice was complicated in the fact of they had a brand new EHR…The baseline data we had wasn't great. I don't think [the scores are]a true reflection of what the practice was doing’.
Practice F	A: −34% B: −24% C: −10% S: −26%	‘[The practice was]so successful with implementation…The culture is so team oriented. Everyone would participate. They organized monthly meetings…so the time was set aside(for H3)without interruption. [They had]full support from administration and the CMO…There was a glitch in the smoking data that was fixed right after 12 months, so the [scores should show]improvement by 18 months. The cholesterol numbers were based on chart review. I’m not sure why there was a decline in Aspirin and Cholesterol. We spent time on both’.
Practice G	A:+10% B:+2% C:+12% S:+1%	‘I’m not surprised by [the gains in]the aspirin score. We first started by looking at numbers and [the practice leaders were]surprised by how low they were. We discovered that many of the visits were for mental or behavioral health, not necessarily primary care. For the primary care visits, providers were not adding aspirin to medication lists.(Under H3), the providers made a concerted effort to look at and pay attention to that. Whether it drove the 10% increase, I don't know’. ‘I am surprised by the cholesterol scores. I don't recall doing PDSA or interventions focused on cholesterol’.
Practice H	A: −20% B: −29% C: Score not available S: −67%	‘[This practice was]not a good fit for H3 just because of limits on my ability to access their EHR. The data were a barrier for this clinic. They were mistakenly thinking it would cost them thousands of dollars to get the data we needed’.

H3, Healthy Hearts in the Heartland.

Discussion

In this evaluation of a PF-led QI intervention, we found a number of practices with sizeable changes in performance scores after 12 months. While the largest changes in scores may not reflect actual changes in care delivery, in practices where data accuracy improved, the changes represent success for the H3 programme. Those practices are now better prepared to engage in QI and pay-for-performance efforts that rely on EHR data.

Our results highlight the importance of mixed methods research, which provides a richer contextual lens to judge the success of QI interventions. A limitation of our study is reliance on ABCS measures as our quality indicators. H3 interventions may have improved care processes uncaptured by the measures. Also, our analysis relied on perceptions of only practice leaders and PFs, and our sample is small. However, our findings are consistent with the broader evaluation of EvidenceNow, and evaluations of similar efforts showing that small practices continue to struggle with EHRs.^{13 14} Federal investments in EHR adoption and technical assistance were made available to practices with the expectation that EHRs would generate meaningful performance data, enabling QI and leading to improved care delivery.¹⁵ However, our findings show that some small practices continue to operate with limited or incorrect performance data. Our results should lend caution to pay-for-performance programmes that rely on EHR data.