Clinical Quality Measure Exchange is Not Easy

Abstract

PURPOSE

The Trial of Aggregate Data Exchange for Maintenance of Certification and Raising Quality was a randomized controlled trial which first had to test whether quality reporting could be a by-product of clinical care. We report on the initial descriptive study of the capacity for and quality of exchange of whole-panel, standardized quality measures from health systems.

METHODS

Family physicians were recruited from 4 health systems with mature quality measurement programs and agreed to submit standardized, physician-level quality measures for consenting physicians. Identified measure or transfer errors were captured and evaluated for root-cause problems.

RESULTS

The health systems varied considerably by patient demographics and payer mix. From the 4 systems, 256 family physicians elected to participate. Of 19 measures negotiated for use, 5 were used by all systems. There were more than 15 types of identified errors including breaks in data delivery, changes in measures, and nonsensical measure results. Only 1 system had no identified errors.

CONCLUSIONS

The secure transfer of standardized, physician-level quality measures from 4 health systems with mature measure processes proved difficult. There were many errors that required human intervention and manual repair, precluding full automation. This study reconfirms an important problem, namely, that despite widespread health information technology adoption and federal meaningful use policies, we remain far from goals to make clinical quality reporting a reliable by-product of care.

INTRODUCTION

Family physicians provide nearly 20% of all clinical outpatient visits, nearly 200 million visits in the United States annually.¹ Frontline clinicians continue to report failures of certified electronic health records (EHRs) to meet federal certification requirements and to meet electronic reporting needs, the latter of which is estimated to cost them $15 billion per year.^2-11 Reducing reporting burden and enhancing the portability and utility of clinical data for quality improvement is an American Board of Family Medicine (ABFM) goal that is aligned with federal policy.^12,13 In 2013, the ABFM began development of a Qualified Clinical Data Registry to assist practices whose EHRs were not serving their quality reporting needs and to reduce reporting burden. That registry, called PRIME, launched in 2016 and now actively extracts data from EHRs in more than 800 practices in 47 states (https://www.primeregistry.org).

Beyond reducing reporting burden, the ABFM aims to help practices easily access their own data for quality assessment and improvement and to reduce continuous certification burden. The ABFM recognized that many primary care physicians would not be able to connect directly to PRIME and that another passive data path was needed by which an EHR vendor, institution, or clinic system could send aggregated quality measures to the PRIME registry.^12,14,15

The objective of this study was to assess whether quality measure reporting could be made a by-product of clinical care and quality improvement. We report here on the functional testing of quality measure exchange, that is to say, a test of the reliable exchange of whole-panel, quality measures from clinical networks. This component of the study was integral to a broader effort called the TRial of Aggregate Data Exchange for Maintenance of certification and Raising Quality (TRADEMaRQ).^14,16,17 The study partners were 4 health systems with mature, established clinician quality-measurement and tracking systems. All 4 systems were willing to test ways to automate direct transmission of clinical measures.

METHODS

While TRADEMaRQ was a randomized, controlled trial, we report on the descriptive study component of the measure exchange testing experience.

Recruitment

Each of our clinical partners recruited family physicians in their networks or participating networks via face-to-face and e-mail solicitations. The ABFM provided study information in written form or via webinar, after which those physicians wanting to participate provided consent. Eligible participants were made aware that participation was voluntary and that the ABFM would not know their identity.

Measures and Data Sharing

Quality measures were calculated per each organization’s protocol and uploaded to the ABFM’s Secure File Transfer Protocol (SFTP) server. Measure exchange was designed to occur every 2 weeks.

Southeast Texas Medical Associates (SETMA) had an extensive history of reporting on quality metrics and publishes them, by clinician, on their website. The only additional work that was required was to format the results per the specifications of the study and to limit the results sent to family physicians participating in the study. They reported that quality control and verification of new files took approximately 4 to 6 hours and finalized files were placed into an automated job on the report server for reporting at 2-week intervals. Before each file set was submitted, they were given a brief inspection for correct formatting to make sure the data did not appear abnormal.

Kaiser Permanente Washington (KPWA) chose to report their clinical measures from their Clinical Improvement & Prevention department’s Healthcare Effectiveness Data and Information Set (HEDIS) measures, which are used both for monthly reporting and Medicare Star Rating calculations; however, KPWA outsourced these calculations to an National Committee for Quality Assurance-certified vendor who consumed KPWA data to produce annual HEDIS measures. KPWA also receives a “measure engine” from the vendor which allows KPWA to run HEDIS calculations themselves on a rolling basis.

Kaiser Permanente Colorado (KPCO) explained that although development of their reporting process was the most labor-intensive effort (350 hours), data processing and maintenance efforts (350 hours combined) were also substantial, resulting in an average 9 hours spent per data file transfer. KPCO developed a program specifically for this study to produce physician-level measures and it was run biweekly by a data specialist who also had to review and approve outputs.

OCHIN (a nonprofit health care innovation center focused on the needs of community health centers, small practices, and critical access hospitals) did not provide documentation about their process or related problems.

The study was approved by the American Academy of Family Physicians’ Institutional Review Board after all parties entered into an Office for Human Research Protections-approved federal-wide authority consortium agreement.

RESULTS

Enrollment and Participation

Across the 4 networks, there were 2,570 eligible family physicians, and 256 participated (Table 1). Participation ranged from 4.4% of OCHIN physicians to 100% of eligible SETMA physicians.

Table 1.

TRADEMaRQ-Enrolled Physicians

Health System	Number Enrolled
Control	125
Kaiser Permanente Washington	90
Kaiser Permanente Colorado	66
OCHIN	94
Southeast Texas Medical Associates	6
Total	256

OCHIN = a nonprofit health care innovation center focused on the needs of community health centers, small practices and critical access hospitals; TRADEMaRQ = Trial of Aggregate Data Exchange for Maintenance of certification and Raising Quality.

Data and Transfers

The 4 partners agreed to 19 common clinical quality measures with our site co-principal investigators; across the 19 measures, however, initially only 4 were used by all health systems (Table 2) and only 5 were used by all systems by the end of the study. More than 15 examples of transmission or receipt errors were identified (Table 3). KPCO documented 17 different ways that their process was interrupted, failed, or had to be revised. Some types of errors, such as breaks in data delivery, changes in measure values, and nonsensical measure results were readily identifiable. Only SETMA had no identified errors—the only system to routinely list physician-level measures on their website for public viewing. KPWA noted 2 complications: (1) HEDIS measures change over time, both the measures required and the measure definitions; and, (2) an annual data validation process that typically delays data availability until the second quarter. Once the data were available from the vendor, KPWA wrote programs to automatically upload a physician-level measure file. KPWA staff noted, “we have internal dashboards with more complete and timely data than what is reported in HEDIS, and this drives our providers absolutely crazy because the two do not agree, and we as a medical group are being judged based on HEDIS measures, not on what we know is data that better reflects our true practice of medicine and delivery of care. But no one is willing to stop producing our internal dashboards because those are used to actually implement changes that improve quality of care, not the HEDIS measures.” This revelation helped explain complaints we were receiving from KPWA physicians who noted that their quality measures in their certification portfolio did not reflect their internal dashboard measures.

Table 2.

TRADEMaRQ Measures across Participating Health Systems

CMS eMeasure ID	NQF #	Measure Title	KPWA	KPCO	OCHIN	SETMA
CMS165v2	18	Controlling High Blood Pressure	X	X	X	X
CMS138v2	28	Smoking Cessation Counseling		X	X
CMS125v2	31	Breast Cancer Screening	X	X	X	X
CMS124v2	32	Cervical Cancer Screening	X	X	X	X
CMS130v2	34	Colorectal Cancer Screening	X	X	X	X
CMS147v2	41	Preventive Care and Screening: Influenza Immunization			X	X
CMS127v2	43	Pneumonia Vaccination Status for Older Adults			X	X
CMS131v2	55	Diabetes: Eye Exam	X	X	X	X
CMS123v2	56	Diabetes: Foot Exam		X	X	X
CMS122v2	59	Diabetes: Hemoglobin A1c Poor Control		X	X	X
CMS134v2	62	Diabetes: Urine Protein Screening	X		X	X
CMS163v2	64	Diabetes: Low Density Lipoprotein (LDL) Management			X	X
CMS164v2	68	Ischemic Vascular Disease (IVD): Use of Aspirin or Another Antithrombotic		X	X
CMS145v2	70	Coronary Artery Disease (CAD): Beta-Blocker Therapy—Prior Myocardial Infarction (MI) or Left Ventricular Systolic Dysfunction (LVEF <40%)	X		X
CMS182v3	75	Ischemic Vascular Disease (IVD): Complete Lipid Panel and LDL Control		X	X
CMS135v2	81	Heart Failure (HF): Angiotensin-Converting Enzyme (ACE) Inhibitor or Angiotensin Receptor Blocker (ARB) Therapy for Left Ventricular Systolic Dysfunction (LVSD)	X	X	X
CMS144v2	83	Heart Failure (HF): Beta-Blocker Therapy for Left Ventricular Systolic Dysfunction (LVSD)			X
CMS2v3	418	Preventive Care and Screening: Screening for Clinical Depression and Follow-Up Plan			X
CMS68v3	419	Documentation of Current Medications in the Medical Record			X	X

CMS = Center for Medicare & Medicaid Services; NQF = National Quality Forum; KPWA = Kaiser Permanente Washington; KPCO = Kaiser Permanente Colorado; SETMA = Southeast Texas Medical Associates; OCHIN = OCHIN is a nonprofit health care innovation center focused on the needs of community health centers, small practices and critical access hospitals.

Table 3.

Common Measure Delivery or Receipt Errors and Hazards Experienced in TRADEMaRQ

Error Types	Examples	Fix
Measure miscalculation	Incorrect numerator or denominator Measures calculated incorrectly sent from inception, noticed 11 months after system launch Incorrect data period (measurement period required 12 months, but 11 months used) Incorrect denominator inclusion criteria used for greater than 1 year Numerator >denominator error Patient panel (erroneously) reduced to 0 Physician moved clinics and changed panels so that measures could not be reconciled; removed from the study Significant change in scores for 5 measures	Corrected data sent and uploaded manually after manual removal of incorrect data Revised measure calculations Error caught internally and repaired; delayed transmission Fixed reporting period compression error
Data delivery error	Delay in data delivery Blank file sent Incorrect NQF number attached to file	Corrected, resent, manual data replacement
Non-enrolled physician data sent	Physician data sent before they were enrolled/randomized Ineligible physician data sent	Physician enrolled and randomized or excluded
Data reporting interrupted	Physician data reported for one period but not another Internal system change caused a measure to not get reported Source database moved and transmission credentials not configured Critical subsystem source failure, 6-week delay	Updated files sent and manually uploaded
Host receiving server not running	System update interruption	Server brought back online
Third party errors	Two years into study, learned that a third-party company was doing measure management and transmitting incorrectly Third-party processes caused several month delays in file transmission around turn of calendar year	Worked directly with vendor to correct calculation or transmission errors Files caught up once data sent by third party

NQF = National Quality Forum; TRADEMaRQ = Trial of Aggregate Data Exchange for Maintenance of certification and Raising Quality.

DISCUSSION

This paper reports on a test of exchange of whole-panel, standardized quality measures from clinical networks and health systems with mature internal quality measure processes, all of whom reported to HEDIS and Centers for Medicare & Medicaid Services (CMS). Of 19 measures that the 4 agreed were common and standard, only 5 measures were used by all 4. Dozens of identified errors or hazards in measure calculation, data management, data delivery, and physician inclusion occurred. These problems often required human intervention and manual repair, precluding fully automated file transfers. In the end, there was a steady stream of reported data and most of the enrolled physicians were able to interact with them on the secure ABFM website. We conclude that it was really a test of system-level breakdown in measure definition, process, capture, and transmission rather than about the particulars of individual clinician quality; it is a symptom of the bigger system problem.

The “passive” data reporting path that TRADEMaRQ sought to test was related to the intentions of the Health Information Technology for Economic and Clinical Health Act.¹⁸ Specifically, it promoted the Office of the National Coordinator for Health Information Technology (ONC) Direct Project which aimed to create a simple, secure, scalable, standards-based way to transmit health information from a sender to a trusted recipient over the Internet.¹⁹ Direct Project priority 2 specifically mentions reporting of quality measures to a variety of potential recipients. The TRADE-MaRQ aims also aligned with the ONC Nationwide Health Information Exchange intentions, namely to support secure health information exchange for measuring and improving quality tied to incentives for improvement and required reporting.^20,21 This study reconfirms an important problem, namely, that despite continued growth in health information technology adoption and calls for interoperability, most clinical practices struggle to produce standard quality measures from their clinical data and to meet a growing number of reporting needs.²² Several large-scale, federal demonstration and research efforts have run headlong into this problem and are struggling to get measure data out of participating practices.¹¹ Although many clinicians are now reporting to the federal Quality Payment Program and hardship exceptions are made for small practices, they are at increasing risk of being left behind because NCQA’s patient-centered medical home recognition program is moving to annual electronic measure reporting, and many public health registries and even some payers are also requiring electronic measure submission.⁵ Quality reporting is unlikely to disappear but the reliability of what is reported is unclear, meaningfulness uncertain, and the cost high.^6,7

A major limitation of our study is that the problems identified may not generalize to other health systems.

CONCLUSION

TRADEMaRQ and other recent studies suggest that we remain far from clinical data liquidity goals of federal agencies. The ABFM’s goals to help physicians make reporting a by-product of care and ongoing quality improvement, and reduce physicians’ reporting burdens, remain aspirations. While small and independent practices struggle with turning their EHR data into reportable measures, large health systems also struggle with managing measure definitions, internal vs external quality processes, and costly processes for managing reporting.