The Development and Endorsement of a Performance Measure for Stroke Misdiagnosis in the Emergency Department

THE NEED FOR DIAGNOSTIC QUALITY PERFORMANCE MEASURES

Diagnostic error remains a problem in public health.¹ Three major disease categories—vascular events, infections, and cancer—account for three-fourths of all serious harms from diagnostic error,² with missed stroke the leading cause of serious harm to patients.³ The misdiagnosis of stroke disproportionately occurs when patients present with symptoms that are non-typical or obvious,^4,5 such as dizziness or vertigo, which can be mistaken for inner ear disease.⁴ Annually in the United States, an estimated 45 000–75 000 patients that present to the emergency department (ED) with dizziness or vertigo caused by stroke are misdiagnosed and erroneously discharged.⁵

ED patients with dizziness and vertigo could be correctly diagnosed as having a stroke if evidence-based interventions are followed,^6,7 but there remains a large evidence-to-practice gap.⁸ Without a timely and accurate diagnosis, these patients suffer misdiagnosis-related harms⁹ because they do not receive prompt treatment for this time-sensitive condition.⁴ The lack of operationally viable performance measures remains a barrier for hospitals to improve their diagnostic quality.^10,11

AVOID H.A.R.M.—ED STROKE/DIZZINESS MEASURE

To address this gap, the team at the Armstrong Institute Center for Diagnostic Excellence developed the Avoid Hospitalization After Release with a Misdiagnosis (Avoid H.A.R.M.)—ED Stroke/Dizziness quality measure, which tracks the rate of missed strokes in the ED. Using the Symptom-disease Pair Analysis of Diagnostic Error (SPADE) framework,¹² the claims-based measure tracks the rate of adult patients treated and released from the ED with either a non-specific, presumed benign symptom-only dizziness diagnosis or a specific inner ear/vestibular diagnosis (collectively referred to as “benign dizziness”) who were subsequently admitted to a hospital for a stroke within 30 days of their ED visit. The measure accounts for the epidemiological base rate of stroke in the population under study using a risk difference approach, comparing the short-term (0–30 days) and long-term (91–360 days) incidence rates of stroke.

MEASURE ENDORSEMENT

The measure was put through the Centers for Medicare and Medicaid Services (CMS) consensus-based entity measure endorsement process, undergoing rigorous reviews by multistakeholder committees against established criteria. Most performance measures used in accountability programs at a national level are endorsed. The endorsement process evaluates measures on four criteria: Importance to Measure, Scientific Acceptability, Feasibility, and Use and Usability.

While the measure was eventually endorsed, our path to endorsement was not straightforward. At two different points in the process, we encountered concerns from the committees charged with evaluating the first two criteria. We needed to submit and resubmit the measure three times before final endorsement.

To establish the Importance to Measure, we needed to demonstrate a relationship between an existing health care structure or process and the outcome of interest. We initially provided evidence that providers trained in eye-movement-based bedside diagnosis of stroke using the head impulse, nystagmus, test of skew (HINTS) exam had improved identification of stroke compared to standard imaging approaches. Use of these bedside diagnostic maneuvers is endorsed by the Society for Academic Emergency Medicine as current best practice.¹³ The committee reviewing this criterion expressed concerns about the limited number of providers currently trained on HINTS, challenges with scaling training, and eventually discounted HINTS as insufficient to produce a difference in reducing misdiagnosed strokes. In response, we submitted data that examined the relationship between a hospital’s use of neuroimaging in patients with “benign dizziness” and the hospital’s performance on the Avoid H.A.R.M measure. We provided comparisons of measure-based outcomes when hospitals used MRI scans versus CT scans, as MRIs are more sensitive for acute strokes in the back part of the brain than CT.

For the Scientific Acceptability criterion, we needed to demonstrate the measure’s validity and reliability. To demonstrate the reliability, we presented a signal-to-noise analysis across all measured entities derived from the full Medicare Fee-For-Service (FFS) dataset for a four-year period. The committee raised concerns about the presented data, both the measure’s signal and its ability to discriminate high-performing from low-performing hospitals. We reran our analyses with a dataset that included a fuller group of patients at a hospital, as Medicare FFS beneficiaries only reflect approximately 20% of a hospital’s total ED visits. For the updated analysis we used the Agency for Healthcare Research and Quality’s Healthcare Cost and Utilization Project (HCUP) data for Florida hospitals, which includes all patients seen at a hospital. With this greater capture of patients, our statistics improved and we were able to better demonstrate the measure’s signal and discrimination.

The respective committees found our updated submissions to meet the criteria and the measure became the first diagnosis-related outcome measure to be endorsed.

OUR LEARNING

The endorsement process was helpful in our work to advance diagnosis-related performance measurement. Our experience could inform others in two primary ways: First, the committee reviewing the Importance to Measure criterion wanted wide-spread and scalable approaches that hospitals could use to improve diagnostic performance, so as others think about developing diagnosis-related outcome measures, having clear, scalable improvement interventions is important. Second, for us to demonstrate the Scientific Acceptability criterion, we needed to employ datasets that were more encompassing than what Medicare FFS alone could offer. We have incorporated this learning into choosing which datasets we’ll use moving forward for this work.

NEXT STEPS

Our next phase focuses on the use of the measure in accountability programs. We are speaking with states, payers, and health care quality organizations about how the measure could be used in their programs. Although endorsement of the measure is helpful for a measure’s utility, endorsement does not guarantee the measure’s real-world use.

The endorsement process highlighted that the uptake of the measure would benefit from the accompaniment of targeted “balancing” measures. For example, concerns were raised during the endorsement process that diagnostic quality measures might prompt inappropriate increases in testing. Specific to the ED-stroke/dizziness measure, there were concerns about the increased use of neuroimaging for “benign dizzy” patients. It would be ideal to create companion measures that track CT and MRI utilization in patients with “benign dizziness” to ensure that there are no unintended consequences of inappropriate imaging.