Underreporting High-Risk Prescribing Among Medicare Advantage Plans

Abstract

Background

Although Medicare Advantage plans are required to report clinical performance using Healthcare Effectiveness Data and Information Set (HEDIS) quality indicators, the accuracy of plan-reported performance rates is unknown.

Objective

To compare calculated and reported rates of high-risk prescribing among Medicare Advantage plans.

Design

Cross-sectional comparison.

Setting

172 Medicare Advantage plans.

Patients

A random sample of beneficiaries in 172 Medicare Advantage plans in 2006 (n = 177 227) and 2007 (n = 173 655).

Measurements

Plan-reported HEDIS rates of high-risk prescribing among elderly persons were compared with rates calculated from Medicare Advantage plans’ Part D claims by using the same measure specifications and source population.

Results

The mean rate of high-risk prescribing derived from Part D claims was 26.9% (95% CI, 25.9% to 28.0%), whereas the mean plan-reported rate was 21.1% (CI, 20.0% to 22.3%). Approximately 95% of plans underreported rates of high-risk prescribing relative to calculated rates derived from Part D claims. The differences in the calculated and reported rates negatively affected quality rankings for the plans that most accurately reported rates. For example, the 9 plans that reported rates of high-risk prescribing within 1 percentage point of calculated rates were ranked 43.4 positions lower when reported rates were used instead of calculated rates. Among 103 680 individuals present in both the sample of Part D claims and HEDIS data in 2006, Medicare Advantage plans incorrectly excluded 10.3% as ineligible for the HEDIS high-risk prescribing measure. Among those correctly included in the high-risk prescribing denominator, the reported rate of high-risk prescribing was 21.9% and the calculated rate was 26.2%.

Limitation

A single quality measure was assessed.

Conclusion

Medicare Advantage plans underreport rates of high-risk prescribing, suggesting a role for routine audits to ensure the validity of publicly reported quality measures.

Primary Funding Source

Health Assessment Lab and National Institute on Aging.

Public reporting of clinical performance data has proliferated in the past decade and is now an established cornerstone of efforts to measure and improve the quality of care (1). The Patient Protection and Affordable Care Act directs the Centers for Medicare & Medicaid Services (CMS) to use performance measures when determining payment rates for hospitals, Medicare Advantage plans, and other health care providers through pay-for-performance incentives or other value-based purchasing programs (2). Publicly reported performance measures also help patients select health plans, hospitals, and providers (3).

Despite the ubiquity and high-stakes consequences of public performance reports, few studies audit the reliability of publicly reported data submitted by insurers and providers (4, 5). Such audits are difficult because performance measures have complex criteria for including patients in the numerator and denominator and require information from medical records (6). To be useful, these audits must accurately identify which patients meet the criteria for inclusion in the performance measure and specify a representative group of these patients.

Prior audits by the U.S. General Accounting Office and Health Care Financing Administration (now the CMS) have not confirmed the reliability of performance reporting. A 2006 General Accounting Office report revealed that the CMS could not ensure the completeness of publicly reported data from U.S. hospitals, noting that quality data may not be reliable if hospitals incorrectly exclude eligible patients (7, 8). Similarly, a 1998 Health Care Financing Administration audit of 7 Healthcare Effectiveness Data and Information Set (HEDIS) measures identified reporting discrepancies in nearly 60% of care plans managed by Medicare (9). More recent federal audits have not been published. Although plans reporting to HEDIS must undergo the annual National Committee for Quality Assurance HEDIS Compliance Audit, this process assesses only an organization’s ability to adhere to reporting specifications without validating performance rates for specific quality measures (10).

The objective of our study was to examine the accuracy and completeness of reporting of the HEDIS 2006 and 2007 Drugs to Avoid in the Elderly indicator (Appendix Table 1, available at www.annals.org) among Medicare Advantage plans. These private managed care plans, which now enroll over 25% of all Medicare beneficiaries, receive capitated payments from the CMS to provide Medicare-covered services for their enrollees (11). Since 1997, the CMS has required all Medicare Advantage plans to publicly report their clinical performance using HEDIS measures (12).

Appendix Table 1.

HEDIS Drugs to Avoid in the Elderly, 2006–2007

Class	Drugs
Antianxiety*	Aspirin–meprobamate
	Meprobamate
Antiemetics	Scopolamine
	Trimethobenzamide
Analgesics*	Ketorolac
Antihistamines*	APAP/dextromethorphan/diphenhydramine
	APAP/diphenhydramine/phenylephrine
	APAP/diphenhydramine/pseudoephedrine
	Acetaminophen–diphenhydramine
	Carbetapentane/diphenhydramine/phenylephrine
	Codeine/phenylephrine/promethazine
	Codeine–promethazine
	Cyproheptadine
	Dexchlorpheniramine
	Dexchlorpheniramine/dextromethorphan/PSE
	Dexchlorpheniramine/guaifenesin/PSE
	Dexchlorpheniramine/hydrocodone/phenylephrine
	Dexchlorpheniramine/methscopolamine/PSE
	Dexchlorpheniramine–pseudoephedrine
	Dextromethorphan–promethazine
	Diphenhydramine
	Diphenhydramine/hydrocodone/phenylephrine
	Diphenhydramine–magnesium salicylate
	Diphenhydramine–phenylephrine
	Diphenhydramine–pseudoephedrine
	Hydroxyzine hydrochloride
	Hydroxyzine pamoate
	Phenylephrine–promethazine
	Promethazine
Antipsychotics, typical	Thioridazine
Amphetamines	Amphetamine–dextroamphetamine
	Benzphetamine
	Dexmethylphenidate
	Dextroamphetamine
	Diethylpropion
	Methamphetamine
	Methylphenidate
	Phendimetrazine
	Phentermine
Barbiturates	Butabarbital
	Mephobarbital
	Pentobarbital
	Phenobarbital
	Secobarbital
Long-acting benzodiazepines*	Amitriptyline–chlordiazepoxide
	Chlordiazepoxide
	Chlordiazepoxide–clidinium
	Diazepam
	Flurazepam
Calcium-channel blockers	Nifedipine†
Gastrointestinal antispasmodics	Dicyclomine
	Propantheline
Belladonna alkaloids*	Atropine
	Atropine/chlorpheniramine/hyoscyamine/phenylephrine/scopolamine
	Atropine/hyoscyamine/phenobarbital/scopolamine
	Atropine–difenoxin
	Atropine–diphenoxylate
	Atropine–edrophonium
	Belladonna
	Belladonna/ergotamine/phenobarbital
	Butabarbital/hyoscyamine/phenazopyridine
	Digestive enzymes/hyoscyamine/phenyltoloxamine
	Hyoscyamine
	Hyoscyamine/methenam/m-blue/phenyl salicyl
Skeletal muscle relaxants*	ASA/caffeine/orphenadrine
	ASA/carisoprodol/codeine
	Aspirin–carisoprodol
	Aspirin–methocarbamol
	Carisoprodol
	Chlorzoxazone
	Cyclobenzaprine
	Metaxalone
	Methocarbamol
	Orphenadrine
Oral estrogens*	Conjugated estrogen
	Conjugated estrogen–medroxyprogesterone
	Esterified estrogen
	Esterified estrogen–methyltestosterone
	Estropipate
Oral hypoglycemics	Chlorpropamide
Narcotics (includes combination drugs)	ASA/caffeine/propoxyphene
	Acetaminophen–pentazocine
	Acetaminophen–propoxyphene
	Belladonna–opium
	Meperidine
	Meperidine–promethazine
	Naloxone–pentazocine
	Pentazocine
	Propoxyphene hydrochloride
	Propoxyphene napsylate
Vasodilators	Dipyridamole†
	Ergot mesyloid
	Isoxsuprine
Others‡	Methyltestosterone
	Nitrofurantoin and macrocrystals
	Nitrofurantoin macrocrystals–monohydrate
	Thyroid desiccated

APAP = N-acetyl-para-aminophenol; ASA = acetylsalicylic acid; HEDIS = Healthcare Effectiveness Data and Information Set; PSE = pseudoephedrine.

^*Includes combination drugs.

^†Short-acting only.

^‡Includes androgens and anabolic steroids, thyroid drugs, and urinary anti-infectives.

We focused on the HEDIS Drugs to Avoid in the Elderly indicator because of the straightforward numerator and denominator inclusion criteria: They apply to all enrollees aged 65 years or older with continuous plan enrollment and specify the use of pharmacy claims to identify high-risk drug use (13, 14). We compared plan-reported and calculated rates of high-risk medication use with the same measure criteria, data sources, period, and eligible population. We further determined whether plans correctly identified individuals who were eligible for the measure denominator and whether those plans correctly classified high-risk drug use in the measure numerator.

Methods

Sources of Data and Study Population

We obtained reported rates of high-risk prescribing in Medicare Advantage plans from HEDIS patient-level files, which contain information about all persons in the plans and indicate whether each individual was included in the numerator and denominator of the plan’s rate calculation. In 2006, a total of 276 plans reported 6 170 590 individual-level entries; in 2007, a total of 316 plans reported 11 070 841 individual-level entries.

We calculated rates of high-risk prescribing in Medicare Advantage plans separately using data from 3 sources. First, we used data from the Medicare Health Outcomes Survey, which includes a random sample of approximately 1000 persons in each plan (15). Between 2006 and 2008, the survey sampled 188 515 persons in 203 plans. Second, we used data from the Medicare Part D Event file, which contains information about every prescription filled in 2006 and 2007 for those with Part D coverage, including survey respondents and nonrespondents. Third, we obtained information about monthly HMO and Part D enrollment from the 2006 and 2007 Medicare enrollment files. These were the most recent data available for this analysis.

For the plan-level analysis, we compared reported with calculated rates of high-risk prescribing in 172 Medicare Advantage plans that had HEDIS measures in 2006 and 2007 and participated in the 2006 to 2007 Medicare Health Outcomes Survey (3 761 706 HEDIS entries and 177 227 Health Outcomes Survey entries in 2006, and 6 466 123 HEDIS entries and 173 655 Health Outcomes Survey entries in 2007). For the individual-level analysis, we examined a subsample of 103 680 persons in 2006 and 85 405 persons in 2007 (Figure 1).

Figure 1. Study flow diagram.

HEDIS = Healthcare Effectiveness Data and Information Set; HOS = Health Outcomes Survey.

Reported and Calculated Rates of High-Risk Prescribing

The primary study variable measured whether a person received 1 or more high-risk prescriptions, with “high-risk” defined by the CMS as a drug to avoid in elderly persons (12, 13). Individuals were included in the denominator of a rate if they were Medicare members aged 65 years or older as of 31 December of the measurement year and continuously enrolled during the measurement year in the Medicare Advantage plan and Part D with no more than 1 gap in enrollment of up to 45 days. Individuals were included in the numerator if they met criteria for being included in the denominator and received at least 1 prescription for any high-risk medication during the measurement year.

Context

Hospitals, managed care plans, and other providers are required to publicly report quality indicators and other performance measures. These efforts are not routinely audited, and few studies have examined their accuracy.

Contribution

This study found underreporting of the frequency of high-risk drug prescribing for elderly patients in Medicare managed care plans.

Caution

The researchers did not study other performance measures or other types of providers.

Implication

Additional studies are needed to determine the accuracy of publicly reported performance measures.

—The Editors

Individual plans applied these criteria to calculate the rates that they reported in HEDIS files. We applied these criteria separately to calculate similar rates by using Medicare data sources. We also calculated the difference between the reported and calculated rates. Positive values indicated that reported rates were higher than calculated rates, and negative values indicated that reported rates were lower than calculated rates.

Statistical Analysis

We conducted a cross-sectional, plan-level analysis of the agreement between plan-reported rates and our calculated rates during the first 2 years of mandatory reporting. All analyses were originally conducted separately for 2006 and 2007. Because findings were nearly identical for both years, we aggregated individual observations in each plan from 2006 and 2007 for the plan-level analyses.

We conducted analyses of variance and t tests to compare the mean differences in calculated and reported rates of high-risk prescribing by plan characteristics. We used an ordinary least-squares regression model with a normal distribution and identity link to estimate the adjusted effects of Medicare Advantage plan characteristics on the difference between reported and calculated rates of high-risk prescribing. Predictors included in the regression model were the profit status of the Medicare Advantage plan (for-profit or nonprofit), model type (staff or group model or nonstaff or nongroup model), duration of plan participation in Medicare (<5, 5 to 10, or >10 years), plan population size (quartiles), a measure of Medicare Advantage market penetration for the states in which the plans were based (<10%, 10% to 25%, or >25%), and U.S. census region (Northeast, Southeast, Midwest, West, or Puerto Rico).

To determine the extent to which plan ranking was affected by using calculated and reported performance rates, we ranked each plan from 1 (best performance) to 172 (worst performance) according to their reported rates of high-risk prescribing and again according to our calculated rates of high-risk prescribing. We then obtained the absolute difference in calculated and reported rank.

We also conducted an individual-level analysis, restricting the analytic sample to beneficiaries aged 65 years or older with 12 months of continuous enrollment in a single Medicare Advantage plan. We matched 103 680 individual beneficiaries sampled in the Health Outcomes Survey to their corresponding HEDIS entries in 2006 and 85 405 persons in 2007 to assess the agreement between reported and calculated high-risk drug measure numerator and denominator assignment.

All analyses were performed with SAS software, version 9.2 (SAS Institute, Cary, North Carolina), using the means, frequency, t test, analysis of variance, and GENMOD procedures. The study was approved by the Institutional Review Boards of Brown University, Providence, Rhode Island; Boston Medical Center, Boston, Massachusetts; and the CMS, Baltimore, Maryland (Data Use Agreement 20544).

Role of the Funding Source

This study was funded by the National Institute on Aging and the Health Assessment Lab. The funding sources played no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, and approval of the manuscript; or the decision to submit the manuscript for publication.

Results

We identified 172 Medicare Advantage plans that reported HEDIS measures in 2006 and 2007 and participated in the 2006 to 2007 Medicare Health Outcomes Survey (177 227 persons in 2006, and 173 655 persons in 2007). Table 1 presents the characteristics of these plans and of the 31 plans that participated in the Health Outcomes Survey but did not report HEDIS measures in 2006 and 2007. Most plans were for-profit and had a nonstaff or nongroup model type.

Table 1.

Characteristics of Medicare Advantage Plans Participating in the 2006 to 2007 Medicare Health Outcomes Survey

Plan Characteristic	Plans Reporting to HEDIS in 2006–2007 (n = 172), n (%)	Plans Not Reporting to HEDIS (n = 31), n (%)
Profit status
For-profit	119 (69.2)	17 (62.1)
Nonprofit	53 (30.8)	11 (37.9)
Model type
Nonstaff/nongroup	143 (86.1)	29 (96.7)
Staff/group	23 (13.9)	1 (3.3)
Time participating in Medicare
≤5 y	46 (26.7)	12 (38.7)
5–10 y	48 (27.9)	6 (19.4)
>10 y	78 (45.4)	13 (41.9)
Beneficiaries
<4845	33 (19.2)	20 (64.5)
4845–16 295	44 (25.6)	6 (19.4)
16 296–32 390	48 (27.9)	2 (6.4)
>32 390	47 (27.3)	3 (9.7)
State Medicare Advantage market penetration
<10%	28 (16.3)	7 (22.6)
10–25%	94 (54.6)	16 (51.6)
>25%	50 (29.1)	8 (25.8)
Census division
New England	7 (4.1)	3 (9.7)
Middle Atlantic	39 (22.7)	4 (12.9)
East North Central	27 (15.7)	4 (12.9)
West North Central	15 (8.7)	5 (16.1)
South Atlantic	24 (14.0)	4 (12.9)
East South Central	8 (4.6)	4 (12.9)
West South Central	10 (5.8)	0 (0.0)
Mountain	18 (10.5)	2 (6.4)
Pacific	20 (11.6)	5 (16.1)
Puerto Rico	4 (2.3)	0 (0.0)
Plans overreporting rates of high-risk prescribing
<1.00%	4 (2.3)	NA
≥1.00%	4 (2.3)	NA
Plans underreporting rates of high-risk prescribing
<1.00%	5 (2.9)	NA
1.00%–4.99%	57 (33.1)	NA
5.00–9.99%	85 (49.5)	NA
≥10.00%	17 (9.9)	NA

HEDIS = Healthcare Effectiveness Data and Information Set; NA = not applicable.

Nearly one half of the plans had participated in Medicare for more than 10 years. Plans were located in all 9 U.S. census divisions and Puerto Rico. Plans not reporting HEDIS measures had smaller plan populations (plans with fewer than 1000 beneficiaries are not required to report these measures).

The mean reported rate of high-risk prescribing among all 276 HEDIS-reporting plans in 2006 and 2007 was 22.7% (median, 22.4% [range, 3.5% to 51.3%]). Among the 172 plans in the final analytic sample, the mean plan-reported rate of high-risk prescribing was 21.1% (95% CI, 20.0% to 22.3%) and the mean rate of high-risk prescribing derived from the Part D claims of a random sample of enrollees was 26.9% (CI, 25.9% to 28.0%).

We plotted the calculated and reported rates of high-risk prescribing for each plan (Figure 2). Approximately 95% of plans reported rates of high-risk prescribing that were lower than the calculated rates derived from Part D claims in 2006 and 2007. On average, plans underreported by 5.8 percentage points (CI, −6.4 to −5.2 percentage points). In adjusted analyses, Medicare Advantage plans located in Puerto Rico had a smaller difference between the calculated and reported rate of high-risk prescribing than those located in other census regions (Appendix Table 2, available at www.annals.org). No other measured plan characteristic was significantly associated with differences between calculated and reported rates of high-risk prescribing.

Figure 2. Calculated and reported rates of high-risk prescribing in Medicare Advantage plans in 2006 and 2007.

Each point represents a single Medicare Advantage plan. CMS = Centers for Medicare & Medicaid Services. Points below the diagonal line represent plans that underreported rates of high-risk prescribing.

Appendix Table 2.

Associations Between Medicare Advantage Plan Characteristics and the Difference Between Observed and Reported Rates of High-Risk Medication Use in 2006 and 2007^*

Plan Characteristic	Persons, n	Mean Unadjusted Difference Between Reported and Calculated Rate (95% CI), percentage points	P Value for Difference	Relative Adjusted Difference Between Reported and Calculated Rate (95% CI), percentage points
Profit status			0.98
Nonprofit	53	−5.8 (−6.9 to −4.7)		Reference
For-profit	119	−5.8 (−6.6 to −5.0)		−0.2 (−1.5 to 1.1)
Time participating in Medicare			0.053
<5 y	46	−4.8 (−6.2 to −3.4)		Reference
5–10 y	48	−6.9 (−8.4 to −5.4)		−1.6 (−3.4 to 0.2)
>10 y	78	−5.7 (−6.3 to −5.0)		−1.2 (−2.9 to 0.6)
Beneficiaries			0.70
<4845	33	−6.3 (−7.9 to −4.8)		Reference
4845–16 295	44	−5.6 (−6.9 to −4.3)		0.9 (−1.1 to 2.8)
16 296–32 390	48	−6.0 (−7.4 to −4.7)		0.6 (−1.5 to 2.7)
>32 390	47	−5.3 (−6.4 to −4.3)		1.1 (−1.1 to 3.3)
Model type			0.88
Nonstaff/nongroup	143	−5.6 (−6.3 to −4.9)		Reference
Staff/group	23	−5.8 (−7.1 to −4.4)		0.2 (−1.6 to 2.0)
State Medicare Advantage market penetration			0.34
<10%	28	−5.9 (−5.8 to −4.9)		Reference
10–25%	94	−6.2 (−7.1 to −5.2)		−0.4 (−2.1 to 1.4)
>25%	50	−5.1 (−6.3 to −3.9)		−1.0 (−3.5 to 1.5)
Census region			0.031
Northeast	46	−6.1 (−7.2 to −5.1)		Reference
Midwest	42	−5.8 (−7.0 to −4.6)		0.3 (−1.5 to 2.1)
South	42	−6.4 (−8.0 to −4.8)		−0.5 (−2.4 to 1.3)
West	38	−5.3 (−6.3 to −4.3)		1.7 (−0.2 to 3.5)
Puerto Rico	4	0.5 (−12.5 to 13.6)		5.9 (1.6 to 10.2)

^*Results adjusted for data shown in the “Plan Characteristics” column. Values with a negative difference signify underreporting. Bold values signify significant data; α = 0.05.

We ranked plans according to their calculated and reported performance. The differences in the calculated and reported rates of prescribing high-risk medications negatively affected quality rankings for the plans that most accurately reported these rates, with the most accurate plans having the greatest penalty and the least accurate plans having the greatest gain (Figure 3). On average, the 9 plans that reported rates of high-risk prescribing within 1 percentage point of calculated rates were ranked 43.4 positions lower when reported rates were used instead of calculated rates. The 18 plans with reported rates that differed from the calculated rate by more than 10 percentage points were ranked an average of 49.1 positions higher when the reported rate was used.

Figure 3. Comparison of ranks from reported versus calculated rates of high-risk prescribing.

The 172 plans are arrayed from most accurate reporting (upper quadrants I and II show the smallest difference between calculated and reported rates) to least accurate reporting (lower quadrants III and IV show the largest difference between calculated and reported rates). Plans in left quadrants I and IV are ranked worse when using reported values; plans in right quadrants II and III are ranked better when using reported values.

In the individual-level analysis, we identified 103 680 persons aged 65 years or older with 12 months of continuous enrollment in a single Medicare Advantage plan and 12 months of Part D enrollment in 2006 who were included in the Health Outcomes Survey sample and HEDIS 2006 patient-level data. By definition, all of these persons should have been included in the Drugs to Avoid in the Elderly measure denominator given their age and enrollment duration. However, plans incorrectly excluded 10.3% of these enrollees from the denominator in the HEDIS reports (Appendix Table 3, available at www.annals.org).

Appendix Table 3.

HEDIS Drugs to Avoid in the Elderly Denominator Agreement for Persons Aged ≥65 Years With 12 Continuous Months of Enrollment in a Single Medicare Advantage Plan and 12 Months of Part D Benefit in 2006^*

Variable	Reported to HEDIS
	Excluded From Denominator, n (%)	Included in Denominator, n (%)	Total
Excluded from denominator	0 (0.0)	0 (0.0)
Included in denominator	10 714 (10.3)	92 966 (89.7)
Total			103 680

HEDIS = Healthcare Effectiveness Data and Information Set.

^*Data in the rows are calculated from the Health Outcomes Study.

Approximately 29% of those who were incorrectly excluded from the HEDIS denominator received at least 1 high-risk medication according to their Part D claims. Among the 92 966 persons who were correctly included in the measure denominator, the reported rate of high-risk prescribing was 21.9% and the calculated rate was 26.2% (Table 2). In this subgroup, reported and calculated numerator assignment differed for 8% of persons. With the calculated values used as a gold standard, the positive predictive value of the reported numerator assignment was 91.5% (CI, 91.1% to 91.9%). The reported numerator assignment allowed for inclusion of some false-positive results, offsetting the exclusion of individuals who should have been included in the measure numerator. Results were similar for the 85 405 persons identified in both data sources in 2007 (Appendix Tables 4 and 5, available at www.annals.org).

Table 2.

HEDIS Drugs to Avoid in the Elderly Numerator (≥1 High-Risk Medication) Agreement for Persons Aged ≥65 Years With 12 Continuous Months of Enrollment in a Single Medicare Advantage Plan and 12 Months of Part D Benefit in 2006 Who Were Included in the Measure Denominator by Both Sources^*

Variable	Reported to HEDIS
	No High-Risk Drug Use, n (%)	High-Risk Drug Use, n (%)	Total
No high-risk drug use	66 846 (71.9)	1735 (1.9)
High-risk drug use	5733 (6.2)	18 652 (20.0)
Total			92 966

HEDIS = Healthcare Effectiveness Data and Information Set.

^*The McNemar test of marginal homogeneity: P < 0.001. Data in the rows are calculated from the Health Outcomes Survey.

Appendix Table 4.

HEDIS Drugs to Avoid in the Elderly Denominator Agreement for Persons Aged ≥65 Years With 12 Continuous Months of Enrollment in a Single Medicare Advantage Plan and 12 Months of Part D Benefit in 2007^*

Variable	Reported to HEDIS
	Excluded From Denominator, n (%)	Included in Denominator, n (%)	Total
Excluded from denominator	0 (0.0)	0 (0.0)
Included in denominator	8687 (10.3)	75 818 (89.7)
Total			84 505

HEDIS = Healthcare Effectiveness Data and Information Set.

^*Data in the rows are calculated from the Health Outcomes Study.

Appendix Table 5.

HEDIS Drugs to Avoid in the Elderly Numerator (≥1 High-Risk Medication) Agreement for Persons Aged ≥65 Years With 12 Continuous Months of Enrollment in a Single Medicare Advantage Plan and 12 Months of Part D Benefit in 2007 That Were Included in the Measure Denominator by Both Sources^*

Variable	Reported to HEDIS
	No High-Risk Drug Use, n (%)	High-Risk Drug Use, n (%)	Total
No high-risk drug use	54 382 (71.7)	1429 (1.9)
High-risk drug use	4226 (5.6)	15 781 (20.8)
Total			75 818

HEDIS = Healthcare Effectiveness Data and Information Set.

^*The McNemar test of marginal homogeneity: P < 0.001. Data in the rows are calculated from the Health Outcomes Study.

Discussion

We compared the rates of high-risk prescribing reported by Medicare Advantage plans to the CMS with those calculated from Part D claims among a random sample of enrollees in the same 172 Medicare Advantage plans. Because Medicare Advantage plans also generate the HEDIS measure by using Part D pharmacy claims, the calculated and reported rates of high-risk drug use should be nearly identical within the bounds of sampling variability. In 2006 and 2007, approximately 95% of Medicare Advantage plans underreported their enrollees’ use of high-risk drugs, with reported rates nearly 6 percentage points lower on average than calculated rates. We identified incomplete reporting due to the exclusion of eligible persons from the measure denominator and inaccurate reporting due to incorrect numerator assignment.

Auditing plan-reported quality measures is not common practice, perhaps because rigorous audits require assessing complex numerator and denominator inclusion criteria or accessing comprehensive patient medical records. A recent study by Kern and colleagues (16) examined the accuracy of “meaningful use” clinical quality measures using electronic health records in a single health center (16). Although their analysis focused on the reporting of quality by providers rather than by health plans, they found similar inaccuracies in 8 quality measures.

The HEDIS Drugs to Avoid in the Elderly measure is uniquely well-suited for an auditing analysis because the measure specifications require relatively few data sources. Although we derived performance rates on the basis of a sample of beneficiaries from each plan, the random sampling method used by the Health Outcomes Survey should ensure that the sampled population does not differ substantially from the full plan population. Assuming no systematic bias in reporting, we would expect accurate reporting to result in similar calculated and reported rates, with the differences reflecting only sampling variability. Instead, the CI around the mean calculated rate of high-risk drug use excludes the mean reported rate, suggesting that the results of this study are not ascribable to chance.

An added strength of this study is the individual-level analysis, which allowed for a direct comparison of numerator and denominator inclusion agreement for the same beneficiaries across both data sources. Our findings suggest that underreporting is predominantly driven by erroneous assignment of the high-risk prescribing measure numerator. HEDIS provides an annual list of Drugs to Avoid in the Elderly (featuring generic names, brand names, and national drug codes) but does not include the statistical code or measurement algorithm to be applied to Part D claims. Plans must therefore generate their own methods of measurement that are compatible with their data systems.

Discrepancies in numerator assignment suggest that many plans cannot identify all persons who receive 1 or more of the 97 high-risk drugs on the HEDIS list. Further, plans may not benefit from devoting additional analytic resources to this task. The most accurate plans had the greatest penalties when ranked according to plan-reported rates rather than the rates we calculated from their enrollees’ Part D claims.

In addition, we noted substantial disagreement with HEDIS denominator classification, with plans incorrectly excluding enrollees who met the measure’s eligibility criteria. This finding underscores the need for audits of reported performance rates to evaluate numerator and denominator classification. Denominator assignment may be more difficult when enrollees change plans during the course of a year. Therefore, we focused our individual-level analyses on persons who were continuously enrolled in a single plan and in Part D for a full calendar year and were aged 65 years or older. Our study shows that plans may have difficulty identifying the eligible sample even for quality measures with straightforward denominator inclusion criteria. The potential for denominator misclassification may be greater with more complex quality measures.

The CMS required Medicare Advantage plans to report on high-risk prescribing beginning in 2006, the first year of Medicare Part D coverage. Plans may have had limited experience generating accurate data for a new measure derived from pharmacy claims. However, rates of underreporting in 2006 and 2007 were nearly identical, suggesting little improvement in reporting accuracy over these first 2 years. Further audits using more recent data should be done. We could not identify the mechanism of exclusion for persons who should have been included in the measure denominator or the reasons for numerator discordance among those correctly identified as eligible for the Drugs to Avoid in the Elderly measure in HEDIS and the Health Outcomes Survey. Finally, our study examined only 1 quality measure.

Our study suggests that approximately 5.8% additional elderly Medicare Advantage enrollees (or an additional half-million beneficiaries) have been exposed to high-risk medications when calculated rather than reported rates were used. Further, we found that the rate of use of high-risk medications among Medicare Advantage enrollees is similar to that reported among Medicare fee-for-service beneficiaries (25.8%) in 2007 (17). However, policymakers evaluating plan-reported rates would erroneously conclude that the Medicare Advantage program has relative rates of high-risk prescribing that are approximately 20% lower than those in the fee-for-service program. In addition, the Patient Protection and Affordable Care Act provides payment bonuses and regulates expansions in plan enrollment according to a Medicare Advantage plan’s quality rankings of 1 to 5 stars (a composite score of approximately 50 quality measures, including HEDIS high-risk drug use). These provisions increase the financial incentives for plans to report better clinical performance (2).

The CMS audit of HEDIS in 1998 found widespread reporting inaccuracies (9). Since then, few studies have been published to suggest substantial improvement. In our study of CMS quality data, we found widespread and substantial underreporting of rates of HEDIS high-risk prescribing among Medicare Advantage plans. Policymakers should consider routine audits of publicly reported quality measures, including the HEDIS indicator of high-risk prescribing, to ensure their validity and reliability for patients and other stakeholders.