Is there an advantage? Considerations for researchers studying the effects of the type of Medicare coverage

Abstract

Objective

To describe common methodological problems that arise in comparisons of Medicare Advantage (MA) and Traditional Medicare (TM) and within‐MA studies and provide suggestions of how researchers can address these issues.

Study Setting

Published research evaluating Medicare coverage options in the United States.

Study Design

We considered key conceptual challenges and promising solutions that have been used thus far and suggest additional directions.

Data Collection

Not available.

Principal Findings

Many existing studies of MA versus TM include significant limitations, such as failing to account for unobserved confounders driving both beneficiary coverage choice and health outcomes once enrolled, not accounting for variation in benefit generosity, provider networks, or plan design across MA plans, and/or having been conducted at a time when MA enrollment was less than a third of all Medicare beneficiaries. We provide a review of methods that can help researchers to overcome these weaknesses and suggest additional methods and data sources that may aid future research.

Conclusions

The MA program is becoming an essential part of the US healthcare system. By accounting for non‐random movement into and out of MA and studying the heterogeneity of beneficiary experience across plan and market characteristics, researchers can provide the high‐quality evidence necessary for policymakers to design the program and reform TM in ways that maximize beneficiary outcomes.

What is known on this topic

• When comparing outcomes across Medicare Advantage (MA) and Traditional Medicare (TM), researchers should be aware of several key challenges: non‐random program participation, commonly known as selection effects; heterogeneity across plans within the MA program; varying coverage choices within TM; spillovers between the programs; and differential coding incentives.

• However, many existing papers do not account for these challenges, limiting generalizability of findings.

What this study adds

• We describe the common conceptual challenges that researchers must confront when comparing Medicare coverage options and summarize methodological strategies that may help with these issues.

• Designing studies that account for the issues raised in this paper will help researchers conduct rigorous evaluations of the MA program, providing important information to policymakers and Medicare beneficiaries about outcomes in MA versus TM.

1. INTRODUCTION

The Medicare Advantage (MA) program is rapidly growing, doubling in enrollment from 13 million beneficiaries in 2012 to 31 million beneficiaries in 2023, ¹ with over half of the Medicare population now enrolled. In MA, private plans are paid on a capitated basis to cover the needs of their enrollees. MA plans can offer additional benefits ² that are not available in Traditional Medicare (TM), but MA plans can also use restricted network designs, prior authorization, and other managed care tools to constrain spending. ³ The explosive growth in the MA program raises simple but important questions about how Medicare beneficiaries fare in MA versus TM, which program beneficiaries should choose, and whether the US government should continue to direct substantial financial resources towards MA.

A growing literature has sought to understand what the effects of the MA program might be on the outcomes and experiences of Medicare beneficiaries. Prior studies for example have found that the MA program may be associated with lower hospital admissions, ⁴ lower post‐acute care use, ⁵ but also increases in diagnosis coding intensity. ⁶ While MA is often included as a single indicator variable, researchers have found substantial, causal impacts of individual MA plans on mortality, ⁷ highlighting the need for more understanding of heterogeneity within the MA program. Investigators have used a range of descriptive and quasi‐experimental study designs. Many of these methods include significant limitations, such as failing to account for unobserved confounders driving both patient decisions to enroll in MA or TM program and health outcomes once enrolled, not accounting for variation in benefit generosity, provider networks, or plan design across MA plans, and/or having been conducted at a time when MA enrollment was less than a third of all Medicare beneficiaries. As a result, there is a major need for research to determine whether the average Medicare beneficiary would have better health or financial outcomes if moving between MA and TM or across plans within their current option if there are specific types of beneficiaries that would fare better or worse with each type of coverage, and on understanding the impacts of MA plans on beneficiary outcomes in general.

This paper describes conceptual and methodological challenges common to MA studies and the growing need for within‐MA research. We then discuss methods that can help to address these concerns, provide an overview of their strengths and limitations, and offer recommendations to strengthen MA research moving forward.

2. METHODS

2.1. Conceptual and methodological issues in studying MA

When comparing outcomes across MA and TM, researchers need to account for five key problems in order to make apples to apples comparisons: non‐random program participation, commonly known as selection effects; heterogeneity across plans within the MA program; varying coverage choices within TM; spillovers between the programs; and differential coding incentives.

For descriptive analyses, researchers may be interested in the average characteristics of MA versus TM enrollees, average utilization conditional on Medicare coverage type, and similar metrics. In these cases, the challenges raised above may not be relevant. This paper is focused on situations where researchers explicitly want to make causal statements about questions like “How does MA enrollment impact health outcomes?” or “Does increasing payment to MA plans benefit Medicare beneficiaries.” The primary issues in conducting causal research are as follows:

2.2. Non‐random program participation

Our goal is often to measure the unbiased average treatment effect of enrollment in one Medicare program compared to the other. Historically, MA plans have benefited from a phenomenon known as positive selection, where the Medicare beneficiaries opting into MA plans were estimated to use significantly less care than the average beneficiary remaining in TM after accounting for differences in observable demographic characteristics such as age, sex, and race. ⁸ , ⁹ , ¹⁰ , ¹¹ , ¹² , ¹³ This is a concern when the same unmeasured factors, for example, beneficiary socio‐economic status, health literacy, or personal beliefs directly impact both the choice of Medicare coverage type as well as health and other outcomes of interest.

In most situations, there are a set of measured confounders like age and sex that we are able to account for using standard regression methods. What we are concerned about however is the set of unmeasured confounders (selection bias), such as beneficiary preferences that are difficult to measure, and confounders such as income, functional status, cognition, and others that could be measurable but are often missing from large datasets. Medicare beneficiaries choosing MA may prefer to accept a limited provider network in exchange for lower cost‐sharing, or may know that they are healthy and will not need to use much medical care that would be impacted by managed care regulations.

Plans may also actively try to impact selection by offering different supplemental benefits that may attract specific profiles of beneficiaries. Supplemental benefits offered by certain MA plans can influence selection. ⁸ , ¹⁴ For example, MA plans offering free gym memberships attracted new members who were 20% more likely to report very good or excellent health. ⁸ , ¹⁴ More recent policy changes allowing MA plans to offer services such as caregiver supports and concurrent palliative care may attract sicker patients to MA. ¹⁵

If a researcher does not adequately account for the potential selection effects that could bias the comparison between MA and TM, a reader will not be able to determine whether lower risk‐adjusted rates of hospitalization in MA result from MA reducing hospitalizations, for example, or enrolling people who would experience fewer hospitalizations regardless of MA versus TM choice. While adjustment methods that use observable variables may help to address these challenges if researchers have a wide set of data, most claims‐based datasets that health services researchers use are unlikely to have sufficient detail to adequately adjust for this selection on the observables alone.

2.3. Heterogeneity across plans

A second potential concern in the analysis of the MA program is that considering MA to be a single exposure may not always be a valid approach for many research questions. Unlike in TM where the program is largely similar across the country, MA is delivered by over 500 different companies that each offers up to dozens of individual plans. The average beneficiaries in 2022 had access to 39 different MA plans they could choose from, each potentially offering different benefits, provider networks, cost sharing, and wrap around services. ¹⁶ While binary comparisons of MA to TM performance may be useful for understanding overall trends in the Medicare program, it is unlikely that differently structured MA plans will have identical impacts on beneficiaries throughout the program. When comparing outcomes between beneficiaries enrolled in TM as compared to beneficiaries enrolled in any MA plan, important differences within the MA program may be ignored, and underlying characteristics and practices of MA plans that drive their performance may be masked, limiting the generalizability and policy relevance of study findings.

Plans can set variable premiums, cost‐sharing, out‐of‐pocket maximums, provider networks, and covered supplemental benefits. ² Plans also vary in whether they offer Part D drug coverage. Plans of different premium levels may be more or less affordable for some beneficiaries, affecting plan choice. Second, plans can vary in their overall structure and type. MA plans can be health maintenance organization or preferred provider organization plans, some plans are special needs plans for dual beneficiaries or those with chronic conditions, and plans can set widely variable network sizes. Third, plans can vary in terms of the demographics of who chooses to enroll in those plans, with some plans disproportionately enrolling minority or low‐income beneficiaries. ¹⁷ Fourth, plans can vary in terms of their overall quality, with some delivering better outcomes for beneficiaries than others. ⁷

Plan decisions about benefits, market participation, quality, and breadth of networks included are generally not random decisions, introducing another potential source of bias. ¹⁸ These decisions are often made at the individual market level, so the same national firm may offer plans and provider networks in competitive markets that differ from offerings in markets the firm dominates. In many cases, including studies using administrative data, researchers may have incomplete information on whether TM beneficiaries also have prescription drug coverage and supplemental benefits from a Medigap or employer plan, compounding the difficulties in making direct comparisons.

With the growth of the MA program, the more pressing research questions moving forward may be about plan‐to‐plan variation within MA rather than MA‐to‐TM differences. When conducting such studies, researchers need to consider whether they are characterizing the average experience of an MA enrollee, or a specialized experience unique to a particular national firm or service area.

2.4. Varying coverage choices in TM

Another important source of heterogeneity in the Medicare program is increasing differences in coverage type in the TM program. While the TM program is still largely paid Fee‐for‐Service and is administered by CMS, the proliferation of alternative payment models (APMs) such as accountable care organizations (ACOs) ACOs and bundled payment programs make it so that the TM program is also not one single exposure. When comparing outcomes between MA and TM, a researcher may find different results when comparing TM beneficiaries attributed to one of these programs, compared to TM beneficiaries not attributed to an APM. ¹⁹ For example, several prior studies have found differences in outcomes when comparing MA to TM without ACO attribution to TM with ACO attribution. ²⁰ Beyond attribution to APMs in TM, TM beneficiaries do actually have some differences in coverage choices that may also be subject to selection bias. TM beneficiaries have the option to enroll in a Part D plan to receive drug coverage. TM beneficiaries may also purchase Medicare Supplemental Insurance, or Medigap, which covers their cost sharing. While using TM data sources such as the enrollment file can provide information on whether a beneficiary selects a Part D plan, most Medicare data sources with the exception of several smaller surveys, do not include information on who enrolls in supplemental insurance, and prior work has found there to be important differences in who chooses to enroll in such plans. ²¹ As a result, if a researcher wants to compare the two programs, it may be unclear if what they are detecting is an MA versus TM effect or a difference related to the levels of cost sharing that could exist within each program. While without additional data this is difficult to solve, researchers must take care when interpreting comparisons between MA and TM as differences in TM benefits may play an important role in driving outcomes.

2.5. Spillovers between MA and TM

A fourth issue that may pose a threat to comparisons between the MA and TM program is the presence of spillover effects between the programs. A robust literature on spillover effects from MA to TM has shown that higher MA penetration in a county can contribute to lower average spending in TM, possibly because managed care promotes more efficient practice styles for all patients. ⁴ , ²² , ²³ The presence of these spillovers implies that in areas with high MA penetration, beneficiary experiences in TM may be more similar to MA because of common practice styles impacting all beneficiaries. At the same time, the characteristics of beneficiaries choosing MA in a market with relatively high penetration relative to those remaining in TM also likely differ across high and low penetration counties. ²⁴ , ²⁵ On the other hand, prior research has also documented spillover effects from TM bundled payment models to MA. ²⁶ This direction of spillover can be significant because TM alternative payment experiments are usually at a substantial scale. The presence of these spillover effects complicates assessments of the performance of MA relative to TM and how researchers and policymakers should value improvements in performance in TM due to higher levels of MA market penetration, while also accounting for contemporaneous payment innovations in TM. The presence of spillovers may also present a methodological challenge in measuring a treatment effect of interest. If the MA and TM programs have a dynamic effect on each other, then the causal effect of interest may not be well identified due to interactions between the two programs.

2.6. Differential coding incentives

A fifth issue that poses a major threat to comparisons between the MA and TM program is a differential incentive to code diagnoses codes between the two programs. In MA, plans are paid a risk‐adjusted rate in an effort to address concerns over adverse selection. Beneficiary diagnoses are classified into Hierarchical Condition Categories (HCC), each of which corresponds with a different weight. A plan's overall risk is then calculated, and plans that have higher risk scores are reimbursed at higher rates. This risk adjustment creates a strong incentive for MA plans to increase the intensity of coding of enrolled beneficiaries. ⁶ As most care received in the TM program is not risk‐adjusted, the same incentive does not exist, which could lead to much higher measured risk in the MA program compared to TM. This is an issue for causal inference since analysts using diagnostic codes to adjust for comorbidities or define cohorts risk including MA beneficiaries who are healthier than their revenue‐maximized claims data may imply.

3. RESULTS

3.1. Strategies to improve MA versus TM comparisons

3.1.1. Accounting for non‐random selection into MA

Researchers have used several quasi‐experimental research designs to improve their ability to compare MA versus TM. Table 1 highlights strategies that have been used previously.

TABLE 1.

Strategies currently used to account for methodological challenges in Medicare Advantage research.

Strategy	How it works (with examples)	Strengths	Limitations
Geographic dimensions of Medicare payment	Uses parts of MA payment rates determined by public policy as an instrumental variable or regression discontinuity leveraging higher payments attract enrollment to predict MA enrollment or penetration. Example papers: 22, 27, 28	Plan payment policy provides a credible source of variation in some cases	At least part of higher payments are passed on to beneficiaries through supplemental benefits, which could directly influence health outcomes through healthcare utilization in addition to MA versus TM choice. Other local area characteristics such as availability of providers may be associated with payment benchmarks and health outcomes. When using an instrumental variable, the local average treatment effect of plan level payment benchmarks may be difficult to interpret.
Pre‐Medicare choices and characteristics	Merge Medicare claims with commercial insurance or survey data from before a beneficiary ages into Medicare and follow beneficiaries into the Medicare program accounting for those pre‐enrollment differences. Example paper: 33	Straightforward interpretation of results, allows us to difference out persistent unobservable characteristics of Medicare beneficiaries	Having ample data pre‐Medicare enrollment to account for selection may be challenging and/or expensive to implement. Differences between groups will likely grow over time, strategy is best for comparing younger beneficiaries.
Changes to choice set: Plan entry/exit	Study beneficiaries who are moved between MA plans or disenrolled to TM because of changes in plan availability in their county. Example papers: 7, 31	Beneficiaries are exposed to new plans without necessarily electing to, which may control for selection well	Beneficiaries are never locked into automatic reassignments, so can only be conducted as an Intent to Treat analysis. When plans enter, exit, or merge, it may be a result of strategic decisions by the plan. To the extent that these strategic decisions could be correlated with patient outcomes, it may introduce its own selection bias.
Matching methods	Use propensity scores, coarsened exact matching, or similar strategy to compare MA and TM enrollees who are similar in observable characteristics. Example papers: 32, 35, 36	Straightforward to implement, and fairly straightforward to interpret	Generally very difficult to account for all potential sources of selection, especially difficult to argue that all unobservables would be correlated with small set of control variables available in claims data studies
Accounting for Medicare Advantage upcoding	Use alternative data sources about beneficiary health (survey data, hospital discharge data). Example papers: 6, 51, 52	Diagnoses coded during a hospitalization are removed from aggressive coding practices plan may employ	Limit generalizability (i.e., only hospitalized patients, surveys have small samples and may preclude specific diseases).

Note: The highlighted papers provide examples of research strategies that can address some of the inherent difficulties with understanding the Medicare Advantage program including comparisons with Traditional Medicare.

Abbreviations: MA, Medicare Advantage; TM, Traditional Medicare.

Geography

Since MA plans have always been paid, at least in part, based on rates set at the county level, geography has provided a useful natural experiment. State‐of‐the‐art methods include isolating parts of plan payment rates that are derived from factors set by public policy, ²² , ²⁷ comparing beneficiaries in zip codes on either side of a county border, or comparing beneficiaries in counties just above or just below 250,000 residents, as the latter were historically eligible for higher reimbursement rates. ²⁸

A challenge with using payment as a source of quasi‐random variation in MA enrollment often referred to as an instrumental variable in the economics literature, is that researchers consistently show that at least some portion of higher plan payments is returned to beneficiaries through supplemental benefits. ²⁹ , ³⁰ , ³¹ Depending on the outcome of interest, it may be difficult to satisfy the key instrumental variables assumption that payment rates impact something like healthcare utilization strictly through a beneficiary's choice to enroll in MA versus TM. This assumption may also be violated if there are other local area characteristics such as availability of providers that are both associated with the payment benchmarks and with the given outcomes of a study.

Pre‐65 choices and characteristics

Medicare beneficiaries use private information about their health and health behaviors to inform their coverage choice, which can bias our estimates of MA effectiveness if we do not account for it. Novel data sets including information about healthcare utilization prior to aging into Medicare, can help incorporate these typically unobserved factors. For example, Schwarz et al. follow a subset of beneficiaries with employer‐sponsored healthcare through a large commercial plan. ³² Similar designs could follow beneficiaries from survey cohorts as they age into Medicare. ³³ This provides a powerful research design because we can account for persistent, potentially unobservable differences between those who choose MA and TM using difference‐in‐difference regression models. Limitations of this approach include the challenges of assembling such datasets, which may be financially costly, difficult to negotiate, and still yield small or non‐representative samples once we restrict to respondents of a particular survey or those who stayed in the same employer‐sponsored plan for a minimum number of years before Medicare eligibility. Another limitation to this approach is that effect estimates are largely limited to younger beneficiaries once they turn 65, as there is more opportunity for selection bias as the time from Medicare entry increases.

Changes to beneficiaries' choice set

Changes in the plans available in a county can also trigger useful beneficiary movement. For example, if a beneficiary's plan leaves the county, they may be assigned another plan that they did not choose. ⁷ Plan exit and entry can create quasi‐experimental variation in market competition, MA penetration, and benefit availability. ³⁴ Plan exits in particular may provide a strong natural experiment to move a beneficiary out of a plan that best matched their preferences. However, we may worry that plan exits are not random, with firms choosing to leave markets where beneficiaries require more care than expected or there are not enough providers for a robust network. Researchers should take care to determine whether there are reasons for plan entry and exit that could also directly impact beneficiary health outcomes and bias comparisons across plans or coverage options. Further, authors will need to be sensitive to whether factors such as plan exits are important for moving beneficiaries out of MA entirely or shuffling beneficiaries across plans within MA.

Matching

Matching models such as propensity score weighting and coarsened exact matching rely on strong assumptions that TM and MA enrollees who are similar in observable characteristics are also similar in unobservable characteristics. ³⁵ , ³⁶ Matching or reweighting so that we only compare the portions of the MA and TM distributions with overlapping support, or similar characteristics, is recommended only if it is used in conjunction with another strategy to address non‐random selection driven by unobserved characteristics. For example, Schwarz et al. combine reweighting with a difference‐in‐difference model accounting for pre‐65 utilization, generating doubly‐robust estimates. ³²

3.1.2. Strategies for future research

Next, we describe several potential natural experiments that have not yet been widely used in the MA versus TM literature that researchers may find applicable to their research.

Choice modeling

An innovative approach is to directly model both the choice between MA and TM and the choice of plans within MA, and to estimate these choice models jointly with equations for care utilization and health outcomes. ³⁴ , ³⁵ , ³⁶ , ³⁷ , ³⁸ , ³⁹ , ⁴⁰ The validity of such “structural econometric” studies hinges on the assumptions made about the choice process and the regression error terms. For example, modeling insurance choices in a multinomial logit framework may be inappropriate because of the well‐known “independence of irrelevant alternatives” assumption (IIA); instead, employing a nested logit model, which relaxes the IIA assumption within MA, may be more realistic. Furthermore, incorporating attitudinal data (e.g., information on how consumers value quality in MA) if available, may better identify consumer preferences. ³⁸

Structural models are also well‐suited to handle the inherent dynamics of plan choice. Significant evidence suggests that MA enrollment is sticky because of the cognitive and time costs associated with coverage changes. ³⁷ , ³⁸ Given inertia with initial choices, the menu of plan options within MA when beneficiaries make their initial election at age 65 are potentially more important than year‐to‐year changes in options. ³⁹ To the extent that quality ratings influence plan choice ⁴⁰ and to the extent that plan choice is persistent, variation in quality star ratings or other plan availability characteristics at age 65 may significantly predict MA enrollment at older ages. The idea would be to instrument for MA plan enrollment with age 65 quality star ratings (or other plan characteristics) in the age 65 county of residence. To our knowledge, such an instrument has not been used in practice, but there is a growing literature on the dynamics of enrollment behavior that is grounded in economic theory.

Market characteristics

Mergers and acquisitions among regional or national insurers cause plausibly exogenous shocks to local insurance market offerings and thus enrollment. Dafny et al. first used the instrument to examine insurance market concentration and premium growth for employer plans, and Lin and McCarthy applied it to assessing the effects of MA insurers' overlap across market segments. ⁴¹ , ⁴² However, mergers are relatively scarce, and the strategies of insurers involved may not be representative of those of all insurers, raising questions about the external validity of such models.

Retiree health benefits provided by employers may sway Medicare beneficiaries into choosing MA or TM. Nearly 3 in 10 of the large employers that offer health coverage also provide retiree health benefits. ⁴³ Larger employers, public employers, and employers with union workers are more likely to do so. Thus, employer characteristics may serve as instruments for local MA enrollment. Nearly half of the large employers offering retiree benefit do so through contracting with MA plans, ⁴³ which encourages MA enrollment. Yet the other half offer supplemental coverage to TM, which increases the likelihood of TM enrollment. With MA program's recent growth and with more employers contracting with MA plans to take advantage of government subsidy, ⁴⁴ retiree health benefit offering may become more predictive of MA enrollment.

Variation in state policies to improve coordination between Medicare and Medicaid benefits for dual‐eligible beneficiaries can be useful for comparing outcomes for these beneficiaries in MA—especially dual eligible special needs plans (D‐SNPs)—and TM. Some states have implemented policies that promote enrollment into D‐SNPs including requiring insurers offering Medicaid plans to also offer D‐SNPs, and automatic assignment of Medicaid beneficiaries to D‐SNPs of the same insurer at Medicare eligibility. ⁴⁵ The limitation of using state policy variation for dual eligibles is that this high‐spending, high‐need group lacks generalizability to the overall Medicare population.

State Medigap (supplemental coverage for TM beneficiaries) policies like guaranteed issue and community rating policies offer another source of variation. In the vast majority of states, Medigap policies are not subject to guaranteed issue or community rating beyond a beneficiary's first 12 months of Medicare eligibility. MA beneficiaries can thus be denied coverage or face substantial premiums if they switch to TM and need supplemental coverage. However, eight states require community rating of Medigap policies for all beneficiaries, half also require guaranteed issue. ⁴⁶ There has been research showing that such consumer protection policies reduce the barrier for beneficiaries to leaving MA. ²

3.1.3. Accounting for heterogeneity across MA and TM coverage choices

Beyond selection bias, the potential heterogeneity in the MA program is also of significant concern. Unfortunately, there has been limited research to date on how to account for the differences across plans, or even to determine the ways that different plan and market characteristics interact with the selection problem. The first step in accounting for heterogeneity is to highlight the different dimensions over which plans can vary. In Table 2, we present important sources of heterogeneity across MA plans and market and point readers to data sources that can allow for stratifications and sub‐analyses.

TABLE 2.

Important dimensions of heterogeneity in Medicare Advantage.

Dimension	Data source
Level of MA payments relative to TM spending	Medicare Advantage ratebook https://www.cms.gov/Medicare/Health‐Plans/MedicareAdvtgSpecRateStats/Ratebooks‐and‐Supporting‐Data
County‐Level MA penetration	CMS Public Use MA Files https://www.cms.gov/Research‐Statistics‐Data‐and‐Systems/Statistics‐Trends‐and‐Reports/MCRAdvPartDEnrolData/MA‐State‐County‐Penetration
Generosity of MA choice set	Number of plans: Medicare Plan Finder Estimated OOP Spending: Medicare Plan Finder https://www.medicareplanfinder.com/ Breadth of Network Coverage
Regional practice patterns	Dartmouth Atlas (Traditional Medicare) https://www.dartmouthatlas.org/data/ MedPAR (all hospitalizations) https://www.cms.gov/Research‐Statistics‐Data‐and‐Systems/Statistics‐Trends‐and‐Reports/MedicareFeeforSvcPartsAB/MEDPAR
Socioeconomic status of beneficiaries	Medicaid or low‐income Part D subsidy eligible: Master Beneficiary Summary File https://www.cms.gov/Research‐Statistics‐Data‐and‐Systems/Files‐for‐Order/LimitedDataSets/MBSF‐LDS
Race/ethnicity of beneficiaries	Medicare Beneficiary Summary File https://www.cms.gov/Research‐Statistics‐Data‐and‐Systems/Files‐for‐Order/LimitedDataSets/MBSF‐LDS
Plan network breadth	Ideon Plan Network Data, Part D Data https://ideonapi.com/researchers/
Generosity of MA plan benefits	CMS MA Plan Benefit Files https://www.cms.gov/Research‐Statistics‐Data‐and‐Systems/Statistics‐Trends‐and‐Reports/MCRAdvPartDEnrolData/Benefits‐Data
Integration with Medicaid programs (SNPs)	CMS Public Use MA Files https://www.cms.gov/Research‐Statistics‐Data‐and‐Systems/Statistics‐Trends‐and‐Reports/MCRAdvPartDEnrolData/MA‐HEDIS‐Public‐Use‐Files
Integration with Health Systems	AHRQ Health System Compendium https://www.ahrq.gov/chsp/data‐resources/compendium.html

Note: Researchers should consider the sources of heterogeneity across Medicare Advantage plans that may be important for beneficiary experiences. The data sources listed above can be used to facilitate these comparisons.

Abbreviations: MA, Medicare Advantage; SNP, special needs plans; TM, Traditional Medicare.

We urgently need more research to determine, for example, whether beneficiaries fare better in markets where the payments to plans are relatively high relative to TM spending to determine the return on public investment in MA. ⁴⁷ However, we know little about how the MA versus TM comparison varies in markets that have high versus low MA penetration. Similarly, we need to understand how robust versus narrow networks, levels of plan payment, and other factors influence these comparisons.

Researchers should consider whether the comparatively rich information about Medigap or other supplemental coverage only found in survey data should be the main data source for their analysis, or whether claims data, particularly with additional market characteristics appended, conveys greater advantage. For many research questions, it will be useful to see models that either separately compare TM beneficiaries in APMs versus “standard” Fee‐for‐Service, or account for APM penetration in the market of interest.

3.1.4. Accounting for spillover effects

While many studies have been focused on identifying spillover effects between MA and TM, researchers may also want to account for these factors when comparing MA and TM. This could be done, for example, by comparing higher versus lower penetration markets or markets with differential rates of MA penetration growth, or looking at market‐level effects combining MA and TM outcomes. ²⁰ , ⁴⁸

3.1.5. Addressing issues around coding intensity

Researchers should use care when conducting analysis that adjusts for comorbidities when the data used can be differentially coded through MA plan influence. While this concern does not necessarily need a separate identification strategy to address, sensitivity analysis can be important for bounding potential concerns. Researchers can consider excluding diagnosis codes that come from chart review records or health risk assessments, which are tools that plans often use to increase coding. Researchers can also consider deflating comorbidity scores, such as CMS does with its standard adjustment factor. However, this approach merits caution as there may be differential coding intensity between MA contracts and a one sized fits all approach may not be appropriate. When possible, investigators should consider using information about comorbidities that are less influenceable by plans such as from surveys of health assessments like the Minimum Data Set for nursing home care.

4. DISCUSSION

In this report, we have sought to outline key challenges inherent to studying the MA program, and to highlight empirical approaches that will help to account for non‐random coverage choices and clarify important sources of heterogeneity within MA. As researchers continue to study the differences between these programs, we have several recommendations that may be helpful to consider.

One option is to look for questions where unobservable patient characteristics are unlikely to be related to the outcomes of interest for the study, for example, uniform treatment guidelines for patients in the emergency department or specific disease cohorts. ⁴⁹ Researchers should start any study of the MA program by clearly thinking about what the potential sources of selection are and how to address them. While restricting to subpopulations of more similar beneficiaries may sacrifice external validity, this may help to remove some patient differences that could bias the effect estimates. This method likely requires a high level of institutional detail about the clinical setting and possibly very detailed data to avoid other challenges discussed in this paper such as upcoding in MA. This method is more appropriate, for example, for comparing treatment immediately after diagnosis among patients with the same tumor size, type, and stage, than care among all patients with a prior cancer diagnosis.

If selection bias cannot be eliminated, researchers should determine whether natural experiments, such as those described above, can adequately address the selection bias in MA versus TM comparisons. When selection cannot be fully accounted for, researchers should try to bound the extent of potential bias. All of the currently used strategies to address this currently have limitations. Thus, efforts should be taken to measure what potential bias could affect the results. One promising strategy for doing so is the use of E‐Values which estimate how large a bias would need to be to reduce the study results to null. Such methods can help to provide more confidence in analyses of the MA program. ⁵⁰ There is a great need for careful, descriptive research that improves our understanding of the variation in beneficiary experience within MA, even without a comparison to TM. We encourage researchers to accompany all studies examining average differences between MA and TM with a robust set of analyses classifying the heterogeneity of experiences across plan types, geographic regions, and beneficiary characteristics.

Rather than making a single MA versus TM comparison, researchers may consider presenting several comparison groups to better characterize potential differences. Depending on the research question, a $0 premium MA prescription drug plan might be compared to TM with Part D with a Medigap supplement to compare the experiences of beneficiaries with similar levels of coverage generosity, or TM with no additional coverage to focus on beneficiaries choosing similar premium expenditures.

Multiple comparison groups and dimensions of heterogeneity can raise concerns about statistical significance following multiple comparisons. Theoretical models and carefully specified research questions can help researchers to isolate key focal areas when determining which beneficiaries, coverage options, providers, and markets to study. Use caution when including measures of patient comorbidities in analysis if the data generation process could include opportunities for differential coding between the MA and TM programs and between plans in the MA program. Consider sensitivity analysis with less biased measures of patient risk, or excluding tools such as chart reviews and health risk assessments.

As the MA program continues to grow, it will be essential for researchers to use rigorous methods to compare the programs. We have outlined several strategies that may be useful when making comparisons in the MA program. Many important questions will require researchers to grapple with a number of the challenges discussed in this paper. None of the approaches currently reviewed will be able to solve all problems, suggesting a need to triage the biggest concerns for each research question. The MA program is becoming an essential part of the US healthcare system, and the more high‐quality research that is conducted on MA, the easier it will be for policymakers to design the program in such a way that maximizes beneficiary outcomes and for beneficiaries to choose the program that is best for their personal situation.

CONFLICT OF INTEREST STATEMENT

The authors have no conflicts of interest.

Nicholas LH, Polsky D, Darden M, Xu J, Anderson K, Meyers DJ. Is there an advantage? Considerations for researchers studying the effects of the type of Medicare coverage. Health Serv Res. 2024;59(1):e14264. doi: 10.1111/1475-6773.14264