Poor Consumer Comprehension and Plan Selection Inconsistencies Under the 2016 HealthCare.gov Choice Architecture

Abstract

Background: Many health policy experts have endorsed insurance competition as a way to reduce the cost and improve the quality of medical care. In line with this approach, health insurance exchanges, such as HealthCare.gov, allow consumers to compare insurance plans online. Since the 2013 rollout of HealthCare.gov, administrators have added features intended to help consumers better understand and compare insurance plans. Although well-intentioned, changes to exchange websites affect the context in which consumers view plans, or choice architecture, which may impede their ability to choose plans that best fit their needs at the lowest cost. Methods: By simulating the 2016 HealthCare.gov enrollment experience in an online sample of 374 American adults, we examined comprehension and choice of HealthCare.gov plans under its choice architecture. Results: We found room for improvement in plan comprehension, with higher rates of misunderstanding among participants with poor math skills (P < 0.05). We observed substantial variations in plan choice when identical plan sets were displayed in different orders (P < 0.001). However, regardless of order in which they viewed the plans, participants cited the same factors as most important to their choices (P > 0.9). Limitations: Participants were drawn from a general population sample. The study does not assess for all possible plan choice influencers, such as provider networks, brand recognition, or help from others. Conclusions: Our findings suggest two areas of improvement for exchanges: first, the remaining gap in consumer plan comprehension and, second, the apparent influence of sorting order—and likely other choice architecture elements—on plan choice. Our findings inform strategies for exchange administrators to help consumers understand and select plans that better fit their needs.

Many health policy experts have endorsed insurance competition as a way to reduce the cost and raise the quality of medical care.¹ In line with this thinking, the Affordable Care Act (ACA) established federal and state exchanges that allow consumers to compare insurance plans online.² Opponents of the ACA have pushed for increased insurance competition as well. For example, Donald Trump has advocated for allowing consumers to purchase insurance plans across state lines,³ and although Paul Ryan has been outspoken about eliminating the ACA’s mandated exchanges, he has proposed private and state exchanges in their place to promote more robust competition.^4,5 In theory, when consumers choose among health plans in a competitive market, insurance companies will be forced to find ways to hold down health care expenditures, or risk losing business to competitors who offer lower priced plans.⁶ In practice, however, it is not clear whether most consumers understand differences between insurance plans well enough to discriminate between high- and low-cost plans and thus foster effective market competition.⁷

Since its initial rollout, the federal exchange website, HealthCare.gov, has incorporated several new design features intended to help consumers understand and compare insurance plans.⁸ For instance, the website now includes pop-up definition boxes, which allow consumers to view insurance term definitions (e.g., premium, copayment, deductible), to improve consumers’ comprehension.⁸ The website also includes an Out-of-Pocket Cost feature that calculates consumers’ estimated total yearly costs for each plan based on their self-reported expected medical use (low, medium, or high) for the next year.⁹ These new features attempt to help consumers better understand the plans, customize plan selection experiences, and estimate and compare costs for each plan.

Although well-intentioned, changes to an exchange website’s design affect the context in which consumers choose their plans (i.e., the website’s “choice architecture”).¹⁰ Choice architecture elements on HealthCare.gov include how plans are sorted, how plan attributes are arranged on the page, and even what attributes are displayed. For example, to browse plans on the 2016 version of HealthCare.gov, consumers scroll through plan options from the top to bottom of the webpage, and the plans are default sorted from lowest to highest premium. Like HealthCare.gov, most public exchanges (9 of the 13 state exchange websites) default sorted plan options from lowest to highest premium in the third open enrollment period.⁹ Thus, consumers in many marketplaces see the lowest premium plans first, even though other, higher premium plans may fit their needs better at a lower cost. In addition, each insurance plan on the 2016 version of HealthCare.gov is depicted as a 3 × 2 table, with plan attributes (premium, deductible, out-of-pocket maximum, estimated total yearly costs, copayments, and physician/drug coverage) displayed in separate blocks. For visual examples of plan displays on HealthCare.gov in 2016, see Appendix Table 1.

Rationally speaking, an exchange website’s choice architecture should not affect consumer choice. The way plans are sorted, how plans are visually depicted, or the font size and colors of plan attributes should not impact what plans consumers select. Instead, consumers enrolling in health insurance should make their decisions by weighing each plan’s attributes and selecting the highest value plan that provides the best coverage for their medical needs at the lowest cost. However, consumers do not always make fully rational decisions when selecting health insurance plans,^11–13 even when highly educated and experienced.^14–17 Their choices are influenced, in part, by the complexity of health insurance concepts and their misunderstanding of plans. But their choices may also be unwittingly influenced by the design of the marketplace website. For example, people’s choices can be unconsciously influenced by the order in which they receive information, or in an insurance context, the order in which they view one plan versus another.^18,19 Additionally, the attribute chosen to sort a list of plans can influence choice, sometimes making the sorting attribute appear more important than otherwise.²⁰ For instance, if plans are clearly sorted by premiums, and premium information is easily accessible,²¹ then consumers may overweigh premiums in their plan choices. When consumers make errors and choose suboptimal plans, they often leave substantial resources on the table and can be exposed to severe financial consequences if care is needed.²²

In this study, we sought to assess how well consumers understand insurance plans on an exchange website, by replicating the 2016 enrollment experience on HealthCare.gov. We then examined whether individuals’ choices would be affected by plan sorting order, a critical aspect of exchange websites’ choice architectures. To do so, we conducted an online comprehension survey to test people’s understanding of HealthCare.gov insurance plans and a randomized choice experiment to examine the effect of plan sorting order on plan choice. As federal and state governments continue to push for increased health care consumerism through efforts such as insurance exchanges, we aim to bring attention to how the choice architecture of exchange websites could lead to plan choice inconsistencies, which consumers likely do not recognize and which can impair efforts toward effective insurance marketplace competition.

Methods

Our participants were 374 U.S. citizens, ages 18 to 74 years. We recruited participants from the Amazon Mechanical Turk (MTurk) online participant pool in March 2016. Research has shown MTurk participants are attentive, provide responses indistinguishable from in-person data collection, and are more diverse than many convenience samples.^23–25 One subset (n = 76) completed a plan comprehension survey, while another subset (n = 298) participated in a plan choice experiment. We assessed plan comprehension and choice separately to adhere as closely as possible to HealthCare.gov enrollment, such that choice experiment participants, like real enrollees, would not spend designated time answering plan comprehension questions before selecting plans.

In both surveys, we replicated the 2016 consumer experience of viewing insurance plans on HealthCare.gov. Five plan options were drawn from the exchange’s metal tiers—catastrophic, bronze, silver, gold, and platinum (Appendix Table 1). Plans and their attributes were copied directly from HealthCare.gov search results in North Carolina. Prior to viewing plans, participants read pop-up definitions of insurance terms and selected their “estimated medical use,” as on HealthCare.gov (Appendix Figures 1 and 2). For simplicity, participants saw the same plans and estimated total yearly costs on their results pages, regardless of selected estimated medical use (Appendix Table 1).

At the end of both the comprehension survey and choice experiment, we assessed participants’ mathematical abilities using the Subjective Numeracy Scale, which is highly correlated with objective numeracy.²⁶ Participants also reported demographics, including age, sex, race, and level of education.

Plan Comprehension

In our comprehension survey, participants viewed three plans (catastrophic, bronze, platinum). Participants answered six plan comprehension questions on premiums, deductibles, out-of-pocket maximum, and estimated total yearly costs (see Table 1 for questions).

Table 1.

Participant Comprehension of Health Insurance Concepts

Concept	Question	% Answering Correctly
Monthly premium	For Plan B, how much will you have to pay each month just to be enrolled in an insurance plan?  [Free Response—Answer: $149 (monthly premium)]	81.6%
Deductible	Imagine that you are covered under Plan A. One day, you break your foot, so the total hospital bill for your medical care is $2,000. You have otherwise been very healthy, so you have not used any other medical services until now.  Not including your premium, how much will you have to pay out of pocket for this medical care under Plan A?  Free Response—Answer: $2,000 (under the $6,850 deductible)]	68.4
Out-of-pocket maximum	Imagine that you are covered under Plan E. You recently had to undergo major surgery and stayed in the hospital for a few days, so the total hospital bill for your medical care was $200,000.  Not including your premium, how much will you have to pay out of pocket for this medical care under Plan E?  [Free Response—Answer: $2,500 (out-of-pocket maximum)]	64.5
Estimated total yearly cost
Apply concept	Based on your “Low/Medium/High” expected medical use, how much will you expect to pay under Plan A for your medical care for the year?  [Free Response—Answer: $8,016 (estimated total yearly cost)]	76.3
Identify lowest cost plan	Based on your “Low/Medium/High” expected medical use, which plan will cost you the least—Plan A, Plan B, or Plan E?  [Multiple Choice – Answer: Plan E, with $6,156 as lowest dollar amount for estimated total yearly cost]	82.9
Identify highest cost plan	Based on your “Low/Medium/High” expected medical use, which plan will cost you the most—Plan A, Plan B, or Plan E?  [Multiple Choice—Answer: Plan B, with $8,652 as highest dollar amount for estimated total yearly cost]	81.6
Average comprehension score (% correct of six questions)		75.9

Plan Choice

In our choice experiment, participants viewed all five plans (catastrophic, bronze, silver, gold, platinum) on one page and were asked to select a plan to hypothetically enroll in. We randomly assigned participants to one of two plan sorting orders. Half of participants (n = 149) viewed the five plans ordered from lowest to highest insurance premiums, or “Low to High Sorting Order.” The other half (n = 149) viewed the same five plans ordered in reverse—from highest to lowest premiums, or “High to Low Sorting Order.” After choosing a plan, participants selected the factor they considered most important to their choice, or “Most Considered Factor.” Options for Most Considered Factor represented the given plan attributes: premium, estimated total yearly cost, deductible, out-of-pocket maximum, copays/coinsurance, other.

Statistical Analyses

For the comprehension survey, we present the proportion of participants who correctly answered each question. We calculated comprehension scores as percent correct of six questions. To test the impact of numeracy on plan comprehension, we used a linear regression, with numeracy as a predictor and comprehension as the outcome variable. We also examined whether comprehension scores varied as a function of education level using one-way ANOVA, comparing participants with at least a bachelor’s degree versus all other participants.

For the choice experiment, we examined the relationship between sorting order (Low to High Sorting Order v. High to Low Sorting Order) and plan choice using chi-squared tests. For each of the five plan options, we used logistic regression to quantify the effect of sorting order and numeracy on plan choice. To examine whether order effects were larger among the less numerate, we tested for an interaction between numeracy and sorting order. We used a hierarchical regression in which the main effects were included in Model 1 and the interaction was added in Model 2. The interaction did not add additional predictive power; thus, our final model includes only the main effects of sorting order and numeracy (Appendix Table 3). Finally, we determined the relationship between sorting order and Most Considered Factor using a chi-squared test.

We used IBM SPSS Statistics software in analyses for the comprehension survey and choice experiment. No funding source had any role in the study.

Results

Appendix Table 2 shows the demographic characteristics of our study participants, with a relatively even mix of female and male respondents in both surveys. The median participant age was 29.5 and 32.0 years, for the comprehension and choice experiment surveys, respectively, younger than the median age range of HealthCare.gov enrollees (35–44 years) in the most recent enrollment period.²⁷ Around 80% of study participants were non-Hispanic White compared to 63% of HealthCare.gov enrollees in 2016.^27,28 Study participants were relatively well-educated (40% college graduates) compared to uninsured adults in the United Studies (15% college graduates), who comprise the main population targeted for enrollment on the exchanges.²⁹ Of those completing the comprehension survey, 95% had been enrolled in an insurance plan, either currently or at some point in the past.

Demographics did not significantly differ across participants in the comprehension survey and choice experiment, nor did they differ between Sorting Order conditions.

Plan Comprehension

Table 1 displays the percent of participants who correctly answered each of the six questions. On average, participants correctly answered 75.9% of the comprehension questions. A total of 27.6% of participants answered all six questions correctly, 34.2% answered five questions correctly, and 19.7% answered four questions correctly. A total of 18.3% of participants correctly answered three questions or fewer. Participants had the highest levels of comprehension for monthly premium (81.6%), ability to identify the highest estimated cost plan (82.9%), and ability to identify the lowest estimated cost plan (81.6%). Participants demonstrated lower plan comprehension when asked to apply the definitions of deductible (68.4%), out-of-pocket maximum (64.5%), and total yearly cost (76.3%) to plans.

Plan Comprehension and Numeracy

Numeracy significantly predicted plan comprehension; for each point increase in numeracy, there was a 7.1% increase in comprehension score (r = 0.291, P = 0.011).

Plan Comprehension and Education

Education level and subjective numeracy score were also significantly correlated (r = 0.390, P = 0.001). Participants with lower levels of education had lower mean comprehension scores than those with higher education (69.7% v. 82.4%; P = 0.017, η² = 0.075).

Plan Choice

Sorting Order and Plan Choice

There was a significant relationship between sorting order and plan choice for three (Plans A, B, D) of five plan options (Figure 1). Plan E (the highest premium plan) was the most popular plan whether participants viewed plans in Low to High or High to Low Sorting Order. Participants had 3.40 and 4.57 higher adjusted odds (P < 0.001) of choosing Plan A (lowest premium plan) and D, respectively, when they viewed plans in High to Low versus Low to High Sorting Order. Participants had 0.13 lower adjusted odds (P < 0.001) of choosing Plan B when they viewed plans in High to Low versus Low to High Sorting Order.

Figure 1.

Participant plan choice as a function of sorting order.

Sorting Order and Most Considered Factor

Figure 2 shows participants’ most considered factor in insurance plan choices. Premiums and estimated yearly costs were the top two factors overall (33.2% and 27.9%, respectively) and in each of the sorting orders. Participants were most likely to select premium as Most Considered Factor regardless of order in which they viewed the plans (34.2% of Low to High participants; 32.2% of High to Low participants; P = 0.941).

Figure 2.

Participant responses for “Most considered factor in choosing plan.”

Discussion

We presented the 2016 version of HealthCare.gov to study participants and found room for improvement in comprehension of the health plan attributes they viewed, with higher rates of misunderstanding among those with poor math skills. We also found that participants’ plan choices were significantly influenced by the order in which they viewed the plans, one component of the website’s choice architecture. Although participants chose different plans, they cited the same Most Considered Factors in their plan choices—premium or estimated total yearly costs for around 60% of participants. Our findings suggest that consumers enrolling on HealthCare.gov do not fully understand insurance plan options and that the exchange website’s choice architecture is likely influencing plan choices without consumers’ knowledge.

By simply reversing the order in which participants viewed the same five plans, we observed significantly different plan choices. Despite different plan choice patterns depending on order, participants cited the same most considered factors for their plan choices (i.e., premium or estimated total yearly cost). This indicates that participants likely did not have conscious or rational explanations for why they chose different plans.

Participants in both sorting order conditions cited premiums more frequently than other plan attributes as the most considered factor in their plan choices, consistent with previous studies on consumer preferences in insurance decisions.^12,15,30 However, Plan E (the highest premium plan) was the most popular plan regardless of which order plans were displayed. This may have resulted from participants’ misunderstanding of how premiums work. Alternatively, given the relatively high comprehension scores for premiums, it may have been that participants knew they would not have to pay for the plans, so were inclined to choose a plan with more generous coverage and disregard costs.

By sorting plans by premiums and prominently displaying the premium amount in each plan, HealthCare.gov’s choice architecture appears to emphasize premiums over other plan attributes. This apparent emphasis on premiums is present even though the Out-of-Pocket Cost Feature (estimated total yearly cost) was introduced to help consumers consider their medical needs and discourage a common instinct to default to the lowest premium plans, which often have high deductibles. Thus, if enrollees come in with a bias toward weighing premiums heavily in their enrollment decisions, HealthCare.gov does little to draw attention to other important attributes, and may even promote this premium bias.

Therefore, while continuing to refine the tools available to consumers on current exchange websites like HealthCare.gov, or developing new exchange websites in the future, we urge administrators to recognize the influential role that choice architecture can play in enrollees’ plan choices. Not only can choice architecture elements steer consumers toward suboptimal plans and higher costs for the year, but because individuals tend to keep their current insurance plans even when better plans are available, poor choices may stick with consumers in the future.¹¹ For example, among the five plans we provided to participants, there was a $2,661 difference in estimated total costs each year between the highest cost plan (Plan B) and the lowest cost plan (Plan D).

If the ultimate goal is to help consumers enroll in plans that best fit their medical needs, then exchange website administrators might consider modifying their current choice architectures to emphasize estimated total yearly costs. For example, by sorting plans from low to high estimated total yearly costs rather than premiums, consumers may be more likely to consider this important attribute when making their choices. California and Kentucky are two states with exchange websites that already sort plans by this attribute.⁹ Likewise, if estimated total yearly costs were moved to a more prominent location in plan displays, such as the top row or upper-left corner (where monthly premiums are currently displayed), this cost estimate might be more salient to consumers as they compare plans. Future experiments can test these suggested improvements to see if they improve the stability of participant plan choices with display order. However, we caution that before emphasizing estimated total yearly cost, it is important to ensure that estimated costs align with the actual costs consumers will experience for the year. Currently, estimated total yearly cost depends on consumers’ selections between three medical use profiles, so its accuracy relies on consumers’ predictions for future use, the available profiles that are able to capture and match actual medical use, and the reliability of the estimates themselves.

In our comprehension survey, our study participants demonstrated suboptimal comprehension of insurance concepts. Given this finding, it is likely that in our choice experiment, participants were viewing and choosing plans without fully understanding the insurance concepts. Without adequate comprehension of insurance terms, it is difficult to make complex tradeoffs between plans and more likely that choice architecture elements, like sorting order, influence plan choice.

Our least numerate participants fared the worst in plan comprehension, underscoring the importance of math skills in understanding and comparing health insurance plans. Of concern, lower numeracy skills are prevalent among those in the target uninsured Marketplace population.³¹ According to recent estimates, more than a quarter (28.8%) of uninsured adults have a Below Basic level of numeracy and only 8.6% have a Proficient level of numeracy.³² Our participants had higher average education levels and therefore likely numeracy skills than typical Marketplace enrollees, so the difficulties in plan comprehension we observed are likely worse among those actually shopping on HealthCare.gov.

To improve comprehension of insurance concepts and encourage more informed decisions about plan enrollment, we encourage exchange website administrators to continue to build on efforts to educate enrollees on complicated insurance concepts, such as the website’s pop-up definitions. For example, consumers have suggested improving the definitions by including narrative examples for insurance concepts, such as how deductibles work with real dollar amounts.³³ These sorts of examples could improve not only comprehension of insurance concepts but also consumers’ ability to apply these concepts to plans, which is where we found the greatest comprehension deficits among our participants. Other strategies to improve plan comprehension include step-by-step tutorials, which could briefly walk enrollees through insurance concepts and allow them to practice applying concepts to plans, and better access to navigators, who help consumers search exchange websites and enroll in plans.³⁴

Future research is needed to determine the impact of choice architecture on actual plan enrollment. Although the influence of choice architecture elements can be tested in experiments such as this one, studies of choice architecture on the exchanges themselves would provide the best evidence. Other variations in choice architecture, some of which are currently used by state-based exchanges, include sorting plans with silver tiered plans first for consumers who qualify for premium tax subsidies,⁹ drawing consumers’ attention to plan attributes other than premiums, or making plan displays narrower so consumers can see more plans at once. As exchanges modify their choice architectures (e.g., switching the default sorting order from premium to estimated total yearly cost), administrators could track how consumer plan choices differ.

Limitations

Our participants were slightly younger, more educated, and more likely to be non-Hispanic White than typical HealthCare.gov enrollees.²⁷ As online participants, they were likely to be more technologically savvy. However, if anything, these limitations make our findings more striking, as the challenges and inconsistencies identified may be more pronounced in the broader population of HealthCare.gov enrollees.

Our study aimed to replicate the HealthCare.gov enrollment experience, but could not perfectly copy all aspects of that process. First, we conducted a simulated experiment with participants who were not actually shopping for insurance, which could have influenced their insurance choices. Also, real consumers might get help from others and take more time before choosing a plan. However, our findings relate to seemingly unconscious influences of choice architecture, which may affect choice regardless of how much time individuals take to select a plan.

Our study does not assess for all possible plan attributes or choice architecture elements that may influence choice (e.g., provider networks, brand recognition, user-controllable sorting, or visual elements such as arrangement and colors). Instead, we focus on the impact of HealthCare.gov’s default sorting order, a particularly prominent aspect of the site’s choice architecture, on plan choice, while replicating the 2016 HealthCare.gov experience as closely as possible.

Conclusion

In a simulated experience of shopping for insurance plans on HealthCare.gov, we found that some participants could not reliably answer basic comprehension questions about the plans. Additionally, their plan choices differed significantly depending on the order in which they viewed the plans, without any change in the plan attributes they cited as most critical to their decisions. Our study highlights both the gaps in consumers’ comprehension of key insurance concepts and how the HealthCare.gov choice architecture likely influences plan choice. By recognizing the influence of choice architecture on plan choice and leveraging this influence to encourage better plan choice, administrators of HealthCare.gov and other exchange websites have the opportunity to help enrollees pick plans that best fit their needs at the lowest cost.