Evaluating the Presence of Cognitive Biases in Health Care Decision Making: A Survey of U.S. Formulary Decision Makers

Abstract

BACKGROUND:

Behavioral economics is a field of economics that draws on insights from psychology to understand and identify patterns of decision making. Cognitive biases are psychological tendencies to process information in predictable patterns that result in deviations from rational decision making. Previous research has not evaluated the influence of cognitive biases on decision making in a managed care setting.

OBJECTIVE:

To assess the presence of cognitive biases in formulary decision making.

METHODS:

An online survey was conducted with a panel of U.S. pharmacy and medical directors who worked at managed care organizations and served on pharmacy and therapeutics committees. Survey questions assessed 4 cognitive biases: relative versus absolute framing effect, risk aversion, zero-risk bias, and delay discounting. Simulated data were presented in various scenarios related to adverse event profiles, drug safety and efficacy, and drug pricing for new hypothetical oncology products. Survey questions prompted participants to select a preferred drug based on the information provided. Survey answers were analyzed to identify decision patterns that could be explained by the cognitive biases. Likelihood of bias was analyzed via chi-square tests for framing effect, risk aversion, and zero-risk bias. The delay discounting section used a published algorithm to characterize discounting patterns.

RESULTS:

A total of 35 pharmacy directors and 19 medical directors completed the survey. In the framing effect section, 80% of participants selected the suboptimal choice in the relative risk frame, compared with 38.9% in the absolute risk frame (P < 0.0001). When assessing risk aversion, 42.6% and 61.1% of participants displayed risk aversion in the cost- and efficacy-based scenarios, respectively, but these were not statistically significant (P = 0.27 and P = 0.10, respectively). In the zero-risk bias section, results from each scenario diverged. In the first zero-risk bias scenario, 90.7% of participants selected the drug with zero risk (P < 0.001), but in the second scenario, only 32.1% chose the zero-risk option (P < 0.01). In the section assessing delay discounting, 54% of survey participants favored a larger delayed rebate over a smaller immediate discount. A shallow delay discounting curve was produced, which indicated participants discounted delayed rewards to a minimal degree.

CONCLUSIONS:

Pharmacy and medical directors, like other decision makers, appear to be susceptible to some cognitive biases. Directors demonstrated a tendency to underestimate risks when they were presented in relative risk terms but made more accurate appraisals when information was presented in absolute risk terms. Delay discounting also may be applicable to directors when choosing immediate discounts over delayed rebates. However, directors neither displayed a statistically significant bias for risk aversion when assessing scenarios related to drug pricing or clinical efficacy nor were there significant conclusions for zero-risk biases. Further research with larger samples using real-world health care decisions is necessary to validate these findings.

What is already known about this subject

• Cognitive biases are psychological tendencies to process information in predictable patterns that result in deviations from rational decision making.

• Previous research with patients and providers has demonstrated that cognitive biases can affect health care decision making.

What this study adds

• This study explores whether pharmacy and medical directors from health care payer organizations were susceptible to cognitive biases when analyzing clinical data and making decisions on drug preference.

• This study highlights the importance of framing clinical data in an appropriate, unbiased manner.

• The length of time before receiving a monetary discount appears to have an apparent but minimal effect on pharmacy and medical directors when making drug formulary contracting decisions.

Behavioral economics is a field of economics that draws on insights from psychology to understand and identify patterns of decision making.¹ In studying decision making, behavioral economists have identified cognitive biases, which are psychological tendencies to process information in a predictable pattern.² These cognitive biases can lead to errors or irrational decisions.² Studies suggest that such errors may be common among patients and providers when making health care decisions.^3-6 Research has yet to investigate the effects of cognitive biases on decision making in pharmacy and therapeutics (P&T) committee processes and formulary management. Such research may provide useful information to understand and improve formulary and reimbursement decision making.

The objective of this study was to understand if cognitive biases affect U.S. pharmacy and medical director decisions involving evidence evaluation, formulary management, and pricing strategies. Assessing cognitive biases in the managed care setting is important, since decisions made by members of P&T committees can potentially affect many patient lives in terms of access to and costs of treatment. Cognitive biases, if present, may lead to misinterpretations of clinical information and making suboptimal choices in adjusting formularies and prioritizing medications for covered patients. Bringing cognitive biases to light can potentially help people avoid these common thinking patterns and improve managed care decision making. Studying cognitive biases may also help improve communication between pharmaceutical manufacturers, clinical researchers, and pharmacy and medical directors.

Numerous cognitive biases have been identified that influence decision making.² This study focused on biases that were hypothesized to most likely play a role in managed care decisions and could be evaluated with survey-based research using hypothetical scenarios. The 4 cognitive biases that met these criteria and were included in this study are (1) absolute versus relative framing, (2) risk aversion, (3) zero-risk bias, and (4) delay discounting.

The framing effect describes the influence of how information is framed or presented on decision making.⁷ The framing effect comes into play in a variety of different situations. One means of framing information that is relevant to health care decision making is whether risks are framed in relative versus absolute risk terms. Research has shown that individuals overvalue risk reductions presented in relative terms when compared with risk reductions presented in absolute terms.⁸ For example, 1 study suggested that patients overestimate the value of mammogram screenings when information is provided in relative versus absolute risk terms.⁹ It is possible that formulary decision makers are also influenced by the framing effect, since clinical decisions are based on evidence that can be framed differently.

Risk aversion describes a preference for sure gains versus potentially greater gains that involve risk of loss.¹⁰ For example, 1 study examined participant choices between 2 hypothetical medical treatments varying in levels of risk. Choices were varied, such that some were more certain but included a lower payoff in terms of number of days of full health, while others were less certain but included the potential for a larger payoff in terms of days of full health. Participants demonstrated a moderate tendency to choose the more certain option with the lower payoff (i.e., the risk-averse choice), deviating from the choice that would maximize expected utility.¹¹ Risk aversion may be applicable to the managed care setting, since pharmacy and medical directors weigh the risks and benefits of different treatment options when making formulary decisions.

Zero-risk bias is related to framing and risk aversion. It describes the tendency to choose to reduce risk to zero, rather than to choose non-zero risk reductions, even when non-zero risk reductions result in an equal or greater absolute risk reduction.¹² To our knowledge, this bias has not been previously studied with respect to medical or health care decisions. One study, which was not related to health care, collected responses from 408 government workers and professionals on their preferences for cleaning up hazardous waste. Participants ranked hypothetical options for cleaning up waste, some of which included partial reductions of risk at multiple sites, while others included complete reduction of risk at 1 site. The researchers found that 42% of the subjects exhibited a bias toward the zero-risk option, even though the other options resulted in a greater net lowering of risk.¹² The zero-risk bias may be relevant in the managed care setting, since formulary decisions makers have to weigh the safety risks associated with different therapeutic options.

Finally, the delay discounting effect is the tendency to prefer an immediate reward, even when a delayed reward is greater in value.¹³ This tendency has been demonstrated for real and hypothetical rewards, and its relationship to health-related decision making has been studied from the patient perspective. For example, in 1 study, individuals who discounted delayed rewards more highly were more likely to report engaging in risky behaviors (e.g., substance use) and less likely to report engaging in protective behaviors (e.g., exercising or wearing a safety belt) than those who discounted delayed rewards to a lesser extent.¹⁴ Delay discounting may be also present in payer-related scenarios, such as when engaging in drug pricing and contract negotiations with pharmaceutical companies.

This study evaluated whether formulary decision making is potentially influenced by the presence of these 4 cognitive biases using a survey-based approach.

Methods

Survey Instrument

Data on study participant characteristics and decision making were collected via responses to an electronic survey designed using Decipher. Decision-making tendencies were assessed via answers to hypothetical scenarios. Scenarios were adapted from previous behavioral economic research and modified to closely mirror actual managed care decision-making situations based on expert opinion. Before survey launch, the survey was piloted by peers separate from the research team to ensure clarity in the question wording.

The main portion of the survey consisted of 33 questions and was divided into 4 randomized sections, 1 for each of the tested behavioral economic concepts. The participants were not made aware that the scenarios were designed to test these behavioral concepts. Rather, they were informed that the survey involved research around blinded, new pharmaceutical therapies. Each section contained questions and scenarios that were designed to mimic actual, blinded payer research studies of new pharmaceutical products and prompted participants to make a decision based on the presented information. All drug therapies, prices, and associated statistics presented in the survey were fictitious and not based on any existing products or studies.

Framing Effect.

The framing effect section consisted of 2 near identical scenarios prompting participants to assess 3 blinded drugs targeted for breast cancer treatment based on their adverse event (AE) profiles. In each scenario, the drugs were compared against a standard of care (SOC) arm, and 2 AEs were reported. In the first scenario, the AE profiles of 3 drugs (F, G, and H) were presented in relative risk terms, with Drug H being the optimal choice based on total AE count. In the second scenario, the AE profiles of 3 other drugs (J, K, and L) were presented in absolute risk terms, with Drug L being the optimal choice based on total AE count. Importantly, the optimal choice was mathematically similar across scenarios (e.g., the calculated AE rate for Drug H was nearly equivalent to the AE rate for Drug L). Further details of the question prompt and scenarios can be found in Appendix A (available in online article).

Risk Aversion.

The risk aversion section also included 2 scenarios. In the first scenario, 2 unspecified oral chemotherapies demonstrating similar efficacy were compared. Two drugs (N and M) were presented, with 1 drug being priced higher than the other, while also exhibiting higher direct pharmacy cost savings due to potentially lower treatment use for chemotherapy-induced nausea and vomiting (CINV) medications. The initial cost of Drug N was $800 higher than that of Drug M, but with a potential long-term savings of $1,200 if Drug N was chosen. However, there was no guarantee that the CINV medication cost savings would manifest with Drug N, while the initial lower cost of Drug M was certain. Consequently, Drug N represented the riskier choice, and if participants were risk averse, they were expected to prefer Drug M.

The second scenario presented 2 other unspecified chemotherapy agents for which prices were equivalent. Clinical trial data were presented for each drug (P and Q) with progression-free survival (PFS) as the primary outcome. Drug P demonstrated a consistent PFS, averaging just over 8 months between its 2 trials. The PFS range of Drug Q was less consistent, with 1 study showing a PFS of 24.2 months and the other a PFS of 4.5 months. Consequently, a choice for Drug P would provide evidence for risk aversion. Participants were asked to choose a preferred therapy based on the clinical trial data. Further details of the question prompt and scenarios can be found in Appendix A.

Zero-Risk Bias.

In the zero-risk effect section, participants were again presented with 2 unrelated scenarios and were asked to choose a preferred therapy based on the information provided. The first scenario presented 2 drug therapies indicated for renal cell carcinoma, each being compared with SOC. Data for patient withdrawals due to severe AEs were presented. Drug A demonstrated 0 withdrawals versus 7 patients withdrawing in the SOC arm, while Drug B had 9 patients withdraw versus 24 patients in the SOC arm. In this case, Drug B represented the optimal choice because the absolute reduction in AEs was greater than for Drug A (15 vs. 7).

The second scenario presented 3 nondescript chemotherapies and the number of cases observed for 2 AEs, simply labeled as AE1 and AE2. Drug C had 6 cases of AE1 and 34 cases of AE2; Drug D had 12 cases of AE1 and 42 cases of AE2; Drug E had 0 cases of AE1 and 47 cases of AE2. Drug C had the fewest total AEs (40 vs. 54 vs. 47), making it the optimal choice. Further details of the question prompt and scenarios can be found in Appendix A.

Delay Discounting.

In the delay discounting section, participants were asked to respond to 27 different drug pricing scenarios based on the Monetary Choice Questionnaire (MCQ). The MCQ is a widely used, validated delay discounting index that requires individuals to choose between a series of hypothetical, smaller immediate monetary rewards and larger delayed monetary rewards.¹⁵ In this section, each scenario offered 2 options from a drug manufacturer hoping to contract with a managed care organization (MCO). Participants were asked to choose between smaller immediate discounts off invoice (i.e., immediate reward) versus a larger delayed rebate (i.e., delayed reward). The per-treated-member-per-month (PTMPM) cost savings and length of delay varied in each of the scenarios, with PTMPM cost savings varying from $25 to $85 and length of delay ranging from 7 days to 186 days. Further details of the question prompt and scenarios can be found in Appendix A and Appendix B (available in online article).

Survey Recruitment and Implementation

The survey was sent in October 2016 to participants recruited through Xcenda’s Managed Care Network. The Managed Care Network is composed of managed care professionals from national and regional health plans in the United States and covers approximately 200 million lives. Approximately 90% of the professionals have served or are currently serving on a P&T committee. Criteria for inclusion in this study required participants to be employed as a pharmacy or medical director at an MCO or similar institution within the previous 2 years before receiving the survey invite and to have served as a member of the organization’s P&T committee. Eligible participants received an email with personalized links to access the survey. Participation in the survey was voluntary. All consented to participate before receiving the survey, and an honorarium was paid by Xcenda after survey completion. The identities of the participants remained blinded throughout the study.

Data Analysis

Descriptive statistics were used to summarize the study participant characteristics. In the framing biases arm, a chi-square test of independence was used to analyze whether decisions were related to framing. The risk-aversion and zero-risk effect arms used a chi-square goodness of fit test to determine whether study participants were more likely to choose a biased option than would be expected by chance. Results from the MCQ in the delay discounting arm were analyzed via a published Excel-based scoring algorithm, and a delay discounting curve was produced.¹⁶ A hyperbolic discounting curve was fit to the observed data, which was plotted as the subjective value of the delayed reward (V) against the length of delay (D). The acceleration of this curve (k) described the magnitude of delay discounting, with k values indicating a greater preference for more immediate rewards (i.e., greater delay discounting). The value of k is calculated using the following equation: with A representing the actual value of the delayed reward.

A supplementary delay discounting assessment was performed to test the magnitude effect. This effect describes the tendency for a greater preference for delayed rewards, when the overall rebate amounts are higher than when using smaller rebate amounts.¹⁷ Appendix B lists the magnitudes (small, medium, and large) of each rebate amount for each scenario.

Results

A total of 76 invitations were sent to members of the Managed Care Network panel, and 54 responses were received, yielding a 71% response rate. The majority of participants were pharmacy directors from MCOs. The characteristics of the survey participants are summarized in Table 1.

TABLE 1.

Study Participant Characteristics^a

	Frequency (n)	%
Participant role
Pharmacy director	35	64.8
Medical director	19	35.2
Organization type
Managed care organization	43	79.6
Pharmacy benefit manager	6	11.1
Integrated delivery health system/integrated delivery network	5	9.3
Plan type
Commercial only	7	13.0
Medicaid/Medicare only	6	11.1
Mixed	37	68.5
Not available	4	7.4
Area of coverage
National	13	24.1
Regionalb	40	74.1
Northeast	16	40.0
Midwest	10	25.0
South	7	17.5
West	15	37.5
Not available	1	1.9
Number of lives covered
Small (<750,000)	27	50.0
Medium (750,000-4,999,999)	20	37.0
Large (≥5,000,000)	7	13.0

^a N = 54.

^b Regional plans sum to greater than 100%, as plans may cover more than 1 region. Percentages for each region were out of the total number of participants who were affiliated with regional plans.

Relative Versus Absolute Framing

In the relative risk framing scenario, 43 (79.6%) participants preferred Drug F over drugs G and H, while in the absolute risk framing scenario, 33 (61.1%) participants chose Drug L over Drugs J and K. As shown in Figure 1, 11 (20.4%) participants versus 33 (61.1%) chose the optimal choice in the relative and absolute framing scenarios, respectively. Participants were less likely to pick the optimal drug based on AE profiles presented in relative terms than when they were presented in absolute terms (chi square = 18.56, P < 0.0001).

FIGURE 1.

Relative and Absolute Risk Framing Results

Risk Aversion

The results of the 2 questions assessing risk aversion are presented in Figure 2. In Scenario 1, which assessed aversion related to financial risk, the majority of participants chose the riskier option (Drug N, 57.4%). In Scenario 2, which presented drug efficacy data, more participants chose the risk-averse option (Drug P, 61.1%). These findings did not reach statistical significance (P = 0.27 for Scenario 1; P = 0.10 for Scenario 2).

FIGURE 2.

Risk Aversion Results

Zero-Risk Bias

For Scenario 1 in the zero-risk bias section, 49 (90.7%) survey participants chose Drug A (the zero-risk option), whereas 5 (9.3%) chose Drug B (chi square = 35.85, P < 0.001). Based on the information provided, the calculated absolute risk reduction for Drug A was 10.1% versus 20.8% for Drug B, making Drug B the objectively optimal choice. In Scenario 2, 1 of the 54 participants answered Drug D, which had the worst AE profile out of the 3 options. Because this selection was not related to zero-risk bias and was only included to make the most optimal choice less obvious, this data point was excluded in the analysis. Of the remaining 53 participants, 36 (67.9%) chose Drug C, whereas 17 (32.1%) selected Drug E (the zero-risk option). Drug C had 6 cases of AE1 and 34 cases of AE2, yielding a combined absolute risk of 22.4% when calculated. Drug E had 0 cases of AE1 and 47 cases of AE2, leading to a higher combined absolute risk of 28.7%. Based on the chi-squared analysis, participants in Scenario 2 significantly deviated from the zero-risk bias choice and preferred the objectively lower-risk drug (chi square = 6.81, P < 0.01). Results of drug preference based on zero-risk are presented in Figure 3.

FIGURE 3.

Zero-Risk Bias Results

Delay Discounting

When assessing delay discounting, 54% of all decisions overall were in favor of a delayed rebate. The rates at which the subjective value of the delayed rebate decreased over time (discount rate, or k) were calculated for each participant. The overall median k value was 0.00254, yielding a shallow hyperbolic delay discounting curve, as shown in Figure 4. This curve demonstrates that participants have a slight preference for rewards with less delay. In an exploratory subgroup analysis, the median k values were found to not be significantly different between pharmacy directors and medical directors. When assessing the magnitude effect, it was found that when rebate amounts were smaller, 50.0% of participants preferred delayed rebates, while 58.6% preferred delayed rebates when rebate amounts were larger. However, the difference in delayed reward preference when using small versus large rebate amounts was not statistically significant.

FIGURE 4.

Delay Discounting Curve

Discussion

This study sought to identify the presence of 4 cognitive biases in formulary decision making when evaluating clinical and economic data. The cognitive bias that appeared to have the most consistent and pronounced results in this study was the framing bias. Pharmacy and medical directors tended to prefer a drug more when data were represented in relative risk terms versus absolute risk terms. This finding mirrors results from a previous framing effect study in the health care field,⁶ which also used surveys to assess decision making related to health care. In this work, patients were asked to recommend a medication indicated for preventive bone disease to a friend based on risk of breast cancer. Patients demonstrated the tendency to make unfavorable recommendations when presented with relative risk statistics.⁶ Our study suggests that pharmacy and medical directors are similarly susceptible to this relative risk framing bias. Importantly, however, the previous study demonstrated that patients corrected their decisions once absolute risk information was shared, suggesting that this bias can be mitigated. These results highlight the need for vigilance when presenting health care data to ensure that information is fairly portrayed and appropriately interpreted. Specifically, confusion can be avoided through the use of absolute risk over relative risk when possible.⁹

Regarding risk aversion, directors were not significantly risk averse regarding cost or survival. However, when presented with risks regarding costs, more directors favored riskier options, whereas when they were presented with risks regarding patient safety and efficacy, directors favored the less risky option. Our sample size may have limited the ability to detect true differences. Despite the nonsignificant results, the findings suggest that directors may be more cautious when evaluating safety and efficacy, which would have a greater effect on patient health, than when addressing drug costs. Until further research is conducted from the perspective of formulary decision makers, these findings on risk aversion remain to be confirmed.

A previous health care study also assessed risk aversion, focusing on physicians. Results from that study suggested that physicians with 15 years of experience or more exhibited risk-averse behaviors versus physicians with less experience.¹⁸ This previous research highlights that risk aversion may influence health care decisions from the provider perspective and is not a phenomenon that is exclusive to patients. Understanding inclination for risks among formulary decision makers may be of value for the development of market access communication strategies and targeted product messaging for formulary reimbursement discussions.

Results of the 2 scenarios comparing drug safety data from the zero-risk bias section conflicted with each other and did not yield conclusive findings. In Scenario 1, in which withdrawal data were presented, directors were significantly susceptible to the zero-risk bias and selected the drug with higher combined risk. In Scenario 2, which presented drugs with different AE rates, directors preferred the drug with objectively lower combined risk. A rationale for the difference between scenarios may be due in part to the survey question design. While in Scenario 1, one must calculate the risk difference versus SOC to find the optimal choice, the second scenario can be successfully answered by merely selecting the option with the least number of total cases of AE1 and AE2, which happened to also be the optimal choice. Replication research is warranted to determine the presence of zero-risk bias in the managed care setting.

In the final section, directors in general displayed a pattern of discounting delayed rewards. Therefore, the length of time that one must wait before receiving a rebate appears to have some degree of effect on the perceived value of that rebate. The hyperbolic curve generated from the median k value of 0.00254 highlights this finding that directors displayed the expected behavior of discounting the value of delayed rewards more heavily as the length of the delay increased. However, the shallowness of the curve suggests that discounting was minimal compared with previously published studies in general populations. For example, the average k value of 60 participants in the control group of a previous study was 0.013, while a more recent study of 111 participants from a general pool of paid survey takers found an average k value of 0.012.^15,17

Both k values in these studies are considerably higher than that found for directors in our study, representing deeper delay discounting curves and greater preferences for immediate rewards than with the scenarios and population in this study. Also, in the majority of scenarios, directors chose to wait for the larger delayed rebate over the smaller immediate discount. Thus, directors of payer organizations may be more willing to wait to obtain greater financial benefits from health care decisions when compared with individuals from the general population. Indeed, payers often have to wait 90-120 days to receive reimbursement from manufacturers after a claim has been paid.

These delay discounting findings could also have potential implications for how value-based contracts with payer organizations can be optimally constructed. For example, some value-based product rebate schemes are set up so that a rebate is given if products do not meet a predetermined clinical benefit threshold. Given that directors of payer organizations minimally discount delayed rewards, this research suggests the feasibility of assessing longer-term outcomes for value-based contracts with payers. Understanding payer perceptions around risks and delayed rewards may be valuable in navigating strategies with payers to share risk and determine a reasonable timeline for price offsets and discounting with drug manufacturers.

Limitations

The findings of this study must be interpreted in light of its limitations. Because of the use of simulated scenarios with unidentified drugs, AEs, and hypothetical data, participants may have made assumptions about drug classes, severity of events, and magnitude of effects beyond the information presented. The data inputs for AE rates, costs, monetary discounts, and other clinical outcomes presented in the survey were specifically selected to test the cognitive biases of interest, which may vary from scenarios found in real-world settings. Any conclusions made regarding the presence of these biases are limited to the findings from the specific scenarios tested. Furthermore, caution should be exercised when generalizing the results of this study, since the sample was limited in size and consisted of only pharmacy and medical directors. Participants responded individually to these scenarios and not in a collective group setting, which may not accurately reflect the decision-making processes of P&T committees. Study participants were also provided limited information, while P&T committee processes involve extensive literature review and thorough assessment of medications. Finally, the study is subject to certain flaws inherent in all self-report methodologies, such as misunderstanding, misrepresentation, and socially desirable responding.

Conclusions

Overall, pharmacy and medical directors displayed susceptibility to some cognitive biases, similar to many other studied populations. Directors tended to overvalue data when presented in relative risk versus absolute risk terms. They were not significantly risk averse, and directors demonstrated a significant bias toward preferring zero risk in one scenario but not in the other scenario. Because of conflicting results, it is inconclusive whether the zero-risk bias is present among directors. Finally, for delay discounting, directors demonstrated expected discounting patterns, albeit to a much lesser extent than among samples previously studied.

To confirm these findings, future researchers could explore cognitive biases with regard to formulary decisions using real-world cases. It may also be insightful to explore whether these results hold true for different disease states. The results from this study and from future work can help us understand how formulary decision makers assess clinical and economic data. These findings may provide insights into how directors from payer organizations frame their decisions with regard to formulary approval, tier placement, rebate and pricing negotiations, and risk-sharing agreements for value-based contracting. A greater understanding of the effect of these cognitive biases may translate into improved communications with clinical researchers, clinicians, marketing agencies, and pharmaceutical representatives to better inform formulary decision making.

APPENDIX A. Survey Questions

NOTE: Survey section headers were not present in the original survey but are included here for clarity.

Main Survey Prompt

Pharmaceutical company X has several chemotherapy agents and medications for palliative care currently on the market and in various stages of development. The purpose of this survey is to seek payers’ initial preferences of medications based on the data provided. Some questions involve hypothetical scenarios that are meant to provide insight on how to best strategize for products in pre-clinical development.

As you read through each question, please assume treatment “preference” to be meant in a general context, without delving into the nuances of any one specific plan. Please treat each scenario as independent from other scenarios unless otherwise stated. Responses to prior questions once submitted cannot changed. Please answer the following questions to the best of your ability with the information provided in each scenario.

Framing Effect

Drug F, Drug G, and Drug H are indicated for breast cancer. All drugs carry a risk of minor adverse events (AEs). A comparative safety and efficacy study compared 3 drugs to standard of care (SOC). In the SOC arm, there were 10 cases of AE1 and 200 cases of AE2. The results are shown below:

• Drug F has a 40% risk of AE1 and 76% of AE2 compared to SOC

• Drug G has a 80% risk of AE1 and 88% of AE2 compared to SOC

• Drug H has a 90% risk of AE1 and 66% of AE2 compared to SOC

Based on the information provided, which drug would you consider preferred therapy?

• Drug F

• Drug G

• Drug H

In a large head-to-head study (N=990), patients were randomized into 3 breast cancer therapies, Drug J, Drug K, Drug L, with 330 patients in each arm. Using an intention-to-treat protocol, adverse events (AEs) were reported. The results are shown below:

• Drug J had 4 cases of AE1 and 154 cases of AE2

• Drug K had 8 cases of AE1 and 176 cases of AE2

• Drug L had 9 cases of AE1 and 132 cases of AE2

Based on the information provided, which drug would you consider preferred therapy?

• Drug J

• Drug K

• Drug L

Risk Aversion

Two new oral chemotherapies are available for multiple myeloma, Drug M and Drug N. Both drugs are comparable in efficacy. A recent cost-effectiveness study demonstrated that Drug N would potentially lead to direct pharmacy savings of $2,000 per patient per month in associated CINV treatment costs.

• Drug M costs $2,400 for 30-day supply

• Drug N costs $3,200 for 30-day supply

Based on the information provided, which drug would you consider preferred therapy?

• Drug M

• Drug N

Drug P and Drug Q are both treatment options for metastatic lung cancer. Drug costs are virtually equivalent. Each drug had 2 different trials measuring progression-free survival (PFS). The results from the trials are provided below:

• For Drug P, the average PFS was 7.8 months and 8.3 months in Trials P1 and P2, respectively

• For Drug Q, the average PFS was 4.5 months and 24.2 months in Trials Q1 and Q2, respectively

Drug Q	Sample Size	PFS (months)
Pivotal Trial Q1	489	4.5
Pivotal Trial Q2	125	24.2

Drug P	Sample Size	PFS (months)
Pivotal Trial P1	315	7.8
Pivotal Trial P2	289	8.3

Based on the information provided, which drug would you consider preferred therapy?

• Drug P

• Drug Q

Zero-Risk Bias

Scenario 1: Drug A and Drug B are currently being studied for the indication of renal cell carcinoma. The 2 studies described below were structured very similarly, and drugs were compared to the same standard therapy regimen. Both Drugs A and B are administered orally once daily until disease progression or unacceptable toxicity occurs.

Drug A was compared to standard therapy in a randomized control trial (n=138, 1:1 randomization). Preliminary data showed that after a 6-month study period, no patients in Drug A group withdrew from therapy due to serious adverse events (SAEs), compared to 7 patients who withdrew due to SAEs under standard therapy.

In a separate randomized control trial, Drug B was compared to standard therapy (n=144, 1:1 randomization). After 6 months, 9 patients withdrew due to SAEs in Drug B group, while 24 patients withdrew due to SAEs under standard therapy.

Based on the information provided, which drug would be considered preferred therapy?

• Drug A

• Drug B

Scenario 2: A large safety study compared the adverse event (AE) rates of 3 chemotherapy agents with comparable efficacy: Drugs C, D, and E. A total of 246 participants (82 participants in each drug group) from multiple institutions around the U.S. were followed over the entire course of therapy. The results are shown below:

• Drug C had 6 cases of AE1 and 34 cases of AE2

• Drug D had 12 cases of AE1 and 42 cases of AE2

• Drug E had 0 cases of AE1 and 47 cases of AE2

Based on the information provided, which drug would be considered preferred therapy?

• Drug C

• Drug D

• Drug E

Delay Discounting

For the questions that follow, imagine that you work for a large MCO in which the pharmacies are wholly owned and operated by the MCO itself. A drug manufacturer wishes to contract with your organization and offers you either an immediate, direct discount off invoice, or a delayed rebate. Please treat each question as an independent scenario. Using only the information provided, choose the preferred proposal, A or B. Cost savings are listed in per-treated-member-per-month (PTMPM) costs for treatment of an unspecified disease.

Which proposal would you prefer?

• Proposal A includes a direct discount resulting in an immediate cost savings of $54 PTMPM

• Proposal B includes a delayed rebate resulting in a cost savings of $55 PTMPM after 117 days of implementation

NOTE: Original survey contained 27 scenarios total in the written format with cost savings and length of delay varied in each scenario. See Appendix B for the full list of values used in the 27 scenarios within the survey.

APPENDIX B. Reward Values for 27 Delay Discounting Scenarios

Immediate Reward Value	Delayed Reward Value	Magnitude of Delayed Reward	Length of Delay for Delayed Reward (Days)	Reward Difference (Delayed Minus Immediate)
$34	$35	Small	186	$1
$54	$55	Medium	117	$1
$78	$80	Large	162	$2
$28	$30	Small	179	$2
$47	$50	Medium	160	$3
$80	$85	Large	157	$5
$22	$25	Small	136	$3
$54	$60	Medium	111	$6
$67	$75	Large	119	$8
$25	$30	Small	80	$5
$49	$60	Medium	89	$11
$69	$85	Large	91	$16
$19	$25	Small	53	$6
$40	$55	Medium	62	$15
$55	$75	Large	61	$20
$24	$35	Small	29	$11
$34	$50	Medium	30	$16
$54	$80	Large	30	$26
$14	$25	Small	19	$11
$27	$50	Medium	21	$23
$41	$75	Large	20	$34
$15	$35	Small	13	$20
$25	$60	Medium	14	$35
$33	$80	Large	14	$47
$11	$30	Small	7	$19
$20	$55	Medium	7	$35
$31	$85	Large	7	$54