Incremental Cost-Effectiveness Ratios (ICERs) and Revised Metformin Cost-Effectiveness Conclusions in the Diabetes Prevention Program/ Diabetes Prevention Program Outcomes Study

Abstract

Introduction: Based on previously published US Diabetes Prevention Program (DPP) cost-effectiveness analyses (CEAs), metformin continues to be promoted as “cost-effective.” We reviewed a 10-year CEA to assess this. Treatment alternatives included placebo, branded metformin and individual lifestyle modification. Following the original CEA, we added group lifestyle as a modeled alternative. Methods: Original published data were taken as given and re-analyzed according to accepted principles for calculating incremental cost-effectiveness ratios (ICERs). With more than 2 treatments, these require attention to the rankings of interventions according to cost or effect prior to stipulating appropriate ICERs to calculate. Results: With appropriate ICER calculations, metformin was not cost-effective. Net Loss calculations indicated substantial costs/health losses to using metformin instead of the optimal lifestyle alternative in response to metformin having been confusingly labeled “cost-saving” in the original CEA. Conclusions: The original DPP CEA, subsequent analyses and citations of such analyses continue to conclude that both metformin and lifestyle modification are cost-effective in diabetes prevention. However, using metformin implies substantial costs and health losses compared to the cost-effective lifestyle modification. It may be that metformin has a role in cost-effective diabetes prevention, but this has yet to be shown based on DPP data.

“The fact that placebo is a dominated alternative reinforces further its inappropriateness for inclusion in any ICER calculations.”

Background

The Diabetes Prevention Program/ Diabetes Prevention Program Outcomes Study (DPP/DPPOS) was a randomized study of 3 interventions: placebo, intensive individual lifestyle modification (hereafter, “lifestyle”) and metformin. The study included over 3000 pre-diabetes patients over a 3-year period (DPP) followed by a 7-year follow-up (DPPOS). Placebo and metformin interventions also included a standard lifestyle behavioral intervention. ¹ These patients had “elevated fasting and post-load plasma glucose concentrations” at trial entry. The study was funded by multiple sources, including the producer of metformin. The DPP showed reductions in cumulative incidences of diabetes of 31% for metformin and 58% for individual lifestyle modification. ¹ These reductions were based on comparisons to a placebo alternative that included basic lifestyle modification advice. After the 7-year follow-up for the DPPOS, these reductions dropped to 18% and 34%. ²

It has become increasingly important to also examine the economics of treatments and preventive efforts. ³ Are their outcomes worth the costs incurred to achieve them? This is particularly important in disease areas like diabetes where large sections of the population are at risk for the disease and may benefit from prevention efforts. Utilizing cost-effective therapies can assist in getting the most from a health care budget—increasing economic efficiencies and ensuring that interventions represent value for money. To that end, the DPP Research Group published a cost-effectiveness analysis (CEA) based on the 10 years of the DPP/DPPOS. ⁴

The basic methods for CEAs are well-documented though they continue to evolve.^5-12 The authors used a standard CEA tool—the incremental cost-effectiveness ratio (ICER). In an ICER a difference in effectiveness (dE) between 2 treatments (e.g., a new treatment compared to an old treatment) is juxtaposed against the cost increment required to bring about that effectiveness change (dC). ¹³ This ICER (dE/dC) is compared to a stipulated societal willingness-to-pay for an incremental unit of effectiveness. An ICER in excess of the willingness-to-pay threshold implies that new intervention is not cost-effective. When the ICER is less than the willingness-to-pay, the new intervention is cost-effective.

The ICER can be deceptive in its apparent simplicity. When analyzing more than 2 treatments (as in the DPP/DPPOS), proper ICER calculation is more complicated. An essential question is which, of many the ICERs that could be calculated in a multiple treatment world, actually should be calculated? Not all ICERs are relevant to economic decision-making. Some may actually be counterproductive in forming conclusions. This process is not immediately intuitive. One must consider the relevant treatment alternatives as well as particular relationships between them (relative effectiveness and costs). Absent careful analysis, particularly when comparing all alternatives to a common one, seemingly valid, but, in fact, misleading results may be produced.^14,15

Based on its assessed ICERs, the authors of the DPP/DPPOS CEA concluded that “lifestyle was cost-effective and metformin was marginally cost-saving compared to placebo.” They further stated that “[w]hen a new treatment is more effective and less costly than usual care [cost-saving], it should be widely adopted and used.” ⁴ This emphasis in describing the proper policy reaction to a treatment’s being “cost-saving” appears to imply some superiority for metformin over lifestyle that received no such positive policy prescription.

This conclusion is similar to earlier DPP-based CEAs and is reflected in its citing literature. Earlier CEAs based on the DPP by some of the same authors had also indicated cost-effective metformin results (though not cost-savings).^16,17 References to the DPP/DPPOS CEA continue with a total of 358 references (60 since 2020). ¹⁸ References, supporting the use of metformin as cost-effective and/or to the purported cost-savings of metformin based on DPP/DPPOS data persist more recently, ¹⁹ including by one of the original CEA’s authors. ²⁰ These references suggest that these historical data and conclusions are still considered to be relevant. Importantly, not only is this CEA of current relevance, in its continuing to be cited, but we show here that its conclusions are incorrect. The original CEA’s age does not mean its errors are only of historical relevance. Our conclusion—and the continuing citations of the original CEA—imply (1) that there is an error of some consequence and (2) that it affects current thinking and decision-making and must be corrected so that these influences are corrected. We reanalyze the 10-year CEA, ⁴ accepting it as largely accurate in its data summary, but showing it to be inaccurate in interpretation. It incorrectly calculates many inappropriate ICERs and leads to inappropriate interpretation and conclusions. A similar re-analysis of the 3-year DPP CEA concluded that the original CEA ¹⁷ was also in error in claiming that both metformin and lifestyle were cost-effective. ²¹ We also suggest that the calculation of the measure of effectiveness used in the 10-year CEA—the Quality-Adjusted-Life-Year (QALY) is unconventional. We recalculate it based on conventional guidelines.^11,22 We then explore the implications of the altered method. Our extensions to the analysis also include a Probabilistic Sensitivity Analysis. This is now considered an essential element of credible CEAs, important in indicating the extent of decision uncertainty. Decision uncertainty is the probability of making an error associated with a CEA’s conclusions).^23,24 We illustrate that with Cost-Effectiveness Acceptability Curves. ⁶ We also supplement that with a calculation of Net Losses associated with implementing metformin instead of lifestyle. This shows, beyond the probability of making an error, the expected costs of such an error. ⁹

Methods

There are, in total, 3 versions of our analysis. Versions 1 and 2 use the original authors’ QALY calculation method. Version 1 is based on reported ICERs and reported differences between treatments in costs and QALYS in the main text of the original CEA (their absolute values were not reported). Version 2 uses the greater detail on costs and utilities (not their differences) provided in original authors’ supplemental data from the original CEA publication. Version 3 required the same input data as Version 2 but substitutes our altered QALY calculation method. Version 1 ICERs are used only to compare with Version 2 ICERs to assess whether our result from the use of the DPP supplemental data is in line with the results presented in the original CEA. Version 3 is suggested as the most appropriate analysis.

After establishing the level of correspondence between Versions 1 and 2, our primary goal was to assess whether our conclusions using Version 3 differ from the original authors’ based on Version 1 where both ICERs and QALYs were calculated differently. A secondary goal is in comparing appropriate ICERs in Version 3 and with the same ICERs from Version 2. In this way we can isolate the influence of our adjustment to QALYS had on conclusions vs those using the original QALYs. As the ICER calculation method will be identical in these versions, the differences will be solely due to the QALY differences.

In previous DPP CEAs, a group-based lifestyle treatment alternative was added in sensitivity analyses.^4,16 Group lifestyle had reduced costs but (by assumption) identical efficacy to individual lifestyle. We present results with and without the group option as its credibility may be questioned since it was not an actual alternative observed in the DPP/DPPOS.

We first describe the health care system perspective cost and effect measures and calculations used in the original CEA and our modifications. The original CEA ⁴ was later revised online and an Erratum published. ²⁵ General conclusions did not change. We use the revision’s supplementary yearly data for our Version 2 and 3 analyses. Direct medical costs (year 2010 dollars) for the health system perspective included “inside” and “outside” program costs. ¹⁷ Inside costs included intervention (e.g., counseling, drug) costs. Outside costs related to treatment performance—effectiveness and safety (e.g., costs of hospitalization). Equal effectiveness (both lifestyle alternatives) implied equal outside program costs.

A treatment’s total costs were built from yearly means (for inside costs) and yearly means for outside costs. The original QALYs were similarly calculated from yearly reported utilities. The exact method for estimating these was not specified in the original CEA. Its supplemental data provided data for outside costs and utilities conditional on diabetes status as well as the number of patients with and without diabetes each year for each treatment. We assumed that these data were used to estimate weighted averages for outside costs and for QALYs reported in the original CEA main text. Our first task was to attempt to reproduce the original CEA results from the supplemental data. If that could be done satisfactorily, we could credibly proceed to modify the QALY and ICER calculations for Version 3 and compare results.

Total outside costs were estimated from yearly mean outside costs conditional on diabetes status. These were weighted by diabetes status proportions in a given year. An example shows the procedure for calculating yearly outside costs below. For each treatment the supplemental data provided mean yearly costs for year y conditional on diabetes status. From the same source, we could calculate the yearly probability of being in a diabetes state (P(D)_y) vs staying in the non-diabetes state that all patients started in (1- P(D)_y). The yearly average outside cost for a treatment is the weighted average of the outside costs for those with diabetes (OC_y^D) and the outside cost for those who have not moved to the diabetes state (OC_y^ND):

$${\text{OC}}_{\mathrm{y}}=\left[1-{\mathrm{P}\left(D\right)}_{\mathrm{y}}\right]*{\text{OC}}_{\mathrm{y}}^{\text{ND}}+{\mathrm{P}\left(D\right)}_{\mathrm{y}}*{\text{OC}}_{\mathrm{y}}^{\mathrm{D}}$$ (1)

This equation is used for each treatment’s OC_y. Yearly total costs were the sum of these yearly OC_y values and the yearly inside cost (IC_y) values.

QALYs in the original CEA were based on mean year-end utility assessments using the Quality of Well Being instrument—shown by year and treatment in the CEA’s supplemental data. In the original QALYs, the end of year utility values were used as QALYS for the year. These were summed to obtain 10-year cumulative QALYs in the original CEA.

We altered that QALY calculation to adhere to more conventional methods, long part of the economic evaluation literature.^{11,13,22,26,27} We calculated an average of 2 utility values for each year so that QALY_y for the year y was the average of beginning and end of year utilities: U_y-1 and U_y. Each of these yearly reported utilities (U_y), like outside costs, were also a weighted average of patients with and without diabetes. The same procedure as with outside costs (an analog to equation (1)) was applied to utility data that was also reported conditional on diabetes status. The same probabilities were used as above to estimate the weighted average utility, U_y, for each year and each treatment (equation (2)). Those values were used to calculate altered QALYs (see “Altered QALY calculations” and “Deterministic Calculations” in Appendix for more details).

$${\mathrm{U}}_{\mathrm{y}}=\left[1-{\mathrm{P}\left(\mathrm{D}\right)}_{\mathrm{y}}\right]*{\mathrm{U}}_{\mathrm{y}}^{\text{ND}}+{\mathrm{P}\left(\mathrm{D}\right)}_{\mathrm{y}}*{\mathrm{U}}_{\mathrm{y}}^{\mathrm{D}}$$ (2)

For comparison with the original CEA that used ICERs, we use ICERs as our primary method. The ICER between 2 treatments, A and B is

$${\text{ICER}}_{\text{BA}}=\left({\mathrm{C}}_{\mathrm{B}}-{\mathrm{C}}_{\mathrm{A}}\right)/\left({\mathrm{E}}_{\mathrm{B}}-{\mathrm{E}}_{\mathrm{A}}\right)$$ (3)

In a simple 2 alternative world, if this ICER is less than the stipulated willingness-to-pay value, then B is the cost-effective choice; otherwise A is. In our more complex situation not all bilateral ICERs should be calculated. Our ICERs were calculated using a conventional systematic process as outlined in Glick ²⁸ (see Appendix, “Systematic Process”). Briefly, the process ranks treatments by effectiveness level and assesses each alternative vs the others, eliminating technically inefficient alternatives (e.g., less effective and more costly than another). Such “dominated” alternatives cannot be cost-effective choices. After removing dominated alternatives, ICERs between remaining contiguous alternatives are calculated and “weakly dominated” alternatives are eliminated. These alternatives also cannot be cost-effective choices and are eliminated. After this, ICERs between contiguous alternatives are again calculated and compared with the stipulated willingness-to-pay threshold to determine the cost-effective treatment. This is the treatment with the highest effectiveness whose ICER is still less than the willingness-to-pay threshold value.

In addition to calculating ICERs correctly, we will present an equivalent alternative that is clearer and less prone to implementation error—Net Monetary Benefits (NMB).^29,30 This uses the same input data as ICER calculations, but for each calculation a willingness-to-pay value must be stipulated (see Appendix, “NMBs”).

The Net Monetary Benefit for a treatment for a given willingness-to-pay value (WTP) is

$$\text{NMB}=\text{WTP}*\text{QALYs}-\text{TC},$$ (4)

where QALYs and TC here are totals over the 10-year period. The WTP value acts as an exchange rate between QALYs and an equivalent monetary valuation of the QALYs. It converts QALYs to a monetary measure based on the stipulated value of a QALY (the WTP for a QALY).

The decision-making criterion for Net Monetary Benefits is to choose the treatment with the maximum value. It is equivalent in its conclusions to those based on ICER criteria. This is because the Net Monetary Benefit decision criteria is derivable from those ICER criteria (a treatment is cost-effective if its ICER is below the threshold defined by that willingness-to-pay^29,30—see Appendix, “NMBs”). In our results, we will present both correct ICERs and Net Monetary Benefits to indicate the relative simplicity of the latter approach and the relative certitude of correct interpretation using the NMB vs the ICER approach that has shown itself to be error-prone. ³¹

For sensitivity analyses, we conduct only a Probabilistic Sensitivity Analysis (PSA). A PSA simultaneously and randomly varies all CEA model parameter values (e.g., inside program cost, outside program cost, etc.). From those values it calculates a cost-effectiveness result. It repeats this process, many times (e.g., 1000-2000), randomly choosing parameter values each time for a new cost-effectiveness calculation. This random variation may imply no difference in the cost-effective treatment among the repeated analyses. If that is the case then there is said to be no decision uncertainty—the cost-effective choice is clear. When the cost-effective treatment changes frequently within the PSA, there is substantial decision uncertainty. The measure of the uncertainty is the frequency among these iterations that each treatment is cost-effective at a given willingness-to-pay threshold and this is summarized in a Cost-Effectiveness Acceptability Curve. PSA details are in the Appendix (“PSA” and Appendix tables). No PSA was done by the original authors.

We use the results from the PSA in 4 ways. First, the mean values for effectiveness and cost for each treatment over all PSA iterations are used as inputs to calculate appropriate ICERs as described previously. Second, the average Net Monetary Benefit based on those same mean inputs determines the optimal treatment (that with the maximum). This calculation is generally done for a varied number of willingness-to-pay values. ²³ The above ICERs and the Net Monetary Benefits will generally correspond to those for the deterministic (baseline) analysis, though the PSA-based conclusions are the more reliable ones. ²³

Third, we can indicate the probability that a particular treatment is optimal. This is simply the proportion of PSA iterations at that willingness-to-pay value for which the treatment was optimal. ⁶ These probabilities for each treatment, indicating decision uncertainty, are displayed in a Cost-Effectiveness Acceptability Curve^32,33 that indicates the confidence in the treatment choice for any willingness-to-pay level (the probability, P(CE)), of being cost-effective.

Lastly, while the probability of making an incorrect decision for each treatment (1-P(CE)) is of clear interest, of even greater value is assessing the consequences of any incorrect decision weighted by those probabilities. This is measured by the expected (i.e., average over all iterations) Net Losses associated with using a treatment. Even for the optimal treatment, Net Loss will be positive unless for every PSA iteration, the overall cost-effective treatment is also cost-effective for every iteration of the PSA. That would indicate no decision uncertainty. The optimal treatment for any willingness-to-pay value will be the treatment with the minimum average Net Loss. Overall, this information is captured by a Net Loss curve for each treatment. ⁹

The difference in Net Loss values by treatment is our primary interest. It indicates the monetized losses from using one alternative instead of another and is equal to the difference in Net Monetary Benefits between the treatments. Of greatest interest is the difference in Net Loss from using any alternative compared to the cost-effective treatment. This indicates the importance of any implementation error, for example, using a non-cost-effective treatment instead of the cost-effective one. This is initially calculated at a per patient level, but it may be (and should be for a complete picture of potential losses) scaled up to the population level for which it is thought appropriate to assess losses. For further details see the Appendix (“Net Losses”). ⁹

Results

Version 1 and 2 results (Table 1) showed a close correspondence, so that we can be relatively confident that our use of the original CEA’s supplemental data in place of its main text-reported incremental data is appropriate.

Table 1.

V1 (Original reported Deterministic CEA ICERs in Revised Publication Based on Reported Incremental Cost and QALYs) Compared to V2 (Original CEA ICERs calculated From Supplemental Data in Original CEA).^1,2

Version of Analysis	Individual Lifestyle vs Placebo	Metformin vs Placebo	Individual Lifestyle vs Metformin	Group Lifestyle vs Placebo
V1 (published)	$10,759	“Cost-saving” a	$13,469	$528
V2 (calculated)	$10,777	$ (1812)	$17,367	$307

CEA is cost-effectiveness analysis; ICER is incremental cost-effectiveness ratio; V1 is Version 1 (original QALYs), undiscounted reported from original CEA publication erratum; V2 is Version 2 (original QALYs); (calculated from supplemental data in original CEA publication erratum.

^aICER value not specified in V2, but reported “cost saving” is consistent with the negative ICER in V2.

Total costs and QALYs for the 4 treatment alternatives are reported in Table 2. The altered yearly QALYs were different from those reported in the published CEA and higher in total. These were used directly in analyses Versions 2 and 3.

Table 2.

Undiscounted Yearly Total Costs and QALYs (Deterministic, Original and Altered) by Intervention.^1,2

	Individual Lifestyle			Metformin
Year	Cost	Original QALY	Altered QALY	Cost	Original QALY	Altered QALY
1	$3658	0.703	0.706	$2554	0.688	0.699
2	$3201	0.695	0.698	$2616	0.683	0.687
3	$3116	0.692	0.692	$2723	0.683	0.683
4	$2653	0.688	0.698	$2813	0.68	0.681
5	$2423	0.682	0.684	$2563	0.678	0.679
6	$2882	0.685	0.683	$3252	0.675	0.676
7	$2943	0.686	0.685	$3183	0.68	0.677
8	$3675	0.682	0.684	$3229	0.675	0.677
9	$3213	0.687	0.685	$3322	0.671	0.673
10	$3618	0.685	0.686	$3413	0.675	0.673
Total	$31,382	6.887	6.894	$29,665	6.788	6.805
	Placebo			Group lifestyle
Year	Cost	Original QALY	Altered QALY	Cost	Original QALY	Altered QALY
1	$2091	0.689	0.699	$2730	0.703	0.706
2	$2396	0.678	0.683	$2877	0.695	0.698
3	$2778	0.674	0.676	$2791	0.692	0.692
4	$3088	0.674	0.674	$2653	0.688	0.698
5	$2760	0.673	0.673	$2423	0.682	0.684
6	$3007	0.671	0.672	$2882	0.685	0.683
7	$3049	0.672	0.671	$2943	0.686	0.685
8	$3324	0.671	0.671	$3675	0.682	0.684
9	$3562	0.667	0.669	$3213	0.687	0.685
10	$3703	0.668	0.667	$3618	0.685	0.686
Total	$29,758	6.737	6.755	$29,805	6.887	6.894

QALYs are quality-adjusted life years; Original vs Altered QALYs described in text; Costs are Total Costs and are the sum of inside and outside program costs.

Table 3 shows our primary altered QALYs deterministic analysis, Version 3, with an ordered ranking of the alternatives by effectiveness to assist in determining relevant ICERs to calculate (see Appendix, “Systematic Process”). ²⁸

Table 3.

Interventions Ordered by Increasing Effectiveness (Altered QALYs; Deterministic Analysis; Health Care Perspective, Version 3).

			Group Lifestyle Included		Group Lifestyle Excluded
(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)
Intervention	Total QALYs	Total costs	ICER vs NMEA	NMEA	ICER vs NMEA	NMEA	NMB (WTP = $50,000)
Placebo	6.756	$29,759	DOM by Metformin	NA	DOM by Metformin	NA	$308,030
Metformin	6.805	$29,655	NA	NA	NA	NA	$310,594
Individual lifestyle	6.894	$31,382	DOM by Group Lifestyle	NA	$19,433	Metformin	$313,295
Group lifestyle	6.894	$29,805	$1585	Metformin			$314,873

DOM is dominated; ICER is incremental cost-effectiveness ratio; NA is not applicable; NMB is Net Monetary Benefit; NMEA is next most effective intervention, not dominated; QALYs are quality-adjusted life years; WTP is willingness-to-pay threshold.

With group lifestyle included among the alternatives, columns 4 and 5 show that placebo and individual lifestyle are strongly dominated (by metformin and group lifestyle, respectively). They have less QALYs and greater costs than their dominating alternatives.^11,23 Once they are eliminated as inefficient, the only relevant ICER is that between metformin and group, with group being cost-effective (ICER = $1585 < assumed willingness-to-pay of $50,000/QALY). When excluding group (columns 6 and 7), individual lifestyle is cost-effective as, again, metformin dominates placebo, so that only one ICER is relevant (individual vs metformin, ICER = $19,433 < $50,000/QALY).

The Net Monetary Benefit conclusion is simplest to convey, showing that group lifestyle (or, in group’s absence, individual lifestyle) has the maximum value for this willingness-to-pay, showing lifestyle as cost-effective (Table 3, column 8). Net Monetary Benefit and correct ICER calculations provide identical conclusions (relevant ICERs—either lifestyle intervention vs metformin—are all below $50,000/QALY).

Table 4 presents analogous data and results for Version 2 (original QALYs). For both it and Version 3, it is clear that the optimal treatment is group lifestyle when that is an included option, or individual lifestyle when the group alternative is excluded. The QALY revision did not materially affect the results in this analysis. Group (or individual) continued to be optimal with only modest changes in relevant ICERs and Net Monetary Benefits.

Table 4.

Interventions Ordered by Increasing Effectiveness (Original QALYs; Deterministic Analysis; Health Care Perspective, Version 2).^1,2

			Group Lifestyle Included		Group Lifestyle Excluded
(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)
Intervention	Total QALYs	Total costs	ICER vs NMEA	NMEA	ICER vs NMEA	NMEA	NMB (WTP = $50,000)
Placebo	6.74	$29,759	DOM by Metformin	NA	DOM by Metformin	NA	$307,042
Metformin	6.79	$29,655	NA	NA	NA	NA	$309,723
Individual lifestyle	6.89	$31,382	DOM by Group Lifestyle	NA	$17,367	Metformin	$312,950
Group lifestyle	6.89	$29,805	$1417	Metformin			$314,527

DOM is dominated; ICER is incremental cost-effectiveness ratio; NA is not applicable; NMEA is next most effective intervention, not dominated; QALYs are quality-adjusted life years; V2 is Version 2 (original QALYs), undiscounted analysis; see Table A2 for input data.

Figure 1 shows these results graphically in a cost-effectiveness plane with points representing Version 2 and 3 analyses. The same conclusions as above are seen. For both Versions 2 and 3, metformin dominated placebo and group dominates individual. Metformin would be on the cost-effectiveness frontier (not shown) with group (or with individual, if group is excluded). That frontier connects technically efficient (undominated) points. The ICER between the relevant lifestyle and metformin, along with the assumed willingness-to-pay value, determines the cost-effective treatment. Note that placebo, being dominated, would not be on the frontier and ICERs with placebo should not be calculated.

Figure 1.

Cost-effectiveness plane, deterministic, undiscounted, versions 2 and 3.

The above primary Version 3 deterministic results are consistent with the results of the Probabilistic Sensitivity Analysis. The optimal treatment there is also group and the decision uncertainty around that choice is low at any commonly used willingness-to-pay values ($50,000-$150,000). The only real decision uncertainty at conventional willingness-to-pay values is between the 2 lifestyle alternatives since the probabilities of either of the other alternatives being cost-effective are minimal. The P(CE) values for the lifestyle alternatives approach each other at high willingness-to-pay values because the influence of those increasingly high values (multiplied by their assumed equal QALYs) overwhelms the relatively small cost difference in their Net Monetary Benefit calculations in equation (4) (Figure 2). At high willingness-to-pay values, the 2 alternatives become essentially equivalent in NMBs so the Probabilistic Sensitivity Analysis treats them as nearly interchangeable.

Figure 2.

Cost-Effectiveness Acceptability Curves, Version 3, Undiscounted, Altered QALYs, Based on PSA Results. QALY is Quality-Adjusted Life Years; P(CE) is probability of being cost-effective; PSA is probabilistic sensitivity analysis; WTP is the willingness-to-pay threshold value.

Figure 3 shows Net Loss functions at a per patient level for Version 3. At a willingness-to-pay of $50,000, the difference in Net Loss from choosing metformin over the cost-effective group is an additional $4239 per patient; the choice of metformin over individual leads to a loss of $2639 per patient. Estimates for these are also possible for the deterministic analysis (the difference in Net Monetary Benefit values—see Table 2) and were $4279 and $2701, respectively. At higher willingness-to-pay values this difference in loss increases as indicated by the increasing vertical distances between these Net Loss curves.

Figure 3.

Net Loss Functions, Version 3, Undiscounted, Altered QALYs Based on PSA Results. QALYs is Quality-Adjusted Life Years; PSA is Probabilistic Sensitivity Analysis; WTP is willingness-to-pay threshold value.

Per patient Net Loss captures the essence of implementing any suboptimal decision, but the true extent of potential loss can only be represented by scaling the loss up to a level of implementation (e.g., a practice, state, national or other population level). The original 2002 DPP clinical paper ¹ suggested at that time that there were 10 million patients in the US that “resembled” those in the DPP study and would benefit from optimized diabetes preventative treatment. This is conservative in terms of impact that the DPP may have had on treatment decisions. Over time, the relevant losses accumulate and the population potentially influenced becomes even larger. Furthermore, the DPP enrolled high-risk individuals. Others at lower risk may have been influenced by medical decisions relying on the results from the DPP (CDC estimated an 84 million pre-diabetes population in 2017). ³⁴ Those outside the US may also have been influenced.^35,36 However, as an illustration, we, conservatively, use the aforementioned 10 million patients for 1 year as a measure of influence for an estimate for scaling up per patient losses. The per patient Net Loss would be multiplied by that population factor. This implies that an inappropriate use of metformin (e.g., based on its having been concluded to be cost-saving) for that entire population would create Net Losses of more than $40 billion in 1 year. Readers may use any scaling factor they like to indicate other size population losses from an individual medical practice to a community or state level, including extending the relevant time beyond our 1-year illustration or to different fractions of patients in the population who may have been given metformin instead of the cost-effective group alternative.

Discussion

The original CEA concluded that lifestyle was cost-effective and that metformin was cost-saving vs placebo. It also stated that the cost-saving nature of metformin meant that it should be “widely adopted and used,” a policy prescription not advocated for the cost-effective lifestyle intervention. However, any implied notion of superiority for metformin has been shown in our re-analysis to not be appropriate. Lifestyle is the cost-effective treatment based on these data (with either QALY calculation method) and it is the lifestyle vs metformin ICERs that make this so, not ICERs of lifestyle vs placebo. The QALY contribution that either lifestyle alternative offers over what metformin offers is worth the added cost incurred—those costs are “worth it”—lifestyle is cost-effective.

Figure 1 showed that metformin is, as claimed, cost-saving—vs placebo. Metformin dominates placebo. Importantly, this is only relevant in its consequential eliminating of placebo from contention as a potentially cost-effective treatment. Placebo is not on the efficiency frontier, so becomes an irrelevant comparison in further determining cost-effective therapy. Because placebo is eliminated, the only relevant ICER that should be calculated is metformin vs lifestyle, indicating the latter as cost-effective. Thus, far from metformin’s cost-saving nature implying it should be “widely adopted and used,” our analysis showed that there would be substantial added costs and lost benefits if metformin were to be used in place of the cost-effective lifestyle alternatives.

In trying to support diabetes prevention, authors of the original CEA may have sought to add to the positive story that there were 2 effective treatments superior to a simple lifestyle advice that accompanied placebo in the DPP/DPPOS.^1,2 That is, not only were these treatments more effective than standard lifestyle advice, but they seemed to both be economically attractive compared to it as well. While this interpretation, based on comparing both those treatments to a common (here, placebo) alternative, may appear valid, it is contradictory to long-standing accepted practice in health economics.^{11,13,15,23,26-28,37} In fact such a practice is called out specifically as a common error to be avoided. ¹⁴ An earlier CEA by the same authors ¹⁷ made the same error; a later publication identified their approach as incorrect. ²¹ In the current work, the fact that placebo is a dominated alternative reinforces further its inappropriateness for inclusion in any ICER calculations.

While the relevant ICER for our conclusion (lifestyle vs metformin) is reported in the original CEA, its central importance seems unrecognized. In fact, this is the only ICER that is relevant and it indicates that lifestyle is the only cost-effective alternative. This brief indication of lifestyle’s possibly higher value carries little weight in the original paper and is not mentioned in its abstract.

The original CEA’s authors presented too many (24) ICERs. They include 4 sets of ICERs (6 ICERs in each set, corresponding to discounted and undiscounted results for each of 3 different perspectives). There are no substantive differences between each group of 6 in a set. Most importantly, 3 of the 4 sets of ICERs are calculated vs placebo and are therefore, as indicated above, ²⁸ in this case are not helpful in choosing an economically efficient treatment. Thus 18 of the 24 ICERs presented in the original CEA are not relevant in informing a cost-effectiveness conclusion. They can only mislead or confuse authors and readers (e.g., by implementing the apparent policy prescription that cost-saving interventions should be “widely adopted and used”). In the original CEA, the only set of ICERs that is relevant for cost-effectiveness (those comparing lifestyle vs metformin) is, apart from 1 key sentence, ignored in the text. An interesting question is whether it has been ignored by readers.

Have readers of the original CEA interpreted it according to the lone sentence suggesting a possible awareness of lifestyle superiority to metformin or have they taken away the messaging that predominates in the paper? We examined subsequent statements by the DPP CEA authors themselves. A 2015 summary of the original CEA ³⁸ by one of its authors may have even further confused interpretation by omitting any mention of the cost-effectiveness of lifestyle vs metformin, the key result. Importantly, the 1 column of relevant ICERs from the original article (lifestyle vs metformin ICERs) was omitted from the otherwise identical table in the 2015 summary which reported only the 18 irrelevant ICERs. Cost-savings conclusions for metformin, with that implied superiority compared to placebo, continued along with the same suggestion that cost-saving treatments should be “widely adopted and used.”

Interestingly, the important qualifying language of metformin’s cost-savings nature (“compared to placebo”) may not always be carefully considered. This seemingly minor difference may cause confusion and misinterpretation. Indeed, a 2019 publication, ³⁹ left the “compared to placebo” out of its metformin summary of the original CEA. This omission moves the metformin cost-savings statement in the original CEA from being true, but largely unimportant for metformin’s attractiveness, to being technically inaccurate. This follows because the original statement only applied to the placebo comparison. The later statement without the “compared to placebo” qualifier can be read (inappropriately) to mean that the cost-savings applies to all alternatives. If accurate, such a cost-savings claim would be quite meaningful. When it is not accurate, it may be confusing to state conclusions without the qualifying language. We note that supporting text for guidelines for the prevention or delay of diabetes for metformin have, in both 2012 and the latest update for 2025, stated that “metformin may be cost-saving over a 10-year period.” Importantly, the guideline developers left out the important “compared to placebo” qualifier that was in the original CEA, perhaps leading to misinterpretation of the 10-year CEA results.^40,41

Other readers not associated with the DPP CEA have also misinterpreted the conclusions from that publication. An earlier search of citing literature in 2021, taking the 50 most recent articles that cited the original CEA, showed minimal evidence that the correct conclusion was understood. None of the citing authors indicated a clear understanding of the appropriate conclusions of the original CEA and none indicated awareness of the singular importance of the original authors’ 1 statement that lifestyle was cost-effective against metformin. At minimum, 73% of the content of relevant citations were assessed as “incorrect”; 27% as “incomplete.” (Appendix, “50 Most Recent Citations”).

Improper ICER calculation and/or interpretation is not unusual. ⁴² Because of various challenges in using and interpreting ICERs, we suggest the Net Monetary Benefit method as superior.^29,30 Using proper ICER calculation, Net Monetary Benefits and/or a graphical depiction of the results (Figures 1–3), all used here, could have easily indicated the errors in the original CEA, but none of these methods were employed in the original work. There is no confusion with the Net Monetary Benefit calculation as to which treatment is optimal—either group or individual lifestyle. There is no possibility of comparing to the wrong treatment as there is with an ICER. There is no confusion as to which comparator is relevant and no misinterpretation is possible. Net Loss curves support this conclusion and Cost-Effectiveness Acceptability Curves also hint at it (while optimality cannot accurately be inferred from them, they often give strong hints).^6,32 The economic evaluation field provides plenty of tools that may be used to confirm general conclusions based on any one of them. We advocate for using this multiplicity of tools, each indicating something slightly different but all, if appropriately used, are consistent as to the designation of the cost-effective treatment.

Conclusion

Metformin has been shown generally to be effective in diabetes prevention in the DPP/DPPOS, though less so than lifestyle modification. ¹ A 2012 DPP/DPPOS-based CEA indicated that both metformin and lifestyle were cost-effective in diabetes prevention. However, that work based its conclusions on ICERs for both products calculated vs a common alternative—here, placebo. The economics literature has specifically called out such comparisons to be inappropriate and potentially misleading. Appropriate ICER calculation in our work has shown that metformin is not cost-effective in the DPP/DPPOS trial. Importantly, we have shown that if metformin has been used in place of a more effective and cost-effective lifestyle alternative due to an unclear presentation of the original CEA results, then there will have been significant losses in cost and in health terms.

An important question is what message readers may have taken from the original CEA. We showed evidence that readers still find the DPP/DPPOS results relevant with more than 50 citations in recent years and that they almost universally took away the incorrect conclusion from the original CEA. Even the original CEA authors ignored their key correct, but not prominent, statement that lifestyle was cost-effective vs metformin in their later summaries of the results.^38,39

Supporting use of metformin in diabetes prevention based on the DPP/DPPOS must appeal elsewhere for justification (e.g., missing data influences, errors in data collection, heterogeneity of treatment effect, updated costing, other data sources, etc.). It seems plausible and it has been found^39,43 that certain patients may perform better than average on metformin. Since at least 2012, including the most recent guidelines for prevention or delay of diabetes, recommendations have included metformin use especially in high-risk individuals.^40,41 However, whether the additional performance in such patients translates to metformin being cost-effective for them has not been shown. We leave it to diabetes experts to determine the optimal use of lifestyle and metformin treatments for diabetes prevention. Economics is intended to be, not determinative, but an aid to decision-making that may appropriately incorporate other factors. We suggest only that CEA results be presented appropriately and clearly so that they may actually aid that decision-making and not potentially confuse it.

ORCID iD

Brian E. Rittenhouse https://orcid.org/0000-0002-9928-2583