The Imaginary Worlds of Cure Proportion Modeling: Survivorship and Reference Case Pricing of Transformative Therapies

Abstract

On August 6, 2019 the Institute for Clinical and Economic Review (ICER) released a set of proposed adaptations to its value assessment framework methods where the intervention under review was considered a ‘single or short-term transformativetherapy’ (SST). These adaptations are intended to ‘complement and build upon the upcoming update to the overall ICER assessment framework…’. The purpose of this commentary is to review the proposed cure proportion modeling reference case framework for assessing the value of SSTs together with ICER’s ‘recommendations for fair value-based pricing …’. Following previous commentaries on the ICER value assessment framework, the question raised is whether the proposed cure proportion modeling standards meet those of normal science: is the modeling proposed capable of generating value claims for the intervention that are credible, evaluable and replicable? The proposed standards for transformative therapies do not change the underlying commitment to reference case modeling. At the same time, the cure proportion modeling proposed adaptations have to be seen in the context of the concerns expressed by ICER that their reference case model can be used to justify substantial one-off SST pricing. This follows from the ICER incremental cost per QALY willingness to pay thresholds where the SST QALY gains are sufficient, at even a $50,000 QALY cut-off, to support SST pricing in the millions of dollars. ICER has two options: (i) abandon the imaginary reference case methodology, which is the ICER core business model and would represent an ironic reversal, or (ii) attempt to bolt-on adaptations, possibly incorporating revised survivorship profiles using cure proportion modeling, that supports a modified imaginary reference case ‘rescue’ model for SSTs designed specifically to generate pricing recommendations that may be considered affordable.

Introduction

On August 6, 2019 the Institute for Clinical and Economic Review (ICER) released a proposed set of standards for value assessment methods for short-term or transformative therapies (SSTs) together with a technical brief on value assessment for potential cures ^1,2. These standards are to be seen as adaptations to the ICER value assessment framework. ICER posted these proposed standards with a one-month public comment period. ICER would then ‘reflect’ on comments received, seek further feedback from stakeholders with its final version of the standards to be posted by end-2019.

The purpose of this commentary is to review and reply to these proposed standards for SST value assessment. Publishing in the public domain is seen as the most effective way of ensuring both transparency in the review process as well as making the review available to a wider audience within the US health care system. Given the role ICER has taken upon itself as the arbiter of value assessments and pricing recommendation in the US, a public debate is critical as there are substantial and fatal objections to the ICER value assessment methodology. Putting these concerns in the public domain follows upon previous commentaries in INNOVATIONS in Pharmacy on ICER evidence reports and associated technical assessments. These have raised the question of whether the ICER modeled reference case meets the standards of normal science ^{3,4,5,6,7,8,9,10}. If, as argued below, the ICER reference case methodology, which applies across the board and not only to SSTs, is judged not to meet the standards of normal science then this should be made clear to health system decision makers and the wider public to include pharmaceutical manufacturers.

A particular concern is to review how ICER might propose to respond to the fact that, in the case of SSTs where a one-off cure is possible, the reference case model is consistent with pricing for SSTs which ICER and others might consider exorbitant. This follows, within the reference case framework, from the substantial QALY gains attributable to SSTs. If ICER is to continue to subscribe to the reference case methodology meme, which after all is its core business model, then adaptations such as core proportion modeling might be seen as a framework for addressing what ICER sees as the potential for exorbitant pricing with manufacturers assumed to be building a surplus (i.e., a rent component) into the price of the SST.

Meeting the Standards of Normal Science

The requirement for testable hypotheses in the evaluation and provisional acceptance of claims made for products and devices is unexceptional. Since the 17^th century it has been accepted that if a research agenda is to advance, if there is to be an accretion of knowledge, there has to be a process of discovering new facts. Indeed, as early as the 16^th century Leonardo da Vinci (1452 – 1519) in notes that appeared posthumously in 1540 for his Treatise on Painting (published in 1641) clearly anticipated the standards for the scientific method which were widely embraced a century later in rejecting thought experiments that fail the test of experience. By the 1660s, the scientific method, following the seminal contributions of Bacon, Galileo, Huygens and Boyle, had been clearly articulated by associations such as the Academia del Cimento in Florence (1657) and the Royal Society in England (founded 1660; Royal Charter 1662) with their respective mottos Provando e Riprovando (prove and again prove) and nullius in verba (take no man’s word for it)¹¹.

By the early 20^th century standards for empirical assessment were put on a sound methodological basis by Popper (Sir Karl Popper 1902-1994) in his advocacy of a process of ‘conjecture and refutation ^12,13. Hypotheses or claims must be capable of falsification; indeed they should be framed in such a way that makes falsification likely. Life becomes more interesting if claims are falsified because this forces us to reconsider our models and the assumptions built into those models. This leads, then, to the obvious point that claims or models should not be judged on the realism or reasonableness of assumptions or on whether the model ‘represents’ for a public advocacy research group such as ICER their perception of a future, yet unknown, reality.

Although Popper’s view on what demarcates science (e.g., natural selection) from pseudoscience (e.g., intelligent design) is now seen an oversimplification involving more than just the criteria of falsification, the demarcation problem remains ¹⁴. Certainly, there are different ways of doing science but what all scientific inquiry has in common is the ‘construction of empirically verifiable theories and hypotheses’. Empirical testability is ‘one major characteristic distinguishing science from pseudoscience’; theories must be tested against data. Indeed, paradoxically, while the development of pharmaceutical products and the evidence standards required by the Food and Drug Administration (FDA) for product evaluation and marketing approval is driven by adherence to the scientific method, once a product is launched and claims made for cost-effectiveness and, in the case of ICER, pricing and access recommendations, the scientific method is put to one side. Pseudoscience succeeds science.

The rejection of a research program that meets the standards of normal science by groups such as ICER is best exemplified by the latest version of the Canadian health technology guidelines where it is stated: Economic evaluations are designed to inform decisions. As such they are distinct from conventional research activities, which are designed to test hypotheses ¹⁵. While this position puts modeled health technology assessment in the category of pseudoscience, it is also what may be described as a relativist position. Rather than subscribing to the position that the standards of normal science are the only standards to apply in health care decisions and value claims, the relativist believes that all perspectives are equally valid. Health care decisions are to be understood sociologically. No one body of evidence is superior to another. Results of a lifetime modeled simulation are on an equal basis with those of a pivotal Phase 3 randomized clinical trial. For the relativist, the success of a scientific research program, in this case one built on hypothetical models and simulations, rests not on its ability to generate new knowledge but on its ability to mobilize the support of the community. Basing decisions on models and simulations underpins the consensus view that evidence is constructed, never discovered. Instead of coming to grips with reality, science is about rhetoric, persuasion and authority¹¹. Truth is consensus.

Models and Assumptions

It is accepted that knowledge is provisional and permanently so. This stems from the obvious point that we can at no stage prove that what we ‘know’ is true. Attempting to believe or justify our belief in a theory is logically impossible. What we can do, by empirical assessment, is to try and demonstrate our preference for one theory over another (and apply it to the best of our knowledge).

Constructing imaginary worlds which were never intended to generate potentially falsifiable claims cannot, therefore, be defended by an appeal to the ‘truth’ of their assumptions. If a health technology assessment claim is built upon a series of assumptions, a reasonable question is to ask what is the status of the various assumptions? Are they to be viewed as ‘reasonable or ‘realistic’ metrics for an unknown future reality? Have they been selected from the literature because they seem appropriate? Is there a belief that the fact that they are based, where feasible, on an empirical study validates the choice of assumption? For example, if the model is intended to incorporate utilities that have been reported in one or two studies (usually as few as that) for progression and time spent in the stages of a disease, then there is an immediate methodological issue. To claim that an assumption is valid is to revisit Hume’s problem (David Hume 1711-1776): induction – an appeal to facts to support a scientific statement. Unfortunately, as Hume pointed out, no number of singular observations can logically entail an unrestricted general statement. Certainly, there may be comfort in reporting that ‘so far’ the claim that all swans are white has not been contradicted (until that vacation in Western Australia) so that one fully expects the next swan to be white. But as Hume also pointed out, this is a fact of psychology and does not entail any general statement. From a utility perspective, the fact that one hundred papers have agreed (within limited bounds) generic utilities from the same instrument for a target population in a disease state stage is immaterial. We cannot secure this assumption: it cannot be ‘ established by logical argument, since from the fact that all past futures have resembled past pasts, it does notfollow that all future futures will resemble future pasts’¹⁶.

Claims, for the relevance of a constructed imaginary world built on the assumption that the model elements have been validated by observation is simply nonsensical.

Despite ICER’s continued embrace, logical positivism is dead. It died some 80 years ago. All knowledge is provisional. Poppers contribution was to make clear that Hume’s problem with induction can be resolved. We cannot prove the truth of a theory, or justify our belief in a theory or attendant assumptions, since this is to attempt the logically impossible. We can only justify our preference for a theory by continued evaluation and replication of claims. Constructing imaginary worlds, even if the justification is that they are ‘for information’ is, to use Bentham’s (Jeremy Bentham (1748-1832) memorable phrase ’nonsense on stilts’. If there is a belief, as subscribed to by ICER, in the sure and certain hope of constructing imaginary worlds, to drive formulary and pricing decisions, then then it needs to be made clear that this is a belief that lacks scientific merit.

A final point: given the emergence of ICER in the mid-2000s, a reasonable question is whether, following the popularity of reality TV shows, ICER should be seen as a reality show. After all, Poniewozik, in his assessment makes the point that Survivor, one of the most popular, is also ‘a masterpiece of applied postmodernism, a faux reality built entirely from simulacra of the real world, an imitation of life more satisfying and pliant to the creator’s will than reality itself could ever be’ ¹⁷.

The ICER Reference Case

Central to the ICER construction of imaginary value claims is the reference case. Standards for model building, the construction of imaginary worlds, are clearly stated with the preference for models that take a long-term or lifetime perspective. Value propositions are in imaginary cost-per QALY terms. Once an imaginary cost-per-QALY estimate (or estimates under different scenarios) has been constructed, the acceptability of a proposed product price is then assessed against cost-per-QALY willingness to pay thresholds (typically $50,000, $100,000 and $150,000 per QALY with exceptions for higher cut-offs for rare diseases). Whether a product ‘adds value’ is then determined in terms of its impact on an imaginary estimated lifetime modeled QALYs set against a proposed lifetime product cost where both are driven by hypothetical constructed evidence.

Unfortunately, the position taken by the Canadian guidelines reflects the consensus view in health technology assessment as reflected in ICER imaginary constructs. Over the past 30 years, literally thousands of modeled claims have been presented in the literature, including leading health technology assessment journals. Annual reviews of the status of cost-effectiveness or modeled claims in the three journals, Pharmacoeconomics, Value in Health and the Journal of Medical Economics found that the majority of models presented non-evaluable claims (typically lifetime cost-per-QALY) ^{18,19,20,21,22}. Where models were funded by a manufacturer a high proportion supported, in their modeled cost-per-QALY assessment, the manufacturer’s product. All too many of the papers were essentially marketing exercises^23,24.

Transformative Therapiesand Adaptive Strategies

The ICER reference case framework fails to meet the standards of normal science. The claims made lack credibility. They are neither evaluable nor replicable. The proposed adaptive strategies for modeling SSTs, through a modified version of the survivorship modeling, that are ‘not mature enough to determine a reference case’, similarly fail to meet those standards.

ICER proposes, as a first step, to determine which single and short-term SSTs are appropriate for the proposed adaptive modeling approach to value assessments. For ICER’s purposes the SSTs are defined as ‘ therapies that are delivered through a single intervention or a short-term course of treatment that demonstrates a significant potential for substantial and sustained health benefits extending throughout patients’ lifetimes’. Two subcategories of SST are identified: (i) potential cures that eradicate a disease or condition; and (ii) transformative therapies that produce sustained major health gains or halt progression of significant illnesses.

The focus of the proposed adaptive strategiesis the application, where deemed relevant by ICER, of cure proportion modeling of survivorship where a ‘ proportion of patients may be expected to be cured or benefit from a complete halt in the progression of a serious illness’. If the data support a cure proportion survival curve with a sustained plateau’ scenario analyses utilizing various survival analytical techniques will be applied to characterize plausible results(emphasis added). ICER proposes to develop a range around estimated survival until ‘more data become available’.Two points to note: (i) who decides on ‘plausibility’ and (ii) at what stage does ICER propose to revisit its imaginary construct to create, presumably, another imaginary world?

Following public comment, ICER will, as arbiter, make the final decision on whether a treatment meets its SST criteria. The key elements of the reference case model are retained with Incremental cost-effectiveness scenarios and value based prices for multiple time horizons with the overall benchmark the standard lifetime horizon of the official ICER value-based benchmark.In both instances, therefore, the base case is the construction of the reference case imaginary world over the projected lifetime of a hypothetical patient cohort. Neither the SST adaptation nor the standard ICER value based price benchmarks are intended to generate evaluable claims. The modeled outcomes are entirely speculative and are, presumably, for ‘information only’.

As evidence for durability of an SST effect will typically not be available, ICER will provide modeled information ‘ on how cost-effectiveness and value-based prices would be impacted under variousassumptions about durability of effect’. ICER will, however, retain the lifetime horizon as its value-based benchmark basis. Whether decision makers will find these various imaginary ‘information’ scenarios’ useful is a moot point. A major concern is that in accepting the various ICER durability scenarios and their corresponding value assessments decision makers may be unaware of the cumulative impact of ICER’s value judgements and choice of assumptions that lie behind the various scenarios. After all, it is unlikely that the majority of health system decision makers in the US will be familiar with the strengths and weakness of the options in cure proportion survivorship projections or the fallback techniques that ICER may apply where the data are ‘not mature’ – whatever that means - or the criteria ICER has applied to give a seal of approval to the ‘maturity’ of the data. Again, ICER asks its audience to ‘take their word for it’ where this could refer, not only to an ICER preferred model (with attendant sensitivity analyses and probability claims), but the possible creation of competing models by interested parties (also with their attendant sensitivity analyses and probability claims) as additional ‘information’ for health system decision makers. Once we admit the role of imaginary worlds in health technology assessment, our imagination can create, in principle, a multiverse of competing worlds; none of which should be taken seriously..

It is not clear what criteria will ICER apply in its decision to apply the adaptive model. One option, of course would be to admit that the data are insufficient to support any application of survivorship techniques. Unfortunately, this would mean that ICER would have to abandon the reference case and any cost-per-QALY value claims. This is unlikely. ICER’s response will be to develop incremental cost-effectiveness scenarios, even in the absence of robust survivorship estimates, at multiple time horizons at the time horizon representing the longest-available follow-up data and also at 5 years, 10 years, 15 years and the standard lifetime horizon.

Hoisting and Petards

Although the focus in the ICER proposed adaptations is on cure proportion modeling, there is a more substantive issue regarding the need to modify QALY based claims where the reference case model is the framework for SST claims. This is the scenario where, due to the substantial QALY incremental gains potentially attributable to SST therapies, the ICER reference case model can support pricing strategies by SST manufacturers that, while seen as exorbitant by groups such as ICER, are consistent with the willingness to pay threshold values of the reference model. Taken at face value, this puts ICER is an embarrassing situation. While the reference case model has been applied over the past decade or more by ICER to support price discounting recommendations, the same model can be used to support SST therapy pricing that may run to millions of dollars for a single cure application.

In the 1604 second quarto version of Hamlet Shakespeare has Hamlet telling his mother:

Let it work,

For ’tis the sport to have the engineer

Hoist with his own petard; and’t shall go hard

But I will delve one yard below their mines

And blow them at the moon. O, tis most sweet

When in one line two crafts directly meet²⁵

Whilst omitted from the first quarto and the first folio, the phrase ‘Hoist with his own petard’ has resonated in the English language. Although ICER and other health technology assessment groups are hardly to be equated with the mission directed by Claudius for Rosencrantz and Guildenstern, it has become proverbial in capturing an event of ironic reversal or poetic justice. For ICER, and other commentators, attempts to model the QALY gains from the application of SSTs over the lifetime of the patients have the potential to justify substantial single prices for SST cure interventions. While ICER has been comfortable in assessing QALY incremental gains for chronic therapies in a range of disease states in cost per QALY terms, the gains have typically been small. Hence modeled imaginary worlds have delivered recommendations for significant price discounting where costs have exceeded willingness to pay thresholds. With SSTs the position is reversed: the modeled imaginary QALY gains are such that substantial one-off or short-term cure prices can be justified by application of conventional modeled willingness to pay thresholds.

Consider, to illustrate the issue faced by ICER and others who subscribe to a lifetime reference case framework for creating imaginary treatment impact, the following example. If a company develops a single-application gene ‘cure’ in a rare disease, the scenario changes from a limited survival expectancy with a low health related quality of life under standard care. After treatment, the patient can presumably (by assumption) look forward to a ‘normal life’ with, for example, incremental modeled discounted QALYS in a hypothetical range of 30 to 50 years or more. If we take the lower bound of 30 QALYs than, to meet a willingness to pay threshold of $50,000 (the lowest of the three typically modeled by ICER) then a manufacturer could set a one-off treatment price marginally less than $1.5 million ($50,000 per QALY) and meet the reference case benchmark for ICER’s seal of approval for pricing. Even if cost-offsets were included the picture would not change materially. Manufacturers could justify the pricing policy on the grounds that they are subscribing to state of the art modeling as advocated by ICER and other professional groups.

The challenge, as presumably seen by ICER, is to attempt to maintain its commitment to imaginary lifetime reference case evidence creation while proposing possible ‘methods adaptations’ to the current standard assessment approaches to accommodate SST pricing.Central to ICER’s concerns is the belief that there needs to be a mechanism that claws back a ‘pricing surplus’ that is believed to be an element in the SST pricing structure. Unfortunately, as ICER recognizes, there is no obvious solution to address either what ICER sees as probably unacceptable SST pricing.The issue ICER has to face is that its reference case evidence construction fails to meet the standards of normal science. Where evidence is constructed by assumption in the ICER reference case or its proposed adaptive SST version, there is no fallback position. A pricing surplus might be assumed, but no evidence presented to support a quantitative assessment across the range of SST therapies. Are we, therefore, to cherry pick a model framework, either versions of the reference case model or versions of possible adaptive cure proportion models to support ICER conclusions and recommendations for SST pricing?

Proposals by ICER for possible innovative payment mechanisms ‘to manage the tensions between high prices and payers’ need to maintain affordability’ reston the willingness of health decision makers to accept ICER imaginary modeled claims under the ICER selected adaptive SST framework. One proposal put forward is for price caps. ICER rejects this on the grounds that a price cap may penalize SST manufacturers, shifting incentives towards investment in chronic therapies. Other proposals are for rate of return pricing and for shared savings where the value of cost offsets generated by SSTs would be shared. Again, any agreement for shared savings would have to agree (i) the offset savings for the health system as contracting party and (ii) the distribution of savings. These would presumably require a monitoring agreement. If the argument is extended to include an agreement to share in the so-called surplus, ICERs problems are compounded. First, there is no evidence to support the hypothesis that manufacturers include a ‘surplus component’ into their pricing policy (if the notion is even entertained). Second, if ICER believes that there is a surplus component (i.e., economic rent), it would be virtually impossible to provide an estimate. ICER would have to demonstrate that they can provide a validated algorithm to generate estimates of the proposed (and mythical) surplus component in one- off pricing that applies across cure therapies in multiple disease areas.

ICER is in an awkward position. After over a decade of therapy intervention assessment, building claims for price adjustments with evidence from the reference case guide to the construction of imaginary worlds, the reference case has become a liability. Unless ICER can convince health system decision makers that the reference case should be put to one side as a basis for pricing recommendations for one-off ‘cure’ therapies, doubt is cast on the application of the reference case framework across multiple disease states.

The irony of the situation is that in following groups such as ISPOR, NICE and CADTH, ICER joins them in applying a reference case framework that can be used to justify, on ICER’s terms, a one-off ‘cure’ price that ICER considers excessive. Indeed, in these cases it is also ironic to consider that manufacturers of one-off cures could utilize a reference case model to establish a pricing structure that meets ICER’s willingness–to-pay standards. It is also not clear that the one-off pricing structures are necessarily exploitive (i.e., rent seeking). In the absence of an assessment by a third party of the costs of ‘cure’ drug development it is impossible to make any claim for a ‘surplus’. To assume that a surplus exists is simply irresponsible. Of course, the irony of the situation does not extend just for one-off cures. There may be in the drug development pipeline chronic therapies that yield similar QALY gains. In which case ICER is also placed in an impossible situation in attempting to maintain the integrity of it’sof claims from imaginary worlds. Perhaps, as the Hamlet stage directions might detail: exeunt ICER with imaginary worlds.

Survivorship, Utilities and QALYS

A recent commentary on the ICER final evidence report for Duchenne muscular dystrophy (DMD) pointed to the limitations for the ICER reference case in the choice of QALY measures²⁶. It was noted that while ICER appears to follow the NICE reference case in recommending (if not mandating) a standard generic measures (e.g. EQ-5D-5L) for QALY assessments by stage of a chronic disease, the commitment is less clear cut.

Putting to one side the obvious objection that the lifetime QALY claims are entirely speculative and fail the standards for demarcation of science from pseudoscience, it appears thatin the case of DMD ICER simply takes the ‘only available’ utility measure, irrespective of its demonstrated ability to capture relevant aspects of quality of life (QoL) or, more narrowly, health related quality of life (HRQoL) in DMD.ICER is not interested in investing resources to generate appropriate QoL or HRQoL instruments for the target disease areas. Even if those resources were invested in a timeframe appropriate to the future ICER assessment schedule, the effort would be questionable given that the ICER reference case is not designed to test hypotheses.

As ICER points out, at the time of marketing approval and product launch the evidence base, and this is not unique to SSTs, is often limited. Unfortunately, as demonstrated by past ICER reports, all too little attention is given to the weaknesses of the evidence base with ICER doggedly going forward to make a modeled value assessment that rests on unsupported or dubious assumptions. This applies in particular to the measurement of utilities appropriate to the target patient population. In many cases there is no validated utility instrument for the target SST that captures the key attributes of the experience of patients, providers, caregivers and the wider family environment. Instead, ICER falls back upon published utility scores that fail to capture the attributes of interest for treatment impact, notably generic scores from instruments such as the EQ-5D-5L, the SF-6D and HUI Mk3. Duchenne muscular dystrophy is a classic example where their application in a reference case model is clearly inappropriate.

In the case of SSTs, ICER faces more intractable issues if it pursues the reference case paradigm. The fact is that, apart from making assumptions regarding the QoL or HRQoL of ‘cure’ patients subsequent to the therapy intervention, or for mixed models where only a proportion of patients respond to the ‘cure’ intervention, there is no evidence base to justify ICERs ‘measured’ QALYs. While this does not imply acceptance of the ICER imaginary world lifetime reference case, ICER recognizes in the discussion paper that in there may be ‘additional dimensions of value’ that ‘should be systematically considered in the evaluation of all new therapies’. These include real option value, the value of hope and insurance value. Although not addressed by ICER there are a range of other issues specific to patients, caregivers and their families that need to be canvassed. ICER is not, at the moment, apparently interested in more ‘robust’ QoL or HRQoL measures with ICER putting them them to one side as premature.

Given the reference case model, ICER needs to come to terms with the issue of utility measurement. It is one thing to cobble together a reference model from’available data’ and another to address the question of the appropriate and defensible measurement of QoLor HRQoL relevant to the disease state. In the case of SSTs, ICER needs to propose standards for the utility scores relevant to stages of a disease and the experience of QALYs in the post-cure target population.The concern, for those subscribing to the information role of imaginary constructs, is that audiences for ICER SST reports and the supporting imaginary modeled claims, will have little faith in the case put forward to justify pricing for SSTs or options for ‘shared savings’ driven by constructed evidence and potentially questionable choice of QoL or HRQoL measure.

Quis custodiet ipsos custodies

Even though the ICER reference case methodology fails to meet the standards of normal science, there can be little doubt that ICER will attempt to defend its position. After all, its business case rests upon the construction of imaginary worlds with the apparent willingness of health system decision makers to take the modeled value claims at face value. Given this, and with ICER’s self-appointed guardian role in value assessment for the US health care system, it is important that ICER is as transparent as possible in interacting with stakeholders. If ICER is to maintain its credibility in the construction of imaginary worlds it needs to make clear the limitations inherent in its reference case methodology.

• ICER should make clear in all reports that the modeled reference case with the base case lifetime or long-term perspective does not meet the standards required for normal scientific inquiry in the construction of empirically evaluable theories and hypotheses;

• ICER should make clear that its imaginary modeled recommendations are always provisional and all too often rest on a limited evidence base. Given this, ICER could commit to a regular review of its models (e.g., every two years) as new data become available following product launch to recast the model structure and assumptions. As there is no basis for saying this is now a more ‘realistic; picture of an unknown future ’reality’; the result would be simply a succession of modeled imaginary worlds with each seen as the latest ‘provisional’ construct;

• ICER should make clear that it sees its value assessments as potentially informing health care decisions and that, as stated clearly in the CADTH guidelines that follow an identical reference case methodology, the ICER value assessments are ‘distinct from conventional research activities, which are designed to test hypotheses’;

• ICER should make clear that the reference case framework can support any number of competing value assessment models and that attempts to differentiate and claim one model is ‘superior’ to another can only rely on an assessment of the assumptions made in constructing the imaginary world;

• ICER should make clear that any claims for ‘value’ based on notional cost per QALY willingness to pay thresholds are not only impossible to evaluate but that they were never intended to meet the standards of normal science and were therefore never intended to be evaluated - which would be impossible over the timeline of the model.

Conclusions: Nullius in Verba

Does ICER have a significant or serious role in the pricing and allocation of health care interventions in the US health care system? Unless a case can be made for the information content of imaginary constructs as a recognized input to decision making, the answer is clearly, no. Irrespective of claims by ICER that their modeling is reflective of the ‘state of the art’ modeling endorsed by professional groups such as ISPOR, the ICER business case that rests on creating imaginary worlds across disease areas, fails the demarcation test; it is pseudoscience.

ICER is not alone in subscribing to the construction of imaginary ‘for information’ worlds in health technology assessment. A significant proportion of single payer health systems also subscribe to this meme – to include Australia, Canada, Ireland, New Zealand the UK and the majority of European Union countries (with Germany a notable exception). Unlike the situation of ICER in the US, the agencies responsible for evaluating reference case models in these countries have a legislative responsibility. They can require a manufacturer to submit a reference case model, which is then subject to review both by the agency and by contracted third parties who have an established track record in assessing modeled imaginary reference case claims. ICER is alone in developing and releasing recommendations from its own model. Certainly, the model is accessible to stakeholders to review (subject to constraints) but there is no formal review process by a third party of the final proposed model. This has to be a concern as there are a number of, primarily academic, centers in for example the UK (e.g., University of Sheffield) and in Australia (e.g., the University of Adelaide) with the necessary expertise. Whether this is worth doing is, of course, problematic as they would be asked to evaluate an imaginary world and potentially propose a modified or alternative imaginary world. This hardly seems to be a basis for advancing knowledge.

There is no reason for the US to embrace imaginary worlds that support hypothetical value assessments. This applies in particular to the application of cost-per-QALY willingness to pay thresholds. The fact that health systems outside the US are prepared to base decisions for pricing and access on the construction of imaginary worlds is no reason for the US to follow suit. In the case of SSTs, ICER faces a critical challenge. How can ICER adapt its reference case methodology when that methodology can be used to support one-off market entry cure pricing that ICER and other commentators might consider excessive?

Importantly, as survivorship is the critical variable driving aggregate estimates of imaginary QALYs the choice and justification for the survivorship algorithm becomes of interest. While cure proportion based claims for QALYs would still fail the demarcation criteria, ICER’s choice of cure proportion application, in particular if viewed as a reference case ‘rescue’, would be a further point of contention between manufacturers, health systems and ICER. The more assumptions that are built into any model, even one that meets the required demarcation criteria, the greater the probability that the claims made are false.

Next Steps

It seems ridiculous, almost to the point of absurdity, to base formulary decisions for access and pricing on the construction of imaginary worlds. While ICER may see these fantasy constructs as the ‘state of the art’ in health technology assessment, it is a poor state of the art to put attempts at creating credible, evaluable and replicable claims for innovative products, especially those that are focused on rare disease and which might offer a one-off cures, to one side. This is not an insoluble problem. Previous commentaries in INNOVATIONS in Pharmacy have addressed this issue. The key point is that initiatives have to be taken to create and monitor platforms for ongoing therapy evaluations. Rather than speculate and then walk away in favor of creating the next imaginary world, it has been proposed that treatment center registry platforms be encouraged to monitor therapy response and the management of patients. An extensive proposal was made for the treatment of chronic pain, with particular emphasis on the abuse of pain medications such as opioids, together with proposals for platforms in behavioral health and medical marijuana^{27,28,29,30,31,32,33}. Indeed, the Program in Social and Administrative Pharmacy at the University of Minnesota has published guidelines for formulary evaluation that specifically address the need for manufacturers in making formulary decisions to propose and if necessary underwrite platforms that will capture the impact of therapies, reporting back to formulary committees in a meaningful time frame^34,35. It is proposed that, subject to peer review, future commentaries will address this issue in more detail while continuing to point out that creating imaginary worlds is hardly the basis for effective decision making.