Alternative Facts and the ICER Proposed Policy on Access to Imaginary Pharmacoeconomic Worlds

Abstract

The Institute for Clinical and Economic Review (ICER) announced at the end of March 2018 that it proposed to share executable draft economic models with relevant drug manufacturers during ICER evidence reviews. Taken at face value, ICER is offering manufacturers and other interested parties the possibility of a greater involvement in the assessment of modeled claims for product pricing. The purpose of this commentary is to point to the obvious limitations to this policy and to raise questions as to the extent to which ICER is willing to commit to transparency in the creation of modeled claims. While a commitment to greater transparency is to be welcomed, the fact remains that the ICER methodology does not meet the standards of normal science. ICER modeled claims not only lack credibility but are impossible to evaluate or replicate in treating environments. In proposing greater transparency ICER runs the risk of undermining its own credibility. If manufacturers have the opportunity to generate alternative models which create alternative claims, then ICER runs the risk of its pole position in health technology assessment being questioned. Rather than focused on non-evaluable lifetime modeled claims, there is the opportunity for ICER to reject its current business model, accepting instead a modelling paradigm that is consistent with the standards of normal science.

Introduction

A recent commentary by the present author in INNOVATIONS in Pharmacy addressed the issue of the apparent unwillingness of the Institute for Clinical and Economic Review (ICER) to engage in a comprehensive and meaningful assessment of the structure and assumptions of the various cost-effectiveness models that support their reviews and pricing recommendations. While the commentary asked readers to reject the ICER methodology, which other previous commentaries have criticized, the absence of standards for a formal review procedure of ICER models was seen as a barrier to understanding the limitations of the ICER methodology. Whether in response to this critique or not, ICER on 30 March 2018 issued a press release announcing a new Program to make available draft executable economic models during drug assessment review process ¹. Although details of how this program is to be implemented have yet to be published, the purpose of this commentary is to raise a number of concerns as to whether this is in fact an invitation to greater transparency or a defense of the status quo.

As detailed in previous commentaries, if the standards of normal science are applied to the ICER modeled claims methodology, then ICER fails those standards². The modeled claims made for competing products lack credibility because they are neither evaluable nor replicable. They should be seen as pseudoscience ^{3 4}. Accepting ICER modeled claim means rejecting standards in normal science that have been in place for the last 350 years ⁵. This point has been made in previous commentaries on ICER, including four evidence reviews together with a more recent comprehensive review of the ICER approach ^{6 7 8 9}. The position taken was that rather than focusing on developing imaginary lifetime-cost-per-quality adjusted life year (QALY) models, ICER would be more usefully employed in focusing on credible short-term models whose claims could be evaluated and replicated in real world treating environments. This critique of the proposed ICER program for transparency should be seen against this background.

ICER is not alone in putting to one side the standards of normal science in health technology assessment. The latest Canadian guidelines for technology assessment are quite clear: Economic evaluations are designed to inform decisions. As such they are distinct from conventional research activities, which are designed to test hypotheses ¹⁰. This ‘unconventional’ role of economic evaluations as research activities opens the door to what has been described in current American political discourse as ‘alternative facts’, where patently absurd claims drive policy decisions. Competing lifetime models generate alternative illusions; they are divorced from reality and have no chance of ever being falsified. Unfortunately, all too many people are prepared to accept this information role with no thought, or awareness, of these standards. The result is that decision makers will be faced with competing modeled claims, none of which can be justified in scientific terms.

Constrained Transparency

In single payer health systems such as the UK, Australia, Canada and New Zealand, agencies can mandate both the standards for modeled formulary submissions as well as the modeling review process ^{11 12 13 14}. In these countries the construction of imaginary worlds are a critical input to pricing and formulary decisions. In the UK, for example, the National Institute for Health and Care Excellence (NICE) mandates a lifetime cost-utility reference case model for formulary submissions with willingness to pay thresholds. A submission by a manufacturer for formulary listing is passed to an external review group and then finally assessed by NICE and a determination made. Typically, the model will be provided in electronic form. The process is transparent with the appropriate documentation, redacted for commercial confidentiality, posted to the NICE web site. The review group’s remit is relatively wide. It can accept the manufacturer’s model, modify it or propose an alternative model framework. NICE in turn, on receipt of the model can consider possible corrections as part of their final review.

A similar process holds in Australia with the Pharmaceutical Benefits Advisory Committee setting standards for formulary submissions to support both the clinical and cost-utility case for the product. An electronic version of the model is also requested. The model and supporting documentation are sent for external reviews by a contracted academic health technology assessment group. The PBAC then factors the review assessment into the decision process. The process is less transparent than that followed by NICE but still relies on the construction of an imaginary world.

ICER, as an independent group, has no ability to ask a manufacturer to submit a modeled case for a product. Indeed, typically no request is made for a manufacturer to voluntarily submit a model for review. Rather than potentially submitting a model submission, ICER has contracted with academic collaborators to develop a draft model. The draft model is presented for review where manufacturers can provide comments. Whether the comments are accepted is a decision for ICER but, again typically, the model remains essentially unchanged. At the same time there is only a limited time for manufacturer’s to comment and for the model builder’s to respond. To date, ICER has refused access to an electronic version of the model itself on the grounds that (i) the model builders have to retain intellectual property rights until possible publication and (ii) that few people are equipped intellectually to offer a critique of the model without the support of the model builders. Understandably, manufacturer’s feel they are being shortchanged.

The proposed new ICER program still insists on the intellectual right of the model builders (academic collaborators) while offering, subject to a payment to the model builders by the manufacturer, access to executable files. Although not entirely clear, it appears to be the intention to provide manufacturers with some version of the draft electronic model. However: To protect the intellectual property of the researchers and their institutions, models will be shared under confidentiality or licensing agreements. Under these agreements, manufacturers will be unable to alter the model for their own purposes. In addition their access will be for a limited timeframe and will be confined to the team of individuals responsible for the review’. Presumably access will be limited with the manufacturer unable to evaluate alternative model structures. Access will be for a limited timeframe – although this is undefined. It is not clear how long the model builder’s will have to engage with manufacturers, agree their (additional) remuneration and provide access to the model so that they can become familiar with ‘a model’s structure, estimates, key assumptions and calculations. The assumption must be, hopefully, that access to the model and supporting data (e.g., systematic reviews) will be on-line and that sufficient time is given for review and comment. Given the complexity of such models this could involve an extend time period and the possible engagement of consultants by the manufacturer to review the model and provide a comprehensive critique. Also, given time and resource constraints it may even be possible for the manufacturer to create ‘alternative fact’ modeled scenarios. Even so, this does not mean that we should lose sight of the imaginary nature of these potentially competing constructs.

Unfortunately, it seems unlikely, from past experience with ICER and their ‘need’ to generate recommendations for products in concert with the product receiving marketing approval, for adequate time to be given for review. A reasonable estimate would be for the manufacturer to have 3 months to review, comment and even generate competing ‘alternative facts’. It might reasonably be asked that if the reviewer is ‘unable to alter the model’ (whatever that really means) what is the object of the exercise? To limit the review to peripheral or marginal comments regarding assumptions? Can the reviewer run alternative simulations within the model framework? Will the model building group have a watching brief to veto untoward attempts to run simulation or test alternative structural assumptions such as the Markov process framework for disease staging and progression? Can the reviewer seek specialized consulting services to support its assessment? Last but not least, who will draft the confidentiality or licensing agreement (again, whatever that means) to facilitate access? A further unresolved question is whether the academic collaborators will have the time and resources to engage with manufacturers in a reasonable time frame. If evidence from NICE as to the resources required to review models is any guide, then the collaborating groups are likely to be well understaffed. Presumably, if they cannot meet the demands placed upon them they can decline requests to contract for any engagement.

As stated, this policy is unlikely to satisfy stakeholders in the ICER assessment process. It limits the ability of manufacturers and other parties to undertake a meaningful assessment of the model. Rather than, in the case of the NICE process where an external assessment group has the remit and is given the time and resources to modify or develop a competing model, ICER seems intent on waiting until the last minute for any review, constraining the options for review and, if past experience is a guide, limiting the time that manufacturers and others have for comment.

Engagement in Model Building

If ICER is to engage in a meaningful process of review to ensure transparency in model development, then stakeholders need to be given access to the modeling teams at an early stage in the development of the model. The model builders should develop a position paper for the model detailing its proposed structure, treatment pathways and key assumptions. Systematic reviews should be documented and the relationship of the proposed model to others that have developed in the disease or treatment area detailed at an early stage.

The model builders should also be asked whether or not they propose to develop, within the model framework, credible claims that can be evaluated within a relatively short timeframe as feedback to formulary committees. There should be an explicit statement that the model is for information only and it is not possible in the proposed framework to develop evaluable claims and test hypotheses. On the other hand, if the model builders propose to develop credible and evaluable claims, then they need to propose how those claims are to be evaluated.

Attempts to argue for an early engagement with the model builders will, no doubt, be resisted by ICER. Unfortunately, this would undercut their case for transparency. Understandably, stakeholders would feel that any engagement with the modeling process has been deliberately minimized. They would have no say in the initial decisions on the scope and content of the model. All they will be asked is if there are minor adjustments that might be considered, and probably rejected, in a brief review process. Their hands are, to use a well-worn cliché, tied. After all, why pay the model builders for access to the model when they are unable to consider changes which may reflect better the clinical and supporting economic evidence.

More Alternative Facts

While it is not the intention here to encourage alternative models generating alternative facts (after all, the technology assessment literature is replete with thousands of non-evaluable modeled claims ¹⁵), the pole position that NICE has taken upon itself to occupy should be challenged. One way to achieve this is to give manufacturers sufficient advance warning so they are in a position to consider the option of developing alternative modeling frameworks which can challenge the ICER version of alternative facts. If manufacturers have already submitted models to agencies such as the NICE and the PBAC they will be in a position to meet ICER on a level playing field (another well-worn cliché). These imaginary worlds would need to be customized for the US market through in-house modification (to include simulating ICER willingness-to-pay thresholds) with their model accommodating the various US market segments (commercial, Medicare, Medicaid, VA). This puts a premium on a meaningful review period.

There is also the question of how robust the evidence base is to support modeling. There is now substantial evidence to point to the lack of reproducibility in clinical claims ^{16 17}. If, for example, key assumptions in a model are built upon a handful of clinical claims where there is no evidence for replication, then the manufacturer may be justified in claiming that assumptions based on those of a model have little validity. Similar concerns can be raised over the often second-hand and interpolated estimates of utility. It is no argument to say that we can overcome limited or weak data or its complete absence by a judicious choice of scenario simulations driven by ‘reasonable yet unsupported assumptions’.

Of particular interest is the modeling of adherence and persistence with therapy. Previous commentaries on ICER models have pointed out that it seems pointless to assume a high level of compliance (indeed, over the lifetime of the modeled target population) when numerous studies have detailed the lack of persistence with therapies with up to 50% of patients non-compliant within two years of an index prescription. If this is expected to be the case, (and who knows for a product being modeled which has yet to enter the market?) then basing cost-per-QALY claims and pricing recommendations on highly dubious compliance assumptions seems a waste of time. To this, it might be added, is the typical ICER modeled assumption that prices for the products and other direct medical costs do not change over the timeframe of the model.

If failure to consider long-term adherence and persistence is a critical weakness in the ICER modeling approach then it might be reasonable to ask whether manufacturers, once they have limited access to the model, can explore different compliance scenarios together with future pricing scenarios. Will ICER allow the results of these scenarios to be presented to the ICER comparative effectiveness public advisory councils? Will these councils be allowed to have access to modeling critiques and alternative scenarios presented within the ‘unalterable’ model?

Access to Prior Modeling

There seems to be no reason why this announced access policy could not be extended to re--assessments by interested parties of prior models that have been developed by ICER. Presumably, after an elapsed timeframe of 18 or 24 months the various academic groups who apparently hold intellectual rights to the models could be asked whether or not they intend to publish. If there is no intention to publish and ICER is assured that there are outside groups capable of reviewing the model, then the model could be reviewed.

Manufacturers, for example, might have an interest in a review of the model if there is a concern that the initial releases of the model and ICER recommendations for pricing have commercially disadvantaged the company. A review by an independent assessor might point out, apart from the obvious point that the claims from the model are non-evaluable, that there are alternative modeled facts that could be developed, again with non-evaluable claims, but which came to quite different conclusions on pricing after applying the ICER willingness-to-pay thresholds. Part of this assessment could involve a review of ICER modeled assumptions. After 18 to 24 months from product launch and formulary listing, there would be evidence, for the various US market segments, on patterns of adherence and discontinuation, the rate of market uptake, quality of life claims and response to therapy. These could be a check on the assumptions in the model with the model recalibrated for clinical, cost-utility claims and simulated pricing points.

Taking ICER Seriously

The question which remains open is whether or not manufacturers and their prospective clients should take the announced ICER program seriously. It may well be that a manufacturer is prepared to put ICER to one side and make the case, quite appropriately, that pricing and formulary decisions should be evidence based. Rather than basing decisions on one set of alternative facts among a potential multiverse of competing alternative facts, the ICER approach should be rejected in favor of a program of establishing short-term modeled claims with protocols developed to demonstrate how those claims can be evaluated. This is, as has been detailed in previous commentaries, the position taken in the proposed Minnesota guidelines for formulary submissions ¹⁸. If, however, the manufacturer wishes to engage with ICER, then the question of transparency should be addressed and the case made for the option of competing economic evaluations.

Conclusions

A reasonable question is whether or not ICER is doing its best to limit any reviews that may undermine the ‘validity’ of the collaborator model. Manufacturers cannot alter the model for their own purposes. This presumably means that they will not have the ability to do a detailed reassessment and explore modified or competing model structures. Indeed, it is far from clear as to whether or not the manufacturer will have access to the full model for review and recalibration purposes or whether they will only have access to stripped down elements of the model and face embargoes on what elements can be recalibrated. If there is a meaningful commitment to transparency then it must involve manufacturers at an early stage and with the option and opportunity of putting forward what they may consider alternative modeled constructs.

ICER has put itself in a difficult situation and one which has significant risks attached. ICER appears to be attempting to draw a fine line between offering limited transparency in the hope that the model framework will not be challenged and offering greater transparency which gives manufacturers the opportunity to develop competing models that generate alternative facts. If alternative facts emerge from competing models then decision makers are in a quandary as to which to believe. In the absence of the ability to discriminate between competing empirical claims, the fallback is to a (possibly unseemly) argument over which model has the most realistic assumptions supporting its imaginary future world (or worlds in competing models).

Given that the ICER business case rests in large part on the construction of imaginary modeled worlds, designed to ‘inform’ decision makers rather than a commitment to developing models that generate evaluable claims, the presence of competing alternative facts would raise doubts as to ICER’s role in pricing and formulary decision making. Opening the door to increased transparency and the possibility of competing imaginary worlds may lead not only to questions over the value of non-evaluable competing clams and pricing recommendations, but to a demand for modeled claims that are evaluable and replicable in treating environments. In short, an acceptance of the standards of normal science.

Conflicts of Interest

I declare no conflicts of interest or financial interests that the I or members of my immediate family has in any product or service discussed in the manuscript, including grants (pending or received), employment, gifts, stock holdings or options, honoraria, consultancies, expert testimony, patents and royalties.