Cost-effectiveness models for Alzheimer’s disease and related dementias: IPECAD modeling workshop cross-comparison challenge

Abstract

Introduction:

The credibility of model-based economic evaluations of Alzheimer’s disease (AD) interventions is central to appropriate decision-making in a policy context. We report on the International PharmacoEconomic Collaboration on Alzheimer’s Disease (IPECAD) Modeling Workshop Challenge.

Methods:

Two common benchmark scenarios, for the hypothetical treatment of AD mild cognitive impairment (MCI) and mild dementia, were developed jointly by 29 participants. Model outcomes were summarized, and cross-comparisons were discussed during a structured workshop.

Results:

A broad concordance was established among participants. Mean 10-year restricted survival and time in MCI in the control group ranged across 10 MCI models from 6.7 to 9.5 years and 3.4 to 5.6 years, respectively; and across 4 mild dementia models from 5.4 to 7.9 years (survival) and 1.5 to 4.2 years (mild dementia).

Discussion:

The model comparison increased our understanding of methods, data used, and disease progression. We established a collaboration framework to assess cost-effectiveness outcomes, an important step toward transparent and credible AD models.

1 |. BACKGROUND

Nations worldwide are challenged with allocating scarce health care resources to ensure that people living with Alzheimer’s disease (AD) and related dementias have access to emerging effective interventions. Mathematical model-based economic studies are central to support policy discussions, especially where value (cost-effectiveness) is a key consideration. Models extrapolate trial outcomes by synthesizing data from multiple sources (e.g., trials, disease registries, and cohort studies) to estimate the long-term costs and benefits of alternative options. The transparency and credibility of models are key for their results to be used by decision-makers and to support timely access to emerging interventions for people living with AD and related dementias.

The AD models published to date have been subject to extensive critical appraisal and have been discussed in terms of methods, data, and reporting standards.^1–6 Recurring themes are limitations in representing the natural history of AD and a lack of empirical evidence for extrapolating trial (cognition) outcomes to long-term outcomes (e.g., function, care needs, and mortality). In addition, real-world data on the natural progression across the disease spectrum and across various outcomes (cognition, function, and behavior) are limited, as well as the utility and costs data to associated with these outcomes. On top of this, models use different structures, reflect different populations/settings, and can lack transparency around data and assumptions.¹ This results in unexplained differences in their predicted outcomes, which has received relatively little attention to date.

Confidence by decision-makers in using health economic models is key to support appropriate decision-making. A lack of clarity and reliability has been cited as a barrier to the use of models in decision-making by policymakers, researchers, and health care managers.⁷ Unexplained differences across models for AD and related dementias are a threat to their credibility and could lead to inappropriate decision-making in the adoption of new interventions, as has been reported, for example, in the field of abdominal aortic aneurysm screening.^8,9

Comparative modeling (i.e., cross-validation), defined as “examining different models that address the same problem and comparing their results” in good research practice guidelines¹⁰ could increase confidence in models if similar results are observed. This has been applied in the fields of diabetes, chronic obstructive pulmonary disease, and cancer, operationalized as a series of workshop meetings on comparing models on a similar hypothetical benchmark scenario,^11–13 externally validating models against clinical trials,^14–17 comparing extreme modeling scenarios to reveal natural history characteristics,¹⁸ independently replicating models from published details,¹⁹ and comparing predictors of trial outcomes.²⁰ Comparative modeling is urgently needed in the field of AD and related dementias to improve our understanding of models and to ensure their credibility for decision-making for new interventions.

We initiated a platform, the International PharmacoEconomic Collaboration on Alzheimer’s Disease (IPECAD) Modeling Workshop, for exchange of ideas on health economic modeling in AD and related dementias, to discuss and align on key methodological issues and to foster the development of robust and transparent models to guide decision-making around novel intervention scenarios. We report results from a workshop on the cross-comparison of models for AD and related dementias.

2 |. METHODS

IPECAD is a broad collaboration of people in academia, government, and industry focused on improving mathematical modeling of AD and related dementias and the methods available on the economic evaluation of interventions in this field. This is accomplished through a recurrent conference, development of open-source modeling methods, and workshops related to (www.ipecad.org). The IPECAD Modeling Workshop Group (consisting of co-authors L.J., C.G., R.H., B.W., A.W., A.G., A.S., and A.K.) initiated a series of modeling workshops to create a dedicated forum. The first workshop was held on September 19 and 20, 2019 in Stockholm, Sweden, and the second workshop was held virtually (due to the coronavirus disease 2019 [COVID-19] pandemic) on September 17 and 18, 2020.

The aim of the first workshop, drawing inspiration from the Mount Hood Diabetes Challenge Network,¹⁴ was to bring together those who are conducting research in the field of decision-modeling for AD and related dementias, to create a dedicated forum for an exchange of ideas on health economic modeling, to discuss and align on key methodological issues, and to foster the development of robust and transparent models to guide decision-making around potential new treatment options.

Participants were identified using available reviews of model-based frameworks for predictive and decision-analytic purposes in the AD context, and also via the IPECAD Modeling Workshop Group’s awareness of current research in the subject area, with potential participants considered eligible if they were research active in the field of AD decision-modeling in a predictive and decision-analytic context. A wide range of modeling groups participated, representing models designed for predictive and decision-analytic purposes with varying research questions, contexts, methods, outcomes, and underlying data. Participants were invited to present their model frameworks, and structured discussions covered important areas of development for the field. The meeting successfully achieved key objectives including participation, collaboration, and the recommendation from participants to develop a format for a second workshop (see Supplementary Material 1 for the agenda and Supplementary Material 2 for the minutes).

The second workshop, which is the focus for this report, involved the development of a benchmark scenario for modeling of hypothetical AD therapies, the cross-comparison of model outcomes submitted by participating modeling groups, and a structured discussion in relation to model design choices and parameterization in terms of eligible populations, treatment settings, disease progression, mortality, treatment operationalization, and model assumptions. Participants for the workshop were identified through a detailed review of the current literature in this area and via networks and conferences related to disease progression and cost-effectiveness modeling in AD and related dementias. Participants were invited if they were research-active in the field of AD modeling and also had a model that (1) was a basic reflection of AD disease severity (e.g., mild, moderate, and severe dementia), (2) was considered able to reflect the benchmark scenario, and (3) was not yet widely represented by other models (e.g., a five-state Markov model using transition probability inputs similar to other models already considered eligible). Eligibility was assessed by the IPECAD Modeling Workshop Group.

All participants supported the co-development of workshop procedures, including the determination of the benchmark (reference case) scenarios and the reporting methods used. Two benchmark scenarios were used (Table 1 and detailed in Supplementary Material 3: chapter 2.2), one for models starting with patients with mild cognitive impairment (MCI) (Scenario A) and one for models with patients starting with mild dementia (Scenario B). Both scenarios can be described in terms of four standardized patients (see Table 2). A hypothetical disease-modifying treatment effect was specified as a 30% reduction in disease progression from the initial disease state to the next (MCI to mild dementia, or mild dementia to moderate dementia, respectively). Input data for natural history, conceptualization of disease stage, mortality, costs, and utilities as well as choice for modeling approach were not pre-specified in order to limit extensive model alterations and thus encourage participation. In addition, variation in inputs allowed discussion about their impact on model outcome predictions.

TABLE 1.

Benchmark scenario and reporting outcomes for Scenarios A (starting in MCI) and B (starting in mild dementia)

Aspect	Specification
Population	Scenario A: persons with MCI due to AD (analyst to detail diagnostic criteria used, for example, on use of biomarkers or clinical criteria).  Scenario B: persons with AD-type mild dementia (analyst to detail diagnostic criteria).
Starting age	70 years.
Setting	Clinical (typically memory clinic), with patients already identified for treatment (no procedures or costs for diagnostics to be included).
Control strategy	Usual care (e.g., including cholinesterase inhibitor treatment).
Intervention strategy	Disease-modifying intervention, resulting in 30% reduction in progression (e.g., rate ratio of 0.70 for conversion from MCI to dementia [Scenario A] or mild to moderate dementia [Scenario B] or other comparable effect as defined by analyst). Analyst determines criteria of mild and moderate dementia, whether treatment effects extend after discontinuation of treatment, and whether treatment effects mortality.
Stopping rule	Scenario A: intervention in MCI state and treated until conversion to dementia.  Scenario B: intervention in dementia state and treated until moderate dementia.  Both in Scenario A and B: treatment for a maximum of 5 years.
Treatment discontinuation	10% per year.
Time horizon	10 and 20 years.
Discount rate	Costs and QALYs both 3.5%.
Half-cycle correction	If applicable (i.e., if Markov-type cycle duration was used).
Outcomes	Mean person-years: alive, in MCI, mild dementia, moderate dementia, severe dementia, on intervention, full-time care, or living in institutionalized setting. Proportion/occupancy of patients in states MCI, mild, moderate, severe, and death by year since baseline. Mean discounted costs (disaggregated to intervention, direct medical, direct nonmedical costs), mean discounted QALY (patient only), incremental cost-effectiveness ratio.

Abbreviations: AD, Alzheimer’s Disease; MCI, Mild Cognitive Impairment; QALY, quality-adjusted life year.

TABLE 2.

Characteristics of standardized patient based on the benchmark scenario, two for each scenario (male and female), as both scenarios did not specify sex as it was intended to reflect the distribution of sexes to be expected in the target population

Age	Sex	Syndrome	Cause
70	Female	MCI	AD
70	Male	MCI	AD
70	Female	Mild dementia	AD
70	Male	Mild dementia	AD

Modeling groups were requested (1) to submit—prior to the workshop—transparent information on their model characteristics (structure, inputs, and assumptions) and model outcomes from applying the specific benchmark scenario over 10- and 20-year time horizons; (2) to participate in the workshop and contribute to the discussions; and (3) to provide submissions after the meeting on a brief set of questions that sought to address the perceived benefits and challenges associated with the cross-comparison and to elicit insights on how their model compared to others.

Details on the workshop were shared with all participants and published on the IPECAD website (www.ipecad.org; see Supplementary Material 3, 4, and 5).

Data were collected centrally in a structured spreadsheet and an overview of the results from the different models was drafted and shared prior to the workshop by the IPECAD Modeling Workshop Group. This overview report included figures presenting model outcomes on mean time and occupancy in each disease state over 10- and 20-year time horizons, along with model descriptions and pre-recorded presentations from each modeling group. After the meeting, modeling groups were requested to provide clarifications or perform additional analyses (e.g., using amendments to input parameters).

Model submissions included a wide variation in inputs for costs and health-related quality of life (QALY) weights that varied in setting (e.g., clinical vs general population) and/or perspective (e.g., health care perspective vs societal perspective). Therefore, outcomes and results relating to costs, QALYs, and incremental cost-effectiveness ratios were considered insufficiently comparable and were not presented.

3 |. RESULTS

3.1 |. Model characteristics and outcomes

Twelve modeling groups contributed data to the workshop (see Supplementary Material 3 for a list of modeling groups and attendees). Eight made a submission for Scenario A (MCI), two for Scenario B (mild dementia), and two for both. Table 3 summarizes the characteristics of the models and references publications containing more detailed information. Nine models had a decision-analytic purpose, two focused on prediction of health and policy outcomes, and one on both. Most were cohort state-transition models (IPECAD,²¹ SVEDEM,²² KP,²³ DAVIS,²⁴ CPEC,²⁵ CEM²⁶) and others (symptom-based) were discrete-time microsimulations (JUTKOWITZ,²⁷ HERRING,²⁸ ADACE,²⁹ FEM³⁰) or discrete-event simulations (MISCAN,³¹ BASQDEM³²).

TABLE 3.

Characteristics of the models

Characteristic	IPECAD	SveDem	KP	FEM	Herring	ADACE	BASQDEM	MISCAN	Davis	CPEC	Jutkowitz	CEM
Abbreviation	International Pharmacoeconomics Collaboration Alzheimer’s Disease	Swedish Dementia Registry	Kungsholmen Project	Future Elderly Model	(author name)	Alzheimer’s Disease Archimedes condition event simulator	Basque dementia model	MIcrosimulation SCreening ANalysis (Dementia)	(author name)	Care Policy Evaluation Centre	(author name)	n/a
Developer(s) who were present during the workshop	CGa, AG,AK, AW, AS,BW, LJ, RH	AWa, RH	ASa, AW	BTa, JH	WLHa	ATa, JM	JM, MS-G	CBa, IdK	MD, SJ, REa,	RAa, RW, RH	EJ, LMLa, PS	MBa, MH
Reference	21	22	23	30	28	29	32	31	24	25	27	26
Starting in	MCI due to AD	MCI due to AD	MCI due to AD	MCI	MCI due to AD	MCI due to AD	MCI due to AD	MCI due to AD	MCI due to AD	MCI due to AD	Mild dementia	Mild AD dementia
Goal	Decision-analytic	Decision-analytic	Decision-analytic	General population health and policy outcomes	Decision-analytic	Decision-analytic	Decision-analytic	General population health and policy outcomes	Decision-analytic	Decision-analytic	Decision-analytic and general population health and policy outcomes	Decision-analytic
Original aim	Estimate cost-effectiveness of treatment in the predementia AD in a transparent open-source way.	Estimate Costs and QALYs of AD patients across disease progression and illustrate the potential health-economic effects of a hypothetical AD-DMT.	Estimate potential economic impact of DMT in AD.	Estimate health and policy outcomes in the US population. Focused on risk factors, chronic conditions, functional limitations, mortality, and health-related economic outcomes.	Estimate the potential cost-effectivenes of disease-modifying treatments targeting the predementia stages of AD.	Economic evaluation on various interventions through close tracking of disease progression and the related clinical outcomes.	Assess the cost-utility of Souvenaid compared to placebo in patients with prodromal AD under the conditions applied in LipiDiDiet trial.	Evaluate the impact of primary (risk factors), secondary (population screening and treatment), and tertiary (care and cure) prevention strategies on population trends in dementia incidence and mortality.	Estimate annual progression rates across the AD continuum and evaluate the impact of a delay in MCI due to AD on the trajectory of AD dementia and clinical outcomes.	Using the threshold of £20,000 per QALY to estimate the maximum price of a hypothetical therapy which reduces transition to AD, and within AD from stage to stage, by given percentages.	Predict the lifetime US cost of dementia and estimate the cost-effectiveness of nondrug interventions that ameliorate declines in physical function, dementia-related behaviors or rate of nursing home transition.	To assess the cost-effectiveness of a disease modifying treatment in patients with mild to moderate AD.
Model type	cohort state-transition	Cohort state-transition	Cohort state-transition	Dynamic microsimulation	Patient-level simulation	Individual-patient simulation (DICE)	Discrete-event simulation	Population-based microsimulation and time-to-event model	Cohort state-transition	Cohort state-transition	Microsimulation	Cohort state-transition
Characteristics of the data from which the starting population was obtained	See table items ‘progression during MCI’	See table items ‘progression during MCI’	See table items ‘progression during MCI’	Representative U.S. based sample from Health and Retirement Survey	Sampled from characteristics in Vos [2013] (preclinical AD stage 3) and Petersen [2010] MCI	See table items ‘progression during MCI’	LipiDiDiet trial [Soininen, 2017]	(non)amnestic and amyloid positive MCI, onset age calibrated to Rotterdam Study dementia incidence.	See table items ‘progression during MCI’	See table items ‘progression during MCI’	Randomly determined from published US incident statistics and other observational data [Jutkowitz, 2017].	Patients with Mild AD in the community from GERAS study (France, Germany, and UK).
Progression during MCI:
Characteristics of underlying empirical data	U.S. NACC memory clinic setting data from 2005-2017 on 3553 individuals with community-dwelling prevalent and incident MCI due to AD, with cause defined as primary or contributing cause (i.e., 2011 NIA-AA criteria or NINCDS-ADRDA); age range = 60-89).	Estimate [Vos, 2015] based on multi-country (EU/U.S.) memory clinic research setting sample of IWG-1 based diagnosis of prodromal AD-MCI with mean 2.4-year follow-up to AD-type dementia.	Estimate [Vos, 2015] based on multi-country (EU/U.S.) memory clinic research setting sample of IWG-1 based diagnosis of prodromal AD-MCI with mean 2.4-year follow-up to AD-type dementia.	HRS data restricted to starting ages 69-70 (mean 70.1, SD 0.54), of which 39% are male, in both community and nursing homes with MCI (operationalized by TICS score of 7-11 points or proxy judgement).	Symptom levels leading up to dementia: MMSE: U.S. population norms, the French Paquid cohort (9-year study of 215 future AD subjects), and ADNI (398 patients with MCI); NPI: 50 patients with AD and 40 controls; DAD: 196 patients with AD, 70 patients with MCI, and 75 controls.	1735 patient-level data from ADNI dataset collected from 2003 to 2018.	LipiDiDiet trial: memory clinic individuals with prodromal AD, IWG-1 criteria: episodic memory disorder and a positive biomarker (either CSF, MRI, or PET). Mean age = 71, 50% male, of 311 participants. Mean starting MMSE was 26.4, 60% APO-e4.	MCI onset calibrated on dementia incidence = Rotterdam Study dementia [Wolters, 2017] incidence by age (68,367 person-years); MCI duration = pooled 5 memory clinic cohorts (ADC, DESCRIPA, ADNI, AIBL, Gothenburg H70) [Vermunt, 2019], n = 729 (57% female, mean age = 72 (MCI defined as amyloid accumulation and (non)amnestic MCI.	U.S. NACC memory clinic setting data from 2005-2014 on 18,103 individuals in community dwelling and assisted living residences with ≥1 visit while ≥65 years, impaired not MCI, MCI, or dementia and a primary etiologic diagnosis of probable or possible AD. Baseline mean age 76, 57% female, 19% (N = 3370) in MCI due to AD.	Estimate [Vos, 2015] based on multi-country (EU/US) memory clinic research setting sample of IWG-1 based diagnosis of prodromal AD-MCI with mean 2.4-year follow-up to AD-type dementia.	n/a	n/a
Model/function/estimate	Weibull function; mean = 2.7, medium = 2.1, maximum follow-up = 10.4 years.	Observed 3-year survival probability was 50% and converted to 20.6% annual transition probability, applied indefinitely.	Observed 3-year survival probability was 50% and converted to 20.6% annual transition probability, applied indefinitely.	Probit model for 2-year incidence of dementia, conditional on having MCI. We follow natural life-time progression of select individuals in 2-year increments.	Exponential curves: MMSE: baseline value = 1.7 below age- and education-adjusted norms; baseline rate of change = −0.7 per year, annual increase in rate of change = 23.0% (NPI: value = 3.9, rate = 0.9, increase = 24.7% DAD: value = 95.5, rate = −1.46, increase = 38.7%.	Interconnected linear mixed modeling disease equations for key AD biomarkers and various relevant patient-level scales of cognition, behavior, function, and dependence (disease progression is decoupled from AD severity).	Mixed model reproducing CDR-SB progression by rewriting the formula to get a time to event for mild dementia and moderate-severe dementia.	Time-to-event model based on mean MCI duration (male = 4.8, female = 4.0), multiplied by gamma distributed random variable. Onset before 65: Gamma (5, 0.25) Onset 65-69: Gamma (3, 0.4) Onset after 70: Gamma (1.75, 0.7).	multivariate ordered probit for age-specific transition probabilities to each health state, controlling for health state at the prior visit and current age. A covariate for days elapsed since the prior visit was included to adjust for variation in visit frequency.	Observed 3-year survival probability was 50% and converted to 20.6% annual transition probability, applied indefinitely.	n/a	n/a
Input and/or simulation-observed conversion rate from MCI to dementia (year 1–3)	Input: year 1 = 19%, year 2 = 14%, and year 3 and further = 8-9%.	Input: 20.6% (indefinitely).	Input: 20.6% (indefinitely).	Input: 14.6% (2-year horizon).	Observed: from Markov trace model outputs (conditioned on survival): 6.5% in year 1; 17.9% in year 2; 34.5% in year 3.	Observed: from model outputs (18.7% with AD after 2 years).	Observed: 41% at year 2.	Observed: Conversion rate (MCI to dementia) in year 1 = 18%, year 2 = 28%, and year 3 = 13%.	Input (predicted from ordered probit age 66): 76.4% to MCI, 22% to mild AD, 0.1% to moderate AD, < 0.1% to severe AD.	Input: 20.6% (indefinitely).	n/a	n/a
Landing state from MCI to dementia (if model reflects states) or connection of continuous outcomes across disease stages (if model reflects continuous scales).	Landing in 27 states distributed as empirically observed from the Weibull analysis (n = 760; 71% mild) and assumed community-dwelling.	All mild dementia.	All mild dementia.	n/a	Dementia: if MMSE ≤1.5 SD below age- and education-adjusted norms, MMSE annual change ≥0.5 SD faster than age- and education-adjusted norms, and DAD ≤93.	Patients progress to mild AD from MCI due to AD once their estimated CDRSB goes above 4.5 during a model cycle.	Patients experience first mild dementia (CDR-SB cut-off of 4.5), then moderate dementia (CDR-SB cut-off of 9.5).	All in mild dementia.	Mild, moderate, and severe AD landing states as predicted from the ordered probit regression.	All in mild dementia.	n/a	n/a
Progression during dementia:
Characteristics of underlying empirical data	Same data source as for progression in MCI (NACC). 4423 participants (age range = 60-89) with prevalent and incident dementia and AD primary or contributing cause and at least 2 consecutive assessments.	Swedish dementia registry (SveDem) any-type dementia (memory clinic, GP, nursing home); 53,880 persons (2007-2016); baseline age = 80, SD = 7.8; 59% female; MMSE = 20.9) [Handels, 2020; Wimo, 2020].	The transition probabilities including mortality figures from general population-based Kungsholmen project in Sweden [Fratiglioni, 1992; Jonsson, 2011].	n/a (model does not reflect staging of dementia). Progression reflected by mortality (see below).	Published symptom trajectories [Guo, 2014]: MMSE: CERAD and 7 donepezil trials; NPI: donepezil trials; DAD: DADE study (249 community/institution patients with possible/probable AD).	Same data source as for progression in MCI (ADNI).	Same data source as for progression in MCI (LipiDiDiet).	Rotterdam Study dementia incidence and survival by age (68,367 person-years).	Same data source as MCI progression (NACC). All patients in AD health states have diagnosis of dementia at ≥1 visit AND primary etiologic diagnosis of probable or possible AD at any time.	Mild to moderate: assumption on nursing home admission after mean 6 years from diagnosis [Brookmeyer, 2007]; Moderate to severe: NACC 3,852 patients aged 77 possible/probable AD [Spackman, 2012].	U.S. NACC community-dwelling newly diagnosed (incident) individuals with any-type dementia, mean age = 80, 55% male [Jutkowitz, 2017].	1495 patients from GERAS 36-month observational study (FR, DE, UK, 2010) community dwelling, probable AD (NINCDS-ADRDA), MMSE≤26, age 78, 38% mild, 32% moderate, 31% moderately severe/severe [Wimo, 2013].
Model/function/estimate	Three ordered probit functions (3-class MMSE, FAQ, and NPI-Q), each including previous year’s categorized MMSE, FAQ, and NPI-Q. Predicted transitions from each function were multiplied to obtain 27 state transition probability matrix.	Ordered probit on 23,146 annualized transitions, adjusted for etiology (AD-specific), and inverse probability of censoring weights to adjust for selective drop-out.	n/s	n/a	MMSE: piecewise linear regression equation predicting annual change rate; NPI: linear regression predicting change since diagnosis; DAD: linear regression predicting DAD from MMSE and NPI [Guo, 2014].	Same as MCI model/function	Mixed model for CDR-SB progression to mild and moderate-to-severe dementia considering CDR-SB longitudinal data to extrapolate progression to the longer term. Death from dementia and other causes behave as a competitive risk.	Time-to-event model based on mean dementia duration (male: 7.5; female: 9.9) multiplied by a Gamma distributed random variable to introduce random variation between the simulated individuals.	Same as MCI model/function.	Annual transition probability: Mild to moderate = 0.167; moderate to severe = 0.20 (rounded ordered probit predicted probability of 0.214 at average demographics characteristics). No variation by age, time in stage or any other factor.	Mixed regression predicting MMSE, FAQ, and NPI-Q. Monthly nursing home transition probability from Weibull function including MMSE, FAQ, and NPI (exit probability from Medicare data), total nursing home time calibrated to Medicare.	Time to events function for institutionalization and death. Regression function for change over time in MMSE, iADL, and bADL [Belger, 2019; Bond, 2012].
Categorization for mild, moderate, and severe dementia (as applied for the benchmark cross-comparison results tables)	MMSE: mild = 21-30, moderate = 10-20, severe = 0-9; FAQ = 0-8, 9-23, 24-30; NPI-Q = each item≤1, each item≤2, at least one item = 3.	MMSE: mild = 21-30, moderate = 10-20, severe = 0-9.	MMSE: mild = 18-23, moderate =10-17, severe = 0-9.	n/a	n/a	CDRSB: mild = 4.5-9.5, moderate = 9.5-16.5, severe = ≥16.5.	CDR-SB: mild = 4.5-9.5, moderate/severe = > 9.5.	n/a	CDR: mild < 2, moderate = 2, severe = 3.	MMSE: mild = 21-30, moderate = 10-20, severe 0-9.	MMSE: mild = > 19, moderates = 19-10, severe = < 10 (only done for this analysis, model include disease stage in these terms).	MMSE: mild = 21-26, moderate = 15-20, moderately severe/severe = < 15 (only for baseline severity stratification).
Mortality:
Background mortality	Age-specific from 2013 U.S. life table.	Age-specific from 2000-2016 Swedish life table.	n/a	2-year mortality rates from HRS up to 2016, controlling for a broad set of risk factors.	Age- and sex-specific from 2010 U.S. life table.	Age-specific from 2017 U.S. life table.	Time until death from other causes using Gompertz function for each sex and Spanish age mortality rates.	Age-specific life table for 1950 birth cohort.	Not explicitly modeled.	Age-specific from 2016-2018 England life table.	n/a	Time to death based on GERAS data.
MCI-related mortality	Assumed same as background mortality.	Assumed same as background mortality.	Static mortality rate of 73-year-old person from the 2005 Swedish life table.	Included as a predictor in the HRS-based mortality model.	HR applied to background mortality [Wilson, 2009] from Chicago Health and Aging Project.	Assumed same as background mortality.	Assumed same as background mortality.	Assumed same as background mortality.	Health state specific mortality rates estimated using a probit function and the NACC data.	Assumed same n/a as background mortality.	n/a	n/a
Dementia-related mortality	RR reflecting excess mortality in mild, moderate, and severe dementia [Andersen, 2010] Danish population cohort (1990’s) adjusted for age and CDR-staging.	HR based on Weibull survival function by dementia severity (SveDem data); very mild dementia as reference (see dementia progression).	Static severity-specific mortality probability as observed in the KP cohort [Jonsson, 2011]. KP Swedish general population study age 75-78 followed up to 9 years.	Endogenously determined in the microsimulation based on a range of comorbidities and risk factors.	HR applied to background mortality (same source as MCI mortality HR).	RR reflecting excess mortality in mild, moderate, and severe dementia [Andersen, 2010] Danish population cohort (1990’s) adjusted for age and CDR-staging.	Modeled as parameterized time from the onset of moderate-to-severe dementia to death [Dodge, 2003: table SM1].	Mortality due to dementia is determined by the duration of the MCI, and dementia stages).	Health state/severity specific mortality rates estimated using a probit function and the NACC data.	Excess mortality per annum over background mortality in moderate and severe dementia [Brookmeyer, 2007].	Background mortality not separated from dementia-related mortality, total mortality based on age of dementia onset, sex, and race [Mayeda, 2017].	Time to death as function of age, cognition, function, and behavior, based on GERAS data. Model assumed no mortality benefit for the intervention.
Input and/or simulation-observed relative risk for dementia-related mortality	Input: mild (2.92), moderate (3.85), and severe (9.52).	Input: mild (1.3), moderate (2.4), and severe (4.3).	n/s	Input: 2-year risk MCI = 16.5% dementia increases marginal risk by 4.8 percentage points.	Input: MCI due to AD (1.48) and AD dementia (2.84).	Input: mild (2.92), moderate (3.85), and severe (9.52).	Observed: mean survival 9.0 years, representing 29% reduction versus 12.7 years in general population.	Observed: RR dementia per age group: 50-59: 5.0, 60-69: 4.3, 70-79: 4.8, 80-89: 3.2, 90-99: 17	Input (predicted value from probit age 70 relative to MCI): mild (3.29) moderate (6.59) and severe (11.88).	Input: excess to mortality probability is 0.11.	See previous.	See previous.
Treatment effect:
Implementation of effect	MCI to AD-dementia rate was multiplied with 0.70.	MCI to AD-dementia rate was multiplied with 0.70.	MCI to AD-dementia rate was multiplied with 0.70.	MCI to AD-dementia rate was multiplied with 0.70.	30% reduction in annual rates of change for MMSE, NPI, and DAD.	Amyloid level calibrated to obtain 30% reduction in AD conversion rate in 2 years.	Time from MCI to mild or moderate dementia multiplied by 1.3.	MCI duration prolonged with 30% of treatment duration.	Transition rates from MCI to each of AD-dementia states multiplied by 0.70.	MCI to AD dementia rate was multiplied by 0.70.	30% reduction in 5-year proportion of transitioning to moderate dementia.	30% reduction in change in MMSE and ADL over 18 months.
Discontinuation	10% per year up to 5 years or conversion to dementia.	10% per year up to 5 years or conversion to dementia.	n/s	20% every 2 years until dementia conversion, or up to 5 years.	10% per year up to 5 years or conversion to AD dementia.	10% per year up to 5 years or conversion to AD dementia.	Treatment effect disappeared after onset moderate dementia.	For year 1-5: 10%, 9%, 8.1%, 7.3%, 6.6%; or transition to mild dementia.	10% per year up to 5 years or conversion to dementia.	10% per year up to 5 years or conversion to dementia.	10% per year above mortality disease status change. At 5-years on treatment.	10% per year until onset moderate dementia.
Indirect effect after discontinuation	Rate reverts to control after 5 years.	Rate reverts to control after 5 years.	n/s	Rate reverts to control after 5 years, accounting for better health due to treatment.	Symptom rates revert to what would have experienced with natural history; gains in actual symptom levels are maintained.	Natural Amyloid disease progression after discontinuation.	Treatment effect disappeared after moderate dementia progression.	Indirect effect on mortality (due to prolonged duration in MCI/dementia) remains.	Rate reverts to control after 5 years.	Rate reverts to untreated after 5 years.	Symptoms return to same level as control group.	Treatment effect extrapolated to 7 years and then maintained; gained progression delay remains over lifetime.
Other characteristics
Cycle/update time and timing of events	Annually.	Annually.	Annually.	2-year cycle, with half-cycle correction.	Symptom levels updated and AD dementia diagnostic criteria checked annually.	Disease equations evaluated every 6 months, other events (e.g., mortality) on continuous scale.	Discrete time until mild dementia and moderate dementia.	MCI, dementia, and mortality timings on continuous scale.	Annually.	Annually.	Monthly.	6-month interval.
Half-cycle correction of mean outcomesb	Yes	Yes	n/s	Yes	No	No	n/a	n/a	Yes	Yes	Yes	No
Half-cycle correction of proportions in each stateb,c	Yes	Yes	n/s	No	No	No	No	No	Yes	Yes	Yes	No
Assumptions	1) separate models for MCI and dementia assumed connected with same underlying progression speed; 2) ordered probit assumes proportional effects of predictors and no interactions were tested; 3) MCI dementia progression assumed independent from age; 4) cognition, function, and behavior are predictors for one another’s next state. 6) mortality life table includes dementia-related death.	1) Effect only apply when treated; 2) Conversion risk independent from age; 3) Costs, outcomes (QALYs), mortality rate dependent on age; 4) ordered probit assumed proportional effects of AD versus other/unspecified dementias and no interactions were tested; 5) mortality life table includes dementia-related death.	1) MCI was assumed to only convert to mild, mild could convert to moderate and severe, and moderate only to severe disease; 2) No backwards transitions were allowed.	1) effect lasts only while being treated, natural progression following discontinuation; 2) benefits of intervention are calculated in QALY terms, cost-effectiveness may not take into account cost savings or cost increase due to the intervention.	1) predementia exponential curves assumed 15 years between normal cognition and AD dementia diagnosis; 2) diagnostic thresholds for AD dementia by clinical diagnostic guidelines [McKhann, 2011] and calibrated based on mean symptom levels at diagnosis; 3) biomarker positivity only reflected in baseline symptom levels; 4) baseline patient characteristics APOE and AD family history not included because mixed evidence of independent effect on progression in biomarker positive patients; 5) symptomatic treatment use not considered in DMT arm, it was initiated in the standard of care arm at AD dementia diagnosis but did not influence disease progression.	1) Disease progression modeling was decoupled from disease staging; 2) The model relies on a noninterventional study to represent disease progression; 3) Modifying the components of the disease pathophysiology that are predictive of future disease progression in natural history is causative and their causal role is reversible.	CDR-SB lineal progression until moderate dementia.	1) dementia stage durations are not age-dependent, the age-dependency observed in literature is assumed to be present due to risk of dying due to competing risks. Stage durations for all age groups are taken from the youngest age group of Vermunt, 2019; 2) progression in the different stages is correlated within a certain individual (individual with fast progressing MCI will also have fast progressing dementia); 3) no transitions back to previous stages.	1) one singular model for MCI and three Alzheimer’s disease dementia states; 2) age-specific transitions; 3) mortality risk was age and health-state dependent and based on observed mortality in the NACC data as opposed a combination of population level mortality and dementia specific hazard ratios; 4) the model did not allow for reversion to less severe health states; 5) half cycle correction impacts first year of treatment but does not impact 5th year of treatment.	1) Ratchet imposed: no transition back to a less severe state. 2) Markov model which included MCI as a preliminary state before AD. 3) In AD analysis model is set to start at the AD mild dementia stage.	1) effect lasts only while being treated, natural progression following discontinuation; 2) treatment has no effect on mortality; 3) mild, moderate, and severe dementia defined only for the cross-model comparisons exercise and based on MMSE thresholds reported in the literature; 4) model did not incorporate utility estimates.	1) treatment effect extrapolated for both cognition and function beyond the initial 18 months as a linear function 2) costs and utilities modeled as function of time before Institutionalization, patients with predicted time < 6 months have cost/utility equal to those institutionalized. 3) institutionalized patients have fixed cost and utility while they remain in that health state. 4) assume treatment does not have a direct impact on mortality.

Abbreviations: AD, Alzheimer’s disease; ADNI, Alzheimer’s Disease Neuroimaging Initiative; bADL, basic activities of daily living; CDR-SB, Clinical Dementia Rating scale Sum of Boxes; CSF, cerebrospinal fluid; DAD, Disability Assessment Dementia; DICE, Discretely Integrated Condition Event; DMT, disease-modifying treatment; EU, European Union; FAQ, functional activities questionnaire; HR, hazard ratio; HRS, Health and Retirement Study; iADL, instrumental activities of daily living; IWG, International Working Group; KP, Kungsholmen project; MCI, mild cognitive impairment; MMSE, mini-mental state examination; MRI, magnetic resonance imaging; n/a, not applicable; n/s not stated; NACC, National Alzheimer’s Coordinating Center; NIA-AA, National Institute on Aging-Alzheimer’s Association; NINCDS-ADRDA, National Institute of Neurological and Communicative Disorders and Stroke and the Alzheimer’s Disease and Related Disorders Association; NPI, Neuropsychiatric Inventory; PET, positron emission tomography; QALY, quality-adjusted life year; RR, relative risk;SD, standard deviation; TICS, Telephone Interview for Cognitive Status; US, United States.

^aPrimary modeler to implement the benchmark scenario and generate the results.

^bAs applied for the benchmark cross-comparison results tables.

^cThat is, outcome reflecting mid-year estimate.

References mentioned in the table:

Andersen K, Lolk A, Martinussen T, Kragh-Sørensen P. Very mild to severe dementia and mortality: A 14-year follow-up – The Odense study. Dement Geriatr Cogn Disord. 2010;29(1):61-7. https://doi.org/10.1159/000265553. Epub 2010 Jan 27. PMID: 20110702.

Belger M, Haro JM, Reed C, Happich M, Argimon JM, Bruno G, Dodel R, Jones RW, Vellas B, Wimo A. Determinants of time to institutionalisation and related healthcare and societal costs in a community-based cohort of patients with Alzheimer’s disease dementia. Eur J Health Econ. 2019 Apr;20(3):343-355. https://doi.org/10.1007/s10198-018-1001-3. Epub 2018 Sep 3. PMID: 30178148; PMCID: PMC6438944.

Bond M, Rogers G, Peters J, Anderson R, Hoyle M, Miners A, Moxham T, Davis S, Thokala P, Wailoo A, Jeffreys M, Hyde C. The effectiveness and cost-effectiveness of donepezil, galantamine, rivastigmine and memantine for the treatment of Alzheimer’s disease (review of Technology Appraisal No. 111): a systematic review and economic model. Health Technol Assess. 2012;16(21):1-470. https://doi.org/10.3310/hta16210. PMID: 22541366; PMCID: PMC4780923.

Brookmeyer R, Johnson E, Ziegler-Graham K, Arrighi HM. Forecasting the global burden of Alzheimer’s disease. Alzheimers Dement. 2007 Jul;3(3):186-91. https://doi.org/10.1016/j.jalz.2007.04.381. PMID: 19595937.

Dodge HH, Shen C, Pandav R, DeKosky ST, Ganguli M. Functional transitions and active life expectancy associated with Alzheimer disease. Arch Neurol. 2003 Feb;60(2):253-9. https://doi.org/10.1001/archneur.60.2.253. PMID: 12580712.

Fratiglioni L, Viitanen M, Bäckman L, Sandman PO, Winblad B. Occurrence of dementia in advanced age: the study design of the Kungsholmen Project. Neuroepidemiology. 1992;11 Suppl 1:29-36. https://doi.org/10.1159/000110958. PMID: 1603245.

Guo S, Getsios D, Revankar N, Xu P, Thompson G, Bobula J, Lacey L, Gaudig M. Evaluating disease-modifying agents: a simulation framework for Alzheimer’s disease. Pharmacoeconomics. 2014 Nov;32(11):1129-39. https://doi.org/10.1007/s40273-014-0203-5. PMID: 25124747.

Handels R, Jönsson L, Garcia-Ptacek S, Eriksdotter M, Wimo A. Controlling for selective dropout in longitudinal dementia data: Application to the SveDem registry. Alzheimers Dement. 2020 May;16(5):789-796. https://doi.org/10.1002/alz.12050. Epub 2020 Mar 22. PMID: 32202077; PMCID: PMC7984348.

Jonsson L, Winblad B, Fratiglioni L, Wimo A. Mortality in relation to disese severity in subjects with dementia (Abstract). ISPOR; September 8- 10 2011; Mexico city 2011.

Jutkowitz E, Kane RL, Gaugler JE, MacLehose RF, Dowd B, Kuntz KM. Societal and Family Lifetime Cost of Dementia: Implications for Policy. J Am Geriatr Soc. 2017 Oct;65(10):2169-2175. https://doi.org/10.1111/jgs.15043. Epub 2017 Aug 17. PMID: 28815557; PMCID: PMC5657516.

Jutkowitz E, MacLehose RF, Gaugler JE, Dowd B, Kuntz KM, Kane RL. Risk Factors Associated With Cognitive, Functional, and Behavioral Trajectories of Newly Diagnosed Dementia Patients. J Gerontol A Biol Sci Med Sci. 2017 Feb;72(2):251-258. https://doi.org/10.1093/gerona/glw079. Epub 2016 Apr 29. PMID: 27129917; PMCID: PMC5344797.

Mayeda ER, Glymour MM, Quesenberry CP, Johnson JK, Pérez-Stable EJ, Whitmer RA. Survival after dementia diagnosis in five racial/ethnic groups. Alzheimers Dement. 2017 Jul;13(7):761-769. https://doi.org/10.1016/j.jalz.2016.12.008. Epub 2017 Feb 5. PMID: 28174069; PMCID: PMC5496783.

McKhann GM, Knopman DS, Chertkow H, Hyman BT, Jack CR Jr, Kawas CH, Klunk WE, Koroshetz WJ, Manly JJ, Mayeux R, Mohs RC, Morris JC, Rossor MN, Scheltens P, Carrillo MC, Thies B, Weintraub S, Phelps CH. The diagnosis of dementia due to Alzheimer’s disease: recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease. Alzheimers Dement. 2011 May;7(3):263-9. https://doi.org/10.1016/j.jalz.2011.03.005. Epub 2011 Apr 21. PMID: 21514250; PMCID: PMC3312024.

Petersen RC, Aisen PS, Beckett LA, Donohue MC, Gamst AC, Harvey DJ, Jack CR Jr, Jagust WJ, Shaw LM, Toga AW, Trojanowski JQ, Weiner MW. Alzheimer’s Disease Neuroimaging Initiative (ADNI): clinical characterization. Neurology. 2010 Jan 19;74(3):201-9. https://doi.org/10.1212/WNL.0b013e3181cb3e25. Epub 2009 Dec 30. PMID: 20042704; PMCID: PMC2809036.

Soininen H, Solomon A, Visser PJ, Hendrix SB, Blennow K, Kivipelto M, Hartmann T; LipiDiDiet clinical study group. 24-month intervention with a specific multinutrient in people with prodromal Alzheimer’s disease (LipiDiDiet): a randomised, double-blind, controlled trial. Lancet Neurol. 2017 Dec;16(12):965-975. https://doi.org/10.1016/S1474-4422(17)30332-0. Epub 2017 Oct 30. PMID: 29097166; PMCID: PMC5697936.

Spackman DE, Kadiyala S, Neumann PJ, Veenstra DL, Sullivan SD. Measuring Alzheimer disease progression with transition probabilities: estimates from NACC-UDS. Curr Alzheimer Res. 2012 Nov;9(9):1050-8. https://doi.org/10.2174/156720512803569046. PMID: 22175655; PMCID: PMC3690579.

Vermunt L, Sikkes SAM, van den Hout A, Handels R, Bos I, van der Flier WM, Kern S, Ousset PJ, Maruff P, Skoog I, Verhey FRJ, Freund-Levi Y, Tsolaki M, Wallin ÅK, Olde Rikkert M, Soininen H, Spiru L, Zetterberg H, Blennow K, Scheltens P, Muniz-Terrera G, Visser PJ; Alzheimer Disease Neuroimaging Initiative; AIBL Research Group; ICTUS/DSA study groups. Duration of preclinical, prodromal, and dementia stages of Alzheimer’s disease in relation to age, sex, and APOE genotype. Alzheimers Dement. 2019 Jul;15(7):888-898. https://doi.org/10.1016/j.jalz.2019.04.001. Epub 2019 Jun 1. PMID: 31164314; PMCID: PMC6646097.

Vos SJ, Verhey F, Frölich L, Kornhuber J, Wiltfang J, Maier W, Peters O, Rüther E, Nobili F, Morbelli S, Frisoni GB, Drzezga A, Didic M, van Berckel BN, Simmons A, Soininen H, Kłoszewska I, Mecocci P, Tsolaki M, Vellas B, Lovestone S, Muscio C, Herukka SK, Salmon E, Bastin C, Wallin A, Nordlund A, de Mendonça A, Silva D, Santana I, Lemos R, Engelborghs S, Van der Mussele S; Alzheimer’s Disease Neuroimaging Initiative, Freund-Levi Y, Wallin ÅK, Hampel H, van der Flier W, Scheltens P, Visser PJ. Prevalence and prognosis of Alzheimer’s disease at the mild cognitive impairment stage. Brain. 2015 May;138(Pt 5):1327-38. https://doi.org/10.1093/brain/awv029. Epub 2015 Feb 17. PMID: 25693589; PMCID: PMC5013930.

Wimo A, Reed CC, Dodel R, Belger M, Jones RW, Happich M, Argimon JM, Bruno G, Novick D, Vellas B, Haro JM. The GERAS Study: a prospective observational study of costs and resource use in community dwellers with Alzheimer’s disease in three European countries–study design and baseline findings. J Alzheimers Dis. 2013;36(2):385-99. https://doi.org/10.3233/JAD-122392. PMID: 23629588.

Wimo A, Handels R, Winblad B, Black CM, Johansson G, Salomonsson S, Eriksdotter M, Khandker RK. Quantifying and Describing the Natural History and Costs of Alzheimer’s Disease and Effects of Hypothetical Interventions. J Alzheimers Dis. 2020;75(3):891-902. https://doi.org/10.3233/JAD-191055. Erratum in: J Alzheimers Dis. 2021;80(2):905. PMID: 32390617; PMCID: PMC7369101.

Modeling groups operationalized the benchmark scenario, although in some cases this required some adjustment to their previously published model frameworks. In some cases, the model structure did not allow for the reporting of the mean time by dementia severity state.

Figure 1 presents the mean time in each state in the control group and its difference from the intervention group over a 10-year time horizon. Supplementary Material 6 provides an alternative representation of Figure 1 for mean survival and time in MCI/mild dementia, showing the control, intervention, and difference in one graph (Figure S6.1) and model outcomes over time (Figure S6.2 and S6.3).

FIGURE 1.

Model outcomes in terms of mean time (years) in disease state in the control setting (left column) and difference in mean time in terms of the intervention minus control setting (right column) over a 10-year time horizon for models in Scenario A (models starting in mild cognitive impairment [MCI]; top row) and models in Scenario B (models starting in mild dementia; bottom row). Some models combined mild/moderate/severe states (gray bars) because they were not designed to generate state-specific outcomes.

3.2 |. Cross-model variation

The mean survival restricted over a 10-year time horizon in the control group ranged from 6.7 to 8.6 years in Scenario A (models starting in MCI) and 5.4 to 7.9 years in Scenario B (models starting in mild dementia). The difference between the intervention and control groups reflected a relatively small impact of the intervention on restricted survival (ranging from no effect to 0.4 years).

The mean time in MCI in Scenario A (starting in MCI) ranged from 3.4 (DAVIS) to 5.6 years (FEM). In Scenario B (starting in mild dementia), the mean time in mild dementia ranged from 1.5 (JUTKOWITZ) to 4.2 years (CPEC). The variation in mean time in mild and moderate dementia in both scenarios was relatively large.

Broad comparisons on disease progression outcomes were possible as a number of models showed a comparable mean time in state (Figures 1 and S6.1) and state occupancy over time (Figure S6.2 and S6.3). Where marked differences occurred, we were able to discuss the likely cause in terms of model inputs or assumptions.

3.3 |. Interpretation of variation in disease progression in the control group

Some differences between models were observed. The SVEDEM model predicted a longer restricted survival than most other models. This could be explained by the relatively low mortality risks by disease severity applied in the model (e.g., mortality hazard ratios 1.3, 2.4, and 4.3 in mild, moderate, and severe, respectively, compared to other models like IPECAD and ADACE(A), which applied 2.9, 3.9, and 9.5, respectively). In the SVEDEM model, dementia mortality risks were estimated relative to very mild dementia (already at some increased mortality risk), and for the IPECAD and ADACE(A) models the risks were obtained from an analysis of persons without dementia as a reference.³³ The KP model predicted a relatively short, restricted survival. This could be explained due to the cohort-specific mortality rates in dementia (based on the KP cohort of participants 75 years of age or older) compared to most other models having implemented a life table starting at age 70 years. The reasons for the relatively large variation in restricted mean survival across models in Scenario B (2.5-year range) were less clear due to input estimates reflecting different settings and different methods applied (severity-specific relative risk applied in ADACE(B), severity-specific absolute risk in CPEC(B), log-logistic model applied to baseline clinical setting data in CEM, and any-stage dementia mortality risk based on clinical record analysis in JUTKOWITZ).

The SVEDEM, CPEC(A), KP, HERRING, and ADACE(A) models operationalized a biomarker-based definition of AD set in a clinical setting. This probably resulted in a faster natural progression from MCI to dementia as compared to, for example, the FEM model, which assumed a general population setting with any-type MCI due to a lack of data on disease pathology (see Figure S6.2, left top). The DAVIS and IPECAD model used similar data on natural progression but applied a different approach to the estimation of time-to-event data for transition from MCI to dementia, which likely explained the differences in predicted mean time in the MCI health state (i.e., 3.4 years in DAVIS using an accelerating rate with age; 4.8 years in IPECAD using a decelerating rate). The HERRING model showed a relatively low level of transitioning from MCI to dementia in the first 2 years followed by a period of high transition (see Figure S6.2, left top), pointing to the potential limitations of capturing heterogeneity in a microsimulation model based strictly on published aggregate data.

The four models in Scenario B represented three different methods for disease progression, comprising a state transition approach in CPEC(B), a time to institutionalization framework in CEM, and use of regression equations on multiple continuous measures of dementia clinical features in ADACE(B) and JUTKOWITZ. This variation in approaches led to difficulty in identifying plausible causes of the variation observed across these models.

3.4 |. Interpretation of variation in the impact of treatment on disease progression

All models predicted that the treatment effect would lead to an increase in time spent in MCI (Scenario A) or mild dementia (Scenario B) and a reduction in time spent in more advanced dementia states. However, there was some variation in the implementation of the treatment effect. The six models classified as cohort state-transition models applied a relative risk of 0.70 to the rate of disease progression, the four models classified as symptoms-based microsimulation models applied a 30%reduction in the rate or transition of symptom change (e.g., cognitive score), and the two models classified as discrete-event simulations increased the time spent in MCI by 30% (see Table 3 for details). These design choices likely produced the differences in time spent in disease states, as well as assumptions on immediate or gradual stopping of treatment effect.

The FEM model showed a relatively small benefit of the treatment. This was possibly related to the relatively slow disease progression in the general population with MCI relative to models assuming biomarker-positive MCI. The slower rate of progression left a smaller window for a relative treatment effect.

The MISCAN and BASQDEM models both represented an individual patient-level discrete-event simulation, with different implementation of the treatment effect of 30%. MISCAN prolonged only the time under treatment during MCI with 30%, and BASQDEM prolonged the full time in MCI with 30%, resulting in a lower treatment effect in the MISCAN model.

The IPECAD model showed a relatively long time in moderate dementia compared to most other models, possibly related to the definition of this particular state including combinations of mild cognition and moderate function, which, if defined by cognition only, would have been classified as mild dementia.

The CEM and JUTKOWITZ models were structured to deliver the same survival for control and treatment groups, in contrast with the CPEC(B) and ADACE(B) models where disease progression was able to reflect an indirect mortality effect.

4 |. DISCUSSION

The IPECAD Modeling Workshop Group engaged with and assessed contributions from 12 modeling teams and cross-comparisons proved feasible. A broadly similar profile of disease progression outcomes was observed across models on treatment populations of MCI and mild dementia, with 10 submissions starting in MCI (Scenario A) and four in mild dementia (Scenario B). The workshop provided a platform to discuss differences between model frameworks and their predicted outcomes.

Precisely identifying sources of variability across models was challenging, since models differed on multiple aspects simultaneously. However, alignment across models was not the objective because different approaches can be valid and provide complementary insights. The meaningfulness of cross-comparison decreases if models are too similar, for example in terms of using identical input parameters and assumptions.¹⁰ Nevertheless, for future cross-comparisons, we identified four factors as potential sources of cross-model variability that could benefit from narrower specification. Alignment on these factors could increase model comparability without forcing participants to change the character of their models or resulting in models that are too similar.

First, implementation of mortality was indicated as a probable cause of variation across model outcomes. This pertains to (1) background mortality representing the age- and sex-specific general population mortality, (2) the increased mortality risk due to AD, and (3) the (in)direct effect of treatment on mortality. In future cross-comparisons, we propose to utilize the same background life table from a specific country (1) and assumptions related to treatment mortality effect (3) to limit this source of variability.

Second, the treatment effect was defined as a 30% reduction in progression from MCI to dementia (Scenario A) or from mild dementia to moderate dementia (Scenario B). This was implemented by the models in different ways depending on the model structure. It is important to note that we did not specify the affected domain or how the treatment effect should evolve after treatment is stopped. Alternative assumptions are possible⁵: (1) symptoms will return to what would be expected in natural history (i.e., symptomatic effect); (2) treatment gains will remain, but progression will return to its natural course (i.e., disease-modifying effect); or (3) gained reduction in progression rate will continue (i.e., curative effect). In future cross-comparison scenarios, we propose to make these assumptions more explicit by expressing the treatment effect in units used as primary end points in a specific clinical trial (e.g., clinical dementia rating [CDR] sum of boxes change from baseline in the intervention group relative to the control or another trial’s outcomes).

Third, the treatment-eligible population was relatively loosely defined in order to accommodate models utilizing different data sources with varying inclusion criteria. Models operationalized the starting population of AD MCI or AD dementia by using data reflecting (1) the etiological diagnosis set in clinical practice (IPECAD, DAVIS, CEM); (2) a biomarker-driven diagnosis from published estimates in Vos et al.³⁴ (SVEDEM, KP, and CPEC) or other source (MISCAN, HERRING, ADACE, and BASQDEM); or (3) a diagnosis of any-type dementia or cognitive impairment reflecting clinical or general population setting (FEM, JUTKOWITZ). Because the starting population is closely related to disease progression, this likely affected model outcomes. In a future cross-comparison scenario, we propose to express the characteristics of the treatment-eligible population more explicitly, specifying a standardized patient population in the reference case.

Fourth, health economic input estimates (e.g., institutionalization; medical, social, and informal care costs; and patient and caregiver utility) were left unspecified. In a future cross-comparison, we propose to standardize health economic inputs for a reference case analysis, as linking them to disease progression comes with challenges and could drive variation in predicted model outcomes.

4.1 |. Strengths and limitations

Participants indicated that the workshop provided them with potential (common) areas for future model development and improved their understanding of differences across models. They expressed appreciation for the transparency and collegiality of the collaboration. Organizers experienced high participation engagement and an interest by most participants to continue in supporting future events.

Our cross-comparison approach had several limitations: (1) no criteria were employed to judge the significance or relevance of cross-model differences; (2) alignment across models is not necessarily a reflection of model validity, as similar outcomes across models might result from using, for example, the same invalid data source; (3) in contrast to predictive or descriptive modeling of real-world evidence, no gold standard of truth was available to benchmark outcomes; (4) multiple differences between models could have effects in opposite directions and thus eliminate differences; (5) the results are based on a non-systematic identification and selection of models and participants, and might not be generalizable to all decision-analytic models in AD; and (6) half-cycle corrections were applied only if applicable to the model and could have led to variability in the output. Therefore, we did not seek to rank or otherwise qualitatively assess individual model performance. Nevertheless, we believe the models represent a rich variety in terms of structure and data sources, the participants originate from leading research groups, and the workshop allowed for a valuable discussion on the possible causes of variation in model outcomes. Furthermore, a gold data standard (i.e., clinical trial with long-term reference data) is not available in AD, which makes comparative modeling even more vital, such as in epidemiology.^35,36

We collected model outcomes on a highly aggregated level (i.e., symptoms categorized as mild, moderate, or severe dementia) to allow a comparison between simple (e.g., categories from composite CDR) and complex (e.g., combined Mini-Mental State Examination [MMSE] neuropsychiatric inventory and disability assessment dementia/functional activities questionnaire scales on continuous level) structures (see Table 3). Validated cross-mappings between single and composite outcome measures are needed to increase the comparability across outcomes.³⁷

Finally, not all models were designed to implement our benchmark scenario, and thus their results do not reflect model validity.

4.2 |. Implications

We expect that the workshop results will increase the understanding of alternative designs and structures when modeling AD and related dementias, which may facilitate further developments in this field. This specifically applies to the methods and/or data used to implement disease progression, mortality, and intervention effect, some aspects of which have received relatively little attention to date. This is important to increase the trust in these models, which is necessary for an appropriate assessment of the cost-effectiveness of interventions for people affected by AD and related dementias.

We propose a future effort to cross-compare models with increased model alignment in four aspects: mortality, intervention effect, population, and health economic input estimates. Based on the results of our work we believe this is feasible, further contributes to model understanding, and does not jeopardize the character of the models. In addition, we recommend following guidance for multi-model comparisons³⁸ to increase the methodological quality of cross-comparison. At last, we recommend transparent and open-source access to models to further increase their understanding.³⁹

5 |. CONCLUSION

The IPECAD modeling workshop collaboration is intended as a step toward increasing the transparency and credibility of health economic models in AD and related dementias. The model cross-comparison results offered a better understanding of models and suggested further validation of methods related to structure, disease progression, mortality, and intervention effect.

RESEARCH IN CONTEXT.

1. Systematic review: Recent (systematic) reviews assessed Alzheimer’s disease (AD) models, and discussed limitations and variation across them. However, comparison of model outcomes has not been reported.

2. Interpretation: Our findings can be considered a step toward increasing the transparency and credibility of health economic models in AD and related dementias, which led to better understanding of models in terms of structure, disease progression, mortality, and treatment effect.

3. Future directions: We propose a future effort to cross-compare models in AD and related dementias, with increased model alignment in four aspects: mortality, treatment effect, population, and health economic input estimates. Based on the current results we believe this is feasible, further contributes to model understanding, and does not jeopardize the character of the models. In addition, we recommend transparent and open-source access to models to further increase their understanding.