SMART Planning of Early-Stage Health Economic Decision Models for Mechanism-Based Precision Treatments in Rare Cancers: An Application and Tool Update

Abstract

The Systematic Model adequacy Assessment and Reporting Tool (SMART) was developed to help ensure that health decision-analytic models (DAMs) are fit for purpose, by listing key model features for which ideal modelling choices are identified. Then the user is required to report and justify any deviations from the ideal, and the consequences of these deviations for model validity and transparency. SMART can be particularly valuable in early development settings, where limited data and resources often require model simplifications. This study illustrates the application of SMART in developing model plans in such early-stage settings and examines whether updates to the tool are needed. The planned DAMs are for evaluating precision and combination treatments in two rare cancers: advanced thyroid cancer and diffuse midline glioma. Using health economic guidelines, literature, and clinical and pre-clinical expert co-author input, SMART was completed and the models were conceptualized. Deviations from ideal modelling choices included assuming homogeneous populations, adopting narrower model perspectives, omitting diagnostic pathways, and relying on simplifying assumptions. SMART highlighted these deviations, indicating a low-to-moderate impact on model validity and variable impact on transparency. Updates to SMART included refining the decision-context section, adding guiding questions and additional space for supporting resources, expanding instructions on justifying deviations and assessing their consequences, and adding a report-generating function. Using SMART enabled a systematic definition of the decision context and supported consensus between model analysts and clinical and pre-clinical experts on clear, well-justified model plans that are adequate for this resource-constrained development stage.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1007/s40273-026-01634-z.

Key Points for Decision Makers

We applied the Systematic Model adequacy Assessment and Reporting Tool (SMART) to plan two early-stage decision-analytic models to perform headroom analysesfor precision combination treatments in rare cancers, to demonstrate its applicability in resource-constrained settings. SMART systematically defined the decision context and ideal modelling choices, while facilitating structured early engagement with clinical and pre-clinical experts. The tool also highlighted the deviations from the ideal modelling choices that were made due to early-stage constraints and their consequences for model validity and transparency.

Based on these applications, several updates were made to SMART. These include the addition of guiding questions for each model feature to support stakeholder discussions and identifying ideal modelling choices, the expansion of guidelines for reporting and justifying modelling deviations, the incorporation of additional space for documenting expert input and supporting resources, and a new function to export SMART inputs as  pdf and summary reports.

SMART, with its add-ons, helps ensure that decision-analytic models remain adequate for healthcare decision-making even with scarce data and resources. It clarifies deviations from the ideal model and their consequences, providing stakeholders with a shared understanding of necessary trade-offs and which deviations should be revisited as new evidence becomes available.

Introduction

We have recently developed the Systematic Model adequacy Assessment and Reporting Tool (SMART) to support the structured and transparent development, review, and assessment of decision-analytic models (DAMs) in health economic evaluation [1]. These models typically encompass the comparison of alternative health interventions in terms of their costs, health outcomes, and cost-effectiveness, and their development requires making many modelling choices that simplifies complex reality [2, 3]. SMART is intended to support clear reporting and justification of these modelling choices and assess their consequences for model validity and transparency. This can help in developing DAMs that are fit for purpose in supporting decision-making, and only as complex as needed. SMART can be used by DAM developers as well as DAM reviewers (e.g. regulatory bodies, health technology assessment [HTA] agencies, and peer reviewers) and DAM users (e.g. decision-makers, policymakers, clinicians, government agencies, and healthcare providers). SMART version 1.0 is publicly available at https://osf.io/4j8qx/?view_only=7c32744f833845a596629e467f7c71ac. However, real-life applications of the tool to date remain limited.

We developed SMART by first reviewing key literature and guidelines on decision-analytic modelling, such as the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Task Force reports. We then translated these findings into a structured framework of key model development steps, defining model features for which simple and complex modelling choices can be made, and specifying the consequences of deviations from decision-context requirements. Next, we designed the tool interface in Microsoft Excel and evaluated its clarity and completeness through interviews and workshops with HTA experts, including modellers, reviewers, and users. Based on this feedback, we refined the tool, added a glossary and user guidance, and applied SMART to a case study in treatment-resistant hypertension. SMART consists of three sections. Section 1 includes a comprehensive list of DAM features where different modelling choices can be made. Section 2 enables users to make modelling choices for each feature, first based on the ideal model required by the decision context and, next, their own choices given data and resource availability in that context. Section 3 requires users to assess the consequences of deviations from the decision-context requirements for validity and transparency and to justify the rationale behind their deviations. The structured SMART approach to making and assessing modelling choices helps ensure, when developing, reviewing, or analysing DAMs, that they are adequate for informing decisions. This is important because DAMs are inevitably a simplification of complex real-world systems [3], and often deviate from what might be considered the ideal model for a given decision context. In settings with limited time, data, or resources, deviations become a necessity and not just a preference.

An example of such a setting is early in the drug development process. Novel drug development and repurposing approaches are increasingly relying on network pharmacology and combination therapies that leverage information on protein–protein interactions [4–6]. Rapid advances in genomic sequencing technologies are also improving diagnosis, especially in rare diseases and cancer, enabling the development of personalized, targeted treatments [7]. These approaches have the potential to deliver more effective and safe treatments compared to conventional treatments. As precision and mechanism-based treatments result in heterogeneous diagnostic pathways, increased number of test-treatment combinations, and smaller patient populations, generating robust evidence on clinical effectiveness and economic value to support reimbursement and adoption decisions can be challenging [8–12].

In this article, we assess the applicability of SMART and identify potential adds-on that may support its use, by applying it to the planning of early-stage DAMs in data-constrained settings. We selected two cases from studies being evaluated within the REPO4EU consortium [13], a European Union (EU)-level platform supporting mechanism-based drug repurposing, which explore the value of personalized and/or combination mechanism-based treatments in rare cancers (defined as cancers with a prevalence of fewer than six per 100,000 persons per year [14]): advanced thyroid cancer and diffuse midline glioma (DMG), located in the pons. Early-stage decision modelling in general, and particularly in rare diseases, is often limited by scarce evidence and patient heterogeneity, while advances in personalized and mechanism-based therapies further reduce the number of treated populations and increase variability in treatment outcomes in the different population subgroups [12, 15, 16]. Therefore, these two cases represent good case examples for a DAM setting that is shaped by limited data and resource scarcity.

The paper is structured as follows. First, for both case studies, we defined the decision context through an iterative process involving the study team (including clinical and pre-clinical expert co-authors), supported by a rapid targeted literature review. The expert co-authors were clinical and pre-clinical specialists in the REPO4EU consortium who contributed domain expertise throughout this process. PS and RNM provided expertise in thyroid cancer, while NG and SM contributed expertise for the DMG case. This step involved specifying the population, intervention, comparator, outcomes, time horizon, perspective, and audience of the decision problem (PICOTPA) and clarifying data and resource availability. Second, we completed SMART Sect. 2 (modelling choices), also in consultation with expert co-authors. This entailed selecting the ideal modelling choices based on the decision-context requirements and informed by reference cases and relevant guidelines, then selecting our modelling choices based on the available data and resources in the setting. Third, we completed SMART Sect. 3 (justification and consequences of modelling choices) by assessing the consequences of deviating from the decision-context requirements for model validity and transparency. After each of these three steps, we assessed how SMART shapes the model development process and identified areas where the tool could benefit from add-ons. Using SMART input, we also develop a draft conceptual model for each case.

Defining the Decision Context

Advanced Thyroid Cancer Case Study

Thyroid cancer is the most common malignancy of the endocrine system and the tenth most common cancer worldwide according to 2020 data [17, 18]. It has a global incidence of 10.1 cases per 100,000 women and three cases per 100,000 in men, with its incidence rates expected to increase by 29% by 2040 [18, 19]. Patients with differentiated papillary and follicular thyroid cancer (DTC), which account for 90% of all thyroid cancer cases, generally have a good prognosis and several treatment options, including thyroidectomy and radioactive iodine therapy [17, 20].

However, patients with advanced thyroid cancer represent a rare subset of the disease (approximately 5–10%), have a particularly poor prognosis, and, due to their low incidence, lack consensus on optimal evidence-based management. These rare subsets include poorly differentiated thyroid cancer (PDTC), radioactive iodine refractory differentiated thyroid cancer (RAI-R-DTC), and anaplastic thyroid cancer (ATC) [21–23]. PDTC represents 2–15% of thyroid cancer cases and has a median survival of 3.2 years, while ATC accounts for 1–3% of cases and is considered the most lethal form, with a median survival of 6 months [21, 22, 24, 25]. Both PDTC and ATC are frequently metastatic at diagnosis, highly aggressive, and resistant to radioactive iodide therapy. In addition, although it is considered a rare condition (4–5 cases per million people per year), up to 15% of DTC patients develop RAI-R-DTC (with a mean survival of 3–5 years), which may be treated with systemic multi-kinase inhibitors [23, 26]. However, treatment with multi-kinase inhibitors in this patient subgroup is not curative and often leads to toxicity and high financial burden, requiring dose reduction or even treatment discontinuation [27]. With a lack of effective treatment options in these advanced thyroid cancer patient subgroups, and best supportive care being the only option, there is an unmet medical need. Precision treatments using drug combinations that target disease interconnected modular pathways could offer a potential benefit beyond single-agent treatments [5].

Headroom analysis in health economic evaluation estimates the maximum price at which a technology remains cost-effective by quantifying the potential improvement over current clinical practice (i.e. unmet need), assuming maximum effectiveness of the technology, and the willingness to pay (WTP) for that improvement in a given patient population [28, 29]. It can inform early-stage pricing policy and return-on-investment decisions by identifying technologies unlikely to be cost-effective in the future. In this context, a headroom analysis is a useful approach to quantify the potential value of precision combination treatments. The planned DAM will assess the headroom of precision combination treatment in advanced thyroid cancer patients, evaluating costs and health outcomes including life years gained, adverse events, and quality-adjusted life years (QALYs), compared to best supportive care, from a societal perspective in the Netherlands. The decision context is summarized in Table 1.

Table 1.

Decision contexts for planned models in advanced thyroid cancer and diffuse midline glioma (DMG)

	Advanced thyroid cancer	DMG
Objective	To assess the headroom (i.e. 100% effectiveness and zero treatment costs) of mechanism-based, individualized repurposed drug combination approach in advanced thyroid cancer patients who can no longer continue standard treatment due to resistance or toxicity and/or have no available systemic therapies	To assess the headroom (i.e. 100% effectiveness and zero treatment costs) of a mechanism-based repurposed drug combination approach in children and young adults with a newly diagnosed pontine H3 K27-mutant DMG
Population	Patients with PDTC, RAI-R-DTC, or ATC	Children and young adults with a newly diagnosed pontine H3 K27-mutant DMG
Intervention	Personalized, mechanism-based treatment and a diagnostic strategy, in addition to best supportive care	Mechanism-based combination treatment and a diagnostic strategy in addition to radiotherapy
Comparators	Best supportive care	Standard of care: radiotherapy
Outcomes	Life years gained, QALYs, adverse events, and costs	Life years gained, adverse events, QALYs, and costs
Time horizon	Lifetime	Lifetime
Perspective	Societal perspective—Netherlandsa	Collective perspective—Franceb
Audience	Clinical development teams, patients, policy makers, payers, and investors	Clinical development team, patients, policy makers, payers, and investors
Setting (data and resources constraints)	This model is primarily exploratory and developed for a very early stage of the development process where no clinical or effectiveness data are yet available. Furthermore, the current lack of effective treatments limits the information on potential efficacy and prognosis for this patient population	This model is primarily exploratory and developed for an early stage of clinical research, with limited data on treatment effectiveness and prognosis. Furthermore, the rarity and heterogeneity of this tumour across patients impose additional limitations on data availability

ATC anaplastic thyroid carcinoma, PDTC poorly differentiated thyroid carcinoma, QALY quality-adjusted life year, RAI-R-DTC radioactive iodine refractory differentiated thyroid cancer

^aDutch societal perspective: Includes all costs across society regardless of who bears them. This covers healthcare costs (e.g. hospital care, procedures, health products), patient and family costs (e.g. informal care, travel, time costs), and costs in other sectors such as productivity losses, education, and the justice system. It requires a full societal accounting of all relevant costs and effects, independent of healthcare budgets or insurers [38]

^bFrench collective perspective: Includes all resources consumed in the production of patient care across the healthcare sector (e.g. stays, procedures, health products), domestic sphere (e.g. informal care), and medico-social sector (e.g. care services and facilities). Indirect costs including productivity losses due to absenteeism or reduced activity in paid or unpaid work from illness or premature death are not part of the reference case but can be considered in a separate supplementary analysis when relevant [39]

Diffuse Midline Glioma Case Study

DMG is a rare and highly aggressive malignant tumour that arises from glial cells in the brain, with an average annual age-adjusted incidence rate of 0.06 per 100,000 according to the Central Brain Tumour Registry of the United States (CBTRUS), and accounts for 10–15% of all brain tumours found in children [30–32]. DMG is the leading cause of brain tumour–related deaths in children, with most patients dying within 12 to 18 months of diagnosis [33]. Because it occurs in midline structures of the brain, DMG can produce a wide range of sensory and motor symptoms depending on tumour location, including speech and swallowing difficulties, balance disorders, and facial paralysis, among other symptoms [30, 31]. The tumour was previously referred to as diffuse intrinsic pontine glioma. However, advances in molecular and tissue sampling revealed distinct characteristics, particularly the presence of H3-K27 mutations, which prompted the World Health Organization (WHO) to reclassify these tumours collectively as DMG, H3-K27M–mutant, in 2016 [34].

These tumours exhibit significant inter-tumour heterogeneity and wide histological variation, often demonstrating invasive spread to other brain regions. Their midline location makes surgical resection of the tumour impossible. Moreover, the blood–brain barrier remains largely intact in DMG, limiting the ability of drugs to penetrate the barrier and achieve therapeutic concentrations [35]. Chemotherapy has shown minimal effectiveness, leaving radiotherapy as the primary standard of care. While radiotherapy provides temporary symptom relief, it typically extends survival by only 3 months in four out of five cases, with limited improvement in quality of life [30, 31, 33]. Together, these factors leave DMG patients with scarce treatment options, and despite ongoing clinical research, their mean 5-year survival rate is less than 5% [33].

Previous research suggests that recent discoveries regarding the underlying mechanisms of DMG and the identification of novel targets make the disease a promising candidate for tailored combination therapies [36, 37]. Such approaches, targeting both tumour cell–intrinsic vulnerabilities and micro-environmental dependencies, may prove more effective than single-agent treatments or radiotherapy. The planned DAM will assess the headroom of this approach in combination with standard-of-care (radiotherapy) compared to standard of care alone in DMG patients. The model will evaluate health outcomes including life years gained, adverse events, QALYs, and costs, compared to current standard of care (radiotherapy) from a collective perspective in France. The full decision context is summarized in Table 1.

Evaluation of SMART Application in Defining the Decision Context

SMART was helpful as a discussion framework for early engagement with the expert co-authors and allowed systematic gathering of inputs and shaping the decision context for both cases. This included refining the PICOTPA and identifying data and resource constraints associated with the early stage of the model and the rarity of the disease that need to be considered when developing the models.

Completing Sect. 2 of SMART: Modelling Choices

Ideal Modelling Choices Based on the Requirements of the Decision Context

The ideal modelling choices required for both advanced thyroid cancer and DMG decision contexts, which can be informed by the decision problem, disease management guidelines, the relevant reference case, and health economic guidelines, are provided in the Electronic Supplementary Material. In both cases, the ideal DAM represents heterogeneous, closed populations to capture the outcomes in different patient subgroups in thyroid cancer and histological heterogeneity in DMG. Both interventions are positioned at a single point in the care pathway, with a direct relationship to model outcomes. Each case currently has a single relevant comparator, as there are no effective alternative treatments for these patient populations. Few final outcomes (life years, QALYs, adverse events, and costs) need to be evaluated in both cases. Lifetime time horizon and a broad societal perspective (in the Netherlands) or a collective perspective (in France) should be incorporated to reflect long-term cancer outcomes and caregiver burden, which is particularly relevant in paediatric DMG and older patients with advanced thyroid cancer. The advanced thyroid cancer case requires a combined model structure comprising of a state transition model paired with a decision tree or  an individual patient-level model,to account for genomic and treatment pathways heterogeneity, where each patient receives a combination of treatments tailored to their genetic profile. In contrast, in the DMG case, a state transition model alone is sufficient to capture the patients’ disease pathway, as all patients receive the same treatment combination. Explicit time modelling is necessary to account for the short survival in these populations.

For model evidence, a systematic literature review, extensive data analysis, and structured expert elicitation are needed to fill the evidence gaps. In line with Dutch health economic guidelines [38], the advanced thyroid cancer model should include probabilistic and value-of-information analyses. The DMG model should include probabilistic analysis in line with French guidelines [39].

Our Modelling Choices

Our selection of modelling choices in both cases deviated from the ideal decision-context requirements (a full assessment can be found in the Electronic Supplementary Material). Simplifications included modelling a homogeneous instead of a heterogeneous population, adopting a narrower healthcare perspective, excluding decision trees for diagnostic pathways and different drug combinations, modelling fewer outcomes by excluding adverse events, using approximate timing for modelled events rather than exact timing, and relying on assumptions instead of structured expert elicitation and extensive data analysis to fill evidence gaps.

Evaluation of the Completion of Sect. 2 of SMART

When completing Sect. 2 in SMART, we noted that the model features related to PICOTPA overlapped with part of the decision context in Sect. 1 (model features). To optimize the use of SMART, we amended the ‘Decision context’ tab in SMART version 1.0 to include only the project title, model objective (including the model purpose and summary of the PICOTPA), and the setting of the decision context, including the data and resources available. We also added a description in the user instructions on how to complete this section.

We also observed that selecting modelling choices may require guiding questions to help users identify the ideal option for each model feature, and that some features were not intuitive for expert co-authors. To address this, we developed targeted questions to help fill in these features and to facilitate discussions with experts, eliciting their input on them (Table 2). Applying SMART to the advanced thyroid cancer and DMG cases further clarified how to define ideal modelling choices based on decision-context requirements. Due to the uncommonness of these diseases and the early stage of the models, many simplified assumptions are still appropriate; however, our choices sometimes deviated from the ideal due to limited data and resource constraints. Clearly defining ideal modelling choices is therefore essential to ensure consistency and reliability when using SMART iteratively alongside the model development cycle. To support this, we have added a dedicated section in SMART that allows users to provide additional explanations for their ideal modelling choices, including supporting sources and expert input.

Table 2.

Questions to guide SMART users and facilitate dialogue with relevant stakeholders when selecting the ideal modelling choices based on the requirements of the decision context

	Model features		Simple modelling choice	Complex modelling choice	Guiding questions
Model conceptualization	Population	Heterogeneity	Homogeneous	Heterogeneous	Do different types of patients respond differently to the treatment, such that outcomes or costs might vary between subgroups?
		Dynamics	Closed population	Open population	Will the number of patients in the population change over time and affect the total costs and effectiveness of the intervention, requiring the model to allow for new entries over time?
	Intervention	Intervention–outcome relationship	Direct	Multifaceted	Could the treatment’s effects depend on other factors that cannot be captured by a simple linear relationship between the intervention and outcomes, such as other treatments patients receive or pre-existing health conditions?
		Placement in the care pathway	Clear/single placement	Unclear/multiple placements	Could the intervention provide benefits at different points in the care pathway? Might the impact of the intervention vary depending on when its given, which requires evaluation of its effects at multiple points in the pathway (e.g. as first-line versus second-line therapy)?
		Capacity constraints	No	Yes	Are there limits in resources that could prevent all patients from receiving treatment and affect their outcomes or costs in this patient’s population, for example, staff, beds, or equipment?
	Comparator	Number of relevant comparators	Single or few	Many	Are there multiple existing treatments for this patient population that should be considered to understand how this new treatment compares?
	Outcomes	Number of outcomes	Few	Many	Does the intervention have multiple consequences for costs and health outcomes, which all need to be captured in the model to avoid bias or uncertainty?
		Type of outcomes	Intermediate	Long term/final	Can the intervention’s relevant costs and health outcomes for patients, healthcare providers, and intended audiences be measured directly and immediately? Or are these outcomes only observable indirectly or in the long term (e.g. survival), thus requiring the use of intermediate outcomes?
	Time	Time horizon	Short	Long	Could the treatment have long-term effects on patient health or costs, such that looking only at short-term outcomes would miss important benefits or harms?
	Perspective	Model perspective	Narrow	Broad	Does the disease have a broader impact beyond direct healthcare costs, such as patients’ ability to work or informal care provided by family members?
	Audience	Intended audience(s)/objective(s)	Single	Multiple	Are there different decision-makers who will use the model results, such as HTA bodies, payers, clinicians, investors, or regulators? Do their objectives or goals need to be considered in the model design?
	Model structure	Number of health states/events	Few	Many	Are there many stages of the disease, events, or clinically distinct health states that need to be distinguished in the model to fully understand the treatment impact?
		Connections between health states/events	Few	Many	Are there multiple disease stages, events, or clinical health states that could influence later stages and therefore need to be considered?
	Modelling approach	Combined modelling approaches	No	Yes	Is it necessary to use a combination of model types to address the decision problem, such as a state transition model and a decision tree?
		Inclusion of memory	No	Yes	Do past events influence future outcomes in ways that should be considered, such as previous complications, adverse events, number of relapses, or time to progression?
		Interaction between individuals	No	Yes	Could patient outcomes be influenced by interactions with other patients in the cohort, due to transmission of disease or resource allocations?
		Explicit modelling of time	No	Yes	Is it important to track how outcomes or costs change over time, rather than just looking at the final costs and health?
		Timing of modelled events and consequences	Approximate	Exact	Is the exact timing of when a specific event occurs important for understanding the treatment’s impact, such as hospitalization or disease progression?
Model inputs		Literature review	Rapid and targeted	Systematic	Are there numerous previous studies that should be examined to better inform the model? Is it necessary to review all available research to ensure the evaluation is based on the best evidence?
		Synthesis	No	Yes	Do we need to combine evidence from multiple studies to obtain reliable estimates of treatment effects or costs, for example, using a meta-analysis?
		Clinical data collection	No	Yes	Is there a need to collect new clinical data and/or perform a new trial because existing information is not sufficient to understand the treatment impact?
		Structured expert elicitation	No	Yes	Is it important to elicit and gather expert opinions in this disease using formal methods, when data are limited or uncertain, to inform the evaluation?
		Data analysis	Limited	Extensive	Are additional analyses needed to process existing data on the disease so it can be used to estimate the treatment impact accurately?
Model analysis		Patient-level stochastic simulations (if applicable)	Few	Many	Do you expect substantial variation in how individual patients may experience different outcomes that should be examined in many simulations?
		Deterministic analyses	Few	Many	Are model structural or parameter uncertainties significant enough to require testing multiple assumptions or different model parameters?
		Probabilistic analysis	No	Yes	Is there substantial uncertainty in this disease area or regarding the new intervention that warrants evaluating multiple scenarios simultaneously in the model?
		Decision uncertainty analysis	No	Yes	Is there likely significant decision uncertainty or risk associated with this decision problem and the intervention in particular? Would collecting additional data or conducting further research improve confidence in the model results and support better-informed decision-making?

Detailed definitions of each model feature, as well as the associated simple and complex modelling choices, are available in the tool and in the glossary provided within the tool

HTA health technology assessment, SMART Systematic Model adequacy Assessment and Reporting Tool

All the changes that were incorporated into the original SMART can be found in SMART version 1.1 at the following link: https://osf.io/4j8qx/overview?view_only=7c32744f833845a596629e467f7c71ac.

Completing Sect. 3 of SMART: Consequences and Justifications of Our Modelling Choices

The Consequences of Deviating from the Requirement of the Decision Context

The deviations made from the ideal decision-context modelling choices in both case studies were due to limited data and time constraints associated with the early development stage, as well as the uncommonness of both diseases, which further limits the availability of cost- and effectiveness-related data. The deviations are likely to have a low to moderate impact on model validity, with varying effects on model transparency. Table 3 summarizes the assessment of these deviations and their justifications.

Table 3.

Summary of SMART assessment of modelling deviations from the requirements of the decision context, and their consequences in the two case studies

Model feature	Ideal modelling choice	Advanced thyroid cancer case		DMG
		Practical modelling choice	Justification and consequences for model validity and transparency	Practical modelling choice	Justification and consequences for model validity and transparency
Population heterogeneity	Heterogeneous	Homogeneous	The model assumes a single homogeneous population due to the lack of subgroup-specific data. This may moderately affect validity by over- or underestimating effects and reduce transparency for stakeholders who require subgroup results	Homogeneous	The model assumes a homogeneous population due to the lack of subgroup data. This may moderately affect validity by over- or underestimating the cost-effectiveness results and reduce transparency by oversimplifying outcomes
Model perspective	Advanced thyroid cancer: broad; DMG: narrow	Narrow	A healthcare perspective will be used due to lack of data. Excluding indirect costs and productivity loss is likely to underestimate the cost-effectiveness of treatment and will have low impact on validity and no impact on transparency	Narrow	–
Audience	Multiple	Single	Due to the very early stage of the model, it will only target the clinical development team and investors. This will have a likely low impact on model validity and improved transparency	Single	Due to the very early stage of the model, it will only target the clinical development team. This will likely have a low impact on model validity and improved transparency
Combined modelling approach	Advanced thyroid cancer: yes; DMG: no	No	Due to lack of subgroup data and time constraints, the decision tree will be excluded. This may moderately impact model validity by either overestimating or underestimating cost-effectiveness outcomes. Transparency might be reduced as assuming all patients share the same prognosis oversimplifies model outcomes for its ideal audience	No	–
Timing of modelled events and consequences	Exact	Approximate	Approximate estimates will be used due to lack of data and survival curves. This will likely have a moderate impact on validity by introducing bias that could either overestimate or underestimate the cost-effectiveness outcomes. No impact on transparency is expected	Exact	–
Literature review	Systematic	Rapid and targeted	Due to the very early stage of the model, a rapid and targeted literature review will be conducted to identify relevant data and studies. Given the uncommonness of the disease and the limited availability of data, this simplified modelling approach is expected to have a likely low impact on model validity and no impact on model transparency	Rapid and targeted	Due to the very early stage of the model, a rapid and targeted literature review will be conducted to identify relevant data and studies. Given the rarity of the disease and the limited availability of data, this simplified modelling approach is expected to have a likely low impact on model validity and no impact on model transparency
Structured expert elicitation	Yes	No	Assumptions will be used due to time constraints and the early nature of the model. This is not expected to have an impact on model validity as experts currently also do not have sufficient information on the outcome. No impact on transparency is expected	No	Only rough estimates will be used due to limited evidence and the early development stage. This is likely to have low impact on validity, as experts currently do not have sufficient information about the outcomes. No impact on transparency is expected
Data analysis	Extensive	Limited	Rough estimates will be used due to limited data. This may introduce a large bias in model outcomes for time to remission, progression, or death. Therefore, the impact on the validity of survival and cost-effectiveness estimates is likely severe. Use of rough estimates may improve transparency	Limited	Rough estimates from literature will be used due to limited data and the headroom nature of the model. This may reduce validity by introducing bias in survival and cost-effectiveness estimates. Clearly reporting these estimates may improve transparency

– No deviation was made from the ideal modelling choice; therefore, justification is not required

DMG diffuse midline glioma, SMART Systematic Model adequacy Assessment and Reporting Tool

Conceptual Models

Based on the SMART outputs for both case studies and clinical expert feedback, we developed conceptual models for advanced thyroid cancer and DMG. The thyroid cancer model is a state transition model consisting of four health states that represent the potential progression of patients within the model after receiving the personalized combination therapy (Fig. 1). The DMG model is also a state transition model, with three health states to capture whether patients will respond to the combination treatment or not and the potentially associated costs and health benefits (Fig. 2).

Fig. 1.

Model concept for advanced thyroid cancer

Fig. 2.

Model concept for diffuse midline glioma

Remarks on Completing Sect. 3 in SMART

During this step, we noted that the original SMART Sect. 3 guidance does not provide detailed instruction on how to fill in this section. Users should start by describing the rationale behind selecting the ideal modelling choice, then explain the data and resource constraints within the decision context that might justify deviation from the ideal model. Moreover, as the assessment of consequences may involve subjectivity, the rationale for the assigned consequences in Sect. 3 should be clearly described for validity (potential impact on cost-effectiveness outcomes, including whether the resulting bias is likely to be positive or negative where possible) and transparency (extent to which the modelling choice affects how easily the model and its results can be understood and reviewed). These explanatory remarks were added to the original SMART ‘User instructions’ tab to ensure the tool is used optimally.

We also noted that having an option to generate SMART outputs, either as a full overview of the completed tool or as a summary report of deviations, can be helpful as a supporting document for HTA submissions and journal publications, or during the review and assessment of DAMs. Therefore, we added a dedicated tab with a button that automatically summarizes SMART inputs into a structured report, which can be exported as a pdf, as well as a separate function to export the completed tool in pdf format. Figure 3 shows an example of a SMART summary report for the DMG case. Full assessment overviews and summary reports for both case studies are provided in the Electronic Supplementary Material. Box 1 presents the key amendments and lessons learned from applying SMART 1.0 across the two cases, while Supplementary Table 1 summarizes the contributions of the study team at each step of the SMART application.

Fig. 3.

Summary report of SMART application in diffuse midline glioma case

Box 1.

Summary of learnings, amendments, and recommendations from SMART application in the two case studies. PICOTPA population, intervention, comparator, outcomes, time horizon, perspective, and audience of the decision problem, SMART Systematic Model adequacy Assessment and Reporting Tool

The full updates can be reviewed by accessing and downloading both SMART 1.0 and the updated version, SMART 1.1, at the following link: https://osf.io/4j8qx/overview?view_only=7c32744f833845a596629e467f7c71ac.

Discussion

In this article, we applied SMART in planning two early-stage DAMs in decision contexts constrained by time and resources: personalized and combination mechanism-based treatments in rare cancers including advanced thyroid cancer and DMG, which currently lack effective treatment options. Using literature and continuous feedback from clinical and pre-clinical expert co-authors, we first defined the decision context. We then completed SMART Sect. 2 by selecting modelling choices, first based on the decision-context requirements, and then our choices based on the available data and resources within the context. We also completed Sect. 3 by assessing the consequences of deviations from the decision-context requirements for model validity and transparency and conceptualized the models.

Applying SMART supported a systematic definition of the decision context and encouraged early engagement with stakeholders, which helped in selecting the modelling choices for both DAMs. It also facilitated agreement between model developers/analysts and clinical and pre-clinical experts on model plans that are most appropriate for this early stage of development. Through this process, we identified opportunities to improve the tool and implemented several amendments. These included revising the decision-context tab, developing guiding questions to support the selection of modelling choices and to facilitate discussions with relevant stakeholders, adding additional space in SMART to summarize sources for informing the ideal modelling choices for a decision context, and providing clearer guidance for completing Sect. 3 of SMART. We have also added the functionality to generate and export an output from SMART based on the completed list of model features and Sect. 3.

The two SMART use cases showed that deviation from ideal modelling requirements is often inevitable in early DAM development in data- and resource-constrained settings. Early-stage decision modelling in rare diseases is often constrained by limited evidence and the heterogeneity of genetic profiles among patients and their response to treatment [40, 41], while advances in personalized and mechanism-based drug development and repurposing lead to smaller numbers of treated patients and highly variable outcomes. Although such interventions have great potential in the prevention and cure of diseases, unlike most symptom-based chronic therapies [42, 43], key healthcare decisions regarding their approval and reimbursement remain major barriers to patient access [10, 44, 45]. Given these challenges, early assessment of their value should involve adequate DAMs that can provide crucial insights into their potential clinical and economic value and the evidence required to support regulatory and reimbursement decisions. Modellers must also consider that early modelling choices influence the model’s adaptability over time. For example, adding tunnel states or a decision tree at later development stages might require structural revision of the original model template. SMART, with its add-ons, provides a transparent and structured framework for planning DAMs at an early stage. It facilitates early engagement with relevant stakeholders, highlights where trade-offs in modelling choices are necessary due to data and resource constraints, and helps identify which simplifications are acceptable and which may need to be revisited as new evidence becomes available.

Although completing SMART may seem time consuming or an additional step when planning models (roughly 1–2 h per case, excluding discussion time with stakeholders), this upfront effort and time may be offset by efficiency gains in later stages, as SMART forces the user to consider issues with model validity early. This in turn may allow the user to address these issues early. Modelling choices in SMART are inherent to the modelling process, and SMART primarily provides a structured framework to ensure they are addressed systematically. The tool is also designed to support practical use, for example, through predefined dropdown options that facilitate rapid input. In addition, SMART supports earlier identification of relevant questions for relevant stakeholders, therefore, enabling more focused and efficient discussions. That said, Sect. 3 (justification and consequences of modelling choices) may require additional time compared with less structured approaches, but the additional effort can strengthen the adequacy of the planned or developed model and enhance methodological rigor and transparency in health economic evaluation. Furthermore, we observed that the process of using SMART, including selecting modelling choices and summarizing relevant information to inform these choices, can be efficiently structured to support the preparation of a model-based health economic analysis plan. Future research can explore how such analysis plans can be derived from the SMART.

Some limitations of this study should be acknowledged. The application of SMART was conducted by the original development team plus clinical experts, which may have influenced perceptions of its usefulness as well as estimates of the time and resources required to complete the tool. In addition, the two rare cancer case studies reflect a specific application setting, which may impact the generalizability of the findings. Future applications and research should involve independent users across different decision contexts to provide more objective and generalizable feedback on usability and the value of SMART. Diverse settings should also be tested, such as those with limited modelling expertise (e.g. non-HTA users), low- and middle-income countries, or later-stage model development with more complex data and modelling requirements. While initial in-depth interviews and focus groups informed the tool development, assessment by model reviewers and model users in real-world applications can also be explored to evaluate SMART’s usefulness. Finally, as there is subjectivity in the assessment of consequences in Sect. 3, which we have tried to address by expanding the user instructions for completing that section, future applications could explore whether there is a need for a formal grading system to standardize SMART assessment of consequences arising from deviations from ideal modelling choice.

Conclusion

Developing DAMs requires trade-offs between ideal decision-context requirements and time and/or other resource constraints, as seen early in the development pathway of personalized, mechanism-based combination treatments for uncommon cancers. SMART (version 1.1), with its add-ons, provides a structured and transparent approach to planning early DAMs, clarifies how such trade-offs impact model validity and transparency, and identifies deviations that may become unjustifiable as evidence emerges. SMART also guides early stakeholder engagement and helps in bridging the communication gap to support the development of early-stage DAMs that are decision relevant across the development stages.

Supplementary Information

Below is the link to the electronic supplementary material.

Funding

This project is funded by the European Union under grant agreement No. 101057619. The views and opinions expressed are, however, those of the authors only and do not necessarily reflect those of the EU or European Health and Digital Executive Agency (HADEA). Neither the EU nor the granting authority can be held responsible for them.

Declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Ethics Approval

Not applicable.

Consent to Participate

Not applicable.

Consent for Publication

Not applicable.

Code Availability

Not applicable.

Authors’ Contributions

TA: Writing, conceptualization, reviewing and editing of the original draft, writing and editing of the research protocol, methodology development, investigation, visualization, validation, data curation, and formal analysis. SG: Conceptualization, review and editing of the original draft, validation, visualization, supervision, and project administration. PC: Conceptualization and review and editing of original draft. NG: Conceptualization and review and editing of original draft. SM: Conceptualization and review and editing of original draft. RNM: Conceptualization and review and editing of original draft. MJ: Conceptualization, review and editing of the original draft, validation, visualization, supervision, project administration, and funding acquisition. We hereby confirm that all authors have read and approved the final version of the article prior to submission.