Putting Stakeholder Engagement at the Center of Health Economic Modeling for Health Technology Assessment in the United States

Abstract

While evidence generated from health economic (HE) models is being used more commonly in health technology assessment (HTA) in the US, it is not consistently adopted by different stakeholder groups or across therapeutic areas. We hypothesize that actively engaging with multiple stakeholder groups throughout the model development process may result in models more widely considered by decision makers. To test this hypothesis, the Innovation and Value Initiative has launched a modeling effort to build an open-source HE model focusing on the disease state ‘major depressive disorder’. A 20-member advisory group has been formed with representatives from patients, employers, clinicians, innovators, payers, and researchers to guide the model development process. While this effort is still in the early stages, the ongoing stakeholder engagement effort has yielded valuable insights that inform the model design. We have also identified several challenges to implementing this new approach. Our early findings suggest that the stakeholder engagement approach to HE model development has the potential to improve HTA in the US.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40273-021-01036-3.

Key Points for Decision Makers

A modeling effort has been launched to test the hypothesis that continuous stakeholder engagement may result in health economic (HE) models more widely considered by decision makers in the US.

Early-stage findings show that such an engagement approach has yielded valuable insights for the model design, and have identified several implementation challenges.

The stakeholder engagement approach to model development may result in HE models more widely considered by decision makers, and has the potential to greatly improve health technology assessment in the US.

Introduction

Under pressure to curb rising health care costs in the US [1], private and public payers, policymakers, clinical societies, and other stakeholders have increasingly turned to formalized health technology assessments (HTA) to assess the relative value of health interventions [2–4]. Health economic (HE) modeling [5] is used to predict and compare the costs and health outcomes of interventions and is commonly utilized to inform HTA outside the US [6, 7]. Although evidence generated from HE models is being used more commonly in coverage and reimbursement deliberations in the US, it is not consistently adopted by different stakeholder groups or across therapeutic areas [3, 8–10].

Experts from multiple disciplines have called for a more open and inclusive process in the development of HE models and HTA frameworks, and they emphasize the importance of broader inclusion of patient and employer perspectives [2, 11, 12]. We hypothesize that actively engaging with multiple stakeholder groups throughout the HE model development process—and particularly during initial planning and model design phases—may result in more relevant and useful models.

To implement such an approach, the Innovation and Value Initiative (IVI) has established a process that empanels an advisory group in the disease state of interest. These advisors provide guidance on key modeling considerations, identify data sources, and generate use cases throughout different phases of the modeling process. As participants in the HE modeling process (Table 1), the advisory group guides modeling decisions from the beginning (Fig. 1).

Table 1.

Primary actors in model development process

Actor type	Description	Role
Primary Organizing Entity	Organization or initiative that initiates, coordinates, directs, and is ultimately responsible for the model developed throughout the process	Provides leadership and scientific oversight in all aspects of model development, including
		- Selecting disease state and acting as final decision-maker in model development process
		- Recruiting and coordinating Multi-Stakeholder Advisory Group and Research Team
		- Ensures adequate funding and other necessary resources
Multi-Stakeholder Advisory Group	Comprises individuals representing key stakeholders in the health care system affected by how HTA is conducted in the disease condition of interest, and may include patients and patient organizations, researchers, life sciences industry, clinicians, employers, and payers	Plays an advisory role throughout the model development process but does not have decision-making authority for the model development process. Key responsibilities include
		- Playing a leading role in establishing the goals and intended applications of the model
		- Ensuring that model objectives and design reflect the real-world decision contexts and metrics considered by end users
		- Contributing specialized knowledge and experience in the disease state, including clinical expertise, patients’ insights, and practical  experience in formulary or benefits design
		- Assisting in identifying data sources and other resources to support model development
Research Team	Organizations or individuals directly contributing to model design and development or related research activities, and may include researchers, academic centers, patient research organizations and other nonprofits, or for-profit health economics consultancies	Members of the research team contribute at all stages of the process, including
		- Developing the overall model design
		- Conducting systematic literature reviews and network meta-analyses
		- Programing the model and developing user interface
		- Managing data
		- Researching gaps in methods or evidence, including emerging scientific methodologies and identification of key determinants of value to patients

HTA health technology assessment

Fig. 1.

Development process for health economic models with multi-stakeholder input

Inclusion of Multiple Stakeholders in Health Economic Model Development

In the US, HE models are often developed by for-profit economic consultants, academic researchers, or health economics and outcomes research experts within life sciences companies [13, 14]. Key model assumptions, specifications, and data inputs are often determined by technical experts, and access to key assumptions or data inputs in the HE models can be limited due to intellectual property concerns [13, 15]. Additionally, HE models seldom consider or incorporate direct patient perspectives and inputs, and they might fail to capture real-world patient experiences valued by decision makers [11, 16]. Perspectives from external stakeholders are often sought ex post and used as contextual inputs or considerations rather than being consistently incorporated into models. The lack of transparency about key modeling decisions and limited stakeholder participation in the model development process can lead to misalignment between model design and decision needs of end users [13]. These limitations cause relevant decision makers to question the credibility and relevance of cost-effectiveness estimates and, ultimately, the ability of models to inform HTA in meaningful ways [17, 18].

In July 2020, IVI launched its third Open-Source Value Project initiative to build an HE model to support HTA for major depressive disorder (MDD). We chose MDD due to its prevalence, significant societal burden, and broad interest among stakeholders who are looking for better treatments and more cost-effective resource allocation [19, 20]. As a first step, IVI convened a 20-member Advisory Group1 (AG) consisting of patients (n = 5), employers (n = 5), clinicians (n = 5), innovators (n = 3), payers (n = 2), and researchers (n = 2) to weigh in throughout the modeling process.

To date, the AG has provided feedback on the model’s conceptual framework [21] as well as the model design, including identifying (1) decision needs (model objectives); (2) gaps in existing economic models; (3) data sources for model inputs; and (4) the most appropriate analytic framework when multiple approaches exist. Feedback was provided through group meetings, surveys, emails, and individual discussions. In prioritizing feedback from the AG, we solicited additional insights from patients and employers, two traditionally under-represented stakeholder groups in the HE modeling process [11, 12, 22].

Early Insights

While still in the early stages, this ongoing stakeholder engagement has already yielded valuable insights, recommendations, and resources (Table 2). This input is meaningfully impacting model design in three key areas discussed here.

Table 2.

Advisory Group input guiding model development

Model specification	Aggregated feedback
Objective	Instead of simply comparing the value of treatment A vs treatment B, the model should be a holistic modeling exercise that examines the treatment pathway of MDD
Treatment settings	A significant proportion of MDD patients were diagnosed and treated in the primary care setting. The model should explore how key clinical and economic outcomes vary by treatment settings (primary care, specialty, and telehealth)
	The factors that are likely to vary across settings include
	- Treatments prescribed, particularly the use of pharmacologic and non-pharmacologic agents
	- Patient characteristics
	- Effectiveness
	- Insurance coverage of patients
Target population	The model should primarily focus on the general or broader MDD population, but exclude patients with the following conditions:
	- Pediatric depression
	- Postpartum depression
	- Terminal illnesses and depression
	- Cognitive impairments
	- Substance use disorder
	- Bipolar or other psychiatric conditions
Subgroups of interest	The model may consider the following patient subgroups in evaluation:
	- Low socioeconomic status (SES)
	- Racial/ethnic subgroups
	- Individuals with co-occurring conditions (e.g., diabetes, cardiovascular disease)
	- Newly diagnosed individuals vs those with 2+ years of treatment
	- Incarcerated populations
	- Insurance type
	The model should try to incorporate subgroup-specific inputs and allow for examining model outputs for specific subgroups
	In existing literature, treatment-resistant depression was noted as a subpopulation associated with significant humanistic burden. However, there is no consensus on the definition of treatment-resistant depression. A more clinically meaningful definition is “patients who did not achieve adequate response after receiving more than two types of interventions''
Time horizon	Existing CE models typically focus on a shorter time horizon (<5 years). In the real world, depression ‘does not go away’ The model should build in flexibility to show both short-term and long-term trajectories, even if such data might not be immediately available
Comparators	The model could consider offering flexibility to evaluate treatments in broader categories (e.g., pharmacologic vs non-pharmacologic), therapy classes (e.g., SSRIs), and individual treatments
	Additional considerations include
	- Increasing use of digital therapy
	- Post-relapse treatment strategies
	- Adding a ‘no active treatment’ arm
Clinical instruments	HAM-D and MADRS are commonly represented in clinical trials, but PHQ-9 is more commonly used in clinical practice
	It is important to recognize that all clinical instruments have their limitations. They do not fully capture the impacts of treatments on patients
	From a payer’s perspective, any measure that is clinically validated can be considered
Inputs	General considerations
	- Use subgroup-specific inputs whenever possible, instead of population-average estimates
	- Consider a mix of data sources (e.g., real-world data) beyond clinical trial data
	Efficacy
	- Note the time lag between diagnosis and treatment—many individuals suffering from MDD episodes were not formally diagnosed
	- Time to treatment effect (if sufficient evidence supports that)
	Costs
	- Long-term cost offsets from improved mental health
	- Given that MDD is a highly co-morbid condition, it might be worth considering separating MDD-specific and all-cause costs
	- The model should consider the various nuances of costs due to loss of productivity including
	Absenteeism (due to treatment, due to symptoms)
	Presenteeism
	Prejudice due to diagnosis of MDD
	Long-term and short-term disability
	- Caregiver burden
Output	The model should offer a range of model outputs that decision makers can reference in decision making
	- QALY is a commonly used but imperfect measure
	- Consider clinically based outcome measures such as ‘number of responders’, or ‘number of remitters’
Other considerations	As the IVI model seeks to incorporate patient-important value elements in the model design, such elements might change over time through the course of their treatment experiences

CE cost effectiveness, HAM-D Hamilton Depression Rating Scale, HE health economic, MADRS Montgomery-Asberg Depression Rating Scale, MDD major depressive disorder, PHQ-9 Patient Health Questionnaire-9, QALY quality-adjusted life-years

AG input has highlighted important decision contexts and factors considered by end users that help inform the model specification For example, the Hamilton Depression Rating Scale and Montgomery-Asberg Depression Rating Scale are the clinical instruments most widely used to define health states in existing models, but stakeholder input indicated that the Patient Health Questionnaire-9 is more commonly used in clinical practice and increasingly in clinical trials and observational studies. In discussions about potential model structure, all stakeholder groups emphasized that care setting (primary vs specialty care) is an important variable that will impact prescribed treatments and their effectiveness. Therefore, we are reviewing additional literature to evaluate how care settings will impact patient trajectory in model simulations. Patient groups also emphasized the importance of prioritizing data inputs from samples more representative of the MDD population in the US.

AG members have shared preliminary, and sometimes proprietary, findings and data from ongoing research projects that can be used as model inputs For example, a researcher shared data on caregiver burden that can be a direct model input [23]. Employer, patient, and researcher representatives highlighted the need to measure different types of productivity impacts (e.g., absenteeism, presenteeism) and provided suggestions on data sources and literature.

AG members who are potential end users are interested in engaging in the modeling process and applying the model in their decision making Despite their differing decisions needs, many stakeholders emphasized similar considerations in model design. These included reflecting patient heterogeneity in treatment experience, facilitating subgroup analyses, and fully measuring different types of productivity loss (Table 2). Moreover, AG members also proactively proposed use case development. For example, one stakeholder suggested that the model design would allow her organization to assess the burden of MDD without active treatments.

The discussions also identified key areas of consensus across stakeholder groups regarding the purpose, design, and use of existing HE models (Table 2 and “Appendix 2”, see ESM). Suggestions included (1) models should be flexibly designed to accommodate emerging and evolving data; (2) models should incorporate real-world data beyond just clinical trials; (3) entire treatment pathways should be modeled rather than just individual treatment comparisons; and (4) models should incorporate patient perspectives and inputs.

Implementation Challenges

Eight months in, we have identified several challenges to implementing this new approach. First, eliciting and incorporating feedback from diverse stakeholders to inform HE model development requires experience with multiple methodologies (both qualitative and quantitative), as well as creativity and flexibility in their use. For example, roundtable discussions required focused questions and well-identified goals to generate specific feedback on research questions. Small-group discussions were sometimes more appropriate for engaging those with specific expertise. Qualitative AG input sometimes needed to be supplemented with additional literature reviews to identify appropriate methods—to understand the measures and estimates of presenteeism, for example. Finally, to ensure inclusive input, discussion materials must account for varying levels of clinical or technical knowledge. While time consuming, these steps are crucial for soliciting useful input.

This approach requires foresight to address differing priorities. Through email communications, meeting summaries, and discussions, we have worked to ensure that all viewpoints are documented and acknowledged, and that methods for prioritizing input are transparent. We are committed to communicating with the AG about our decision-making process, and plan to prioritize considerations with the broadest stakeholder buy-in, inputs from traditionally under-represented stakeholders, and inputs that are most feasible to implement [24].

Finally, engaging a large, diverse stakeholder group increases the scope and complexity of model development. The research team leading the process must invest considerable time and resources to planning and managing engagement activities, synthesizing conflicting viewpoints, and determining how to incorporate feedback into the model design. This may be particularly challenging for those lacking engagement experience. Refining the methodology through further implementation may increase efficiency, however.

Advisory Group members must also commit their time and energy to an ongoing process, and the research team may need to account for missing stakeholder perspectives. Some key stakeholder groups were not able to participate in our Advisory Group, namely pharmacy benefit managers, benefit consultants, and government agencies (e.g., Centers for Medicare & Medicaid Services). As these are key decision makers and potential users, we intend to use public comment periods and individual outreach to solicit feedback.

Learning Through Action

As the next phases of the MDD development transpire, we will continue to evaluate the process with the AG. Specifically, we will assess how this engagement has impacted model design, whether it will increase the consideration of HE models by different stakeholders, and how well it facilitates open dialogue and trust across stakeholder groups.

As US policymakers and payers continue to prioritize a transition to value-based care, IVI is advocating a concurrent evolution of methods to better reflect the inputs and needs of broader stakeholders. Our multi-stakeholder engagement approach to developing an open-source HE model has the potential to greatly improve HTA in the US.

Supplementary Information

Below is the link to the electronic supplementary material.

Declarations

Funding

No funding was received for the preparation of this manuscript.

Conflict of Interest

Richard Z. Xie, Erica deFur Malik, Mark T. Linthicum, and Jennifer L. Bright hold the positions of HEOR Manager, Director of Membership, Director of Scientific Communications, and Executive Director, respectively, at the Innovation and Value Initiative (IVI), a 501(c)(3) non-profit research organization whose members include pharmaceutical companies, patient advocacy organizations, and health systems. Richard Z. Xie was previously employed by Analysis Group, an economics consultancy with clients in the life science industry. Erica deFur Malik is the Managing Director of Malik Group, LLC, and consultant for Momentum Health Strategies. Mark. T. Linthicum was previously employed by Precision Health Economics, a health economics consultancy with clients in the life science and insurance industries. Jennifer L. Bright is also President, Momentum Health Strategies.

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent to publication

Not applicable.

Availability of Data and Material

Information on the development of the IVI-MDD model is available at https://www.thevalueinitiative.org/ivi-mdd-value-model/. Additional information is available upon request from the authors.

Code availability

Not applicable.

Author Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by RZX, EFM, and MTL. The first draft of the manuscript was written by RZX and EFM and all authors provided comment and revision on subsequent versions of the manuscript. All authors read and approved the final manuscript.