The Use of Expert Elicitation Among Computational Modeling Studies in Health Research: A Systematic Review

Abstract

Background:

Expert elicitation (EE) has been used across disciplines to estimate input parameters for computational modeling research when information is sparse or conflictual.

Objectives:

We conducted a systematic review to compare EE methods used to generate model input parameters in health research.

Data Sources:

PubMed and Web of Science.

Study Eligibility:

Modeling studies that reported use of EE as the source for model input probabilities were included if they were published in English before June 2021 and reported health outcomes.

Data Abstraction and Synthesis:

Studies were classified as ‘formal’ EEs methods if they explicitly reported details of their elicitation process. Those that stated use of expert opinion, but provided limited information were classified as ‘indeterminate’ methods. In both groups, we abstracted citation details, study design, modeling methodology, a description of elicited parameters, and elicitation methods. Comparisons were made between elicitation methods.

Study Appraisal:

Studies that conducted a formal EE were appraised on the reporting quality of the EE. Quality appraisal was not conducted for studies of indeterminate methods.

Results:

The search identified 1,520 articles, of which 152 were included. Of the included studies, 40 were classified as formal EE and 112 as indeterminate methods. Most studies were cost-effectiveness analyses (77.6%). Forty-seven indeterminate method studies provided no information on methods for generating estimates. Among formal EEs, the average reporting quality score was 9 out of 16.

Limitations:

Elicitations on non-health topics and those reported in the grey literature were not included.

Conclusions:

We found poor reporting of EE methods used in modeling studies, making it difficult to discern meaningful differences in approaches. Improved quality standards for EEs would improve the validity and replicability of computational models.

Introduction

Computational models are widely used to evaluate the effects of policies and interventions in silico for health research.^1,2 Modeling can incorporate disparate sources of data in a coherent framework, compare different interventions that may not be ethical or feasible to evaluate through trials or observational studies, and account for uncertainty in the variables or effect sizes of the interventions of interest.^3,4 Crucial to any modeling project is the use of the best available data inputs for the question at hand. However, in many instances, modeling is applied to a problem or area with limited or even non-existent data. For example, the measures of the impact of new policies on behaviors or health outcomes, especially those that have not been previously implemented, are often unavailable. While the likelihood of some events can be extrapolated from similar policies that have been implemented, the potential impacts of new policies may be substantially different from past policies. In instances where policy effects are unknown, elicitation of expert judgement, also referred to as expert elicitation (EE), has been used as a method to obtain input parameters for computational models.⁵

Decision-makers have used EEs to quantify the unknown potential effects and levels of uncertainty of a given policy or intervention. Alternatively, existing evidence on the effects of an intervention may be inconsistent and elicitation can be used to build consensus. For example, EE has been used by the US Environmental Protection Agency, Food and Drug Administration, and other federal agencies, as well as international bodies such as the Intergovernmental Panel on Climate Change,^2,5 for model development⁶ and to gauge uncertainty in climate modeling.^7,8 In the field of health, the use of EE is most prevalent in pharmacoeconomic modeling to inform reimbursement decisions,^9–11 but has been used in models to inform cancer screening recommendations,¹² childhood obesity interventions¹³ and tobacco control policy.^14–16 The process has been used extensively by the National Institute for Health and Care Excellence in the UK for treatment coverage decision-making.¹⁷

EE refers to a variety of methods by which opinions of authorities in the field are collected and collated. Despite the widespread use of expert opinion as a source of data for health analyses, no consensus nor gold standard exist on the proper methods of planning, implementing, presenting, and using data from an EE.^9,18 There exists a range of formal to informal EE methods.^{10,11,18–24} The levels of complexity and rigor of the methods vary. For instance, elicitations can be conducted in-person, over the phone, online via video conferences, or with paper or electronic surveys. The measures elicited can include point estimates (e.g., medians or means), ranges of uncertainty (e.g., quartiles or confidence intervals), or probability distributions. Probability distributions are often developed by asking each expert placing a series of markers on a frequency chart to represent points on a probability density function. This method is commonly referred to as the histogram method,²² or alternatively ‘chips and bins’²⁵ or the Trial Roulette method.²⁶

Differences also appear in the methods by which expert responses are aggregated. Behavioral approaches to data aggregation include the nominal group technique or decision conferencing, where experts exchange views to draw forth a single expression of their judgment.²² Mathematical aggregation may involve linear pooling (i.e., simple averaging), logarithmic opinion pooling, weighted averaging, or Bayesian methods. In addition, a series of hybrid methods exist that combine both behavioral and mathematical approaches, such as the commonly used Delphi method.^9,27 In this approach, the experts provide initial estimates and are then asked to revise their estimates based on those of the other experts. This can take place over multiple rounds in an attempt to build consensus among all members of the elicitation.²⁷ All interaction in a Delphi study is facilitated by an investigator who understands the objectives and can effectively engage experts.

Finally, packages, such as the SHeffield ELicitation Framework (SHELF) and EXPLICIT, have been developed to aid researchers in the conduct elicitations. These packages provide templates and software that can assist with implementation and aggregation. SHELF in particular supports facilitator-guided group interactions and information sharing to arrive at a consensus.²⁸ EXPLICIT is an Excel-based package that provides real-time visualizations based on the expert’s elicited values.²⁹

Previous reviews on EE methods have highlighted these different approaches, but at the same time emphasize the need for increased methodological rigor and standardization.^10,11 To that end, Iglesias and colleagues, developed guidelines for reporting results from EEs.⁹ More recently still, Bojke et al. developed a set of guiding principles and criticisms to foster the development of a reference case for health-related EEs.¹⁸

Given the widespread use of EE in modeling health behaviors and outcomes, it is important to understand how EEs have been conducted in practice in order to better assess their utility for computational modeling studies. The purpose of this review is to systematically identify and compare methods used to elicit expert opinion to generate input parameters in computational modeling studies with particular emphasis on the way in which decisions regarding EE methods are reported. Two earlier reviews evaluated the use of EE limited to health technology assessments,^10,11 while a third looked at the use of EE when examining enteric illness.²⁰ We expand upon this work by including all health-related modeling studies as well as those that do not use formal EE methods. Consistent with previous reviews,^10,11 we focused exclusively on studies that elicited probability estimates, given the challenges in eliciting probabilities.^5,19 While EE can also be used to estimate costs or other outcomes, these inputs are more directly observable and their elicitation involves a separate set of issues.^30–33

Methods

This review follows the guidelines set by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) (Appendix 1), the protocol for the review was not registered in advance.³⁴

Data Sources and Searches:

Searches were conducted in PubMed and Web of Science for articles without date restrictions. The last search was conducted on June 1, 2021. A combination of MeSH and general terms were used to develop a search in PubMed (Appendix 2) which was then amended for Web of Science. No date or geographic restrictions were used. We also searched bibliographies and used citation tracking in Scopus to identify other potentially relevant studies missed by the database searches.

Study Selection:

Study inclusion criteria were determined a priori. To be eligible, peer-reviewed health-related computational modeling studies published in English needed to include one or more probabilistic model inputs generated through a de novo expert opinion elicitation process. Health-related modeling studies were defined as those in which the model reports a health outcome in humans. We consider only probabilistic inputs, such as transitions between different behaviors or health risks. As such, studies that elicited only cost estimates, resource use estimates, or utility values (such as quality-adjusted life years or disability-adjusted life years) were excluded. Additionally, studies were excluded if the elicitation process was previously reported in an earlier publication that meets the inclusion criteria. There were no restrictions on the methods used for elicitation. Studies that state the use of expert opinion but reported no information on how it was conducted or used were excluded.

Following the removal of duplicates, all study titles and abstracts were screened to determine eligibility by one reviewer (CC). Abstract and title screening was purposefully inclusive and any abstract that was deemed potentially relevant, or where relevance was not clear was included. The full texts of selected studies were then screened to determine final eligibility by two reviewers and reasons for exclusion were documented (CC & MK). Any uncertainty over exclusion was discussed and resolved by consensus.

Data Abstraction and Quality Assessment:

A data abstraction template was developed in Excel prior to data abstraction. Two reviewers independently abstracted relevant information from selected studies (CC, MK). Selected studies were classified into two groups, those that conducted a formal EE, and those that used indeterminate methods to elicit expert opinion. The determination was based solely on the information available in the study report and appendices. The former group, classified as “formal EE methods”, consists of studies that present a formal method to elicit and synthesize estimates from experts. The latter group, classified as “indeterminate EE methods”, includes studies that reported using experts to provide model inputs, but did not present a formal process, or provided insufficient details to determine the process by which expert judgments were elicited and synthesized. These studies are characterized by limited information regarding how expert judgements were elicited, collected, and collated.

For those classified as formal EE studies, four types of information were abstracted: 1) study-specific details (authors, date, location, study design, modeling method, and disease or condition of interest); 2) expert selection methodology (the number of, definition of, and identification and selection process for experts); 3) elicitation methods (the elicited parameters, elicitation process, aggregation methods, uncertainty estimation); and 4) incorporation of elicitation results (type of model input, and how uncertainty was modeled). Given the limited amount of information provided in studies classified as indeterminate EE methods, it was not possible to consistently extract study characteristics for comparison. We abstracted 1) study-specific details (authors, date, location, modeling method, and disease or condition of interest); 2) expert opinion details (elicited parameters, details on experts and methods, other sources used to complement expert opinion).

There is no widely accepted standard for assessing the quality of an EE used for generating input parameters of a health-related modeling study. Recent work by Bojke et al.¹⁸ has presented a framework, but consensus has not yet been reached on how to assess EEs. Therefore, we assessed the reporting quality of EEs based on an amended version of the reporting guidelines for expert opinion from Iglesias et al.⁹ Studies received one point for each of the criteria they satisfied for a total of 16 points (Appendix 3). Briefly, these criteria included a description of the research rationale, the definition of an expert and expert selection criteria, the development of a structured elicitation protocol, the piloting of the elicitation process, as well as clearly reporting the methodology for administering the EE, collecting data, aggregating data, and the use of performance measures. Finally, studies needed to present and interpret their EE results in the context of their model. Given the limited information provided by studies classified as indeterminate, it was not feasible to conduct a similar quality assessment.

Data Synthesis and Analysis:

Following full-text review and data abstraction, direct comparisons between studies were made to highlight the differences between formal EE studies and those that used indeterminate methods. Comparisons also illustrate the range of formal EE methods currently used to recruit experts, elicit data, aggregate responses, and account for uncertainty. Methods were compared qualitatively across studies to highlight the various ways in which expert opinion was utilized. For those classified as indeterminate EE methods, we present a summary of the types of experts and number of experts consulted.

Results

Study Selection and Characteristics

Searches identified 1,520 unique studies. The screening process and reasons for exclusion are outlined in the PRISMA diagram (Figure 1). Following title and abstract review, 355 articles were deemed as potentially eligible. The full-text review excluded 203 studies. The most common reason for exclusion was that the study did not use computational modeling (n=84). An additional 36 studies did not report health outcomes, while 34 used expert elicitation to generate model inputs that were not probabilities (cost estimates, resource use estimates, or utility values). Notably, 12 studies were excluded because they cited expert opinion as a source of model input parameters in the abstract of the paper but provided no further information in the full text.

Figure 1.

PRISMA diagram of identified studies of Expert Elicitations conducted in health services research-based modeling studies.

In total, 40 studies met the inclusion criteria for using formal EE methods for model inputs. An additional 112 studies used indeterminate methods to elicit expert opinion.

Formal Expert Elicitation Studies

Of the 40 EEs studies included (Table 1), the most common study types were cost-effectiveness, cost-utility, or cost-benefit analysis (n=29). Other study types included the evaluation of hypothetical policy interventions (n=2), EE methods papers that included modeling (n=3), and a series of one-off study types, such as a risk management analysis or value of information analysis. The type of computational model used in the majority of studies was decision-analytic (n=24).³⁵ A wide range of diseases/conditions were examined with no single one being most prevalent.

Table 1.

Study characteristics of Expert Elicitations conducted in health research-based modeling studies.

Author, Year	Location	Study Design	Model Type*	Disease/Condition
Akpo, 201760	Belgium	Cost-Utility Analysis	State Transition Markov Model	Chemotherapy Induced Neutropenia
Annemans, 201061	Belgium	Cost-Effectiveness Analysis	Markov Model	Herpes Zoster Vaccination
Apelberg, 201814	US	Hypothetical Policy Evaluation	Dynamical Systems Model	Tobacco Use
Banz, 200568	Germany	Cost-Effectiveness Analysis	Decision Analytic Model	Heart Failure
Bojke, 201041	UK	Comparative Elicitation Methods for Cost-Effectiveness Analysis	Probabilistic Decision Analytic Model	Psoriatic Arthritis
Chiêm, 201240	Belgium	Chronic Disease Epidemiological Modeling	Rule-based Modeling	Elderly Depression
De Graeve, 200536	Belgium	Cost-Effectiveness Analysis	Decision Tree Model	Schizophrenia
Dewilde, 200962	UK	Cost-Effectiveness Analysis	Microsimulation	Severe Chronic Pain
Dubinsky, 200272	US	Cost-Effectiveness Analysis	Decision Analytic Model	Inflammatory Bowel Disease
Duenas-Meza, 202037	Colombia	Cost-Utility Analysis	Decision Analytic Model	Childhood Asthma
Ekaette, 200742	Canada	Risk Management Analysis	Probabilistic Fault Tree	Cancer
Firoz, 200643	US	Comparative-Effectiveness Analysis	Decision Analytic Model	Graft-vs-Host Disease
Garay, 201963	Argentina	Cost-Utility Analysis	Markov Model	Acute Kidney Injury
Garthwaite, 200864	UK	Elicitation Methodology for Costs and Benefits Analysis	Treatment Pathway Model	Bowel Cancer
Gordon, 201244	Australia	Cost-Effectiveness Analysis	Decision Analytic Model	Esophageal Cancer and High-grade Dysplasia
Grand, 201967	UK	Cost-Effectiveness Analysis	State Transition Markov Model	Endometriosis-related Pain
Grigore, 201645	UK	Comparative Elicitation Methods for Health Technology Assessment	Probabilistic Decision Analytic Model	Prostate Cancer
Gyllensten, 201253	Sweden	Cost-of-Illness and Disease Estimation	Decision Tree Model	Drug-related Morbidity
Handels, 201765	The Netherlands	Cost-Utility Analysis	Decision Analytic Model	Alzheimer’s
Hinde, 201558	UK	Cost-Effectiveness Analysis	Decision Analytic Model	Lung Cancer
Hitimana, 201956	Rwanda	Cost-Effectiveness Analysis	Monte Carlo Simulation	Antenatal Care
Hsu, 201159	Taiwan	Cost-Effectiveness Analysis	State Transition Model	Colon Cancer
Kauf, 201066	US	Cost-Utility Analysis	Decision Tree Model and Lifetime Discrete Event Simulation	HIV
Keren, 200254	US	Cost-Effectiveness Analysis	Decision Analytic Model	Newborn Hearing Impairment
Kip, 201846	The Netherlands	Cost-Effectiveness Analysis	Decision Tree	Myocardial Infarction
Loeve, 200039	US	Cost-Effectiveness Analysis	Microsimulation	Colorectal Cancer
Lubell, 201157	Africa, South and Southeast Asia	Elicitation Methods and Cost-Effectiveness Analysis	Decision Tree	Untreated Febrile Illness
McKenna, 201069	UK	Cost-Effectiveness Analysis	Decision Analytic Model	Stable Angina
Meeyai, 201538	Thailand	Cost-Effectiveness Analysis	Dynamic Transmission and Decision Analytic Model	Seasonal Influenza
Petersohn, 202150	Netherlands	Comprehensive Uncertainty Assessment for Cost-Effectiveness Analysis	State Transition Model	Peripheral Artery Disease
Poncet, 201570	Switzerland	Cost-Effectiveness Analysis	Decision Analytic Model	Sudden Cardiac Death in Psychiatric Patients
Prosser, 201849	US	Cost-Effectiveness Analysis	Decision Analytic Model	Pompe Disease
Sankasting, 202012	The Netherlands	Cost-Effectiveness Analysis	Microsimulation Model	Breast Cancer
Schumock, 200751	US	Cost-Benefit Analysis	Decision Analytic Model	Bipolar Disorder
Soares, 201347	UK	Value of Further Research Cost-Effectiveness Analysis	Decision Analytic Model	Pressure Ulcers
Stevenson, 200948	UK	Expected Value of Sample Information Analysis	Decision Analytic Model	Fracture
van Hoeven, 201955	The Netherlands	Cost-Effectiveness Analysis	Decision Analytic Model	Blood Transfusion Parvovirus B19 Infection
Veerman, 200913	US	Hypothetical Policy Evaluation	ODE Model	Childhood Obesity
Walton, 201052	US	Cost-Benefit Analysis	Decision Analytic Model	Osteoarthritis
Wielage, 201371	US	Cost-Utility Analysis	Discrete-State, Semi-Markov Model	Osteoarthritis

^*As specified in the study’s methods.

Definitions of Expert

Experts were generally defined by their topical experience. For instance, clinical experts on the health condition of interest were selected in 35 studies (Table 2). In seven studies, non-clinical experts were recruited. These included epidemiologists, health economists, and academic researchers who were consulted in combination with clinicians in five studies.^12,36–39 Studies recruited ‘academic researchers’ in two instances.^13,14 One study defined experts as medical, pharmacy and public health students who were recruited along with ‘professional researchers.’⁴⁰

Table 2.

Expert selection methods for Expert Elicitations conducted in health research-based modeling studies.

Author, Year	Definition of Expert	Expert Selection Method	Number of Experts
Akpo, 201760	Oncologists and hematologists	NR	7
Annemans, 201061	Primary care physicians, dermatologists, ophthalmologists and pain specialists or neurologists	NR	20
Apelberg, 201814	Canadian and US-based researchers with extensive publication records on tobacco smoking	Scopus was used to identity the top 30 tobacco policy and tobacco science experts by citation count. Those with an H-Index above 20 were screened for potential bias.	8
Banz, 200568	Physicians with expertise treating heart failure	NR	9
Bojke, 201041	Senior Rheumatologists	Convenience sample; based on advice from clinical collaborator and proximity to study team	5
Chiêm, 201240	Medical, Pharmacy and Public Health students at the Université Catholique de Louvain-la-Neuve and professional researchers in elderly care from the Institute of Health and Society	NR	28
De Graeve, 200536	Belgian psychiatrists and a health economist	NR	11
Dewilde, 200962	Clinical experts experienced in analgesia	NR	5
Dubinsky, 200272	Gastroenterologists	NR	5
Duenas-Meza, 202037	Pediatric pulmonologists, asthma experts, physical therapists, health economists	NR	9
Ekaette, 200742	Unclear: medical physicists, oncologists, and radiation dosimetrists/therapists at Tom Baker Cancer Center	Convenience sample; may have used experts available at the cancer center	12
Firoz, 200643	Oncologists and dermatologists	Convenience sample; from academic medical centers	16
Garay, 201963	Local nephrology experts	NR	6
Garthwaite, 200864	Medical experts	NR	4
Gordon, 201244	Esophageal surgeons	Convenience sample; chosen from high-volume centers with an active interest in esophageal cancer research.	5
Grand, 201967	Gynaecologist	NR	1
Grigore, 201645	Consultant urologists or oncologists	Convenience sample; through networks within the research department and websites of professional associations.	10
Gyllensten, 201253	Pharmacists with clinical expertise	Strategic sample to represent a wide range of practice and care levels identified by contacting Drug and Therapeutics Committees in Sweden	29
Handels, 201765	Family physicians; neurologists; psychiatrists; geriatrician with 8 to 30 years of clinical experience	NR	7
Hinde, 201558	Clinicians	NR	3
Hitimana, 201956	Specialists trained in obstetrics having practiced in Rwanda for at least 5 years	All OB/GYNs in Rwanda who met the criteria	8
Hsu, 201159	Taiwanese colorectal cancer surgeons and medical oncologists	NR	12
Kauf, 201066	Infectious disease specialists specializing in HIV care	NR	5
Keren, 200254	National experts on hearing screening and language development in the deaf	Authorities in the field were asked to identify individuals with extensive experience and a balanced perspective on effectiveness of hearing screening.	4
Kip, 201846	Cardiologists	Convenience sample; selected from clinicians at two Dutch teaching hospitals	10
Loeve, 200039	Cancer modelers and health services researchers	NR	19
Lubell, 201157	Malaria Specialists	Selected based on expertise and international reputation in managing malaria	24
McKenna, 201069	Expertise and knowledge of enhanced external counterpulsation	NR	5
Meeyai, 201538	Epidemiologists and clinicians with relevant experience	NR	10
Petersohn, 202150	Vascular surgeons, cardiologists, internal medicine specialists, vascular medicine specialists, GPs and vascular surgery nurses treating periphertal artery disease in the Netherlands	Experts were selected if they had a publication record on the topic and proven expert knowledge. Experts could recommend other experts to the researchers	12
Poncet, 201570	Cardiologists and Psychopharmacologists at Universities of Geneva, Lausanne and Milano	NR	13
Prosser, 201849	Clinical and scientific experts in Pompe disease	Experts identified through a systematic evidence review	6
Sankasting, 202012	Experts in breast cancer screening	Involved in digital breast tomosynthesis trials or affiliated with the Dutch Expert Centre for Screening.	9
Schumock, 200751	Psychiatrists	History of prescribing two bipolar treatments then ranked by number of prescriptions	15
Soares, 201347	Nurses with experience treating people with pressure ulcers	Convenience sample; recruited nurses from hospital and community settings	23
Stevenson, 200948	Clinicians	Convenience sample; members of the study team	3
van Hoeven, 201955	Physicians specializing in haematology, neonatology and immunology	Snowball sampling	25
Veerman, 200913	Academic experts with a publication record or knowledge of the field	Recent publications in the field or known to the authors to be experts	8
Walton, 201052	Rheumatologists and pain specialists.	Stepwise process based off of prescribing history	30
Wielage, 201371	US-based pain specialists	NR	5

Expert selection methods were unreported in 20 of the 40 studies. Eight studies reported a selection method that used convenience sampling in which experts were selected due to their proximity to the study team.^41–48 Of the remaining studies, four used publication records;^13,14,49,50 two used prescribing records to identify clinicians who have used the medications being evaluated;^51,52 two consulted external advisors;^53,54 and the rest used snowball sampling;^50,55 trial involvement;¹² all certified obstetrician/gynecologist in the country of study with years of experience;⁵⁶ or international reputation.⁵⁷ The average number of experts recruited per EE was 11 with a range of 1 to 30. No studies gave reasons for their choice regarding the number of experts recruited. Additionally, it was not possible to verify the suitability of experts or to assess the potential for bias introduced in the selection process, because most studies did not provide sufficient details on the selection process of experts or provide a list of the experts.

Elicitation Methods

Elicitation methods for each included study are reported in Table 3. No methods describing the process for implementing the EE were reported in three studies.^37,58,59 Among the remaining studies that did report elicitation methods, considerable variation existed regarding the level of detail included in the methods descriptions. The most frequent methods were Delphi or modified Delphi process (n=10)^{13,36,49,54,56,57,60–63} while two used the SHELF method,^38,50 both methods rely on multiple rounds of discussion to build consensus among experts. Elicitations were frequently done virtually or over the phone (n=13)^{14,36,43,49,51,52,55,56,60,64–67} while only five specified that the elicitation was conducted in-person.^{39,41,45,64,68} Web- or spreadsheet-based elicitation tools were used to facilitate data collection in 9 elicitations.^{12,40,45,47,50,64,67,69,70}

Table 3.

Elicitation methods for Expert Elicitations conducted in health research-based modeling studies.

Author, Year	Elicited Parameters	Elicitation Method	Aggregation Method	Uncertainty Method
Akpo, 201760	Transition probabilities; resource use	Modified Delphi* process with two rounds. 11 responses to online survey which were analyzed using median, 25th and 75th percentiles. 7 experts revised/validated their responses via email or in an interview.	NR	Tornado diagrams to assess one-way effects and probabilistic sensitivity analysis.
Annemans, 201061	Use of primary and secondary care, medications, diagnostic tests, and hospitalizations	Delphi* method over two rounds	NR	Probabilistic sensitivity analysis
Apelberg, 201814	Probabilities of smoking initiation, cessation and switching	Three online workshops present background material, discuss relevant material. After the second panel, individual interviews elicited minimum, maximum, 5%, 95%, 25%, 75% and 50% values	Individual experts responses were sampled using the Latin Hypercube method and 1000 simulations were run per expert and aggregated into one distribution.	The median, 5th and 95th percentiles of the aggregated distribution is recorded.
Banz, 200568	Clinical effects of compared therapeutic approaches; drug usage; percentage of patients requiring prolonged hospitalization	In-person interviews	NR	NR
Bojke, 201041	Initial Health Assessment Questionnaire gain; disease progression following treatment response; disease progression following withdrawal from treatment	In-person interviews. Mathematically elicit beliefs using a spreadsheet based exercise. Correlation between the two parameters was incorporated with responses conditional upon previous answers. Background materials were provided. Histogram method* was used to represent beliefs and experts were asked to place 20 crosses on a frequency chart representing their beliefs about the distribution of a particular quantity	Expert responses are calibrated using the distribution of initial HAQ gain from experts and the distance from experts expected values to the known parameters. Results were aggregated by either linear pooling or random effects meta-analysis. Both equal weights and calibrating weights were used.	Compare all four methods to two alternative scenarios.
Chiêm, 201240	Effects of social contacts on depressed and non-depressed elderly	Excel spreadsheet with questions and visualizations of responses was sent out to potential participants	Linear pooling of all experts and a subset of experts	Simulations were run using overall responses and then subsets of experts
De Graeve, 200536	Treatment transition probabilities; probability of remaining relapse-free for specific treatments; rates of hospitalization and resource utilization	Two expert groups were consulted. No methods reported for one. The other used modified Delphi method* using 11 individual conference calls. Multiple rounds of interviews were planned, but only one took place given that only minor fluctuations were seen when extrema were removed.	NR	The only parameters from the “expert committee” were included in sensitivity analysis. Specifically: long-term treatment efficacy and improvements in remaining relapse free. Delphi panel parameters were not varied.
Dewilde, 200962	Patient contacts with healthcare professionals, concurrent therapy use, treatment discontinuation	Two-round Delphi method* with repeated circulation of a questionnaire. Experts were asked to estimate average resource utilization. Mean and Median were circulated and experts provided feedback. Second round results were used for inputs.	Linear pooling	Final cost parameters were varied using a Gamma distribution.
Dubinsky, 200272	Gold standard sensitivity and specificity; probability of IBD given symptoms	Completed a 17 item questionnaire that addressed clinical scenarios	Reached consensus	One-way sensitivity analyses were conducted on elicited parameters.
Duenas-Meza, 202037	Transition probabilities	NR	Reached unanimous consensus	NR
Ekaette, 200742	The probabilities of rare but potentially serious incident-prone tasks when preparing radiation treatment over the course of one year of operation	Nominal group technique. Following training in probability, participants individually present their initial judgments and then discussed all the estimates in a facilitated sessions which was followed by an opportunity to revise. Consensus was not required in order to capture uncertainty.	PERT distributions were derived for each expert and integrated into one distribution per incident using Monte Carlo simulation.	Consensus was not sought between experts, Monte Carlo simulation was used to develop probability distributions. Elicited probabilities were compared to incident reports
Firoz, 200643	Prevalence of graft-vs-host; sensitivity and specificity of skin biopsy	A 15-minute phone interview including a clinical scenario to determine treatment decisions and estimate probability of graft-vs-host as well as sensitivity and specificity. Also provided ratings of the desirability of different outcomes used to estimate utility benefits	Linear pooling	Multiple sensitivity analyses (one-way and multi-way) explore impact of parameters. Individual physician estimates were modeled and compared to physician’s stated preferences
Garay, 201963	Length of ICU stay; probability of switching treatment; survival post ICU discharge, 60 after treatment, 180 days after treatment; health utilities	Delphi-like panel method*	Consensus was reached	One-way sensitivity analyses were conducted on all parameters. Probabilistic sensitivity analysis simulated results 1000 times
Garthwaite, 200864	9 parameters including: time from symptom onset to GP visit; proportion of patients referred for investigation; various treatment related proportions; survival duration for metastatic patients with best supportive care	Experts were questioned in separate sessions three of which were conducted in-person and the last over the phone. Experts provided median values and a series of quantiles. An elicitation software was used to determine a multivariate normal distribution to represent experts opinions for two parameters. For others, the medians and quartiles were assessed, the point estimates and extrema, or the point estimates on their own were used.	Single expert’s values were used for each parameter.	Where experts values differed, individual values from experts were assessed separately.
Gordon, 201244	Likelihood of treatments and select outcomes of treatments	Independent interviews followed by email correspondence to build consensus through agreement/disagreement with the group range.	Linear pooling	Range of elicited parameters was used in probabilistic sensitivity analysis.
Grand, 201967	Utility estimates; State transition probabilities	For utility values, a gynaecologist was presented with available evidence and asked to estimate utility values using a web-based elicitation tool (MATCH Uncertainty Elicitation Tool) using a “chips and bins” method*. For state transition probabilities, 1000 hypothetical patients were assigned to each states and an expert estimated the rates for monthly transitions into other states.	NA-only one expert was consulted	Probabilistic sensitivity analysis was used for all elicited parameters with wide ranges. Alternative QALY estimates based on the literature were also considered.
Grigore, 201645	Proportion of patients experiencing complications from testosterone ‘flare’; proportion of patients experiencing paraplegia	In-person interviews were used to ask each expert to provide answers using each method. Excel-based exercise was used. Compares the histogram method and the hybrid method. Histogram method* asks experts to pace a number of crosses on a frequency chart to represent their uncertainty. The hybrid method elicits the lowest, highest and most likely quantity and intervals are built using a formula to divide the distance between estimates into equal bins. Experts are then asked to enter the probability that their estimated value lays within each interval. A cooling off period between sessions was used to avoid anchoring to results of the first session.	Linear pooling method assuming equal weights and calibrated based on responses to seed question (testosterone ‘flare’).	Probabilistic sensitivity analysis and associated expected value of perfect information analyses we evaluated. The uniform distribution of uncertainty was simulated and the inverse cumulative beta probability density function. Results from each of the four EE methods are compared.
Gyllensten, 201253	Probabilities for all events in the decision tree	Two round guided processes where experts worked through each arm of the decision tree. Experts were mailed and their results and interquartile ranges of other experts and given the opportunity to revise responses	Linear pooling	First and third quartiles of experts’ estimates were used in sensitivity analyses.
Handels, 201765	Patient management; diagnostic impact	A survey was sent to experts with detailed questions and sample answers indicating the background and scale for each question.	NR	Experts were not asked to provide uncertainty estimates. Uncertainty in the elicited parameters was not evaluated
Hinde, 201558	Prior probabilities for lung cancer transitions to be calibrated to lung cancer incidence	NR	NR	NR
Hitimana, 201956	Mortality reduction from Antenatal Care interventions	Modified Delphi method* with phone interviews that included background information, elicitation questions, and policy recommendations. Experts were given the opportunity to revise their responses based on the group responses. Second round responses were used as final inputs.	Linear pooling of two outcome scenarios. An “optimistic” of the most favourable half of responses and a “pessimistic” based on the least favourable half of expert responses.	ICERs were calculated for the optimistic and pessimistic scenarios
Hsu, 201159	Local treatment regimes; drug administration patterns; adverse effect management	NR	NR	NR
Kauf, 201066	Treatment regimens, frequency of office visits, ED contacts, hospitalizations, and symptomatic treatment among other resource use parameters	Individual phone based surveys.	Linear pooling	Some elicited parameters were varied in one-way sensitivity analysis.
Keren, 200254	Effectiveness of interventions on improving language outcomes	Modified Delphi method*.	NR	Performed extensive sensitivity analyses around these base-case estimates
Kip, 201846	Improved diagnostic performance; treatment received; discharge rates; treatment changes due to improved diagnostics	Individual written questionnaire with no interaction between experts	Linear pooling	Probabilistic sensitivity analysis included elicited parameters.
Loeve, 200039	Sojourn time; sensitivity of sigmoidoscopy by adenoma size; screening costs	Two expert meetings at the National Cancer Institute and email correspondence	NR	Sensitivity analyses conducted on elicited parameters.
Lubell, 201157	Probability that untreated malaria would become severe/fatal; probability that a non-malaria illness requires antibiotic treatment; the probability that bacterial infection would become severe/fatal	Delphi method* with an initial questionnaire providing point estimates and feedback on the questionnaire. Mean, median and range were graphed and provided to experts with a revised survey.	Linear pooling and median response	The range of each elicited parameter was used in one-way sensitivity analyses and plotted in a tornado diagram.
McKenna, 201069	Duration of treatment effects beyond 12 months	Independent Excel-based exercise where experts placed 20 crosses on a frequency chart to derive a distribution. Mean and standard deviations of the probability distribution	Linear pooling with equal weights	Probabilistic sensitivity analysis included elicited parameter, as well as a best and worst case assuming full benefit and no benefit
Meeyai, 201538	Probability of immunity by age; initial proportion infected in each age group	Sheffield Elicitation Framework* using both individual and group elicitation	Probability distributions were obtained using expert elicitations as prior distributions with the likelihood of observed data to obtain posterior distributions	Sensitivity analyses test the impact that of elicited parameters on the results.
Petersohn, 202150	Effectiveness of clopidogrel on major adverse cardiovascular or limb event, and risk of bleeding; patient loss of quality of life after acute or post-acute cardiovascular events; health state costs of peripheral artery disease;	Excel-based EXPLICIT tool* to facilitate remote elicitation.	Linear pooling with equal and unequal weights; random effects meta-analysis. Probability distributions for the effectiveness of clopidogrel were aggregated with evidence from the literature to obtain posterior distributions	Multiple sensitivity analyses including probabilistic sensitivity analysis, cost-effectiveness acceptability, and value of information
Poncet, 201570	Risk of Torsades-de-Pointes	Individually and independently asked to estimate risk using the histogram method.	Linear pooling with equal weights	One elicited parameter was used for sensitivity analysis; all were varied in probabilistic sensitivity analysis
Prosser, 201849	Prevalence of classic and non-classic infantile-onset of Pompe disease in universal newborn screening and clinical identification groups; sensitivity and specificity of screening	Modified Delphi method* with three rounds. Expert panel met via webinar and provided feedback on model structure and input parameters. Following meetings, parameter estimates were revised to reflect the expert opinion of the group.	Consensus was sought during expert panel meetings.	One-way sensitivity analyses were conducted on all parameters.
Sankasting, 202012	Sensitivity, positive predictive value, and costs of digital breast tomosynthesis	Excel-based EXPLICIT tool*	Linear pooling of unweighted responses by averaging the highest, lowest and most likely values.	Elicited parameters were varied using the boundaries from a beta-PERT distribution of the elicited values
Schumock, 200751	Probability of treatment success, adverse events and subsequent healthcare utilization.	A trained interview contacted experts via phone to estimate the number of patients who would experience an event out of 100.	Linear pooling	Monte Carlo simulation using a uniform probability distribution based on the minimum and maximum value from expert responses.
Soares, 201347	Probability of ulcers of a certain grade; proportion of patients healed; probability of complications; effectiveness of different treatments, among others	An Excel-based instrument was developed based on a modified histogram approach which allows experts to enter conditional probabilities (among the same expert).	Linear pooling fitted with parametric distributions.	Probabilistic sensitivity analysis including elicited parameters. Various scenarios considered different combinations of existing evidence, elicited evidence, and pilot trial data.
Stevenson, 200948	Efficacy of extended treatment	Bisection method; experts provide their median 25th and 75th percentile for each parameter and a parametric distribution is fitted to each percentile using a least squares approach. This was reported back to clinicians to ensure they corresponded to clinicians beliefs	Individual expert’s distributions were sampled from and the combined values were approximated in a statistical distribution.	The distribution of elicited parameters is used for modeling
Van Hoeven, 201955	Adverse events due to infection	An online questionnaire asks experts to estimate the most likely value, minimum and maximum.	Expert responses are used to fit one distribution per expert. Expert distributions are pooled by specialism with equal weights for expert responses. Pooled distribution is randomly sampled to generate a point estimate and 95% confidence interval.	Various sensitivity analyses explore the range of adverse events including number of infections, and mortality
Veerman, 200913	Dose-response relationship between TV advertisements and total energy intake	Delphi method* over two rounds; asked for upper and lower bounds and the central estimate.	Non-parametric bootstrapping on the central estimate using Ersatz programme and 5000 iterations	NR
Walton, 201052	Probability of adverse events	An online survey conducted by Inspire Opinions, a subsidiary of a large national healthcare marketing research firm. Experts were asked to estimate the number of patients who would experience an event out of 100.	Linear pooling	Sensitivity analyses incorporated various estimates from the literature using the implied relative risks from the expert panel.
Wielage, 201371	Treatment utilization; treatment management; adverse drug events;	Some questions asked for the least, most, and usual which were averaged as the physician’s response; other questions asked for single percentages	NR	Discontinuation rates and length of treatment were varied in one-way sensitivity analyses and probabilistic sensitivity analysis

^*Additional details on methods provided in the introduction.

Eleven studies elicited point estimates from experts. Of those, the most frequent estimates were the median (n=5), mean (n=3), and ‘most likely value’ (n=3). Expert uncertainty was elicited by asking experts to provide ranges or different quantiles in seven studies. While seven other studies used the histogram method or a modified histogram method (where experts mark a frequency chart) to elicit expert uncertainty.^{41,45,47,50,67,69,70}

Aggregation Methods

Aggregation methods for each included study are reported in Table 3. No aggregation method was reported in ten studies,^{36,39,54,58–61,65,68,71} one study did not aggregate responses as only one expert was consulted,⁶⁷ and one study modeled individual expert responses separately in lieu of aggregation.⁶⁴ Among those that reported aggregation methods, four used behavioural aggregation (where experts were asked to reach consensus)^37,49,63,72 and the remainder used mathematical aggregation. The most frequently used form of mathematical aggregation was linear pooling (n=18).^{12,40,41,43–47,50–53,56,57,62,66,69,70} Performance measures,^73,74 where expert’s estimates are weighted based on their responses to seed variables (also known as calibration questions) with answers known to the facilitator but unknown or not readily available to the experts, were only used in two studies.^64,66 No other weighting, such as by expert’s reported level of confidence, was used. Those that did not use linear pooling reported a diverse set of aggregation methods. These included aggregation based on a simulated distribution of expert estimates such as through the use of Latin Hypercube sampling¹⁴, random-effects meta-analysis,⁵⁰ or Monte Carlo simulation.⁴² One study compared three alternative pooling approaches and concluded that differences in results based on pooling method existed, but that it remains unclear which method was preferable.⁶⁶

Uncertainty

Methods that modelers used to account for the uncertainty in EE-based inputs are reported in Table 3. In most cases (n=26), one-way or probabilistic sensitivity analysis was used to evaluate the uncertainty of outcomes due to elicited parameters by varying the input value across expert estimates or using a distribution generated from the estimates. Another approach was to simulate the responses of each expert individually (n=3). In seven studies, sensitivity analyses were not conducted.

Reporting Quality

We found substantial variation in the reporting quality of EEs (Appendix 4). The average score based on our 16-point quality scale was 9.4. The most underreported characteristics were lack of an explicit reference to an elicitation protocol (n=3), reporting whether or not performance measures were used (n=6), and lack of piloting the elicitation (n=7). Less than half of studies reported whether training materials were used or made available, described the number or phrasing of questions, or presented the individual as well as aggregated expert responses.

Indeterminate Expert Elicitation Methods

Of the 112 studies that presented indeterminate EE methods (Appendix 5), the most common study design was cost-effectiveness or cost-utility analysis (n=89). Experts were asked to provide estimates for a wide range of parameters, from the sensitivity and specificity of testing instruments, to state transition probabilities or the occurrence of adverse events. Eight studies cited the use of experts but did not specify the parameters that were elicited.

Among the studies where the type of expert was specified, clinical experts were most common (n=36). In eight instances, combinations of experts (clinicians, researchers, and/or patients, among others) were consulted and in five studies experts were members of the study team. In 63 studies, no specifications on the type of expert were provided.

Eighteen studies, elicited information from 2-5 experts, five utilized 6-10 experts, three elicited information from only one expert, and two included more than 10 experts. A total of 77 studies provided no information on the number of experts used in the elicitations, while seven studies consulted an unspecified number of experts.

A total of 47 studies included no details on the size or composition of the panels. Across the board, almost no information was provided regarding how expert opinion was elicited or aggregated. Expert opinion was supplemented by additional data mainly from the published literature or literature reviews in 65 studies.

Discussion

Our systematic review focused on the use of expert opinion to inform computational modeling health studies. Expert elicitation is a common approach used to generate parameter estimates employed in computational modeling health outcome studies. We found 152 studies utilizing some form of expert opinion for the generation of probabilistic model input parameters. However, detailed descriptions of the methods used to elicit expert opinions were often inconsistent and incomplete. Twelve studies could not be included because, while the use of expert opinion was highlighted in the abstract, they failed to mention its use in the text of the study. We classified 40 studies as using formal EE methods and 112 as using indeterminate EE methods.

Consistent with previous reviews of formal EE methods, we found poor levels of EE methods reporting.^10,11 Unlike previous reviews, our results extend beyond studies exclusively in pharmacoeconomics and cost-effectiveness research. While most of the included studies were still cost-effectiveness analyses, we identified 11 non-cost-effectiveness studies that used formal EE methods. Additionally, our study is the first to evaluate indeterminate EE methods in addition to those with formal EE methods. Indeterminate EE methods were more common than formal EE studies and reported minimal information about how experts were defined and selected or how inputs were generated, raising substantial concerns regarding the reliability and generalizability of the elicitation process, the transparency of derived modeling inputs and the validity of modeling results.

Among formal EE method studies, lack protocols and piloting of elicitations were most absent. However, a substantial number of studies either failed to report key methodological information, such as the expert recruitment/sampling process and how expert estimates were collected and collated, or reported incomplete results from elicitation without including all experts’ responses prior to aggregation. Studies with indeterminate methods did not report any of this information.

Our review identified two important avenues of future work on the use of EE. The first, is the development of EE best practices. The second is determining when EE is most valuable to researchers conducting health-related modeling studies.

Best Practices

With expert opinion playing a key role in studies that are directed at health policy and medical reimbursement decisions, ensuring the highest methodological quality is essential. Yet, modeling and cost-effectiveness best practices, such as those set forth by the 2012 ISPOR-SMDM Modeling Good Research Practices Task Force and the Second Panel on Cost-Effectiveness in Health and Medicine, among others, make limited mention of when and how expert opinion should be relied upon for input parameter estimation.^75–78 While the more recent ISPOR guidelines discuss the value of EE, limited guidance is offered on the proper methodologies.⁷⁹ The lack of consensus and guidance on methods have likely contributed to the limited reporting found in this and previous reviews. Earlier this year, Bojke et al. sought to develop reference case methods for EEs in the healthcare setting offering nine principles to guide the use of EE.¹⁸ However, modelers are still left with considerable discretion to determine when and how expert opinion should be elicited, aggregated, modeled, and communicated, and readers with almost no ability to evaluate the quality of the resulting parameters and results if editors and reviewers do not know of these guiding principles.

While a lack of reporting does not necessarily equate to poor quality or indicate bias,⁸⁰ persistent underreporting of essential information can reduce the credibility and replicability of modeling studies that use EE. Recent reporting guidelines provide an extensive outline of what health-related EEs should report, but do not go so far as to highlight best methods.⁹ At a minimum, studies should report the composition of the expert panel and selection process, the preparation materials provided and the elicitation and aggregation methods. These pieces of information are essential to ensure the replicability of EEs and allow reviewers and readers to assess potential bias. A range of contextual biases can influence EE results including anchoring, availability bias, the base rate fallacy, and overconfidence which can be amplified by inappropriate EE methods.^19,22,81,82 Better reporting is needed to facilitate the evaluation of potential bias and highlight any value of the EE. In addition, consistent and extensive poor reporting may call into question the general value of EEs, thereby making it difficult to use this methodology when there is a clear need for a well-conducted EE.

In the absence of methodological details, at best, it is not clear whether EE provides modelers information that improves upon a review of the literature. However, at worst, it could enable the use of EE results which highly bias the model findings. Research societies whose members make extensive use of EE and computational modeling should reach a consensus on how to conduct and report on their use. The development of reporting guidelines and best practices can promote transparency and allow for a fuller assessment of any sources of bias.^80,83,84

Beyond research society guidelines, journals and peer-reviewers also have a responsibility to present high-quality EE research. Since expert knowledge has been considered by some to be a lower form of evidence than observational or clinical trial data,² editors and reviewers may disregard stand-alone EE manuscripts or modeling studies that rely heavily on EEs regardless of their methodological rigor. Thereby, journals may limit the discussion, review and development of EE methods that could advance the field. Journals, particularly those that publish high-quality model-based research, should open venues to facilitate the publication and peer review of stand-alone EEs with emphasis on adhering to principles and reporting guidelines for EEs.^9,18 Separate peer-reviewed publication of EEs should be encouraged, as this would enable independent review of the EE and the subsequent modeling study.

Value of Expert Elicitation

Greater reporting of methods could allow for a more robust discussion about the application of EEs, which are generally considered to be of value when data is scarce^10,11 or a lack of consensus exists.⁵ However, limited research exists on when the use of EE provides value over other methods to generate model input parameters. In some instances, EE is the only way to generate necessary model inputs given the lack of existing data. In others, it can be used to synthesize disparate sources or build consensus when evidence is conflictual. As seen in 65 of the studies with indeterminate methods, EE can also be combined with other sources of data. Rarely, justifications were provided as to why expert opinion was better than the information available in the published literature. The value of EE may also vary by the type of intervention being studied, where some interventions may be better suited to other types of data collection methods. EE may be useful when primary data collection is unethical, infeasible, or time sensitive. Even in studies that conducted formal EEs, few provided a robust discussion of the merits of conducting an EE relative to other data collection methods.

Limitations

Despite a large number of studies that represent the spectrum of expert opinion, we note that our review is limited to only those studies where expert opinion was conducted to generate probability inputs for modeling health outcomes. EE may be used for the development of other types of model input parameters (such as costs, or utilities) or to inform the model development process, but these may require alternative methods to those described here. Additional insights on the use of EE may be gleaned from studies that focus on different types of outcomes beyond health. Regardless of the methods used, EEs for any model input parameters should be subject to the same critical review as EEs that elicit probabilistic inputs. Additionally, we only included peer-reviewed journal articles and did not search the grey literature which is likely to be much larger than what is represented in this review. Expert opinion generally, and formal EE in particular, is used by a diverse array of government agencies.^5,10,11,14 Reports in the grey literature may present other EEs that have been used directly for policy modeling. We also limited our search to elicitations that were used to inform computational modeling. Studies of EEs conducted outside of modeling settings may provide valuable information to modelers on methods for conducting and aggregating data from EEs.

Conclusion

Despite these challenges and limitations, our review illustrates the extensive use of expert opinion, both formally and indeterminately in health-related modeling studies to generate input probabilities. We highlight the pervasive lack of reporting of important methodological details about expert elicitation. Among formal EEs, we found considerable variation in the methods used for expert selection, data collection and aggregation, and in accounting for uncertainty. The comparison of these methods may be useful to modelers looking to implement their own elicitations. Further work is needed to outline when EE provides additional benefits beyond that of other methods such as literature reviews and primary data analysis. Going forward, the lack of reporting among studies with indeterminate EE methods should raise concerns. As EE continues to proliferate, reporting guidelines and recommended methods need to be established to guide future researchers looking to incorporate EEs into their model-based analyses of health outcomes.

Highlights.

• We find extensive use of expert elicitation for the development of model input parameters, but the majority of studies do not provide adequate details of their elicitation methods.

• Lack of reporting hinders greater discussion of the merits and challenges of using expert elicitation for model input parameter development.

• There is a need for the establishment of expert elicitation best practices and reporting guidelines.

Appendix 1: 2020 PRISMA Checklist¹

Section and Topic	Item #	Checklist item	Location where item is reported
TITLE
Title	1	Identify the report as a systematic review.	1
ABSTRACT
Abstract	2	See the PRISMA 2020 for Abstracts checklist.	3-4
INTRODUCTION
Rationale	3	Describe the rationale for the review in the context of existing knowledge.	5-6
Objectives	4	Provide an explicit statement of the objective(s) or question(s) the review addresses.	6
METHODS
Eligibility criteria	5	Specify the inclusion and exclusion criteria for the review and how studies were grouped for the syntheses.	7-8
Information sources	6	Specify all databases, registers, websites, organisations, reference lists and other sources searched or consulted to identify studies. Specify the date when each source was last searched or consulted.	7
Search strategy	7	Present the full search strategies for all databases, registers and websites, including any filters and limits used.	Appendix 2; PubMed Only
Selection process	8	Specify the methods used to decide whether a study met the inclusion criteria of the review, including how many reviewers screened each record and each report retrieved, whether they worked independently, and if applicable, details of automation tools used in the process.	7-8
Data collection process	9	Specify the methods used to collect data from reports, including how many reviewers collected data from each report, whether they worked independently, any processes for obtaining or confirming data from study investigators, and if applicable, details of automation tools used in the process.	8
Data items	10a	List and define all outcomes for which data were sought. Specify whether all results that were compatible with each outcome domain in each study were sought (e.g. for all measures, time points, analyses), and if not, the methods used to decide which results to collect.	NA
	10b	List and define all other variables for which data were sought (e.g. participant and intervention characteristics, funding sources). Describe any assumptions made about any missing or unclear information.	8
Study risk of bias assessment	11	Specify the methods used to assess risk of bias in the included studies, including details of the tool(s) used, how many reviewers assessed each study and whether they worked independently, and if applicable, details of automation tools used in the process.	9
Effect measures	12	Specify for each outcome the effect measure(s) (e.g. risk ratio, mean difference) used in the synthesis or presentation of results.	NA
Synthesis methods	13a	Describe the processes used to decide which studies were eligible for each synthesis (e.g. tabulating the study intervention characteristics and comparing against the planned groups for each synthesis (item #5)).	9
	13b	Describe any methods required to prepare the data for presentation or synthesis, such as handling of missing summary statistics, or data conversions.	NA
	13c	Describe any methods used to tabulate or visually display results of individual studies and syntheses.	9
	13d	Describe any methods used to synthesize results and provide a rationale for the choice(s). If meta-analysis was performed, describe the model(s), method(s) to identify the presence and extent of statistical heterogeneity, and software package(s) used.	NA
	13e	Describe any methods used to explore possible causes of heterogeneity among study results (e.g. subgroup analysis, meta-regression).	NA
	13f	Describe any sensitivity analyses conducted to assess robustness of the synthesized results.	NA
Reporting bias assessment	14	Describe any methods used to assess risk of bias due to missing results in a synthesis (arising from reporting biases).	NA
Certainty assessment	15	Describe any methods used to assess certainty (or confidence) in the body of evidence for an outcome.	NA
RESULTS
Study selection	16a	Describe the results of the search and selection process, from the number of records identified in the search to the number of studies included in the review, ideally using a flow diagram.	9–10; Figure 1
	16b	Cite studies that might appear to meet the inclusion criteria, but which were excluded, and explain why they were excluded.	NA
Study characteristics	17	Cite each included study and present its characteristics.	Tables 1–3
Risk of bias in studies	18	Present assessments of risk of bias for each included study.	Appendix 4
Results of individual studies	19	For all outcomes, present, for each study: (a) summary statistics for each group (where appropriate) and (b) an effect estimate and its precision (e.g. confidence/credible interval), ideally using structured tables or plots.	10-13
Results of syntheses	20a	For each synthesis, briefly summarise the characteristics and risk of bias among contributing studies.	13
	20b	Present results of all statistical syntheses conducted. If meta-analysis was done, present for each the summary estimate and its precision (e.g. confidence/credible interval) and measures of statistical heterogeneity. If comparing groups, describe the direction of the effect.	NA
	20c	Present results of all investigations of possible causes of heterogeneity among study results.	NA
	20d	Present results of all sensitivity analyses conducted to assess the robustness of the synthesized results.	NA
Reporting biases	21	Present assessments of risk of bias due to missing results (arising from reporting biases) for each synthesis assessed.	NA
Certainty of evidence	22	Present assessments of certainty (or confidence) in the body of evidence for each outcome assessed.	NA
DISCUSSION
Discussion	23a	Provide a general interpretation of the results in the context of other evidence.	14
	23b	Discuss any limitations of the evidence included in the review.	14-17
	23c	Discuss any limitations of the review processes used.	16-17
	23d	Discuss implications of the results for practice, policy, and future research.	14-17
OTHER INFORMATION
Registration and protocol	24a	Provide registration information for the review, including register name and registration number, or state that the review was not registered.	6
	24b	Indicate where the review protocol can be accessed, or state that a protocol was not prepared.	6
	24c	Describe and explain any amendments to information provided at registration or in the protocol.	NA
Support	25	Describe sources of financial or non-financial support for the review, and the role of the funders or sponsors in the review.	1
Competing interests	26	Declare any competing interests of review authors.	1
Availability of data, code and other materials	27	Report which of the following are publicly available and where they can be found: template data collection forms; data extracted from included studies; data used for all analyses; analytic code; any other materials used in the review.	NA

NA = Not applicable

Appendix 2: Final PubMed Search Term

The final search term for PubMed included MeSH terms for key modeling concepts (computer simulation, theoretical models, cost-benefit analysis, and biomedical technology assessment) and expert elicitation key words selected in part from previous reviews on expert opinion.^2,3 The modeling MeSH terms encapsulate keywords for modeling methodologies including cost-effectiveness, utility and benefit analysis; computation modeling, microsimulation, Markov models, and decision analytic models among others. The full search term was:

((“computer simulation”[MeSH Terms]) OR (“models, theoretical”[MeSH Terms]) OR (“cost-benefit analysis”[MeSH Terms]) OR (“technology assessment, biomedical”[MeSH Terms]) OR (“simulation model”)) AND ((“expert testimony”[MeSH Terms]) OR (expert elicit*) OR (“expert judgment”) OR (“expert opinion”)) AND ((elicit*) OR (probabilit*) OR (value) OR (distribution))

The PubMed search term was adapted to search Web of Science by removing MeSH terms and expanding the list of modeling and expert testimony related keywords that are encapsulated by the PubMed MeSH terms.

Appendix 3: Quality Assessment of Studies Based on Guideline for Reporting Expert Judgments. (Adapted from Iglesias et. al)⁴

Criterion	Description
Research rationale	The need for using an expert elicitation exercise should be described
Research problem	All uncertain quantities (model input parameters) that will be elicited should be described
Measurement type of uncertain quantities	The rationale for the measure type of each uncertain quantity elicited should be described
Definition of an expert	The nature of the expert population should be described to clearly state what topic of expertise they represent and why
Number of experts	The selection criteria and final number of experts recruited to provide expert judgment should be reported
Preparation	There should be clear reference made to a protocol that describes the design and conduct of the elicitation exercise
Piloting	It should be clearly reported if the elicitation exercise process was piloted and a summary of any modifications made
Data collection	The approach to collect the data should be reported
Administration	The mode of administering the elicitation exercise should be reported
Training	The use of training materials should be reported and made available
The exercise	The number and framing of questions used in the exercise should be reported and made available; alternatively in Delphi consensus building exercises the number of sessions and processes for building consensus should be reported
Data aggregation	The type of aggregation method (mathematical or behavioral) should be reported together with a description of the method or process used to aggregate the data
Measures of performance for data aggregation	The processes followed to estimate measures of performance (calibration/information) for data aggregation need to be fully described
Presentation of results	The individual, and aggregated, point estimate(s) and distribution for each uncertain quantity (quantities) should be presented
Interpretation of results	The interpretation of uncertain quantities elicited should be presented together with a description of how the results will be used in the model-based analysis
Parameter impact	The modeling results explicitly describe how changes in the elicited parameter(s) impact the results and conclusions of the study

Appendix 4: Quality assessment of Expert Elicitations conducted in health services research-based modeling studies.

Author,	Research Rationale	Research Problem	Measurem	Definition of Expert	Number of Expert	Preparatio	Piloting	Data Collection	Administra	Training	The Exercise	Aggregatio	Performan Measures	Present Results	Interpret Results	Parameter Impact	TOTAL
Akpo, 2017 5	1	0	1	1	1	0	0	1	1	0	0	0	0	0	0	0	6
Annemans, 2010 6	1	1	1	1	1	0	0	1	0	0	0	0	0	0	0	0	6
Apelberg, 2018 7	1	1	1	1	1	1	1	1	1	1	1	1	0	1	1	1	15
Banz, 2005 8	1	1	0	1	0	0	0	0	1	0	0	0	0	0	0	0	4
Bojke, 2010 9	1	1	1	1	1	0	0	1	1	1	1	1	1	1	1	1	14
Chiem, 2012 10	1	1	1	1	1	0	0	1	1	0	1	1	0	0	1	0	10
De Graeve, 2005 11	1	1	1	1	1	0	0	0	0	0	0	0	0	0	0	0	5
Dewilde, 2009 12	1	1	1	0	1	0	0	1	1	0	0	1	0	0	0	0	7
Dubinsky, 2002 13	1	1	1	1	0	0	0	1	0	0	0	0	0	1	1	1	8
Duenas-Meza, 2020 14	1	1	0	1	0	0	0	0	0	0	0	0	0	0	1	0	4
Ekaette, 2007 15	1	1	1	0	0	0	0	1	1	1	0	1	1	1	1	0	10
Firoz, 2006 16	1	1	1	1	1	0	1	1	1	0	1	1	0	0	1	1	12
Garay, 2019 17	1	1	1	1	1	0	0	1	0	0	0	1	0	0		1	9
Garthwaite , 2008 18	1	1	1	0	1	0	0	1	1	1	0	1	0	0	0	0	8
Gordon, 2012 19	1	0	0	1	1	0	0	1	1	0	0	1	0	0	0	0	6
Grand, 2019 20	1	1	1	0	0	0	0	1	1	1	0	NA*	0	1	1	0	8
Grigore, 2016 21	1	1	1	1	1	0	1	1	1	1	1	1	1	1	1	1	15
Gyllensten, 2012 22	1	1	1	1	1	0	0	1	1	0	0	1	0	0	1	1	10
Handels, 2017 23	1	1	1	1	1	0	0	1	1	0	1	0	0	1	0	0	9
Hinde, 2015 24	1	1	1	0	1	0	0	0	0	0	0	0	0	0	0	0	4
Hitimana, 2019 25	1	1	1	1	1	0	0	1	1	1	1	1	0	0	1	1	12
Hsu, 2011 26	1	1	1	1	0	0	0	0	0	0	0	0	0	0	0	0	4
Kauf, 2010 27	1	1	1	1	1	0	0	1	1	0	0	1	0	0	0	0	8
Keren, 2002 28	1	1	1	1	1	0	0	1	0	0	0	0	0	0	0	1	7
Kip, 2018 29	1	1	1	1	1	0	0	1	1	0	1	1	0	0	1	0	10
Loeve, 2000 30	1	1	1	1	0	0	0	0	1	0	0	0	0	0	1	7	13
Lubell, 2011 31	1	1	1	1	1	0	1	1	1	0	1	1	0	1	1	1	13
McKenna, 2010 32	1	1	1	1	0	0	1	1	1	1	1	1	0	1	1	1	13
Meeyai, 2015 33	1	1	1	1	1	0	0	1	1	0	0	1	0	0	0	1	9
Petersohn, 2021 34	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	16
Poncet, 2015 35	1	1	1	1	0	0	0	1	1	0	1	1	0	0	0	0	8
Prosser, 2018 36	1	1	1	1	1	0	0	1	1	1	1	1	0		1	1	13
Sankasting, 2020 37	1	1	1	1	1	0	0	1	0	0	0	1	0	1	1	1	10
Schumock, 2007 38	1	1	1	1	1	0	0	1	1	0	1	1	0	0	1	1	11
Soares, 2013 39	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	16
Stevenson, 2009 40	1	1	1	0	0	0	0	1	0	0	0	1	0	0	0	0	5
Hoeven, 2019 41	1	1	1	1	1	0	0	1	1	1	1	1	1	1	1	1	14
Veerman, 2009 42	1	1	1	1	1	0	0	1	0	1	0	1	0	0	0	0	8
Walton, 2010 43	1	1	1	1	1	0	0	1	1	0	0	1	0	0	0	0	8
Wielage, 2013 44	1	1	1	1	0	0	0	1	1	0	1	0	0	0	0	1	8
TOTAL	40	38	37	34	29	3	7	34	29	13	17	27	6	13	21	26	9.4

^*Only one expert was consulted making aggregation not applicable.

Appendix 5: Characteristics of modeling studies using Expert Opinion from probabilistic model inputs

Author, Year	Location	Study Design	Model Type	Disease/Condition	Opinion Parameters	Opinion Solicitation Details	Other Sources Considered for Opinion Parameters
Aballéa, 200745	Brazil, France, Germany, Italy	CEA	Decision Analytic	Influenza Vaccination	Proportion currently eligible for vaccination reimbursement	NR	Data from other countries
Abernethy, 200346	US	Decision Analysis	Clinical Decision and Economic Analysis Model	Cancer	Probability that cancer pain patient would require intervention	Clinical experts	Published intervention rates
Amin, 200447	US	Decision Analysis	Markov Decision Analytic	Liver transplantation	Replantation; probability of spontaneous recovery following primary graft failure	Personal communication with an expert	None
Annemans, 201448	Belgium	CEA	Decision Analytic	Major Depressive Disorder	Difference in suicide risk post remission or relapse between treatment types; time to remission or therapy change; most commonly prescribed medications; validation of model structure and assumptions;	Experts were used to validate assumptions and provide inputs	Published literature
Banz, 20058	Europe (Switzerland)	CEA	Decision Analytic	Heart Failure	Adverse events; treatment utilization	Nine medical experts	Published literature
Banz, 200949	Switzerland	CEA	Dynamic Transition Decision Analytic	Varicella Vaccination	Coverage rate; relative susceptibility; risk of neurological complications; hospitalization from bacterial super infection, neurological complication, otitis media; death due to super infection, otitis media, or other complications; physician contacts in cases of infection; antiviral drug use; length of stay for severe complications;	Expert panel	Published literature
Beck, 198750	US	Decision Analysis and CEA	Decision Analytic	Symptomatic Bifascicular Block	Adverse events; treatment plans	NR	Published literature
Benneyan, 202151	US	Epidemiological Modeling	Differential Equation and Agent-Based Models	COVID-19	Cross exposure to COVID-19 between campus and community; proportion of the exposed who become infected; effects of reopening plans	NR	Published literature; grey literature
Bennison, 201652	UK	CEA	Economic model-Decision Analytic	Vitreomacular Traction and Macular hole	Treatment success	NR	None
Bernard, 201753	Singapore	CEA	Markov Cohort	Atopic Dermatitis	Treatment use; parent time lost	Three Singapore based clinicians who treat children with atopic dermatitis provided their opinion	Previous models, and the published literature
Bhanegaonkar, 201554	Malaysia	CEA	Markov	Atopic Dermatitis	Distribution of cases and probability of flares by severity and age group; clinical management resource use	Malaysian pediatricians with experience treat atopic dermatitis	Published literature
Bieri, 202055	Switzerland	Comparative Effectiveness	Discrete Event Simulation	Prostate Cancer	Time to start of treatment; time to end of treatment by receipt of radiotherapy; progression free interval;	Expert opinion was obtained from nephrologists with longstanding expertise in nephrology and/or fellowship trained uro-oncologists	Published literature
Boakye, 201256	US	Decision Analysis	Decision Analytic	Adjacent Segment Disease	Probability of adjacent segment disease in a patient with a previous asymptomatic degenerative disc	NR	Published literature
Bramley, 200357	New Zealand	CEA	Decision Analytic	Universal antenatal screening for HIV	HIV prevalence in New Zealand; clinical protocols for care of women with HIV	Infectious disease experts provided information.	None
Broekhuizen, 201858	The Netherlands	Multi-Criteria Decision Analysis	Decision Analytic	Lung Cancer Screening	Sensitivity and specificity, radiation burden attributes, and lower and upper bounds	Clinical expertise of two authors	Published literature
Buchanan, 201759	UK	CEA	Markov Model	Chronic Lymphocytic Leukaemia	Probability of undergoing refractory ofatumumab treatment by mutation type	Clinicians estimated the proportion of patients treated with ofatumumab that would be fit enough for a bone marrow transplant which was then converted to a 28-day transition probability.	None
Bullement, 201960	UK	CEA	Partitioned Survival Model	Metastatic Merkel Cell Carcinoma	Improvement in overall survival for avelumab, frequency of GP visits; adverse event disutilities	An advisory board and follow-up one-to-one consultations validated the model	None
Burd, 200261	US	Decision Analysis	Decision Analytic	Protective Anti-reflux Procedures in Neurologically Impaired Children	Postoperative outcomes	NR	Published Literature
Burg, 202162	The Netherlands	CEA	Decision Analytic Model	Endometrial Cancer	Percentage of patients with: treatment side effect symptomatic lymphocele and no lymph node metastasis; recurrence rate; radiotherapy induced toxicity rate; lymphedema rate; local recurrence rate; regional failure rate; mortality rate	Consulted two experts in the field	Published Literature
Burgette, 201863	US	CEA	Decision Tree	Childhood Dental Caries	Various (but not all) probabilities associated with treatment success by sedation method	Six-member panel with pediatric dentistry expertise	Published literature
Campbell, 199864	US	Epidemiological Modeling	Deterministic	Lyme Disease	Lyme disease cases; and people with unrecognized bites	Expert opinion of the authors	Unpublished data for select parameters
Cannon, 201865	Australia	CEA	Markov	Streptococcus Skin Infections	Proportions of outcomes attribute to Group A Streptococcus and disability weights	Infectious disease experts provided information	Published literature and local pathology studies
Castro-Jaramillo, 201266	England and Colombia	CEA	Markov	Pompe Disease	Life expectancy	NR	None
Chang, 202167	Taiwan	Cost Utility Analysis	Partitioned-Survival Model	Metastatic Merkel Cell Carcinoma	Treatment duration; incidence of adverse events	Three Taiwanese medical oncology experts were consulted.	Literature Review
Chiou, 200368	US	Economic Analysis	Markov	Female Contraception	Unspecified probability estimates	Expert opinion was used to derive estimates and verify accuracy of estimates	None
Chirikov, 201969	US	CEA	Markov	Multiple Sclerosis	Probability, disutility and duration of treatment related adverse events	NR	FDA warning labels
Cipriano, 200770	Canada	CEA	Decision Analytic	Newborn Screening	Incremental mortality of metabolic diseases; social and physical development by age	NR	Published literature
Crivellaro, 201971	US	Decision Analysis	Markov	Benign Prostatic Hyperplasia	State transition probabilities	NR	Results from a systematic review were used to adjust expert opinion using the Lagrangian approach
Das, 201372	UK	CEA	Markov	Advanced Breast Cancer	Treatment skipping, duration, costs, and proportion of patients per sequence	NR	Published Literature
de Rooij, 201473	The Netherlands	CEA	Decision Tree and Markov	Prostate Cancer Diagnosis	Probability of treatment with insignificant tumor	NR	None
Delgado, 200774	US	Decision Analysis	Markov Decision Analytic	LVAD Removal	Some probabilities (no specifics listed); health utilities	A range of cardiac transplant and surgical practitioners were interviewed	Literature reviews
Demarteau, 201275	Taiwan	CEA	Markov Model	HPV Vaccination	Seven screening and treatment related probabilities; proportions of nine HPV variants and cervical cancer	Local experts updated data for the Taiwanese setting	Published literature, country-specific databases
Dieleman, 202076	The Netherlands	CEA	Discrete Event Simulation	Cardiac Surgery	Probability of complicated cardiopulmonary bypass weaning by treatment; probability of surgery given complicated weaning by treatment type; probabilities of long-term cardiovascular events; costs	Two clinical experts were consulted	Published Literature
Dijkstra, 201777	The Netherlands	CEA	Decision Tree and Markov	Prostate Cancer Screening	Treatment probabilities	NR	Previous CEA on prostate cancer
Dornan, 201778	US	CEA	Markov Decision Analytic	Rotator Cuff Repair	Distribution of long-term complication and failure rates; and disutilities	Senior authors on the study	None
Druais, 201679	France	CEA	Markov	Schizophrenia	Therapeutic sequence; treatment choice; treatment interruption; adverse events; outpatient visits; minimums and maximums for sensitivity analyses	Advisory panels and questionnaires	Published literature
Duff, 200380	US, UK, Canada	CEA	Decision Analytic	Food Borne Illnesses	Efficacy of cleaning products; causes of food borne infections; costs of sequelae; utility estimates	Expert panel	Published literature and unpublished data
Ekaette, 200781	Canada	Value of Information	Monte Carlo Simulation	Cancer Staging	Comorbidities; staging; primary metastatic site assignment;	Two co-author experts	None
Enaoria, 201682	US	Epidemiological Modeling	Agent-Based Network Model	Measles	Behavioral inputs of the model	Personal communication with an expert	None
Fournier, 201383	Canada	Resource Utilization	Discrete Event Simulation	Neonatal Intensive Care Unit Capacity	NR	NR	NR
Gaidos, 200884	US	Decision Analysis	Decision Tree	Non-alcoholic Fatty Liver Disease	Probability of an insufficient biopsy sample; liver biopsy complications	NR	None
Garside, 200485	UK	CEA	Markov	Heavy Menstrual Bleeding	Wait time for hysterectomy; percentage of women receiving hysterectomy over repeat ablation	NR	Published literature
Gorelik, 202086	US	CEA	Decision Analytic	Sacroiliitis Diagnosis	Recovery following first line treatment; costs of complications; adverse event disutilities	NR	None
Guyat, 201787	UK	CEA	Parametric Survival Model	Cancer	Overall survival	NR	Published literature, SEER data, preliminary RCT data
Gyftopoulos, 201888	US	CEA	Decision Analytic	Adhesive Capsulitis	Probability of symptoms decreasing after treatment success or failure	Two senior physician authors (one orthopedic surgeon, one musculoskeletal radiologist)	None
Gyftopoulos, 201789	US	CEA	Decision Analytic	Symptomatic full-thickness supraspinatus tendon tears	Probabilities of going on to surgery after imaging after a false-positive imaging study, after a false-negative imaging study, and after a truepositive MRI study following a false-negative ultrasound examination.	Two shoulder and elbow fellowship-trained orthopedic surgeons	None
Hensen, 201090	Sweden	CEA	Discrete-event simulation	Schizophrenia	Treatment compliance and location	Three Swedish clinicians were interviewed	Published literature
Imperiale, 199591	US	Cost Analysis	Decision Analytic	Duodenal Ulcer	Probability of recurrence and re-endoscopy	NR	None
Iskedjian, 200892	Canada	CEA	Decision Analytic	Generalized Anxiety Disorder	Medication adherence switching and augmentation rates; resource use for follow-up	Clinical experts and physician questionnaire	None
Jin, 202093	UK	CEA	Decision Analytic	Schizophrenia	Treatment wait time; uptake of monitoring high risk individuals; medication use	Academic researchers, National Health Service practitioners, commissioners of mental health services and service users	None
Johal, 201394	UK	CEA	Markov	Non-metastatic Osteosarcoma	Adverse events; resource utilization; frequency of routine monitoring;	NR	Studies from the European context
John, 201795	India	CEA	Decision Analytic	Glaucoma	Glaucoma severity	NR	None
Karanicolas, 200796	Canada	CEA	Decision Analytic	Esophageal achalasia	Probability of recurrence; major and minor complication rates; adverse event rates	Two gastroenterologists and three thoracic surgeons	Literature review
Keshavarz, 201697	Iran	Cost Utility and Cost Effectiveness Analysis	Markov Microsimulatio n Model	HBeAg-Negative Chronic Hepatitis B	Disease states of patients and transition probabilities; costs; treatment effectiveness	NR	Published literature; patient self-report; local observations
Kruger, 199898	US	CEA	Decision Analytic	Arthritis	Treatment completion, treatment session number and duration	Opinion came from researchers and Arthritis Foundation staff	None
Kruyt, 202099	The Netherlands	CEA	Markov Model	Hearing Implants	Complications per-year; incidence of complications over time; skin overgrowth	Two clinical experts with bone-anchored hearing implant experience were consulted.	None
Kulkarni, 2007100	Canada	Decision Analysis	Decision Analytic	High-Risk T1G3 Bladder Cancer	Estimates of probabilities or utilities	Uro-oncologists	None
Larsen, 1992101	US	CEA	Decision Analytic	Sudden Cardiac Death	Clinical event rates and probabilities	NR	None
Leeftink, 2020102	The Netherlands	Risk Analysis	Monte Carlo Simulation	Post-Surgical Drug Adherence	NR	Clinical and simulation experts provided model inputs	NR
Leong, 2010103	The Netherlands	CEA	Markov	Idiopathic Overactive Bladder	Treatment effects and duration	A survey sent to 13 urologists	None
Little, 2007104	South Africa	Epidemiological Modeling	Deterministic State Transition	Vertical HIV infections	Efficacy of opportunistic infection treatments	NR	None
Locker, 2007105	US	CEA	Markov	Breast Cancer	Duration of risk of potentially fatal adverse events, resource utilization	Opinion from the Arimidex, Tamoxifen Alone or in Combination Trial steering committee	None
Lubinga, 2015106	Uganda	CEA	Decision Analytic	Postpartum Hemorrhage	Proportion of patients with access to emergency care	consultant obstetrician and gynecologist in Uganda	None
Lumie, 2016107	The Netherlands	CEA	Decision Analytic	Arthritis	Test characteristics and costs; utility values; frequency of tests	NR	Published literature; government health authority
Lundqvist, 2020108	Sweden	CEA	Decision Analytic Model	Prostate Cancer	Risks of adverse events; resource use	Expert opinion was elicited via questionnaires devised by the research group. The respondents were identified with help from the Regional Cancer Center South East. Two different questionnaires were devised and the questions were addressed to oncologists and urologists.	Published Literature
Magee, 2020109	UK	CEA	Markov Model	Gastric Antral Vascular Ectasia	Disease pathway; treatment clinical efficacy; probability of adverse events	Elicited via structured telephone interviews of four NHS consultant gastroenterologists.	None
Maklin, 2011110	Finland	CEA	Decision Tree and Markov	Bariatric Surgery	Proportions of surgical technique, patient characteristics; prevalence of diabetes and sleep apnoea; surgical mortality; reoperation rate	Bariatric surgeons	Hospital data; representative health survey data; published literature
Makris, 2003111	Canada	CEA	Clinical Decision Analytic	Dyspepsia	Estimates and ranges for the prevalence h. pylori; median length of time to treatment failure	Expert panel of gastroenterologists	None
Mandavia, 2020112	UK	CEA	Decision Analytic Model	Hearing Loss	Decision tree and state transition probabilities; utility scores; type of treatment used by severity of hearing loss; complication rates	Clinicians, audiologists, health economic modellers. industry representatives and patients were consulted.	Published scientific and grey literature
Mankowski, 2016113	Scotland	CEA	Decision Tree	Peripheral Neuropathic Pain	Efficacy of the final line of therapy	Expert panel included pain physicians (n = 3), pain nurses (n = 4), a senior academic (n = 1); and a general practitioner (n = 1)	None
Mansel, 2007114	UK	CEA	Markov	Breast Cancer	Duration of risk of adverse events; risk of hip fractures	Six practicing UK breast cancer specialists	Clinical trial data
Masucci, 2019115	Canada	CEA	Decision Analytic	Biliary Atresia	Time to first transplant; probability of second transplant	Clinical hepatology experts	None
Mavranezouli, 2020116	UK	CEA	Decision Analytic	Post-traumatic Stress Disorder	Annual risk of relapse; costs	Clinical academic, and health providers with service user and carer representatives with expertise and experience in the field of PTSD	Published literature and national databases
McCrone, 2009117	UK	CEA	Decision Analytic	Psychosis	Discharge to community mental health team	NR	None
Mnatzaganian, 2015118	Australia	CEA	Decision Analytic-Markov	Cobalamin Deficiency	NR	Co-authors and an external expert	Published evidence
Mohiuddin, 2015119	UK	CEA	Decision Analytic	Otitis Media with Effusion	Potential hearing gain after 12 months; utilities; costs	Three clinical experts	Published expert opinion
Moretti, 2018120	Canada	CEA	Decision Analytic	CYP2D6 Ultrarapid metabolizer phenotype	Probability of receiving codeine; probability of hospital admission	One expert was consulted	None
Najib, 2000121	US	CEA	Decision Analytic	Community-Acquired Pneumonia	NR	NR	Published literature
Nazir, 2015122	UK	CEA	Markov	Overactive Bladder	Final line therapy; number of consultations	Six clinical experts	None
Nietert, 2000123	US	Decision Analysis	Decision Analytic-Markov model	Ischemic stroke	State transition probabilities; adverse events; quality of life	Expert panel of clinicians	Published literature
Ondhia, 2019124	Canada	CEA	Three-state survival model	Non-Small Cell Lung Cancer	Duration of adverse events; utilities; subsequent therapies; resource utilization	Canadian clinical experts	None
Ophuis, 2018125	The Netherlands	CEA	Markov	Panic Disorder	Intervention uptake; Intervention costs	NR	Health economic guidelines
Panchmatia, 2016126	Sweden	CEA	Markov	Macular Degeneration	Monitoring frequency in clinical practice; ranges for sensitivity analyses	NR	Published literature for sensitivity analyses
Pitt, 2006127	UK	Cost Utility Analysis	Decision Analytic	Mild and Moderate Atopic Eczema	Transition probabilities	NR	None
Prosser, 2011128	US	CEA	Decision Analytic	H1N1	Incidence of hospitalizations for pneumonia or other respiratory conditions due to pH1N1 influenza per 100,000	NR	Published data
Rebenitsch, 2011129	US	Decision Analysis	Decision Analytic-Markov model	Keratoconus	Medical treatment regimens after surgery	NR	None
Richards, 2002130	US	CEA	Decision Analytic	Diverticulitis	Transition state probabilities, probabilities of clinical events	NR	Published literature
Rogers, 2008131	UK	CEA	Markov	High-grade Gliomas	Rate of health utility decline in progressive state	Three clinical experts	None
Scheckter, 2020132	US	CEA	Decision Tree	Partial Thickness Burns	Treatment effects of allografting; utilities	NR	NR
Scholte, 2019133	Netherlands	CEA	Decision Analytic- state-transition model	Vestibular Schwannoma	Treatment outcomes; costs of post-treatment care	Otolaryngology, radiology, and neurosurgery experts	None
Shamout, 2018134	Canada	Cost Utility Analysis	Decision Analytic-Markov model with Monte Carlo simulation	Post Prostatectomy Stress Urinary Incontinence	Annual Transition Probabilities; health-utilities	NR	Published literature
Shen, 2016135	China	CEA	Decision Analytic	Influenza-like Illness	Sensitivity and specificity of a rapid influenza diagnostic test	NR	Published literature
Shrestha, 2010136	US	Budget Impact Analysis	NR	Pediatric Vaccine Stockpile	Potential impacts of unvaccinated child by vaccine type	NR	Published literature
Simon, 2019137	UK	CEA	Decision Tree and Markov Model	Depression	Treatment effects and discontinuation; relapse rate; remission; costs	Interviews with practitioners (n=14) and online survey of primary care providers (n=50)	National databases
Smits, 2007138	The Netherlands	CEA	Decision Analytic	Major Depressive Disorder	Treatment outcomes by genotype; all sensitivity ranges	Several experts in psychopharmacology	Published literature
Stam, 2008139	Italy	CEA	Decision Analytic	Empiric Antifungal Treatment in Patients with Neutropenic Fever	Probability of ICU stay by treatment branch; length of ICU stay for patients; costs	Four clinical experts led by one of the coauthors	None
Stellato, 2019140	US	Budget Impact Analysis	Decision Analytic	Melanoma	Proportion of Stage III cases successfully resected; proportion of different treatments by year	NR	National databases; market research data
Stockdale, 2017141	UK	CEA	Decision Analytic	Tinnitus	Transition probabilities; costs; clinical pathways	Six researchers, four of whom were co-authors who through discussion formulated consensus on missing parameters	Survey of British Tinnitus Association, and published literature
Strand, 2016142	Ethiopia	CEA	Decision Analytic	Neuropsychiatric Disorders	Disease prevalence	Hospital and health department experts	Global Burden of Disease estimates
Tappenden, 2013143	UK	CEA	Decision Analytic - Bayesian Markov chain Monte Carlo methods	Colorectal Cancer	The probabilities that patients receive particular treatments	NR	Unpublished survey data
Teppakdee, 2002144	Thailand	CBA	Markov	Hepatitis A vaccination	NR	NR	Supplemented with published data when available
Tiwana, 2012145	US	CEA	Decision Analytic - Markov model	Newborn Screening	Event probabilities in screened and unscreened infants; costs related to false positive screen	NR	Published literature
Treur, 2012146	Spain	CEA	Discrete Event Simulation	Schizophrenia	Compliance rates; monitoring costs	NR	None
Udeh, 2008147	US	CEA	Decision Analytic	Adenoviral Conjunctivitis	NR	NR	Primary data sets; published literature
Verklejj, 2020148	Albania	CEA	Microsimulation	Neonatal Hearing	Screening participation rates; probability of treatment type; costs of special education and early family interventions;	Multiple expert meetings including sessions that update data from other countries to the Albanian context.	Data from other countries
Vodicka, 2020149	Philippines	CEA	Markov Model	Japanese Encephalitis	Probability of treatment for sequelae by severity; vaccine coverage; cost of vaccine delivery	In-country expert opinion in addition to key administrative and clinical personnel at the national and regional Departments of Health, facility-level and national laboratories, vaccine storage facilities, and the Western Pacific Regional Office of WHO.	None
Von Bargen, 2015150	US	Cost Utility Analysis	Decision Analytic	Stress Urinary Incontinence	Probability of expectant management success, utility value for pelvic floor muscle therapy	NR	None
Walzer, 2018151	UK	CEA	Decision Analytic-Markov	Venous Leg Ulcers	Transition probabilities for the different treatments	Clinician and wound nurse	None
Watkins, 2016152	Africa	CEA	Decision Analytic-Markov	Rheumatic Fever and Rheumatic Heart Disease	NR	NR	Historical Data
Wilson, 2007153	UK	CEA	NR	Lupus Nephritis	NR	NR	Systematic Reviews
Wirth, 2017154	Germany	CEA	Markov	Multidrug-Resistant Tuberculosis	Treatment algorithm; costs of adverse events	NR	Clinical guidelines and clinical study data
Yanagi, 2017155	Japan	CEA	Markov	Macular Degeneration	Treatment discontinuation; number of monitoring visits and maintenance injections	The authors and colleagues provided expert opinion	Published literature

US = United States, UK = United Kingdom, CEA = Cost-Effectiveness Analysis; NR = Not Reported