How to determine if a model is right for neglected tropical disease decision making

Abstract

Mathematical and computational modeling can transform decision making for neglected tropical diseases (NTDs) if the right model is used for the right question. Modeling can help better understand and address the complex systems involved in making decisions for NTD prevention and control. However, all models, modelers, and modeling are not the same. Thus, decision makers need to better understand if a particular model actually fits their needs. Here are a series of questions that a decision maker can ask when determining whether a model is right for him or her.

Mathematical and computational modeling can transform decision making for neglected tropical diseases (NTDs) if it is the right model for the right question. Modeling has revolutionized decision making in many other fields such as meteorology, manufacturing, transportation, and air traffic control. For example, prior to the advent of simulation models that could generate weather maps, weather forecasting and planning had to rely on much more haphazard observations. Modeling can help better understand and address the complex systems (e.g., biological, behavioral, environmental, cultural, and economic) involved in making decisions for NTD prevention and control, which unaided can be challenging. Fig 1 shows the timeline/life cycle for a product or policy and how modeling can aid decision making at each step. However, all models, modelers, and modeling are not the same. Models range from those primarily oriented towards scientific inquiry to those that assist decision making with many variations in between. Therefore, decision makers need to better understand if a model actually fits their needs. What follows are the questions that a decision maker can ask when determining whether a model is right for him or her.

Fig 1. Examples of how modeling can help decision making at each stage in the product and policy development and implementation timeline.

Question 1: Does the modeler understand me?

In any business, understanding the client is paramount to designing and providing a product or service. Similarly, without understanding the decision maker, there is risk of developing a model that does not fit or capture the needs of the decision maker. These needs can be complex, subtle, and not apparent to a modeler who is unable to place himself or herself in the shoes of the decision maker. Understanding the decision maker requires humility, a desire to listen and learn, and preferably direct experience. Nothing can really replace experience. However, without true experience, the modeler has to put in considerable extra work to truly relate. For example, a common complaint among physicians and policy makers is that a model does not accurately represent the realistic options and constraints that they face. This raises the second question.

Question 2: Does the model reflect my real-world constraints?

While models are simplifications of the real world, to be useful to a decision maker, the model needs to include the constraints that the decision maker may face. This is a key distinction between models that are designed more for pure scientific inquiry and those that aim to assist decision making. Common real-world constraints include economics (e.g., some interventions may be cost prohibitive), logistics (e.g., getting medications to a population requires functional and far-reaching supply chains) and behavior (e.g., people may not comply with completing a course of medications).

Question 3: Does the model incorporate all the considerations?

Models for decision making are by definition, multi- and inter-disciplinary. A decision is rarely specific to a discipline. In other words, a decision is not purely a biology decision, an epidemiology decision, or an economic decision. Instead, a decision typically spans multiple disciplines and considerations. There may be biological, epidemiological, or economic considerations that need to be integrated.

Question 4: Was the model originally designed for this purpose?

Choosing the right model is important. All models are not the same. Instead, a tremendous diversity of models exist, and different models serve different purposes. For example, some modelers and models are more oriented towards specific biological or transmission processes and scientific inquiry. Others help understand and guide clinical processes. Still others focus more on policy making. No single model or modeler can do all of these or serve other similar disparate purposes at once. Similarly, a wide variety of modeling methodologies exist, ranging from decision analytic models, to compartment models, to network models, to agent-based models, and many other types and hybrids along the spectrum of possibilities. No single methodology is appropriate for all situations or necessarily superior or more complex than other methodologies. Repurposing a model that was originally designed for another purpose can be analogous to using a car as a boat.

Question 5: Is the model too simple or too complex?

The principle of parsimony specifies that a model should not be more complex than needed. In other words, a model should only include what’s necessary. A model is not a replica (i.e., something that is supposed to mirror everything in real life), but instead a distillation of the important factors and relationships germane to a question or decision. For instance, detailed representations of pathogen transmission that do not significantly affect cost or effectiveness may not be necessary in an economic model. On the other extreme, models that oversimplify important processes or leave out key factors or elements may produce misleading results (e.g., a model that determines the value of a treatment but does not account for the costs and health effects).

Question 6: Does the model take the appropriate perspective?

The modeler should clearly identify the perspective of the model, which in turn should match the perspective (i.e., viewpoint) of the decision maker. Different decision makers have different perspectives. For example, an employer may be interested in how a vaccine affects company costs and productivity, whereas an individual patient may be more interested in how a vaccine affects his or her own health and finances.[1–3] A third party payer (or insurance company) may focus on the direct medical costs incurred and saved by a vaccine, which represent what the third party must pay. The societal perspective accounts for both direct medical costs and productivity losses. Model results can differ considerably by perspective (e.g., the annual direct medical costs of Chagas disease may be US$474 but the societal costs are much higher at US$4,660[4]).

Question 7: Does the model use the appropriate time frame?

The modeler should also clearly identify the time frame of the model. Time frame matters in decision making. The model should represent the duration of time that is relevant to decision making. A model that only represents a day after the administration of a vaccine would not capture its potential costs and benefits. For example, capturing the value of a vaccine with a 20-year duration of protection would require running a model for at least 20 years.[5]

Question 8: Does the model require too much data?

Decision makers frequently have to make decisions under uncertainty and data sparse situations. A modeler asking for extensive amounts of data that require too much time to analyze can defeat a model’s utility. A common myth is that models are only as good as the data, i.e., garbage in, garbage out. Depending on the type and structure of the model, this is not necessarily true. Some models rely more heavily on data, such as those in which the modeler essentially fits mathematical equations onto specific datasets. However, other models attempt to represent different relevant mechanisms by other means besides extensive datasets. They can make certain assumptions as long as they are transparent and justifiable, draw learnings from other fields and settings, or develop representations based on observations and theories. Extensive sensitivity analyses can help test the robustness of assumptions and various representations. This is especially relevant for NTDs, for which data is often sparse, of poor quality, or absent. In fact, trying to fit models on poor-quality data can actually lead to misleading findings. The relative lack of data should not prevent the building of models. In fact, building models can help determine what data is needed and the value of additional data, which can guide and prioritize data collections.

Question 9: Does the model generate relevant measures and outcomes?

Each type of decision maker tends to rely on particular measures to make decisions. For example, those with financial responsibilities will focus on financial measures. Health care workers will be very interested in key health outcomes, such as deaths, hospitalizations, and incidence of various types of diseases. Model outcomes should lead to actionable steps. For example, the reproductive rate (R0) may be of interest to an epidemiologist, but have little meaning to a policy maker. Additionally, modelers need to be able to generate graphs and visualizations that communicate findings in a concise and easily comprehendible manner. Many scientific methods of presenting data just do not resonate with decision makers.

Question 10: Is the modeler transparent about what the model can and cannot do?

Just like every tool, product, or service, every model has its strengths and limitations. Misuse of models is common and is analogous to using a hammer as a saw. A model that is very good for a given purpose can be very bad for another. Ultimately, decision makers should be wary of any model that is a “black box” to them. Modelers need to be willing and able to effectively communicate their model’s structure and components, strengths and weaknesses, and associated assumptions and limitations. In many ways, sensitivity analyses (exploring the effects of varying key model parameters) are the most important model results, as they show how it performs under different conditions and circumstances. The analogy is experiencing how a car behaves during a test drive.

Table 1 shows examples of questions decision makers may ask and the models that have been used to answer such questions. While modeling can help address many current challenges of NTD decision making (e.g., complex systems impacted by environmental, economic, behavioral, and biologic factors[6]; subtle and indirect impacts that accumulate over time[7]), both modelers and decision makers have to better appreciate all of the above considerations so that they can choose the right model to fit the right decision.

Table 1. Examples of questions decision makers may ask and models that have answered such questions.

Example Questions	Example Models
How significant is an NTD, and how much time, effort, and resources should be dedicated to prevention and control?	Globally, human hookworm infection costs society US$7.5 billion to US$138.9 billion in 2016 and results in 4.1 million DALYs. The cost per hookworm infection varies by geographic region, from US$99 in Southeast Asia to US$447 in the Americas.[8]
What is the impact of an NTD on different locations, sectors, and industries?	Chagas disease cost US$627 million in health care costs and US$7 billion in societal costs. Annually, an infected individual costs US$4,059 in Latin America, US$13,580 in Europe, and US$15,762 in the United States, Canada, and Australia.[4]
What level of control (e.g., eradication, elimination, or long-term control) of the NTD is possible?	MDA for lymphatic filariasis would need to be drastically scaled up (85% coverage rate with all countries treating 100% of at-risk population annually) to achieve eradication. Maintaining current rates would mean eradiation would not be possible until 2050, with the last round of MDA being given then.[9]
Which existing policies, products, and strategies should be used to prevent and control an NTD?	A combination of treatment of infective individuals and insecticide spraying was the most effective and cost-effective (economically dominant) method of control for visceral leishmaniasis. Insecticide spraying would require less effort when used in combination with treatment policies.[10]
What new policies, products, and strategies for NTD prevention and control should be developed and what should be their characteristics?	A human hookworm vaccine would be more cost-effective than currently used MDA except when vaccination was less efficacious (20% efficacy, 5-year duration) and MDA coverage was 75%. The prevalence in children decreased to 14.6% after 20 years with vaccination, compared to 54.1% with MDA.[5]
How should NTD policies, products, or strategies be implemented?	Hookworm vaccination in combination with drug treatment would be cost-effective when vaccinating both school-aged children and women of child-bearing age but may yield a greater economic return among school-aged children, as the coverage may be greater in this population.[11]
How much should be invested in the development of new policies, products, and strategies for NTD prevention and control?	The return-on-investment for a therapeutic Chagas vaccine would reach US$2 million to US$18 million if 3% of the target population is vaccinated and development costs were $100 million or if 11% are vaccinated and development costs were $400 million.[12]
How should a population be screened for a NTD?	Screening Latin American pregnant women and their newborns for Chagas disease in nonendemic settings economically dominated (cost less and provided health benefits) no screening. Even at a low prevalence of Chagas (0.9%) with a low probability of vertical transmission (2.24%), screening was still better than no screening at a cost of €37.5 per test.[13]
What should be the price of a NTD product or service?	To cost less to avert a case than the cost of treatment, a vaccine for cutaneous leishmaniasis that is US$0.5 per dose (2 doses) would need to have an efficacy of 70% and 5-year protection duration with a ≥0.1% infection risk.[14]
Which of the available NTD policies, products, and strategies should be used?	Among four drug treatment regimens currently used for visceral leishmaniasis, amphotericin B deoxycholate was the most effective, averting 87.2% of attributable deaths, while miltefosine was the most cost-effective.[15]

DALY, disability-adjusted life year; MDA, mass drug administration; NTD, neglected tropical disease.

Funding Statement

This work was supported by the Agency for Healthcare Research and Quality (AHRQ) via grant R01HS023317; the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), Office of Behavioral and Social Sciences Research (OBSSR), and the Global Obesity Prevention Center (GOPC) via grant U54HD070725; NICHD via grant U01 HD086861; and USAID via grant AID-OAA-A-15-00064. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.