On Representations and Quantifications of Uncertainty

Many of the phenomena and systems that you, reader, wish to model involve uncertainties. Your models are low-resolution representations of the real things—the empirical systems they abstract. Whether explicitly or intuitively, you abstract salient aspects of the empirical world as mathematical objects, relations, or operations, establishing a homeomorphism between the empirical system and the model.¹ In health, uncertainties in empirical systems are often of salience, and thus, they ought to be represented and quantified in the models you build. This is a long way of saying that you have a need for uncertainty models.

Most applications in health represent and quantify uncertainties exclusively with probability measure models. Probability measure models are satisfactory uncertainty models for well-understood chance mechanisms, such as the throw of a coin, or for phenomena that are conceptually equivalent to stylized chance mechanisms, such as when modeling the number of observationally equivalent patients who experience an outcome in an experiment.^2,3

However, we often need to represent and quantify uncertainties in situations in which we have more fundamental ignorance about the empirical world or when we do not believe that our uncertainties are conceptually equivalent to some stylized chance mechanism. Examples of such deep uncertainties or ambiguities abound. Early in the COVID-19 pandemic, epidemiologists did not know whether SARS-COV-2 is transmitted by aerosols, and we still do not know whether the virus emerged naturally or escaped from a lab. Structural uncertainties about the empirical world, such as which of several carcinogenesis paths are most correct, are almost always ambiguities.

Our thesis is that ambiguities are best described with uncertainty models that are more general than the probability measure model.⁴ We will briefly discuss intuition about why probability measure models are overly restrictive models for ambiguities, examples of nonprobability uncertainty models that are not as restrictive, and the catch with using the latter: propagating uncertainty and optimizing decisions in a probabilistic setting is a well-understood technology. However, using nonprobability uncertainty models changes the calculus of uncertainty propagation and complicates notions of optimality and the identification of optimal acts. We will conclude with what you should do.

Why Probability Measure Uncertainty Models Are Not Always Satisfactory

In applied modeling in health, almost all uncertainties are described with probability measures, to the extent that the phrase “probability measure uncertainty models” rings almost redundant. However, there are several reasons why a single probability measure may be less attractive as a model of uncertainty.

First, no single probability distribution captures well situations of ignorance or of very minimal information. Consider ignorance about the probability Pr[A] that event A obtains. The correct description is that

$$0\le \text{Pr}[A]\le 1,\text{or}\text{Pr}[A]\in 𝒰=[0,1].$$ (1)

Assigning any single probability measure to Pr[A] provides more information than (1), which is the only information we really have. For example, a uniform probability distribution between 0 and 1 satisfies (1) but also imposes the extraneous constraint that all values are equally likely. The oft-rehearsed justification for the uniform distribution in cases of ignorance invokes Laplace’s Principle of Insufficient Reason, which professes that we should take all events to be equally likely, unless there is a reason to the contrary. However, the Principle is an esthetic criterion that imposes a restrictive constraint to (1). More generally, if all we know is that $\text{Pr}[A]\in 𝒰=[L,U]\subseteq [0,1]$, no single probability distribution works as an uncertainty model. A reasonable uncertainty model may simply be the set $𝒰$.

A second reason is that a probability measure uncertainty model implies statements about the probability mass that is contained in arbitrary intervals. Under ambiguity, it may be difficult to justify such statements. For example, if we assume a uniform distribution in [L, U], we may infer that the probability mass between L and (L + U)/2 is 1/2. This inference is not implied by (1), which is compatible with scenarios in which the probability mass between these bounds is anything between 0 and 1, inclusive.

Other Uncertainty Models

When it comes to ambiguities, various uncertainty models can be considered that are more general than single probability measures.

One example is an uncertainty model that comprises a set Γ of several probability measures. For a random quantity X with such a model, we would write that

$$X~G,\text{with}G\in \Gamma ,$$ (2)

to state that X can be distributed as per any one of the probability measures in Γ, although we would not know which one. This is the idea behind robust Bayesian analysis⁵ and probability boxes.⁶ It is different from model averaging in that there is no model over the elements of Γ. Defining Γ is a “you” problem. It can be finite (e.g., 3 nonparametric distributions) or infinitely large (e.g., any distribution with mean and variance fixed to specific values).

Another example of a more general uncertainty model was already introduced in (1). It is the same as the model in (2) if Γ includes all distributions whose entire mass is in a subset of $𝒰$.

Several other uncertainty models exist, but to provide intuition about them, we should discuss for each of them how they deduce whether, and in which way, new evidence “narrows” the extant uncertainty and how they propagate uncertainty from inputs to outputs (what calculus they use).

The catch with using uncertainty models that are different from a single probability uncertainty model is that propagating uncertainty and identifying optimal acts gets complicated. Discussing these complications is out of scope, but we can hazard a few observations.

Propagation of More-General Uncertainty Models

Uncertainty in a model’s inputs induces uncertainty in its outputs. When the inputs have probability measure uncertainty models, the outputs also have probability measure uncertainty models. For applied health modeling, propagating probability measures through a mathematical model is turn-key technology, achieved, usually, with some Monte Carlo approach. Typically, this is just forward Monte Carlo (“probabilistic sensitivity analysis” in this field’s somewhat misleading terminology).

When the inputs have more-general models, say uncertainty sets as per (1), they induce analogous more-general uncertainty models in the outputs. For example, if the inputs have uncertainty sets as uncertainty models, so will the outputs. To propagate uncertainty in this setting is to characterize the uncertainty sets in the outputs, which generally involves specifying and solving mathematical programs.⁶ Thus, at least for the health modeler, the propagation of more-general uncertainty models is not exactly a turn-key technology.

Optimizing Decisions

If your model is a decision model, your goal is to identify acts that are optimal in some well-defined sense. To fix ideas, stipulate that you want to know the acts that improve a decision-relevant quantity, which we will take to be a functional of a 1-dimensional utility outcome. In most applications, this functional appears to be the mean, but we generalize to any scalar statistic. The idea is then to find the acts that optimize said statistic according to the model.

When the utility has a probability distribution, that is, is a point mass distribution for deterministic models or a probability distribution for nondeterministic models, the decision-relevant quantity with each act is a scalar, and you can always order scalars from small to large. In technical parlance, you can assess a weak order over your acts.

When the uncertainty model that is induced for the utility is not a single probability distribution, but, for example, an uncertainty interval, you cannot point-identify the decision-relevant quantity. For example, you would not know the mean utility with each strategy—you would know only the interval in which the mean utility lives. In this case, you cannot always order the acts. For any two acts, sometimes you may be able to identify one as clearly better (if the worst possible utility with one is better than the best possible utility with the other). Still, sometimes you may not be able to do so. You cannot order uncertainty sets the same way you order scalars. The technical term is that you can only assess a partial order over the acts.

There are several notions of optimality over a partially ordered set of acts, and each captures different attitudes toward risk and ambiguity. Some involve excluding an act if it is worse than one other act over the whole uncertainty set. Others optimize for the worst possible case, the best possible case, or some hybrid of the two extremes, and many others exist.

How Should You Model Uncertainty and Parting Comments

Modeling uncertainty in your problem is your responsibility and prerogative. For some applications and under ambiguity, you may be unwilling or unable to assess probability measure uncertainty models. In such cases, you have other options that more “honestly” represent and quantify your uncertainty but complicate its propagation and the choice of optimal acts.

If you decide to use only probability measure models, your typical approach might be to adopt a “working” probability measure and explore the implications of your analytic choices in sensitivity analyses. You would still run into difficulties if we were to ask that you summarize your uncertainty propagation results and the “optimal” acts over all your sensitivity analyses. (And this reasonable request is rarely raised.) You may also run into computational intractability once you have more than a handful of model inputs in your sensitivity analyses. A key notion is that sensitivity analysis is a related but distinct idea from what we discussed here.

Obviously, for many applied problems, using uncertainty models other than single probability distributions will require techniques that are less familiar to the health modeler. Then, it would not be a practical solution, and other analyses may fit the bill. It may also be that the uncertainty is so extreme that the goal is not to propagate it and describe its implications but to examine the behavior of the model and intuit about the empirical world it abstracts.

How one models uncertainties and ambiguities is, in the end, an esthetic, philosophical choice. You have seen this vividly in probability-land: many statisticians reject the concept of subjective probability because they do not believe that the uncertainty around humans’ beliefs is conceptually equivalent to a stylized chance mechanism, while others are famously comfortable with this abstraction.⁷

We would be excited to see in the health literature many applications that use more-general uncertainty models to address worthy problems. And if you side with some of our interlocutors who do not see the point in all this, all is well. Different strokes and all that.