Embracing Uncertainty: The Value of Partial Identification in Public Health and Clinical Research

Abstract

Introduction:

This paper describes the methodology of partial identification methods and its applicability to empirical research in preventive medicine and public health.

Methods:

The authors summarize findings from the methodological literature on partial identification.

Results:

The applicability of partial identification methods is demonstrated via three empirical examples drawn from published literature.

Conclusions:

Partial identification methods are likely to be of considerable interest to clinicians and others engaged in preventive medicine and public health research.

1. Introduction

“Turning Discovery into Health” summarizes succinctly the mission of the U.S. National Institutes of Health, an organization that devotes $40 billion dollars annually towards funding research. For most researchers, “discovery” entails obtaining exact knowledge of the magnitude of a phenomenon of interest, such as the prevalence of a given disease, the effect of a treatment or exposure, or how the results of a clinical trial apply to a specific set of patients. Attempting to do so, researchers rely on statistical inference, which aims to characterize the knowledge of a population that can be gained from a particular sample, such as the Framingham Heart Study or the Nurses’ Health Study, using observational or experimental data. To inform clinical care or population health policy, estimates are then applied to broader populations.

However, studies seeking to obtain exact estimates have to deal with a number of challenges. These include unmeasured confounders of causal relationships, availability of only surrogate health measurements, non-random patient attrition, absence of covariate measures for extrapolating results from one population to another, and many others. These challenges are referred to in the statistical literature as identification challenges. They often may be formalized as problems of missing data.

Missing data challenges pose obvious threats to complete discovery. Unfortunately, these problems persist in spite of researchers’ best efforts to address them. Unlike problems of statistical inference, identification challenges cannot be resolved simply by amassing so-called “big data,” although some may be mitigated by collecting better data. Pursuing complete discovery—that is, obtaining exact estimates of a given quantity of interest—when data are missing inevitably requires making strong, often unjustifiable assumptions that can lead to misleading estimates of the magnitudes of interest.

To best guide patients, clinicians, and policy makers, we argue that any medical research agenda ought to start by considering identification challenges. Given the pervasiveness of missing data problems and the difficulty of addressing them fully without making questionable assumptions¹, we contend that researchers should consider alternatives to attempting to achieve complete discovery. In many situations, partial discovery, also known as partial identification, may yield useful clinical and policy insights that do not rest on untenable assumptions.

Partial identification is not a new concept: early contributions were made close to a century ago. A large modern literature has developed in the past thirty years, with many applications in social science research. Manski^2,3 provides technical and non-technical reviews of the literature. However, the basic ideas and specific findings have only rarely been exploited in medical research. Contributions to public health and clinical research are growing rapidly^4–12.

Our goal in this brief essay is to introduce to clinicians working in public health and preventive medicine the fundamental ideas of partial identification, hopefully thereby convincing many of the value of thinking about data through such a lens.

2. Partial Identification as a Research Strategy

Partial identification seeks to determine the range of feasible values for a phenomenon—rather than an exact value—making transparent and realistic assumptions about the nature of missing data. In many cases, the best one can credibly hope to learn is that the answer to a specific clinical or policy question lies within some bounded range. An important aspect of the research strategy is to begin with weak—and therefore more credible—assumptions and then examine how the bounded range changes as stronger—and perhaps less defensible—assumptions are made. In this critical way, the partial identification approach differs from the dominant approach used by medical researchers to date, which reports a single estimate of the quantity by relying on assumptions that may be controversial.

Drawing back from the unrealistic hope for complete discovery, partial identification analysis can yield informative bounds. For example, a patient may want to know whether a particular drug could cause harm or a particular dietary regimen is effective. A health policymaker may want to know whether a specific intervention can be applied to a new population. Bounded answers to these types of questions can be useful in both informing the direction of impact and determining best and worst-case scenarios. Even situations where bounds are wide are still informative, as they indicate the need for better (though not necessarily “big”) data. Table 1 provides several examples of different forms of missing data related to the COVID-19 pandemic, in which policymakers and clinicians are making decisions under tremendous uncertainty. Partial identification methods may yield useful insights to guide these decisions.

Table 1:

Examples of Missingness and Corresponding Identification Problems in the Absence of Additional Assumptions

Magnitude of Interest	Nature of Possible Missingness and Identification
Disease prevalence rate	Problem(s): Nonrandom sample of population. Imprecise testing. Example: Many infected with COVID-19 are asymptomatic or do not seek medical attention; many symptomatic are not tested; testing may produce false positives and/or negatives. Implication: Prevalence can be only partially identified.
Case mortality rate	Problem(s): Nonrandom sample of population cases. Missing data on true cause. Example: Many infected with COVID-19 are asymptomatic or do not seek medical attention; may not distinguish between deaths caused by COVID-19, pneumonia, or other disease Implication: Mortality rate can be only partially identified.
Difference between subgroup mean outcomes	Problem(s): Nonrandom sample of the population. Example: COVID-19 case mortality rate in males versus females. Many infected with COVID-19 are asymptomatic or do not seek medical attention and this may vary by gender; may not distinguish between deaths caused by COVID-19, pneumonia, or other disease and this may vary by gender. Implication: Difference can at best be partially identified.
Average treatment effect	Problem(s): Outcome in counterfactual state is not observable. Nonrandom sample of the population in the treatment and control groups due to self-selection into groups (without randomization or with randomization and noncompliance) or loss to follow-up. Health outcome unmeasured, but surrogate outcome is available. Example: Hydroxychloroquine and azithromycin treatment for COVID-19 initially assessed based on nonrandom assignment to treatment and control groups, using a binary outcome measure of disease prevalence rather than a continuous measure of viral load. Implication: Average treatment effect is identified under “perfect” randomization/unconfoundedness, otherwise at best partially identified.
Features of treatment-effect distribution, e.g. median(outcome(treated)-outcome(control))	Problem(s): Outcome in counterfactual state is not observable. Nonrandom sample of the population in the treatment and control groups due to self-selection into groups (without randomization or with randomization and noncompliance) or loss to follow-up. Health outcome unmeasured, but surrogate outcome is available. Example: Hydroxychloroquine and azithromycin treatment for COVID-19 initially assessed based on nonrandom assignment to treatment and control groups, using a binary outcome measure of disease prevalence rather than a continuous measure of viral load. Implication: Features of treatment-effect distribution can at best be only partially identified even under “perfect” randomization/unconfoundedness.

3. Applications of Partial Identification Methods to Public Health and Preventive Medicine

To further understand partial identification in real-world clinical and population health settings, we offer three examples that are typical of those faced by professionals working in public health and preventive medicine settings.” In the first, the fraction of sample cases with missing data is large, so the observed data are only weakly informative. This case illustrates how dangerous it can be to assume without foundation that data are missing at random. In the second, the fraction of cases with missing data is relatively small, so highly informative inferences are possible even without making assumptions on the distribution of the missing data. The third example uses partial identification methods to obtain credible bounds on SARS-CoV-2 cumulative infection rates.

Example 1: Estimating Child Vaccination Rates

The proportion of children that are vaccinated against an infectious disease has important implications for population health. The quantity of interest is the probability that a child is vaccinated. Suppose that data are missing for a fraction of children; perhaps they cannot be located by data collectors or perhaps parents refuse to provide information. If vaccination information is missing for some children, the probability of vaccination cannot be completely discovered even with “big” data.

However, vaccination rates are partially identified so long as some data are available (See Supplementary Appendix, Part A, for technical details.) To illustrate numerically, consider CDC data on vaccination rates in the U.S. through the National Immunization Surveys (NIS). In 2017, the NIS overall response rate was 26.1%¹³. The observed vaccination rate for Measles, Mumps, and Rubella (MMR) was 92.0%, which the CDC reports as the population immunization rate in Health, United States, 2018¹⁴. However, considering this estimate to be the population vaccination rate requires the strong and untenable assumption that the vaccination rate in the 73.9% of the sample who were not interviewed is the same as in the sample that was observed.

Conservatively assuming that none of those who were not interviewed were vaccinated, the partial identification approach yields a lower bound estimate of vaccination probability of 24.0%. This estimate cautions public decision makers that the reported CDC vaccination rate of 92.0% may be wildly optimistic. Nevertheless, the partial identification bounds do provide some information, showing that the actual vaccination rate cannot be lower than 24.0% and not higher than 97.9%. Stronger assumptions will result in narrower bounds, although a better approach to estimating vaccination probabilities might be to invest in collecting better data (See Supplementary Appendix, Part B).

Example 2: Missing Outcome and Risk-Factor Data in a Trial of Treatments for Hypertension

Horowitz and Manski⁵ analyzed how to estimate treatment responses in clinical trials when key data on trial outcomes or patient attributes are missing. Focusing on cases in which the outcomes is meeting a binary clinical endpoint, they derived bounds on the probability of achieving the endpoint without imposing any assumptions about why data may be missing. This approach contrasts sharply with the conventional practice in medical research, which often assumes that missing data in clinical trials are missing at random.

Horowitz and Manski applied their partial identification approach to data from the U.S. Department of Veteran Affairs (DVA) antihypertensives trial conducted by Materson et al. (1993). Male veteran patients at 15 DVA hospitals were randomly assigned to receive one of 6 antihypertensive drugs [(1) hydrochlorothiazide, (2) atenolol, (3) captopril, (4) clonidine, (5) diltiazem, or (6) prazosin] or (7) placebo. The trial had two phases. In the first, the dosage that brought diastolic blood pressure (DBP) below 90 mm Hg was determined. In the second, it was determined whether DBP could be kept below 95 mm Hg for a period. Treatment was defined to be successful if DBP < 90 mm Hg on two consecutive measurement occasions in the first phase and DBP ≤ 95 mm Hg in the second. Treatment was deemed unsuccessful otherwise.

Horowitz and Manski obtained the trial data and used them to demonstrate how treatment effects varied with serum renin. Renin response was measured at baseline, but data were missing for some subjects in the trial. Horowitz and Manski also removed the intention-to-treat interpretation of attrition as lack of success and instead viewed subjects who leave the trial as having missing outcome data. The pattern of missing attribute and outcome data is shown in Table 2.

Table 2:

Missing Data in the DVA Hypertension Trial

Treatment	Randomized	Observed Successes	None Missing	Missing Only Outcomes	Missing Only Attributes	Missing Both
1	188	100	173	4	11	0
2	178	106	158	11	9	0
3	188	96	169	6	13	0
4	178	110	159	5	13	1
5	185	130	164	6	14	1
6	188	97	164	12	10	2
7	187	57	178	3	6	0

Horowitz and Manski used partial identification analysis to estimate bounds on the success probabilities for the seven treatments without imposing assumptions on the distribution of missing baseline renin and missing outcome data. Table 3 shows the estimates of the implied bounds on average treatment effects (relative to placebo).

Table 3:

Bounds on Success Probabilities Conditional on Renin Response

	Treatment
Renin Response	1	2	3	4	5	6	7
Low	[0.54, 0.61]	[0.52, 0.62]	[0.43, 0.53]	[0.58, 0.66]	[0.66, 0.76]	[0.54, 0.65]	[0.29, 0.32]
Medium	[0.47, 0.62]	[0.60, 0.74]	[0.53, 0.68]	[0.50, 0.69]	[0.68, 0.85]	[0.41, 0.65]	[0.27, 0.32]
High	[0.28, 0.50]	[0.64, 0.86]	[0.56, 0.75]	[0.63, 0.84]	[0.55, 0.78]	[0.34, 0.59]	[0.28, 0.40]

Though the findings are bounds rather than precise success probabilities, many of the estimated bounds are sufficiently narrow to enable a clinician to conclude that certain treatments are surely inferior for some patients than others. For example, for patients with low renin response, treatments 1, 2, 3, 4, 6, and 7 are all dominated by treatment 5, which has the greatest lower bound (0.66). For patients with medium renin response, treatments 1, 3, 6, and 7 are dominated by treatment 5, which again has the greatest lowest bound (0.68). For patients with high renin response, treatments 1, 6, and 7 are dominated by treatment 2, which has the greatest lowest bound (0.64). Thus, without imposing any assumptions on the distribution of missing data, a clinician can reject treatments 1, 6, and 7 for all patients, reject treatment 3 for patients with medium renin response, and determine that treatment 5 is optimal for patients with low renin response.

Example 3: Bounding SARS-CoV-2 Infection Rates

Manski and Molinari⁹ addressed the well-known problem that accurate characterization of the time path of the coronavirus pandemic has been hampered by a serious problem of missing data. Confirmed cases have been measured by rates of positive findings among persons who have been tested for infection. Infection data are missing for persons who have not been tested.

The persons who have been tested differ considerably from those who have not been tested. Criteria used to determine eligibility for testing have typically required demonstration of symptoms associated with presence of infection or close contact with infected persons. This gives reason to believe that some fraction of untested persons are asymptomatic or pre-symptomatic carriers of the COVID-19 disease. Hence, the actual cumulative rate of infection is likely higher than the reported rate.

A second problem of data quality is that measurement of confirmed cases is imperfect because the prevailing tests are not fully accurate. The accuracy of these tests is asymmetric, with positive predictive values close to one but negative predictive values possibly much less than one. Given this asymmetry, the actual rate of infection is again likely higher than the reported rate.

Combining the problems of missing data and imperfect test accuracy yields the conclusion that reported cumulative rates of infections have been lower than actual rates. Thus, point estimates of infection rates put forward by researchers and public health agencies necessarily rely on strong assumptions. The specific assumptions vary, and so do the findings. No assumption or estimate has been thought sufficiently credible as to achieve consensus.

Manski and Molinari argue that it is more informative to determine the range of infection rates implied by weaker, but credible assumptions. In particular, they assume that the infection rate among untested persons is lower than the rate among tested persons. They also assume a bound on the accuracy of nasal swab tests. Applying these and other assumptions to public health statistics on test positivity rates, they derive bounds on population infection rates.

To illustrate, they analyze data from Illinois, New York, and Italy in March and April 2020. They obtain bounds that are wide but yield some information. For example, they find that the cumulative infection rates on April 24 were in the intervals [0.4%, 52.5%], [1.7%, 61.8%], and [0.6%, 47.1%] respectively. They show that these bounds can be tightened by adding further assumptions, for example by using knowledge of the rate of asymptomatic cases of COVID-19.

4. Moving Forward

Medical science finds itself at a remarkable juncture. Important highly publicized debates regarding the reliance on p-value thresholds, the inability to replicate empirical findings, and the limitations of randomized trials have called seriously into question many well-entrenched research practices. Amidst all this, we believe the time is ripe to consider a broad role for partial identification methods in public health and clinical research.

One of us has written about the importance of avoiding the lure of “incredible certitude” in empirical research^6,8. Instead, being appropriately confident about magnitudes that are partially knowable provides medical decision makers with a firmer foundation for their actions. For evidence-based research to help patients, findings must be trustworthy. Trust arises from understanding the data at hand, what they reveal and what they do not reveal. Research that creates false hope or discouragement because of the incredible certitude on which it is based should be discarded in favor of research based on a recognition and appreciation of the limited signals often sent by real data. Patients and those who care for them undoubtedly yearn for unambiguous statements about magnitudes that matter to them. Assisting them in appreciating that partial knowledge is a feature and not a bug is of paramount importance.

Partial identification approaches can yield trustworthy and informative estimates, even if they present a range of possibilities. These approaches can be applied to a variety of settings. While applications in healthcare are still few, whole domains of research could benefit from their application. These approaches can be useful in many clinical and population health settings, including the COVID-19 global pandemic: estimating the prevalence and lethality of infection; applying treatment results from non-randomized trials to other patients; making decisions based on incomplete clinical trials of potential vaccines and treatments; and many more.

While researchers, clinicians, and the patients they seek to help desire unambiguous statements about magnitudes, the partial identification approach is more grounded in what everyone knows about the real world where decisions are made based on the best information available. If our considerable investment in “turning discovery into health” is to pay off, then understanding when research efforts do and do not yield complete discovery is essential. When research falls short of yielding complete discovery, the fact that it may partially identify magnitudes of interest should be celebrated, not bemoaned.