Early human judgment forecasts of human monkeypox, May 2022

Beginning in May, 2022, clusters of monkeypox virus infection have been observed in multiple countries, including the UK, Spain, Portugal, and the USA.¹ In response, the US Centers for Disease Control and Prevention and WHO have increased surveillance for orthopoxvirus infections and have asked health-care providers to be alert for patients who present with a rash-like illness that is consistent with human monkeypox.1, 2

We present an ongoing human judgment effort in collaboration with Metaculus—a forecasting technology platform—to make crowdsourced ensemble predictions of (1) the estimated number of reported human monkeypox infections globally as of Dec 31, 2022, (2) the estimated number of deaths due to human monkeypox globally as of Dec 31, 2022, (3) the number of monkeypox cases reported across Europe as of July 1, 2022, (4) the number of monkeypox cases reported in the USA as of July 1, 2022, (5) the number of monkeypox cases reported in Canada as of July 1, 2022, (6) the number of countries that will report at least one case of human monkeypox by July 31, 2022, (7) the number of US states that will report at least one case by July 1, 2022, and (8) whether WHO will declare the monkeypox outbreak a Public Health Emergency of International Concern (PHEIC) before Dec 31, 2022.

As demonstrated during previous infectious disease outbreaks, crowdsourced human judgment can successfully estimate many different quantities that have epidemiological meaning or are of importance to public health decision makers.3, 4, 5, 6, 7, 8 During the 2014–15 and 2015–16 influenza seasons, crowdsourced forecasts of influenza-like illness were collected and found to be among the most accurate when compared to computational forecasts.³ Beginning on Feb 18, 2020, just before WHO declared COVID-19 a pandemic, experts in infectious disease modelling were called upon to generate predictions of a diverse set of quantities related to COVID-19, including the number of weekly incident cases at the US national level, the cumulative number of deaths by the end of 2020, and counterfactual values (eg, the reported number of cases at the state level under two different potential policy changes).⁴ Experts also sensibly predicted the safety, timing, and efficacy of a SARS-CoV-2 vaccine 6 months before the first vaccine approval.⁵ Additional studies have found that human judgment and computational modelling might complement one another.6, 7, 8

We collected 686 unique and revised predictions (358 unique and 328 revised) from May 19 to May 24, 2022, on the Metaculus human judgment crowdsourcing platform (methods, limitations, and TRIPOD reporting guidelines are in the appendix pp 15–19).

The probability that the crowdsourced ensemble placed on 1000 to 10 000 estimated global infections was 0·51, with a 0·46 probability on 10 000 or more infections (figure A ). The probability that the crowd placed on 100 to 1000 deaths was 0·59, with a 0·23 probability on 1000 or more deaths (figure B). Predictive quantiles for the estimated number of human monkeypox infections and deaths stabilised after the first few days of forecasting (figure C).

Figure.

Human judgment ensemble forecasts from May 19 to May 24, 2022, of the ongoing human monkeypox outbreak

(A) Ensemble predictive density of the estimated number of human monkeypox infections that will occur globally in the year 2022. (B) Ensemble predictive density of the number of deaths due to human monkeypox in 2022. (C) Median (line), and 25th and 75th percentiles (shaded area) of predictions of A–B from May 19, 2022, when these questions were first posed, to May 24, 2022. (D) Ensemble predictive density of the number of countries that will report at least one case by July 31, 2022. (E) The number of US states that will report at least one case by July 1, 2022. (F) Median (line), and 25th and 75th percentiles (shaded area) predictions of E and F from May 19, 2022, when the questions were first posed, to May 24, 2022. Ensemble predictions of the number of human monkeypox cases that will be reported in the United States (G), Canada (H), and Europe (I) by July 1, 2022. (J) Median (line), and 25th and 75th percentiles (shaded area) predictions of G–I from May 19, 2022, when the questions were first posed, to May 24, 2022. (K) Median prediction of the probability that WHO will declare the monkeypox virus a Public Health Emergency of International Concern from May 19, 2022, when the questions were first posed, to May 24, 2022.

The probability assigned that 30 to 100 countries would report one or more infections by July 31, 2022, was 0·75, with a probability of 0·09 for 100 or more countries (figure D). The probability that the crowd assigned to more than one reported infection in less than ten US states by July 1, 2022, was 0·34, in 10–20 states was 0·33, in 20–30 states was 0·20, and in 30 or more states was 0·13 (figure E). Quantiles for the number of countries predicted to report an infection increased during the forecasting period, and the median prediction of the number of US states to report an infection decreased from 35 to 14 (figure F).

The crowdsourced probabilities were 0·56 for 10–100 cases, 0·37 for 100–1000 cases, and 0·03 for 1000 or more cases reported in the USA by July 1, 2022 (figure G). By the same date, the probability that the crowd assigned to less than 100 reported cases in Canada was 0·51 and to between 100 and 1000 was 0·48. However, the probability was less than 0·01 for 1000 or more cases (figure H). In Europe, the probability that the crowd placed on 500 to 1000 cases by July 1, 2022, was 0·32, and the probability placed on 2000 or more cases was 0·23 (figure I). After the first day of forecasting, the 25th and 75th percentiles for the cumulative reported cases for all three locations included a large range of values, which might represent forecasters' uncertainty in how the present outbreak will evolve (figure J).

The ensemble median prediction of the probability that WHO will declare the monkeypox virus a PHEIC before Dec 31, 2022, was approximately 25%. When this question was first posted, the median prediction was above 80% and decreased within 4 days (figure K).

Several limitations should be considered when aggregating human judgment predictions. Forecasts might be biased by (1) those in the crowd who might submit forecasts that are poorly calibrated; (2) anchoring effects that might act on forecasters when they view current aggregate forecasts; (3) correlations—if present—between those experts who are more likely to respond to a survey and their perceptions of the intensity of an outbreak; and (4) potential correlations between the probability that a forecaster submits a forecast and their confidence in their ability to generate an accurate prediction. Humans, like computational models, might also be biased by current surveillance data that is available during an outbreak. Additional work might focus on measuring the influence of an observable aggregate forecast on a forecast submitted by an individual and stratifying by subject matter experts and non-experts.

These early results suggest that a human judgment forecasting platform can quickly generate probabilistic predictions for targets of public health importance. Predictions from a human judgment forecasting platform might be especially important when data are sparse, limiting the accuracy of statistical models, or when historical data has been collected in different locations than the present outbreak, limiting the accuracy of mechanistic models. Further, human judgment forecasts can augment epidemiological models with forecasts of policy decisions (eg, WHO declaration), which might influence epidemic trajectories. Even in dynamic situations, human judgment-based systems might aggregate information across diverse areas of expertise, disparate scientific documents (such as outbreak reports and preprints), and word-of-mouth.⁹ Quantiles of aggregated forecasts stabilised in a short period of time, suggesting a consensus which might be driven by a combination of observed data and human intuition. Human judgment forecasts therefore provide a crucial health tool to fill this gap in assessing the potential for widespread community transmission of the monkeypox virus.

MSM has been awarded funds from the National Science Foundation and Boston Children's Hospital. TB is an advisor to Metaculus and holds stock options with Metaculus. SV has been awarded funds from Centers for Disease Control and Prevention, Council for State and Territorial Epidemiologists, Virginia Department of Health, National Science Foundation, Defense Threat Reduction Agency, and the University of Pittsburgh. GD holds leadership and board roles at Metaculus, which is the public benefit corporation that develops the forecasting platform used to elicit forecasts in this Comment. All other authors declare no competing interests. We would like to thank all individuals who dedicated their time and energy to submitting probabilistic forecasts of the transmission and burden of the monkeypox virus. The forecasting data used to generate figures in this Comment are available on GitHub. Real time forecasts are available on the Metaculus website. Because no personal information about forecasters was collected for this work, we determined, in consultation with an international review board analyst, that this study should not be considered Human Subjects Research.