Evaluating probabilistic dengue risk forecasts from a prototype early warning system for Brazil

Rachel Lowe; Caio AS Coelho; Christovam Barcellos; Marilia Sá Carvalho; Rafael De Castro Catão; Giovanini E Coelho; Walter Massa Ramalho; Trevor C Bailey; David B Stephenson; Xavier Rodó

doi:10.7554/eLife.11285

. 2016 Feb 24;5:e11285. doi: 10.7554/eLife.11285

Evaluating probabilistic dengue risk forecasts from a prototype early warning system for Brazil

Rachel Lowe ^1,^*, Caio AS Coelho ², Christovam Barcellos ³, Marilia Sá Carvalho ³, Rafael De Castro Catão ^1,⁴, Giovanini E Coelho ⁵, Walter Massa Ramalho ⁶, Trevor C Bailey ⁷, David B Stephenson ⁷, Xavier Rodó ^1,⁸

Editor: Simon I Hay⁹

PMCID: PMC4775211 PMID: 26910315

Abstract

Recently, a prototype dengue early warning system was developed to produce probabilistic forecasts of dengue risk three months ahead of the 2014 World Cup in Brazil. Here, we evaluate the categorical dengue forecasts across all microregions in Brazil, using dengue cases reported in June 2014 to validate the model. We also compare the forecast model framework to a null model, based on seasonal averages of previously observed dengue incidence. When considering the ability of the two models to predict high dengue risk across Brazil, the forecast model produced more hits and fewer missed events than the null model, with a hit rate of 57% for the forecast model compared to 33% for the null model. This early warning model framework may be useful to public health services, not only ahead of mass gatherings, but also before the peak dengue season each year, to control potentially explosive dengue epidemics.

DOI: http://dx.doi.org/10.7554/eLife.11285.001

Research Organism: None

eLife digest

Dengue is a viral infection spread by mosquitoes and is widespread in tropical and sub-tropical regions. Dengue epidemics in Brazil often occur without warning, and can overwhelm the public health services. Forecasts of seasonal climates combined with early data from a dengue surveillance system could help public health services anticipate dengue outbreaks several months in advance. However, this information has not been previously exploited to predict dengue epidemics in a practical real-life framework.

Recently, a group of researchers developed a prototype of a dengue early warning system based on 13 years worth of data, and used it to predict the risk of dengue three months ahead of the 2014 FIFA World Cup in Brazil. Now Lowe et al. – including most of the researchers involved in the earlier work – have evaluated the prototype against the actual reported cases of dengue during the event. Brazil is divided into over 550 'microregions', and the forecasts correctly predicted high risk of dengue for 57% of the microregions reporting high levels of dengue during the games. Forecasts based on seasonal dengue averages would have only detected high risk in 33% of these microregions. The forecasts also correctly predicted the dengue risk level in seven out of the twelve cities where the World Cup games were hosted. However, the prototype failed to predict the high risk in both São Paulo and Brasília. Lowe et al. speculate that this may have been due to changes in how water was stored in these cities (standing water is a breeding site for mosquitoes) and the circulation of a new strain of the dengue virus.

The implementation of seasonal climate forecasts and early reports of dengue cases into an early warning system is now a priority for public health authorities. This action is likely to help them to prepare for and minimize epidemics of dengue and other diseases that are spread by mosquitoes, such as chikungunya and Zika virus.

DOI: http://dx.doi.org/10.7554/eLife.11285.002

Introduction

Dengue is an arboviral infection of major international public health concern (Guzman and Harris, 2015). Dengue is endemic in more than 100 countries in the tropics and sub-tropics, with Brazil reporting more cases than any other country in the world (Bhatt et al., 2013; Teixeira et al., 2009). Dengue is caused by four distinct dengue virus serotypes (DENV 1–4), which are transmitted to humans by Aedes mosquitoes (Guzman and Harris, 2015). The distribution of both Ae. aegypti and Ae. albopictus is widespread across Brazil, with Ae. aegypti found predominantly in urban settings, breeding in artificial containers, and Ae. albopictus more commonly found in rural and peri-urban settings (Kraemer et al., 2015). However, dengue incidence is unequally distributed in Brazil, with higher and sustainable incidence along the Atlantic coast and in the central region. Temperature and rainfall regimes seem to control the magnitude and seasonality of dengue transmission (Campbell et al., 2015). Large outbreaks are typically observed after rainy and warm periods, at the end of summer, particularly in densely populated urban areas (Teixeira et al., 2009). The presence and abundance of dengue mosquitoes is a necessary but not sufficient condition for dengue transmission and the occurrence of large outbreaks. Besides vector infestation, an important factor regulating transmission is the introduction of new serotypes of virus in areas with a high susceptible population. This may be facilitated by the increasing international and internal mobility across the country. Thus, large and touristic cities are prone to the introduction and maintenance of virus circulation. International mass gathering events have become an important health issue in the recent years (Abubakar et al., 2012) as they create the opportunity for the introduction of new pathogens in a susceptible population, as well as exposing visitors to new and unknown local risks (Matos and Barcellos, 2010).

Early warning systems, which take into account multiple dengue risk factors, can assist public health authorities to implement timely control measures ahead of imminent dengue outbreaks. Seasonal climate forecasts combined with early dengue surveillance system data provide an opportunity to anticipate dengue outbreaks several months in advance (Lowe et al., 2014). To date, several studies have assessed the use of climate information in early warning systems for diseases, such as malaria and Rift Valley fever (Anyamba et al., 2009; Thomson et al., 2006). The incorporation of climate information for dengue early warning systems has also been explored (Degallier et al., 2010; Lowe et al., 2011; Stewart-Ibarra and Lowe, 2013). However, to our knowledge, real-time climate forecasts have not been previously applied to predict dengue epidemics in a practical real-life framework.

From 12 June to 13 July 2014, Brazil hosted the 2014 Fédération Internationale de Football Association (FIFA) World Cup, a mass gathering of more than 3 million Brazilian and international spectators, travelling between 12 different host cities. Before the event, the potential risk of transmission of several communicable diseases, including dengue fever, was highlighted (Gallego et al., 2014; Wilson and Chen, 2014). Several research groups published dengue outlooks ahead of the World Cup. Approaches included analysing historical time series distributions of city or state level data (Hay, 2013) and mapping of historical averages, while accounting for seasonality and areas of permanent transmission (Barcellos and Lowe, 2014a). Some groups formulated deterministic (Massad et al., 2014) and statistical (van Panhuis et al., 2014) models to estimate the number of tourists expected to contract dengue fever. Another study (Lowe et al., 2014) assessed the potential for dengue epidemics during the tournament by providing probabilistic forecasts of dengue risk for the 553 microregions of Brazil with risk-level warnings issued for the 12 cities where the matches were played. The dengue early warning system, formulated using a Bayesian spatio-temporal model framework (Lowe et al., 2011, 2013), was driven by real-time seasonal climate forecasts for the period March-April-May and the dengue cases reported to the Brazilian Ministry of Health in February 2014. This information was combined to produce a dengue forecast at the start of March 2014. Predicted probability distributions of dengue incidence rates (DIR) were summarised and translated into risk warnings, using a two-tier threshold approach. First, the probability of DIR falling into categories of low, medium and high risk was determined using dengue risk thresholds of 100 and 300 cases per 100,000 inhabitants, defined by the National Dengue Control Programme of the Brazilian Ministry of Health (Ministério da Saúde, 2008). Second, probability trigger thresholds were calculated by selecting optimal cut-off values that maximised sensitivity and specificity, for each dengue risk threshold (medium and high). Using criteria related to the probability trigger thresholds, forecast warnings of low, medium or high dengue risk were determined for the 12 microregions hosting World Cup matches (see Materials and methods for further details). The forecasts were produced and made available three months ahead of the event (see Lowe et al., 2014). In this article, we provide an evaluation of the forecast model predictions by using the observed dengue incidence rates for June 2014 to assess the ability of the model framework to successfully assign dengue risk warning categories for the host microregions and all microregions in Brazil. We also compare the forecast model framework to a null model, based on seasonal averages of previously observed dengue incidence. We then discuss the challenges and limitations of producing disease risk forecasts in a real-time setting, such as the use of incomplete surveillance data to drive the model, the coarse spatial resolution of the forecasts, the definition of risk and alarm trigger thresholds, the lack of information regarding the (re)introduction of different serotypes or vector control activities, and the difficulties in communicating probabilistic forecasts. Finally, we suggest future model developments and advocate a multi-model approach to dengue prediction in the future.

Results

Comparison of probability forecast and observed dengue risk categories

Table 1 shows the dengue forecasts for June 2014 for the microregions where stadiums were located, issued three months before the World Cup, and published before the event (Lowe et al., 2014). For comparison, the observed DIR values are included in the table, along with the observed risk categories, determined using the medium and high dengue risk thresholds. Note that the dengue risk thresholds used by the National Dengue Control Programme are based on yearly dengue incidence rates (Ministério da Saúde, 2008). Therefore, we converted the monthly incidence to yearly equivalent incidence to make use of the medium and high dengue risk thresholds at a monthly time scale (see Materials and methods). Figure 1 shows the corresponding predictive distributions, the posterior predicted mean DIR and upper 95% prediction interval (dashed and dotted lines) and observed DIR (marked with an arrow). The comparison of the second and last columns of Table 1 reveals that the model correctly predicted dengue risk categories (highlighted in blue) for Fortaleza and Natal (high), Belo Horizonte, Manaus and Salvador (medium) and Curitiba and Porto Alegre (low). In Recife, the predicted category was high, but the observed category was medium. However, for Recife, the mean predicted DIR was almost equal to the observed DIR and the point estimate fell within the medium category (see Table 1, Figure 1). The definition of the alarm trigger threshold placed this microregion in the high category. This result highlights the difficulties of translating probabilistic information into simpler warnings, based on predefined probability trigger thresholds. The model 'missed' the unprecedented high incidence that was observed in Brasília and São Paulo in June 2014. However, the model predicted a 7% forecast probability of observing high risk in Brasília.

Table 1.

Dengue risk forecast warnings and corresponding observations for June, 2014 for host microregions. Dengue risk forecast warnings and observed category for June 2014, for the microregions hosting the World Cup tournament. Low risk was defined as fewer than 100 cases per 100,000 inhabitants, medium risk as between 100 and 300 cases per 100,000 inhabitants, and high risk as greater than 300 cases per 100,000 inhabitants. If the probability of low risk was less than 68%, a medium risk forecast warning was issued. If the probability of high risk was concurrently greater than 18%, the forecast warning was upgraded to high risk. The observed DIR value is included. Microregions where the observed DIR fell into the same category as forecast (e.g. the forecast warning category was high and the observed DIR category was high), are shaded.

DOI: http://dx.doi.org/10.7554/eLife.11285.003

Microregion	Forecast warning	Probability (p_L, p_M, p_H)	Observed DIR	Observed category
Belo Horizonte	Medium	p(65%, 24%, 11%)	126	Medium
Brasília	Low	p(73%, 20%, 7%)	725	High
Cuiabá	Low	p(71%, 22%, 7%)	168	Medium
Curitiba	Low	p(100%, 0%, 0%)	4	Low
Fortaleza	High	p(34%, 20%, 46%)	507	High
Manaus	Medium	p(63%, 25%, 12%)	110	Medium
Natal	High	p(32%, 20%, 48%)	780	High
Porto Alegre	Low	p(100%, 0%, 0%)	1	Low
Recife	High	p(57%, 24%, 19%)	161	Medium
Salvador	Medium	p(56%, 27%, 17%)	149	Medium
São Paulo	Low	p(99%, 1%, 0%)	161	Medium
Rio de Janeiro	Medium	p(62%, 25%, 13%)	32	Low

Open in a new tab

Figure 1. — Posterior predictive distributions of dengue incidence rates (DIR) (base-10 logarithmic scale) for June 2014 showing the probability of low risk (blue), medium risk (orange) and high risk (pink) for June 2014, in the microregions hosting the World Cup tournament: (a) Belo Horizonte, (b) Brasília, (c) Cuiabá, (d) Curitiba, (e) Fortaleza, (f) Manaus, (g) Natal, (h) Porto Alegre, (i) Recife, (j) Salvador, (k) São Paulo and (l) Rio de Janeiro. Observed DIR indicated by black arrow. Posterior predictive mean and upper 95% prediction (credible) interval of the distribution indicated by a dashed and dotted line, respectively.

**DOI:** http://dx.doi.org/10.7554/eLife.11285.004

Probabilistic forecasts were generated not only for the twelve host microregions, but for all 553 microregions of Brazil. This gives an idea of how the model framework might contribute towards a nationwide dengue early warning system in the future. Figure 2 shows a ternary probabilistic forecast map (Jupp et al., 2012; Lowe et al., 2014) and the corresponding observed dengue incidence rate categories (low, medium and high). The model correctly predicted, with high certainty (the greater the colour saturation, the greater the certainty), low dengue risk in South Brazil and large areas of the Amazon. Areas with a higher chance of observing high risk were correctly detected for areas in North East Brazil. Actual dengue incidence rates were higher than expected in Brasília, although the likelihood of observing higher dengue incidence for the surrounding region was relatively greater than observing lower incidence. For some microregions in the state of São Paulo, the model was uncertain of the most likely category (indicated by pale colours). Some of these areas experienced high dengue incidence rates in June 2014.

Figure 2. — (a) Probabilistic dengue forecast for June 2014. The continuous colour palette (ternary phase diagram) conveys the probabilities assigned to low-risk, medium-risk, and high-risk dengue categories. Category boundaries defined as 100 cases per 100,000 inhabitants and 300 cases per 100,000 inhabitants. The greater the colour saturation, the more certain is the forecast of a particular outcome. Strong red shows a higher probability of high dengue risk. Strong blue indicates a higher probability of low dengue risk. Colours close to white indicate a forecast similar to the benchmark (long-term average distribution of dengue incidence in Brazil, June, 2000–2013: p_L=68%, p_M=16%, p_H=16%), marked by a cross. (b) Observed dengue incidence rate (DIR) categories for June, 2014.

**DOI:** http://dx.doi.org/10.7554/eLife.11285.005

Figure 3 shows the probability of DIR falling in the category that was actually observed. The deeper the colour shading, the greater the probability of observing the correct category. This gives an indication of the certainty of the model in predicting correct outcomes. In general, a high degree of certainty in the forecast is found in the south region, parts of the Amazon and many densely populated cities along the eastern coastline. However, as the historical distribution of DIR is not symmetrical, with a greater proportion of the distribution in the low category, compared to the high category (as epidemics can be considered as 'extreme events'), it is interesting to consider each category individually. Figure 4a–c show conditional maps of the forecast probability given that low, medium and high DIR was observed, respectively. The grey areas indicate areas where the observed DIR fell in the other two categories and are therefore not considered in each individual map. The probability trigger thresholds defined in Lowe et al., (2014) are taken into account to weight the graduated colour bars, ranging from 0% to 100% chance of the observed category. Using the forecasting model, if the probability of low risk were greater than 68%, a low risk warning would have been assigned. If the probability of low risk were less than or equal to 68%, a medium risk warning would have been assigned (giving a medium trigger threshold of 32%). If simultaneously, high risk were greater than 18%, a high risk warning would have been assigned. Therefore, lower probabilities are assigned more weight (represented by colour darkness) in the high category plot than the low category plot. Given that low risk was observed, the model framework would have correctly assigned a low risk warning for 67% of the microregions. Given that high risk was observed, the model framework would have correctly assigned a high risk warning for 57% of the microregions. High risk was correctly forecast with considerable certainty in microregions in the north east of Brazil near Fortaleza (see Figure 4c). Although the model 'missed' the high risk observed in Brasília, it was able to correctly detect, with a relatively high degree of certainty high risk in surrounding microregions.

Figure 4. — Forecast probability given that (a) low, (b) medium and (c) high DIR was observed. Grey areas indicate that other DIR categories were observed and are therefore not considered. The graduated colour bar represents the probability of observing the given category (ranging from 0%, pale colours, to 100%, deep colours). Note, the alarm trigger thresholds are marked with a star (*). For (a) low risk warnings, p_L > 68%, for (b) medium risk warnings, p_M > 32% and for (c) high risk warnings, p_H > 18%. Colour bars are weighted, with increased saturation beyond the alarm trigger threshold to reflect the correct assignation of warnings.

**DOI:** http://dx.doi.org/10.7554/eLife.11285.007

Comparison of forecast model framework to a null model

Useful predictions from a forecasting system are likely to be those that recommend changes from the activities that would otherwise have taken place anyway, which are typically based on the 'normal' dengue season. Beyond that, predictions that forecast higher than expected incidence are critical, as they could advocate increased interventions. To assess the performance of the forecast model framework beyond a simple seasonal profile, we defined a null model as the average DIR in each microregion for June 2000–2013. We consider the ability of both the forecast model and the null model to predict 'high risk' dengue across Brazil. Table 2 shows a summary of contingency table results for observed DIR exceeding the high risk epidemic threshold (300 cases per 100,000 inhabitants) using the probabilistic category forecast model and the null model for June 2014. Results show that for the June 2014 event, the forecast model predicted a greater number of true positives (hits) and fewer false negatives (misses) than the null model (see Table 2, Materials and methods). This gave a hit rate of 57% (miss rate of 43%) when using the forecast model and a hit rate of 33% (miss rate of 67%) when using the null model. However, the forecast model also tended to produce more false positives (or false alarms) than the null model (see Table 2). The two types of error (false alarms and missed events) have very different consequences for public health. For example, failing to predict an epidemic that then occurs (type II error – a miss) is much more damaging than predicting an epidemic that does not materialise (type I error – a false alarm) (Stephenson, 2000). Figure 5 shows hit rates and false alarm rates for both the forecast and null model, calculated in 'leave one year out' cross-validation mode from 2000–2013, i.e. by excluding the year for which the prediction is valid when estimating model parameters (see Materials and methods). Results for the 2014 event are also included. The hit rate for the forecast model exceeds that of the null model for all years, expect 2004, when dengue incidence was at its lowest across the whole of Brazil.

Table 2.

Summary of contingency table results for observed DIR exceeding the epidemic risk threshold. Summary of contingency table results for observed DIR exceeding the high risk epidemic threshold (300 cases per 100,000 inhabitants) using the probabilistic category forecast model and the null model (mean DIR, June 2000–2013) for June 2014.

DOI: http://dx.doi.org/10.7554/eLife.11285.008

Performance measures	Forecast model probabilistic	Null model seasonal mean
Hit	81	46
False alarm (type I error)	94	55
Miss (type II error)	60	95
Correct rejection	318	357
Hit rate	57%	33%
False alarm rate	23%	13%
Miss rate	43%	67%

Open in a new tab

Figure 5. — Comparison of (a) hit rates and (b) false alarm rates for the event of observed DIR exceeding the high risk epidemic threshold (300 cases per 100,000 inhabitants) using the probabilistic category forecast model (blue circles) and the null model (orange triangles) for June 2000–2014. The vertical bars around each point represent the 95% confidence intervals. The vertical dotted line separates the leave-one-out cross validation results (2000–2013) from the true predicted results for 2014.

**DOI:** http://dx.doi.org/10.7554/eLife.11285.009

To assess the additional value of the forecasting system beyond that of the seasonal profile, it is useful to consider the full posterior predictive distributions from the model, compared to the null model and associated prediction intervals. Figure 6 shows time series of observed and predicted dengue incidence rates for June 2000–2014 for the 12 host microregions. The posterior predictive mean and upper 95% prediction (credible) interval from the forecast model, and the sample mean and upper 95% prediction interval from the null model are also included. The forecast and null model predictions are calculated in 'leave one year out' cross-validation mode from 2000–2013, i.e., by excluding the year for which the prediction is valid when estimating model parameters (see Materials and methods). Note, the predictions for 2014 are the only 'true' forecasts (i.e., no information is included beyond the forecast issue date). When considering the posterior predictive mean of the forecast model, some of the inter-annual variations in the observations are captured by the model, for example in Belo Horizonte, Manaus and Salvador. However, in some other places, the mean prediction from the forecast and the null model are either very similar or, in some years, the null model mean is closer to the observed value, for example, in Cuiabá in 2009. During years with relatively low DIR, the predictions from the forecast model tend to be more precise than the null model, with narrower prediction (credible) intervals. Further, when DIR is exceptionally high, the forecast model is able to account for this increased possibility of an outbreak in most cases, compared to the null model. This is evident for the dengue epidemics that occurred in Belo Horizonte in 2010 and 2013 (Figure 6a), Salvador in 2010 (Figure 6j) and Manaus in 2011 (Figure 6f). Although the forecast model is far from perfect, in general, it is better able to detect extreme dengue incidence rates than the null model.

Figure 6. — Observed DIR (pink squares), posterior mean DIR (blue circles) and upper 95% prediction (credible) interval from forecast model (blue dashed line) and mean DIR (orange triangles) and upper 95% prediction interval (orange dashed line) from null model, June 2000–2014 in the host microregions (a) Belo Horizonte, (b) Brasília, (c) Cuiabá, (d) Curitiba, (e) Fortaleza, (f) Manaus, (g) Natal, (h) Porto Alegre, (i) Recife, (j) Salvador, (k) São Paulo and (l) Rio de Janeiro. The vertical dotted line separates the leave-one-out cross validation results (2000–2013) from the true predicted results for 2014.

**DOI:** http://dx.doi.org/10.7554/eLife.11285.010

Discussion

In this study, we have evaluated the ability of the dengue early warnings produced three months ahead of the 2014 World Cup to anticipate dengue risk categories, using pre-determined risk and alarm trigger thresholds. The forecasts correctly predicted dengue risk categories for seven of the twelve host microregions: Fortaleza and Natal (high), Belo Horizonte, Manaus and Salvador (medium) and Curitiba and Porto Alegre (low). The model was able to detect medium risk for Recife (posterior mean DIR=142, observed DIR=161, see Figure 1), although the definition of the alarm trigger thresholds placed this microregion in a higher risk category (the probability of high risk just exceeded 18%). The model missed the higher incidence that was observed in Brasília and São Paulo. The observed dengue incidence rate for Rio de Janeiro was lower than usual for this time of year. When comparing the ability of the forecast model to the null model to predict high dengue risk across Brazil, the forecast model produced more hits and fewer missed events than the null model, with hit rates of 57% for the forecast model compared to 33% for the null model. The hit rate was almost always greater for the forecast model during previous years, 2000–2013. Despite the tendency of the forecast model to over predict dengue risk compared to the null model, we generally found the forecast model better able to detect extreme dengue incidence rates. For catastrophic events, such as epidemics, failing to predict an epidemic (e.g. forecast warning is for low risk but observed incidence is high) is considered much more damaging than announcing that an epidemic is about to occur and then does not happen (e.g. forecast warning is for high but observed incidence is low). Besides, if the dengue control programme intervene based on the forecast warnings, this could result in a decrease in cases compared to what would have happened otherwise.

The month of June is the transition in the southern hemisphere between autumn and winter, with the intensification of lower temperatures in central and southern Brazil (from Rio Grande do Sul to São Paulo and high plateaus of Minas Gerais), leading to areas with lower incidence rates. The model could, in general, distinguish climate-limiting areas for dengue fever (Barcellos and Lowe, 2014b). However, the model was unable to predict the high risk category for Brasília and São Paulo, which experienced unprecedented dengue incidence for the month of June. As the model is formulated using past data, and driven by climate forecasts and dengue data with a 3–4 month lead time, such irregularities and sudden shifts in transmission patterns can be difficult to capture. However, potential variability is accounted for in the random component of the model and correctly reflected in the probabilistic nature of our predictions.

The areas with greater model uncertainty (see pale areas in Figure 2a) are located along the fringes of climatic zones: the southern limits between the tropical warm and temperate climates, near the tropic of Capricorn; the Amazon rainforest contact with the tropical savanna vegetation (Cerrado); and the semiarid boundaries in the Northeast region. In fact, these fringes were the areas where there was a low probability of observing the correct category (see Figure 3). From the meteorological perspective, these zones should not be assumed as fixed boundaries, as they may vary over the years under the influence of the El Niño-Southern Oscillation, dominant air masses and other transient drivers. Important dengue outbreaks have been observed along these fringes that may be seen as a prelude of the expansion of dengue transmission area (Barcellos and Lowe, 2014b).

The dengue early warning model takes into account seasonal climate forecasts and epidemiological data for the months preceding the event, which create the conditions (environmental and epidemiological) for vector density and distribution, as well virus circulation. These conditions largely influence the development of an epidemic. Ideally, data about health care provision and vector control intensity between regions and over time would be included in the model. However, in the absence of such data, spatial and seasonal heterogeneity and dependency structures are accounted for in the model via random effects at the scale of calendar month and microregion, to try and ensure that the variance in the predictions includes uncertainty that could be generated by these unknown features of the disease system. Temporal random effects were not included in the model. Such effects are useful to help describe variations in temporal patterns and understand the relative contribution of various temporal explanatory variables. However, when it comes to predicting into the future, these effects become obsolete. In reality, dengue outbreaks are also modulated by factors that occur in more restricted temporal and spatial scales, which are not accounted for in the model framework, thus limiting the results of this study (Lowe, 2015). Such risk factors include the implementation and efficiency of vector control activities, circulating serotypes and herd immunity, along with other complex socio-economic factors, such as water storage practices.

From the start of 2014, a high pressure system over Southeast Brazil prevented the entrance of cold frontal systems from the south and the transport of humid air from the Amazonian Region in northern Brazil towards the southeast region of Brazil (Coelho et al., 2015b). This led to severe drought conditions in the South East region, which particularly affected the states of São Paulo and Minas Gerais. This situation led to a water supply system crises that began in the first semester of 2014 (Coelho et al., 2015a). This likely resulted in an increase of artificial water storage containers, increasing mosquito breeding sites and thus intensifying dengue transmission in this region (Otto et al., 2015). The combination of introducing new serotypes of dengue virus in a large city, with a huge number of susceptible inhabitants, with high mobility and circulation of tourists, during a water crisis may have had an explosive effect on dengue transmission in this region. The unforeseen dengue outbreak in the microregion of Brasília may have resulted from a change in the dominant serotype to DENV1 (Ministério da Saúde, 2014) and the immunological status of the population. However, this requires further investigation. The four month lag between the target forecast date (June 2014) and the dengue cases observed before the forecast issue date (February 2014) was perhaps too long to capture these unprecedented changes in the dengue dynamics. In an ideal situation, the forecast would be updated each month leading up to the forecast target date, to incorporate the latest dengue and climate information available.

The objective of the model framework was to predict the probability of exceeding pre-defined 'epidemic' thresholds (see Materials and methods). The idea of a forecast that provides the probability of exceeding an epidemic threshold is to allow decision makers to quantify the level of certainty of the model predictions. For example, farmers often consult seasonal climate forecasts for their region of interest to decide when, how and what to plant their crops (Meinke and Stone, 2005). The information provided is given in a probabilistic context, e.g. “there is a 70% chance of observing temperatures above normal next season” rather than a specific value, e.g. “the temperature next season will be 27°C”. Considering a dengue forecast, if the probability of exceeding the high dengue risk threshold for June in a given location is 90%, public health officials may be inclined to increase dengue control measures in that area (Lowe et al., 2015a). This information is likely to be more informative and beneficial to decision makers than a deterministic forecast indicating that “there will be a DIR of 182 in June”.

The advantage of issuing probabilistic forecasts is that we can more easily quantify prediction uncertainty. However, effectively communicating probabilities to decision makers, the general public and even other scientists can be challenging. The use of set incidence (risk) thresholds and static probability decision (i.e. alarm trigger) thresholds means that the translation of probabilities into discrete warnings (low, medium and high) might not always reflect the predictive power of the model. For example, even though the mean estimate of the predictive distribution might be close to the observations (e.g. for the case of Recife), the predetermined cut-off alarm trigger thresholds might result in the designation of an incorrect categorical warning. Risk and alarm trigger thresholds need to be carefully chosen, as this influences the outcome of the 'translation' of probabilistic information into categorical risk warnings for decision makers.

The dengue incidence rate baseline has been increasing over the years. Since 1998, the overall incidence rate reached a plateau, situated between 100 and 300 cases per 100,000 inhabitants (Barreto et al., 2011). However, this seems to be a consequence of the spread of the disease in the country rather than an increase of incidence rates in cities (Barcellos and Lowe, 2014b). For example, the number of cities reporting dengue cases increased since the late 1980's from a few hundred to almost 4000, reaching 72% of the Brazilian territory (Barreto et al., 2011). The three class dengue risk levels (low<100, 100<medium<300 and high>300 cases per 100,000 inhabitants per year) used by the National Dengue Control Programme have been shown to appropriately capture the occurrence of large epidemics for certain cities, such as Rio de Janeiro (Lowe et al., 2013). For this city, the thresholds of 100 and 300 cases per 100,000 inhabitants has been adequate for distinguishing the recent outbreaks during the peak season (February-April) in the city since the year 2000. Using these thresholds, three large outbreaks in 2002, 2008 and 2012 (DIR>300), and smaller outbreaks in 2011 and 2013 (DIR>100) would have been detected (data not shown). However, these thresholds may not be appropriate for other cities, regions or seasons. This could be revised by taking into account population variability between microregions. For example, in the state of São Paulo, the decision to clinically or epidemiologically confirm dengue cases (see Materials and methods) is based on dengue incidence rates exceeding the high risk category threshold in small cities and the medium risk category threshold for large cities (SES, 2010). Recently, several different outbreak definitions have been tested using dengue data for Brazil, including moving averages, cumulative mean and fixed thresholds (Brady et al., 2015). However, there is still no clear recommendation as to the most appropriate threshold(s) to apply for a nationwide dengue early warning system in Brazil. The current dengue risk thresholds are defined using yearly dengue incidence rates and applied to the whole country. These could be refined by considering seasonal dengue variations in different regions across Brazil to create location specific thresholds, better able to detect the onset of dengue outbreaks at different times throughout the year.

The model framework could be improved by revising the definition of the alarm trigger thresholds, which are currently determined by assessing the ability of the model to distinguish medium and high risk categories in the past. These could be developed by stochastically simulating the optimum threshold for each category boundary multiple times, to produce a probability distribution of alarm trigger thresholds. Therefore, credible intervals could be assigned to the alarm trigger thresholds. This could help to flag warnings that might border two categories, e.g. low-medium or medium-high. For example, for Recife, the probability of high risk was 19% and the high risk alarm trigger threshold was a point estimate of 18%, which placed Recife into the high category, even though the predicted probability was just 1% greater than the trigger threshold. Ideally, a cost-benefit analysis should be performed by the public health services to assess the economic cost of false positives versus false negatives, to define optimum alert trigger thresholds. Another point to consider is the extent to which forecast warnings, issued several months in advance, influence vector control and personal protection activities. If a dengue forecast warning provokes more intense interventions or increased education and public awareness, this could potentially change the course of the dengue transmission patterns expected due to the environmental conditions and epidemiological trends.

Despite these limitations, the analyses performed in this study revealed several areas across Brazil with successful predictions. These results are promising for the future development of a routine dengue early warning system for Brazil. This will be conducted in consultation with the Brazilian Ministry of Health to align model capabilities with decision-maker requirements and data availability. In practice, early warning systems for such a complex disease should include several surveillance activities. Our model provides advance warning of the likely dengue risk three months later, based on climate forecasts and the current dengue risk situation at the time of forecast. This information should be updated and complimented with surveillance on the (re)introduction of new serotypes and notification tracking to ensure the ability of local health services to stay alert for dengue diagnosis. More widespread and better designed surveys to track changes in dengue seroprevalence should be routinely conducted. Establishing a new seroprevalence surveillance system at the scale of the whole country is a challenge. A national system could be initiated in key capital cities, to capture the entrance of new pathogens from other states or countries. The probability of introducing new serotypes could be determined by the number of people entering local airports and bus terminals from endemic regions. Given prior knowledge of seroprevalence, an estimate of the population at risk stratified by serotype could be incorporated in the model offset. By integrating data from the sentinel dengue seroprevalence surveillance systems at the time of forecast, predictions for each serotype could then be produced, to assess the epidemic potential generated by a re-emerging or dominant stereotype.

The model framework could also be downscaled and refined to account for within city disease dynamics and consider the impact of climate of other mosquito-transmitted viruses, such as chikungunya and Zika, which have recently emerged in Brazil (Nunes et al., 2015; Zanluca et al., 2015). This could help to assess the risk of arboviral disease importation and transmission in Rio de Janeiro ahead of the 2016 Olympics.

Several research groups developed dengue risk maps and forecasts ahead of the World Cup (Barcellos and Lowe, 2014a; Hay, 2013; Lowe et al., 2014; Massad et al., 2014; van Panhuis et al., 2014). While results varied between studies, all agreed that the northeast cities of Fortaleza and Natal would be at greater risk of dengue transmission and that the risk for the southern cities of Porto Alegre and Curitiba would be very low. Massad et al. (2014) also forecast a greater risk to tourists in the city of Rio de Janeiro. Only one study corrected predicted elevated dengue risk for teams and tourists in Brasília, using an Empirical Bayes model and weekly dengue incidence data up to May 2014 (i.e., one month lead time) (van Panhuis et al., 2014). Given the different research questions, methods, data and underlying populations that were used for each study, the direct comparison of these model results is not feasible in the absence of a consensus to translate results into equivalent quantities (Lowe et al., 2015). For example, the system evaluated here generated probabilistic distributions of yearly equivalent dengue incidence rates in the Brazilian population, at the monthly time scale and microregion spatial scale with 3–4 months lead-time, while other studies estimated the number of visitors at risk of dengue infection in specific cities. This emphasises the importance of dengue model inter-comparison projects, such as the 'Epidemic Prediction Initiative' launched by the White House Office of Science and Technology Policy (http://dengueforecasting.noaa.gov/). The comparison of different models, forecasting common targets, will identify strengths and weakness in model formulation and gaps in data and methods.

Conclusion

Several studies have developed models for dengue fever, using climate and other risk factors, and tested predictive performance in retrospective mode. However, none of these studies have incorporated real-time seasonal climate forecasts and epidemiological data to predict future dengue risk. Therefore, to our knowledge, this work constitutes the first evaluation of a nationwide dengue early warning, issued before a global mass gathering. The dengue early warnings were disseminated to the Ministry of Health, the general public and visitors travelling to Brazil, prior to the World Cup. The predictions were incorporated into the European Centre for Disease Control (ECDC) health risk assessment (ECDC, 2014) and reported by more than 18 international press outlets. As a result, the forecast by Lowe et al. (2014), along with others (Hay, 2013; Massad et al., 2014), further contributed by raising general awareness about dengue fever and the risk of contracting the disease when travelling to endemic regions. This dengue early warning framework may be useful, not only ahead of mass gatherings, but also before the peak dengue season each year, to control or contain potentially explosive dengue epidemics. The use of real-time seasonal climate forecasts and early epidemiological reports in routine dengue early warnings is now a priority for the Brazilian Climate and Health Observatory (www.climasaude.icict.fiocruz.br), in collaboration with the Brazilian Institute for Space Research. We hope this prototype will serve as a demonstration for scientists, health surveillance teams and decision makers of the data and tools required to produce, communicate and evaluate timely predictions of climate-sensitive disease risk.

Materials and methods

Data

We obtained dengue data for June 2014 from the Notifiable Diseases Information System (SINAN), organised by the Brazilian Ministry of Health. We then aggregated the cases for the 5570 municipalities, 42% of which have less then 10,000 inhabitants, to the microregion level. A microregion is defined as an aggregate of neighbouring municipalities, with common economic interests and frequent population exchanges. This helps to alleviate problems of low population numbers and misreporting, due to variations in availability of health services/epidemiological facilities at the municipality level. This data includes confirmed cases of dengue fever, including mild infections, dengue haemorrhagic fever, and shock syndrome. Dengue cases can be confirmed by laboratory exams or clinical and epidemiological evidence. In the second case, a patient must present at least two of the following symptoms: high fever, severe headache, severe eye pain, joint and muscle pain, mild bleeding manifestation or low white cell count. In addition to these symptoms, the patient must have been in areas where dengue is being transmitted or where there has been an infestation of Ae. aegypti in the past 15 days (Ministério da Saúde, 2005). Dengue notification records are considered a priority in the epidemiological surveillance system in Brazil. Data flow is accelerated in relation to other diseases. About 50% of cases are reported within 3 days after the first symptoms, and 90% of cases are digitised within 7 days of notification (Barbosa et al., 2015). During outbreaks, this flow tends to be speeded up, adopting optimisation measures, for example, using 'clinical and epidemiological evidence' to confirm cases that could not be submitted for laboratory confirmation. These criteria allow accelerated case reporting, prioritising sensitivity and opportunity rather than specificity of information (Duarte and França, 2006). In fact, during high incidence periods, the proportion of cases confirmed by laboratory criteria is lower than during low incidence periods and 'clinical and epidemiological evidence' is a common procedure for case confirmation. On the other hand, about 50% of suspect cases are subsequently confirmed after laboratory tests in epidemic periods, while in periods of low transmission intensity, approximately 30% of cases are confirmed, revealing good predictive value of suspected cases (Barbosa et al., 2015). Therefore, after post-processing, some cases are discarded because they have negative serology. However, beyond these 30% of cases, identification of serotype is not carried out by laboratory tests, which hinders the understanding of transmission dynamics and population susceptibility level. Since recent data are still subject to confirmation of cases and elimination of duplicate registers, the initial figures of dengue cases may be modified in the following months and will most likely be official by the end of 2015. These data will then be made publicly available via the Health Information Department (DATASUS, http://dtr2004.saude.gov.br/sinanweb/).

We used 2014 population estimates obtained from the Brazilian Institute for Geography and Statistics (IBGE, 2014) to convert the case data into dengue incidence rates (DIR), per 100,000 inhabitants. Other estimates of population for inter census years are produced by national institutions and may present discrepancies, mainly for small populations and newly created municipalities. According to the methodology used by the Ministry of Health, the DIR is calculated for a geographical space in a given year (PAHO, 2008). As the dengue risk thresholds used by the National Dengue Control Programme are based on yearly dengue incidence rates (Ministério da Saúde, 2008), it is necessary to use a proportion (1/12) of yearly population estimate as the denominator in the dengue incidence rate calculation, to make use of this at a monthly time scale. Therefore, we converted the monthly incidence to yearly equivalent incidence to make use of the risk thresholds of 100 and 300 cases per 100,000 per year. This is consistent with the metrics published in the Epidemiological Bulletins of the Ministry of Health. Our goal was to provide measures that can be easily interpreted by the Dengue Control Programme and translated into well understood risk levels (low, medium, high).

Dengue forecast formulation and translation

A spatio-temporal Bayesian hierarchical model (Lowe et al., 2011, 2013, 2014) was formulated, using monthly dengue cases, from 2000 to 2013, for 553 Brazilian microregions as the response variable. Based on findings from previous studies (Lowe et al., 2013), the climate variables used to formulate the model were three-month average temperature (Fan and Van den Dool, 2008) and precipitation (Adler et al., 2003) anomalies (departures from the long-term average), over the three months preceding the dengue month of interest. This is equivalent to a two month lag when considering the mid-point of the three month average. Lags of 1–3 months are typically used when modelling dengue (Lowe et al., 2015b), to try and capture the impact of rainfall on mosquito breeding sites and the effect of temperature on the mosquito life cycle, although these relationships are still not well understood. Other explanatory variables included population density, altitude, and dengue relative risk (ratio of observed to expected cases) lagged by four months. Zone-specific seasonality was accounted for using autocorrelated annual cycles (i.e. by allowing each calendar month to depend on the previous month) for different Brazilian ecological zones (e.g. Amazon, Caatinga, Cerrado, Atlantic Pampa, Pantanal). Unknown confounding factors (e.g. health care and vector control disparities between microregions) and dependency structures (i.e., human mobility between neighbouring areas) were allowed for using area-specific unstructured and structured random effects (see Lowe et al. (2014) for further details).

To produce the forecast for June 2014, the model was driven by (1) real-time seasonal precipitation and temperature anomaly forecasts (Coelho et al., 2006), produced in mid-February by the Center for Weather Forecasting and Climate Research (CPTEC) (valid for the March-May [MAM] season) and (2) the observed epidemiological situation (ratio of observed to expected cases) for February, 2014, collated in March, 2014 by the Ministry of Health [see Lowe et al. (2014) for details]. Note, the precipitation seasonal forecasts used in this study were produced by CPTEC as part of EUROBRISA: A Euro-Brazilian Initiative for improving South American seasonal forecasts (http:eurobrisa.cptec.inpe.br).

Posterior predictive distributions were simulated for every microregion to determine the probability of dengue incidence rates exceeding predefined risk thresholds (see Figure 1). Probability forecasts (p_L, p_M, p_H) were issued for low (fewer than 100 dengue cases per 100,000 inhabitants), medium (between 100 and 300 dengue cases per 100,000 inhabitants) and high (more than 300 dengue cases per 100,000 inhabitants) risk. These results were presented using a visualisation technique (Jupp et al., 2012), where the forecast for each microregion was expressed as a colour determined by a combination of three probabilities, with colour saturation used to indicate certainty for a particular category (see Figure 2a). We then used a receiver operating characteristic (ROC) analysis of past forecasts and observations from 2000–2013 to define optimal probability thresholds for warnings. If the probability of low risk was less than 68%, a medium risk forecast warning was issued. If the probability of high risk was concurrently greater than 18%, the forecast warning was upgraded to high risk (see Table 1).

Dengue forecast evaluation

After the event, we compared the published probabilistic predictions with observed DIR data for June 2014. We defined a null model as the seasonal average of past dengue incidence (i.e., mean DIR for June 2000–2013). We assessed the ability of the forecast model and the null model to determine the binary event of DIR exceeding 300 cases per 100,000 inhabitants (i.e. the high risk threshold) for n=553 microregions in Brazil (Table 2). Table 3 shows the two ways for the forecast to be correct (either a hit or a correct rejection) and two ways for the forecast to be incorrect (either a false alarm or a miss). Cell count a is the number of events correctly forecast to occur, i.e. the number of hits; cell count b is the number of events incorrectly forecast to occur, i.e., the number of false alarms; cell count c is the number events incorrectly forecast not to occur, i.e., the number of misses; and cell count d is the number of event correctly forecast not to occur, i.e., the number of correct rejections (Jolliffe and Stephenson, 2012). We calculated performance measures, such as the hit rate; the proportion of events (i.e., epidemics) that were correctly predicted (a/(a+c), also know as true positive rate or sensitivity) and the false alarm rate; the proportion of events that were predicted but did not occur (b/(b+d), also know as false positive rate or 1-specificity). The false alarm rate can be interpreted as the rate of making a 'type I error', whereas the 'miss rate,' equal to one minus the hit rate, measures the rate of making a 'type II error' (Stephenson, 2000).

Table 3.

The four possible outcomes for categorical forecasts of a binary event.

DOI: http://dx.doi.org/10.7554/eLife.11285.011

		Event observed
		Yes	No	Total
Forecast warning issued	Yes	Hit (a)	False alarm (b)	a+b
	No	Miss (c)	Correct rejection (d)	c+d
	Total	a+c	b+d	a+b+c+d=n

Open in a new tab

We compared time series of observed and predicted DIR from the forecast and null model for June 2000–2014 in the twelve host microregions. The mean of the posterior predictive distribution from the forecast model and the 95% prediction (credible) interval, obtained from the 2.5% and 97.5% percentiles of the distribution were calculated for June 2014, using the forecast model fitted to data from 2000–2013 (note, the lower 95% prediction interval from the forecast model was nearly always equal to zero and is therefore not shown in the figures). For the years 2000–2013, the model was fitted 14 times, excluding one year at a time when estimating model parameters, to produce 'cross validated' predictions to test against 'out-of-sample' data (i.e. the year for which the predictions are valid).

For the null model, the 95% prediction intervals for the sample mean was calculated as $\bar{y} \pm t_{n - 1, α / 2} s \sqrt{1 + \frac{1}{n}}$ , where $\bar{y}$ is the sample mean, $t_{n - 1, α / 2}$ is the 100(1-α/2)^th percentile of T Distribution, with n−1 degrees of freedom, and s is the standard error. For the 2014 null model prediction, the mean, standard error and 95% prediction intervals were calculated using past data for June 2000–2013 (n=14). For 2000–2013, the null model mean, standard error and 95% prediction intervals were calculated in cross-validated mode, by excluding one year at a time (n=13) (note, the lower 95% prediction interval for the null model was nearly always less than zero and is therefore not shown in the figures).

Acknowledgements

We are grateful to the Brazilian Ministry of Health for providing the dengue data. The dynamical ensemble forecast data was kindly provided by the European Centre for Medium-Range Weather Forecasts (ECMWF) as part of the EUROBRISA license agreement.

Funding Statement

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Funding Information

This paper was supported by the following grants:

Seventh Framework Programme DENFREE project,FP7-HEALTH.2011.2.3.3-2; 282378 to Rachel Lowe, Xavier Rodó.
Seventh Framework Programme EUPORIAS project, FP7-ENV.2012.6.1-1; 308291 to Rachel Lowe, Xavier Rodó.
Conselho Nacional de Desenvolvimento Científico e Tecnológico Produtividade em Pesquisa - PQ - 2013, 306863/2013-8 to Caio AS Coelho.
Seventh Framework Programme SPECS project, FP7-ENV-2012-1; 308378 to Caio AS Coelho, David B Stephenson.
Financiadora de Estudos e Projetos Brazilian Research Network on Global Climate Change, Rede Clima - FINEP, 01.13.0353-00 to Christovam Barcellos.
Conselho Nacional de Desenvolvimento Científico e Tecnológico Brazilian Observatory of Climate and Health, 552746/2011-8 to Christovam Barcellos.
Conselho Nacional de Desenvolvimento Científico e Tecnológico Produtividade em Pesquisa - PQ - 2013, 309692/2013-0 to Marilia Sá Carvalho.
Fundação de Amparo à Pesquisa do Estado do Rio de Janeiro E-23557/2014 to Marilia Sá Carvalho.
Fundação de Amparo à Pesquisa do Estado de São Paulo BEPE 2014/17676-0 to Rafael De Castro Catão.

Additional information

Competing interests

The authors declare that no competing interests exist.

Author contributions

RL, Conception and design, Analysis and interpretation of data, Drafting or revising the article.

CASC, Conception and design, Analysis and interpretation of data, Drafting or revising the article.

CB, Conception and design, Analysis and interpretation of data, Drafting or revising the article.

MSC, Conception and design, Analysis and interpretation of data, Drafting or revising the article.

RDCC, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article.

GEC, Acquisition of data, Drafting or revising the article, Contributed unpublished essential data or reagents.

WMR, Acquisition of data, Drafting or revising the article, Contributed unpublished essential data or reagents.

TCB, Conception and design, Analysis and interpretation of data, Drafting or revising the article.

DBS, Conception and design, Analysis and interpretation of data, Drafting or revising the article.

XR, Conception and design, Analysis and interpretation of data, Drafting or revising the article.

References

Abubakar I, Gautret P, Brunette GW, Blumberg L, Johnson D, Poumerol G, Memish ZA, Barbeschi M, Khan AS. Global perspectives for prevention of infectious diseases associated with mass gatherings. The Lancet Infectious Diseases. 2012;12:66–74. doi: 10.1016/S1473-3099(11)70246-8. [DOI] [PubMed] [Google Scholar]
Adler RF, Huffman GJ, Chang A, Ferraro R, Xie P-P, Janowiak J, Rudolf B, Schneider U, Curtis S, Bolvin D, Gruber A, Susskind J, Arkin P, Nelkin E. The version-2 global precipitation climatology project (GPCP) monthly precipitation analysis (1979–present) Journal of Hydrometeorology. 2003;4:1147–1167. doi: 10.1175/1525-7541(2003)004<1147:TVGPCP>2.0.CO;2. [DOI] [Google Scholar]
Anyamba A, Chretien J-P, Small J, Tucker CJ, Formenty PB, Richardson JH, Britch SC, Schnabel DC, Erickson RL, Linthicum KJ. Prediction of a rift valley fever outbreak. Proceedings of the National Academy of Sciences of the United States of America. 2009;106:955–959. doi: 10.1073/pnas.0806490106. [DOI] [PMC free article] [PubMed] [Google Scholar]
Barbosa JR, Barrado Jean Carlos dos Santos, Zara Ana Laura de Sene Amâncio, Siqueira JB. Avaliação da qualidade dos dados, valor preditivo positivo, oportunidade e representatividade do sistema de vigilância epidemiológica da dengue no Brasil (2005 a 2009) Epidemiologia e Serviços de Saúde. 2015;24:49–58. doi: 10.5123/S1679-49742015000100006. [DOI] [Google Scholar]
Barcellos C, Lowe R. Dengue and the World Football Cup: a matter of timing. PLoS Neglected Tropical Diseases. 2014a;8:e11285. doi: 10.1371/journal.pntd.0003022. [DOI] [PMC free article] [PubMed] [Google Scholar]
Barcellos C, Lowe R. Expansion of the dengue transmission area in Brazil: the role of climate and cities. Tropical Medicine & International Health. 2014b;19:159–168. doi: 10.1111/tmi.12227. [DOI] [PubMed] [Google Scholar]
Barreto ML, Teixeira MG, Bastos FI, Ximenes RAA, Barata RB, Rodrigues LC. Successes and failures in the control of infectious diseases in Brazil: social and environmental context, policies, interventions, and research needs. The Lancet. 2011;377:1877–1889. doi: 10.1016/S0140-6736(11)60202-X. [DOI] [PubMed] [Google Scholar]
Bhatt S, Gething PW, Brady OJ, Messina JP, Farlow AW, Moyes CL, Drake JM, Brownstein JS, Hoen AG, Sankoh O, Myers MF, George DB, Jaenisch T, Wint GR, Simmons CP, Scott TW, Farrar JJ, Hay SI. The global distribution and burden of dengue. Nature. 2013;496:504–507. doi: 10.1038/nature12060. [DOI] [PMC free article] [PubMed] [Google Scholar]
Brady OJ, Smith DL, Scott TW, Hay SI. Dengue disease outbreak definitions are implicitly variable. Epidemics. 2015;11:92–102. doi: 10.1016/j.epidem.2015.03.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
Campbell KM, Haldeman K, Lehnig C, Munayco CV, Halsey ES, Laguna-Torres VA, Yagui M, Morrison AC, Lin C-D, Scott TW. Weather regulates location, timing, and intensity of dengue virus transmission between humans and mosquitoes. PLOS Neglected Tropical Diseases. 2015;9:e11285. doi: 10.1371/journal.pntd.0003957. [DOI] [PMC free article] [PubMed] [Google Scholar]
Coelho CA, Cardoso DH, Firpo MA. Precipitation diagnostics of an exceptionally dry event in São Paulo, Brazil. Theoretical and Applied Climatology. 2015a:1–16. doi: 10.1007/s00704-015-1540-9. [DOI] [Google Scholar]
Coelho CAS, de Oliveira CP, Ambrizzi T, Reboita MS, Carpenedo CB, Campos JLPS, Tomaziello ACN, Pampuch LA, Custódio Maria de Souza, Dutra LMM, Da Rocha RP, Rehbein A. The 2014 southeast Brazil austral summer drought: regional scale mechanisms and teleconnections. Climate Dynamics. 2015b:1–16. doi: 10.1007/s00382-015-2800-1. [DOI] [Google Scholar]
Coelho CAS, Stephenson DB, Balmaseda M, Doblas-Reyes FJ, van Oldenborgh GJ. Toward an integrated seasonal forecasting system for South America. Journal of Climate. 2006;19:3704–3721. doi: 10.1175/JCLI3801.1. [DOI] [Google Scholar]
Degallier N, Favier C, Menkes C, Lengaigne M, Ramalho WM, Souza R, Servain J, Boulanger J-P. Toward an early warning system for dengue prevention: modeling climate impact on dengue transmission. Climatic Change. 2010;98:581–592. doi: 10.1007/s10584-009-9747-3. [DOI] [Google Scholar]
Duarte HHP, França EB. Qualidade dos dados da vigilância epidemiológica da dengue em Belo Horizonte, MG. Revista De Saúde Pública. 2006;40:134–142. doi: 10.1590/S0034-89102006000100021. [DOI] [PubMed] [Google Scholar]
ECDC Risk assessment Brazil 2014 FIFA World Cup, 12 June–13 July 2014 2014.
Fan Y, van den Dool H. A global monthly land surface air temperature analysis for 1948–present. Journal of Geophysical Research. 2008;113 doi: 10.1029/2007JD008470. [DOI] [Google Scholar]
Gallego V, Berberian G, Lloveras S, Verbanaz S, Chaves TS, Orduna T, Rodriguez-Morales AJ. The 2014 FIFA World Cup: communicable disease risks and advice for visitors to Brazil--a review from the Latin American Society for Travel Medicine (SLAMVI) Travel Medicine and Infectious Disease. 2014;12:208–218. doi: 10.1016/j.tmaid.2014.04.004. [DOI] [PubMed] [Google Scholar]
Guzman MG, Harris E. Dengue. The Lancet. 2015;385:453–465. doi: 10.1016/S0140-6736(14)60572-9. [DOI] [PubMed] [Google Scholar]
Hay S. Football fever could be a dose of dengue. Nature. 2013;503:439. doi: 10.1038/503439a. [DOI] [PubMed] [Google Scholar]
IBGE . Population estimates for the Brazilian municipalities from July 1st 2014. 2014. [Google Scholar]
Jolliffe IT, Stephenson DB. Forecast verification: a practitioner’s guide in atmospheric science. John Wiley & Sons; 2012. [Google Scholar]
Jupp TE, Lowe R, Coelho CAS, Stephenson DB. On the visualization, verification and recalibration of ternary probabilistic forecasts. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 2012;370:1100–1120. doi: 10.1098/rsta.2011.0350. [DOI] [PubMed] [Google Scholar]
Kraemer MUG, Sinka ME, Duda KA, Mylne AQN, Shearer FM, Barker CM, Moore CG, Carvalho RG, Coelho GE, Van Bortel W, Hendrickx G, Schaffner F, Elyazar IRF, Teng H-J, Brady OJ, Messina JP, Pigott DM, Scott TW, Smith DL, Wint GRW, Golding N, Hay SI. The global distribution of the arbovirus vectors Aedes aegypti and Ae. albopictus. eLife. 2015;4:e11285. doi: 10.7554/eLife.08347. [DOI] [PMC free article] [PubMed] [Google Scholar]
Lowe R, Bailey TC, Stephenson DB, Graham RJ, Coelho CAS, Carvalho MS, Barcellos C. Spatio-temporal modelling of climate-sensitive disease risk: towards an early warning system for dengue in Brazil. Computers & Geosciences. 2011;37:371–381. doi: 10.1016/j.cageo.2010.01.008. [DOI] [Google Scholar]
Lowe R, Bailey TC, Stephenson DB, Jupp TE, Graham RJ, Barcellos C, Carvalho MS. The development of an early warning system for climate-sensitive disease risk with a focus on dengue epidemics in southeast Brazil. Statistics in Medicine. 2013;32:864–883. doi: 10.1002/sim.5549. [DOI] [PubMed] [Google Scholar]
Lowe R, Barcellos C, Coelho CA, Bailey TC, Coelho GE, Graham R, Jupp T, Ramalho WM, Carvalho MS, Stephenson DB, Rodó X. Dengue outlook for the World Cup in Brazil: an early warning model framework driven by real-time seasonal climate forecasts. The Lancet. Infectious Diseases. 2014;14:619–626. doi: 10.1016/S1473-3099(14)70781-9. [DOI] [PubMed] [Google Scholar]
Lowe R, Carvalho MS, Coelho CAS, Barcellos C, Bailey TC, Stephenson DB, Rodó X. Interpretation of probabilistic forecasts of epidemics. The Lancet Infectious Diseases. 2015a;15:20. doi: 10.1016/S1473-3099(14)71031-X. [DOI] [PubMed] [Google Scholar]
Lowe R, Cazelles B, Paul R, Rodó X. Quantifying the added value of climate information in a spatio-temporal dengue model. Stochastic Environmental Research and Risk Assessment. 2015b doi: 10.1007/s00477-015-1053-1. [DOI] [Google Scholar]
Lowe R. Understanding the relative importance of global dengue risk factors. Transactions of the Royal Society of Tropical Medicine and Hygiene. 2015;109:607–608. doi: 10.1093/trstmh/trv068. [DOI] [PubMed] [Google Scholar]
Massad E, Wilder-Smith A, Ximenes R, Amaku M, Lopez LF, Coutinho FAB, Coelho GE, Silva Jr Jarbas Barbosa da, Struchiner CJ, Burattini MN. Risk of symptomatic dengue for foreign visitors to the 2014 FIFA World Cup in Brazil. Memórias Do Instituto Oswaldo Cruz. 2014;109:394–397. doi: 10.1590/0074-0276140133. [DOI] [PMC free article] [PubMed] [Google Scholar]
Matos V, Barcellos C. Relações entre turismo e saúde: abordagens metodológicas e propostas de ação. Revista Panamericana De Salud Pública. 2010;28:128–134. doi: 10.1590/S1020-49892010000800009. [DOI] [PubMed] [Google Scholar]
Meinke H, Stone RC. Seasonal and inter-annual climate forecasting: the new tool for increasing preparedness to climate variability and change in agricultural planning and operations. Climatic Change. 2005;70:221–253. doi: 10.1007/s10584-005-5948-6. [DOI] [Google Scholar]
Ministério da Saúde . Informe Epidemiológico da Dengue Janeiro a Junho de 2008. 2008. [Google Scholar]
Ministério da Saúde Boletim Epidemiológico. 2014;45 [Google Scholar]
Ministério da Saúde. Secretaria de vigilância em saúde . Guia De Vigilância Epidemiológica. 6 Ed. Ministério da Saúde: 2005. pp. 07–21. [Google Scholar]
Nunes MRT, Faria NR, de Vasconcelos JM, Golding N, Kraemer MUG, de Oliveira LF, Azevedo Raimunda do Socorro da Silva, da Silva DEA, da Silva EVP, da Silva SP, Carvalho VL, Coelho GE, Cruz ACR, Rodrigues SG, da Silva Gonçalves Vianez JL, Nunes BTD, Cardoso JF, Tesh RB, Hay SI, Pybus OG, da Costa Vasconcelos PF. Emergence and potential for spread of chikungunya virus in Brazil. BMC Medicine. 2015;13 doi: 10.1186/s12916-015-0348-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
Otto FEL, Haustein K, Uhe P, Coelho CAS, Aravequia JA, Almeida W, King A, Coughlan de Perez E, Wada Y, Jan van Oldenborgh G, Haarsma R, van Aalst M, Cullen H. Factors other than climate change, main drivers of 2014/15 water shortage in southeast Brazil. Bulletin of the American Meteorological Society. 2015;96:S35–S40. doi: 10.1175/BAMS-EEE_2014_ch8.1. [DOI] [Google Scholar]
PAHO . Indicadores básicos para a saúde no Brasil: conceitos e aplicações. Brasil: Ministério da Saúde; 2008. [Google Scholar]
SES . Programa De Vigilância De Dengue. Secretaria de estado de Saúde de São Paulo; 2010. [Google Scholar]
Stephenson DB. Use of the “odds ratio” for diagnosing forecast skill. Weather and Forecasting. 2000;15:221–232. doi: 10.1175/1520-0434(2000)015<0221:UOTORF>2.0.CO;2. [DOI] [Google Scholar]
Stewart-Ibarra AM, Lowe R. Climate and non-climate drivers of dengue epidemics in southern coastal Ecuador. The American Journal of Tropical Medicine and Hygiene. 2013;88:971–981. doi: 10.4269/ajtmh.12-0478. [DOI] [PMC free article] [PubMed] [Google Scholar]
Teixeira MG, Costa Maria da Conceição N., Barreto F, Barreto ML. Dengue: twenty-five years since reemergence in Brazil. Cadernos De Saúde Pública. 2009;25:S7–S18. doi: 10.1590/S0102-311X2009001300002. [DOI] [PubMed] [Google Scholar]
Thomson MC, Doblas-Reyes FJ, Mason SJ, Hagedorn R, Connor SJ, Phindela T, Morse AP, Palmer TN. Malaria early warnings based on seasonal climate forecasts from multi-model ensembles. Nature. 2006;439:576–579. doi: 10.1038/nature04503. [DOI] [PubMed] [Google Scholar]
van Panhuis WG, Hyun S, Blaney K, Marques ETA, Coelho GE, Siqueira JB, Tibshirani R, da Silva JB, Rosenfeld R. Risk of dengue for tourists and teams during the World Cup 2014 in Brazil. PLoS Neglected Tropical Diseases. 2014;8:e11285. doi: 10.1371/journal.pntd.0003063. [DOI] [PMC free article] [PubMed] [Google Scholar]
Wilson ME, Chen LH. Health risks among travelers to Brazil: implications for the 2014 FIFA World Cup and 2016 Olympic Games. Travel Medicine and Infectious Disease. 2014;12:205–207. doi: 10.1016/j.tmaid.2014.04.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
Zanluca C, Melo Vanessa Campos Andrade de, Mosimann ALP, Santos Glauco Igor Viana dos, Santos Claudia Nunes Duarte dos, Luz K. First report of autochthonous transmission of zika virus in Brazil. Memórias Do Instituto Oswaldo Cruz. 2015;110:569–572. doi: 10.1590/0074-02760150192. [DOI] [PMC free article] [PubMed] [Google Scholar]

eLife. 2016 Feb 24;5:e11285. doi: 10.7554/eLife.11285.013

Decision letter

Editor: Simon I Hay¹

In the interests of transparency, eLife includes the editorial decision letter and accompanying author responses. A lightly edited version of the letter sent to the authors after peer review is shown, indicating the most substantive concerns; minor comments are not usually included.

Thank you for submitting your work entitled "Evaluating probabilistic dengue risk forecasts from a prototype early warning system for Brazil" for consideration by eLife. Your article has been reviewed by two peer reviewers, and the evaluation has been overseen by Simon Hay (Reviewing Editor) and Prabhat Jha (Senior Editor). One of the two reviewers, Jane Messina, has agreed to share her identity.

The reviewers have discussed the reviews with one another and the Reviewing editor has drafted this decision to help you prepare a revised submission.

Summary:

This aims to evaluate the predictive accuracy of a dengue forecasting model trained on data 2000-2013 for real-time prediction in the year 2014. The manuscript is well written, detailed and comprehensive. However, while I appreciate the various challenges of the field, my main concern is that, based on the data presented, the forecasts do not seem to perform very favorably and it is only really the inclusion of the near-miss category (when there are only 3 categories) that results in attractive headline figures. If based solely on direct classifications (7 world cup sites are predicted correctly and 5 incorrectly and only 54% of all areas are correctly identified). A forecasting system is only useful when it correctly recommends changes in activities that would have happened anyway. It would be therefore nice to see these predictions compared to simple seasonal averages (as a null model) and over more than one year.

Essential revisions:

1) Performance measured against a wider range of continuous performance metrics. Hit, miss and near miss is not a sufficiently high benchmark.

2) Performance above and beyond simple seasonal profile should be the null model.

3) Finally, a much more nuanced and objective appraisal of the success of this warning system is required.

These concerns are expanded below.

It would be good to include a comparison with a null model based on historical averages. Useful predictions from a forecasting system are only likely to be those that recommend changes from the activities that would otherwise have taken place anyway (which are typically based on the normal dengue season) – and beyond that predictions that forecast higher than expected incidence, as recommending a reduction in vector control efforts is politically challenging). What are the model predictive performance statistics in areas where the seasonal forecast is wrong?

I would suggest reconsidering the scoring system for hit, near-hit and miss. With just three categories and with two of them counting positively to performance statistics, even a random prediction would perform well, particularly if you excluded areas in the south and north that have highly predictable dynamics. Surely a predicted low but observed high is worse than a predicted high and observed low?

What is the difference between pre and post processed dengue cases reports?– it would be nice to see some numbers on this- how close do they correlate and is this different in an outbreak year where the number of reports mean surveillance personnel are overwhelmed and there is a backlog

I appreciate that the risk thresholds are defined by the ministry of health and that there is already an extensive discussion on the limitations this place on the analysis, but it might be nice to discuss the impact of changing incidence over time on these thresholds (the number of cases and presumably incidence has increased significantly since 2000, but so to presumably has treatment and control capacity – does this mean outbreaks should be more frequent now than before?)

The manuscript is slightly longer than it needs to be in all sections with some repetition of methods of the current or previous Lowe et al. paper. Reference these details out where possible.

eLife. 2016 Feb 24;5:e11285. doi: 10.7554/eLife.11285.014

Author response

1) Performance measured against a wider range of continuous performance metrics. Hit, miss and near miss is not a sufficiently high benchmark.

We are very grateful to the reviewers for identifying the potential weakness of including a 'near hit' category to calculate evaluation metrics. We have now removed this from the analysis and included further performance metrics, including:

– Maps to show the probability of observing the correct category overall and for each observed category, to highlight the value of providing probabilistic information and identify areas where the model was more certain of observing the correct category.

– An assessment of the ability of the forecast model to correctly predict high dengue incidence rates across Brazil. The hit rate (the relative number of times an event was forecast when it occurred) and the false alarm rate (the relative number of times the event was forecast when it did not occur) are compared between the forecast model (the probability of exceeding the high risk category threshold) and a null model (seasonal averages, based on historical data for June 2000-2013). These performance metrics are shown for the 2014 event and for previous years.

– Time series of observed and predicted dengue incidence rates for June 2000-2014 for the twelve host microregions. Posterior predictive mean and 95% prediction (credible) intervals from the forecast model and sample mean and 95% prediction intervals from the null model are compared. Note, the forecast and the null model predictions are calculated in 'leave one year out' cross-validation mode from 2000-2013, i.e. by excluding the year for which the prediction is valid when estimating model parameters.

2) Performance above and beyond simple seasonal profile should be the null model.

As described above, we have compared the ability of the forecast model (the probability of exceeding the high risk threshold) to the null model (seasonal averages, based on historical data for June 2000-2013) to distinguish high dengue risk across Brazil. We have also provided time series of observed and predicted dengue incidence rates (using both the forecast and the null model) for all years in host microregions, June 2000-2014.

3) Finally, a much more nuanced and objective appraisal of the success of this warning system is required.

Performance metrics are now reported and discussed for both the dengue forecast model and the null model, to gauge an idea of the additional skill and value gained by using the reported early warning system.

These concerns are expanded below. It would be good to include a comparison with a null model based on historical averages. Useful predictions from a forecasting system are only likely to be those that recommend changes from the activities that would otherwise have taken place anyway (which are typically based on the normal dengue season) – and beyond that predictions that forecast higher than expected incidence, as recommending a reduction in vector control efforts is politically challenging). What are the model predictive performance statistics in areas where the seasonal forecast is wrong?

Thank you for this comment. We have now compared the ability of the forecast model and null model (seasonal averages, based on historical data for June 2000-2013) to distinguish high risk dengue. We have also provided time series of observed and predicted dengue incidence rates (using both the forecast and the null model) for all years, June 2000-2014.

We agree with the reviewers. We have removed the inclusion of a near hit category and have instead compared the ability of the forecast model and the null model to distinguish dengue incidence rates exceeding the high risk threshold. We also compare time series of observed and predicted dengue incidence rates for both the forecast model and the null model, to gauge an idea of the additional skill gained from the proposed early warning system. A discussion of the potential impact of false alarms (predicted high but observed low, i.e. type I error) and missed events (predicted low but observed high, i.e. type II error) is now included.

The following text has been added to the data section of the Materials and methods:

“Dengue notification records are considered a priority in the epidemiological surveillance system in Brazil. […]However, beyond this 30% of cases, identification of serotype is not carried out by laboratory tests, which hinders the understanding of transmission dynamics and population susceptibility level.”

Thank you for this interesting question. Yes, the incidence rate baseline has been increasing over the years (see Author response image 1). Since 1998, the overall incidence rate reached a plateau, situated between 100 and 300 cases per 100,000 inhabitants. However, this seems to be more a consequence of the spread of the disease in the country rather than an increase of incidence rates in cities. The number of cities reporting dengue cases increased since the late 1980's from a few hundred to almost 4000, i.e., 72% of the Brazilian territory (Barreto et al., 2011).

These processes have several consequences: 1) The emergence of large outbreaks in cities without prior transmission (Barcellos and Lowe, 2013); 2) An increase in the frequency of outbreaks with low amplitude in cities with favourable climatic and socioeconomic conditions (e.g., Fortaleza and Recife); 3) the occurrence of large outbreaks with low frequency in some large and favourable cities (e.g., Rio de Janeiro and Belo Horizonte). However, the frequency of outbreaks is regulated by population susceptibility to the viruses. As a result, the peaks tend to occur every 4 years; a necessary time frame for replacement of susceptible population or the arrival of a new type of virus.

We have added the following text to the Discussion to summarise this point:

“The dengue incidence rate baseline has been increasing over the years. […] For example, the number of cities reporting dengue cases increased since the late 1980's from a few hundred to almost 4000, reaching 72% of the Brazilian territory (Barreto et al., 2011).”

The manuscript is slightly longer than it needs to be in all sections with some repetition of methods of the current or previous Lowe et al. paper. Reference these details out where possible.

Repetition has been removed and details previously published have been referenced, where appropriate.

[bib1] Abubakar I, Gautret P, Brunette GW, Blumberg L, Johnson D, Poumerol G, Memish ZA, Barbeschi M, Khan AS. Global perspectives for prevention of infectious diseases associated with mass gatherings. The Lancet Infectious Diseases. 2012;12:66–74. doi: 10.1016/S1473-3099(11)70246-8. [DOI] [PubMed] [Google Scholar]

[bib2] Adler RF, Huffman GJ, Chang A, Ferraro R, Xie P-P, Janowiak J, Rudolf B, Schneider U, Curtis S, Bolvin D, Gruber A, Susskind J, Arkin P, Nelkin E. The version-2 global precipitation climatology project (GPCP) monthly precipitation analysis (1979–present) Journal of Hydrometeorology. 2003;4:1147–1167. doi: 10.1175/1525-7541(2003)004<1147:TVGPCP>2.0.CO;2. [DOI] [Google Scholar]

[bib3] Anyamba A, Chretien J-P, Small J, Tucker CJ, Formenty PB, Richardson JH, Britch SC, Schnabel DC, Erickson RL, Linthicum KJ. Prediction of a rift valley fever outbreak. Proceedings of the National Academy of Sciences of the United States of America. 2009;106:955–959. doi: 10.1073/pnas.0806490106. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib4] Barbosa JR, Barrado Jean Carlos dos Santos, Zara Ana Laura de Sene Amâncio, Siqueira JB. Avaliação da qualidade dos dados, valor preditivo positivo, oportunidade e representatividade do sistema de vigilância epidemiológica da dengue no Brasil (2005 a 2009) Epidemiologia e Serviços de Saúde. 2015;24:49–58. doi: 10.5123/S1679-49742015000100006. [DOI] [Google Scholar]

[bib5] Barcellos C, Lowe R. Dengue and the World Football Cup: a matter of timing. PLoS Neglected Tropical Diseases. 2014a;8:e11285. doi: 10.1371/journal.pntd.0003022. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib6] Barcellos C, Lowe R. Expansion of the dengue transmission area in Brazil: the role of climate and cities. Tropical Medicine & International Health. 2014b;19:159–168. doi: 10.1111/tmi.12227. [DOI] [PubMed] [Google Scholar]

[bib7] Barreto ML, Teixeira MG, Bastos FI, Ximenes RAA, Barata RB, Rodrigues LC. Successes and failures in the control of infectious diseases in Brazil: social and environmental context, policies, interventions, and research needs. The Lancet. 2011;377:1877–1889. doi: 10.1016/S0140-6736(11)60202-X. [DOI] [PubMed] [Google Scholar]

[bib8] Bhatt S, Gething PW, Brady OJ, Messina JP, Farlow AW, Moyes CL, Drake JM, Brownstein JS, Hoen AG, Sankoh O, Myers MF, George DB, Jaenisch T, Wint GR, Simmons CP, Scott TW, Farrar JJ, Hay SI. The global distribution and burden of dengue. Nature. 2013;496:504–507. doi: 10.1038/nature12060. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib9] Brady OJ, Smith DL, Scott TW, Hay SI. Dengue disease outbreak definitions are implicitly variable. Epidemics. 2015;11:92–102. doi: 10.1016/j.epidem.2015.03.002. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib10] Campbell KM, Haldeman K, Lehnig C, Munayco CV, Halsey ES, Laguna-Torres VA, Yagui M, Morrison AC, Lin C-D, Scott TW. Weather regulates location, timing, and intensity of dengue virus transmission between humans and mosquitoes. PLOS Neglected Tropical Diseases. 2015;9:e11285. doi: 10.1371/journal.pntd.0003957. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib11] Coelho CA, Cardoso DH, Firpo MA. Precipitation diagnostics of an exceptionally dry event in São Paulo, Brazil. Theoretical and Applied Climatology. 2015a:1–16. doi: 10.1007/s00704-015-1540-9. [DOI] [Google Scholar]

[bib12] Coelho CAS, de Oliveira CP, Ambrizzi T, Reboita MS, Carpenedo CB, Campos JLPS, Tomaziello ACN, Pampuch LA, Custódio Maria de Souza, Dutra LMM, Da Rocha RP, Rehbein A. The 2014 southeast Brazil austral summer drought: regional scale mechanisms and teleconnections. Climate Dynamics. 2015b:1–16. doi: 10.1007/s00382-015-2800-1. [DOI] [Google Scholar]

[bib13] Coelho CAS, Stephenson DB, Balmaseda M, Doblas-Reyes FJ, van Oldenborgh GJ. Toward an integrated seasonal forecasting system for South America. Journal of Climate. 2006;19:3704–3721. doi: 10.1175/JCLI3801.1. [DOI] [Google Scholar]

[bib14] Degallier N, Favier C, Menkes C, Lengaigne M, Ramalho WM, Souza R, Servain J, Boulanger J-P. Toward an early warning system for dengue prevention: modeling climate impact on dengue transmission. Climatic Change. 2010;98:581–592. doi: 10.1007/s10584-009-9747-3. [DOI] [Google Scholar]

[bib15] Duarte HHP, França EB. Qualidade dos dados da vigilância epidemiológica da dengue em Belo Horizonte, MG. Revista De Saúde Pública. 2006;40:134–142. doi: 10.1590/S0034-89102006000100021. [DOI] [PubMed] [Google Scholar]

[bib16] ECDC Risk assessment Brazil 2014 FIFA World Cup, 12 June–13 July 2014 2014.

[bib17] Fan Y, van den Dool H. A global monthly land surface air temperature analysis for 1948–present. Journal of Geophysical Research. 2008;113 doi: 10.1029/2007JD008470. [DOI] [Google Scholar]

[bib18] Gallego V, Berberian G, Lloveras S, Verbanaz S, Chaves TS, Orduna T, Rodriguez-Morales AJ. The 2014 FIFA World Cup: communicable disease risks and advice for visitors to Brazil--a review from the Latin American Society for Travel Medicine (SLAMVI) Travel Medicine and Infectious Disease. 2014;12:208–218. doi: 10.1016/j.tmaid.2014.04.004. [DOI] [PubMed] [Google Scholar]

[bib19] Guzman MG, Harris E. Dengue. The Lancet. 2015;385:453–465. doi: 10.1016/S0140-6736(14)60572-9. [DOI] [PubMed] [Google Scholar]

[bib20] Hay S. Football fever could be a dose of dengue. Nature. 2013;503:439. doi: 10.1038/503439a. [DOI] [PubMed] [Google Scholar]

[bib21] IBGE . Population estimates for the Brazilian municipalities from July 1st 2014. 2014. [Google Scholar]

[bib22] Jolliffe IT, Stephenson DB. Forecast verification: a practitioner’s guide in atmospheric science. John Wiley & Sons; 2012. [Google Scholar]

[bib23] Jupp TE, Lowe R, Coelho CAS, Stephenson DB. On the visualization, verification and recalibration of ternary probabilistic forecasts. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 2012;370:1100–1120. doi: 10.1098/rsta.2011.0350. [DOI] [PubMed] [Google Scholar]

[bib24] Kraemer MUG, Sinka ME, Duda KA, Mylne AQN, Shearer FM, Barker CM, Moore CG, Carvalho RG, Coelho GE, Van Bortel W, Hendrickx G, Schaffner F, Elyazar IRF, Teng H-J, Brady OJ, Messina JP, Pigott DM, Scott TW, Smith DL, Wint GRW, Golding N, Hay SI. The global distribution of the arbovirus vectors Aedes aegypti and Ae. albopictus. eLife. 2015;4:e11285. doi: 10.7554/eLife.08347. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib25] Lowe R, Bailey TC, Stephenson DB, Graham RJ, Coelho CAS, Carvalho MS, Barcellos C. Spatio-temporal modelling of climate-sensitive disease risk: towards an early warning system for dengue in Brazil. Computers & Geosciences. 2011;37:371–381. doi: 10.1016/j.cageo.2010.01.008. [DOI] [Google Scholar]

[bib26] Lowe R, Bailey TC, Stephenson DB, Jupp TE, Graham RJ, Barcellos C, Carvalho MS. The development of an early warning system for climate-sensitive disease risk with a focus on dengue epidemics in southeast Brazil. Statistics in Medicine. 2013;32:864–883. doi: 10.1002/sim.5549. [DOI] [PubMed] [Google Scholar]

[bib27] Lowe R, Barcellos C, Coelho CA, Bailey TC, Coelho GE, Graham R, Jupp T, Ramalho WM, Carvalho MS, Stephenson DB, Rodó X. Dengue outlook for the World Cup in Brazil: an early warning model framework driven by real-time seasonal climate forecasts. The Lancet. Infectious Diseases. 2014;14:619–626. doi: 10.1016/S1473-3099(14)70781-9. [DOI] [PubMed] [Google Scholar]

[bib28] Lowe R, Carvalho MS, Coelho CAS, Barcellos C, Bailey TC, Stephenson DB, Rodó X. Interpretation of probabilistic forecasts of epidemics. The Lancet Infectious Diseases. 2015a;15:20. doi: 10.1016/S1473-3099(14)71031-X. [DOI] [PubMed] [Google Scholar]

[bib29] Lowe R, Cazelles B, Paul R, Rodó X. Quantifying the added value of climate information in a spatio-temporal dengue model. Stochastic Environmental Research and Risk Assessment. 2015b doi: 10.1007/s00477-015-1053-1. [DOI] [Google Scholar]

[bib30] Lowe R. Understanding the relative importance of global dengue risk factors. Transactions of the Royal Society of Tropical Medicine and Hygiene. 2015;109:607–608. doi: 10.1093/trstmh/trv068. [DOI] [PubMed] [Google Scholar]

[bib31] Massad E, Wilder-Smith A, Ximenes R, Amaku M, Lopez LF, Coutinho FAB, Coelho GE, Silva Jr Jarbas Barbosa da, Struchiner CJ, Burattini MN. Risk of symptomatic dengue for foreign visitors to the 2014 FIFA World Cup in Brazil. Memórias Do Instituto Oswaldo Cruz. 2014;109:394–397. doi: 10.1590/0074-0276140133. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib32] Matos V, Barcellos C. Relações entre turismo e saúde: abordagens metodológicas e propostas de ação. Revista Panamericana De Salud Pública. 2010;28:128–134. doi: 10.1590/S1020-49892010000800009. [DOI] [PubMed] [Google Scholar]

[bib33] Meinke H, Stone RC. Seasonal and inter-annual climate forecasting: the new tool for increasing preparedness to climate variability and change in agricultural planning and operations. Climatic Change. 2005;70:221–253. doi: 10.1007/s10584-005-5948-6. [DOI] [Google Scholar]

[bib34] Ministério da Saúde . Informe Epidemiológico da Dengue Janeiro a Junho de 2008. 2008. [Google Scholar]

[bib35] Ministério da Saúde Boletim Epidemiológico. 2014;45 [Google Scholar]

[bib36] Ministério da Saúde. Secretaria de vigilância em saúde . Guia De Vigilância Epidemiológica. 6 Ed. Ministério da Saúde: 2005. pp. 07–21. [Google Scholar]

[bib37] Nunes MRT, Faria NR, de Vasconcelos JM, Golding N, Kraemer MUG, de Oliveira LF, Azevedo Raimunda do Socorro da Silva, da Silva DEA, da Silva EVP, da Silva SP, Carvalho VL, Coelho GE, Cruz ACR, Rodrigues SG, da Silva Gonçalves Vianez JL, Nunes BTD, Cardoso JF, Tesh RB, Hay SI, Pybus OG, da Costa Vasconcelos PF. Emergence and potential for spread of chikungunya virus in Brazil. BMC Medicine. 2015;13 doi: 10.1186/s12916-015-0348-x. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib38] Otto FEL, Haustein K, Uhe P, Coelho CAS, Aravequia JA, Almeida W, King A, Coughlan de Perez E, Wada Y, Jan van Oldenborgh G, Haarsma R, van Aalst M, Cullen H. Factors other than climate change, main drivers of 2014/15 water shortage in southeast Brazil. Bulletin of the American Meteorological Society. 2015;96:S35–S40. doi: 10.1175/BAMS-EEE_2014_ch8.1. [DOI] [Google Scholar]

[bib39] PAHO . Indicadores básicos para a saúde no Brasil: conceitos e aplicações. Brasil: Ministério da Saúde; 2008. [Google Scholar]

[bib40] SES . Programa De Vigilância De Dengue. Secretaria de estado de Saúde de São Paulo; 2010. [Google Scholar]

[bib41] Stephenson DB. Use of the “odds ratio” for diagnosing forecast skill. Weather and Forecasting. 2000;15:221–232. doi: 10.1175/1520-0434(2000)015<0221:UOTORF>2.0.CO;2. [DOI] [Google Scholar]

[bib42] Stewart-Ibarra AM, Lowe R. Climate and non-climate drivers of dengue epidemics in southern coastal Ecuador. The American Journal of Tropical Medicine and Hygiene. 2013;88:971–981. doi: 10.4269/ajtmh.12-0478. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib43] Teixeira MG, Costa Maria da Conceição N., Barreto F, Barreto ML. Dengue: twenty-five years since reemergence in Brazil. Cadernos De Saúde Pública. 2009;25:S7–S18. doi: 10.1590/S0102-311X2009001300002. [DOI] [PubMed] [Google Scholar]

[bib44] Thomson MC, Doblas-Reyes FJ, Mason SJ, Hagedorn R, Connor SJ, Phindela T, Morse AP, Palmer TN. Malaria early warnings based on seasonal climate forecasts from multi-model ensembles. Nature. 2006;439:576–579. doi: 10.1038/nature04503. [DOI] [PubMed] [Google Scholar]

[bib45] van Panhuis WG, Hyun S, Blaney K, Marques ETA, Coelho GE, Siqueira JB, Tibshirani R, da Silva JB, Rosenfeld R. Risk of dengue for tourists and teams during the World Cup 2014 in Brazil. PLoS Neglected Tropical Diseases. 2014;8:e11285. doi: 10.1371/journal.pntd.0003063. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib46] Wilson ME, Chen LH. Health risks among travelers to Brazil: implications for the 2014 FIFA World Cup and 2016 Olympic Games. Travel Medicine and Infectious Disease. 2014;12:205–207. doi: 10.1016/j.tmaid.2014.04.001. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib47] Zanluca C, Melo Vanessa Campos Andrade de, Mosimann ALP, Santos Glauco Igor Viana dos, Santos Claudia Nunes Duarte dos, Luz K. First report of autochthonous transmission of zika virus in Brazil. Memórias Do Instituto Oswaldo Cruz. 2015;110:569–572. doi: 10.1590/0074-02760150192. [DOI] [PMC free article] [PubMed] [Google Scholar]

PERMALINK

Evaluating probabilistic dengue risk forecasts from a prototype early warning system for Brazil

Rachel Lowe

Caio AS Coelho

Christovam Barcellos

Marilia Sá Carvalho

Rafael De Castro Catão

Giovanini E Coelho

Walter Massa Ramalho

Trevor C Bailey

David B Stephenson

Xavier Rodó

Roles

Abstract

eLife digest

Introduction

Results

Comparison of probability forecast and observed dengue risk categories

Table 1.

Figure 1. Predictive distributions and observed DIR for June 2014 for host microregions.

Figure 2. Probabilistic dengue forecast and observed dengue incidence rate categories for Brazil, June 2014.

Figure 3. Forecast probability of observed DIR categories for June 2014.

Figure 4. Forecast probability of observed DIR in the low, medium and high category for June 2014.

Comparison of forecast model framework to a null model

Table 2.

Figure 5. Hit rate and false alarm rate for predicting dengue in the high risk category for June 2000–2014 using the forecast model and null model.

Figure 6. Time series of observed and predicted DIR for June 2000–2014 for host microregions.

Discussion

Conclusion

Materials and methods

Data

Dengue forecast formulation and translation

Dengue forecast evaluation

Table 3.

Acknowledgements

Funding Statement

Funding Information

Additional information

Competing interests

Author contributions

References

Decision letter

Roles

Author response

Author response image 1. Dengue incidence and number of municipalities reporting dengue cases in Brazil, 1985-2010 (Source: Barreto et al., 2011).

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases