Validation analysis of Global Health Security Index (GHSI) scores 2019

Abstract

Introduction

The COVID-19 pandemic powerfully demonstrates the consequences of biothreats. Countries will want to know how to better prepare for future events. The Global Health Security Index (GHSI) is a broad, independent assessment of 195 countries’ preparedness for biothreats that may aid this endeavour. However, to be useful, the GHSI’s external validity must be demonstrated. We aimed to validate the GHSI against a range of external metrics to assess how it could be utilised by countries.

Methods

Global aggregate communicable disease outcomes were correlated with GHSI scores and linear regression models were examined to determine associations while controlling for a number of global macroindices. GHSI scores for countries previously exposed to severe acute respiratory syndrome (SARS), Middle East respiratory syndrome and Ebola and recipients of US Global Health Security Agenda (GHSA) investment were compared with matched control countries. Possible content omissions in light of the progressing COVID-19 pandemic were assessed.

Results

GHSI scores for countries had strong criterion validity against the Joint External Evaluation ReadyScore (rho=0.82, p<0.0001), and moderate external validity against deaths from communicable diseases (−0.56, p<0.0001). GHSI scores were associated with reduced deaths from communicable diseases (F(3, 172)=22.75, p<0.0001). The proportion of deaths from communicable diseases decreased 4.8% per 10-point rise in GHSI. Recipient countries of the GHSA (n=31) and SARS-affected countries (n=26), had GHSI scores 6.0 (p=0.0011) and 8.2 (p=0.0010) points higher than matched controls, respectively. Biosecurity and biosafety appear weak globally including in high-income countries, and health systems, particularly in Africa, are not prepared. Notably, the GHSI does not account for all factors important for health security.

Conclusion

The GHSI shows promise as a valid tool to guide action on biosafety, biosecurity and systems preparedness. However, countries need to look beyond existing metrics to other factors moderating the impact of future pandemics and other biothreats. Consideration of anthropogenic and large catastrophic scenarios is also needed.

Key questions.

What is already known?

• The report on the Global Health Security Index (GHSI) was published in October 2019 to benchmark preparedness for biothreats and showed that the world is not prepared.

• A number of global organisations have described health security weaknesses and called for action, including the United Nations, World Bank, WHO and Global Preparedness Monitoring Board.

• Few studies have attempted to validate metrics of global health security and no studies have linked GHSI scores to global health security investments or previous experience with outbreaks, and there is little published critical appraisal of the GHSI to date.

What are the new findings?

• This study determined the association of GHSI scores with outcomes of communicable diseases, therefore providing some degree of concurrent and external validation of the tool.

• Results reveal associations between GHSI scores and general education, Global Health Security Agenda investment and previous experience with severe acute respiratory syndrome.

• The study highlights the breadth of the GHSI in comparison to Joint External Evaluations (JEE), but reveals some factors relevant to mitigating high-consequence biothreats yet to be addressed by the GHSI.

What do the new findings imply?

• The GHSI is the most comprehensive tool released to date for benchmarking countries’ capability to prevent and manage communicable disease and exhibits some validity against related tools and common disease outcomes.

• Given the clear lack of health security capability, countries (especially those who have not completed a JEE) should look to the GHSI, which goes beyond the requirements of the International Health Regulations and JEE, for actionable steps they can take to reduce the impact of future biothreats.

• However, action also needs to move beyond the GHSI to successfully anticipate and mitigate potentially very high consequence and as yet unprecedented events.

Introduction

International efforts to mitigate biothreats are grounded in the 2005 International Health Regulations (IHR), which require countries to prevent, detect and respond to emerging threats.¹ Nations self-report their IHR compliance annually through a monitoring tool (International Health Regulations Monitoring Tool, IHRMT) developed by the WHO and can voluntarily undertake a Joint External Evaluation (JEE) to evaluate their level of compliance. The JEE is a multisectoral process and consists of a country self-assessment, which is shared with an external team of WHO and non-WHO experts, who then visit and make a week-long independent assessment.²

The Global Health Security Agenda (GHSA) was established in 2014 to stimulate better IHR compliance and improve epidemic detection, preparedness and response. A number of countries have additional programmes focused on global health security capabilities, such as Public Health England’s IHR Strengthening Project. However, reports from the United Nations, World Bank, Global Preparedness Monitoring Board and US National Academy of Sciences have criticised ongoing lack of preparedness globally.^3–6 GHSA investment has built capacity equivalent to raising JEE scores in some countries,⁷ and was renewed to 2024.^{8 9}

Given the large self-assessment component and the facilitation of JEE by discussions, the JEE process may be subject to self-assessment bias. There may also be selection bias in which countries have so far requested a JEE. As of 31 July 2020, 110 countries had completed such an assessment.¹⁰ Furthermore, many factors contribute to the emergence and mitigation of biothreats beyond those measured by the JEE. Prior to 2019, there existed no objective, sufficiently broad, universal metric assessed independently. This led to the development of the Global Health Security Index (GHSI) by the Nuclear Threat Initiative, Johns Hopkins Centre for Health Security and the Economist Intelligence Unit.^11–13 A similar metric is the Epidemic Preparedness Index tool, also published in 2019 but with fewer countries and indicators included.¹⁴ This present paper focuses on the GHSI.

The GHSI is scored by examining published and publicly available evidence across six categories (prevention, detection, rapid response, health system, commitments to international norms and risk environment) with the intention of encouraging nations to document and publicise their preparations.¹² The GHSI points explicitly to very high-consequence events, robust health systems, commitments to international norms, the risk environment and is therefore a much broader assessment than the JEE. By requiring published evidence, the GHSI aimed to ensure there is no suspicion that nations are not prepared and shows all nations their preparedness gaps. Like the JEE, the GHSI may be prone to certain biases, such as overlooking measures that are in place but not documented. However, to date there is no gold standard in measuring global health security and the JEE and GHSI approaches are both likely to be limited and incomplete.

Commentary on the GHSI generally laments weak preparedness,^{11 15} even among relatively rich countries.¹⁶ New Zealand authors identified the nation’s low scores and highlight the implications for Pacific regional health security.¹⁷ A letter has critiqued the UK and US responses to COVID-19, and calls into question the value of GHSI if these are the top scoring nations.¹⁸ Another brief commentary has critiqued the focus of the GHSI, its reliance on published data only and questions whether the GHSI offers any added value.¹⁹ However, there is as yet no in-depth critical appraisal or external validation of the GHSI tool in the literature published to date to our knowledge.

The IHRMT, JEE and GHSI are only useful if they have reasonable validity, if they cover the relevant domains, if scores bear some relationship to experience and actual health outcomes, if action can improve scores and if countries then implement what they recommend. Few attempts to validate the IHRMT and JEE have been made,^{20 21} and the validity of IHR compliance and JEE evaluations has been called into question.²² Therefore, the validity of the GHSI needs to be assessed.

Assessing the validity of an index requires several evaluations. These include the face validity (does it look right?), the content validity (does it include all the items it should?), the construct validity (do scores have the expected relationships with other variables?), criterion validity (do scores align with other measures of the same thing?), concurrent or predictive validity (do scores map to outcomes of interest?), as well as internal consistency and test–retest reliability.²³

After the COVID-19 pandemic experience in 2020, countries will be looking for actionable steps to find weaknesses, mitigate risk and where to direct efforts to reduce any future impact of COVID-19 and other biothreats. The GHSI is the most comprehensive tool with the aim of supporting these tasks. Therefore, it was our aim to examine the content, construct, criterion and concurrent validity of the GHSI and therefore provide countries with some account of its relevance and limitations. In evaluating the validity of the GHSI we expected that GHSI scores should be positively correlated with existing measures of health security and negatively correlated with harm from communicable diseases. On the basis of our validation analyses, along with our interpretation of the GHSI scores as published, we explore how the GHSI might be used by countries.

Methods

We evaluated the criterion validity of the GHSI by correlating GHSI with the JEE ReadyScore (as explained below). We evaluated the construct validity by comparing the GHSI scores of countries receiving US GHSA investments to those that did not, as well as the GHSI scores of countries previously exposed to severe acute respiratory syndrome (SARS), Middle East respiratory syndrome (MERS) and Ebola versus those that were not. We evaluated the concurrent validity by correlating aggregate measures of communicable disease outcome with GHSI scores. Finally, we evaluated the content validity by considering whether responses to the COVID-19 pandemic suggest items the GHSI has overlooked.

Criterion validation of GHSI was performed by calculating correlations between GHSI overall score and the JEE ReadyScore. The JEE ReadyScore is a single summary metric of JEE results and is derived as the average of the scores for the 19 JEE domains expressed as a score from 0 to 100.²⁴ Given that both metrics can be interpreted as global measures of health security, we therefore expected GHSI scores to correlate with ReadyScore.

Concurrent validation was performed by calculating correlations between GHSI and the proportion of deaths in each country due to aggregate measures of communicable disease. We obtained data for the percentage of the population in each country dying from communicable diseases.²⁵ Given that the dataset for this metric included nutritional diseases and maternal and neonatal deaths, we also obtained more focused data for an aggregate of deaths per capita per annum from five relevant communicable diseases as reported in data from the Global Burden of Disease Study (diarrhoeal disease, HIV, lower respiratory tract infection, meningitis and tuberculosis).²⁶

Linear regression was performed to evaluate associations between GHSI as the independent variable and communicable disease deaths as the dependent variable. In order to control for potential major confounders in our analyses, we obtained data describing countries’ gross domestic product (GDP) per capita,²⁷ percentage of GDP spent on health,²⁸ population size,²⁹ political stability (as measured by the World Bank’s political stability and absence of violence/terrorism index)³⁰ and education (as measured by the proportion of the population over 25 years of age that completed secondary school).³¹ To determine correlations of these macroindices with GHSI, Spearman’s rho and Pearson’s r were calculated. We included in the multiple regressions, variables showing statistically significant correlations with GHSI scores. We also determined the strength of association of the various macroindices with GHSI scores by multiple linear regression.

It is unclear whether health security is higher in countries receiving certain health security investments. Therefore, the association of health security investments and GHSI scores was determined by comparing the GHSI scores of the recipient nations of US GHSA aid during 2014–2019 with the scores of matched control countries. The USA was chosen for parsimony of analysis. Although there are many contributing countries to the GHSA (and other) programmes, the US investment was by far the largest and US GHSA aid has gone to 31 countries, as reported by the US Centers for Disease Control and Prevention.⁷ Country control matching was done within WHO region by matching GHSA recipient countries to the country with the closest GDP per capita that was not a recipient of US GHSA aid. GDP per capita was chosen given that the GHSI report authors describe a moderate correlation between GHSI and GDP per capita. A list of all countries and controls is presented in online supplemental table S5.

It might be supposed that countries can learn from previous epidemics and therefore those countries that were exposed to some previous epidemic might have improved health security capability. It has been specifically suggested that countries exposed to the SARS pandemic in 2003 have managed the COVID-19 pandemic better than other countries.³² We assumed that some institutional memory exists and chose SARS as a previous major biothreat that affected a sufficiently large number, though not all, countries. If GHSI is a valid measure of health security, we expected scores would be higher in countries that had cases of SARS in 2003 compared with matched controls. For this analysis countries were matched by GDP per capita alone, rather than within WHO region, due to difficulty matching the USA and Canada to countries with near GDP per capita within WHO region. SARS exposure was obtained from the WHO summary of probable SARS cases.³³ We performed the same analysis for countries that experienced MERS cases³⁴ and those having Ebola outbreaks.³⁵

Content validity must be assessed by expert inspection of the items in an index. We examined the WHO COVID-19 situation reports and sought examples of responses to COVID-19 by high and low-scoring GHSI countries in order to determine whether there appeared to be factors important for health security that are not adequately captured by the GHSI. We report on a number of such factors.

Correlations and multiple regressions were performed with R (V.3.1.0). Differences between means were calculated using two-tailed paired t-tests in Microsoft Excel (V.16.37).

Funding and patient/public involvement

This study was funded by the Strategic Priorities Fund and the Open Philanthropy Project. These funders had no input into study design, collection, analysis, or interpretation of data, written outputs or the decision to submit for publication. Patients or the public were not involved in the design, or conduct, or reporting or dissemination plans of our research.

Results

Macroindices and GHSI scores

For reference, the mean GHSI scores by WHO region are reproduced in online supplemental table S2. Correlations between macroindices and GHSI scores are displayed in table 1. There is a very strong correlation between GHSI and ReadyScore. The proportion of the population that completed upper secondary school shows a moderate but highly significant correlation with GHSI score. This correlation is stronger than that for GDP, population, health spending and political stability, although all showed statistically significant correlations and were included in regression analyses. In multiple linear regression to examine any relationship between GHSI and the other macroindices (excluding ReadyScore), a significant regression equation was found (F(5,149)=29.51, p<0.0001), with an R² of 0.50. The GHSI increased 1.6 points for each 10% rise in secondary school completions, three points for each US$10 000 rise in GDP per capita, and 0.8 points for each 1% extra proportion of GDP spent on health.

Table 1.

Spearman correlations (p value) between GHSI categories and some global macroindicators

	ReadyScore (average of 19 JEE domains)	Education (population over 25 years that completed upper secondary)	GDP per capita (US$)	Population size	Health spend (% of GDP)	Political stability and absence of violence/terrorism index
GHSI total	0.82 (<0.0001)	0.57 (<0.0001)	0.56 (<0.0001)	0.36 (<0.0001)	0.32 (<0.0001)	0.25 (0.00041)
Prevent	0.79 (<0.0001)	0.56 (<0.0001)	0.52 (<0.0001)	0.40 (<0.0001)	0.35 (<0.0001)	0.18 (0.012)
Detect	0.66 (<0.0001)	0.41 (<0.0001)	0.37 (<0.0001)	0.42 (<0.0001)	0.25 (0.0006)	0.05 (0.50)
Respond	0.67 (<0.0001)	0.37 (<0.0001)	0.40 (<0.0001)	0.35 (<0.0001)	0.28 (0.00012)	0.17 (0.017)
Health system	0.80 (<0.0001)	0.56 (<0.0001)	0.56 (<0.0001)	0.34 (<0.0001)	0.33 (<0.0001)	0.22 (0.0019)
Commitments	0.41 (<0.0001)	0.30 (<0.0001)	0.24 (0.0013)	0.35 (<0.0001)	0.19 (0.0098)	0.10 (0.17)
Risk environment	0.81 (<0.0001)	0.64 (<0.0001)	0.91 (<0.0001)	−0.24 (0.001)	0.32 (<0.0001)	0.78 (<0.0001)

Spearman’s rho reported, note that Pearson’s r differed substantially from Spearman’s rho only for population size, where correlation with GHSI overall score was r=0.14, p<0.05.

GDP, gross domestic product; GHSI, Global Health Security Index; JEE, Joint External Evaluation.

Communicable diseases and GHSI scores

Table 2 shows the correlations between ReadyScore as well as GHSI scores and two measures of communicable disease deaths. We found a strong negative correlation between ReadyScore and communicable disease deaths and a moderate but significant negative correlation between GHSI overall score and communicable disease deaths.

Table 2.

Spearman correlation (p value) between ReadyScore or GHSI metrics and indicators of disease outcome or management

Score/score component	Proportion of deaths from communicable, maternal, neonatal and nutritional diseases (2017)	Deaths from five specified communicable diseases per capita per annum (2017)
ReadyScore (n=93 countries)	−0.78 (<0.0001)	−0.73 (<0.0001)
GHSI overall (n=195 countries)	−0.56 (<0.0001)	−0.43 (<0.0001)
Prevent	−0.57 (<0.0001)	−0.44 (<0.0001)
Detect	−0.39 (<0.0001)	−0.30 (<0.0001)
Respond	−0.37 (<0.0001)	−0.29 (<0.0001)
Health system	−0.60 (<0.0001)	−0.49 (<0.0001)
Commitments	−0.24 (0.00084)	−0.08 (0.31)
Risk environment	−0.71 (<0.0001)	−0.56 (<0.0001)

ReadyScore is the average across the 19 JEE categories; ‘Proportion of deaths from communicable, maternal, neonatal and nutritional diseases’ is the proportion of annual deaths by country due to communicable diseases derived from 2017 Global Burden of Disease study data, this category includes maternal and neonatal deaths and deaths due to nutritional disease. ‘Deaths from five specified communicable diseases’ is a composite of: diarrhoeal disease, HIV, lower respiratory infection, meningitis and tuberculosis, expressed as cases per capita per annum.

GHSI, Global Health Security Index;JEE, Joint External Evaluation.

Multiple linear regression to explore the association between GHSI and communicable disease deaths, while controlling for GDP per capita and population size, revealed a significant regression equation (F(3, 172)=22.75, p<0.0001), with an R² of 0.28. The proportion of the population dying from communicable diseases decreased 4.8% for each 10-point rise in GHSI (figure 1). GDP per capita was also associated with communicable deaths, but this association was no longer significant when controlling for education, political risk and health spending. In the regression model including all five macroindices, only education and political stability had statistically significant associations with communicable disease deaths.

Figure 1.

Relationship between proportion of the population dying from communicable diseases and countries’ GHSI overall score. Communicable deaths are derived from Global Burden of Disease Study and include maternal, neonatal and nutritional deaths. GHSI, Global Health Security Index.

GHSA investment and GHSI scores

The GHSI scores of individual countries receiving US GHSA aid during 2014–2019, along with individual matched controls, are displayed in online supplemental table S5. Mean scores and between-group differences (cases vs controls) are displayed in table 3. The GHSI scores of countries receiving aid was six points higher than matched controls (p=0.0011). The GHSI category ‘Detect’ showed the largest difference (+9.9, p=0.0062) and ‘Risk Environment’ the smallest (+1.1, p=0.52).

Table 3.

Mean (SD) GHSI scores for countries receiving US GHSA investment versus GHSI scores of matched controls not receiving US GHSA investment

	GHSI total	Prevention	Detection and reporting	Rapid response	Health system	Compliance with international norms	Risk environment
US GHSA recipient countries	40.6 (10.3)	34.9 (13.5)	46.7 (15.7)	39.6 (13.2)	24.2 (14.7)	55.0 (9.4)	44.0 (10.9)
Non-recipient countries	34.6 (8.3)	29.1 (10.5)	36.8 (16.4)	34.9 (10.1)	20.4 (11.1)	45.8 (8.5)	42.9 (11.4)
Difference in GHSI score	+6.0	+5.8	+9.9	+4.7	+3.8	+9.3	+1.1
P value	0.0011	0.0252	0.0062	0.0477	0.1125	0.0001	0.5207

Countries were matched within WHO region by next closest GDP per capita, with small island nations of <1 million population excluded.

GDP, gross domestic product; GHSA, Global Health Security Agenda; GHSI, Global Health Security Index.;

SARS, MERS and Ebola exposure and GHSI scores

Countries that had experienced one or more SARS cases in 2003 (n=26) had a GHSI 8.2 points higher (p=0.0010) than matched controls, and those with more than one case of SARS (n=18) had a GHSI 13.8 points higher (p=0.0002), see table 4. The greatest difference was in ‘Detect’, where countries with more than one SARS case scored 26.0 points higher (p=0.0003). ‘Risk Environment’ had the smallest difference (+5.4, p=0.0089).

Table 4.

Mean (SD) GHSI scores of countries exposed to one or more SARS cases in 2003 and matched control countries with no SARS cases

	GHSI total	Prevention	Detection and reporting	Rapid response	Health system	Compliance with international norms	Risk environment
Countries that had any SARS cases (n=26)	60.5 (11.7)	56.0 (13.6)	70.7 (19.1)	58.5 (13.8)	48.6 (13.2)	60.5 (12.0)	69.6 (11.7)
Matched countries that had no SARS cases	52.3 (12.0)	48.5 (14.3)	56.3 (16.7)	47.6 (12.9)	42.0 (16.7)	56.8 (11.5)	66.1 (13.8)
Difference in GHSI score	+8.2	+7.5	+14.5	+10.9	+6.6	+3.7	+3.5
P value	0.0010	0.0122	0.0025	0.0016	0.0281	0.2454	0.0205
Countries that had more than one SARS case (n=18)	63.1 (12.0)	58.5 (15.1)	76.5 (15.8)	60.1 (14.2)	50.7 (13.8)	63.4 (12.6)	69.5 (12.1)
Matched countries that had no SARS cases	49.3 (13.0)	46.8 (15.3)	50.5 (22.0)	47.3 (13.3)	39.2 (17.7)	51.6 (10.0)	64.1 (14.7)
Difference in GHSI score	+13.8	+11.7	+26.0	+12.8	+11.5	+11.7	+5.4
P value	0.0002	0.0070	0.0003	0.0003	0.0075	0.0073	0.0089

Countries were matched globally by GDP per capita.

GDP, gross domestic product; GHSI, Global Health Security Index; SARS, severe acute respiratory syndrome.

Countries that experienced MERS cases (n=27) had a GHSI 5.6 points higher (51.8 vs 46.2, p=0.055) than matched controls, with scores for ‘Prevent’ (+6.2, p=0.082), ‘Detect’ (+8.5, p=0.043), ‘Respond’ (+9.2, p=0.023) and ‘Health System’ (+5.8, p=0.088) all higher in MERS-exposed countries than controls. However, only ‘Detect’ and ‘Respond’ reached statistical significance at the p<0.05 level. Results are displayed in online supplemental tables S7 and S8.

Countries that experienced Ebola outbreaks in the period 1976–2020 (n=13) had similar overall GHSI scores to matched controls (33.6 vs 32.4, difference +1.2, p=0.69). Category differences were not statistically significant but were greatest in ‘Respond’ (+5.3), ‘Norms’ (+5.2) and ‘Risk Environment’ (−3.9), see online supplemental tables S9 and S10.

With regard to content validity, we found that the WHO situation reports on the COVID-19 pandemic up to 31 July 2020 indicated that a number of countries with high GHSI scores have had large numbers of cases and deaths from COVID-19,³⁶ while some others with modest GHSI scores appear to have successfully contained or suppressed pandemic spread. We found a number of items in the content of the GHSI that might account for this finding. For example, GHSI item-level scores show that the UK scores half-marks or less for: number of field epidemiologists, number of doctors, nurses and hospital beds per capita, access to care, ability to receive foreign health personnel, evidence of a plan for vulnerable populations or plan to prioritise healthcare for health workers, plan for personal protective equipment supply issues, domestic spending on health per capita, public–private healthcare communication, public confidence in government, international tensions. Additionally, the US exhibits low scores for political and security risks, socioeconomic resilience and public confidence in government, all of which can impact pandemic response. However, given the COVID-19 responses, the content of the GHSI may not fully capture health security and we explore this further in the Discussion section below.

Discussion

Individual GHSI scores for countries (and the average global score of 40.2/100) led the GHSI authors to conclude that collective international preparedness and health security are weak and that, political, socioeconomic and environmental vulnerabilities can amplify these deficiencies.¹²

Our results suggest that the GHSI shows some validity as a measure of health security but may not fully suffice as a stand-alone metric. The GHSI has face validity, being the culmination of a programme of work that began with a systematic search for factors that underpin health security.¹³ The GHSI correlates with the ReadyScore as well as aggregate outcomes for communicable diseases. GHSI scores are typically higher for countries with past domestic cases of SARS and those that received US investment through the GHSA programme. This latter finding may imply that investment in health security more generally (other countries and other programmes) might be associated with improving GHSI scores.

Our results also suggest other factors associated with higher GHSI scores including GDP per capita and the proportion of the population over 25 who have completed secondary education. Although there is likely collinearity among the variables evaluated, the relationship between education and GHSI is interesting. It might merely reflect that education is a proxy for other factors that could be causally related to GHSI scores, such as human and workforce capital, a welfare state, beliefs about the determinants of health, or many other variables. These relationships should be investigated in future work.

The strong correlation between the ReadyScore and GHSI overall score is important. This more than likely reflects the substantial overlap in the design of these two metrics. Two further points should be made. First, the GHSI exists for a further 85 countries that at the time of writing had not yet completed a JEE. This makes GHSI a potentially very useful reference for these countries. Second, the fact that GHSI measures many factors that JEE does not, and yet still correlates strongly with JEE, suggests that the additional domains of health system, commitments to international norms and features of the risk environment are all aspects of a stable overarching construct, namely ‘health security’.

ReadyScore is more strongly associated with deaths from communicable disease than the GHSI. This is not surprising given that the JEE approach is more specific to preventing, detecting and responding to known communicable diseases, whereas the GHSI has a substantial focus on commitments to norms, the risk environment and planning for unknown catastrophic risks such as emerging anthropogenic threats. That said, only one country with a GHSI overall score greater than 50 has a proportion of deaths from communicable disease greater than 20% (figure 1).

We also suspect that given that GHSI depends on public and published information, there will be instances where preparations have been made, but not published. This means that the GHSI might systematically underestimate countries’ health security capabilities. We also note the correlation with ReadyScore is not perfect and there is high variance in ReadyScore, for example where the GHSI score is moderate (see online supplemental figure S1). This variance may help explain why some countries have low GHSI and low communicable disease deaths (figure 1). The fact that default GHSI scoring weights all six categories approximately equally (12.8%–19.2% of overall score depending on the category) means that a significant contribution to the overall score comes from aspects that JEE does not measure. These contributions are largely derived from the ‘Norms’ and ‘Risk environment’ categories. Adjusting the weighting of various GHSI categories up and down would likely produce stronger and weaker correlation with JEE.

We found that countries previously exposed to SARS had higher overall and category GHSI scores than those that did not have SARS cases. SARS largely occurred in Asian countries and possible regional biases in factors such as national leadership and history of other region-specific outbreaks could bias results. However, a similar pattern was seen with MERS, although the findings were not as statistically robust. No trend was observed among countries having Ebola outbreaks, however the sample size of n=13 is small.

The GHSI has been criticised given that some high-scoring nations appear to have performed poorly in their initial 6-month response to the COVID-19 pandemic.¹⁸ We believe it is premature to dwell on this and at some future point a thorough analysis of GHSI scores and COVID-19 response will be needed.

A list of reasons why GHSI may not correlate with COVID-19 outcomes at present includes: low GHSI countries may not have identified many of their cases and deaths, outcome will depend on country strategy (eg, exclusion, elimination, suppression or mitigation); countries impacted later have the advantage of learning from those impacted earlier; the pandemic as at 29 July 2020 was still accelerating and far from over, perhaps still beginning, countries that have done well so far may yet be overwhelmed; and various items in the GHSI might apply to COVID-19 and others may not, meaning that subsets of items (not necessarily GHSI categories) might need to be analysed in isolation. It is possible that countries performing ‘poorly’ at one point in time may yet look successful in the future, for example, if they develop vaccines and roll out vaccination quickly.

On the one hand we found that the content of the GHSI provides some explanation for the poor performance of some high GHSI countries (namely the UK and US, see case studies in online supplemental material). High overall GHSI scores may mask many of these specific shortcomings.³⁷ However, there appear to be factors that are not part of the GHSI that are important for health security capabilities. In particular, it appears that the GHSI indicators may not adequately measure the ability to leverage capacity at the right time. Also, the GHSI is based on national information, and cannot determine local capacity in federal systems. Issues such as trust in government, leadership challenges, health insurance coverage, undocumented immigrants, polarised news sources or rampant misinformation and disinformation on social and mainstream media and of course value trade-off between economics and health, all appear to be conspiring in some countries to subvert an optimal health response and many of these factors are not adequately covered in the content of the GHSI.

Given that the COVID-19 pandemic is still developing at the time of writing, the predictive validity of the GHSI remains to be explored. GHSI scores should eventually be compared with COVID-19 pandemic outcomes but only once the pandemic is over (or at least a 2-year review point is analysed) and using strong evidence such as excess deaths stratified by country strategy, for example, elimination, suppression or mitigation. Future research should also identify subsets of GHSI items specific to particular events such as COVID-19 or laboratory accidents and validate these subsets of items against prospective external data.

Overall the GHSI is very heterogeneous and more like a maturity model than an index of one concept (eg, pandemic response). In fact, a single and highly valid universal index encompassing all aspects of health security, and applying to all countries, may not be possible. In the GHSI, some items may be very important for some countries such as screening all DNA sequence orders (and customer vetting) and others not important for some nations (eg, dual-use research oversight in Tuvalu). What the GHSI appears to show is where countries may have weaknesses and where the world might need to cooperate to bolster gaps in preparedness.

Many of these gaps can be seen in the GHSI 2019 Report, which provides a heat map of global GHSI scores (p18–19). The main weaknesses in health security appear to lie in Africa where scores for ‘health system’ are particularly low (mean 14.8/100). Low scores for small island nations are also noted by the GHSI Report authors.¹² The GHSI demonstrates that low scores in the WHO Africa region are in areas of biosecurity, biosafety, dual-use research oversight, zoonotic disease, data integration, emergency preparedness, linking public health and security agencies, health system capacity, medical countermeasures, communications with healthcare workers and infection control and availability of equipment. We further observed that African nations, small island nations, war-torn nations and nations isolated from the global community are over-represented in the bottom 20 scores across the six categories (see online supplemental material).

We are aware that our analyses have focused mainly on naturally occurring and endemic disease. However, important biothreats are not limited to naturally occurring human pathogens and emerging zoonoses. The risk of biothreats arising from laboratory error or failure of oversight, intentional misuse of biotechnology or deliberate attack are a major concern.³⁸ This is why the GHSI is also explicitly inclusive of anthropogenic threats.

This aspect of the GHSI will also require validation. This could be performed by analysing the incidence of, or response to, laboratory accidents for example (749 potential incidents were reported in US Federal Select Agent Programme incident data 2009–2015).³⁹ Methods should be sought to correlate aspects of the GHSI with Biological Safety Level (BSL)-3 and BSL-4 laboratory incidents, with deliberate biological events, and the response to these type of events.

Such work would be valuable given that the GHSI indicates that evidence for mitigation of anthropogenic biological hazards is scant. For example, the GHSI shows insufficient evidence of oversight of dual-use research and screening requirements for genetic material (186 countries), insufficient biosafety systems, laws, agencies or standardised training (121 countries) and many scores of zero for biosecurity (89 countries).

The vast majority of all the harm due to biological events, comes from very rare catastrophic events.^{40 41} The highest consequence events are catastrophic risks that threaten to permanently curtail the potential of humanity,⁴² and existential threats, which could eliminate humanity,⁴³ but critically, there is no way to determine external validity of GHSI (or JEE) with respect to unprecedented events. This is where the face and content validity of tools like the GHSI is important and where the very low scores across many countries in indicators of biosecurity, biosafety and dual-use research oversight are so concerning.

Beyond the GHSI

The impact of COVID-19 makes clear that we need to address factors that the JEE and GHSI do not measure and perhaps cannot measure. Institutional knowledge, a sense of urgency, relationships established in living memory, cross-sectoral logistics and resilience, all these factors may be critical to responses, and difficult to measure. Other overlooked factors may include root cause analysis of laboratory accidents and near misses; gathering intelligence on biological threats; cyber-biosecurity practices; the capability to strictly manage borders and quarantine; the elimination of wet markets and wildlife trade; and the apparent negative impact of sanctions, federalism, the politicisation of media and misinformation and disinformation on social and mainstream media.⁴⁴ Future research should examine these factors not covered by the GHSI, many of which appear to have confounded efforts to respond to the COVID-19 pandemic. Our findings suggest that the role of general education in health security should also be explored further.

Global biological catastrophe stems from high-risk environments, which the GHSI identifies. However, it is clear that countries need more than internal efforts as measured by GHSI or JEE scores. Efforts will require global leadership and strong international coordination mechanisms to overcome the failures and piecemeal responses to the COVID-19 pandemic.

To facilitate international cooperation there have been repeated calls for heads-of-state summits on global health security and to bolster the Biological Weapons Convention.^{5 12 43} Lessons learnt in global health security have been collated.⁴⁵ The IHR may need to be reconceived for a world where border closures, lockdowns and anthropogenic risks are the new norm. The IHR may also need to facilitate enhanced information sharing, early in an outbreak. Concern for the well-being of future generations may need to be written into international law, and the disagreements and grievances of present states must not be allowed to risk the flourishing of humanity’s future. Given all of these potentially important factors, it seems clear that even high GHSI scores will not be enough to ensure global health security.

Study strengths and limitations

This study has the strengths of being the first analytical attempt to consider the validity of the GHSI and being able to consider the GHSI in the context of previous global outbreaks such as SARS, MERS and Ebola. Nevertheless, an ecological study like ours has a number of limitations. This includes the use of aggregate and composite data, which can introduce collection bias. We note that gaps in the available data mean that not every metric has a score for every country. We are only able to show correlation, and so can not demonstrate causation. However, in most instances the correlation went in the a priori expected direction. Any analysis of the relationship between COVID-19 response and the GHSI is subject to many limitations where the outcome of interest is still developing. The GHSI is based on published documents and may fail to score capabilities that have not been written up or published on websites. Although we assessed a number of measures of validity, we have not specifically assessed the validity of the GHSI with respect to anthropogenic biothreats. Furthermore, the GHSI is a heterogeneous index, and it may be the case that subsets of items (not necessarily the defined six GHSI categories) are the best indicators of certain capabilities, perhaps pandemic response. The GHSI has some overlap with some of the metrics we compare it to (eg, the JEE—ReadyScore—and the GHSI ultimately share a number of items), however, the GHSI uses different methods to assess capabilities and therefore correlation remains useful.

Conclusion

Our results suggest that the GHSI is a somewhat valid measure of health security, but remains incomplete. The GHSI identifies considerable potential gaps in global health security capabilities and that anthropogenic threats remain neglected, although further validation of the GHSI as a measure of preparedness to mitigate these threats is desirable. It appears that US investments through GHSA and prior experience with the SARS pandemic were associated with higher GHSI scores, suggesting the importance of improved epidemic detection, preparedness and response and institutional experience.

The GHSI authors conclude that collective international preparedness and health security are weak. We conclude that all countries, particularly those with advanced biotechnology programmes, need to ensure sufficient biosecurity measures, however, scoring high on the GHSI, while desirable, is likely not enough to adequately mitigate threats. It is particularly relevant that the GHSI authors identified political, socioeconomic and environmental vulnerabilities as factors that can amplify deficiencies in preparedness.¹² We have suggested that some of these factors are not yet accounted for in measures of global health security.

The GHSI anticipates and calls on the world to mitigate the greatest of threats. Countries need to overcome the failure of imagination demonstrated by lack of preparedness ahead of the COVID-19 pandemic and the complacency that strong JEE assessments were sufficient. The world is in a challenging situation, clearly needing to do more, but many countries are still not compliant with the 15-years-old IHR. This highlights the magnitude of what is needed. Countries need a new scale of thinking that is focused on high-consequence future threats, with coordinated, well-financed and qualitatively different approaches to preparedness.

GHSI assessments have gone where JEEs have not, to 195 States Parties to the IHR to highlight the many risks and preparedness gaps. Biological catastrophe could strike soon and suddenly,⁴³ and the world should consider the vision of the GHSI alongside lessons learnt from COVID-19 and institutional knowledge and trialled experience.