Validating Joint External Evaluation reports with the quality of outbreak response in Ethiopia, Nigeria and Madagascar

Abstract

To date more than 100 countries have carried out a Joint External Evaluation (JEE) as part of their Global Health Security programme. The JEE is a detailed effort to assess a country’s capacity to prevent, detect and respond to population health threats in 19 programmatic areas. To date no attempt has been made to determine the validity of these measures. We compare scores and commentary from the JEE in three countries to the strengths and weaknesses identified in the response to a subsequent large-scale outbreak in each of those countries. Relevant indicators were compared qualitatively, and scored as low, medium or in a high level of agreement between the JEE and the outbreak review in each of these three countries. Three reviewers independently reviewed each of the three countries. A high level of correspondence existed between score and text in the JEE and strengths and weaknesses identified in the review of an outbreak. In general, countries responded somewhat better than JEE scores indicated, but this appears to be due in part to JEE-related identification of weaknesses in that area. The improved response in large measure was due to more rapid requests for international assistance in these areas. It thus appears that even before systematic improvements are made in public health infrastructure that the JEE process may assist in improving outcomes in response to major outbreaks.

Summary box.

• After Action Reviews (AARs) are a key part of improving Global Health Security.

• AARs in 3 countries closely tracked the strengths and weaknesses seen in each of the country's Joint External Evaluation.

• AARs can be used to monitor progress and gaps in health security.

Introduction

Structured evaluation of a country’s ability to respond to health security threats has garnered a great deal of attention and effort in the last 2 years with implementation of the Joint External Evaluation (JEE) system.¹ At the time of this writing, 95 countries had engaged in the full JEE process involving a national self-study followed by a 5 day, on the ground review involving international experts.² JEEs are intended to provide a thorough review and evaluation of a country’s capacities in 19 key areas of public health.³ The scores for each of 49 indicators in 19 domains is measured on a five-point scale combining quantitative and qualitative characteristics. The accompanying narratives summarise major strengths and limitations in each country’s public health systems, and recommendations for improvements are made.

The JEE scores are subsequently published as part of the JEE report and publicly available on several websites. Much effort has gone into developing and carrying out the JEEs; little validation of the JEE scores and recommendations are available to date.

We review a disease outbreak in each of three countries having undergone the JEE process. We compare scores and recommendations from the JEEs to conditions identified during postoutbreak reviews affecting that country’s response to the outbreak. Such a comparison provides a field-based validation in outbreak-related areas of the scores and recommendations from the JEE.

Scores and recommendations from each country’s JEE were drawn from the JEE summary documents published on the WHO’s website.⁴ Information on each outbreak was collected using a combination of sources and methods, including the following:

• Documented first and last reports from US Centers for Disease Control and prevention (CDC)’s Global Disease Detection Operations Center;

• On-line media reports, UN Situation Reports and journal articles;

• Interviews with staff following each outbreak at the CDC Operations Center;

• Interviews with international and national responders during the outbreaks. These responders included staff from CDC, other international agencies, and national Ministries of Health. Questions specific to that outbreak were elaborated for these interviews on the bases of the above sources. In some cases, follow-up questions were posed to these informants in an iterative process to probe further prior responses and in triangulating information from the various sources.

• Finally, preliminary conclusions were shared with field and headquarter staff to further refine an understanding of the collected responses.

Correspondence between the strengths and limitations in national systems relevant to an outbreak are summarised from review of text in the JEE document. Summarised information on each outbreak was compared with the relevant country’s JEE scores and text in these topical areas. The topical areas of relevance included IHR Coordination, National Laboratory Systems, Surveillance, Public Health Workforce, Preparedness, Emergency Operations, Medical Countermeasures and Risk Communication.

A subjective assessment of similarity and difference between these two sets of information was created independently as judged by each of the three authors on a three-level scale. Each of the authors is involved in global health security work professionally and has taken part in JEEs and postoutbreak reviews, though not in the countries evaluated. The three reviewers did not consult in creating their agreement scores. High correspondence existed if both the JEE and description of an outbreak raised a common concern. For example, if both described highly effective systems for laboratory diagnosis a ‘high’ level of correspondence was recorded. If in the outbreak, instead, an inadequate response from the laboratory system was reported, ‘low’ correspondence was recorded. Similarly, if the JEE reported poor surveillance capacity, and surveillance during the outbreak was considered poor, a ‘high’ correspondence was reported.

A Kappa statistic was generated to identify how likely the level of agreement among raters could have happened by chance. The MAGREE macro in SAS was used as it is a multiple-rater kappa statistic which omits missing values.

Background

The earliest JEE in this set was carried out in Ethiopia, during March/April 2016. The other two JEEs were carried out in June and July of 2017. The outbreak in Ethiopia was identified as beginning a little over a year after the JEE. In Madagascar and Nigeria, outbreaks began 2 and 6 months after the JEE, respectively. See table 1 and figure 1.

Table 1.

Summary of reviewed outbreaks

Country	Disease	Outbreak dates	Cases	Deaths	How many regions or states affected
Ethiopia	Acute watery diarrhoea	1/1/2017–23/7/2017	39 344	801	7 of 11
Nigeria	Lassa	24/3/17–15/12/17	376	86	19 of 36
Madagascar	Pneumonic plague	19/8/2017–27/11/2017	1293	209	55 of 114

Figure 1.

Time line from JEE to outbreak, 2016–2018. JEE, joint external evaluation.

​Acute watery diarrhea in Ethiopia

Detection and treatment

• The outbreak of Acute Watery Diarrhoea (AWD) is identified as starting 1/1/2017 as an index case was not identified. The Global Disease Detection programme of CDC stopped following the outbreak as it wound down after 23/7/2017.⁵

• A total of 39 344 clinical cases, with 801 deaths, were attributed to AWD.

• Cases were reported from seven regions of the country, with the majority of cases coming from the Somali region and believed to have begun from cases that came from Somalia.

• Outbreaks in neighbouring countries occurred in 2014, 2015 and 2016. The outbreak in 2016 resulted in registration of more than 20 000 cases in Ethiopia, including in the capital city.

• 2016 and 2017 outbreaks were exacerbated by drought, a high number of displaced people in areas with inadequate sanitation and poor food safety practices.⁶

Context and organisation

• The outbreak was called by its causative agent—Vibrio Cholera—in South Sudan and Yemen. It was referred to as AWD in Sudan, Ethiopia and Somalia.

• Leadership in the response in Ethiopia was provided by the Federal Ministry of Health, Regional Health Bureaus and the WHO.

• A major activity in response was to drill bore holes and truck water, and the provision of emergency food rations to millions of people.

• Thousands of national staff were deployed for water treatment activities and to staff AWD treatment centres. They were supported by dozens of international staff.

• Water quality testing and chlorination was a major activity, both among national and international partners.

• Treatment centres were set up in all affected states. Much activity was focused on infection prevention and control in treatment centres, social mobilisation to identify cases and get them to treatment centres, and training by case management teams for treatment centres.

​Lassa fever outbreak in Nigeria

Detection and treatment

• The first cases of the current outbreak were identified during mid-December of 2017.⁷ As of 18/3/2018 a total of 376 confirmed cases and 86 deaths were recorded. A further 1495 suspected cases were identified, 1084 of which were determined to be negative. Of note, 3675 contacts of confirmed or suspected cases were followed. Among these contacts, 59 were symptomatic but only 23 were confirmed as positive cases; 805 were still being followed at the time of publication, and a total of seven had been lost to follow-up.⁸

• The number of new cases identified peaked in late February. By mid-March, the number of new cases declined rapidly. Nine states left the active phase of the outbreak and 38 people were still receiving treatment in six of the remaining nine ‘active’ states.

• Lassa fever is endemic to Nigeria and other West African countries. Small, disseminated outbreaks are common as the vectors in the animal kingdom infect people in close contact. In 2017 there were two peaks of infection, indicating a potential for expanded transmission from animal vectors as came to occur in 2018. There were 247 cases recorded in 2016 and 85 deaths. The death rate in 2018 appears to be about 1/3rd lower, probably due to earlier treatment and better identification of cases.⁹

Context and organisation

• Three states account for 83% of all confirmed cases. Cases were identified in 56 local areas of 19 states across the country.

• The Nigerian CDC (NCDC) and WHO led response activities out of the NCDC Emergency Operations Centre in Abuja. Rapid Response Teams composed of NCDC staff, Ministry of Health staff and Field Epidemiology Training Programme residents led the response in affected states.¹⁰

• Three laboratories in country confirmed infection using a PCR method. The laboratory system is supported by the Bernhard Nocht Institute for Tropical medicine in Germany.

• Three hospitals provide all the in-patient care for Lassa Fever cases.

• A total of 17 health workers became confirmed cases in six states. No new infections occurred among healthcare workers in later weeks.

• Rapid Response Teams went to four states that bordered Benin to improve disease surveillance as nine suspected cases and several confirmed cases in Benin appear to have imported the infection from Nigeria.

​Pneumonic plague in Madagascar

Detection and treatment

• The Index case had become symptomatic in mid-August 2017. Travelled via taxi from central highlands through the capital city on 27/8/2017. Diagnosed first case 11/9/2017. WHO notified on 13/9/2017. Twenty-seven other cases traced to the index case.

• Bubonic plague is endemic with cases reported every year. The last outbreak was 8/2016–1/2017 with around 300 cases. Pneumonic plague was reported in northern Madagascar last in 2015 with 14 cases. Seven of those were treated and four of them survived.

• In 2017, a total of 402 confirmed cases and 209 deaths occurred due to plague through 27/11/2017. Some of these deaths occurred among unconfirmed cases and are thus probably not all pneumonic plague. Total of 2417 cases (including 700 with negative laboratory tests) reported; 1293 of the total are considered confirmed, probable or suspected. Of these 1854 were classified as pneumonic; others were bubonic or unclassified.¹¹ Government of Madagascar then called the epidemic contained, while WHO said more cases could be anticipated through the April end of plague seasonal transmission.

Context and organisation

• Ministry of Public Health led response, co-led by WHO, focusing mainly on case finding, diagnosis, treatment of cases and isolation. Preventive chemoprophylaxis provided to 7318 identified contacts of cases.¹²

• Institute Pasteur de Madagascar provided all laboratory support for diagnosis and treatment. Awareness campaigns led by government throughout country.

• Nine plague treatment centres and six mobile centres were established with the support of international organisations.¹³

• Fifty-five of 114 districts reported cases. Capital city had the most cases.

Comparison of JEE reports and outbreak results

Tables 2–4 present the results of JEE and outbreak reviews in Ethiopia, Nigeria and Madagascar, respectively. In the final column of each of these tables, scores from the three raters on the level of correspondence between the JEE and outbreak is presented as low (L), medium (M), high (H) or no response (N/A).

Table 2.

Comparison of JEE and outbreak review in Ethiopia

JEE domain/indicator	JEE score	Major JEE recommendation	Outbreak response capacity	Degree of correspondance
IHR Coordination P.2.1	3	Coordination mechanisms planned to be established but not yet in place across sectors	Coordination between health and water authorities was weak.	M H H
National Lab System D.1.1	4	System strong at the national level but poor supply chain and staff turnover	Nearly all cases received a clinical diagnosis only; laboratory services were weak. More laboratory specialists were desired but funds to bring internationals were inadequate.	L M H
Surveillance D.2.3	3	Plans to develop, national commitment in place and skill but not strong at state level	Case finding and active surveillance to identify clusters of cases was strong.	M L M
Workforce D.4.1	3	Need more staff and FETP resident advisor; WHO role key	FETP staff key in regional level coordination.	H H H
Preparedness R.1.2	2	Risk assessments done but mapping of resources lacking	Equipment, supplies and training not ready for the outbreak.	H H H
Emergency Operations Activation R.2.1	2	No manager and lack of permanent staff	Regional coordination good; national level was inadequate and frustrating,. Emergency Operations Centre (EOC) not activated until August.	H H H
Emergency Operations Function R.2.3	2	No specific training for staff	Regional coordination good; national level was inadequate and frustrating,. EOC not activated until August.	H H H
Emergency Operations Case Management R.2.4	2	Cholera guidelines exist	Training done rapidly for staff when outbreak hit a new area. Guidelines were essential.	H H H
Medical Countermeasures R.4.1	4	No warehouse, weak logistics, no established international agreements	Regional Health Bureau led the response to 100 treatment centres, with good WHO support. UN Children’s Fund (UNICEF), WHO, Medecins Sans Frontieres (MSF), Oxford Famine Relief) Oxfam, CDC, Islamic Relief among other non-governmental organisations very involved. Not adequate for Water, Sanitation and Hygiene activities.	M M L
Risk Communication R.5.4	3	Dedicated local staff in place	Epi identification of cases was used to tailor social mobilisation and education activities in some areas. Weaker in nomadic areas.	H H H

CDC, Centers for Disease Control and prevention; FETP, Field Epi Training Programme; JEE, joint external evaluation.

Table 3.

Comparison of JEE and outbreak review in Nigeria

JEE domain/indicator	JEE score	Major JEE comment/ recommendation	Outbreak response observed	Degree of correspondance
IHR Coordination P.2.1	2	SOPs exist	Coordination weak especially in first month	H L H
Zoonotic Diseases P.4.3	1	Better coordination for response needed	Spread of vectors went unrecognised	H H H
Biosafety P.6.1	1	Funding and planning weaknesses	Use of Personal Protective Equipment and training inadequate; 17 health workers infected	H H H
National Lab System D.1.1	3	Capable laboratories but need for standardisation	International supply of reagents and training was essential. EOC actions key to make this happen	H M H
Surveillance D.2.3	3	Weak capacity in many states	Needed RRT staff from national level to take over from states	M H H
Reporting D.3.2	2	Officers in each state	Variable response from states. Needed RRT teams from central level to make this happen	H L H
Workforce D.4.1	3	Strong Field Epi Training Programme (FETP)	FETP trainees essential to response	H H H
Preparedness R.1.2	1	Logistics system weak; risk mapping needed	Lack of preparedness or awareness of rising risk of outbreak	H HH
Emergency Operations Activation R.2.1	2	SOPs not fully developed; state level EOCs missing	EOC key for this response; developing while doing	H M H
Emergency Operations Function R.2.3	3	Experience coordinating responses; procedures not standardised	EOC key for this response; developing while doing	M H H
Emergency Operations Case Management R.2.4	2	Some case management guidelines available	Not useful until adapted mid-epidemic	H M M
Medical Countermeasures R.4.1	1	Need for stockpile and logistics	Effective supply system created on the fly	H L M
Risk Communication R.5.4	3	Coordination from Federal to States weak	No Info (anecdotally, seemed to be weak)	M N/A N/A
Points of entry PoE.2	1	Contingency plans needed	Proactive response initiated on the fly	N/A L M

EOC, Emergency Operations Centre; IPC, Infection Prevention and Control; JEE, joint external evaluation; RRT, Rapid Response Teams; SOP, Standard Operating Procedures.

Table 4.

Comparison of JEE and outbreak review in Madagascar

JEE domain/indicator	JEE score	Major JEE recommendation	Outbreak response capacity	Degree of correspondance
IHR Coordination P.2.1	2	Intersectoral committee exists but plan of work and response plan needed	A high-level inter-Ministerial coordination forum had to be established by the Prime Minister’s office to lead the response as the Inter-sectoral Support Group for Plague Control (GIALP) was not operational.	H M H
Zoonotic Diseases P.4.3	2	Need to elaborate and fund plan	Weak plan for seasonal vector control	N/A H H
Biosafety P.6.1	2	Need intersectoral coordination and funding	Inadequate and insufficient IPC supplies	N/A H H
National Lab System D.1.1	4	International accords existing; 13 national labs in place	Rapid Diagnostic kits not adequate and insufficient, delay for PCR confirmation. Laboratory testing was led in-country at Institut Pasteur.	H L H
Surveillance D.2.3	3	Need for training and experience	Surveillance could have been improved to better reflect burden of disease; case definitions weak; weak detection capacity at the community level	M H M
Reporting D.3.2	2	Not in place in all parts of the country	No clear reporting channels led to delay in response	H H H
Workforce D.4.1	2	Inadequate outside of national level	Seemed to have adequate personnel for contact tracing but needed to train community health workers on plague surveillance and control	M H H
Preparedness R.1.2	1	Need to analyse and map	Existence of national contingency plan which was not implemented and shared with all regions. No coordination of preparedness activities	H H H
Emergency Operations Activation R.2.1	2	Able to establish RRTs	Insufficient trained and equipped multisectoral teams at the regional level	H M M
Emergency Operations Function R.2.3	2	Experienced in plague response in prior years	Internal Incident Management System coordination was lacking	H M M
Emergency Operations Case Management R.2.4	2	Existing guidelines	Revised treatment protocol developed, but not implemented; limited experience managing pulmonic plague cases	H M H
Medical Countermeasures R.4.1	1	Need to establish procedures, stock control and logistics	Weak supply management; the logistics function not included in the national contingency plan	H H H
Risk Communication R.5.4	2	Personnel exist but lack of feedback and local operationalisation	Mobilised 9000 to assist with risk communication and community engagement but there was still miscommunication regarding dignified safe burial and stigma of contacts	H M H
Points of entry PoE.2	1	Lack of personnel and plans	Lack of Standard Operation Procedures and trained personnel at points of entry; implemented by international partners during outbreak	H H H

JEE, joint external evaluation; RRT, Rapid Response Teams.

Thirty-seven variables were compared between JEE scores and field operation levels, by three raters, among these three outbreaks. This created a total of 111 scores representing a low, medium or high level of correspondence between the JEE and the outbreak response review.

For 13 of the 37 variables, all three raters agreed that the correspondence was high. For additional 13 variables, two raters rated the correspondence a high, while one rater considered it to be only moderate. Only eight times did a rater consider the correspondence to be low, and for none of the 37 variables did more than one rater consider it low.

While 37 variables were evaluated, a reviewer occasionally chose not to respond with a ‘low, medium or high’ response. In total, 107 scores were recorded among the three reviewers. The Coefficient of Concordance produced via the MAGREE routine for a Kappa test was 0.457; this represents an F statistic of chance probability of 0.037. In simple terms, the level of agreement was high (SAS summary reference ofvarious measures of concordance are presented in https://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm%23statug_freq_a0000000647.htm The SAS routine used for MAGREE is presented in http://support.sas.com/kb/25/006.html).¹⁴

The comparison did not show any consistent or dramatic conflicts between JEE and outbreak information. Thus, though interpretations may vary regarding the degree of agreement between JEE and outbreak information, JEEs overall appear to provide a very good guide to strengths and weaknesses in actual outbreaks.

The comparisons made here had two important limitations. First, response capacity at the end of the outbreak often had improved a great deal from the beginning. Thus, comparisons depend on when the comparison is made. Second, much of the action in the outbreak is local, so laboratory, social mobilisation or treatment characteristics in one area may be very different from another. JEE scores seldom took variable capacity in a country into account.

In summary, it appears that after-action reports can provide a strong check on information in the JEE and can provide a near real-time update on capacities of the public health system.

Lessons

Low JEE scores existed in critical areas across all three countries with implications on the ability of those countries to detect and respond to the outbreaks. Those low scores, however, were not always critical limiting factors. In the case of Ethiopia and Nigeria, even with limited skill and personnel, large country systems were able to mobilise an adequate number of skilled personnel.

Some of the inconsistencies found between the JEE and outbreak review can be explained by particular details of the outbreak. For example, though Nigeria has a low JEE score for border health, the national EOC mobilised teams to border areas to coordinate response. Because of the needs in that outbreak, one strong area of the JEE made up for weakness in another. In each of the three countries, international staff strengthened the response in areas rated low during the JEE. The quality of outbreak response, when inconsistent with JEE scores, was generally better than that predicted by the country’s JEE.

Where internal skill, equipment, training and personnel were lacking, in each of the three outbreaks, national resources were supplemented by international resources that to a large extent made up for national limitations. The quality and variety of those international personnel and supplies, and the ability of national systems to absorb them, was described to be important in strengthening the outbreak response; this could not be captured by the JEE. It appears to the authors that the JEE strongly sensitised national authorities to their areas of weakness and to opportunities to supplement national resources with international staff and equipment.

The JEE assessment identified systems capacity at the time of evaluation. What cannot be assessed from a single outbreak event is the influence of the JEE experience over time. It appears to us that for all three countries, the JEE created a framework to understand key roles and activities needed to respond to an outbreak more effectively.

Major outbreaks following a JEE may provide an opportunity for more rapid improvements in situation awareness of systematic weaknesses to be addressed than existed prior to the JEE. A fuller evaluation of this would require comparison of the review of several outbreak after a JEE, compared with several outbreaks before the JEE.

In Nigeria, Lassa outbreaks occur every year. In the outbreak prior to the JEE, an observer noted that authorities would send epidemiologists out to collect data and as we identify problems in the response we will address them. In the post-JEE Lassa outbreak, there was a much stronger focus on technical activities needed in the areas of laboratory, surveillance, reporting, Emergency Operations Centres, Medical Countermeasures, Points of Entry, communication and biosafety. Specific weaknesses in human/animal surveillance and laboratory systems triggered discussion to involve the Ministries of Agriculture and Environment. The phrase ‘supply chain’ had become part of the vocabulary. There was discussion about weaknesses in newly established legislation. The JEE experience provided an intense context with which to focus on these issues, raising the level of understanding and discourse and creating a shared vision, which otherwise would likely have been far less.

Although the JEE raised the level of understanding for key roles and actions that shared vision can be expected to deteriorate over time as staff rotation occurs and people not involved in the country’s JEE assume relevant posts. The need to refresh the reflection that occurred during the JEE can, in part, be met by improving After Action Reviews (AARs). AARs should consider the trend among several outbreaks over time. They should move from a focus on the specifics of the current outbreak to a more general reflection on the JEE indicators and levels.

What would the response to the outbreak have looked like if the JEE had not occurred? In Nigeria, weaknesses may not have been recognised as well or as quickly. Having recognised those weaknesses, prioritisation of critical functions occurred (in the case of Nigeria) which would have been far weaker and the coordination among state and national authorities would have been far less had not the JEE sensitised a large group of people to critical functions. It is less clear that this occurred in Madagascar, where leadership authority was not clearly established, or in Ethiopia, where political considerations limited the ability of health leaders to organise and mobilise.

Several key qualities of these outbreak responses were not captured by each country’s JEE:

• In federal system countries, such as Nigeria, the coordination of roles between national and state level authorities and the assessment of variable levels of capacity in various states.

• Quality of AARs and their integration into International Health Regulations and interim JEE internal country assessments.

• The level and timing of intersectorial participation in public health activities during an outbreak. Early large-scale mobilisation can greatly reduce transmission, and obviate the need for later panic-level participation. This goes beyond coordination with security authorities or risk communication to affected communities. It most closely tracks to the PREVENT indicator of ‘IHR Coordination, Communication, and Advocacy’, but is a key part of response that is easy to see but difficult to measure.

Opportunities to focus on these issues in JEEs, and in After Action Reviews, Simulation Exercise Evaluations and annual national IHR reporting can be used to improve post-JEE National Action Plans and outbreak response in the future.