Abstract
Objective
To investigate the reliability of the public health event notification assessment process under the International Health Regulations (2005) (IHR).
Methods
In 2009, 193 National IHR Focal Points (NFPs) were invited to use the decision instrument in Annex 2 of the IHR to determine whether 10 fictitious public health events should be notified to WHO. Each event’s notifiability was assessed independently by an expert panel. The degree of consensus among NFPs and of concordance between NFPs and the expert panel was considered high when more than 70% agreed on a response.
Findings
Overall, 74% of NFPs responded. The median degree of consensus among NFPs on notification decisions was 78%. It was high for the six events considered notifiable by the majority (median: 80%; range: 76–91) but low for the remaining four (median: 55%; range: 54–60). The degree of concordance between NFPs and the expert panel was high for the five events deemed notifiable by the panel (median: 82%; range: 76–91) but low (median: 51%; range: 42–60) for those not considered notifiable. The NFPs identified notifiable events with greater sensitivity than specificity (P < 0.001).
Conclusion
When used by NFPs, the notification assessment process in Annex 2 of the IHR was sensitive in identifying public health events that were considered notifiable by an expert panel, but only moderately specific. The reliability of the assessments could be increased by expanding guidance on the use of the decision instrument and by including more specific criteria for assessing events and clearer definitions of terms.
ملخص
الغرض
تقصّي معوّليّة عملية تقييم التبليغ عن أحداث الصحة العمومية في اللوائح الصحية الدولية (2005).
الطريقة
في عام 2009، دُعِي 193 منسقاً وطنياً لاستخدام أداة اتخاذ القرار في الملحق 2 التابع للوائح الصحية الدولية لتحديد ما إذا كان ضرورياً التبليغ لمنظمة الصحة العالمية عن 10 أحداث صحية عمومية افتراضية. وجرى تقييم ضرورة التبليغ عن كل حدث على نحو مستقل من قبل فريق من الخبراء. وقد اُعتبر أن درجة التوافق بين المنسقين الوطنيين والتواؤم بين المنسقين الوطنيين وفريق الخبراء مرتفعة إذا تعدّت 70% من الاتفاق على الاستجابة.
النتائج
استجاب إجمالياً 74% من المنسقين الوطنيين. كانت درجة الوسيط للتوافق بين المنسقين الوطنيين حول قرارات التبليغ 78%. وكانت مرتفعة في ستة أحداث اعتقد غالبيتهم أن التبليغ عنها ضرورياً (الوسيط: 80%؛ المجال: 76-91) ولكنها كانت منخفضة في الأربعة أحداث المتبقية (الوسيط: 55%؛ المجال: 54-60). وكانت درجة التواؤم بين المنسقين الوطنيين وفريق الخبراء مرتفعةً في خمسة أحداث اعتقد فريق الخبراء أن التبليغ عنها ضرورياً (الوسيط: 82%؛ المجال: 76-91) ولكنها كانت منخفضة في الأحداث الأخرى التي اعتقد أن التبليغ عنها غير ضروري (الوسيط: 51%؛ المجال: 42-60). حدد المنسقون الوطنيون الأحداث الواجب التبليغ عنها بحساسية عالية مقارنة بالنوعية (قيمة الاحتمال P أقل من 0.001).
الاستنتاج
كان استخدام المنسقين الوطنيين لعملية تقييم التبليغ في الملحق2 من اللوائح الصحية الدولية حساساً في تحديد الأحداث الصحية العمومية التي اعتقد فريق الخبراء أن من الواجب التبليغ عنها، ولكن النوعية كانت متوسطة. يمكن زيادة معوّليّة التقييم عن طريق توسيع الإرشادات حول استخدام أداة اتخاذ القرار وإدراج المزيد من المعايير النوعية لتقييم الأحداث مع توضيح أفضل لتعاريف المصطلحات.
Résumé
Objectif
Examiner la fiabilité du processus d’évaluation de la notification d’un événement de santé publique dans le cadre du Règlement sanitaire international (RSI) de 2005.
Méthodes
En 2009, 193 points focaux nationaux RSI (PFN) ont été invités à utiliser l’instrument de décision figurant à l’annexe 2 du RSI afin de déterminer s’il convenait de notifier 10 événements fictifs de santé publique à l’OMS. L’obligation de notification de chaque événement a été évaluée de manière indépendante par un groupe d’experts. Le degré de consensus entre les PFN et de concordance entre les PFN et le groupe d’experts était considéré comme étant élevé lorsque plus de 70% s’accordaient sur une réponse.
Résultats
En tout, 74% des PFN ont répondu. Le degré moyen de consensus parmi les PFN sur les décisions de notification était de 78%. Il était élevé pour les six événements considérés comme étant notifiables par la majorité (moyenne: 80%; éventail: 76–91), mais comme étant faible pour les quatre restants (moyenne: 55%; éventail: 54–60). Le degré de concordance entre les PFN et le groupe d’experts était élevé pour les cinq événements jugés comme étant notifiables par le groupe (moyenne: 82%; éventail: 76–91), mais comme étant faible (moyenne: 51%; éventail: 42–60) pour ceux qui n’étaient pas considérés comme étant notifiables. Les PFN ont identifié les événements devant être notifiés avec plus de sensibilité que de spécificité (P < 0,001).
Conclusion
Lorsqu’il était utilisé par les PFN, le processus d’évaluation de la notification figurant à l’annexe 2 du RSI était sensible à l’identification d’événements de santé publique considérés comme devant être notifiés par un groupe d’experts, mais il n’était que modérément spécifique. La fiabilité des évaluations pourrait être accrue par une généralisation des recommandations sur l’utilisation de l’instrument de décision et par l’ajout de critères plus spécifiques en matière d’évaluation des événements et des définitions terminologiques plus claires.
Resumen
Objetivo
Investigar la fiabilidad del proceso de evaluación de las notificaciones de episodios incluido en el Reglamento Sanitario Internacional (RSI) de 2005.
Métodos
En 2009, se propuso a 193 Centros Nacionales de Enlace (CNE) la utilización del instrumento de decisión incluido en el Anexo 2 del RSI para determinar cuáles de los 10 acontecimientos ficticios de salud pública debían notificarse a la OMS. Un grupo de expertos evaluó la perceptibilidad de cada suceso de manera independiente. El grado de consenso entre los CNE y de concordancia entre los CNE y el grupo de expertos se consideró alto cuando coincidían más del 70% en una respuesta.
Resultados
En total respondió el 74% de los CNE. El grado medio de consenso entre los CNE sobre las decisiones de notificación fue de un 78%. Resultó alto para los seis acontecimientos considerados como notificables por la mayoría (media: 80%; intervalo: 76–91) pero bajo para los cuatro restantes (media: 55%; intervalo: 54–60). El grado de concordancia entre los CNE y el grupo de expertos fue elevado para los cinco sucesos que el grupo de expertos consideró como notificables (media: 82%; intervalo: 76–91) pero bajo para aquellos casos que no consideraron notificables (media: 51%; intervalo: 42–60). Los CNE identificaron los sucesos notificables con una mayor sensibilidad que especificidad (p < 0,001).
Conclusión
Cuando los CNE utilizaron el proceso de evaluación de notificaciones incluido en el Anexo 2 del RSI, este resultó ser sensible para identificar los acontecimientos de salud pública que un grupo de expertos consideró notificable, aunque con una especificidad moderada. La fiabilidad de las evaluaciones podría aumentar si se ampliara la orientación sobre el uso del instrumento de decisión y si se incluyeran criterios más específicos para evaluar los sucesos y las definiciones de los términos fueran más claras.
Резюме
Цель
Исследовать надежность схемы оценки уведомлений о событиях в области общественного здравоохранения, предусмотренной Международными медико-санитарными правилами (ММСП) 2005 года.
Методы
В 2009 году 193 национальным координаторам (НК) ММСП было предложено воспользоваться инструментом принятия решений, содержащимся в Приложении 2 ММСП, чтобы определить, следует ли уведомлять ВОЗ о 10 вымышленных событиях в области общественного здравоохранения. Группа экспертов оценивала, является ли каждое из событий предметом обязательного уведомления. Уровень единодушия между НК и единогласия между НК и группой экспертов считался высоким, если более 70% соглашались принять ответные меры.
Результаты
В общей сложности ответы получены от 74% НК. Медианный уровень единодушия между НК в отношении принятия решений об уведомлении составлял 78%. Он был высоким для шести событий, которые, по мнению большинства, подлежали обязательному уведомлению (медиана: 80%; диапазон: 76–91), а для остальных четырех событий – низким (медиана: 55%; диапазон: 54–60). Уровень единогласия между НК и группой экспертов был высоким для пяти событий, которые, по мнению группы экспертов, подлежали обязательному уведомлению (медиана: 82%; диапазон: 76–91), а для событий, которые не были признаны подлежащими уведомлению, был низким (медиана: 51%; диапазон: 42–60),. НК выделяли события, подлежащие уведомлению в большей степени проявляя чувствительность, чем специфичность (P < 0.001).
Вывод
Будучи использованной национальными координаторами, схема оценки уведомлений, предусмотренная Приложением 2 ММСП, была чувствительной к выявлению событий в области общественного здравоохранения, которые группа экспертов считала предметом обязательного уведомления, но лишь в умеренной степени была специфичной. Надежность оценок можно повысить, если расширить руководство по использованию данного инструмента принятия решений и предусмотреть более конкретные критерии оценки событий и более четкие определения терминов.
摘要
目的
旨在调查2005年《国际卫生条例》（IHR）中公共卫生事件通报评估过程的可靠性。
方法
2009年，193个《国际卫生条例》国家中心（NFPs）受邀运用《国际卫生条例》附件二中的决策工具，以确定10个虚构的公共卫生事件是否应向世界卫生组织（WHO）通报。每个事件的可通报性均由专家小组进行独立评估。当对某一反应的一致意见超过70%，则认为国家中心间的共识度以及国家中心与专家小组间的共识程度高。
结果
总体而言，74%的国家中心做出了回应。国家中心就通报决定达成一致意见的中位数为78%。多数认为需予以通报的6个事件的中位数高（中位数：80%；范围：76-91），然而其余4个事件的中位数低（中位数：55%；范围：54-60）。就专家小组视为应予以通报的5个事件而言，国家中心和专家小组间的一致性程度高（中位数：82%；范围：76-91）；但是，就专家小组视为应不予通报的事件而言，此一致性程度低（中位数：51%；范围：42-60。国家中心在确定需予以通报的事件过程中，敏感性大于特异性（P < 0.001）。
结论
在国家中心的运用过程中，《国际卫生条例》附件二中的通报评估过程在确定专家小组认为应予以通报的公共卫生事件时敏感性强，然而特异性则一般。评估的可靠性可通过加强对决策工具使用的指导，以及通过收录更多事件评估特定标准和更明确的术语定义来提高。
Introduction
The International Health Regulations (2005) (IHR), which entered into force in June 2007, are a legally binding agreement between 194 States Parties, including states that are not Member States of the World Health Organization (WHO).1 Under the IHR, States Parties are required to notify WHO of “all events which may constitute a public health emergency of international concern”. Whether a given event is notifiable is determined using an algorithm: the decision instrument contained in Annex 2 of the IHR.1
The decision instrument defines an event as notifiable if it satisfies two or more of the following four criteria: (i) the event has a serious public health impact; (ii) the event is unusual or unexpected; (iii) there is a significant risk of international spread; (iv) there is a significant risk of international travel or trade restrictions.
In addition, all cases of smallpox, wild-type polio, novel-subtype human influenza virus infection and severe acute respiratory syndrome (SARS) are “intrinsically” notifiable, without the need to apply the four criteria. Annex 2 of the IHR is designed to heighten the sensitivity of the notification process and thus ensure a timely assessment of and response to critical public health events.2
The IHR stipulate that each State Party designate an office with which WHO can communicate at all times: the National IHR Focal Point (NFP). The NFPs receive guidance and training from WHO.2–6 They are responsible for contacting WHO about notifiable events and WHO recommends that they coordinate the notification assessment process, though they may not themselves be responsible for actually assessing the public health risk.3
Article 54.3 of the IHR,1 as well as World Health Assembly resolutions 58.37 and 61.2,8 mandate WHO to conduct studies to review and evaluate the functioning of Annex 2 of the IHR. A 2008 WHO technical consultation recommended that an assessment be carried out to evaluate agreement among NFPs when judging certain described events using Annex 2.9 The University of Geneva Hospitals’ Infection Control Programme in Switzerland was commissioned to carry out this assessment.
Methods
Survey design
The survey exploring NFPs’ use of Annex 2 of the IHR was based on several fictitious scenarios that described events devised to represent a wide range of public health risks with a varying likelihood of being considered notifiable. The number of scenarios included was limited to 10 and the vignettes (Box 1) were kept succinct to minimize the time and effort required from participants and to maximize the response rate. Box 2 provides summaries of the 10 scenarios.
Box 1. Full description of the public health event in one scenario used to evaluate the use of Annex 2 of the International Health Regulations (2005).
Several hospitals in province X have reported an abnormal increase of admissions due to pneumonia in the last two months.
A significant number of health-care workers are themselves affected by the respiratory disease and three health care workers aged between 20 and 50 years have died from respiratory failure. Influenza was suspected based on the clinical presentation, but tests for this and other common respiratory pathogens have been negative so far. Infection control measures, including the isolation of suspected cases, are being implemented.
Box 2. Short descriptions of the public health eventsa in the 10 scenarios used in the survey to evaluate implementation of Annex 2 of the International Health Regulations (2005).
Scenario 1. Pneumonia of unknown etiology linked to deaths among health-care workers
Scenario 2. Probable large-scale arsenic poisoning among refugees in a border region due to contaminated ground water
Scenario 3. International distribution of an ineffective HIV drug by a pharmaceutical company due to a production error
Scenario 4. Klebsiella pneumoniae resistant to all currently available antimicrobials detected for the first time in this country among a cluster of cases of neonatal sepsis
Scenario 5. Moderate rise in the incidence of dengue fever in an area dependent on international tourism
Scenario 6. Outbreak of salmonellosis linked to chicken meat in a product distributed within a single country
Scenario 7. Imported case of wild-type polio
Scenario 8. Fuel explosion with many casualties; only a single country affected
Scenario 9. Ochratoxin A detected in an imported wheat shipment at the port of entry
Scenario 10. Outbreak of foot-and-mouth disease among livestock
a The full scenarios are available with the survey at https://www.surveymonkey.com/s.aspx?sm=GX8_2f9IUtDrEbMrNQpAFylg_3d_3d or from the corresponding author on request.
For each scenario, six items about the notifiability of the event had to be answered either “yes”, “no” or “don’t know” (Table 1). These items were: the personal opinion of the individuals who responded on behalf of the NFPs about whether the event was notifiable (item 1); the NFP’s judgement of whether each of the four criteria of the decision instrument in Annex 2 were satisfied (items 2–5); and the NFP’s judgement of whether an actual notification decision under the IHR should be made (item 6). The structure of the survey required participants to go through all four criteria for each scenario, including scenario 7, which described an intrinsically notifiable event (i.e. a case of wild-type polio).
Table 1. Decisions made by an expert panel and 141 National International Health Regulations’ Focal Points when using Annex 2 of the International Health Regulations (2005) to assess whether the public health events described in 10 survey scenarios should be reported to the World Health Organization.
| Scenario | Responses | Survey item |
|||||
|---|---|---|---|---|---|---|---|
| 1 |
2 |
3 |
4 |
5 |
6 |
||
| Personal opiniona | Criterion 1 of IHR Annex 2b | Criterion 2 of IHR Annex 2c | Criterion 3 of IHR Annex 2d | Criterion 4 of IHR Annex 2e | Notification decisionf | ||
| 1 | Expert panel’s | NA | Yes | Yes | No | No | Yes |
| NFPs “yes”, %g | 87.2 | 95.7 | 92.9 | 38.3 | 19.1 | 90.8 | |
| NFPs “no”, % | 8.5 | 3.5 | 5.7 | 24.8 | 52.5 | 7.8 | |
| NFPs “don’t know”, % | 4.3 | 0.7 | 1.4 | 36.9 | 28.4 | 1.4 | |
| 2 | Expert panel’s | NA | Yes | Yes | Undecided | No | Yes |
| NFPs “yes”, %g | 76.6 | 97.2 | 80.1 | 45.4 | 14.2 | 78.0 | |
| NFPs “no”, % | 22.0 | 2.1 | 17.0 | 51.8 | 82.3 | 18.4 | |
| NFPs “don’t know”, % | 1.4 | 0.7 | 2.8 | 2.8 | 3.5 | 3.5 | |
| 3 | Expert panel’s | NA | Yes | Yes | Undecided | Undecided | Yes |
| NFPs “yes”, %g | 78.0 | 80.9 | 77.3 | 48.2 | 21.3 | 75.9 | |
| NFPs “no”, % | 19.9 | 16.3 | 19.1 | 47.5 | 75.9 | 21.3 | |
| NFPs “don’t know”, % | 2.1 | 2.8 | 3.5 | 4.3 | 2.8 | 2.8 | |
| 4 | Expert panel’s | NA | Yes | Undecided | No | No | No |
| NFPs “yes”, %g | 55.3 | 90.1 | 56.0 | 19.1 | 3.5 | 55.3 | |
| NFPs “no”, % | 39.7 | 9.2 | 43.3 | 70.2 | 92.9 | 41.8 | |
| NFPs “don’t know”, % | 5.0 | 0.7 | 0.7 | 10.6 | 3.5 | 2.8 | |
| 5 | Expert panel’s | NA | No | No | No | No | No |
| NFPs “yes”, %g | 32.6 | 50.4 | 8.5 | 41.1 | 17.7 | 38.3 | |
| NFPs “no”, % | 66.0 | 49.6 | 90.8 | 55.3 | 78.7 | 60.3 | |
| NFPs “don’t know”, % | 1.4 | 0.0 | 0.7 | 3.5 | 3.5 | 1.4 | |
| 6 | Expert panel’s | NA | Yes | No | No | No | No |
| NFPs “yes”, %g | 48.9 | 74.5 | 63.8 | 16.3 | 17.0 | 53.9 | |
| NFPs “no”, % | 50.4 | 24.1 | 31.9 | 81.6 | 80.1 | 46.1 | |
| NFPs “don’t know”, % | 0.7 | 1.4 | 4.3 | 2.1 | 2.8 | 0.0 | |
| 7 | Expert panel’s | NA | NA | NA | NA | NA | Yes |
| NFPs “yes”, %g | 87.9 | 77.3 | 82.3 | 61.0 | 12.8 | 88.7 | |
| NFPs “no”, % | 11.3 | 21.3 | 17.7 | 36.2 | 82.3 | 10.6 | |
| NFPs “don’t know”, % | 0.7 | 1.4 | 0.0 | 2.8 | 5.0 | 0.7 | |
| 8 | Expert panel’s | NA | Yes | No | No | No | No |
| NFPs “yes”, %g | 36.2 | 69.5 | 77.3 | 7.1 | 2.1 | 41.8 | |
| NFPs “no”, % | 61.7 | 26.2 | 19.1 | 90.8 | 96.5 | 55.3 | |
| NFPs “don’t know”, % | 2.1 | 4.3 | 3.5 | 2.1 | 1.4 | 2.8 | |
| 9 | Expert panel’s | NA | Undecided | No | Undecided | Yes | Yes |
| NFPs “yes”, %g | 78.7 | 77.3 | 81.6 | 52.5 | 58.9 | 82.3 | |
| NFPs “no”, % | 19.1 | 17.7 | 14.2 | 39.0 | 34.0 | 14.9 | |
| NFPs “don’t know”, % | 2.1 | 5.0 | 4.3 | 8.5 | 7.1 | 2.8 | |
| 10 | Expert panel’s | NA | Undecided | Undecided | Undecided | Yes | Undecided |
| NFPs “yes”, %g | 63.8 | 39.7 | 74.5 | 83.0 | 85.8 | 78.0 | |
| NFPs “no”, % | 33.3 | 57.4 | 22.7 | 11.3 | 12.8 | 20.6 | |
| NFPs “don’t know”, % | 2.8 | 2.8 | 2.8 | 5.7 | 1.4 | 1.4 | |
IHR, International Health Regulations; NA, not applicable; NFP, National IHR Focal Point.
a The personal opinions of the individuals who responded on behalf of the NFPs were obtained by asking the question: Before using the decision instrument or any relevant guidance, please tell us whether you personally consider that the event should be reported to the World Health Organization.
b For criterion 1, the question was: Is the public health impact of the event serious?
c For criterion 2, the question was: Is the event unusual or unexpected?
d For criterion 3, the question was: Is there a significant risk of international spread?
e For criterion 4, the question was: Is there a significant risk of international travel or trade restrictions?
f For the notification decision, the question was: Does this event need to be notified to the World Health Organization under Article 6 of the IHR?
g Percentages indicate the proportions of NFPs who chose a given answer. The totals for each scenario may not equal 100% due to rounding.
Survey implementation
The survey was made available in the six languages used by the World Health Assembly via a secure web site (available at: https://www.surveymonkey.com/s.aspx?sm=GX8_2f9IUtDrEbMrNQpAFylg_3d_3d) and a printable version was provided on request. All 193 NFPs for which WHO had e-mail addresses were invited to participate by the University of Geneva Hospitals’ Infection Control Programme. Each State Party could submit only one completed survey. Access to the web survey was achieved by individualized links, which ensured that responses could be correctly attributed to a specific State Party. Participants were asked to base their notification assessment on only the information contained in the scenarios. Participation was voluntary and anonymous. The study was exempted from review by the WHO Research Ethics Review Committee.
Gold standard responses
To obtain an independent assessment of responses to survey items 2–6, seven internationally recognized public health experts recommended by the six WHO regions were invited to complete the survey. The names of the experts are listed in the acknowledgements section. Six of these experts had participated in the 2008 technical consultation on the implementation of Annex 2 of the IHR.9 A response was accepted as a “gold standard” response if it was supported by at least five members of the expert panel.
Analysis
The degree of consensus among NFPs on a particular survey item was expressed as the proportion of participants who chose the most commonly selected response for that item. The degree of concordance between NFPs and the expert panel was expressed as the proportion of participating NFPs who selected the panel’s standard response for that item, where applicable. The degree of consensus or concordance was arbitrarily defined as being high when more than 70% of respondents agreed on a given response. In addition, the degree of concordance between an individual NFP and the expert panel was assessed using a concordance score in which one point was awarded for each notification assessment that matched the “gold standard”.
Differences in continuous variables were assessed using the Wilcoxon rank-sum or signed-rank test or Student’s t test, if values were normally distributed. Differences between paired proportions were tested for statistical significance using McNemar’s test. The association between concordance scores and demographic variables was explored using multivariate linear regression and Stata statistical software, release 11 (StataCorp LP, College Station, United States of America). Economic and population data were extracted from the WHO Statistical Information System (2007 data, available at: http://www.who.int/whosis). The association between gross national income and survey responses was assessed by dividing countries into four groups on the basis of gross national income quartiles.
Results
Response rate
The survey of the NFPs’ use of Annex 2 of the IHR was completed by 142 States Parties between November 2009 and January 2010, giving a response rate of 74%. Three NFPs provided participants for both the expert panel and the survey, and one NFP participated in the expert panel only. Data on population and gross national income per capita were available for 134 of the States Parties who responded (94%) and 41 who did not (79%). There was no significant difference in population or gross national income per capita between those who did and did not respond.
Respondents
Half of the individuals who responded on behalf of NFPs were medical doctors and one-third defined themselves as epidemiologists. Twenty-six (18%) reported that they had not applied Annex 2 in the last 12 months, 72 (51%) said they had applied it less than once a month, 20 (14%) once every month, and 23 (16%) at least once a week.
Consensus among NFPs
The results of the NFPs’ notification assessments are shown in Table 1. The overall median degree of consensus among NFPs on whether WHO should be notified (item 6) was 78% (interquartile range, IQR: 55–82). The degree of consensus was high (i.e. > 70%) for six of the 10 scenarios: 1, 2, 3, 7, 9 and 10. For each of the corresponding events, most participating NFPs (median: 80%; range: 76–91) felt that the notification requirement had been met. For the remaining four scenarios, the NFPs’ judgements were much more divided (median degree of consensus: 55%, range 54–60). Overall, NFPs judged a median of 7 events (IQR: 6–9) notifiable under the IHR.
When the four criteria of the decision instrument were assessed individually, the degree of consensus was found to be high (i.e. > 70% of NFPs provided the same response for a criterion) on 24 of 36 occasions (67%). Application of the four criteria was not required for scenario 7 (i.e. a case of wild-type polio). On 10 occasions (28%), the degree of consensus among NFPs was minimal (i.e. < 60% for the most frequently chosen response). Five of these occasions, in scenarios 1, 2, 3, 5 and 9, concerned the third criterion (i.e. a significant risk of international spread).
Concordance between NFPs and the expert panel
The expert panel considered that events in five scenarios met the requirements for notification under the IHR: scenarios 1, 2, 3, 7 and 9 (see Table 1). Most of the NFPs (median: 82%; range: 76–91) concurred with the panel on the notifiability of these events. The median sensitivity of the NFPs in identifying an event considered notifiable by the expert panel was 4 out of 5 (IQR: 4/5 to 5/5). For the four events not deemed notifiable by the expert panel (i.e. in scenarios 4, 5, 6 and 8; Table 1), the degree of concordance between NFPs and the panel was considerably lower (median: 51%; range: 42–60). Correspondingly, the median specificity of the NFPs in identifying an event not considered notifiable by the expert panel was 2 out of 4 events (IQR: 1/4 to 3/4), which was significantly lower than the sensitivity (P < 0.001). Finally, for scenario 10 (i.e. a foot-and-mouth disease outbreak), there was a high degree of consensus among NFPs (i.e. 78% for notification), but disagreement among expert panel members.
The expert panel gave a clear opinion on whether or not an individual assessment criterion was fulfilled on 27 of 36 occasions (75%): on 11 occasions, individual criteria were considered fulfilled by the expert panel, while on 16 occasions they were considered “not met”. On the remaining nine occasions (25%), there was no clear majority among panel members (Table 1). For the 11 occasions on which criteria were considered fulfilled by the expert panel, most NFPs concurred (median: 81%; range 59–97); on nine of those 11 occasions, more than 70% of NFPs agreed. For the 16 occasions on which criteria were not considered fulfilled by the expert panel, the median proportion of NFPs concordant with the panel was 74%, but responses were highly variable (range: 14–96%). Overall, more than 70% of NFPs agreed with the expert panel on their judgment using the decision instrument criteria in 18 of 36 occasions (50%).
Application of Annex 2 of the IHR
According to Annex 2 of the IHR, events that “meet any two of the four criteria (I–IV) in the algorithm” are notifiable to WHO. The seven members of the expert panel adhered to this rule in their notification assessments wherever it was applicable (i.e. in all scenarios except scenario 7).
In contrast, 64% of NFPs disregarded this rule at least once and this affected 15% of all individual NFP notification assessments. For all scenarios except scenario 7, two or more criteria were considered fulfilled by 71% of the NFPs, but only 66% deemed the events to be notifiable under the IHR, while only 62% considered that the events should be notified to WHO solely on the basis of the respondents’ personal judgment (P < 0.001 for all comparisons by McNemar’s test). If all events for which two or more criteria were considered fulfilled were regarded as notifiable, the specificity of the NFPs’ assessments would have been significantly lower (median: 1 out of 4; IQR: 1/4 to 2/4; P < 0.001), but the sensitivity would not have been significantly higher.
Concordance score
For the nine scenarios (i.e. scenarios 1–9) on which the expert panel expressed a clear opinion on notifiability under the IHR, the mean concordance score achieved by individual NFPs was 6.19 (standard deviation: 1.33). Univariate analysis showed a significant association between the concordance score and both gross national income (P = 0.04) and self-reported frequency of use of Annex 2 (P = 0.015). The corresponding levels of significance of these two variables found on multivariate analysis were P = 0.03 and P = 0.005, respectively. However, the two variables alone explained only 15% of the observed variation in concordance score, which suggests that other unmeasured factors may have had a more important effect.
Discussion
The survey was completed by NFPs from nearly three-quarters of the States Parties to the IHR (2005). This suggests that there is considerable interest among NFPs in Annex 2 of the IHR and bears witness to the excellent support provided by WHO regional and country offices in implementing the survey. Although the degree of consensus between NFPs in assessing the 10 scenarios was variable, comparison with the “gold standard” responses provided by the expert panel showed that the sensitivity of the notification assessment process when used by NFPs to identify events considered notifiable by the expert panel was quite high. In contrast, the specificity was only moderate.
The scope of the IHR is intentionally broad and non-specific, which is a major strength that makes the IHR future-proof against new and unforeseeable threats.10 At the same time, the lack of specificity of the decision instrument in Annex 2 leaves considerable room for users’ perceptions, experience and knowledge to have an influence. Consequently, the disagreement observed between NFPs in assessing some events should not be surprising.
The level of agreement tended to be high when the implications of an event were “obvious” and could be judged using common sense. Examples of survey items that were not subject to dispute are the seriousness of drinking water being heavily contaminated by arsenic11 (scenario 2), the seriousness and unusualness of an infection that causes symptoms similar to SARS and results in deaths among health-care workers12 (scenario 1), and the absence of the risk of international spread or of international travel and trade restrictions due to accidental blast and burn injuries (scenario 8).
The level of agreement was lower when the applicability of the criteria was debatable in the absence of more detailed guidance. Thus, the NFPs disagreed on whether the moderate increase in dengue cases described in scenario 5 or the foot-and-mouth disease outbreak detailed in scenario 10 should be considered as posing a serious public health threat or not (i.e. the first criterion). The seriousness of the dengue scenario depends in part on how much the incidence increases.13 Consequently, a complete consensus on this criterion would only be achieved by defining epidemiological thresholds for seriousness. Foot-and-mouth disease was considered a notifiable event by many NFPs and even by some members of the expert panel, even though human cases of this animal disease are a rarity.14 Annex 2 may not be clear enough about how it should be applied to animal diseases. There was also a lack of consensus among NFPs and expert panel members on whether the local emergence of pan-resistant Klebsiella pneumoniae infection (scenario 4) was unusual or unexpected (i.e. the second criterion). The phenomenon of increasing antimicrobial resistance could, somewhat cynically, be called expected,15 but this particular scenario is unusual at present.16 Interestingly, the decision on whether the event in scenario 4 was notifiable hinged on the applicability of the second criterion, as most participants considered it as serious.
Agreement was also limited on the use of the third criterion: a significant risk of international spread. Scenario 1 (i.e. pneumonia with deaths among health-care workers) could represent the beginning of an outbreak of a viral infection that causes symptoms similar to SARS, with a clear potential for global spread.12 However, the examples given in the second part of Annex 2 put more weight on the circumstances of the event than on the known or expected properties of the disease agent (e.g. transmissibility) and suggest that only an imminent risk of spread is significant for the purposes of the IHR.1 The variability in the responses given when applying the fourth criterion in scenarios 1, 3 (i.e. HIV drug failure) and 9 (i.e. ochratoxin A in wheat) may in part be explained by the additional level of complexity introduced by the need to assess the risk of international travel or trade restrictions, which may be heavily influenced by political considerations.
These examples illustrate that users of Annex 2 have a considerable amount of freedom in judging events. In the present study, for example, the rule that the fulfilment of two criteria should lead to notification was not strictly followed. We found indications that the appropriate use of Annex 2 might increase the likelihood of certain events being notified to WHO: 66% of events were deemed notifiable under the IHR, while 62% were considered notifiable on the basis of personal judgment alone.
In accordance with the recommendations of the WHO technical consultation,9 the principal aim of this study was to assess the reliability of the responses given by public health professionals involved in the IHR notification process at the national level when applying Annex 2 of the IHR. In addition, we explored the validity of the assessments by reference to “gold standard” responses. Overall, the findings suggest that, in the hands of professionals, Annex 2 is a sensitive instrument for identifying events that are notifiable to WHO under the IHR. In contrast, a recent WHO audit revealed that, in practice, NFPs “are not yet a major source of early information to WHO on events”.6 Our findings indicate that this is unlikely to result from a malfunctioning of Annex 2 but instead may be due to barriers within countries. These barriers may include inadequate surveillance infrastructure or a poor flow of information within countries, perhaps resulting from limited resources or the administrative structure.17 In addition, political and economic considerations may also play a role.18,19 These considerations had already been recognized as important obstacles to reporting under the previous version of the IHR.19
Limitations
First, to some extent, this survey is a theoretical exercise since the results tell us only what respondents think should be notified under the IHR, not what would be notified in practice; for example, notification may depend on how much relevant information is available to NFPs or on political considerations. The sensitivity of the notification assessment process observed in this survey could have been increased by a social desirability bias, as participants may have felt that it was more acceptable to make an unnecessary notification than not to notify an event. Second, the condensation of real-life events into short fictitious scenarios necessarily involves a high degree of simplification. For example, real events develop over time and have to be re-assessed at intervals. It would be difficult for a global survey to reflect this. Moreover, the notification assessment process in the present study may also have been simplified because fewer individuals may have been involved than under real-life conditions. Third, the number of scenarios had to be limited to 10 to make the survey manageable for participants. Although the aim was that the scenarios should cover a large variety of public health risks, the selection may not be representative of the risks encountered in the real world. Fourth, NFPs may not necessarily play the key role in the risk assessment of an event occurring within the territory of a given IHR State Party. Assessment may involve decision-makers based outside the NFP,2,9 such as other public health experts within the country or at WHO, individuals from governmental agencies not normally involved in health matters, or politicians. Moreover, depending on the nature of the event, fundamentally different stakeholders may be involved in the notification assessment process. Ideally, all these individuals and organizations should have been involved in responding to this survey, but the practical barriers to their systematic inclusion were deemed too high. Therefore, the survey was addressed to the NFPs, the only national bodies that could be clearly identified and contacted by WHO. Fifth, due to the anonymous nature of the survey we do not know whether the answers we received from NFPs represent the views of individual risk assessors within the NFP, of the entire NFP team or of a group of collaborators that included national experts outside the NFP. Sixth, although a response rate of 74% was achieved, our findings may have been affected by a non-response bias. Finally, the “gold standard” proposed in this study cannot claim to provide responses that are universally correct or applicable to all settings.
Implications
The low specificity of the notification assessment process in Annex 2 when used by NFPs to identify an event not considered notifiable by the expert panel is not a major concern as long as the volume of notifications is low. However, WHO might want to consider whether guidance on the use of Annex 2 of the IHR should be expanded. For example, additional scenarios could be described and more specific criteria for common public health events and a clearer definition of terms such as “a significant risk of international spread” could be given. In addition, the variability observed in the use of the Annex 2 decision instrument by NFPs in this study and the low self-reported frequency of its application in the past suggest that the proficiency of the NFPs in using Annex 2 could be further improved.
Acknowledgements
We would like to thank all participating NFPs, all staff at WHO headquarters and at regional and country offices who assisted us in the development and implementation of the study, and Angèle Gayet-Ageron for statistical advice. We are grateful to Preben Aavitsland, Abdel Hakim AlKohlani, Eduardo Hage Carmo, Ambrose Talisuna, Theresa Tam, Enrique A Tayag and Kumnuan Ungchusak for serving on the expert panel.
Funding:
The study was funded by WHO.
Competing interests:
Helge Hollmeyer and Max Hardiman are employed by WHO at the International Health Regulations Coordination Department, WHO Headquarters, Geneva, Switzerland.
References
- 1.International Health Regulations (2005). 2nd ed. Geneva: World Health Organization; 2008. Available from: http://www.who.int/ihr/9789241596664 [accessed 27 September 2010]. [Google Scholar]
- 2.WHO interim guidance for the use of Annex 2 of the International Health Regulations (2005). Decision instrument for the assessment and notification of events that may constitute a public health emergency of international concern. Geneva: World Health Organization; 2008. Available from: http://www.who.int/ihr/Annex_2_Guidance_en.pdf [accessed 27 September 2010].
- 3.National IHR focal point guide. Designation/establishment of National IHR focal points Geneva: World Health Organization; 2006. Available from: http://www.who.int/ihr/English2.pdf [accessed 31 January 2011].
- 4.IHR training site [internet]. Geneva: World Health Organization; 2010. Available from: https://extranet.who.int/ihr/training [accessed 31 January 2011].
- 5.Securing global health: second IHR implementation course [internet]. Geneva: World Health Organization; 2010. Available from: http://www.who.int/ihr/training/ihrcourse [accessed 31 January 2011].
- 6.Implementation of the International Health Regulations (2005): report by the Director General Geneva: World Health Organization; 2009. Available from: http://apps.who.int/gb/ebwha/pdf_files/A62/A62_6-en.pdf [accessed 31 January 2011]. [Google Scholar]
- 7.Fifty-eighth World Health Assembly. Resolutions and decisions annex Geneva: World Health Organization; 2005. Available from: http://apps.who.int/gb/ebwha/pdf_files/WHA58-REC1/english/A58_2005_REC1-en.pdf [accessed 31 January 2011]. [Google Scholar]
- 8.Sixty-first World Health Assembly. Resolutions Geneva: World Health Organization; 2008. Available from: http://apps.who.int/gb/ebwha/pdf_files/WHA61-REC1/A61_Rec1-part2-en.pdf [accessed 31 January 2011]. [Google Scholar]
- 9.WHO technical consultation on the implementation and evaluation of Annex 2 of the International Health Regulations (2005). Geneva: World Health Organization; 2008 (WHO/HSE/IHR/2009.10). http://www.who.int/ihr/summary_report_annex2.pdf [accessed 31 January 2011].
- 10.Rodier G, Greenspan AL, Hughes JM, Heymann DL. Global public health security. Emerg Infect Dis. 2007;13:1447–52. doi: 10.3201/eid1013.070732. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Tchounwou PB, Centeno JA, Patlolla AK. Arsenic toxicity, mutagenesis, and carcinogenesis–a health risk assessment and management approach. Mol Cell Biochem. 2004;255:47–55. doi: 10.1023/B:MCBI.0000007260.32981.b9. [DOI] [PubMed] [Google Scholar]
- 12.Weinstein RA. Planning for epidemics–the lessons of SARS. N Engl J Med. 2004;350:2332–4. doi: 10.1056/NEJMp048082. [DOI] [PubMed] [Google Scholar]
- 13.Runge-Ranzinger S, Horstick O, Marx M, Kroeger A. What does dengue disease surveillance contribute to predicting and detecting outbreaks and describing trends? Trop Med Int Health. 2008;13:1022–41. doi: 10.1111/j.1365-3156.2008.02112.x. [DOI] [PubMed] [Google Scholar]
- 14.David W, Brown G. Foot and mouth disease in human beings. Lancet. 2001;357:1463. doi: 10.1016/S0140-6736(00)04670-5. [DOI] [PubMed] [Google Scholar]
- 15.Spellberg B, Guidos R, Gilbert D, Bradley J, Boucher HW, Scheld WM, et al. Infectious Diseases Society of America The epidemic of antibiotic-resistant infections: a call to action for the medical community from the Infectious Diseases Society of America. Clin Infect Dis. 2008;46:155–64. doi: 10.1086/524891. [DOI] [PubMed] [Google Scholar]
- 16.Elemam A, Rahimian J, Mandell W. Infection with panresistant Klebsiella pneumoniae: a report of 2 cases and a brief review of the literature. Clin Infect Dis. 2009;49:271–4. doi: 10.1086/600042. [DOI] [PubMed] [Google Scholar]
- 17.Wilson K, McDougall C, Fidler DP, Lazar H. Strategies for implementing the new International Health Regulations in federal countries. Bull World Health Organ. 2008;86:215–20. doi: 10.2471/BLT.07.042838. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Wilson K, von Tigerstrom B, McDougall C. Protecting global health security through the International Health Regulations: requirements and challenges. CMAJ. 2008;179:44–8. doi: 10.1503/cmaj.080516. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Baker MG, Fidler DP. Global public health surveillance under new international health regulations. Emerg Infect Dis. 2006;12:1058–65. doi: 10.3201/eid1207.051497. [DOI] [PMC free article] [PubMed] [Google Scholar]