Abstract
Objective
To develop a simple tool for assessing the severity of disability resulting from Japanese encephalitis and whether, as a result, a child is likely to be dependent.
Methods
A new outcome score based on a 15-item questionnaire was developed after a literature review, examination of current assessment tools, discussion with experts and a pilot study. The score was used to evaluate 100 children in Malaysia (56 Japanese encephalitis patients, 2 patients with encephalitis of unknown etiology and 42 controls) and 95 in India (36 Japanese encephalitis patients, 41 patients with encephalitis of unknown etiology and 18 controls). Inter- and intra-observer variability in the outcome score was determined and the score was compared with full clinical assessment.
Findings
There was good inter-observer agreement on using the new score to identify likely dependency (Κ = 0.942 for Malaysian children; Κ = 0.786 for Indian children) and good intra-observer agreement (Κ = 1.000 and 0.902, respectively). In addition, agreement between the new score and clinical assessment was also good (Κ = 0.906 and 0.762, respectively). The sensitivity and specificity of the new score for identifying children likely to be dependent were 100% and 98.4% in Malaysia and 100% and 93.8% in India. Positive and negative predictive values were 84.2% and 100% in Malaysia and 65.6% and 100% in India.
Conclusion
The new tool for assessing disability in children after Japanese encephalitis was simple to use and scores correlated well with clinical assessment.
ملخص
الغرض
إعداد أداة بسيطة لتقييم شدة الإعاقة الناتجة عن الالتهاب الدماغي الياباني واحتمال أن يصير الطفل نتيجة لذلك عالة على غيره
الطريقة
أُعد حرز جديد للنتائج استناداً إلى استبيان مكوّن من 15 بنداً بعد مراجعة المؤلفات الطبية، وفحص أدوات القياس الحالية، ومناقشة الخبراء، وإجراء دراسة ارتيادية. واستخدم الحرز لتقييم 100 طفل في ماليزيا (56 مريضاً بالالتهاب الدماغي الياباني، و مريضين بالتهاب دماغي غير معروف، و 42 طفلاً من الشواهد) و 95 طفلاً في الهند (36 مريضاً بالالتهاب الدماغي الياباني، و 41 مريضاً بالتهاب دماغي غير معروف، و 18 طفلاً من الشواهد). وجرى تحديد الاختلافات بين المراقبين ولدى كل مراقب في تسجيل حرز النتائج، وقورن الحرز بالتقييم السريري الشامل.
الموجودات
كان هناك توافق جيد بين المراقبين على استخدام الحرز الجديد لتحديد احتمال حاجة الأطفال للإعالة (Κ= 0.942 للأطفال الماليزيين؛ Κ= 0.786 للأطفال الهنود) وكان هناك توافق جيد بالنسبة للمراقب الواحد (Κ= 1.000 و 0.902 على التوالي). بالإضافة إلى أن التوافق بين الحرز الجديد والتقييم الإكلينيكي كان جيداً أيضاً (Κ= 0.906 و 0.762 على التوالي). وكانتا حساسية ونوعية الحرز الجديد في تحديد الأطفال المرجح حاجتهم للإعالة هما 100% و 98.4% في ماليزيا، و 100% و 93.8% في الهند. وكانت القيّم التكهنية الإيجابية والسلبية هي 84.2% و 100% في ماليزيا، و 65.6% و 100% في الهند.
الاستنتاج
الأداة الجديدة لتقييم الإعاقة بين الأطفال بعد الإصابة بالالتهاب الدماغي الياباني كانت يسيرة الاستخدام وارتبط الحرز ارتباطاً جيداً مع التقييم السريري .
Résumé
Objectif
Développer un outil simple pour évaluer la gravité de l’invalidité résultant de l’encéphalite japonaise et, en conséquence, la probabilité de dépendance de l’enfant.
Méthodes
Une nouvelle échelle d’évaluation basée sur un questionnaire de 15 questions a été développée après une analyse de la littérature, l’examen des outils d’évaluation actuels, une discussion avec les experts et une étude pilote. L’échelle a été utilisée pour évaluer 100 enfants en Malaisie (56 patients atteints d’encéphalite japonaise, 2 patients atteints d’encéphalite d’étiologie inconnue et 42 contrôles) et 95 en Inde (36 patients atteints d’encéphalite, 41 patients atteints d’encéphalite d’étiologie inconnue et 18 contrôles). La variabilité inter et intra-observateurs dans l’échelle d’évaluation a été déterminée et l’échelle a été comparée avec une évaluation clinique complète.
Résultats
Il a été observé une bonne correspondance inter-observateurs dans l’utilisation de la nouvelle échelle pour identifier la dépendance probable (Κ=0,942 pour les enfants malais; Κ=0,786 pour les enfants indiens), ainsi qu’une bonne correspondance intra-observateurs (Κ=1,000 et 0,902, respectivement). De plus, la correspondance entre la nouvelle échelle et l’évaluation clinique a été bonne (Κ=0,906 et 0,762, respectivement). La sensibilité et la spécificité de la nouvelle échelle d’évaluation de la probabilité de dépendance des enfants ont été de 100 % et de 98,4 % en Malaisie et de 100 % et de 93,8 % en Inde. Les valeurs prédictives positives et négatives ont été de 84,2 % et de 100 % en Malaisie et de 65,6 % et de 100 % en Inde.
Conclusion
Le nouvel outil pour l’évaluation de l’invalidité des enfants après encéphalite japonaise a été simple à utiliser et les résultats présentent une bonne corrélation avec l’évaluation clinique.
Resumen
Objetivos
Diseñar una herramienta sencilla para valorar la gravedad de la incapacidad causada por la encefalitis japonesa y la posibilidad de que un niño sea dependiente como consecuencia de la misma.
Métodos
Se ha elaborado una nueva escala de resultados, basada en un cuestionario de 15 puntos, realizado tras una revisión bibliográfica, en el estudio de las herramientas de valoración actuales, en el debate con expertos y en un estudio preliminar. La escala se empleó para evaluar a 100 niños en Malasia (56 pacientes con encefalitis japonesa, 2 pacientes con encefalitis de etiología desconocida y 42 controles) y 95 en India (36 pacientes con encefalitis japonesa, 41 pacientes con encefalitis de etiología desconocida y 18 controles). Se determinó la variabilidad interobservador e intraobservador en la escala de resultados y se comparó la escala con una valoración clínica completa.
Resultados
Hubo un consenso interobservador favorable respecto a la utilización de la nueva escala para identificar la posible dependencia (Κ = 0,942 en el caso de los niños malasios; Κ = 0,786 para los niños indios) y un consenso intraobservador favorable ( Κ = 1,000 y 0,902, respectivamente). Además, el consenso entre la nueva escala y la valoración clínica también fue bueno (Κ = 0,906 y 0,762, respectivamente). La sensibilidad y la especificidad de la nueva escala para identificar a los niños que pueden ser dependientes fue del 100% y del 98,4% en Malasia, y del 100% y del 93,8% en India. Los valores predictivos positivos y negativos fueron del 84,2% y del 100% en Malasia, y del 65,6% y del 100% en India.
Conclusiones
La nueva herramienta de valoración de la incapacidad infantil tras la encefalitis japonesa fue fácil de usar y los resultados estaban relacionados con la valoración clínica.
Introduction
Neurological disability is a major problem among children in resource-poor countries but the true burden of disability is unknown because there is no simple and reliable way of measuring it.1 The ability to measure disease burden is especially important for Japanese encephalitis, which is a major cause of death and disability in Asia. The disease is caused by the mosquito-borne flavivirus, Japanese encephalitis virus, and is spreading. Recently, there have been large outbreaks in India and Nepal and it is estimated that there are 20 000 to 175 000 cases globally each year.2–5 Although vaccines against Japanese encephalitis have been available for many years, they have not been widely used, partly because policy-makers lack information about disease burden.4,6,7 Moreover, the proportion of patients reported to have severe sequelae after infection varies widely, from 19 to 71%.8–11 A major reason for this uncertainty is the lack of a standard method for assessing the outcome of Japanese encephalitis and other forms of acquired brain injury among children in resource-poor countries.
Even in industrialized countries, tools for assessing disability in children are not as well developed as for adults.12 The gold-standard method requires a large multidisciplinary team and involves multiple lengthy assessments over an extended period of time. Although some tools have recently been redeveloped for use in resource-poor settings, they often still require lengthy assessments by trained personnel.13,14 We set out to develop a simple score for assessing disability in children affected by Japanese encephalitis that can be applied by health-care workers with minimal training. We focused on whether the disability was likely to make a child dependent on others, because this is the key issue in terms of disease burden, as well as the single most important parameter for the children themselves. The score we developed, which has become known as the Liverpool Outcome Score,15 was field-tested at two sites in south-eastern Asia: Bellary in India and Sibu in Malaysia. It is also now being used in Bangladesh, Cambodia, Indonesia,16,17 the Lao People’s Democratic Republic and Viet Nam (S Hills, et al. unpublished data, 2008).
Methods
Setting
The new post-encephalitis disability assessment score was developed, piloted and tested in two different clinical settings representative of locations across Asia where Japanese encephalitis occurs: (i) the Vijayanagar Institute of Medical Sciences, which is a government hospital in Bellary in southern India with basic diagnostic facilities but no paediatric intensive care facilities that serves the city (population: 0.5 million) and district (population: > 2 million) of Bellary;18 and (ii) Sibu Hospital, which is a referral hospital in Sarawak, Malaysia, with full intensive care facilities that serves the town of Sibu (population: 250 000) and the central region of Sarawak (population: 650 000).19
The outcome score
A pilot version of the outcome score based on 20 questions was developed after a literature review and the examination of assessment tools used in developed and resource-poor countries, including the Ten Questions screening questionnaire for childhood disability, the Denver II child development screening test, the Paediatric Evaluation of Disability Inventory (PEDI) and three other assessment tools (Fig. 1).20–23 Written informed consent was obtained from the parent or guardian of each child. Approval for the study was granted by the ethics committees of the University of Liverpool in the United Kingdom of Great Britain and Northern Ireland and the Vijayanagar Institute of Medical Sciences in India, and the director of health of the state of Sarawak and the hospital director of Sibu Hospital in Sarawak.
Fig. 1.
The process of devising a new outcome score for assessing disability after Japanese encephalitis, India and Malaysia, 2005–2007
AES, acute encephalitis syndrome; AMPS, Assessment of Motor and Process Skills; JE, Japanese encephalitis.
Participants
After a pilot study in 2006 involving 51 children in India, the score questionnaire was revised and applied in its current 15-question format in 2006 to a cohort of children who had had Japanese encephalitis in Sibu, Malaysia,11 and to controls. The questionnaire is available at: http://liv.ac.uk/neuroscience/brain-infections/education_presentations.htm Subsequently in 2007, after minor modification and clarification of the questions, the score questionnaire was applied to a further cohort of children with suspected Japanese encephalitis (defined according to the World Health Organization surveillance standard definition)24 in Bellary, India. Children were invited to attend a follow-up assessment by post in Bellary and via the radio message system in Sarawak. Japanese encephalitis was confirmed using an enzyme-linked immunosorbent assay on cerebrospinal fluid and serum, and patients who tested negative were classified as having acute encephalitis syndrome of unknown etiology. These patients may also have had Japanese encephalitis but, because of sample timing, we were unable to confirm this. Controls were selected at both sites from the siblings of patients who were assessed using the new score and from children who were well and attending the outpatient department for non-neurological conditions.
Application of the score
The score questionnaire requires the assessor to ask the child’s parent or carer to answer direct questions about the child’s ability to perform various daily activities or functions, such as speaking and feeding, in comparison with other children of the same age in their community. It was decided to compare children with others because expected norms vary enormously across communities and no normative data are available. The child is also observed performing simple motor functions, as described in the questionnaire, available at: http://liv.ac.uk/neuroscience/brain-infections/education_presentations.htm
For each question, a set of possible answers scored from 2 to 5 is provided. A child whose response to a particular question is completely normal would score 5 for that question. One having minor sequelae that are reported, for example, as mild behavioural problems would score 4. A child having moderate sequelae that affect function but would not lead to dependence (e.g. difficulty walking) would scores 3. A child whose impairment is so great that it would lead to dependence in that setting (e.g. being unable to walk in rural India) would score 2.
Although impairments do change with time, particularly during childhood, it is difficult to predict the change.11 Consequently, for the purposes of the assessment tool, the child is classified on the basis of the individual evaluation alone. The final outcome score for each child, which ranges from I to V, corresponds to the lowest individual score recorded for any single question in the completed score sheet. For example, children whose impairment is severe enough in one domain to make them dependent will be dependent however well they might score in other domains. A score of I is given if the child has died; children who died were not considered further in this study. A score of II corresponds to a lowest single question score of 2 and indicates severe sequelae. Correspondingly, a score of III indicates moderate sequelae, IV indicates minor sequelae and V indicates full recovery.
Although the assessment tool can identify the specific domains in which each child has difficulty, for the purposes of health economic and epidemiological analyses it is more useful to dichotomize children as either “dependent” or “independent” (i.e. likely to be capable of independent living). Children with a score of II were classed as dependent, while those with a score of III to V were classed as independent. Scores in individual domains could also be examined and a total score ranging from 33–75 could be derived from the sum of all the individual scores, but these parameters were not assessed in this study.
Local doctors were trained to use the new outcome assessment tool by discussing cases and with the aid of a PowerPoint (Microsoft, Redmond, United States of America) teaching tool. In both India and Malaysia medical education and training is mainly conducted in English, hence English versions of the questionnaire forms were used.15 Although less than ideal, this was felt to be a practical approach as more than 20 languages are in use in Sibu and more than 6 languages, in Bellary. It was not felt appropriate to translate the written questionnaire into local languages as the written format of some of the languages used is a more formal format than that used in everyday speech. The new outcome score questionnaire was applied by junior physicians who were not otherwise involved in the study.
To investigate the inter-observer and intra-observer variability in the outcome score each child was assessed twice by each of two independent assessors. For practical reasons, these assessments were performed on the same day. However, assessors were unlikely to remember the classification they had given earlier because so many children were assessed in a single day: in Sibu the median number per day was 7 (range: 4–10); in Bellary, it was 6 (range: 2–9).
The new outcome score was validated by comparing each child’s score with the results of a full clinical consultation carried out on the same day. The consultation comprised an assessment by a physician, including history-taking and developmental and full neurological examinations, and an examination by a specialized occupational therapist using the Assessment of Motor and Process Skills,25 which has been validated internationally and cross-culturally for children aged 3 years and older. For children aged under 3 years, the doctor’s assessment alone was performed. Children were classified on the basis of the clinical assessment as having “severe” sequelae, which were likely to make the child dependent, or “moderate”, “minor” or “no” sequelae. The latter three categories were compatible with independent living. The clinical assessors were blinded to the outcome score and vice versa.
In both India and Malaysia, the presence of Japanese encephalitis virus infection was confirmed using standard local assays for detecting Japanese encephalitis virus-specific immunoglobulin-M antibody, as described previously.26,27
Statistical analysis
To give a measure of item redundancy and the internal consistency of the questionnaire, Cronbach’s α was determined during development of the assessment tool for both the pilot 20-question and the final 15-question scores.28 Inter- and intra-observer agreement for the new outcome score and the comparison between the new score and full clinical assessment were all assessed using the kappa (Κ) statistic and 95% confidence intervals (CIs) were computed using the large-sample modified formula.29 The sensitivity, specificity and positive and negative predictive values of the new assessment score relative to full clinical assessment were determined and their 95% CIs were computed using exact binomial formulae. Predictive validity was calculated as the correlation between the new score and clinical assessment. Data were analysed using SPSS version 15 (SPSS Inc., Chicago, United States of America).
Results
The new outcome score
Cronbach’s α was determined for the data on all children assessed using the pilot questionnaire in India and the results were used to revise the questionnaire and to produce the current 15-question version, shown in the questionnaire, available at: http://liv.ac.uk/neuroscience/brain-infections/education_presentations.htm
In Sibu, Malaysia, of the 72 children (78%) invited for a follow-up assessment, 56 attended and were evaluated using the 15-question outcome score. The children were assessed a median of 69 months (range: 6–114) after their acute illness. Their median age was 11 years (range: 5–20; interquartile range, IQR: 8–13) and 18 (32%) were female. Forty-two control children (median age: 8 years; range: 3–18; IQR: 6–10) were also assessed, as were two children who had initially been diagnosed with Japanese encephalitis but who were subsequently classified as having acute encephalitis syndrome of unknown etiology after a review of virology results.
The score questionnaire was then used in Bellary, India, in a cohort of 36 children with prior Japanese encephalitis (median age: 8.5 years; range: 4–15; IQR: 6–11; 19 [53%] female) and 41 with acute encephalitis syndrome of unknown etiology (median age: 7 years; range: 2–17; IQR: 5–10; 22 [55%] female). These children were assessed a median of 15 months (range: 1–38) after acute illness. In addition, 19 healthy control children were also assessed (median age: 7 years; range: 3–13; IQR: 4–11; 10 [53%] female).
Each assessment took approximately 10 minutes for individuals experienced in using the new score questionnaire. In total, 779 assessments were made with the new clinical score in 196 children. If problems were identified, they were discussed with carers and referrals were made to local agencies, where available.
Redundancy in the score questions
In the Malaysian cohort, Cronbach’s α for the 15-question outcome score was 0.927 for all observers and children combined and 0.894 for children who had had Japanese encephalitis. In addition, in the Indian cohort, Cronbach’s α was 0.787 for all observers and children combined and 0.849 for those who had had Japanese encephalitis, 0.708 for controls and 0.585 for those with acute encephalitis syndrome of unknown etiology. No significant improvement in the internal consistency of the questionnaire could be made by excluding any of the 15 items and inter-item correlations were acceptable (data not shown).
Inter- and intra-observer agreement
In the Malaysian cohort, there was very good inter-observer agreement (Κ = 0.714) on the outcome score when children were classified according to the severity of their sequelae; the intra-observer agreement was also very good (Κ = 0.943). Moreover, when children were classified according to the dichotomous outcome of being dependent (i.e. a final outcome score of II) or independent (i.e. a final outcome score of III, IV or V), inter-observer agreement was very good (Κ = 0.942) and intra-observer agreement was perfect (Κ = 1.000). In the Indian cohort, inter-observer agreement was moderate (Κ = 0.584) and intra-observer agreement was good (Κ = 0.799) when the severity of sequelae was examined and inter- and intra-observer agreement were good (Κ = 0.786) and very good (Κ = 0.902), respectively, when the dichotomous outcome was examined. Details of these results are shown in Table 1, Table 2, Table 3 and Table 4.
Table 1. Inter-observer agreementa for new 15-question outcome score for assessing post-encephalitis disability in children, Sibu, Malaysia, 2006.
| Observer 1 |
Observer 2 |
|||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Likely dependence | Final outcome score (sequelae) | Final outcome score (sequelae) |
Likely dependence |
|||||||
| II (severe) | III (moderate) | IV (mild) | V (none) | Total | Dependent | Independent | Total | |||
| Dependent | II (severe) | 18 | 2 | 0 | 0 | 20 | 18 | 2 | 20 | |
| Independent | III (moderate) | 0 | 23 | 4 | 3 | 30 | 0 | 180 | 180 | |
| IV (mild) | 0 | 0 | 9 | 13 | 22 | |||||
| V (none) | 0 | 3 | 5 | 120 | 128 | |||||
| Total | 18 | 28 | 18 | 136 | 200 | 18 | 182 | 200 | ||
| Kappa value | 0.714 (95% CI: 0.622–0.806)b | 0.942 (95% CI: 0.862–1.000)b | ||||||||
CI, confidence interval.
a Agreement is shown for outcomes classified both in terms of four severity levels of sequelae and in terms of a dichotomous outcome: dependent (i.e. final outcome score: II) or independent (i.e. final outcome score: III–V).
b Kappa values were interpreted as follows: 0.0–0.2, poor agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, good agreement; and 0.81–1.00, very good agreement.30
Table 2. Intra-observer agreementa for new 15-question outcome score for assessing post-encephalitis disability in children, Sibu, Malaysia, 2006.
| Observer 1 |
Observer 2 |
|||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Likely dependence | Final outcome score (sequelae) | Final outcome score (sequelae)b |
Likely dependenceb |
|||||||
| II (severe) | III (moderate) | IV (mild) | V (none) | Total | Dependent | Independent | Total | |||
| Dependent | II (severe) | 19 | 0 | 0 | 0 | 19 | 19 | 0 | 19 | |
| Independent | III (moderate) | 0 | 28 | 0 | 1 | 29 | 0 | 179 | 179 | |
| IV (mild) | 0 | 0 | 18 | 1 | 19 | |||||
| V (none) | 0 | 1 | 3 | 127 | 131 | |||||
| Total | 19 | 29 | 21 | 129 | 198 | 19 | 179 | 198 | ||
| Kappa value | 0.943 (95% CI: 0.897–0.988)c | 1.000c | ||||||||
CI, confidence interval.
a Agreement is shown for outcomes classified both in terms of four severity levels of sequelae and in terms of a dichotomous outcome: dependent (i.e. final outcome score: II) or independent (i.e. final outcome score: III–V).
b Two repeat scores were omitted and the inter-observer agreement for that item was calculated by comparing one score with the average of two from the second observer.
c Kappa values were interpreted as follows: 0.0–0.2, poor agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, good agreement; and 0.81–1.00, very good agreement.30
Table 3. Inter-observer agreementa for new 15-question outcome score for assessing post-encephalitis disability in children, Bellary, India, 2007.
| Observer 1 |
Observer 2 |
|||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Likely dependence | Final outcome score (sequelae) | Final outcome score (sequelae) |
Likely dependence |
|||||||
| II (severe) | III (moderate) | IV (mild) | V (none) | Total | Dependent | Independent | Total | |||
| Dependent | II (severe) | 25 | 4 | 4 | 2 | 35 | 25 | 10 | 35 | |
| Independent | III (moderate) | 0 | 31 | 7 | 4 | 42 | 1 | 153 | 154 | |
| IV (mild) | 0 | 10 | 14 | 15 | 39 | |||||
| V (none) | 1 | 3 | 6 | 63 | 73 | |||||
| Total | 26 | 48 | 31 | 84 | 189 | 26 | 163 | 189 | ||
| Kappa value | 0.584 (95% CI: 0.495–0.674)b | 0.786 (95% CI: 0.666–0.906)b | ||||||||
CI, confidence interval.
a Agreement is shown for outcomes classified both in terms of four severity levels of sequelae and in terms of a dichotomous outcome: dependent (i.e. final outcome score: II) or independent (i.e. final outcome score: III–V).
b Kappa values were interpreted as follows: 0.0–0.2, poor agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, good agreement; and 0.81–1.00, very good agreement.30
Table 4. Intra-observer agreementa for new 15-question outcome score for assessing post-encephalitis disability in children, Bellary, India, 2007.
| Observer 1 |
Observer 2 |
|||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Likely dependence | Final outcome score (sequelae) | Final outcome score (sequelae)b |
Likely dependenceb |
|||||||
| II (severe) | III (moderate) | IV (mild) | V (none) | Total | Dependent | Independent | Total | |||
| Dependent | II (severe) | 28 | 0 | 0 | 1 | 29 | 28 | 1 | 29 | |
| Independent | III (moderate) | 2 | 38 | 3 | 4 | 47 | 4 | 155 | 159 | |
| IV (mild) | 2 | 5 | 25 | 7 | 39 | |||||
| V (none) | 0 | 0 | 3 | 70 | 73 | |||||
| Total | 32 | 43 | 31 | 82 | 188 | 32 | 156 | 188 | ||
| Kappa value | 0.799 (95% CI: 0.729–0.868)c | 0.902 (95% CI: 0.818–0.987)c | ||||||||
CI, confidence interval.
a Agreement is shown for outcomes classified both in terms of four severity levels of sequelae and in terms of a dichotomous outcome: dependent (i.e. final outcome score: II) or independent (i.e. final outcome score: III–V).
b One repeat score was omitted and the inter-observer agreement for that item was calculated by comparing one score with the average of two from the second observer.
c Kappa values were interpreted as follows: 0.0–0.2 poor agreement; 0.21–0.40 fair agreement; 0.41–0.60 moderate agreement; 0.61–0.80 good agreement; and 0.81–1.00 very good agreement.30
Validation
Outcomes obtained using the 15-question score and clinical assessment were compared (Table 5). Four scores were used for each child: one from each of the two assessments carried out by each of the two observers. When the outcome compared was the severity of the sequelae, a moderate level of agreement was found between the new score and clinical assessment: Κ = 0.544 for the Malaysian cohort and Κ = 0.467 for the Indian cohort. When the outcome compared was the child being dependent or independent, very good agreement was found, with Κ = 0.906 and Κ = 0.762 for the Malaysian and Indian cohorts, respectively.
Table 5. Outcomes obtained with new 15-question outcome score for assessing post-encephalitis disability compared with the outcomes of clinical assessment in 196 children in Malaysia and India, 2006–2007.
| Outcome score assessmenta |
Clinical team assessment |
|||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Likely dependenceb | Final outcome score (sequelae) | Final outcome score (sequelae) |
Likely dependenceb |
|||||||
| II (severe) | III (moderate) | IV (mild) | V (none) | Total | Dependent | Independent | Total | |||
| Malaysia | ||||||||||
| Dependent | II (severe) | 32 | 0 | 2 | 4 | 38 | 32 | 6 | 38 | |
| Independent | III (moderate) | 0 | 28 | 19 | 11 | 58 | 0 | 362 | 362 | |
| IV (mild) | 0 | 9 | 25 | 6 | 40 | |||||
| V (none) | 0 | 7 | 46 | 211 | 264 | |||||
| Total | 32 | 44 | 92 | 232 | 400 | 32 | 368 | 400 | ||
| Kappa value | 0.544 (95% CI: 0.473–0.616)c | 0.906 (95% CI: 0.832–0.980)c | ||||||||
| India | ||||||||||
| Dependent | II (severe) | 40 | 11 | 7 | 3 | 61 | 40 | 21 | 61 | |
| Independent | III (moderate) | 0 | 23 | 49 | 19 | 91 | 0 | 318 | 318 | |
| IV (mild) | 0 | 6 | 51 | 13 | 70 | |||||
| V (none) | 0 | 0 | 37 | 120 | 157 | |||||
| Total | 40 | 40 | 144 | 155 | 379 | 40 | 339 | 379 | ||
| Kappa value | 0.467 (95% CI: 0.400–0.534)c | 0.762 (95% CI: 0.666–0.858)c | ||||||||
CI, confidence interval.
a Most children were assessed twice using the 15-question outcome score by each of two observers. Overall, there were 779 assessments in 100 children in Malaysia and 96 in India.
b Children were classified as likely to be dependent if their final outcome score was II and as independent if their final outcome score was III–V.
c Kappa values were interpreted as follows: 0.0–0.2 poor agreement; 0.21–0.40 fair agreement; 0.41–0.60 moderate agreement; 0.61–0.80 good agreement; and 0.81–1.00 very good agreement.30
The sensitivity and specificity of the new score in identifying children likely to be dependent, as determined by clinical assessment, were 100% (95% CI: 89.1–100) and 98.4% (95% CI: 96.5–99.4), respectively, in Malaysia and 100% (95% CI: 91.2–100) and 93.8% (95% CI: 90.7–96.0), respectively, in India. The positive predictive values were 84.2% (95% CI: 68.7–94.0) and 65.6% (95% CI: 52.3–77.3) for the Malaysian and Indian cohorts, respectively, and the negative predictive values were 100% (95% CI: 98.6–100) and 100% (95% CI: 98.5–100), respectively. Overall only 3.8% of children categorized as independent on clinical assessment were incorrectly classified by the outcome score as dependent.
Discussion
The inability to measure disability using a simple tool has been identified as one of the key reasons for the lack of data on disease burden among children living in poor countries.1 The resulting gaps in knowledge mean that there is often insufficient evidence to drive changes in public health policy.31 Nothing provides a better example of this problem than the failure to control Japanese encephalitis over the past 40 years. Without good data on disease burden, the impetus to implement vaccination programmes has been haphazard. As more vaccines become available and as they become cheaper, countries will have to make important decisions about public health priorities.32,33 In particular, simple reliable ways of measuring disability are needed for diseases such as Japanese encephalitis, whose morbidity rate is much higher than the 8–30% mortality rate.10,11
Our aim was to design and validate a disability assessment tool that can be applied relatively quickly and easily by a range of health-care workers in different settings. None of the currently available scores, such as the Ten Questions, Denver II or PEDI score, meets this need. The Ten Questions was devised as a community screening tool to identify children who should be referred for neurological assessment but is too nonspecific for use as an assessment tool.34 The Denver II tool assesses disability in children and is widely used in Europe and North America.21 However, it is usually applied by paediatricians and requires at least 35 minutes. It is also dependent on the cultural setting, though it has recently been adapted for use in Malawi.14 The PEDI is another well-established and widely-used tool. However, it was designed for use in the developed world.20 Finally, the World Health Organization Disability Assessment Schedule II (WHO DAS II), which is in development, assesses patients’ needs, functioning and outcomes but is designed for an adult population.35
In developing the new outcome score we faced considerable challenges and had to accept many compromises. We had to accept that a scoring system would never match an assessment performed over several months by a multidisciplinary team. However, it would still be better than the disease outcome “discharged alive” so often recorded in hospital notes in many parts of rural Asia. We chose to focus on a single disease, Japanese encephalitis, because it is one of the most important causes of acquired brain injury in Asian children. However, the brain injury resulting from Japanese encephalitis is very similar to that associated with other infectious or noninfectious causes, such as trauma. Most disability assessment tools are generic and, with further validation, the new score can perhaps be used more generally across the spectrum of acquired neurological disability.
One limitation of our study was that the proportion of children that responded to a request to attend a follow-up assessment was limited, especially in rural India, where distances to hospital are great. However, we felt it was important to develop the score in real-life settings where it will be used in practice rather than in the logistically easier, but less relevant, setting of a large teaching hospital. We were concerned that sicker and more disabled children might not be able to attend follow-up assessments, but our visits to rural villages to track down nonattendees indicated that it was those who recovered fully that were less likely to attend. Ideally the new score would have been compared with a full multidisciplinary team assessment performed over several visits, but again this was not practical: even assessment by the clinician and occupational therapist took 60–90 minutes.
One of the challenges was to develop a single scoring system that could be applied in a wide range of age groups, in different settings and in areas where there are no normative data. Our solution was to ask the caregiver to compare the child with other children of the same age in the same community. Although this is a crude measure that is dependent on the caregiver, a parent’s judgement of a child’s level of development and abilities is usually correct. This approach allows for cultural differences across Asia; for example, Indian children feed themselves at a younger age than Malaysian children. Cultural differences and the child’s living conditions could mean that an inability to walk would make the child dependent in one setting, for example in rural India, but not another, for example in urban Malaysia, where wheelchairs are available. We felt this was a pragmatic approach because, when looking at disease burden, the impact of a disability is more important than neurological observations or biological dysfunction.
We did not attempt to classify or quantify disablement in terms of impairment, disability (i.e. in activity) or handicap (i.e. in participation). Rather, we developed an assessment tool that identifies children who, after having Japanese encephalitis, suffer a loss of functional ability compared to their peer group. For practical reasons, junior physicians applied the tool in our study, though other health-care workers have now used it without difficulty (unpublished observations). The tool is, if anything, oversensitive in predicting disability, but only 3.8% of children were incorrectly classified as dependent by the outcome score. We felt this was a reasonable proportion since we wanted to ensure that no dependent child was missed.
Recent data show that children with Japanese encephalitis may improve or deteriorate many months after the initial insult.11 Consequently, further work needs to be done in following up a prospective cohort to determine the correlation between the outcome score at hospital discharge with that 3 months and 3 to 5 years later. This information will enable us to determine the time at which the new outcome score will give the best prediction of long-term outcome. In addition, the test–retest reliability of the score now needs to be examined, as does its sensitivity to change over time and its potential for use in acute brain injury due to other causes.
In summary, we have developed a simple outcome score for detecting disability in children affected by Japanese encephalitis, a common cause of acquired neurodisability in Asia. Although the tool has limitations, its ability to identify children with “likely disability”, as judged by the clinical team, was good, with good to very good inter- and intra-observer agreement. It is now being used in several Asian countries affected by Japanese encephalitis and should be suitable for modification to assess acquired neurodisability due to other causes in children in resource-poor countries.
Acknowledgements
We are grateful to all the patients and their carers for assisting with this work. Thanks are also due to staff and patients at the Vijayanagar Institute of Medical Sciences, Bellary, who assisted with the study, and, in particular, the Medical Director and also Kailash Soni and Thomas Schulz. In Malaysia, we thank the hospital staff and patients at Sibu Hospital. We also thank JM Lewthwaite. We also thank Julie Jacobson and Susan Hills of the PATH Japanese Encephalitis Project for their support and encouragement, Mary Gainsborough for help with the initial development of the score and Janet Daly for help with the manuscript. Mong How Ooi and Rachel Kneen also work with the Brain Infections Group, University of Liverpool, Liverpool, United Kingdom and Tom Solomon is Chair of Neurological Science at the Walton Centre for Neurology and Neurosurgery, Liverpool, United Kingdom.
Funding:
The project was supported by the Bill & Melinda Gates Foundation-funded Japanese encephalitis programme at PATH. Mong How Ooi is funded by the Wellcome Trust in the United Kingdom. Tom Solomon is funded by the United Kingdom Medical Research Council.
Competing interests:
None declared.
References
- 1.Mung'ala-Odera V, Newton CR. Identifying children with neurological impairment and disability in resource-poor countries. Child Care Health Dev. 2007;33:249–56. doi: 10.1111/j.1365-2214.2006.00714.x. [DOI] [PubMed] [Google Scholar]
- 2.Immunization and vaccine development: Japanese encephalitis. New Delhi: World Health Organization Regional Office for South-East Asia; 2006. Available from: www.searo.who.int/en/section1226/section2073.asp [accessed 20 January 2010].
- 3.Tsai TF. New initiatives for the control of Japanese encephalitis by vaccination: minutes of a WHO/CVI meeting, Bangkok, Thailand, 13–15 October 1998. Vaccine. 2000;18(Suppl 2):1–25. doi: 10.1016/S0264-410X(00)00037-2. [DOI] [PubMed] [Google Scholar]
- 4.Solomon T. Control of Japanese encephalitis — within our grasp? N Engl J Med. 2006;355:869–71. doi: 10.1056/NEJMp058263. [DOI] [PubMed] [Google Scholar]
- 5.Halstead SB, Jacobson J. Japanese encephalitis. Adv Virus Res. 2003;61:103–38. doi: 10.1016/S0065-3527(03)61003-1. [DOI] [PubMed] [Google Scholar]
- 6.Beasley DW, Lewthwaite P, Solomon T. Current use and development of vaccines for Japanese encephalitis. Expert Opin Biol Ther. 2008;8:95–106. doi: 10.1517/14712598.8.1.95. [DOI] [PubMed] [Google Scholar]
- 7.Ding D, Kilgore PE, Clemens JD, Wei L, Zhi-Yi X. Cost-effectiveness of routine immunization to control Japanese encephalitis in Shanghai, China. Bull World Health Organ. 2003;81:334–42. [PMC free article] [PubMed] [Google Scholar]
- 8.Richter RW, Shimojyo S. Neurologic sequelae of Japanese B encephalitis. Neurology. 1961;11:553–9. doi: 10.1212/wnl.11.7.553. [DOI] [PubMed] [Google Scholar]
- 9.Kumar R, Mathur A, Singh KB, Sitholey P, Prasad M, Shukla R, et al. Clinical sequelae of Japanese encephalitis in children. Indian J Med Res. 1993;97:9–13. [PubMed] [Google Scholar]
- 10.Rayamajhi A, Singh R, Prasad R, Khanal B, Singhi S. Study of Japanese encephalitis and other viral encephalitis in Nepali children. Pediatr Int. 2007;49:978–84. doi: 10.1111/j.1442-200X.2007.02495.x. [DOI] [PubMed] [Google Scholar]
- 11.Ooi MH, Lewthwaite P, Lai BF, Mohan A, Clear D, Lim L, et al. The epidemiology, clinical features, and long-term prognosis of Japanese encephalitis in central Sarawak, Malaysia, 1997–2005. Clin Infect Dis. 2008;47:458–68. doi: 10.1086/590008. [DOI] [PubMed] [Google Scholar]
- 12.Jones P, Drummond A. Occupational therapy for children with acquired brain injury: a review of the literature. Br J Occup Ther. 2005;68:324–30. [Google Scholar]
- 13.Carter JA, Lees JA, Murira GM, Gona J, Neville BG, Newton CR. Issues in the development of cross-cultural assessments of speech and language for children. Int J Lang Commun Disord. 2005;40:385–401. doi: 10.1080/13682820500057301. [DOI] [PubMed] [Google Scholar]
- 14.Gladstone MJ, Lancaster GA, Jones AP, Maleta K, Mtitimila E, Ashorn P, et al. Can Western developmental screening tools be modified for use in a rural Malawian setting? Arch Dis Child. 2008;93:23–9. doi: 10.1136/adc.2006.095471. [DOI] [PubMed] [Google Scholar]
- 15.Japanese encephalitis teaching resources and outcome score. Liverpool: University of Liverpool; 2009. Available from: http://liv.ac.uk/neuroscience/brain-infections/education_presentations.htm [accessed 28 January 2010].
- 16.Liverpool outcome score for assessing children at discharge. Seattle: PATH; 2009. Available from: http://www.path.org/vaccineresources/details.php?i=676 [accessed 28 February 2009].
- 17.Maha MS, Moniaga VA, Hills SL, Widjaya A, Sasmito A, Hariati R, et al. Outcome and extent of disability following Japanese encephalitis in Indonesian children. Int J Infect Dis. 2009;13:e389–93. doi: 10.1016/j.ijid.2009.01.009. [DOI] [PubMed] [Google Scholar]
- 18.Health: general information and important statistics. Bellary: National Informatics Centre, India; 2010. Available from: http://www.bellary.nic.in/Health.htm [accessed 31 May 2008].
- 19.Department of Statistics. Population and vital statistics. Yearbook of statistics. Sarawak 2006. Kuala Lumpur: DS;2006. [Google Scholar]
- 20.Haley SM, Coster WJ, Ludlow LH, Haltiwanger JT, Andrellos PJ. Pediatric evaluation of disability inventory: development, standardization, and administration manual, version 1.0. Boston: Trustees of Boston University, Health and Disability Research Institute; 1992. [Google Scholar]
- 21.Frankenburg WK, Dodds J, Archer P, Shapiro H, Bresnick B. The Denver II: a major revision and restandardization of the Denver Developmental Screening Test. Pediatrics. 1992;89:91–7. [PubMed] [Google Scholar]
- 22.Durkin MS, Davidson LL, Desai P, Hasan ZM, Khan N, Shrout PE, et al. Validity of the ten questions screened for childhood disability: results from population-based studies in Bangladesh, Jamaica, and Pakistan. Epidemiology. 1994;5:283–9. doi: 10.1097/00001648-199405000-00005. [DOI] [PubMed] [Google Scholar]
- 23.Solomon T, Dung NM, Kneen R, Thao le TT, Gainsborough M, Nisalak A, et al. Seizures and raised intracranial pressure in Vietnamese patients with Japanese encephalitis. Brain. 2002;125:1084–93. doi: 10.1093/brain/awf116. [DOI] [PubMed] [Google Scholar]
- 24.WHO recommended standards for surveillance of selected vaccine-preventable diseases. Geneva: World Health Organization; 2003. Available from: http://www.who.int/vaccines-documents/DocsPDF06/843.pdf [accessed 29 January 2010].
- 25.Robinson SE, Fisher AG. A study to examine the relationship of the Assessment of Motor and Process Skills (AMPS) to other tests of cognition and function. Br J Occup Ther. 1996;59:260–3. [Google Scholar]
- 26.Cardosa MJ, Wang SM, Sum MS, Tio PH. Antibodies against prM protein distinguish between previous infection with dengue and Japanese encephalitis viruses. BMC Microbiol. 2002;2:9. doi: 10.1186/1471-2180-2-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Ravi V, Desai A, Balaji M, Apte MP, Lakshman L, Subbakrishna DK, et al. Development and evaluation of a rapid IgM capture ELISA (JEV-Chex) for the diagnosis of Japanese encephalitis. J Clin Virol. 2006;35:429–34. doi: 10.1016/j.jcv.2005.11.004. [DOI] [PubMed] [Google Scholar]
- 28.Streiner DLN, Norman GR. Health measurement scales: a practical guide to their development and use. 3rd ed. Oxford: Oxford University Press; 2003. [Google Scholar]
- 29.Fleiss JL, Cohen J, Everitt BS. Large sample standard errors of kappa and weighted kappa. Psychol Bull. 1969;72:323–7. doi: 10.1037/h0028106. [DOI] [Google Scholar]
- 30.Altman DG. Practical statistics for medical research. Padstow: TJ International; 1993.
- 31.Maulik PK, Darmstadt GL. Childhood disability in low- and middle-income countries: overview of screening, prevention, services, legislation, and epidemiology. Pediatrics. 2007;120(Supplement 1):S1–55. doi: 10.1542/peds.2007-0043B. [DOI] [PubMed] [Google Scholar]
- 32.Solomon T. New vaccines for Japanese encephalitis. Lancet Neurol. 2008;7:116–8. doi: 10.1016/S1474-4422(08)70004-8. [DOI] [PubMed] [Google Scholar]
- 33.Japanese encephalitis vaccines. Wkly Epidemiol Rec. 2006;81:331–40. [PubMed] [Google Scholar]
- 34.Durkin MS, Hasan ZM, Hasan KZ. The ten questions screen for childhood disabilities: its uses and limitations in Pakistan. J Epidemiol Community Health. 1995;49:431–6. doi: 10.1136/jech.49.4.431. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.World Health Organization Disability Assessment Schedule II (WHODAS II). Geneva: World Health Organization; 2001. Available from: http://www.who.int/icidh/whodas/ [accessed 25 October 2009]. [Google Scholar]