Validity of a Screening Instrument for Neurologic Disability in Resource-Poor African Communities

Abstract

Background

There have been no recent population-based studies on all-cause adult neurological morbidity in Sub Saharan Africa. We have developed a screening survey to improve the feasibility in performing these studies.

Methods

Our screening instrument contains both history questions and examination items. We pilot tested this instrument in the Hai District, Tanzania, and Butajira, Ethiopia using trained individuals from the local communities. To measure sensitivity, we applied the instrument blindly to 25 previously-identified subjects with Parkinson’s disease, stroke or epilepsy. To measure specificity, we examined 42 randomly selected previously screened subjects. We also compared the validity of the entire instrument to the history-only section.

Results

There were 669 adult subjects screened in both communities (150 screen-positives, and 519 screen-negatives). The sensitivity of the instrument was 100% (95% CI 84.2–100%) and the specificity was 82.4 % (95% CI 66.1–92.0%). However, when restricting the instrument to the history-only section, the sensitivity remained unchanged, but the specificity became 91.2% (95% CI 76.3–97.7%; p=0.48).

Conclusions

We have created a valid tool to screen adults for neurologic morbidity in resource-poor communities. The use of the history-only section of the tool is adequate as a screen and will improve feasibility.

Introduction

In 2003, the Institute of Medicine outlined six strategies to reduce the burden of brain disorders in the world. Strategy Four included the directive to “Conduct…epidemiological research to monitor the incidence, prevalence and disease burden of brain disorders in developing countries.”(1) This is especially true in Sub Saharan Africa where policy makers require accurate prevalence data to make evidenced-based decisions on scarce health care resource allocation.

The demand for this data will increase as the burden of neurologic disease in Africa changes. The nations of Africa are experiencing an epidemiologic transition.(2) As nations advance economically, the prevalence of infectious diseases decreases and non-communicable conditions increases. It is estimated that the population over age 50 years in Africa will increase from 83 million in 1990 to 186 million in 2015. (2) This will increase the prevalence of neurodegenerative diseases.

In the last ten years, there has been an increase in epidemiologic literature on neurologic disease from sub-Saharan Africa. This has included population-based data about individual diseases such as stroke,(3, 4) parkinsonism,(3, 5, 6) headache,(7–9) epilepsy,(3, 10–13) dementia and mental disorders(3, 14–16)and essential tremor.(17) There have also been hospital-based studies on all-cause neurologic morbidity.(18–21) These studies suggest that neurologic diseases make up 18–25% of medical admissions. The Global Burden of Disease Study suggests however that neurologic disease makes up only 3.9% of total disease burden in sub Saharan Africa. (22) The explanation for this large discrepancy remains unclear. There has only been one recent population-based study on all-cause neurologic morbidity, and that was in children.(23) The only two population-based assessments of adult neurologic impairment in Sub-Saharan Africa predated the HIV epidemic.(24, 25)

It is difficult to obtain this prevalence data on adult neurologic morbidity in Sub Saharan Africa. Demographic data on births and deaths are scarce and sometimes non-existent. Medical records are not computerized and are scattered amongst various clinics. There is a paucity of epidemiologic infrastructure.

In 1982, Schoenberg first discussed the difficulties in measuring the prevalence of neurologic disorders in resource-poor countries.(26) He and coworkers developed a two phased approach to community surveys where non medical personnel used a screening instrument to first screen a population, followed by a referral to a neurologist for examination if the individual screened positive.(27, 28)

We have previously reported the difficulties with that original screening instrument, and have shown in a Tanzanian outpatient setting that a new instrument we created significantly improved its specificity from 29.2% to 61.0% (p=0.001).(29) Like the original instrument, our instrument contained history questions and examination items. We have previously shown that a non medical interviewer can be taught to perform a simple neurologic exam(29)- but it is unclear whether that leads to improved sensitivity in the field. We now wish to present our results on the overall validity of our instrument as assessed in two diverse African communities, one in the Hai District, Tanzania and the other in Butajira, Ethiopia. This validation study was part of a greater pilot study to test the methods necessary to perform a population-based prevalence study of neurologic disability in these two communities. We present both the results of the entire screening instrument, and those of the history-only section of the instrument.

Methods

We created a screening instrument for adults (≥14 years) that consisted of 22 history questions, and 16 simple tasks (appendix). To improve specificity, ten questions had sub questions. At least one of the sub questions needed to be answered affirmatively for the overall question to be considered affirmative. If one history question was answered affirmatively, or one examination item was positive, that subject was considered a “Screen Positive.” (The instrument contained three additional questions on headache and spine pain, as part of a separate pilot study. These questions were not considered when determining screen status.) We administered the instrument in two diverse African communities, chosen because of their differing epidemiologic infrastructure, and diverse geographic, religious, language and cultural attributes. Ethical approval was obtained through the Institutional Review Boards of Mayo Clinic (Dr. Bower- for both sites), Kilimanjaro Christian Medical College (Tanzania), and Addis Ababa University School of Medicine (Ethiopia).

The Hai District is situated in the Kilimanjaro region of northern Tanzania. The project area is almost entirely rural and most of the population live on the slopes of Kilimanjaro. A demographic surveillance site (DSS) has been established in the area to measure census data. The current population is 161,000. Most of the population are subsistence farmers, but some cash crops such as coffee, are grown. The predominant language is Swahili, but Chagga, the language of the predominant local tribe, is also widely spoken. The main religion is Christianity.(30) The Hai project was completed in the eight-week period May 23- July 16, 2010.

Butajira, Ethiopia is situated 130 km south of the capital city Addis Ababa. It has a population of 259,689 with 87% residing in rural areas. The climate of the highlands is temperate and the climate of the lowlands is tropical. Farming of teff (a cereal staple), maize, millet, barley and legumes is the main mode of living. The main language is Guraje, and the majority follow the Islamic religion.(31) The Butajira Rural Health Program (BRHP) was established in 10 of the 82 villages in Butajira to study the demographics of the population. These ten villages contain 12,798 households with an estimated population of 54,096. The Butajira project was completed in the seven-week period April 17- June 3, 2011.

At each site, we recruited a nurse to act as a translator for the neurologist. In the Hai District, we chose one village large enough to recruit an adequate number of subjects, and recruited the two village health workers to act as screeners. They had no formal medical training other than basic first aid, but were familiar with the topography of the village. In Butajira, we chose two villages, one rural and one urban. We recruited two local BRHP workers who were also familiar with the topography of the villages. At both sites we discovered that if one household was screened, the neighbors also wanted to participate. We therefore chose to screen clusters of households. Inclusion criteria included everyone residing in that cluster on the prevalence day (defined as the first day of the screening period for each site). There were otherwise no exclusion criteria. Because we chose clusters of households to study, some areas of the three villages were not screened.

At every selected household, the screener obtained the names of the inhabitants from the head of the household. They screened every inhabitant in that household directly. (As part of a different pilot study, children (<14 years) were also screened using different instruments that have been described elsewhere.(32, 33)). They obtained signed consent from every subject. They returned to the household at designated times to complete the screening on individuals not home. Three attempts were made to screen the individuals before they were considered “never home.”

The intent of the screening instrument was to improve the feasibility of performing a population-based prevalence study of neurologic disease by reducing the workload of the neurologists- not to make a neurologic diagnosis. (For example, if a person complained of visual loss in the screen, they were not given a neurologic diagnosis. They were considered a screen positive and got a further neurologic examination. If the neurologist determined that the cause was cataracts, they were not given a neurologic diagnosis. If the loss was due to optic neuritis, they were given a neurologic diagnosis.)

This study was performed to test feasibility in recruitment, participation rates, productivity of screeners and neurologists, and to perform pilot studies on mobile phone use and pain surveys fro pain syndromes. These results will be reported separately. In this paper, we report on the testing of the validity of our adult survey.

To measure sensitivity, previously identified subjects with neurologic disability are required. This is difficult in a population-based setting as previously identified subjects are usually not known a priori. However, in the Hai District, previous epidemiologic surveys had already identified subjects with Parkinson disease, stroke and epilepsy. We randomly selected 25 of these cases. All 25 of these subjects lived in different villages from our screener; therefore he was not familiar with the subjects or their villages. He blindly administered the instrument to the 25 subjects. A neurologist (JHB) examined each subject after the screen to confirm the diagnoses. Amongst the 25 cases, if the instrument screened positive, that subject was classified a True Positive (TP). If the instrument screened negative, that subject was classified a False Negative (FN). Sensitivity was calculated as TP divided by the sum of TP and FN. Sensitivity could not be calculated in Butajira because there were no previously identified cases.

To measure specificity, after approximately 250 subjects were screened in Butajira, we used a random number generator to select 50 of these subjects. If the subject selected was a child, we used the next higher number associated with an adult to act as the subject for the specificity arm. The neurologist (JHB or SA) saw those subjects and determined if they had any neurologic disability (the gold standard). Of the 50 subjects, eight ( all screen negatives) were never examined by a neurologist (two refusals, six never home). Of the 42 remaining examined subjects, 33 had screened negative and nine had screened positive. If a subject had no disability and had screened negative, they were considered a True Negative (TN). If the subject had no disability and had screened positive, they were considered a False Positive (FP). Specificity was calculated as TN divided by the sum of TN and FP.

Sensitivity and Specificity were calculated with 95% Confidence Intervals using the modified Wald method.

Results

Table 1 shows the demographics of the two communities. There were 1153 subjects that were screened, of whom 669 (58.0%) were adults and thus were screened using our instrument. Of the 669 adults, 149 (22.3%) screened positive, and 520 (77.7%) screened negative. Butajira had a higher female:male ratio and children:adult ratio than the Hai District.

Table 1.

Demographics of the Two African Communities

	Hai District	Butajira	Total
Total #Subjects	657	519	1176
Refused Consent	6 (0.9%)	3 (0.6%)	9 (0.8%)
Never Home	6 (0.9%)	8 (1.5%)	14 (1.2%)
Total #Screened Subjects	645	508	1153
Children (<14 years)	206 (31.9%)	278 (54.7%)	484 (42.0%)
Adults (≥14 years)	439 (68.1%)	230 (45.3%)	669 (58.0%)
Median age adults (range), in years	35 (14–115)	28 (14–82)	33 (14–115)
Gender adults (female:male)	223:216	132:98	354:315

Table 2 shows our results on the validity of the entire screening instrument, including both the history and examination items. Of the 42 examined subjects in the specificity arm, 36 had screened negative, and six had screened positive.

Table 2.

Validity of the Entire Instrument

	TP	TN	FP	FN	Validity
Sensitivity	25	-	-	0	Sens1= 100% (95% CI 84.2–100%)
Specificity	3	28	6	5	Spec2= 82.4% (95% CI 66.1–92.0%)

¹Sens= TP/(TP + FN)

²Spec= TN/(TN + FP)

Table 3 shows our results on the validity of the history-only section of the instrument. The sensitivity of the instrument does not change. In the specificity arm of the study, three subjects had screened positive by exam only. When the neurologist evaluated these three subjects, they were not found to have neurologic disability. By eliminating the examination section, their status thus changed from false positive to true negative. This changed the specificity from 82.4% to 91.2% (p=0.48).

Table 3.

Validity of the History-Only Section of the Instrument

	TP	TN	FP	FN	Validity
Sensitivity	25	-	-	0	Sens1= 100% (95% CI 84.2–100%)
Specificity	3	31	3	5	Spec2= 91.2% (95% CI 76.3–97.7%)

¹Sens= TP/(TP + FN)

²Spec= TN/(TN + FP)

Discussion

We have created a valid instrument that can screen adults in resource-poor communities for neurologic disability. This is a crucial initial step in determining the true prevalence of neurologic disease in regions such as sub Saharan Africa, where the epidemiologic infrastructure is poor, but the need to have these data is great. Population-based data are a more accurate measure of true prevalence as compared to hospital-based data. Without the demographic and disease registries common in the West, door-to-door surveys will be necessary to measure prevalence. This instrument will better allow African neurologists to obtain these data.

Our instrument is highly sensitive in detecting subjects with Parkinson disease, epilepsy and stroke. Because the measurement of sensitivity requires previously recognized cases, it is difficult to test it in a community setting. We were therefore only able to test our instrument’s sensitivity to these three diseases. Furthermore, patients with more subtle partial epilepsy, mild parkinsonism and mild stroke or TIA may have been missed in the prior epidemiologic surveys, and may go undetected by our instrument. This is a common problem in all population-based epidemiologic surveys in Africa. However, we have previously shown that our instrument’s sensitivity in detecting 71 different neurologic diagnoses amongst 63 subjects in an outpatient clinic was 100% as well.(29)

Of course, by definition, these 25 cases were already aware of having their disease, which certainly improved their chances of screening positive. Individuals aware of having a disease are more likely to be aware of the symptoms of the disease, and thus would screen positive in a symptom-based screening instrument. Subjects unaware of having a disease may be more likely to screen falsely negative. We examined 42 randomly selected screen-negative subjects in the Hai District. Only three of those were found to have a neurologic disability, giving a false negative rate of only 3/42 (7.1%). In fact, one of the false negative cases was a result of a mistake in translation. The individual had had cerebral malaria two months prior to the screen but answered negatively to the question “Have you ever had fever with loss of consciousness in the last year?” We determined that the Swahili translation of the sentence did not differentiate between “in the last year” (i.e. the previous 12 months) and “last year” (i.e. the previous calendar year). He understood the question as being about the prior calendar year, and answered negatively. The translation was corrected. The other two false negative cases were individuals with essential tremor. Although present, the tremor caused them no concern. Some might argue that this shouldn’t be classified as a “disability” as it remained subclinical to the patients. We are confident that our instrument, when stringently translated, will detect close to 100% of neurologic disability in subjects living in a resource poor setting.

Equally as important, our instrument is also successful in screening out subjects with no neurologic disease, as evidenced by the high specificity. We randomly selected 50 screened subjects for the specificity arm of the study, and of those, only three were false positives, giving an overall FP rate of 6% in the community. The specificity was 91.2%. This means that our instrument was successful in screening out over 90% of those with no neurologic disability. This will greatly improve the feasibility of any prevalence study as the neurologist will not need to examine these subjects.

Finally, we found that the validity of the history-only section was not statistically different from the validity of the full instrument. Omitting the examination section will greatly improve the feasibility of future prevalence studies. The median time to administer the history section is six minutes, and the median time for the examination section is six minutes. By eliminating the examination, screeners will be able to see more people in a single day. Furthermore, for this pilot study we were fortunate to recruit four individuals who were all highly capable and trainable to perform a simple neurologic examination. In a more widespread study, this may not be the case. A poor examiner could greatly increase the number of false positives, and thus reduce the feasibility of the study. By eliminating the examination section, the training of the screeners is simplified, and the dependence on highly competent screeners is reduced.

One potential limitation of the study was that we did not perform a random selection of the households screened. Because neighbors wanted to be screened as well, we chose to screen households in a cluster. Therefore, some areas of the villages were not screened. On the other hand, we had no reason to believe that the population unscreened was in any way different in behavior or disease status than the population that was screened.