A screening questionnaire for generalized tonic‐clonic seizures: Hospital‐based validation vs field‐validation method

Summary

The majority of the screening questionnaires for epilepsy have been validated in hospital settings. We previously developed and used for field validation a screening tool to detect generalized tonic‐clonic seizures (GTCS) in the rural communities of the Chaco region of Bolivia. The objective of the present study was to perform a hospital‐based validation of the same questionnaire and to compare the levels of accuracy obtained when validated in the field or in a hospital‐based context. We carried out a hospital‐based validation in the Hospital Hernandez Vera of Santa Cruz, Bolivia, where we enrolled patients affected by epilepsy with GTCS and controls. Sensitivity, specificity, and positive and negative predictive values (PPV and NPV) were calculated. One hundred twenty questionnaires were administered to 59 patients (27 men [45.8%]; mean age ± SD = 32.4 ± 14.2 years) and 61 controls (27 men [44.3%]; mean age ± SD = 32.6 ± 14.3 years). We obtained levels of accuracy of 100%. Sensitivity and PPV were significantly higher than the estimates obtained in the field‐validation study (sensitivity 100% vs 76.3%; PPV 100% vs 69.0%). Our screening questionnaire showed a significantly lower level of sensitivity when validated in the field, confirming that hospital‐based validation can lead to an overestimation of sensitivity.

1. INTRODUCTION

Epilepsy is one of the most prevalent noncommunicable neurologic diseases, affecting approximately 70 million people worldwide, of whom the majority live in low‐ and middle‐income countries (LMICs).1 Several epidemiologic studies have been carried out during the last decades in LMICs to estimate the prevalence of epilepsy, often using a two‐stage study design consisting of a screening phase during which the population is interviewed face‐to‐face through the use of validated screening questionnaires and a second phase in which positive subjects are evaluated by the neurologists.2, 3

Therefore, the choice of the screening tool is an important step of any epidemiologic study and the use of a validated instrument is recommended to quantify the potential inaccuracy of a screening test.

Several questionnaires to screen epilepsy have been developed and validated, showing on average a high sensitivity, generally higher than 90%, and lower specificity levels.3 However, as highlighted by a recent review, one of the main limitations of these screening tools is represented by the fact that the majority of them have been validated in hospital‐based settings, using affected‐cases vs unaffected‐controls study designs.2, 4, 5 This design may in fact overestimate sensitivity, due to the higher likelihood for the affected cases to be positive at screening, thus limiting the generalizability of the results.6

We recently developed and field‐validated a screening tool to specifically detect generalized tonic‐clonic seizures (GTCS).7 Our screening questionnaire was validated in the general population of the rural communities in the Chaco region of Bolivia, using a field‐validation design. It showed a level of sensitivity (76.3%) and specificity (99.6%) representing a potentially valuable instrument to screen the population living in the rural communities of Latin American countries. However, it did not reach the levels of accuracy usually obtained by other questionnaires that have been validated in a hospital‐based context.4, 5 The objective of the present study was to perform a hospital‐based validation of the screening questionnaire developed by our group7 and to compare the levels of accuracy obtained for the same questionnaire when validated in the field or in a hospital‐based context.

2. METHODS

2.1. Study population

Our study sample included 120 subjects, consecutively recruited in the neurology department of the Hospital Hernandez Vera of Santa Cruz de la Sierra, in the department of Santa Cruz de la Sierra. The group of patients with epilepsy were referred as outpatients of the center of epilepsy of the hospital. The group of controls was recruited among subjects attending as other outpatients of the same hospital for different neurologic diseases and for whom the diagnosis of epilepsy was excluded.

2.2. Screening tool

We used the screening tool already field‐validated by our group in the rural communities of Chaco region in Bolivia.7 The questionnaire is a modified and translated version of the questionnaire developed by Anand et al.5 The questionnaire is composed by two parts: a first screening question directed toward the proxy responder (the most reliable relative of the index subject) and the same screening question plus six confirmatory questions directed toward the index case (Appendix S1). Subjects answering YES to the first screening question were considered to be positive. The more‐specific features of the questionnaire have been described elsewhere.7

2.3. Validation

Questionnaires were administered to both patients and controls. The first part of the questionnaire was administered to a relative of the subject and the second part of the questionnaire was administered directly to the subject. All positive and negative subjects at the questionnaire underwent a neurologic evaluation to detect true‐positive (TP) false‐positive (FP), true‐negative (TN) and false‐negative (FN) subjects. The neurologic evaluation was performed by a neurologist (E.B.C.G.), who was blinded to the condition of the subjects. The administration of the questionnaires and their validation were performed between September and November 2016. The accuracy of both the indirect and the direct parts of the questionnaire was calculated. GTCS were defined according to the International League Against Epilepsy (ILAE) definition of 1993.8 Seizures were classified according to the most recent classification of seizures.9 The diagnosis of epilepsy was made according to the 2017 classification of epilepsies.10 The study was approved by the Bolivian Society of Neurology and by the ethics committee of the University Hospital “Policlinico Vittorio Emanuele” of Catania, in Italy (41/2016/PO). Verbal consent was obtained from the majority of the participants due to illiteracy. Written consent was obtained whenever possible. The study was developed in accordance with Standards for Reporting of Diagnostic Accuracy Studies guidelines.11

2.4. Statistical analysis

The questionnaires were collected and entered into an ad hoc created database (Excel 2013) by two investigators of the study (L.G., C.E.C.). Qualitative variables were described as percentages and quantitative variables as mean ± standard deviation (SD). Sensitivity, specificity, and positive and negative predictive values (PPV and NPV) were calculated for both the indirect and direct parts of the questionnaire. Pearson χ² test was used to compare the values of accuracy of the questionnaire obtained with either the field or the hospital‐based validation in two different samples; 95% confidence intervals (CI) were estimated. Data were analyzed using STATA 12 software packages (version 12.0; College Station, TX).

3. RESULTS

One hundred twenty questionnaires were administered to 59 patients (27 men [45.8%]; mean age ± SD = 32.4 ± 14.2 years) and 61 controls (27 men [44.3%]; mean age ± SD = 32.6 ± 14.3 years) and their relatives. Six controls and 10 patients were younger than 18 years of age.

No differences were found in the baseline characteristics of patients and controls.

For the patients, the proxy responders were the fathers in 14 cases, the mothers in 22, wives or husbands in 11, and brothers in 5 cases, and other subjects for the remaining 7 cases. In the group of controls, the proxy were the fathers in 18 cases, the mothers in 19, wives or husbands in 15, and brothers in 4 cases, and others for the remaining 5 cases.

All patients had convulsive seizures but a definite epilepsy diagnosis was available for 47 (79.7%): the majority (33; 55.9%) of the patients had focal to bilateral tonic‐clonic seizures with a known etiology, and the remaining had focal to bilateral tonic‐clonic seizures without a known etiology (8; 13.5%) and idiopathic GTCS (6; 10.2%). For 12 patients (20.3%) the information available allowed a diagnosis of undetermined epilepsy.9

When the questionnaire was indirectly performed to the relative of the enrolled subjects, one subject was incorrectly identified as negative (false negative) in the group of patients and one subject was incorrectly identified as positive (false positive) among controls, giving a sensitivity of 98.3% (95% CI 90.9‐99.6) and a specificity of 98.4% (95% CI 91.2‐100) with a PPV of 98.3% (95% CI 89.2‐99.7) and an NPV of 98.4% (95% CI 89.6‐99.8).

On the other hand, when the screening questionnaire was directly administered, neither false‐negative nor false‐positive subjects were found, giving a sensitivity and specificity of 100%.

The values of accuracy of the indirect and direct parts of the questionnaire were almost overlapping. Five subjects answered “I don't know” to the screening question, and thus they were not considered for the computation of the accuracy estimates. The accuracies of each single confirmatory question are shown in Table 1. Sensitivity and PPV obtained in the hospital‐based validation study were significantly higher when compared with the estimates obtained in our previous field‐validation study (sensitivity 100% vs 76.3%, P‐value <0.0001; PPV 100% vs 69.0%, P‐value <0.0001) as shown in Table 2.

Table 1.

Values of accuracy of the six confirmatory questions of the questionnaire

Hospital‐based validation	Sensitivity % (95% CI)	Specificity % (95% CI)	PPV % (95% CI)	NPV % (95% CI)
Have you ever peed or pooped during the attack?	41.0 (25.6‐57.9)	100 (94.1‐100)	100	72.6 (67.1‐77.5)
Have you ever bitten your tongue or got injured during the attack?	69.4 (54.6‐81.7)	100 (94.1‐100)	100	80.3 (72.7‐86.1)
Have you ever drooled during the attack?	88.9 (75.9‐96.3)	100 (94.0‐100)	100	92.3 (84.0‐96.5)
Have you ever had the attack while you were sleeping (night or early morning)?	95.0 (83.1‐99.4)	96.7 (88.6‐99.6)	95.0 (82.9‐98.7)	96.7 (88.4‐99.1)
Do you remember what happened during the attack?	92.6 (82.1‐97.9)	0 (0.0‐6.1)	45.9 (44.0‐47.7)	0.0
How often did you have this kind of attack?	92.3 (90.8‐100)	95.6 (78.1‐99.9)	92.3 (89.3‐99.7)	95.6 (75.9‐99.4)

CI, confidence interval; PPV, positive predictive value; NPV, negative predictive value.

Table 2.

Comparison between accuracy estimates of the questionnaire when field‐ or hospital‐based validated

Accuracy	Field validation	Hospital‐based validation	P valuea
Sensitivity % (95% CI)	76.3 (59.8‐88.6)	100 (93.6‐100)	<0.0001b
Specificity % (95% CI)	99.6 (99.4‐99.8)	100 (93.9‐100.0)	0.49
PPV % (95% CI)	69.0 (52.9‐82.4)	100	<0.0001b
NPV % (95% CI)	99.7 (99.5‐99.9)	100	0.54

CI, confidence interval; PPV, positive predictive value; NPV, negative predictive value.

^a“N‐1” Pearson chi‐square test.

^bStatistically significant.

4. DISCUSSION

The validation of a screening tool is an important process because, on the basis of a screening instrument, prevalences and incidences of diseases in the general population are estimated.12 There are two main strategies to perform a validation: in a hospital‐based setting or through a field‐validation method.13 In the first approach, patients with the disease and subjects that are known to be free from the disease are selected and the instrument under study is applied to both groups. This method is easy to perform; however, generalization of the results deriving from this approach is limited by the possible presence of selection bias. In fact, the most common methodology used in the hospital‐based validations is the case‐control design, comparing cases of known epilepsy to controls who are known not to have epilepsy. It is well known that such a study design inflates the diagnostic accuracy of a test. In fact, cases with more severe forms of disease and greater awareness of their condition have a higher chance of being enrolled; these subjects are more likely to be positive at the screening instrument, possibly leading to an overestimation of the of sensitivity.6 As a consequence, there is a real risk of obtaining incorrect estimates of the disease prevalence when applying a hospital‐validated questionnaire to the general population. Furthermore, socioeconomic and cultural differences between urban and rural areas are quite common, thus, a tool developed and validated in an urban setting could show a different level of accuracy if it is used in a different context. Therefore, to avoid selection bias, a screening tool should be validated in the general population. However, field validations are cost and time consuming13 because the screening questionnaire should be administered in a large sample of the population and, in order to detect false negative subjects, at least an appropriate random sample of screened negative subjects should also undergo a neurologic examination in the second phase.2, 7, 13, 14 This kind of validation provides more accurate estimates but is difficult to perform, above all in rural settings where the population is spread in large areas and specialists are rarely available to confirm the diagnosis. For these reasons, almost all the available questionnaires developed to detect people with epilepsy in the general population have been validated using hospital‐based designs.2 We recently developed and used for field validation a screening questionnaire for GTCS in the rural communities of the Chaco region in Bolivia. Our questionnaire was developed to detect GTCS because, as recently highlighted by the World Health Organization (WHO),3 the detection of epilepsy associated with GTCS is a priority in rural areas of LMICs due to its high level of associated comorbidities, injuries, and mortality.15, 16 The questionnaire has been developed to be used in a three‐stage design and showed a sensitivity of 76.3% and a specificity of 99.6%.

However, when validated in a hospital setting, the same questionnaire showed a sensitivity and a PPV significantly higher with respect to its field validation, confirming the presence of selection bias in the clinical‐based design. These findings, in fact, support the hypothesis that patients selected from a hospital context are more aware of their disease and more prone to correctly answer the questionnaire, thereby increasing the levels of sensitivity.

In conclusion, according to our experience, the use of hospital‐based validated screening instruments in epidemiologic studies could be a relevant source of incorrect estimates of the real burden of the diseases in the general population. Such a difference should be taken into account when a hospital‐based validated screening tool is used in a rural setting. When possible, field validation designs of a screening tool should be preferred to provide more accurate estimates.

DISCLOSURE

The authors have nothing to declare. We confirm that we have read the Journal's position on issues involved in ethical publication and affirm that this report is consistent with those guidelines.

Supporting information

 

Giuliano L, Cicero CE, Crespo Gómez EB, Sofia V, Zappia M, Nicoletti A. A screening questionnaire for generalized tonic‐clonic seizures: Hospital‐based validation vs field‐validation method. Epilepsia Open. 2019;4:339–343. 10.1002/epi4.12315