Abstract
Background
Two simples scoring systems for a self-completed postal respiratory questionnaire were developed to identify adults who may have obstructive airways disease. The objective of this study was to validate these scoring systems.
Method
A two-stage design was used. All adults in two practice populations were sent the questionnaire and a stratified random sample of respondents was selected to undergo full clinical evaluation. Three respiratory physicians reviewed the results of each evaluation. A majority decision was reached as to whether the subject merited a trial of obstructive airways disease medication. This clinical decision was compared with two scoring systems based on the questionnaire in order to determine their positive predictive value, sensitivity and specificity.
Results
The PPV (positive predictive value) of the first scoring system was 75.1% (95% CI 68.6–82.3), whilst that of the second system was 82.3% (95% CI 75.9–89.2). The more stringent second system had the greater specificity, 97.1% (95% CI 96.0–98.2) versus 95.3% (95% CI 94.0–96.7), but poorer sensitivity 46.9% (95% CI 33.0–66.8) versus 50.3% (95% CI 35.3–71.6).
Conclusion
This scoring system based on the number of symptoms/risk factors reported via a postal questionnaire could be used to identify adults who would benefit from a trial of treatment for obstructive airways disease.
Background
The aim of the study was to validate two simple scoring systems for a self-completed postal questionnaire designed to identify adult patients likely to have asthma/COPD (obstructive airways disease).
It has been reported that asthma is under-diagnosed and under-treated in adults [1,2]. In the short term this probably leads to increased morbidity for sufferers, and in the long term it may have a detrimental effect on their lung function and clinical state [3]. Chronic obstructive pulmonary disease is also thought to be under-diagnosed in adults [4]. This may lead to increased morbidity and the loss of an effective opportunity for giving smoking cessation advice. There may be clinical benefits to the individual and health economic benefits to society from identifying and treating patients who have asthma or COPD and who are unknown to the medical services. Before this hypothesis can be tested, however, mechanisms for identifying such patients need to be established. Full clinical review of everyone in a community is neither cost efficient nor practical. An alternative approach is to use a screening questionnaire, designed to identify individuals most likely to have the condition, and therefore most likely to benefit from clinical review.
The Wythenshawe Community Asthma Project (WYCAP) is a long term prospective study of the natural history of respiratory symptoms in two general practice populations in Manchester, UK. One of its aims is to develop a method for identifying patients in the community with asthma and COPD as the first stage in evaluating the benefits and costs of treating such individuals. In 1993 and 1995, postal respiratory questionnaire surveys were carried out in both practice populations. We did not feel that it would be possible to differentiate asthma from COPD using a simple postal questionnaire and so two simple scoring systems were developed, based on the number of symptoms/risk factors reported from key questions, both having the aim of identifying adults with either asthma or COPD. This paper examines the validity of these scoring systems, using data relating to respondents to the 1995 postal survey. A similar exercise has been performed on children in the two practices [5]. This study had ethical approval from the local research ethics committee and informed consent was obtained from all subjects attending for clinical review.
Method
In September 1995 postal questionnaires [2] were sent to all adults registered with two practices in South Manchester, with reminders sent after four and eight weeks to those who had not yet responded. From the respondents, a stratified random sample of adults was selected to undergo full clinical examination. To determine the sample, adults were first stratified according to the number of symptoms/risk factors reported from six key questions (Table 1).
Table 1.
Six key questions from the respiratory questionnaire and two markers of disease severity
| Six key questions: |
| 1) Have you had wheezing or whistling in your chest at any time*? |
| 2) Have you woken up with a feeling of tightness in your chest*? |
| 3) Have you been woken by an attack of shortness of breath at any time*? |
| 4) Have you been woken by an attack of coughing at any time*? |
| 5) Has any person in your family (parents, grandparents, sisters, brothers, or your children) had asthma? |
| 6) Have you ever had hay fever or eczema? |
| *(in the last 12 months) |
| Markers of severity: |
| a) Breathlessness when wheezing was present. |
| b) Wheezing or whistling without a cold. |
Random samples were taken from each of five strata (those with one to three positive responses were aggregated for sampling purposes). The sampling fraction for each stratum was chosen in advance in order to give approximately equal numbers of subjects with and without obstructive airways disease in the overall sample. As this was a new scoring system, estimates of the expected prevalence of obstructive airways disease in each stratum had to be derived empirically from the researchers' clinical experience. The number from each stratum invited to the clinical review and the number who attended is shown in Table 2. Subjects were invited via telephone or home visit by their general practitioner to attend their local hospital for clinical review.
Table 2.
Response categories of respondents, those invited for and those attending clinical review
| Number of symptoms/risk factors | 0 | 1–3 | 4 | 5 | 6 | Total |
| All respondents | 2158 | 3765 | 521 | 355 | 148 | 6947 |
| Number invited for clinical review | 60 | 60 | 150 | 105 | 45 | 420 |
| Number who attended review (% of those invited) | 26 (43.3) | 30 (50.0) | 75 (50.0) | 53 (50.5) | 18 (40.0) | 202 (48.1) |
Clinical assessment
The clinical assessment (carried out by a clinician, TLF) involved a structured medical history and physical examination. A research assistant performed the investigations, supervised and assisted by the clinician. Neither individual had access to the postal questionnaire results for the subject being examined.
The investigations performed have been described in a previous study [5] and included forced expiratory volume in one second (FEV1), forced vital capacity (FVC), FEV1 / FVC, a peak expiratory flow diary, skin prick testing using the skin prick/puncture technique [6] and reversibility testing to β2 agonists [7]. A bronchial challenge test to histamine was also performed by the YAN technique [8]. A positive histamine challenge was defined as a fall in FEV1 of 20% from the pre challenge FEV1.
For each subject, three independent consultant chest physicians were sent full details of the clinical assessment but excluding the details of the postal questionnaire. The consultants were asked to decide whether the subject merited a trial of asthma/COPD (obstructive airways disease) medication. Patients warranting a trial of treatment were said to have a diagnosis of possible obstructive airways disease.
The consultants' opinions were compared with two sets of questionnaire scoring systems chosen in advance on the basis of clinical experience as likely predictors of obstructive airways disease:
1) Four or more symptoms/risk factors reported from the six key questions.
2) Four or more symptoms/risk factors reported from the six key questions plus at least one marker of severity (Table 1).
Statistical analysis
The positive predictive value, sensitivity, specificity and their confidence intervals were calculated for each scoring system using the majority consultant opinion as the "gold standard". The method used was appropriate for the two-stage process [9]. The prevalence of subjects who would merit a trial of obstructive airways disease medication was also estimated using the two-stage process.
Results
Of 10,429 questionnaires sent out in 1995, 7,580 were returned after three mailings. After a 5.5% adjustment for adults no longer at the mailing address [2], this represented a 78% response rate. 6947 responses contained sufficient information for the present analyses. 420 adults were invited for clinical review of whom 202 (48%) attended.
Agreement between consultants about the decision to treat is detailed in Table 3. There was unanimous agreement concerning the merits of a trial of treatment in 119 (59.2 %) adults. The majority verdict was used to classify the remaining subjects.
Table 3.
Level of consultant agreement Number of adults categorised according to whether they merit a trial of obstructive airways disease medication by the level of consultant agreement.
| Meriting a trial of obstructive airways disease medication | |||
| Consultant agreement | Yes | No | Total |
| All agreed | 73 | 46 | 119 |
| 2 out of 3 agreed | 43 | 39 | 82 |
| Final categorisation | 116 | 85 | 201 |
One subject was not categorised by one of the consultants and the other two disagreed on the decision on whether a trial of obstructive airways disease treatment was indicated so no majority decision could be made. This individual was excluded from analysis.
Table 4 compares the majority expert opinions against the two sets of questionnaire scoring systems. Of 146 adults with four or more symptoms/risk factors, 109 (75%) merited a trial of medication. 100 (82%) of the 122 adults who reported four or more symptoms/risk factors and who had a marker of severity were thought by the consultants to merit a trial of asthma medication.
Table 4.
Comparison between majority expert opinions and questionnaire scoring systems
| Categories of score | "Meriting a trial of medication" | "Not meriting a trial of medication" | Total | |
| All responses | 116 | 85 | 201* | |
| System 1 | Four or more symptoms/risk factors | 109 | 37 | 146 |
| Fewer than four or more symptoms/risk factors | 7 | 48 | 55 | |
| System 2 | Four or more symptoms/risk factors plus one marker of severity | 100 | 22 | 122 |
| Other responses- | 16 | 63 | 79 |
* One subject was not categorised by one of the consultants and the other two disagreed on the decision on whether a trial of obstructive airways disease treatment was indicated so no majority decision could be made. This individual was excluded from analysis.
The positive predictive values (PPV), sensitivity and specificity of the two scoring systems are shown in Table 5. The PPV for the first scoring system when compared against the consultants' opinion was 75.1% (95% confidence interval (CI) 68.6–82.3), whilst that of the second system was 82.3% (95% CI 75.9–89.2). The more stringent second scoring system had a greater specificity but poorer sensitivity than the first scoring system.
Table 5.
Positive predictive value, sensitivity and specificity of the two scoring systems Discriminative properties of two scoring systems for questionnaire responses compared with the majority expert opinion about whether the patients merit a trial of obstructive airways disease therapy.
| Scoring system | 1. Four or more symptoms/risk factors | 2. Four or more symptoms/risk factors plus one marker of severity |
| PPV % [95% CI] | 75.1 [68.6–82.3] | 82.3 [75.9–89.2] |
| Sensitivity % [95% CI] | 50.3 [35.3–71.6] | 46.9 [33.0–66.8] |
| Specificity % [95% CI] | 95.3 [94.0–96.7] | 97.1 [96.0–98.2] |
PPV = Positive predictive value CI = Confidence interval
The prevalence of adults "meriting a trial of obstructive airways disease therapy" (a surrogate for obstructive airways disease) in our survey was calculated [5] to be 22% (95% CI 15.5–31.4).
Discussion
Two simple scoring systems, (based on the number of symptoms/risk factors reported on a respiratory questionnaire) produced high positive predictive values when compared with majority opinion of three respiratory physicians regarding possible obstructive airways disease.
Two important sources of selection bias could have occurred. First, non-respondents to the questionnaire survey could have been materially different from respondents. This was examined after the first questionnaire survey in 1993, when a comparison was made between the practice medical records of a random sample of 100 respondents and 100 non-respondents. No important differences were found with respect to age, gender, and total number of consultations and consultations for respiratory problems in the previous year. There is no reason to suspect that the situation was different in 1995. Second, the adults attending for clinical review may have been different from those who were invited but who did not attend. These two groups were compared with respect to their age, gender and the number of cigarettes smoked as reported on the questionnaire. No significant differences were found between the two groups.
The purpose of a screening test is to identify individuals with a good chance of having disease and who require further clinical assessment to confirm or refute the diagnosis. When defining the threshold at which a screening test is deemed to be positive, consideration has to be taken of the balance between false positive results (which can lead to extra distress because of unnecessary further investigations) and false negatives (which result in some cases of disease being missed). The positive predictive value of a test reflects the frequency of disease in those with a positive screening test. The cost effectiveness of a screening programme will depend on the cut-off values chosen for the screening procedure as these determine the number of new cases detected and requiring treatment, unnecessary investigations undertaken etc.
When assessing a screening test it can be difficult to know what diagnostic standard to use, particularly for conditions such as obstructive airways disease which do not have a universally accepted clinical definition. In our study, we defined adults as having possible obstructive airways disease if the majority opinion of three consultant physicians was that a trial of treatment for obstructive airways disease was merited (majority decisions of specialists has been used in previous studies to diagnose asthma) [1,5]. Many clinicians would probably agree that it is reasonable to assess an adult who might merit a trial of obstructive airways disease treatment. Comparing the two scoring systems against this "standard", we were able to determine the test characteristics (sensitivity, specificity and positive predictive value) of each system. The more stringent system (four or more symptoms/risk factors plus a marker of severity) had a greater positive predictive value and greater specificity (so gave fewer false positive results) than the less stringent system (four or more symptoms/risk factors alone), but at the cost of reduced sensitivity (more cases were missed).
Any system, which increased sensitivity, would almost certainly have reduced positive predictive value probably resulting in and unacceptable rate of false positives. In contrast with screening for malignant conditions when detection of all cases is of prime importance (i.e. high sensitivity), it could be argued that in population screening for diseases such as asthma/COPD the PPV carries more weight than sensitivity.
A recent article by Grimes and Schultz [10] illustrates this point:
"Although sensitivity and specificity are of interest to public-health policymakers, they are of little use to the clinician. Stated alternatively, sensitivity and specificity (population measures) look backward (at results gathered over time). Clinicians have to interpret test results to those tested. Thus what clinicians need to know are the predictive values of the test (individual measures, which look forward)".
In a previous paper [2], 1112 patients (13.8% of responders) were identified positively by the screening questionnaire. The PPV would indicate that approximately 834 of these had obstructive airways disease. Further, 529 of those who screened positively had no recorded diagnosis of obstructive airways disease or received inhaled medication in the previous 12 months. Thus despite the disadvantage of a relatively low sensitivity for the scoring system significant numbers of patients with obstructive airways disease would be identified. It should be noted that the prevalence of adults "meriting a trial of obstructive airways disease therapy" in our survey was calculated [5] to be 22% (95% CI 15.5–31.4). This may be an overestimate of the true prevalence of obstructive airways disease as it will include patients who would not benefit from a trial of treatment.
The three specialists reviewing the clinical information relied on detailed written data supplied to them by the research team; they did not have direct contact with patient. Neither the experts nor the examining team were aware of the results of the postal questionnaire at the time of their involvement in the study. Total agreement by the consultants on whether a subject warranted a trial of medication was reached in 59.2% of adults. Difficulties in making a diagnosis without personally seeing the patient may also have contributed to disagreement between consultants although it is likely that some difference of opinion would have persisted even with personal examination. Expert opinion concerning asthma diagnosis has been used to define asthma in previous epidemiological studies [1,11,12]., whilst others have used patient recall of asthma diagnosis or treatment when measuring prevalence of the disease or its underdiagnosis [13]. The majority of diagnostic decisions are made in primary care and we acknowledge that choosing three consultant respiratory physicians as the diagnostic gold standard therefore has limitations. They were however chosen to reflect a spread of secondary and tertiary respiratory opinions.
An advantage of the scoring systems used was their simplicity. Techniques such as discriminant analysis and logistic regression could have been used to assess the predictive value of each question and a different scoring system developed from them. This would have the disadvantage that validation would then have been based on the same data as was used to derive the scoring system and therefore would tend to be over optimistic. In addition the scoring system might be more difficult to implement.
In clinical practice, a simple scoring system to identify patients requiring further review is attractive. Choices about which system to use will depend on a number of factors; the balance between positive predictive value and sensitivity of each system and available health care resources. It is important to remember that if this questionnaire were to be used in another setting the positive predictive value would have to be recalculated as this value is dependent on the prevalence of disease in the population studied.
Conclusions
When assessed against the majority expert opinion of "meriting a trial of obstructive airways disease medication", two simple scoring systems based on responses to a brief postal questionnaire provided a good method for identifying adults likely to benefit from a trial of treatment for obstructive airways disease and so warranting clinical review.
Authors' contributions
TLF and PIF conceived the study and TLF, PIF, PCH, RMM, SH, and CACP participated in its design and coordination. TLF, JAC and MLH contributed to data analysis and interpretation. RRM provided statistical advice, data analysis and interpretation. TLF wrote the manuscript and all the authors critically revised it and approved the final manuscript
Pre-publication history
The pre-publication history for this paper can be accessed here:
Acknowledgments
Acknowledgements
The authors would like to thank the three consultants, Dr J Miles, Dr S Burge and Dr J Williams who provided our expert majority diagnosis and Tamsin Wright who helped with the clinical evaluations; the staff of Bowland Medical Practice and Tregenna Group Practice and the North West Lung Research Centre; and the subjects who answered the postal questionnaire and who attended for clinical review. We would like to thank the Royal College of General Practitioners (RCGP) Manchester Research Unit (now relocated to Aberdeen). This study was supported by Glaxo-Wellcome, Zeneca Pharmaceuticals, Manchester Airport, The Royal College of General Practitioners and The North West Lung Centre.
Contributor Information
Timothy L Frank, Email: gpresearchunit@yahoo.co.uk.
Peter I Frank, Email: gpresearchunit@yahoo.co.uk.
Jennifer A Cropper, Email: gpresearchunit@yahoo.co.uk.
Michelle L Hazell, Email: gpresearchunit@yahoo.co.uk.
Philip C Hannaford, Email: gpresearchunit@yahoo.co.uk.
Roseanne R McNamee, Email: gpresearchunit@yahoo.co.uk.
Sybil Hirsch, Email: gpresearchunit@yahoo.co.uk.
Charles AC Pickering, Email: gpresearchunit@yahoo.co.uk.
References
- De Marco R, Cerveri I, Bugiani M, Ferrari M, Verlato G. An undetected burden of asthma in Italy: the relationship between clinical and epidemiological diagnosis of asthma. Eur Respir J. 1998;11:599–605. [PubMed] [Google Scholar]
- Frank PI, Ferry S, Moorhead T, Hannaford P. Use of a postal questionnaire to estimate the likely under-diagnosis of asthma-like illness in adults. Br J Gen Pract. 1996;46:295–297. [PMC free article] [PubMed] [Google Scholar]
- Agertoft L, Pedersen S. Effects of long-term treatment with an inhaled corticosteroid on growth and pulmonary function in asthmatic children. Respir Med. 1994;88:373–381. doi: 10.1016/0954-6111(94)90044-2. [DOI] [PubMed] [Google Scholar]
- van den Boom G, Rutten-van Molken MP, Folgering H, van Weel C, van Schayck CP. The economic effects of screening for obstructive airway disease: an economic analysis of the DIMCA program. Preventive Medicine. 2000;30:302–308. doi: 10.1006/pmed.2000.0633. [DOI] [PubMed] [Google Scholar]
- Frank TL, Frank PI, McNamee R, Wright T, Hannaford P, Morrison J, et al. Assessment of a simple scoring system applied to a screening questionnaire for asthma in children aged 5–15 yrs. Eur Respir J. 1999;14:1190–1197. doi: 10.1183/09031936.99.14511909. [DOI] [PubMed] [Google Scholar]
- Basomba A, Sastre A, Peleaz A, Romar A, Campos A, Garcia-Villalmanzo A. Standardisation of the prick test. Allergy. 1985;40:395–399. doi: 10.1111/j.1398-9995.1985.tb02677.x. [DOI] [PubMed] [Google Scholar]
- American Thoracic Society. Standards for the diagnosis and care of patients with chronic obstructive pulmonary disease (COPD) and asthma. Am Rev Respir Dis. 1987:225–244. doi: 10.1164/ajrccm/136.1.225. [DOI] [PubMed] [Google Scholar]
- Yan K, Salome C, Woolcock AJ. Rapid method for measurement of bronchial responsiveness. Thorax. 1983;38:760–765. doi: 10.1136/thx.38.10.760. [DOI] [PMC free article] [PubMed] [Google Scholar]
- McNamee R. Optimal designs of two-stage studies for estimation of sensitivity, specificity and positive predictive value. Statistics in Medicine. 2002;21:3609–3625. doi: 10.1002/sim.1318. [DOI] [PubMed] [Google Scholar]
- Grimes DA, Schulz KF. Uses and abuses of screening tests. Lancet. 2002;359:881–884. doi: 10.1016/S0140-6736(02)07948-5. [DOI] [PubMed] [Google Scholar]
- Jenkins MA, Clarke JR, Carlin JB, Robertson CF, Hopper JL, Dalton MF, et al. Validation of questionnaire and bronchial hyperresponsiveness against respiratory physician assessment in the diagnosis of asthma. Int J Epidemiol. 1996;25:609–616. doi: 10.1093/ije/25.3.609. [DOI] [PubMed] [Google Scholar]
- Burr ML. Editorial: Diagnosing asthma by questionnaire in epidemiological surveys. Clin Exp Allergy. 1992;22:509–510. doi: 10.1111/j.1365-2222.1992.tb00158.x. [DOI] [PubMed] [Google Scholar]
- Kaur B, Anderson HR, Austin J, Burr M, Harkins LS, Strachan DP, et al. Prevalence of asthma symptoms, diagnosis, and treatment in 12–14 year old children across Great Britain (international study of asthma and allergies in childhood, ISAAC UK) BMJ. 1998;316:118–124. doi: 10.1136/bmj.316.7125.118. [DOI] [PMC free article] [PubMed] [Google Scholar]