Validation of the COPD Diagnostic Questionnaire in an Australian general practice cohort: a cross-sectional study

Abstract

Background:

The gold standard for the diagnosis of chronic obstructive pulmonary disease (COPD) is spirometry, but there are barriers to its use in primary care.

Aims:

To externally validate the COPD Diagnostic Questionnaire (CDQ) as a diagnostic tool in patients at increased risk in Australian general practice and to compare its performance with other CDQ validation studies.

Methods:

Patients were recruited from 36 general practices in Sydney, Australia. Former or current smokers aged 40–85 years with no prior COPD diagnosis were invited to a case-finding appointment with the practice nurse. The CDQ was collected and pre- and postbronchodilator spirometry was performed. Cases for whom complete CDQ data were present and the spirometry met quality standards were analysed.

Results:

Of 1,631 patients who attended case-finding recruitment, 1,054 (65%) could be analysed. Spirometry showed 13% had COPD. The ability of the CDQ to discriminate between patients with and without COPD was fair, represented by the area under the receiver operating characteristic curve of 0.713. With a CDQ cut-off point value of 16.5 the sensitivity was 80% and specificity 47% and, at a cut-off point value of 19.5, the sensitivity was 63% and specificity 70%.

Conclusions:

The CDQ did not discriminate between patients with and without COPD accurately enough to use as a diagnostic tool in patients at increased risk of COPD in Australian general practice. Further research is needed on the value of the CDQ as a tool for selecting patients for spirometry.

Introduction

Chronic obstructive pulmonary disease (COPD) is a common respiratory disease characterised by airflow obstruction. The major risk factor in developed countries is tobacco smoking.¹ COPD is an important cause of mortality and disability worldwide, ranked globally as the fifth leading cause of death and 11th leading cause of disability-adjusted life years lost in 2002.² In Australia in 2007, COPD was the fourth and sixth leading causes of death for men and women, respectively, and made up 1% of all hospitalisations.³ In a recent multi-regional Australian study, 14.5% of the population aged ≥40 years had COPD based on spirometric diagnosis, with the prevalence higher in older age groups.⁴

The gold standard for COPD diagnosis is post-bronchodilator (post-BD) spirometry assessment in the clinical context of any patient with dyspnoea, chronic cough, or sputum production with a history of exposure to risk factors such as tobacco smoke and occupational dusts.⁵ Performing screening spirometry in the general population — which includes asymptomatic people — is controversial, with insufficient evidence to indicate that screening improves morbidity, mortality, or smoking cessation rates.^5,6 The barriers to spirometry use in general practice include lack of expertise in performing spirometry, poor access to a well maintained spirometer, the time consuming nature of pre- and post-BD spirometry, and low confidence in spirometry interpretation.⁷ This can lead to underdiagnosis and misdiagnosis of COPD, particularly if general practitioners rely on a symptom-based assessment.^7,8 High-quality management of COPD requires accurate diagnosis to relieve symptoms, improve health status, prevent exacerbations and disease progression, and reduce early mortality.⁵

Although the CDQ was not developed as a diagnostic tool, it has been validated in comparison with spirometry in subjects selected from primary care settings in Europe and Australia and hospital outpatient clinics in Japan.^15,19–21 Two of these studies investigated the validity of the Piko-6^® flow meter (a form of micro-spirometry) as a screening tool for COPD diagnosis in primary care.^15,21 These studies also validated the CDQ and compared it with micro-spirometry.^15,21 Other questionnaires such as the COPD Population Screener Questionnaire (COPD-PS) and a condensed version of the COPD Assessment Test (CAT) have been developed but have not undergone external validation.^13,14

The aim of this study was to validate the CDQ externally in a large sample of current and former smokers with no prior diagnosis of obstructive lung disease recruited from general practices in Sydney, Australia. We wanted to determine if the CDQ could be used as a COPD diagnostic tool in patients at increased risk in Australian general practice. This complements an earlier CDQ study in a smaller cohort recruited from Australian general practices.²¹ The analysis methods were based on a protocol outlined by Price et al.¹² The study measured the relationship between CDQ scores and COPD proportions determined by spirometry and these results were compared with the findings in the study by Price et al., one Australian and three international external validation studies.^{12,15,19–21} The variability in how the CDQ performed in this and other external validation studies was examined.

Methods

Patient recruitment for external validation

Patients in this validation study were from a case-finding recruitment group for a cluster randomised controlled trial of early intervention in COPD by practice nurse-general practitioner teams. Written informed consent was obtained from all patients. The details of the protocol of this trial have been described elsewhere.²² Ethics approval was granted by the University of New South Wales Human Research Ethics Committee.

Patients aged 40–85 years who were former or current smokers with no previous diagnosis of COPD or other obstructive lung disease were invited to a case-finding appointment with a practice nurse in one of the 36 study general practices. The nurses had attended eight hours of training in spirometry and how to administer the CDQ.²² The CDQ was completed by the practice nurse prior to performing pre- and post-BD spirometry using 400μg salbutamol or 500μg terbutaline (for those refusing salbutamol) via a metered dose inhaler based on the American Thoracic Society and the European Respiratory Society (ATS/ERS) 2005 lung function guidelines.^23,24 The practice nurses used the practice's own spirometer which had been calibrated by the research team. Several different models of spirometer were used across the study, with each model being found in more than one practice. Spirometry tracings were independently reviewed by a respiratory physiologist (AJC). Cases where spirometry met quality standards based on the ATS/ERS 2005 criteria and for whom complete CDQ data were present were included in the analysis.²⁴ A study diagnosis of COPD was assigned to subjects who had a post-BD forced expiratory volume in one second/forced vital capacity (FEV₁/FVC) ratio <0.7, which was based on the Global Initiative for Chronic Obstructive Lung Disease (GOLD) guidelines.⁵ Calculation of the CDQ scores was performed by the project officer using predetermined scoring criteria.¹²

CDQ score and statistical analysis

A three-tier scoring system was used to assign subjects into groups of low, intermediate, and high likelihood of COPD based on the CDQ score 0–38 with cut-off points at 16.5 and 19.5.¹² The receiver operating characteristic area under the curve (ROC_AUC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for the two cut-off points using the non-parametric method. The Pearson χ² test and independent sample t tests were used for comparing demographic information between included and excluded subject groups. A p value of <0.05 indicated a statistically significant difference. The raw CDQ score was used as the screening test variable and the COPD diagnosis as the dichotomised classification variable. Analyses were performed using SPSS (IBM, Armonk, NY, USA) and Microsoft Excel (Microsoft, Redmond, WA, USA) software.

Results

Study population selection

A total of 1,631 people attended for case-finding recruitment. Of these, 1,054 (65%) had the complete CDQ recorded and spirometry meeting quality criteria for analysis. Figure 1 shows the reasons for exclusion of the remaining 35% of subjects. The mean±SD age (62.5±11.6 years vs. 61.0±11.3 years, p=0.009) and percentage of males (58.9% vs. 51.8%, p=0.006) was significantly higher for excluded patients than for the included group. There was no significant difference between excluded and included groups in current smoking status (22.7% vs. 22.3%, p=0.825).

Figure 1. Flow chart showing study numbers and reasons for exclusion from statistical data analysis. Percentages in brackets represent the proportion of the initial 1,631 subjects. CDQ= COPD Diagnostic Questionnaire.

Characteristics of participants

Complete data for 1,054 patients were used in the study group analyses. Table 1 shows the population characteristics of the two groups based on COPD diagnosis. After post-BD spirometry, 13.1% of the total population were diagnosed with COPD. The mean post-BD FEV₁/FVC ratio for the COPD group was 63% compared with 80% for the non-COPD group and 78% for all subjects. The COPD group had more men, a higher proportion of current smokers and CDQ scores (by four points), and were on average five years older than the non-COPD group.

Table 1. Characteristics of study population.

Performance of the CDQ in this study and comparison with other CDQ validation studies

Following application of the two cut-off points of 16.5 and 19.5 after CDQ calculation in accordance with the original CDQ study,¹² 43.4% had a low likelihood of COPD (<16.5), 22.4% had an intermediate likelihood (16.5–19.5), and 34.3% of patients had a high likelihood of COPD (>19.5). In terms of the distribution of subjects with COPD within the CDQ zones, 20% had scores in the lowest zone but 63% had scores in the highest zone (Table 1). Within the three zones, 6.1% of subjects had COPD in the low likelihood zone (28 of 457 subjects), 9.7% in the middle zone (23 of 236), and 24.1% (87 of 361) in the high zone.

The trend of increasing COPD proportions from the low to the high zone was published in three of the studies, as shown in Table 2. The ability of the CDQ to discriminate between patients with and without COPD is represented by the ROC curve (Figure 2). The ROC_AUC was 0.713, which is lower than the original study and two other validation studies.^12,19,21 This is shown in Tables 2 and 3, with results of the other studies derived from their respective papers.^{12,15,19–21}

Table 2. CDQ validation studies compared with original study¹².

Figure 2. Receiver operating characteristic (ROC) curve for the CDQ score compared with chronic obstructive pulmonary disease diagnosis. Area under ROC curve (ROC_AUC) was 0.713. A ROC_AUC of 1.0 would have the optimal operating point indicated by the star. A ROC_AUC of 0.5 is indicated by the solid diagonal line. CDQ= COPD Diagnostic Questionnaire.

Table 3. Performance of COPD Diagnostic Questionnaire across comparison studies.

Sensitivity and specificity at the cut-off point value of 16.5 (cut-off point A) were 79.7% and 46.8%, respectively and, at 19.5 (cut-off point B), the sensitivity and specificity were 63.0% and 70.1%, respectively. When compared with other validation studies (Table 3), sensitivity was lower at both cut-off points. However, with the exception of the study by Sichletidis et al., the specificity was higher.¹⁵ The PPV at cut-off points A and B were 18.4% and 24.1% for this study compared with 30.3% and 37%, respectively, in the original study.¹² The NPV was comparable to the original study at both cut-off points (93.9% this study vs. 92.7% in the original study at point A and 92.6% vs. 89.0% at B).¹²

Discussion

Main findings

In this study, using the three-tier scoring system, the CDQ did not perform well in identifying people with COPD when compared with spirometry. The ROC_AUC of 0.713 is fair, and is higher than that in the study by Kotz et al. who considered their ROC_AUC of 0.65 to be very low.^20,25 The ROC_AUC in this study is less than two other external validation studies and less than the original study which had an ROC_AUC of 0.816.^12,19,21 The ROC_AUC in this study is closer to 0.5 than 1.0, where a test with an area under the curve of 1 would represent a perfect test with no overlap between true positives and false positives and the optimal operating point corresponding to the upper left-hand corner of the ROC graph (Figure 2).²⁶ An ROC_AUC of 0.5 indicates a test with no discriminative power and would be essentially worthless.^25,26

The ability of the CDQ to identify patients with and without COPD varies between populations. This is influenced by characteristics such as smoking and age. When applying the two cut-off points from the CDQ in this study, the questionnaire achieved sensitivities of 79.7% and 63% at cut-off points A (16.5) and B (19.5), with specificities of 46.8% and 70.1%, respectively. This means that, in this population, at cut-off point A about 80% of patients with a COPD diagnosis on spirometry were correctly identified by the questionnaire, but 55% of patients without COPD were incorrectly identified by the CDQ as having COPD. At cut-off point B, while about two thirds of patients with COPD were correctly identified by the CDQ, 30% of patients without COPD were incorrectly identified as having COPD. The PPV of 24.1% at cut-off point B indicates that a person with a CDQ score above 19 has a one in four chance of spirometry-diagnosed COPD. When designing the CDQ, Price et al. considered that a reasonable PPV would be at least 50% for the higher cut-off point, but the PPV at cut-off point B was less than 50% in the three studies which published these results (Table 3).¹²

Strengths and limitation of this study

A particular strength of the study is that it is a large sample of primary care patients from Australia made up of former and current smokers, the number of participants only exceeded by the study by Sichletidis et al.¹⁵ The number of patients excluded from the study may have biased the outcome. For example, the invalid subset was older by an average of 1.5 years. Potentially this could have increased the proportion of COPD in the final population as higher CDQ scores reflect higher COPD prevalence and COPD diagnoses increase with age.⁴ The proportion of subjects excluded from the statistical analysis (i.e. 35%) is almost double the cases excluded from the CDQ validation study by Kotz et al.²⁰ (18%) and substantially higher than the Greek and Australian external validity studies as well as the original study (9%).^12,19,21 This could be due to the different operators at each of the 36 sites for spirometry and the use of several different spirometry models, although every practice nurse received spirometric training. The other studies used independently trained operators such as respiratory physicians or research assistants to perform the spirometry and each study used one spirometer model/brand. While the latter design may help to reduce variability in spirometry performance and improve interpretation of adequate spirometry attempts as per the ATS/ERS 2005 criteria, this study's design better represents real-world use of the CDQ and spirometry in Australian general practice.

As our sample included patients without respiratory symptoms, this may underestimate the ability of the CDQ to detect cases of COPD as defined by the GOLD guidelines.⁵ This limitation also applies to the other CDQ studies that looked at detecting spirometrically-defined COPD in patients with risk factors for COPD but not necessarily symptoms.^{12,15,19–21} Given the controversy of detecting asymptomatic airflow obstruction, this could be seen as an advantage of the CDQ in that asymptomatic patients would be less likely to score above the CDQ cut-off points. Further research is needed to determine if the CDQ performs better as a diagnostic tool for detecting only those cases meeting all the GOLD criteria.

Interpretation of findings in relation to previously published work

Two of the validation studies compared micro-spirometry with full spirometry, with or without the CDQ. Sichletidis et al. used post-BD micro-spirometry with the Piko-6^® meter whereas Frith et al. used this device in the pre-BD stage prior to full spirometry.^15,21 The Piko-6^® meter proved to be a more specific and less sensitive test, with a higher PPV than CDQ for COPD diagnosis. The ROC_AUC of 0.85 for the Piko-6^® meter in the study by Frith et al. outperformed the CDQ across all studies.²¹ The performance of the CDQ/Piko-6^® combination vs. spirometry in the study by Sichletidis et al. reduced the sensitivity and increased the specificity compared with the tests individually.¹⁵ Frith et al. concluded that measuring FEV₁/FEV₆ from a simple flow meter could optimise early referral for spirometry.²¹ Using a pre-BD peak flow meter to screen individuals with respiratory symptoms and at high risk of COPD for diagnostic spirometry has been discussed previously but has not been externally validated.²⁷

Another possible role for the CDQ is as a selection tool for spirometry, removing the need for this test in patients at low risk of COPD. The concept of using the CDQ to ‘pre-screen’ those smokers at highest likelihood of COPD for spirometry using the higher cut-off point instead of performing spirometry on all current and former smokers was discussed previously by Price et al.¹² Subjects in the intermediate zone of likelihood in the CDQ could undergo spirometry but, where there were limited resources to do spirometries in this group, these subjects could be followed up clinically and spirometry deferred to a later date to minimise the number of unnecessary spirometries.¹² The International Primary Care Respiratory Group (IPCRG) have recommended a diagnostic process where all patients over 35 years of age should be evaluated for their risk of developing COPD by completing the CDQ and/or ‘case-identification’ spirometry prior to standard diagnostic spirometry.²⁸ The IPCRG assigned the lower cut-off point as a singular cut-off point for the CDQ in this diagnostic process, with subjects scoring ≥17 going on to have diagnostic spirometry.^12,28

Implications for future research, policy and practice

Further research is needed on the use of the CDQ as a selection tool for proceeding to spirometry. This could potentially save both time and money if found to be effective. It may be more practical to set one instead of two cut-off points for this application of the questionnaire. At the lower cut-off point the sensitivity of 80% would be considered suboptimal when compared with the other external validation CDQ studies with sensitivities ranging between 89% and 94% (Table 3). If high sensitivity is preferable for diagnosing more COPD cases, then the cut-off point can be set as low as is practical, but at the expense of more negative spirometry tests. Alternatively, one could select a cut-off point that finds the right balance between sensitivity and specificity for selecting patients for spirometry in a given setting. Therefore, in a future study it would be important to look at how the CDQ performs using a two-tier versus a three-tier scoring system.

Conclusions

The results of this external validation study suggest that the questionnaire does not discriminate between patients with and without spirometrically-defined COPD accurately enough to use as a stand-alone diagnostic tool in Australian general practice. Further research is needed on the value of the CDQ as a tool for selecting patients to proceed to spirometry. Setting one instead of two cut-off points for the CDQ could make it more practically applicable for this purpose.

Appendix 1. COPD Diagnostic Questionnaire (CDQ)