Criteria To Screen for Chronic Sinonasal Disease

Abstract

Background:

Sinusitis and rhinitis are associated with uncontrolled asthma. There are no simple, validated tools to screen for these diseases. The objective of this study was to assess instruments to assist in the diagnosis of chronic sinonasal disease.

Methods:

Participants without acute sinonasal symptoms underwent an extensive evaluation. The results were submitted to an expert panel that used the Delphi method to achieve consensus. Using the consensus diagnosis of the panel, we determined the sensitivity and specificity of test procedures to diagnose chronic sinonasal disease. We determined the reproducibility of the most sensitive and specific instrument in a separate cohort.

Results:

Fifty-nine participants were evaluated, and the expert panel reached consensus for all (42 participants with chronic sinonasal disease, 17 participants without chronic sinonasal disease). A six-item questionnaire based on the frequency of nasal symptoms was the most sensitive tool used to diagnose sinonasal disease (minimum specificity, 0.90). Reproducibility testing in a separate cohort of 63 participants (41 chronic sinonasal disease with asthma, 22 chronic sinonasal disease without asthma) showed a concordance correlation coefficient of 0.91 (95% CI, 0.85 to 0.94) when this questionnaire was limited to five items (ie, excluding a question on smell). This five-item questionnaire had a sensitivity of 0.90 (95% CI, 0.77 to 0.97), a specificity of 0.94 (95% CI, 0.71 to 1.00), and an area under the receiver operating characteristic curve of 0.97 (95% CI, 0.93 to 1.0). Sinus CT scans and nasal endoscopy lacked sensitivity for use in the diagnosis of chronic sinonasal disease.

Conclusions:

We have developed a sensitive, specific, and reproducible instrument to screen for chronic sinonasal disease. Validation studies of this five-item questionnaire are needed, including in patients with asthma.

Rhinitis and sinusitis are very common in patients with asthma.^1,2 Sinonasal disease may lead to poorly controlled asthma, so guidelines³ have recommended screening for rhinitis and sinusitis, but there is no consensus on how this should be done. Although some investigators have suggested using sophisticated testing such as CT imaging or endoscopy, these tests are expensive and inconvenient. In practice, physicians often make diagnoses and treat patients on the basis of vague clinical criteria, and only when patients do not respond to therapy or have symptoms suggesting complicated disease are CT scans or endoscopy used.

The purpose of this study was to determine the most sensitive, specific, and reproducible instrument for use in the diagnosis of chronic sinonasal disease. Given the overlap between rhinitis and sinusitis, we did not attempt to distinguish between the two disease entities. We evaluated a cohort of participants with a standardized set of clinical, laboratory, and imaging procedures. We excluded acute disease. The data from this evaluation were submitted to an expert panel that determined whether the participants had chronic rhinitis or sinusitis. Using the diagnoses of the panel, we determined the sensitivity and specificity of individual clinical instruments to distinguish patients with chronic rhinitis or sinusitis from those without. We determined the repeatability of the most sensitive and specific instrument, a questionnaire, in a separate cohort.

Materials and Methods

An overview of both phases of the study method is presented in Figure 1 and discussed herein.

Figure 1.

Overview of study protocol.

Phase 1

The study was approved by the institutional review boards of all six participating American Lung Association-Asthma Clinical Research Centers, and informed consent was obtained from all phase 1 participants. From the centers, we enrolled participants with and without asthma who were ≥ 18 years of age. Participants were recruited from local clinics and through advertising. We recruited persons without asthma to ensure the inclusion of participants without chronic rhinitis and sinusitis for sensitivity and specificity calculations. Details of the eligibility criteria are included in the online supplement. Recruitment took place out of allergy season between November 2006 and March 2007 at each individual center.

Spirometry and methacholine challenge testing were performed according to American Thoracic Society^4,5 guidelines. Participants answered questionnaires^6–13 and underwent nasal endoscopy and sinus CT scanning¹⁴ (details in the online supplement).

We submitted a standardized data set on each participant to an expert panel who had determined its contents at the beginning of the study. The data set included multiple questions pertaining to specific sinonasal symptoms (13 total, shown in the online supplement) that have been reported in the literature,^6,7 were believed to be relevant by the expert panel, and would be useful and relevant in a clinical setting. In addition to the answers to these questions, the data set included the results of a previously published sinus symptom scale score,¹⁰ and the endoscopy and CT scanning.

The expert panel determined whether each participant had sinusitis, rhinitis, unknown disease, or no disease. We used the Delphi method¹⁵ to reach consensus among panelists, who had no direct contact with one another. When disagreement existed between panelists, the data coordinating center returned the data set (with the anonymous results of the first round) to the panel for a second and third round, and the panelists were again asked to classify the participant until they reached consensus on a diagnosis. The panel consisted of two otolaryngologists and one allergy-immunologist, all of whom were experts in the field of sinonasal disease.

Phase 2: Reproducibility Testing

Based on previous reports in the literature,^6,7 we identified 6 questions from the original total of 13 questions that were related to the frequency of specific symptoms, and that we hypothesized would be both sensitive and specific for identifying patients with sinonasal disease. We tested the performance characteristics of these items and compared them with other standard questionnaires and specific testing procedures.

Results from phase 1 of the study showed that this six-item questionnaire was the tool with the highest sensitivity and specificity for use in diagnosing chronic sinonasal disease. To determine the reproducibility of these items, participants in phase 2 of the study completed the questionnaire on two occasions at least 1 week apart.

The second cohort of participants was recruited outside of allergy season in Vermont from a pulmonary clinic and local college campuses. We included participants ≥ 18 years of age with and without a self-report of asthma. We excluded participants who had reported upper airway symptoms only in the past 6 weeks, had a > 10 pack-year smoking history, or had smoked in the past 6 months.

Statistical Analysis

For phase 1, descriptive statistics (mean, SD, count, and proportion) were used to summarize baseline demographics. The population was divided into the following two groups: participants with asthma; and participants without asthma. The number of participants identified as either having or not having chronic sinonasal disease was tabulated for each group. The agreement between the first and last diagnosis rounds was calculated for each panel member using a κ statistic with a 95% CI. We compared continuous measurements of lung function (eg, symptom scores and pulmonary function) among participants with and without a condition using the Wilcoxon rank sum test because not all of the measurements followed a Gaussian distribution. Sinonasal symptom scoring measurements with p < 0.05 in the association analysis were considered in the classification analysis described later in this article. Classification techniques (receiver operating characteristic [ROC], area under the curve [AUC], sensitivity, and specificity) were used to identify clinical tools with potential use in reliably diagnosing chronic sinonasal disease in the overall population. For each of the measurements of upper airways disease, a ROC curve was constructed. The empirical AUC was calculated with a 95% bootstrap CI.¹⁶ The goal was to obtain a high level of specificity while maintaining an acceptable level of sensitivity. Therefore, for each measurement, the cutoff point was selected in order to maximize sensitivity while requiring that the specificity be ≥ 0.90. The sensitivity and specificity with 95% CIs were calculated for each measurement based on these cutoff points.

For phase 2, the reproducibility of the questionnaire, both with (six items) and without a question pertaining to smell (five items), was evaluated in a separate cohort made up of both participants with asthma and participants without asthma. Summary statistics (mean [SD]) were used to describe each visit as well as the change between visits. A Wilcoxon signed rank test was used to determine whether a significant difference existed in the scores between the two visits. Pearson correlation coefficients and the concordance correlation coefficients¹⁷ were calculated in order to assess the amount of variability between time points relative to the overall variability. The percentage of the times that the classification changed based on the optimal cutpoints was calculated, and comparisons between the two versions of the questionnaire were made using the McNemar test. We repeated the classification analysis for the five-item questionnaire. All analyses were performed using a commercial statistical software package (STATA 9; StataCorp; College Station, TX) and open-source software (environment R; www.r-project.org).

Results

Phase 1

Sixty-one participants were enrolled in phase 1 of the study. We excluded one participant who had previously undergone sinus surgery, and another participant for whom the consensus panel believed it could not make a definitive diagnosis from the analysis. The final cohort comprised 41 participants with asthma and 18 participants without asthma in the final analysis (Table 1).

Table 1.

Baseline Characteristics of Phase 1 Study Participants

Characteristics	With Asthma(n = 41)	Without Asthma(n = 18)
Age at enrollment, yr*	38.3 (15.9)	32.2 (13.0)
Male gender†	12 (29.3)	4 (22.2)
Race or ethnic group†
White	29 (70.7)	16 (88.9)
Black	8 (19.5)	2 (11.1)
Hispanic	3 (7.3)	0
Other	1 (2.4)	0
Lung function
Pre-bronchodilator therapy FEV1, %predicted	79.2 (19.9)	100.2 (10.3)
FEV1/FVC ratio	0.7 (0.1)	0.8 (0.1)
Baseline Juniper asthma control score	2.04 (0.49)	NA
Other conditions (self-reported)†‡
Eczema	7 (17.1)	1 (5.6)
Allergic rhinitis	29 (70.7)	0
Food allergies	7 (17.1)	0
Gastroesophageal reflux disease	14 (34.1)	0

NA = not applicable.

*Values are given as the mean (SD).

†Values are given as No. (%).

‡Participants were asked whether they had these conditions “now or within the past year.”

We submitted a standardized set of data to the expert panel three times. Table 2 summarizes the level of agreement from each round. The κ statistics for each reviewer between the first and the final diagnosis for all three reviewers were 0.85 (95% CI, 0.73 to 0.96), 0.61 (95% CI, 0.46 to 0.76), and 0.61 (95% CI, 0.46 to 0.76). If sinusitis and rhinitis are considered as a single category (sinonasal disease), the κ statistics were 1 (95% CI, 1 to 1), 0.84 (95% CI, 0.69 to 0.99), and 0.70 (95% CI, 0.49 to 0.91), which correspond to 0 (0%), 4 (7%), and 7 (11%) changes in diagnosis, respectively.

Table 2.

A Summary of the Delphi Process for Phase 1

Agreement*	Round 1	Round 2	Round 3
3/3	36	14	3
2/3	20	7	5
No agreement	3	1	1†
Missing‡	0	1	0

The table shows agreement of the panel through three rounds of data submission on 59 participants. The panelists agreed that their final diagnosis on one participant was “unsure,” and this participant is not included in this table and excluded from further analysis.

*Number of panelists who agreed.

†One panelist voted for rhinitis, one for sinusitis, and one for unsure. The individual participant was classified for analysis as having “sinusitis/rhinitis” because a majority of the panel agreed that sinonasal disease was present.

‡Missing refers to the fact that one of the expert panel diagnoses was missing for one participant during the second round.

Of the participants with asthma, 36 of 42 participants (87%; 95% CI, 71% to 95%) were deemed to have chronic sinonasal disease compared with 6 of the 18 participants without asthma (33%; 95% CI, 13% to 59%). Participants with chronic sinonasal disease had significantly higher scores on all measures of symptoms and imaging and more impaired quality of life (Table 3).

Table 3.

Nasal Symptoms and Findings in Participants With and Without Asthma With and Without Upper Airways Disease

	No Sinusitis or Rhinitis		Sinusitis or Rhinitis
Measurements	Mean (SD)	Median (Range) [n = 17]	Mean (SD)	Median (Range) [n = 42]	p Value*
SSS†	5.18 (6.67)	1.00 (0.00–24.00	19.19 (13.42)	19.00 (0.00–57.00)	0.0001
RQLQ‡	0.31 (0.52)	0.63 (0.00–1.60)	1.68 (0.94)	1.69 (0.00–3.68)	< 0.0001
SNOT-20§	0.36 (0.58)	0.10 (0.00–2.00)	1.48 (0.89)	1.55 (0.00–3.70)	< 0.0001
Six-item questionnaire‖	0.30 (0.30)	0.17 (0.00–0.83)	1.37 (0.54)	1.33 (0.33–2.67)	< 0.0001
SNQ¶	0.35 (0.34)	0.20 (0.00–1.00)	1.54 (0.54)	1.60 (0.40–2.60)	< 0.0001
Sum of endoscopy scores#	1.29 (1.36)	1.00 (0.00–4.00)	2.98 (2.37)	3.00 (0.00–8.00)	< 0.0001
Sinus CT score**	0.73 (1.16)	0.00 (0.00–4.00)	2.93 (3.38)	2.50 (0.00–19.00)	0.0023

RQLQ = Rhinitis Quality of Life Questionnaire; SNOT-20 = 20-item Sino-Nasal Outcomes Test; SSS = sinus symptom score.

*Calculated using the Wilcoxon rank sum test.

†The SSS scale is 0 to 60, where a higher score indicates more severe symptoms.^10,11

‡The RQLQ scale is 0 to 6, where a higher score indicates more severe impairment.¹² The value shown for RQLQ is without activities (n = 56).

§The SNOT-20 scale is 0 to 5, where a higher score indicates more severe impairment.¹³ One participant with asthma was missing data for this instrument (n = 58).

‖The six-item questionnaire scale is 0 to 3, where a higher score indicates more frequent symptoms.

¶The SNQ scale is 0 to 3, where a higher score indicates more frequent symptoms.

#The endoscopy scale is 0 to 20, where a higher score indicates more extensive disease.

**The sinus CT scale is 0 to 30, where a higher score indicates more severe disease. Fifty-eight participants had sinus CT scores, and one participant without disease was missing a sphenoid reading; thus, the total sinus CT score is generated by calculating the sum of the area scores.

We analyzed the sensitivity, specificity, and ROC curve for measures of sinonasal disease in participants with and without asthma (Table 4, Fig 2). Initially, we found that the combination of the six questions we had identified based on the frequency of nasal symptoms was a sensitive and specific measure for use in diagnosing upper airways disease. Furthermore, we tested the impact of adding additional frequency questions on sensitivity, specificity, and ROC for diagnosing sinonasal disease and found no significant difference with the addition of any further questions to the six-item (or five-item) questionnaire.

Table 4.

Sample Size, Optimal Cutpoints, Sensitivity, and Specificity for the Seven Measurements

Measurements	Participants, No.	Optimal Cutpoint*	Sensitivity, % (Range)	Specificity, % (Range)
SSS	59	14.00	69.05 (52.91–82.38)	94.12 (71.31–99.85)
RQLQ	56	1.56	64.29 (48.02–78.45)	94.12 (71.31–99.85)
SNOT-20	58	1.55	52.38 (36.41–68.00)	94.12 (71.31–99.85)
Six-item questionnaire	59	1.00	80.95 (65.88–91.40)	100.00 (80.49–100.00)
SNQ	59	1.00	90.48 (77.37–97.34)	94.12 (71.31–99.85)
Sum of endoscopy scores	59	5.00	26.19 (13.86–42.04)	100.00 (80.49–100.00)
Sinus CT score	56	3.00	50.00 (34.19–65.81)	93.75 (69.76–99.84)

See Table 3 for abbreviations not used in the text.

*The optimal cutpoint is the value that maximizes the sensitivity of the measurement while achieving at least 90% specificity.

Figure 2.

ROC, including the AUC, with a bootstrap estimate of the 95% CI. Value shown for RQLQ is without activities, n = 56. Fifty-eight participants had sinus CT scores, and 1 participant without asthma was missing a sphenoid reading. The total sinus CT score is generated by calculating the sum of the area scores. See Table 3 for other abbreviations.

Nasal endoscopy and CT scanning lacked sensitivity, given the specificity constraint. For diagnosing sinusitis alone (present in 13 participants), sinus CT scanning, using a score of 4.0, had the highest sensitivity (83.3; 95% CI, 51.6 to 97.8) and specificity (93.5; 95% CI, 82.1 to 98.6).

Phase 2

We administered the six questions to a different cohort of 64 participants to determine reproducibility. One participant was excluded because information on the sense of smell was missing. Of the remaining 63 participants, 41 self-reported having asthma, and 22 did not. Table 5 summarizes the behavior of the six-item and five-item questionnaires at two time points. For both questionnaires, the score differed significantly (p = 0.0002 vs p = 0.0018, respectively); however, there were only three changes (4.76%) in the diagnosis of sinonasal disease for the five-item questionnaire, but 13 changes (20.63%) for the six-item questionnaire. The five-item questionnaire had a significantly lower number of changes in diagnosis (p = 0.0094). The concordance correlation coefficient improved from 0.71 to 0.91 when the question pertaining to smell was eliminated. We termed this five-item questionnaire the sinonasal questionnaire (SNQ) [Fig 3].

Table 5.

Correlation of Questionnaires in Reproducibility Cohort (Phase 2)

Summary Statistics	Six-Item Questionnaire	SNQ
Visit 1
Mean (SD)	1.33 (0.50)	1.18 (0.59)
Median (range)	1.33 (0.16–2.33)	1.20 (0.00–2.40)
Visit 2
Mean (SD)	1.16 (0.59)	1.28 (0.59)
Median (range)	1.17 (0.00–2.17)	1.40 (0.00–2.20)
Difference
Mean (SD)	0.16 (0.39)	−0.10 (0.24)
Median (range)	0.17 (−0.83–1.17)	0.00 (−1.00–0.60)
p Value*	0.0002	0.0018
Concordance correlation coefficient (95% CI)	0.71 (0.56–0.81)	0.91 (0.84–0.94)
Change in diagnosis, No. (%)†	13 (20.63)	3 (4.76)

Total, n = 63.

*Calculated using the Wilcoxon rank sum test.

†The SNQ has a significantly lower number of diagnosis changes based on the McNemar test (p = 0.0094).

Figure 3.

The SNQ.

The optimal cutpoint for the SNQ based on the frequency of symptoms of nasal obstruction was 1, which is equivalent to symptoms occurring, on average, one to four times per month. Participants with symptoms one or more times per month are classified as abnormal. As the SNQ score increases, the sensitivity decreases and specificity increases. Using a cutpoint of 1, the SNQ was highly sensitive (0.90; 95% CI, 0.77 to 0.97) [Table 4] and specific (0.94; 95% CI, 0.71 to 1.00) [Table 4] for distinguishing between the presence and absence of upper airways disease, with the ROC (Fig 2) having an AUC of 0.97. Among participants with asthma only, the sensitivity and specificity were 0.94 (95% CI, 0.81 to 1.00) and 1.00 (95% CI, 0.47 to 1.00), respectively.

Discussion

We have identified a sensitive, specific questionnaire to screen for chronic rhinitis and sinusitis. We focused on chronic sinonasal disease because patients typically seek treatment for acute symptoms but may ignore more chronic symptoms. We have shown that the diagnostic information from this questionnaire is highly reproducible. The initial diagnosis of chronic sinonasal disease can be made without resorting to more expensive, inconvenient testing.

Chronic rhinitis and sinusitis are part of a disease spectrum in which chronic inflammation leads to leukocyte infiltration, edema, and remodeling in the contiguous nasal and sinus mucosa. Therefore, we included both chronic rhinitis and sinusitis together because the two disease processes represent a continuum. This continuum may explain why the expert panel members disagreed on the specific diagnosis of rhinitis and sinusitis in some participants. We refer to the disease continuum as sinonasal disease because the related term rhinosinusitis usually is reserved for patients who have sinusitis accompanied by rhinitis.^18,19

Rhinitis is reported in up to 25% of the general population, but afflicts up to 90% of patients with asthma.^2,20,21 Sinusitis is reported in 30% to 40% of patients with asthma.^2,22,23 The prevalence reported varies across studies¹ because validated methods rarely are used to make the diagnosis, particularly in cases of rhinitis.

The “gold standard” in this study was the diagnosis rendered by an expert panel. Previous practice guidelines²⁴ have recommended a symptom-based diagnosis for rhinitis, though questions have not been prospectively validated. Guidelines¹⁴ on the diagnosis of chronic sinusitis have recommended using symptom-based criteria, though many²⁵ now recommend imaging. Imaging, either by CT scanning or endoscopy, are frequently used as “gold standards,” though they give complementary information and are not always in agreement.^6,26 Our questionnaire is unique in that we have chosen to diagnose chronic sinonasal disease rather than to focus specifically on rhinitis and sinusitis, as these diseases represent a continuum, particularly in patients with asthma. Because there is no “gold standard” for the diagnosis of chronic sinonasal disease, we chose to incorporate symptoms, and CT scan and endoscopic findings into our evaluation, and to submit these findings to an expert panel. The clinical “gold standard” for diagnosing many disease processes often is contentious; so, for many disease processes, the best “gold standard” available^27,28 represents a consensus opinion among experts.

We used the Delphi method to achieve consensus among panel members rather than a method involving face-to-face meetings.^29,30 With the Delphi method, information is submitted separately to each panelist; there is no contact among members of the panel. The panelists submit an opinion without any discussion. In this study, each panelist communicated only with the data-coordinating center. The advantage of the Delphi method is that one panelist cannot exert undue influence on the group process, but opinions may be altered based on the anonymous feedback from previous rounds of panel review. Thus, this method is effective in achieving a balanced consensus among all panelists.

Our results show that the most sensitive method to diagnose chronic rhinitis and sinusitis was a brief questionnaire using questions adapted from previously published studies.^7–10 Separate reproducibility testing showed that the questionnaire could be significantly improved by eliminating a question pertaining to smell, resulting in the SNQ. A cutpoint of 1 (experiencing each symptom an average of one to four times per month) was highly sensitive and specific for diagnosing chronic sinonasal disease. In these circumstances, the SNQ was superior to endoscopy and CT scan assessment using standard scoring systems. Endoscopy and CT scans provide important anatomic data in patients with complicated sinus disease, especially those who require surgery or a similar intervention, but are not necessary to make the initial diagnosis of chronic rhinitis and sinusitis in patients with asthma.

This tool was not developed to assess the response to treatment, so it should not be used for that purpose. We did not collect information on education status when we administered this questionnaire, which may have implications for its use in certain populations. The SNQ was developed in a post hoc analysis and needs to be validated in a separate cohort.

The SNQ represents an important new tool that could be useful in the care of patients with asthma and for future research studies. It suggests that the initial diagnosis of sinonasal disease can be made by a simple questionnaire rather than by sophisticated imaging studies. Further studies are required to validate this instrument in different populations, to examine the individual components of the instrument, and to determine whether the treatment of disease diagnosed through the use of this questionnaire is warranted.

Abbreviations:

AUC

area under the curve

ROC

receiver operating characteristic

SNQ

sinonasal questionnaire

Appendix: American Lung Association-Asthma Clinical Research Centers

Clinical Research Sites

Nemours Children's Clinic-University of Florida Consortium, Jacksonville, FL; North Shore-Long Island Jewish Health System, New Hyde Park, NY; University of Vermont, Colchester, VT; University of Alabama at Birmingham, Birmingham, AL; University of South Florida, Tampa, FL; and Washington University/St. Louis University, St. Louis, MO.

Members of the Research Group

Nemours Children's Clinic-University of Florida Consortium (J. Lima [principal investigator], G. Josephson and D. Schaeffer [coinvestigators], A. Santos [principal clinic coordinator], and L. Duckworth [coordinator]), Jacksonville, FL; North Shore-Long Island Jewish Health System (R. Cohen [co-principal investigator], J. Karpel [co-principal investigator], and R. Ramdeo [principal clinic coordinator]), New Hyde Park, NY; Vermont Lung Center at the University of Vermont (C. G. Irvin [principal investigator]; A. E. Dixon and D. A. Kaminsky [co-principal investigators]; S. M. Burns [principal clinic coordinator]; and L. M. Bourassa, S. E. Lang, and L. V. Griffes [coordinators]), Colchester, VT; University of Alabama at Birmingham (L. B. Gerald [principal investigator]; R. Grad [coinvestigator]; and S. Erwin, D. Laken, and A. Lewis [coordinators]), Birmingham, AL; University of South Florida (R. Lockey [co-principal investigator], S. Mohapatra [co-principal investigator], M. Grandstaff [principal clinic coordinator], and S. McCullough and B. Fimbel [coordinators]), Tampa, FL; Washington University/St. Louis University (M. Castro [co-principal investigator]; R. Slavin [co-principal investigator]; and M. E. Scheipeter, J. Tarsi, and D. Keaney [coordinators]), St. Louis, MO; and the Data Coordinating Center (R. Wise [center director]; J. Holbrook [deputy director]; E. Brown [principal coordinator]; and C. Levine, J. Jones, R. Masih, S. Modak, D. Nowakowski, N. Prusakowski, D. Shade, and E. Sugar), Johns Hopkins University Center for Clinical Trials, Baltimore, MD.