Sensitivity Analysis and Diagnostic Accuracy of the Brief Smell Identification Test in Patients with Chronic Rhinosinusitis

Abstract

Background

The Brief Smell Identification Test (BSIT) is an abbreviated version of the Smell Identification Test (SIT) used to assess olfactory function. Although the BSIT can be efficiently administered in under 5 minutes, the accuracy of the BSIT in relation to the SIT in patients with chronic rhinosinusitis (CRS) is unknown.

Methods

Patients with CRS were recruited as part of an ongoing multi-institutional observational cohort study. A total of 183 participants provided both BSIT and SIT olfactory function scores during initial enrollment. Linear associations between BSIT and SIT scores were evaluated using Pearson’s correlation coefficients (r_p). Sensitivity and specificity of BSIT scores were determined using SIT scores as the ‘gold standard’.

Results

A strong bivariate linear association was found between BSIT and SIT scores (r_p=0.893; p<0.001) for all participants. A significantly lower proportion of patients were identified as having abnormal olfaction using the BSIT compared to the SIT (47% vs. 68%, respectively; p<0.001). Using the currently defined score of ≤ 8 as a cut-point for abnormal olfactory function, the BSIT demonstrated a sensitivity of 63% and specificity of 88% with an overall accuracy of 71%. Increasing the cut-point to ≤ 9 resulted in an increased sensitivity of 86%, a specificity of 76%, and an improved overall accuracy of 83%.

Conclusion

In patients with CRS, BSIT scores strongly correlate with SIT scores; however, the BSIT underestimates olfactory dysfunction as defined by the suggested cut-point of ≤ 8. Increasing the cut-point to ≤ 9 increased the sensitivity and accuracy of the BSIT.

INTRODUCTION

Olfactory dysfunction is one of the leading complaints in patients with chronic rhinosinusitis (CRS) and is associated with decreased quality of life (QOL).[1, 2] Established guidelines have categorized a decreased or lost sense of smell as a cardinal symptom in the diagnosis of CRS underscoring its diagnostic significance.[3, 4] Several objective instruments are available to quantify olfactory function; however, evaluation can prove burdensome in a busy clinical practice.

Two of the most widely used tests to evaluate olfactory function in the United States include the Smell Identification Test (SIT) and the Brief Smell Identification Test (BSIT). The SIT is a highly reliable, 40-item odor-identification test with a proven history assessing olfactory function in a broad array of patient populations.[5–10] Despite its’ investigational utility and diagnostic accuracy, the SIT is relatively time consuming which may prevent wide-spread adoption in a robust clinical setting. In contrast, the BSIT is an abbreviated version of the SIT incorporating 12 cross-cultural yet analogous odorants with the primary advantage of being easily administered in under 5 minutes.[11]

Although the BSIT is an efficient tool to assess olfactory function, the ability of the BSIT to accurately identify olfactory function in patients with CRS is not yet known. This study aims to determine the sensitivity and accuracy of the BSIT as compared to the SIT in patients with CRS. By understanding how the BSIT performs in this patient population, clinicians will be able to better select the appropriate test for their clinical and investigational needs.

MATERIALS and METHODS

Patient population and Inclusion Criteria

Study participants were recruited and prospectively enrolled into a multi-site, observational cohort investigation designed to evaluate various treatment outcomes for CRS in a non-randomized fashion. Adult patients (≥ 18 years) were diagnosed with medically refractory CRS defined by current criteria described by both the American Academy of Otolaryngology and the European Position Paper on Rhinosinusitis and Nasal Polyps 2012 (EPOS 2012).[3, 4] All patients had completed previous medical therapy aimed towards alleviation of symptoms related to CRS including, but not limited to, at least one course (≥ 14 days) of broad spectrum or culture-directed antibiotics and at least one course of either topical corticosteroids (≥ 21 days) or oral corticosteroid (≥ 5 days) therapy. Before baseline enrollment meetings, patients elected either endoscopic sinus surgery (ESS) or further medical management for continued symptom mitigation. Preliminary findings from this investigation have been previously reported.[12–16]

The Institutional Review Board (IRB) at each enrollment site governed all study protocols and informed patient consent documentation. Study enrollment sites were comprised of sinus and skull base surgery clinics within academic, tertiary hospital systems in the United States including Oregon Health & Science University (OHSU; Portland, OR, eIRB#7198), the Medical University of South Carolina (MUSC; Charleston, SC, IRB#12409), and the University of Utah (UoU, Salt Lake City, UT, IRB#61810). Study participants were assured study participation was voluntary and standard of care surrounding subsequent treatment decisions was in no way altered due to study procedures.

During baseline enrollment meetings participants were asked to provide demographic information, as well as medical and social history cofactors including, but not limited to: age, gender, race, asthma, nasal polyposis, history of prior sinus surgery, septal deviation, depression, obstructive sleep apnea, allergy, acetylsalicylic acid (ASA) sensitivity, current smoking/tobacco use, gastroesophageal reflux, ciliary dyskinesia, immunodeficiency, corticosteroid dependency, and diabetes mellitus.

Comparative Measures of Olfactory Function

Olfactory function was evaluated at baseline using the Smell Identification Test (SIT; Sensonics, Inc., Haddon Heights, NJ; formerly known as the University of Pennsylvania Smell Identification Test). The SIT is a validated, non-invasive, 40-item, test with high test-retest reliability (r > 0.90).[5, 17] The SIT employs 40 microencapsulated odorant strips which are activated using a #2 pencil utilizing a “scratch ‘n sniff” format used to operationalize olfactory function (score range: 0–40). The SIT is considered a forced choice test wherein each correctly identified odorant is selected out of 4 total options including 3 additional distractor odorants. Total SIT scores are categorized into olfactory dysfunction diagnoses based on gender-adjusted normative data with higher scores representing greater olfactory function [17]. Participants with SIT scores between 6–33 for males or 6–34 for females were categorized as abnormal while SIT scores ≤ 5 were categorized as probable malingering and removed from final analysis. The categorization of probable malingering for scores ≤ 5 is based on the statistical likelihood that a subject with total anosmia will score 10 out of 40 items correctly by pure chance alone as well as evidence that subjects asked to feign total anosmia will consistently score ≤ 5 items correctly.[5]

The Brief Smell Identification Test (BSIT; Sensonics, Inc., Haddon Heights, NJ) is a validated 12-item, non-invasive test of olfactory function utilizing odorant strips in a duplicate fashion to the SIT instrument. The BSIT is an abbreviated instrument using 12 cross-cultural analogous odorant items embedded within the 40-item SIT instrument but incorporating varying distractors and test item positions. BSIT scores were extracted from the SIT test by scoring the corresponding odorants embedded within the SIT. Total BSIT scores (score range: 0–12) can be categorized into olfactory function diagnoses based on gender-adjusted adult normative data where higher scores represent greater olfactory function.[11, 18] Both male and female respondents of all ages are categorized as having abnormal olfaction if BSIT scores are ≤ 8.

Data Management and Statistical Analysis

Study data was coded using a unique study identification number to ensure patient confidentiality, and transferred to OHSU from each enrollment site. All data was manually entered into a relational database (Microsoft Access, Microsoft Corp., Redmond, WA) and statistical analyses were conducted using SPSS v.22 (IBM Corp., Armonk, NY). Baseline patient cofactors and olfactory function scores were evaluated descriptively while normality was verified for all continuous measures. The prevalence of normal and abnormal olfaction was described for both instruments and compared using McNemar’s chi-square (χ²) test for matched pairings. Linear associations between SIT and BSIT scores was evaluated using Pearson’s correlation coefficients (r_p) for both total and patient subgroups – including participants with and without nasal polyposis (CRSwNP and CRSsNP, respectively) and with and without a history of prior sinus surgery. Further sensitivity and specificity was calculated to determine the diagnostic accuracy of BSIT scores to detect abnormal olfaction using gender-adjusted SIT scores as the ‘gold standard’ for both the total cohort and patient subgroups. The overall accuracy was calculated using the formula: [((True [+]) + (True [−]))/Total N]. The ability of BSIT scores to accurately discriminate abnormal vs normal olfaction was evaluated using receiver operating characteristics (ROC) curves. The resultant areas under the ROC curve (AUC), standard error [SE], and 95% confidence intervals were calculated. Conventional definitions for AUC findings were used.[19] All reported p-values less than 0.050 were considered significant.

RESULTS

Cohort Characteristics

A total of 187 study participants completed baseline olfactory function evaluations between June, 2013 and March, 2015. A total of 183 were available for final analysis after exclusion of 4 participants due to probable malingering. Participant characteristics and average olfactory test scores are described in Table 1.

Table 1.

Study cohort characteristics (n=183)

Characteristics:	N (%)
Age (years), mean (SD, range)	49.3 (16.4, 18–80)
Males	89 (49%)
Females	94 (51%)
White	164 (90%)
Asthma	78 (43%)
Nasal polyposis	69 (38%)
Prior sinus surgery	105 (57%)
Septal deviation	46 (25%)
Depression	33 (18%)
Obstructive sleep apnea	23 (13%)
Allergy (mRAST/skin prick confirmed)	116 (63%)
ASA sensitivity	18 (10%)
Current smoker/tobacco use	5 (3%)
GERD	59 (32%)
Ciliary dyskinesia	6 (3%)
Immunodeficiency	13 (7%)
Corticosteroid dependency	26 (14%)
Diabetes mellitus (Type I/II)	16 (9%)
SIT Olfactory Function Score, mean (SD, range)	27.7 (9.4, 6–38)
BSIT Olfactory Function Score, mean (SD, range)	7.9 (2.9, 1–12)

SD, standard deviation; LL, lower limit; UL, upper limit; mRAST, modified radioallergosorbent test; ASA, acetylsalicylic acid; GERD, gastroesophageal reflux disease; SIT, Smell Identification Test; BSIT, Brief Smell Identification Test.

Categorizations of diagnosis frequency for olfactory dysfunction for both the SIT and BSIT scores are described for the total study cohort in Table 2. Compared to SIT diagnoses, a significantly lower proportion of patients were identified as having abnormal olfaction using the BSIT instrument for the total cohort (68% vs. 47%, respectively; p<0.001). Similar differences in the prevalence of abnormal olfaction categorized by SIT and BSIT scores were found for both study participants with CRSwNP (84% vs. 68%, respectively; p=0.007) and CRSsNP (59% vs. 34%, respectively; p<0.001), as well as patients with a history of prior sinus surgery (66% vs. 47%, respectively; p<0.001) and those without prior sinus surgery (72% vs. 47%, respectively; p<0.001).

Table 2.

Olfactory dysfunction scoring diagnostic categories for total study cohort (n=183)

	SIT	BSIT
Diagnoses:	N (%)	N (%)
Abnormal olfaction	125 (68%)	86 (47%)
Complete anosmia	36 (20%)	----
Severe hyposmia	16 (9%)	----
Moderate hyposmia	27 (15%)	----
Mild hyposmia	46 (25%)	----
Abnormal relative to age	----	65 (36%)
Deficit relative to younger persons	----	19 (11%)
Normal olfaction	58 (32%)	97 (53%)

SIT, Smell Identification Test; BSIT, Brief Smell Identification Test.

Highly significant bivariate linear association was found between SIT and BSIT scores for the total cohort (r_p=0.893; p<0.001). Similar strong magnitudes of correlations were found for both CRSwNP (r_p=0.899; p<0.001) and CRSsNP (r_p=0.857; p<0.001), as well as patients with a history of prior sinus surgery (r_p=0.913; p<0.001) and those patients without prior sinus surgery (r_p=0.861; p<0.001).

Analysis of Diagnostic Accuracy

Using SIT as the ‘gold standard’, the frequency of true positives, true negatives, false positives, and false negatives of categorized normal and abnormal BSIT scores using a score of ≤ 8 to define abnormal olfaction are described for the entire cohort in Table 3. Scores from the BSIT were determined to have a positive predictive value (PV+) of 92% (95% CI: 86% – 98%) in this patient population with CRS, with a sensitivity of 63% (95% CI: 55% – 72%) and a specificity of 88% (95% CI: 80% – 96%). The overall accuracy of the BSIT for the entire cohort of patients with CRS was 71% (95% CI: 65% – 78%).

Table 3.

Comparison of normal and abnormal SIT and BSIT scores for the total cohort

BSIT Scores		SIT Scores “Gold standard”
		Abnormal (+)	Normal (−)	Total N
	Abnormal (+)	79 (True +)	7 (False +)	86
	Normal (−)	46 (False −)	51 (True −)	97
	Total	125	58	183

SIT, Smell Identification Test; BSIT, Brief Smell Identification Test.

Categorized scores of the BSIT instrument in study participants with CRSwNP had a PV+ of 96% (95% CI: 90% – 99%) with a sensitivity of 78% (95% CI: 67% – 88%) and a specificity of 82% (95% CI: 59% – 98%). The overall accuracy of the BSIT in the CRSwNP subgroup was 78% (95% CI: 69% – 88%). Similarly, in study participants with CRSsNP, the BSIT had a PV+ of 87% (95% CI: 77% – 98%) with a sensitivity of 51% (95% CI: 39% – 64%) and specificity of 89% (95% CI: 81% – 98%). The overall accuracy of the BSIT in the CRSsNP subgroup was 67% (95% CI: 58% – 75%).

Categorized scores of the BSIT instruments in study participants with a history of prior sinus surgery had a PV+ of 90% (95% CI: 81% – 98%) with a sensitivity of 64% (95% CI: 52% – 75%) and a specificity of 86% (95% CI: 75% – 97%). The overall accuracy of the BSIT in this subgroup was 71% (95% CI: 63% – 80%). Likewise, in study participants without a history of prior sinus surgery, the BSIT had a PV+ of 95% (95% CI: 87% – 99%) with a sensitivity of 63% (95% CI: 50% – 75%) and specificity of 91% (95% CI: 79% – 99%). The overall accuracy of the BSIT in this subgroup was 71% (95% CI: 60% – 81%).

ROC Curve Analysis

The frequencies of true normal and abnormal olfaction as designated by the SIT, are tabulated for each BSIT score in Table 4. Sensitivity and specificity for each discrete BSIT cut-point are listed in Table 5. The overall area under the ROC curve (Figure 1) was determined to be 0.873 (SE 0.027; 95% CI: 0.819, 0.927), indicating that the BSIT has excellent discriminative ability in identifying abnormal olfaction. Increasing the cut-point to ≤9 improved the sensitivity of the BSIT to 86% (Table 4) and optimized the overall accuracy at 83% though the specificity declined to 76%. Additionally, the proportion of patients identified as having olfactory dysfunction using a cut-point of ≤ 9 on the BSIT was more comparable to the proportion of olfactory dysfunction as identified by the SIT (67% vs. 68%, respectively; p=0.720).

Table 4.

Frequency of SIT normal and abnormal scores per discrete BSIT scores

BSIT Score:	1	2	3	4	5	6	7	8	9	10	11	12	Totals
SIT Normal	0	0	1	0	0	0	0	6	7	16	19	9	58
SIT Abnormal	7	6	8	5	10	13	14	16	29	11	4	2	125

SIT, Smell Identification Test; BSIT, Brief Smell Identification Test;

Table 5.

Sensitivity and specificity values for discrete BSIT scores

Coordinates of the ROC Curve
BSIT Score cut-point for abnormal olfaction:	Sensitivity	Specificity	1-Specificity
≤ 0	0.000	1.000	0.000
≤ 1	0.056	1.000	0.000
≤ 2	0.104	1.000	0.000
≤ 3	0.168	0.983	0.017
≤ 4	0.208	0.983	0.017
≤ 5	0.288	0.983	0.017
≤ 6	0.392	0.983	0.017
≤ 7	0.504	0.983	0.017
≤ 8	0.632	0.879	0.121
≤ 9	0.864	0.759	0.241
≤ 10	0.952	0.483	0.517
≤ 11	0.984	0.155	0.845

ROC, receiver operating characteristics; BSIT, Brief Smell Identification Test;

Figure 1.

Receiver operating characteristic curve for BSIT scores (AUC=0.873, 95% CI: 0.819–0.927). The green diagonal line represents uninformative test (eg. sensitivity + specificity = 1.0, a diagnostic no better than chance alone). The area between the blue and green lines represents the AUC. BSIT, Brief Smell Identification Test; AUC, area under the curve; CI, confidence interval.

DISCUSSION

This study evaluates the diagnostic accuracy of the BSIT as compared to the SIT in detecting olfactory dysfunction in patients with CRS. The analysis demonstrates a strong correlation between the two tests; however, the BSIT was found to significantly underestimate the presence of olfactory dysfunction when compared to the SIT (47% vs. 68%, respectively; p<0.001). Furthermore, the overall diagnostic accuracy for the BSIT is relatively poor using the suggested cut-point of ≤ 8 for abnormal olfaction. When the analysis was broken down by CRS sub-types (CRSwNP and CRSsNP), similar trends were demonstrated. Likewise, the BSIT performed in analogous fashion when evaluated in patients who had either a positive or negative history of prior sinus surgery.

This sensitivity analysis suggests that a different cut-point could be used to define abnormal olfaction in patients with CRS, as assessed by the BSIT, in order to better reflect the diagnostic abilities of the SIT. The currently defined cut-point of ≤ 8 carries a sensitivity and specificity of 63% and 88%, respectively, with an overall accuracy of 71%. Increasing the cut-point to ≤ 9 improved the sensitivity of the BSIT to 86% and optimized the overall accuracy to 83%. Additionally, the proportion of patients identified as having olfactory dysfunction using the cut-point of ≤ 9 (67%) on the BSIT is more comparable to the proportion of patients with olfactory dysfunction as identified by the SIT. Although the specificity declined to 76% using the increased cut-point, the improvements in sensitivity and overall accuracy would be expected to have greater clinical relevance due to the high pre-test probability of olfactory dysfunction in patients with CRS, which attenuates the impact of a lower specificity.

When the BSIT was developed by Doty et al., it was introduced as an efficient yet reliable substitute for the SIT when olfactory function needed to be assessed in less than 5 minutes.[11] In the initial report, comparability between the BSIT and the 12 analogous SIT items was established using scores from 198 healthy, nonsmoking individuals without a history of sinonasal or olfactory disorders. Although a sensitivity analysis of the BSIT was not performed, Doty acknowledged that the BSIT is less sensitive in detecting minor alterations in smell function due to the test’s lower reliability (r=0.71 for the BSIT compared to r=0.92 for the SIT) and narrower range of scoring leading to potential overlap of scores near the defined cut-point.[20] Supporting this notion, 25% of study participants in our current analysis scored in the “mild hyposmia” category using the SIT instrument. This is a rather large percentage of our cohort with a relatively subtle alteration in smell function and may, in large part, account for the discrepancies between the BSIT and SIT. In fact a total of 38/46 (83%) subjects categorized with “mild hyposmia” using SIT scoring results were alternatively determined to have “normal” olfactory function using analogous BSIT scores as defined by the standard ≤ 8 cut-point.

With over 28.5 million dollars in NIH funding tied to olfactory research in 2015 alone, this study has substantial value in guiding both clinical rhinology and refining olfactory research methodology.[21] This study is the first to assess the sensitivity of the BSIT as compared to the SIT in detecting olfactory dysfunction in patients with CRS. While the sensitivity of the BSIT is decreased in patients with CRS using the currently defined cut-point, the strong correlation and efficient administration suggests that the BSIT has the potential to be a useful tool for identification of olfactory dysfunction when clinical time restrictions do not permit the use of the UPSIT. Providers administering the test to patients with CRS should consider using the ≤ 9 cut-point but must consider the trade off between accuracy and time spent completing the test when assessing olfaction in a robust clinical practice. For investigational purposes, the UPSIT should be preferentially used due to its higher reliability and ability to sub-classify olfactory dysfunction. Consideration should be made, however, to incorporate BSIT scores as an investigational tool in addition to UPSIT scoring if it is expected that the majority of clinicians will use the BSIT so that findings can be translated and externally validated within the clinical setting.

It is important to note that for this analysis, the BSIT was not administered in its commercially available form. Rather, BSIT scores were extracted from the UPSIT test by scoring the corresponding odorants. Differing test item positions and response alternatives in the commercially available BSIT could affect the outcome of the test and introduce potential limitations to the reported data and subsequent conclusions. In the development of the BSIT, however, a comparability analysis of the BSIT items to the 12 analogous UPSIT items was performed and no statistical difference in mean score or frequency distribution was identified. [11] A second potential study caveat is that despite the multi-institutional nature of this study, the optimized cut-point identified may be a reflection of our patient population and should not be generalized to those without CRS or to patients in alternate care settings. Future study will be necessary to determine if similar findings are present in other patient populations.

CONCLUSION

The BSIT is an efficient tool to assess olfactory function. In patients with CRS, BSIT scores correlate strongly with SIT scores; however, the BSIT demonstrates suboptimal sensitivity and accuracy using the currently defined cut-point of ≤ 8 for abnormal olfaction. Increasing the cut-point to ≤ 9 improved the sensitivity and overall accuracy of the BSIT in this patient population. Providers employing the BSIT in their clinical practice should consider using this new cut-point when evaluating patients with CRS.