Does Medical Therapy Improve SNOT-22 Domain Scores?: An Analysis of Clinically Important Differences

Abstract

Objectives:

Minimum clinically important differences (MCIDs) for the 22-item SinoNasal Outcomes Test (SNOT-22) in patients with chronic rhinosinusitis (CRS) electing endoscopic sinus surgery (ESS) are well described. However, similar estimations for the MCID have not been investigated for patients electing continued appropriate medical therapy (CAMT). We sought to determine MCID values for a medically treated CRS cohort and compare them to historical MCIDs associated with ESS.

Methods:

120 patients with refractory CRS electing CAMT were prospectively enrolled from academic referral clinics into an observational cohort study. Baseline and post-treatment SNOT-22 survey responses were collected. Four distribution-based methods for calculating MCIDs (eg. half-standard deviation, Cohen’s d, standard error of measurement, and minimum detectable change) were used to identify a range of MCID values for SNOT-22 total and domain scores.

Results:

The average MCID value for SNOT-22 total scores was 8.0 while mean MCID values for rhinologic, extra-nasal rhinologic, ear/facial, psychological, and sleep symptom domain scores were 3.9, 2.5, 3.3, 3.4, and 2.9 respectively, comparable to previously reported values for patients electing ESS. While change in SNOT-22 total scores following CAMT exceeded the MCID, none of the average SNOT-22 domain score improvements surpassed their respective MCID thresholds.

Conclusion:

MCID values for SNOT-22 total and domain scores in patients electing CAMT are similar to previously published MCID values associated with ESS, indicating that MCID values are independent of treatment modality selection. Therefore, despite evidence of statistical significance, CAMT for CRS may not be associated with clinically discernable improvements in average SNOT-22 domain scores.

INTRODUCTION

Chronic rhinosinusitis (CRS) is a multifactorial disease with several suspected underlying etiologies^1,2, but few definitive answers. Even basic questions involving appropriate treatment of CRS are controversial, with conflicting evidence regarding the relative efficacy of medical and surgical treatments for CRS.^3–6 In order to further address these issues, outcomes research involving disease-specific quality of life (QOL) measures such as the 22-item SinoNasal Outcome Test (SNOT-22) has become widespread.⁷ Unfortunately, the SNOT-22 score is an abstract, multi-faceted measure of QOL, and thus it is difficult to compare population means through tests of statistical significance alone. For example, it is intuitive to compare means reported in tangible metrics such as dollars, kilograms, or years, but defining the difference between 5.0 points and 3.0 points of post-treatment improvement on the rhinologic domain of the SNOT-22 is more challenging. Thus, the concept and calculation of a minimum clinically important difference (MCID) was derived to identify a minimum threshold by which a clinically meaningful result is likely discernible by an individual patient.

Several papers have investigated potential MCID values for the SNOT-22 total score and the five subdomains of the SNOT-22 instrument in surgically managed CRS patients with general agreement.^8,9 However, a similar endeavor has not been attempted for patients electing continued, appropriate medical therapy (CAMT). We could assume that CAMT shares the same SNOT-22 MCIDs, but, as seen in Table 1, MCID threshold values for patients electing CAMT is consistently less than a patient population electing ESS.¹⁰

Table 1:

Comparison between previously published values for average post-treatment improvement in SNOT-22 domain scores in patients undergoing CAMT for CRS and MCID values following ESS.

SNOT-22 domain:	Average Improvement following CAMT10 Mean [±SD]	MCID value following ESS9	Exceeds MCID Threshold?
Rhinologic symptoms	3.0 ± 7.4	3.8	No
Extra-nasal rhinologic symptoms	1.6 ± 3.8	2.4	No
Ear/facial symptoms	1.6 ± 4.6	3.2	No
Psychological dysfunction	2.4 ± 7.6	3.9	No
Sleep dysfunction	1.5 ± 6.7	2.9	No

SNOT-22, 22-item SinoNasal Outcome Test; CAMT, continued appropriate medical therapy; CRS, chronic rhinosinusitis; MCID, minimal clinically important difference; ESS, endoscopic sinus surgery; SD, standard deviation.

Therefore, on a population level, the absolute mean change in SNOT-22 domain scores for CAMT did not meet the MCID criteria defined in surgically managed patients, even though it met criteria for significance testing. Several possible explanations exist for these findings: 1.) the net effect of CAMT across a population of refractory CRS patients is not sufficient, on average, to induce clinically important differences in SNOT-22 symptom scores, or 2.) the specific, previously defined values for the MCID are not generalizable beyond a surgically treated cohort.¹¹ We therefore proposed an independent investigation into the MCID of the SNOT-22 survey instrument in a cohort of patients undergoing CAMT for recalcitrant CRS to address this crucial gap in the literature. By calculating the appropriate MCID thresholds for both the SNOT-22 total and subdomain scores in this population, clinicians and researchers will be better equipped to determine if CAMT is indeed associated with clinically important post-treatment changes.

MATERIALS and METHODS

Study Design and Inclusion Criteria

Findings from this treatment outcomes investigation have been previously published.^12–14 This study is a retrospective analysis of prospectively collected patient cohort data. Study participants (age ≥ 18 years) were sampled from medical practices located within academic, regional referral centers in North America including: Oregon Health & Science University (OHSU; Portland, OR), the Medical University of South Carolina (Charleston, SC), Stanford University (Palo Alto, CA), and the University of Calgary (Calgary, Alberta, Canada). All study participants were diagnosed with CRS by fellowship trained rhinologists per current criteria defined by the American Academy of Otolaryngology¹⁵ between March, 2011 and March, 2015. The Institutional Review Board (IRB) at each enrollment site provided study authorization, annual review, and continual safety monitoring while central IRB, study coordination, and data compilation were conducted at OHSU. Minimal risk study protocols were explained to patients prior to providing informed, adult, written consent in English per good clinical practice guidelines outlined by the International Conference on Harmonisation.¹⁶

Study participants provided extensive social and medical histories during baseline enrollment meetings. Participants were also asked to complete the SNOT-22 survey during initial baseline evaluations and approximately 12-months later during either routine clinical appointments or surveys distributed through personalized mailings. Future treatment modality was not randomized or assigned for investigational reasons, however patients were considered for study inclusion if they voluntarily selected continued medical therapy for on-going control for symptoms associated with CRS. Patients were considered for study enrollment if they had previously completed, at a minimum, one course (14 days) of culture-directed or empiric antibiotics, either corticosteroid nasal sprays (21 days) or systemic corticosteroid therapy (5 days), and daily saline solution irrigation (~240ml). Appropriate continued medical therapy during the observation period was not standardized but prescribed on an individual basis per refractory symptom severity and clinical evidence obtained during routine examinations.

Exclusion Criteria

Study participants with comorbid ciliary dyskinesia/cystic fibrosis or systemic corticosteroid dependency were excluded from final analyses as these disease processes are typically managed differently than other forms of CRS and could potentially skew results. Additionally, enrolled study participants who did not return for routine follow-up appointments or respond to study related follow-up surveys were considered lost to follow-up and excluded from final analyses.

22-item SinoNasal Outcome Test (SNOT-22) Questionnaire

Study participants were asked to complete the SNOT-22, a validated, 22-item survey designed to quantify sinonasal symptom severity (©2006, Washington University, St. Louis, MO). The 22-items of the SNOT-22 have been previously re-categorized into 5 symptom domain scores including as identified through previous factor analysis.⁷ A MCID value for SNOT-22 total scores has been historically defined as within-subject postoperative improvement of at least 8.9 points in patients with CRS electing ESS.^8,9 Study participants were observed through the standard of care up to 12-months after initial baseline enrollment meetings and asked to complete the SNOT-22 during both baseline and follow-up time points.

Data Management and Analysis

Study data was stripped of all protected health information and confidentiality was assured through the use of unique study identification number assignment and data collection using a HIPAA compliant, closed-environment database (Access, Microsoft Corp., Redmond, WA). Secondary analysis of this cohort was completed using commercial statistical software (SPSS v.24; IBM Corp., Armonk, NY). Continuous SNOT-22 data distributions were assessed for normality to guide appropriate statistical comparisons.

Distribution-based methods were used to obtain the MCID of the overall SNOT-22 score and the individual domain scores. The half standard deviation estimate was computed by calculating the standard deviation (SD) of each distribution and multiplying this by 0.5.¹⁷ The standard error of measurement (SEM) MCID score was calculated by first obtaining the internal reliability of each distribution using Cronbach’s alpha.¹⁸ This reliability score (R) was then used to compute the SEM using the formula: $SEM = SD*\sqrt{1-R}$. ^19,20 The minimum detectable change (MDC) MCID score incorporated the SEM using a different formula $\text{MDC}\text{=}\text{1.96}\ast \sqrt{2}\ast SEM$.²¹ Finally, the effect size based MCID used Cohen’s small effect size estimation (0.20) multiplied by the baseline standard deviation of scores to obtain an MCID estimate.²²

RESULTS

All study participants meeting inclusion criteria (n=120) were screened and enrolled between March, 2011 and March, 2015. Final cohort selection was determined after further exclusions for comorbid cystic fibrosis (n=3) and systemic corticosteroid dependency (n=14). A total of 59/103 (57%) patients provided follow-up SNOT-22 evaluations approximately 12 months after baseline enrollment. Patient characteristics and comorbid conditions provided during baseline enrollment meetings for the final cohort are described in Table 2.

Table 2:

Baseline patient characteristics and comorbid conditions in patients with 12-month follow-up after continued appropriate medical therapy (n=59)

Characteristics:	Mean [±SD]	Range	N (%)
Age at enrollment (years)	56.5 [±12.6]	(33 – 82)	-----
Males*	-----	-----	28 (48%)
White/Caucasian	-----	-----	53 (90%)
African American	-----	-----	3 (5%)
Asian	-----	-----	1 (2%)
Hispanic/Latino	-----	-----	1 (2%)
Nasal polyposis	-----	-----	24 (41%)
Turbinate hypertrophy	-----	-----	3 (5%)
Asthma	-----	-----	21 (36%)
AERD/ASA intolerance	-----	-----	5 (9%)
Allergic rhinitis^	-----	-----	21 (36%)
Depression*	-----	-----	4 (7%)
Current tobacco use/smoking	-----	-----	0 (0%)
Current alcohol use	-----	-----	34 (58%)
Diabetes mellitus (Type I / II)	-----	-----	5 (9%)
SNOT-22 total score:	40.1 [±14.8]	(8 – 80)	-----
Rhinologic symptoms	13.8 [±6.1]	(1 – 27)	-----
Extra-nasal rhinologic symptoms	7.1 [±3.2]	(0 – 12)	-----
Ear/facial symptoms	7.3 [±4.8]	(0 – 19)	-----
Psychological dysfunction	10.5 [±5.8]	(0 – 23)	-----
Sleep dysfunction	9.9 [±5.7]	(0 – 24)	-----

^*identified through self-report;

^{^}confirmed via modified radioallergosorbent or skin prick testing; CRS, chronic rhinosinusitis; SD, standard deviation; N, sample size; AERD, aspirin exacerbated respiratory disease; ASA, acetylsalicylic acid; SNOT-22, 22-item SinoNasal Outcome Test.

Improvement in SNOT-22 After Continued Medical Therapy

Scaled data distributions for both baseline and follow-up SNOT-22 total and domain scores were found to be approximately normal. Significant improvement in mean SNOT-22 total and domain scores was reported approximately 12 months after baseline (Table 3).

Table 3:

Mean improvement in SNOT-22 scores following continued appropriate medical therapy for CRS

	12-month Follow-up	Δ value	Paired t-testing statistic	p-value
	Mean [±SD]	Mean [±SD]
SNOT-22 total score:	30.6 [±17.3]	−9.4 [±16.3]	4.4	<0.001
Rhinologic symptoms	10.6 [±6.0]	−3.3 [±6.4]	3.9	<0.001
Extra-nasal rhinologic symptoms	5.5 [±3.0]	−1.8 [±3.2]	4.3	<0.001
Ear/facial symptoms	5.4 [±4.5]	−1.9 [±4.2]	3.3	0.002
Psychological dysfunction	8.2 [±7.1]	−2.2 [±6.2]	2.7	0.008
Sleep dysfunction	7.5 [±5.8]	−2.2 [±5.6]	3.0	0.004

SNOT-22, 22-item SinoNasal Outcome Test; SD, standard deviation. Δ, within-subject change over time.

Distribution-based Methods for MCID Determination

Distribution-based statistical methods for calculating MCID values from SNOT-22 total and domain scores were estimated for the final cohort with 12-month follow-up (Table 4). Cronbach’s alpha reliability estimates of internal consistency of pre-treatment scores were utilized for the calculation of SEM (Table 5). Comparing average MCID values for study participants electing CMT and previously published study results using the same distribution-based methods for patients electing ESS⁹ found that MCID values were similar for all SNOT-22 domains, with the exception of SNOT-22 total scores (Table 6).

Table 4:

Results of distribution-based methods of calculating MCID values for SNOT-22 scores in patients electing CAMT for CRS

	MCID Value Determinations				Mean MCID values
	0.5 standard deviation value	Standard error of measurement (SEM) value	Cohens effect size (d) value*	Minimum detectable change (MDC) value
SNOT-22 total score:	7.4	5.7	3.0	15.9	−8.0
Rhinologic symptoms	3.1	3.0	1.2	8.3	−3.9
Extra-nasal rhinologic symptoms	1.6	2.1	0.6	5.7	−2.5
Ear/facial symptoms	2.4	2.6	1.0	7.3	−3.3
Psychological dysfunction	2.9	2.5	1.2	7.0	−3.4
Sleep dysfunction	2.9	2.0	1.1	5.5	−2.9

MCID, minimal clinically important difference; SNOT-22, 22-item SinoNasal Outcome Test; CAMT, continued appropriate medical therapy; CRS, chronic rhinosinusitis

^*Cohens effect size (d) determined by multiplying baseline standard deviation value by 0.20 to reflect an MCID equal to a minimal effect size.

Table 5:

Cronbach’s reliability estimations for baseline SNOT-22 scores in CAMT

Baseline measures:	Cronbach’s alpha reliability estimate (R)
SNOT-22 total score:	0.85
Rhinologic symptoms	0.76
Extra-nasal rhinologic symptoms	0.58
Ear/facial symptoms	0.70
Psychological dysfunction	0.81
Sleep dysfunction	0.88

SNOT-22, 22-item SinoNasal Outcome Test; CAMT, continued appropriate medical therapy

Table 6:

Comparison of average MCID values for SNOT-22 scores between study participants electing CAMT and ESS.

	Treatment Modalities:
	Continued Appropriate Medical Therapy (n=59)	Endoscopic Sinus Surgery9 (n=276)
SNOT-22 total score:	−8.0	−9.0
Rhinologic symptoms	−3.9	−3.8
Extra-nasal rhinologic symptoms	−2.5	−2.4
Ear/facial symptoms	−3.3	−3.2
Psychological dysfunction	−3.4	−3.9
Sleep dysfunction	−2.9	−2.9

MCID, minimal clinically important difference; SNOT-22, 22-item SinoNasal Outcome Test; CAMT, continued appropriate medical therapy; ESS, endoscopic sinus surgery.

DISCUSSION

Similar to prior investigations of this patient cohort, we confirmed that CAMT of refractory CRS improves SNOT-22 scores over time, achieving a mean improvement of 9.4 points (Table 3), beyond the CAMT MCID threshold of 8.0 points. However, when comparing mean improvements in SNOT-22 domain scores with their respective MCID thresholds, we observe a notable divergence. Study participants electing CAMT did not report average improvements which met or exceeded the MCID threshold across any of the five symptom domain scores. This is despite the fact that SNOT-22 MCID threshold values are remarkably similar between the CAMT cohort and the surgical cohort (Table 6). One way to interpret this finding is that the average CRS patient electing CAMT reports a ‘global’ improvement over their overall condition without noticing that any particular symptom group is meaningfully improved. While somewhat paradoxical, this is a result of looking at population means rather than an individual patient’s outcome. These findings are in contrast to ESS, where prior work has shown clinically discernable improvements in sleep, psychological dysfunction, and rhinologic symptom groups.^10,23,24

Interestingly, the calculated MCID for the SNOT-22 total score was exactly 1.0 point less for the CAMT group compared to previous determinations for the surgical cohort, perhaps reflecting a lower threshold for clinically meaningful improvement. Due to the distribution-based methodology used to compute the MCID, this difference may be partially reflective of reduced data variability in pre-treatment SNOT-22 scores within the CAMT group. Each potential MCID value then carries a trade-off between sensitivity and specificity, with lower values increasing the number of false positives and higher values increasing the number of false negatives. While it is difficult to ascribe much clinical value to this difference, one could posit that this may be indicative of a higher expectation of improvement in patients electing ESS. Alternatively, it is possible that this difference is merely statistical variance, and these MCID determinations approximate enough that the more conservative value of 9 is appropriate for the general CRS population. Ultimately, MCID determinations using these methods are relative and a useful heuristic, rather than an absolute ‘gold standard’. Given the approximation of the two total MCID scores, the similar range of domain scores, and the more robust data in the ESS group, it may be reasonable to generalize the MCID beyond treatment modality selection in this patient population, but we are unable to definitively state this conclusion without further validation in a larger group.

There are two major findings from this study that potentially enhance our understanding of clinical populations with CRS. First, post-treatment improvements associated with CAMT for refractory CRS do not meet MCID criteria for the SNOT-22 domains as previously described within patient populations undergoing ESS. This distinction may have clinical implications when counseling patients regarding potential treatment modalities. For example, consider a patient presenting with notable ear/facial symptom complaints attempting to decide between ESS or CAMT as subsequent treatment options. Prior work has shown that the mean improvement in this domain with ESS is 4.9 points⁹, while only 1.9 points of change were with CAMT (Table 3). Both values were significantly improved with respect to baseline measures, indicating that there is likely some degree of improvement on average with both therapies, but only ESS meets the MCID criteria of 3.2 points. Therefore, ESS is more likely to be associated with a patient reporting an improvement in ear/facial symptoms when compared to CAMT. Of course, there are multiple other factors to consider when choosing a treatment option, but in terms of efficacy and mean improvement in symptom severity reduction, it appears that ESS has an advantage on a population level. Second, the similarity of values between the medical and surgical cohorts suggests that we can generalize MCID domain values beyond treatment modality selection and use a single set of values to analyze changes in status.

Several limitations of this study should qualify our findings. First, the medical cohort had a smaller sample size compared to the previously published cohort electing sinus surgery. As patients enrolled in the study selected a preferred treatment modality, this discrepancy reflects a higher propensity to elect ESS in an observational study cohort without randomly assigned treatments in our patient population. Thus, confirmation of these findings in a larger, medically treated cohort would help externally validate these results. Additionally, only 56% of patients returned for the 12-month follow-up, which could overweigh outcomes of non-responding patients who are seeking care for persistent symptoms. Second, these results are reflective of a tertiary-care academic rhinology practice with refractory CRS patients and may not necessarily be generalizable to a wider CRS population in alternative care settings. Further study with high quality, prospective data collection of both CAMT and surgically treated patients would be necessary to externally validate these findings. Third, these reported MCID values are reflective of current standards of care for both medical and surgical therapy, and may not hold true if significant shifts in either treatment modality occur in the future, such as increased use of biologics for CRS or similar targeted therapies.¹¹ Fourth, an anchor-based determination of the MCID was not able to be performed in this study, which could assist in independently confirming the distribution-based findings.²⁵ Fifth, CAMT was left to the discretion of each individual practitioner by design, thus introducing potential heterogeneity to the medical treatment arm. This was chosen to allow providers the flexibility to select treatments as the clinical picture of the patient evolved, but it does limit generalizability compared to the surgical cohort, where there is more standardization of treatment. Ultimately, this may be a reflection of our limited understanding of how medical therapy modifies the inflammatory process of CRS compared to the structural issues that surgery tends to address. Further dedicated work is needed to determine what the optimal CAMT should consist of, ideally through a randomized trial. Finally, while MCID determinations are useful tools for comparison of outcomes over time between populations, they are not necessarily applicable at an individual level. Patients may have their own independent measure of clinically important differences in perceived health status. Thus, this data and its findings should be used in a judicious manner to supplement the individualized treatment decision process rather than serve as rigid guidelines. There is still a significant role for continued medical therapy for many patient subsets of CRS and further research is needed to better identify which patients are better suited for certain treatment modalities than others.

CONCLUSION

Determinations of MCID threshold values for SNOT-22 domain scores in patients electing CAMT for refractory CRS are similar to published MCID values associated with surgical intervention. In contrast to ESS, longitudinal SNOT-22 domain outcomes of CAMT after 12 months are less than the MCID, suggesting that CAMT may not be associated with individually discernable changes in sinonasal QOL symptom domains. Further validation is needed to confirm these findings.