Establishing the Minimal Clinically Important Difference for the Questionnaire of Olfactory Disorders

Abstract

Introduction

Olfactory-specific quality of life (QOL) can be measured using the Questionnaire of Olfactory Disorders Negative Statements (QOD-NS). Changes in the QOD-NS after treatment can be difficult to interpret since there is no standardized definition of clinically meaningful improvement.

Methods

Patients with chronic rhinosinusitis (CRS) completed the QOD-NS. Four distribution-based methods were used to calculate the minimal clinically important difference (MCID): 1) one-half standard deviation (SD), 2) standard error of the mean (SEM), 3) Cohen’s effect size (d) of the smallest unit of change, and 4) minimal detectable change (MDC). We also averaged all four of the scores together. Finally, the likelihood of achieving a MCID after sinus surgery using these methods, as well as average QOD-NS scores, was stratified by normal vs. abnormal baseline QOD-NS scores.

Results

Outcomes were examined on 128 patients. The mean improvement in QOD-NS score after surgery was 4.3 [SD±11.0] for the entire cohort and 9.6 [SD±12.9] for those with abnormal baseline scores (p<0.001). The MCID values using the different techniques were: 1) SD = 6.5; 2) SEM = 3.1; 3) d = 2.6; 4) MDC = 8.6. The MCID score was 5.2 on average. For the total cohort analysis, the likelihood of reporting a MCID ranged 26%–51%, and49%–70% for patients reporting preoperative abnormal olfaction.

Conclusions

Distribution-based MCID values of the QOD-NS range between 2.6 to 8.6 points, with an average of 5.2. When stratified by preoperative QOD-NS scores the majority of patients reporting abnormal preoperative QOD-NS scores achieved a MCID.

INTRODUCTION

Olfactory dysfunction (OD) is a known sequela of chronic rhinosinusitis (CRS).¹ OD affects 40% – 80% of patients with CRS depending on the method of measurement and population studied, while significantly decreasing olfactory-specific quality of life (QOL).^1–3 Olfactory-specific QOL can be measured using the Questionnaire of Olfactory Disorders Negative Statements (QOD-NS) survey, which is a previously validated instrument used to analyze multiple aspects of how changes in olfaction impact an individual’s daily life.^4,5

The QOD-NS is being used with increasing frequency in treatment outcome studies relating to CRS-associated OD and endoscopic sinus surgery (ESS). Particularly, changes in QOD-NS after ESS or medical treatment modalities for CRS are of increasing interest. While post-treatment improvement in objective olfaction is of importance, the impact of OD on QOL is equally relevant. We have previously shown that QOD-NS scores improve after ESS³, and that changes in QOD-NS have a stronger association with economic productivity than changes in psychophysical olfactory parameters.² However, while it is understood that improvements in the QOD-NS reflect better olfactory-specific QOL, the interpretation of the magnitude of change after treatment remains uncertain.

Defining a minimal clinically important difference (MCID) is one method to ascertain whether post-treatment changes in patient-reported outcome measures are clinically significant and perceptible to patients.⁶ The MCID is an individual threshold value that would be considered meaningful or worthwhile by a patient if he/she were to consider intervention again in the future.⁶ Our objective was to develop an MCID value for the QOD-NS in a heterogeneous CRS population that would improve utility of this instrument in future studies, and to determine the proportion of the patients in our population that achieved MCID in the QOD-NS after ESS.

MATERIALS and METHODS

Study Population and Inclusion Criteria

Adult (≥18 years) patients were prospectively enrolled from academic, tertiary care centers in North America with medically refractory symptoms of CRS. Confirmed diagnosis of CRS was provided using criteria provided by the American Academy of Otolaryngology.^7,8 Participating enrollment sites included Departments/Divisions of Otolaryngology-Head and Neck Surgery within: Oregon Health & Science University (OHSU, Portland, OR.), the Medical University of South Carolina (Charleston, SC.), Stanford University (Palo Alto, CA.), and the University of Calgary (Calgary, AB, Canada). Previous outcomes of olfactory dysfunction from this observational, multi-centered cohort study have been described in the literature.^3,9–12 Study participants provided written informed consent in English during initial enrollment meetings. The Institutional Review Board at each enrollment site provided annual review and safety monitoring with central regulatory study oversight at OHSU.

Study participants were asked to provide medical histories to verify previous attempt at recent therapeutic management including: at least one course (≥14 days) of empiric or culture-directed antibiotics, either corticosteroid nasal spray application (≥21 days) or oral corticosteroid therapy (≥5 days), and daily nasal saline irrigations as needed (~240ml. daily).

Following extensive physician counseling, participants elected ESS due to insufficient symptom resolution from previous therapeutic management. Surgical intervention was not randomized or assigned for study purposes. Surgical approach was formulated by each enrolling physician using radiographic imaging and endoscopic examinations to determine disease extent and anatomic location. Surgery was either primary or revision ESS and conducted under general anesthesia for all cases. Procedures consisted of unilateral or bilateral maxillary antrostomy, partial or total ethmoidectomy, sphenoidotomy, and/or Draf (type: 2a, 2b, or 3) frontal sinusotomy as needed. Anatomic ventilation was further maximized by incorporating either inferior turbinate reduction and/or septoplasty if indicated. Postoperative management included continued nasal saline irrigations (QD) and topical corticosteroid sprays/rinses to facilitate optimal postoperative healing. Study participants were followed through the postoperative standard of care up to 18 months. Follow-up evaluations occurred during routine clinical appointments or mailed responses using self-addressed, stamped envelopes sent to study participants at regular 6 month intervals.

Radiographic Imaging and Endoscopic Examinations

Assessments of disease severity, collected during standard preoperative clinical assessment, were used concurrently for study purposes. High resolution computed tomography (CT) was utilized to evaluate sinonasal inflammation and quantified by each enrolling physician using the Lund-Mackay staging system (score range: 0–24), which grades the severity of opacification in the maxillary, ethmoid, sphenoid, osteomeatal complex, and frontal sinuses.¹⁴ Postoperative CT images were not collected for study purposes due to risk of radiation exposure.

The paranasal sinuses were also preoperatively evaluated using rigid, fiberoptic endoscopes (Karl Storz, Tuttlingen, Germany) and quantified by each enrolling physician using the bilateral Lund-Kennedy endoscopy scoring system (score range: 0–20) which quantifies attributes within the paranasal sinuses including the presence and severity of: nasal polyposis, discharge, edema, scarring, and crusting.¹⁵ Higher overall total scores on both staging systems represent worse disease severity.

Primary Patient Reported Outcome Measure (PROM)

As the primary outcome of interest to this study, patients were asked to complete the Negative Statements portion of the Questionnaire for Olfactory Disorders (QOD-NS) both preoperatively and postoperatively in order to quantify patient perception of olfactory function. The QOD-NS is a validated, olfactory specific QOL survey that consists of 17 discrete survey items summarized using Likert score responses from 0=“disagree” to 3=“agree” (score range: 0–51) where higher scores represent worse olfactory impairment.⁴ Study participants were asked to complete the QOD-NS at baseline and after ESS to determine the prevalence of postoperative improvement in OD. The follow-up intervals ranged between 6 and 18 months. Postoperative scores were defined using the last available evaluation provided by study participants due to known statistical stability in mean olfactory function. ⁹

Further, we stratified our analysis for “high” vs. “low” QOD-NS scores at baseline with a cut-off score of 12.5. We have previously published this cut-off score for the QOD-NS. Originally, the score cut-off of 38.5 has been previously reported that stratifies QOD-NS scores in patient with normal vs. abnormal olfaction on objective psychophysical testing (hyposmia and anosmia).¹² For this study, we reverted our scoring to the original scoring method reported by Hummel et al.¹³, where high scores reflect poor QOL and low scores reflect good QOL. As such, we used the numerical inverse of 38.5 to obtain a cut-off score of 12.5 in order to reflect normal vs. abnormal scores.

Data Management and Statistical Analyses

Patient data was safeguarded thorough unique identification number assignments and removal of protected health information on clinical report forms. Study data was transferred to OHSU for data entry into a closed, relational database (Access; Microsoft Corp., Redmond, WA) in compliance with the Health Insurance Portability and Accountability Act. Secondary data analysis was completed using commercial software (SPSS v.24; IBM Corp., Armonk, NY). All data was evaluated descriptively and distributions of ordinal and continuous measures were assessed for normality.

Improvements in mean QOD-NS total scores were compared for the entire cohort using two-sided, matched pair t-testing. Determinations of MCID values for the QOD-NS was performed using preoperative QOD-NS scores and four distinct analytical approaches. First, a calculation of ½ the standard deviation of the preoperative mean score was used.¹⁶ This method of calculating an MCID is widely accepted and appears to be rooted in basic human psychology, where the ability of individuals to make absolute discriminations on a variety of tasks or questions frequently falls in the values corresponding to ½ the standard deviation of the baseline mean.¹⁶

Second, we calculated an MCID value based on determining one standard error of measurement (SEM).¹⁷ The value of the SEM is that it is a fixed characteristic of any measure in a population, regardless of the subjects included in the sample. As such, a change smaller than the SEM is likely the result of measurement error rather than true change. Furthermore, using a one SEM threshold has been shown to closely correlate with external anchor methods when determining MCID values in QOL metrics.¹⁷

Third, we estimated Cohen’s effect size (d) of the smallest unit change (0.2)^18,19 An effect size estimates the magnitude of the between-group differences. In the context of QOL research, it estimates the magnitude of the effect that a given intervention had on the QOL of the subjects studied.¹⁹ Effect sizes have also been shown to correlate with external anchors when attempting to establish the MCID.¹⁹ Cohen standardized effect sizes into small (0.2), moderate (0.5), and large (0.8).¹⁹ Given that we are interested in the minimal amount of noticeable change when calculating and MCID, then the small effect size threshold is used.

Finally, we evaluated the minimum detectable change (MDC) value.⁶ The MDC is the smallest change that can be considered above the measurement error within a given confidence interval.⁶ Given that the purpose of MCID calculations is to identify meaningful changes in any given measure, theoretically any MCID value should be greater than the MDC. All four of these MCID calculations were then averaged to provide an MCID threshold for post-treatment improvement on the QOD-NS instrument.

RESULTS

Final Cohort Characteristics and Intervention Procedures

A total of 180 study participants with medically refractory CRS completed all preoperative materials between June, 2013 and June, 2015. There were no significant differences between those patients with and without follow-up across any of the measures in Table 1. For this study, the mean preoperative QOD-NS score for those with follow-up 14.1 [±13.0] while the mean preoperative QOD-NS score for those without follow-up is 16.4 [±13.5], a non-significant difference (p=0.253). Follow-up QOD-NS questionnaires were collected for 128 (71%) study participants. Study participants were followed for an average 14.5 [±5.0] months. Cohort characteristics and preoperative measures of the final cohort with postoperative follow-up (n=128) disease severity are described in Table 1.

Table 1.

Description of preoperative cohort characteristics, comorbidity, and measures of disease severity in study participants with CRS (n=128)

Preoperative measures / comorbidity:	Mean [±SD]	Range: [LL - UL]	N (%)
Age (years at enrollment)	50.8 [±16.4]	[18 – 81]
Male			55 (43%)
Female			73 (57%)
White / Caucasian			112 (88%)
African American			6 (5%)
Asian			7 (6%)
Hispanic / Latino			7 (6%)
Current tobacco use			4 (3%)
Asthma			57 (45%)
Allergy (mRAST/skin prick confirmed)			83 (65%)
Nasal polyposis			46 (36%)
Previous endoscopic sinus surgery			83 (65%)
OSA			9 (7%)
AERD			11 (9%)
Ciliary dyskinesia / CF			6 (5%)
Septal deviation			35 (27%)
Turbinate hypertrophy			16 (13%)
Depression			16 (13%)
Corticosteroid dependency			15 (12%)
Diabetes mellitus (Type I / II)			16 (12%)
GERD			40 (31%)
CT total score	11.8 [±6.3]	[0 – 24]
Endoscopy total score	5.7 [±3.7]	[0 – 18]

SD, standard deviation; LL, lower limit; UL, upper limit; OSA, obstructive sleep apnea; AERD, aspiring exacerbated respiratory disease; CF, cystic fibrosis; mRAST, modified radioallergosorbent testing; GERD, gastroesophageal reflux disease; CT, computed tomography; QOD-NS, Questionnaire for Olfactory Disorders, Negative Statements; CRS, chronic rhinosinusitis.

Average Postoperative Improvements in QOD-NS

Mean postoperative improvements in matched pairs of QOD-NS total scores are described in Table 2. Statistically significant improvements were reported, on average, following endoscopic sinus surgery for the total cohort, however not for those study participants reporting normal, preoperative QOD-NS total scores.

Table 2.

Average improvement in QOD-NS for study participants with postoperative follow-up

	N	Preoperative Mean [±SD]	Postoperative Mean [±SD]	Change Mean [±SD]	Paired t-test statistic	p-value
QOD-NS total score	128	14.1 [±13.0]	9.7 [±11.5]	−4.3 [±11.0]	4.43	<0.001
Abnormal (> 12.5) preoperative QOD-NS total scores	59	25.7 [±9.9]	16.1 [±13.4]	−9.6 [±12.9]	5.70	<0.001
Normal (< 12.5) preoperative QOD-NS total scores	69	4.1 [±3.6]	4.3 [±5.9]	0.2 [±6.4]	−0.23	0.821

PROMs, patient reported outcome measures; SD, standard deviation; QOD-NS, Questionnaire for Olfactory Disorders, Negative Statements;

Distribution-based Determinations of MCID for the QOD-NS

Estimations of distribution-based methods for determining the MCID values from QOD-NS scores (n=128) were calculated and compared (Table 3). Strong levels of Internal consistency between preoperative QOD-NS scale items were found using Cronbach’s alpha (α) value (α=0.943) and utilized as the reliability estimate in the calculation of the SEM value. For the entire cohort, the MCID values ranged from 2.6 to 8.6 depending on the method used, with a percent of patients achieving MCID after surgery ranging from 26 to 43% depending on the method. For the patients who started with an abnormal QOD-NS values, 49 to 68% of them achieved MCID. The average MCID value was determined to be 5.2 points or ~10% of the score range of the QOD-NS survey instrument. Based on the average MCID value calculated via these distribution-based methods, 38% of all patients achieved MCID after ESS, compared to 63% of those patients who began with abnormal QOD-NS values.

Table 3.

Distribution-based methods for determining MCID values for QOD-NS scores

	MCID Value Determinations
Measure	Cohens effect size (d) value*	1.0 Standard error of baseline measurement (SEM) value	0.5 baseline standard deviation value	Minimum detectable change (MDC) value	Mean
QOD-NS	2.6	3.1	6.5	8.6	5.2
Likelihood of achieving MCID for entire cohort	51%	43%	31%	26%	38%
Likelihood of achieving MCID in patients with abnormal QOD-NS (> 12.5) at baseline	70%	68%	58%	49%	63%

MCID, minimal clinically important difference; QOD-NS, Questionnaire for Olfactory Disorders, Negative Statements;

DISCUSSION

The distribution-based methods used in this study are widely-accepted techniques that harness statistical methodology to measure the responsiveness of an instrument and to determine changes in instrument scores that are outside of measurement error.¹ Since there are multiple distribution-based methods available, and different statistical calculations are likely to yield differing values, we aimed to explore a range of distribution-based MCID values and obtain an average of those determined values. This approach would make threshold values more conservative and decrease the threshold of overly restrictive MCID determinations for those methods with higher MCID values. Clearly, there is some variation in distribution-based MCID values depending on the method chosen, and this analytical approach does raise some problematic issues related to external validity to other treatment groups or in other patient populations. Nonetheless, this analysis provides us a range of values from which we can begin to understand what magnitude of change in the QOD-NS might be related to a clinically significant improvement.

In addition to distribution-based methods, MCID values can be determined using external methods. External methods require the use of a true “gold-standard”, anchor-based measure to guide various analytical approaches including: sensitivity/specificity, social comparisons, and either within-subject or between-subject score changes. Anchor-based methodology was considered for this investigation, but ultimately it was not deemed feasible. Several barriers exist to using external anchor-based measures to determine the MCID of the QOD-NS. First, it is not clear what the external gold-standard would be. One could consider objective olfactory data like psychophysical testing. However, objective smell loss is likely not a good anchor for olfactory-specific QOL, since we have previously shown that in CRS there are several factors that affect QOD-NS scores in subjects, of which objective olfactory testing is but one of them.² Furthermore, the collection of psychophysical olfactory testing is time consuming and labor intensive. Nonetheless, this alternative approach could be considered in future investigations and additional cohorts may help refine MCID values over time.

The distribution-based calculations yielded a range of QOD-NS MCID values from 3.1 to 8.6, and an average score of 5.2. Since QOD-NS item scores are discrete whole number values, the MCID would be rounded upward for a more conservative estimate. It is important to note that the different distribution-based methods are simply statistical manipulations, and one technique is not inherently better than the other. Interestingly, the average MCID value of these different methods is approximately 10% of the maximum score range. This 10% value is a phenomenon that is seen in other MCID calculations in the literature. One example is the VAS pain scale, where a 10% change is considered the MCID.²⁰

In this study cohort, 26 to 51% of all patients achieved MCID after surgery. This relatively low level of improvement is consistent with our previously report changes of the QOD-NS after ESS which differed by a value of 4 points.³ However, the degree of improvement and likelihood of achieving MCID is dependent on the olfactory status of the population being treated. As detailed in Table 3, when stratifying by normal vs. abnormal QOD-NS scores at baseline, the likelihood of reaching MCID is much higher in the patients with abnormal QOD-NS since patients with normal scores would have less room for improvement after surgery. Nonetheless, a significant proportion of patients may not achieve MCID in the QOD-NS, and this requires further study to better understand why certain patients do not improve, and what might be influencing their improvement, or lack thereof. The establishment of an MCID value will now allow future investigations to better judge the magnitude of change observed in the QOD-NS, and will facilitate our understanding of what factors may impact meaningful improvement in QOD-NS scores.

While this study was performed prospectively and recorded demographics, comorbidities, and CRS-specific disease severity measures it does have limitations. This is a cohort recruited from tertiary rhinology practices with a high burden of disease and large incidence of prior sinus surgery, and so the results may not be externally generalizable to other patient populations undergoing alternative treatment regimens. After all, the purpose of a MCID is to determine clinical importance and distribution-based methods from a single, relatively small cohort are inherently unable to fully address this concept without more concise anchor-based metrics. However, without a suitable external measure, investigators must largely rely on this type of analytical approach for accurately defining MCID values.

Furthermore, the interpretation of the clinical importance of a distribution-based MCID should be taken cautiously. First, this cohort is composed of a heterogeneous group of CRS patients. Ideally, MCID calculations are performed in a homogenous sample, and this is an inherent problem of all CRS research, where clinical subgroups are not well defined and sample sizes are relatively small. Additionally, different subgroups are likely to skew the nature of the data calculations. For example, the inclusion of the patients with normal baseline QOD-NS scores likely skews the MCID results in a conservative fashion. While these limitations are important, our goal was to further the understanding of how changes in the QOD-NS can be interpreted in the overall CRS cohort. We hope that future investigators can use this information to guide MCID studies within different CRS subgroups.

CONCLUSION

The distribution-based MCID value determinations of the QOD-NS survey instrument range between 2.6 to 8.6 points, with an average value of 5.2. Once stratified by normal vs. abnormal QOD-NS scores at baseline, the majority of patients with abnormal pre-operative QOD-NS achieve an MCID in olfactory-specific QOL. A better understanding of why some patients perceive improved olfactory QOL and prognostic factors is needed.