Development and Psychometric Validation of the Olfactory Dysfunction Outcomes Rating

This study aims to develop and validate a concise and visually appealing smell loss–associated quality-of-life patient-reported outcome measure for olfactory dysfunction.

Key Points

Question

Can a quality-of-life instrument be developed and validated for assessing the physical problems, functional limitations, and emotional consequences associated with olfactory dysfunction (OD)?

Findings

In this survey creation and validation study among 283 patients, the newly created 28-item Olfactory Dysfunction Outcomes Rating (ODOR) was internally consistent, reliable, and valid. The minimal clinically important difference was 15 out of 112 total points.

Meaning

The ODOR is a novel, concise, and valid patient-reported outcome measure of OD-associated quality of life that captures change after a given intervention; thus, it can be used as an outcome measure in future OD trials.

Abstract

Importance

Olfactory dysfunction (OD) is an increasingly common and morbid condition, especially given the ongoing COVID-19 pandemic. Thus, the ability to reproducibly measure smell loss–associated quality of life (QOL) and its response to treatment is paramount.

Objective

To develop and validate a concise and visually appealing smell loss–associated QOL patient-reported outcome measure for OD.

Design, Setting, and Participants

A secondary analysis of comments to an online survey by 1000 patients with olfactory dysfunction published in 2013 was used as the primary source to generate items of the Olfactory Dysfunction Outcomes Rating (ODOR). In addition, 30 patients with OD enrolled in 2 clinical studies at a tertiary care medical center (Washington University) were asked to identify their main concerns associated with smell loss. And finally, 4 otolaryngologists reviewed the items generated from the online survey and the patients’ interviews to identify any additional items. Prospective study design was used for data collection from the 30 patients and 4 otolaryngologists. Prospective study design was used for survey validation. Validation of the ODOR was performed with 283 patients enrolled in several prospective studies at a single institution that completed the ODOR as an outcome measure.

Main Outcomes and Measures

Item generation and selection were the outcomes of ODOR development. The psychometric and clinimetric measures evaluated for validation were internal consistency, test-retest reliability, face and content validity, concurrent validity, and discriminant validity. Minimal clinically important difference was also determined.

Results

The ODOR is a 28-item instrument with each item scored as either no difficulty or very rarely bothered (0) to complete difficulty or very frequently bothered (4) with a total instrument score range of 0 to 112 points. Higher scores indicate higher degree of dysfunction and limitation. Validation in the cohort of 283 patients (mean [SD] age, 47.0 [14.4] years; 198 female participants [73%]; 179 White participants [80%]) revealed that the instrument has high internal consistency (Cronbach α = 0.968), test-retest reliability (r = 0.90 [95% CI, 0.81-0.95]), face validity, content validity, concurrent validity (r = 0.87 [95% CI, 0.80-0.91] compared with the Questionnaire of Olfactory Disorders–Negative Statements; ρ = −0.76 [95% CI, −0.81 to −0.71] compared with a patient-reported symptom severity scale), and divergent validity (mean score difference, −33.9 [95% CI, −38.3 to −29.6] between normosmic patients and hyposmic/anosmic patients). The minimal clinically important difference was 15 points. The estimated time for survey completion was approximately 5 minutes.

Conclusions and Relevance

In this survey creation and validation study, the ODOR was shown to be a novel, concise, reliable, and valid patient-reported outcome measure of OD-associated QOL. It can be used to measure physical problems, functional limitations, and emotional consequences associated with OD and how they change after a given intervention, which is clinically applicable and particularly pertinent given the growing burden of OD associated with COVID-19.

Introduction

Up to 25% of adults in the US report some degree of impaired olfactory function per large population-based studies.^1,2 These numbers are likely now underestimates due to the prevalence of patients with previous COVID-19.^3,4,5 Due to the sense of smell’s integral role in vital functions, such as avoidance of hazards, eating, drinking, maintenance of hygiene, and mating, olfactory dysfunction (OD) has a profound effect on quality of life.⁶ Individuals with OD describe their condition as “living a life without color” or “living in a plastic bag.”^7,8 In addition to the subjective complaints, there are measurable changes to neural connectivity of the olfactory network in patients with OD compared with healthy controls.⁹ Moreover, older adults with OD have a greater than 3-fold risk of 5-year mortality compared with normosmic adults of similar age.¹⁰

The objective loss of olfactory function can be measured with a variety of psychophysical assessments and tests. Subjective assessment can be obtained through the use of patient-reported outcome measures (PROMs), which are crucial for capturing the physical problems, functional limitations, and emotional consequences of OD. Subjective assessments are especially important for conditions where symptoms and functional impairment define the severity of the illness, such as in OD.¹¹ PROMs, which are usually self-completed questionnaires, measure functional status, health-related quality of life (HRQOL), symptom burden, and health-related behaviors.¹² Transitional PROMs can also be used to assess for patient response to treatment or to monitor change over time,¹¹ which are mandated by both the US Food and Drug Administration and the European Medicines Agency to support labeling claims.¹³ However, as with many conditions, there is often a large difference noted in the degree of OD as defined by objective testing and subjective recording, and this discrepancy creates challenges when deciding what outcomes to include in clinical research. Accordingly, a well-designed disease-specific PROM of HRQOL should capture and quantify physical problems, functional limitations, and emotional consequences associated with the disease process.^14,15

There are several PROMs of OD, and only a few have undergone rigorous psychometric validation.¹⁶ The Questionnaire for Olfactory Disorders (QOD) is the most commonly used HRQOL PROM of OD, which consists of 52 statements divided into 3 categories: 39 negative statements, 5 positive statements, and 8 “socially desired” statements.^16,17 The socially desired statements are designed to evaluate the truthfulness of participant responses. The instrument’s long length, inconsistency in scoring, and unknown clinical utility of the positive and socially desired statements prompted investigators to shorten the instrument by including only the 17 negative statements, aptly named the Questionnaire for Olfactory Disorders–Negative Statements (QOD-NS).¹⁸ Each item is scored from 0 to 3, with higher scores indicating worse olfactory function, and the minimal clinically important difference (MCID) for the QOD-NS is 5.2 points.¹⁹ The QOD-NS was further shortened to the 7-statement brief QOD-NS (B-QOD) after psychometric item reduction.^20,21

While the QOD has been used as an outcome measure in various olfaction studies, it has its limitations. Originally translated from German, its syntax and readability are at times confusing to English-speaking participants.⁶ It also used the distribution-based approach in determining the MCID rather than the more clinically meaningful within-patient health transition global ratings method, a type of anchor-based approach.^19,22 While the principles of rigorous psychometric validation suggest including approximately 10 participants per item,²³ the investigators validating the QOD and QOS-NS used sample sizes of 205 and 68 patients with OD, respectively, which are relatively small given the 52-item and 17-item instruments.^17,18 For these reasons, this study’s objective was to develop and validate the Olfactory Dysfunction Outcomes Rating (ODOR), a novel disease-specific HRQOL PROM of olfactory function that is concise, visually appealing, and patient centered in accurately assessing the physical problems, functional limitations, and emotional consequences associated with OD.

Methods

Development Methods

The items of the ODOR were derived using clinimetric strategies described by Feinstein.^24,25 Secondary analysis of the open-ended narratives of the experience of 1000 patients with OD collected from 2009 to 2012 by Keller and Malaspina⁸ were analyzed using principles of Grounded Theory to identify key themes. This process was completed after the review of the first 500 excerpts once thematic saturation was reached. In addition, in-person discussions with 30 patients with OD enrolled into 2 clinical studies conducted at Washington University were used to explore additional patient concerns associated with smell loss. Finally, in the discussion with 4 otolaryngologists who treat patients with OD, a dissected intuition approach was used to finalize items identified from interviews.

Validation Methods

Population Under Study

Participants who were actively enrolled in several institutional review board–approved clinical trials and prospective cohort studies at Washington University and who completed the ODOR questionnaire between July 1, 2019, and June 30, 2021, constituted the study sample.^{9,26,27,28,29,30} In total, the ODOR questionnaire was included in 6 separate studies (2 pending publication), and this current ODOR validation study pooled data from the studies for secondary analysis. Research investigators of each study identified participant race and ethnicity by participant self-report and/or reviewing the Demographics section of the electronic health record. Completion of the questionnaire constituted patient informed consent.

Patient Surveys: Classification of Disease Severity

Each study captured each participant’s baseline Clinical Global Impressions–Severity (CGI-S), QOD-NS, and University of Pennsylvania Smell Identification Test (UPSIT).^{26,27,28,29,30} The CGI-S is a validated static PROM³¹ and asks, “Overall, please rate your sense of smell now. Response options: (A) excellent, (B) very good, (C) good, (D) fair, (E) poor, or (F) absent.” The UPSIT includes 4 odor-impregnated booklets that participants “scratch and sniff” to identify 40 various odors. Out of 40 total points, normosmia is defined as 34 or higher for male individuals and 35 or higher for female individuals, and the cutoff for anosmia is 18.^32,33 While MCID has not been psychometrically calculated, a change of 4 or more points has been inferred to be a clinically meaningful change based on face validity in prior studies.^34,35,36

From our longitudinal studies, participants who underwent follow-up testing were administered the Clinical Global Impressions–Change (CGI-C), QOD-NS, and UPSIT. The CGI-C is a validated transitional PROM,^{27,29,30,31,37} and it asks, “Overall, how would you rate your change in smell after (certain intervention)?” These participants were enrolled in other clinical trials or cohort studies, and interventions included intranasal theophylline,^27,29,30 olfactory training, mometasone spray or irrigations,²⁸ and endoscopic transsphenoidal surgery. Response options were as follows: much better, somewhat better, slightly better, neither better nor worse, slightly worse, somewhat worse, or much worse. A clinically meaningful change was considered to be a response of “somewhat better” or “much better.” Follow-up testing was administered after 4 to 12 weeks of an intervention.

Reliability

Reliability was measured by both internal consistency and test-retest reliability. Internal consistency is a measure of how well items within an instrument relate to each other as measured by Cronbach α.³⁸ The minimum acceptable score is 0.7.³⁹

Test-retest reliability is a measure of the instrument’s stability over time with repeated testing by the same user with the same symptomatic severity. The test-retest reliability was assessed by administering ODOR at baseline and again at 4 weeks in participants who reported no subjective change in their olfaction during this interval. Pearson correlation was used to measure level of agreement between the test and retest scores.

Validity

Validity refers to the degree of which the index instrument or test measures what it is designed to measure. Measures of psychometric validity include face validity, content validity, concurrent validity, and discriminant validity. Face validity refers to the agreement between experts and users that the content of the instrument is measuring its intended aim. Content validity assesses whether the instrument measures a representative and encompassing sample of the construct it is supposed to be measuring. Content validity was assessed by evaluating whether the ODOR items represented relevant aspects of the physical, functional, and emotional aspects of OD in adults and whether they were relevant for the assessment of treatment effects.

Construct validity, separated into concurrent and discriminant validity, assesses the suitability of an instrument to measure what it is supposed to measure. Concurrent validity refers to how closely the scores of the index instrument are correlated with the scores of an instrument that measures the same construct. To assess concurrent validity, ODOR scores were compared with CGI-S, QOD-NS, and UPSIT scores at baseline. A 1-way analysis of variance was used to explore the association of the ODOR and CGI-S scores. Pearson correlation was used to explore the association of ODOR scores with QOD-NS and UPSIT scores.

Discriminant validity was assessed by comparing ODOR scores between patients with OD and normosmic patients. The diagnosis of OD, which includes anosmia and hyposmia, was measured by both self-report and psychophysical testing such as the UPSIT. Normosmia was also defined according to both self-report and the UPSIT. Participants who scored as normosmic on the UPSIT and also “good,” “very good,” or “excellent” on the CGI-S were considered normosmic, while all others were categorized as hyposmic/anosmic, indicating presence of OD. Independent samples t test was used to evaluate discriminant validity.

Responsiveness to Change and MCID

Responsiveness to change is the instrument’s ability to capture change in a patient’s condition and symptoms. The MCID is the minimum difference in the instrument’s total score that is considered clinically meaningful by treating physicians and afflicted patients for discriminating between different levels of OD severity. The MCID was calculated using the anchor-based approach based on the CGI-C, which captures patients’ self-reported perceived change between 2 time points, which is clinically relevant and practical.²² The MCID was defined as the mean change in ODOR score between patients who self-reported feeling “somewhat better” or “much better” vs patients who self-reported feeling “neither better nor worse.” We also used the distribution-based method, which defines the MCID as 0.5 SDs of the ODOR score at baseline.

Results

Development Results

Instrument Domains and Items

The themes identified from patient online excerpts and from discussions with otolaryngologists were divided into 5 domains: diet/appetite, environmental safety, interpersonal relationships, hygiene, and mood. A 28-item ODOR instrument was then created with 6 items on diet/appetite, 4 items on environmental safety, 5 items on interpersonal relationships, 5 items on hygiene, and 5 items on mood (see eAppendix in the Supplement). There were also 3 miscellaneous items related to perceived insensitivity, parosmia, and phantosmia.

Instrument Scoring

Each item is scored on a 5-point Likert scale graded from 0 to 4. The prompt for the first 19 items, which covers all 5 domains, asks, “Due to your sense of smell, how difficult is it to perform the following tasks or activities?” Response options (and scores) were as follows: no difficulty (0), mild difficulty (1), moderate difficulty (2), severe difficulty (3), or complete difficulty (4). The prompt for the following 9 items, which covers all 5 domains and also the 3 miscellaneous items, asks, “Due to your sense of smell, how often are you bothered by the following?” Response options (and scores) were as follows: very rarely bothered (0), rarely bothered (1), occasionally bothered (2), frequently bothered (3), or very frequently bothered (4). The total score is the sum of all items with a maximum score of 112, and higher values indicate higher degree of dysfunction and limitation in daily life.

Validation Results

Reliability and Validity Measures

The study population comprised 283 participants from 6 studies; the majority were female (n = 198 [73%]) and White (n = 179 [80%]), and the mean (SD) age was 47.0 (14.4) years (Table 1). The main causes of OD reported from participants were COVID-19 (n = 113 [40%]), iatrogenic cause (n = 54 [19%]), viral infection other than COVID-19 (n = 31 [11%]), chronic rhinosinusitis (n = 26 [9%]), other causes (n = 17 [3%]), and unknown (n = 23 [8%]). Participants had a median (range) baseline UPSIT score of 32 (5-40), median (range) ODOR baseline total score of 32 (0-108), and the following distribution among all CGI-S categories: absent (n = 41 [14%]), poor (n = 82 [29%]), fair (n = 65 [23%]), good (n = 30 [11%]), very good (n = 44 [16%]), and excellent (n = 21 [7%]). The estimated time for survey completion was approximately 5 minutes, and all participants were able to complete the survey.

Table 1. Baseline Characteristics of the Study Population Stratified by Olfactory Function.

Characteristic	No. (%)
	Normosmic (n = 54)	Hyposmic/anosmic (n = 229)
Age, mean (SD), ya	51.4 (13.4)	45.8 (14.4)
Sexb
Female	37 (69)	162 (74)
Male	17 (31)	57 (26)
Race and ethnicityc
Asian	0	4 (2)
Black	4 (13)	21 (11)
Hispanic	1 (3)	8 (4)
White	27 (84)	152 (79)
Multiple races	0	5 (3)
Otherd	0	2 (1)
UPSIT baseline
Median (range)e	37 (34-40)	30 (5-39)
Normosmia	54 (100)	42 (19)
Mild hyposmia	0	65 (29)
Moderate hyposmia	0	34 (15)
Severe hyposmia	0	37 (16)
Anosmia	0	47 (21)
Probable malingering	0	1 (<0.1)
CGI-S baseline
Absent	0	41 (18)
Poor	0	82 (36)
Fair	0	65 (28)
Good	14 (26)	16 (7)
Very good	31 (57)	13 (6)
Excellent	9 (17)	12 (5)

Abbreviations: CGI-S, Clinical Global Impressions–Severity; UPSIT, University of Pennsylvania Smell Identification Test.

^a Total participants with age data = 273; 10 (4%) missing from table.

^b Total participants with sex data = 273; 10 (4%) missing from table.

^c Total participants with race and ethnicity data = 224; 59 (21%) missing from table.

^d “Other” was based on self-report; therefore, further details cannot be elucidated.

^e Total participants with UPSIT baseline data = 280; 3 (1%) missing from table.

Participants who scored as normosmic on UPSIT and also “good,” “very good,” or “excellent” on the CGI-S were considered normosmic (n = 54), while all others were categorized as hyposmic/anosmic (n = 229), indicating presence of OD (Figure 1). The hyposmic/anosmic patients had higher total ODOR scores (median [range], 37 [0-108]) than normosmic patients (median [range], 3.5 [0-39]) with a median ODOR score difference of 31 (95% CI, 27-38); hyposmic/anosmic patients also had higher domain-specific ODOR scores (Table 2).

Figure 1. Flow Diagram of Study Participation.

Study A: Randomized clinical trial to evaluate mometasone lavage vs spray for patients with chronic rhinosinusitis without nasal polyps who have not undergone surgery. Study B: Smell Changes and Efficacy of Nasal Theophylline (SCENT) irrigation: a randomized clinical trial for treatment of postviral olfactory dysfunction. Study C: Anosmia and olfaction with Sellar minimally invasive approaches. Study D: Resting-state functional magnetic resonance imaging results of healthy normosmic controls. Study E: Development and validation of a novel at-home smell assessment. Study F: Natural trajectory of recovery of COVID-19–associated olfactory loss. CGI-S indicates Clinical Global Impressions–Severity; ODOR, Olfactory Dysfunction Outcomes Rating; UPSIT, University of Pennsylvania Smell Identification Test.

Table 2. Comparison of ODOR Total Score and Domain-Specific Scores Stratified by Olfactory Function.

	Median (range)
	Normosmic (n = 54)	Hyposmic/anosmic (n = 229)
ODOR total score	3.5 (0-39)	37.0 (0-108)
ODOR domain score
Diet	1.0 (0-11)	7.0 (0-24)
Environmental safety	0.0 (0-9)	8.0 (0-16)
Interpersonal relationships	0.0 (0-9)	1.0 (0-19)
Hygiene	0.0 (0-12)	11.0 (0-20)
Mood	0.0 (0-12)	7.0 (0-20)

Abbreviation: ODOR, Olfactory Dysfunction Outcomes Rating.

The ODOR had high internal consistency as measured by a Cronbach α value of 0.968. Thirty-two participants of 2 studies that displayed no change in UPSIT scores from baseline to postintervention were used for evaluating test-retest reliability. The mean UPSIT score difference preintervention to postintervention was −0.56 (95% CI, −1.99 to 0.86). There was a strong correlation (r = 0.90; 95% CI, 0.81-0.95) between the preintervention and postintervention ODOR scores of these 32 participants.

Face and content validity of ODOR was assessed by a team of 4 otolaryngologists and more than 30 patients with either postviral or posttraumatic OD who were participating in other trials at our institution. The clinicians and patients determined that all items were clinically relevant to OD-associated quality of life and relevant to the target population. No concerns about need for additional items were raised during open-ended questioning of patients with OD.

Concurrent validity was evaluated by exploring the correlation of ODOR total score with QOD-NS, UPSIT, and CGI-S scores. As expected, there was a strong positive correlation with QOD-NS (r = 0.87; 95% CI, 0.80-0.91), a moderate negative correlation with UPSIT score (r = −0.45; 95% CI, −0.54 to −0.35) (Figure 2), and a strong negative correlation with the CGI-S (ρ = −0.76; 95% CI, −0.81 to −0.71) (Figure 3). The ODOR displayed good discriminant validity as measured by its ability to differentiate between patients with normal smell and those with hyposmia or anosmia (mean ODOR score difference, −33.9 [95% CI, −38.3 to −29.6]).

Figure 2. Olfactory Dysfunction Outcomes Rating (ODOR) Scores as a Function of University of Pennsylvania Smell Identification Test (UPSIT)–Defined Categories.

The top and bottom of the boxes represent the upper and lower quartiles, respectively; the horizontal lines within the boxes indicate the medians; the whiskers show the range; and the plotted points represent outliers.

Figure 3. Olfactory Dysfunction Outcomes Rating (ODOR) Scores as a Function of Clinical Global Impressions–Severity–Defined Categories.

The top and bottom of the boxes represent the upper and lower quartiles, respectively; the horizontal lines within the boxes indicate the medians; the whiskers show the range; and the plotted points represent outliers, with the shaded plot points indicating extreme outliers.

Responsiveness to Change and MCID

There were 93 participants from 3 different studies that assessed the ODOR instrument at 2 different assessments. During the last assessment, these participants also completed the CGI-C, which measures their perceived impression of change. A total of 36 (39%) participants reported no change in their sense of smell, and the mean (SD) reduction in ODOR scores between the 2 assessments was 2.4 (8.4) points. There were 20 participants (22%) who reported “somewhat better” or “much better” sense of smell, and these participants had a mean (SD) reduction in ODOR scores of 17.7 (15.6) points. The MCID, defined as the mean difference between the group of participants self-reporting “somewhat better” or “much better” sense of smell and the group self-reporting no change, was 15 points. Using the distribution-based method, the calculated MCID of 15 is equivalent to 0.56 SDs of the ODOR scores (mean [SD], 34.9 [26.9]), corresponding to a moderate effect size. The mean (SD) ODOR scores of participants in the “poor” and “fair” categories of CGI-S were 49.4 (20.6) and 33.6 (21.1), respectively, also corresponding to a mean difference in ODOR scores of 15 points (Figure 3).

Discussion

After reviewing more than 500 open-ended narratives of patients with reported smell loss, we developed the 28-item ODOR questionnaire. A total of 283 patients with and without OD who completed the ODOR were included for psychometric validation of the instrument, which revealed excellent internal consistency, test-retest reliability, face validity, content validity, concurrent validity, and discriminant validity. The MCID was calculated as 15 out of a total of 112 points. The average time needed to complete the questionnaire was approximately 5 minutes.

Compared with other HRQOL PROMs of olfactory function that have been described in the literature,^{16,17,18,20,40,41,42,43,44} the ODOR has undergone rigorous psychometric and clinimetric validation (see eTable in the Supplement). It is the only instrument to assess for responsiveness to change and to calculate the MCID through both the distribution-based and anchor-based methods, thereby increasing its potential clinical utility. With the acute surge of COVID-19–associated OD and ongoing studies of novel therapeutics for OD in response to this burgeoning public health crisis, the ability to reproducibly measure smell loss–associated quality of life and its response to treatment is paramount. Current and future efficacy studies of novel therapeutics should include not only psychophysical measures of olfaction, such as UPSIT and Sniffin’ Sticks scores, but also HRQOL PROMs.

Strengths and Limitations

There are several strengths and limitations to this study. One strength is the adequately powered psychometric validation given that there are greater than 10 participants per item (n > 280).²³ Another strength is the inclusion of various causes of OD, including postviral, postsurgical/traumatic, and inflammatory, thereby increasing generalizability. There are also several limitations. Inherent to any survey study, participants may be subject to response bias. Because the validation cohort consisted of enrollees in prospective studies at our institution, volunteer bias may be present. While the ODOR total scores appear psychometrically valid, the clinical utility of ODOR domain-specific scores is not yet known. A future validation study is needed to confirm the validation results and to explore the distribution of ODOR scores across different causes of OD.

Conclusions

In this survey creation and validation study, the ODOR was shown to be a novel, concise, and valid PROM of OD-associated quality of life. The ODOR can be used to measure the physical, functional, and emotional limitations associated with OD and how they change after a given intervention, which is clinically applicable and particularly pertinent given the growing burden of COVID-19–associated OD.

Supplement.

eAppendix. Olfactory Dysfunction Outcomes Rating (ODOR) questionnaire.

eTable. Comparison among patient-reported outcome measures for olfactory dysfunction.

eReferences