A systematic review to examine the relationship between objective and patient‐reported outcome measures in sinonasal disorders: recommendations for use in research and clinical practice

Abstract

Background

Common sinonasal disorders include chronic rhinosinusitis (CRS), allergic rhinitis (AR), and a deviated nasal septum (DNS), which often coexist with shared common symptoms including nasal obstruction, olfactory dysfunction, and rhinorrhea. Various objective outcome measures and patient‐reported outcome measures (PROMs) are used to assess disease severity; however, there is limited evidence in the literature on the correlation between them. This systematic review aims to examine the relationship between them and provide recommendations.

Methods

A search of MEDLINE and EMBASE identified studies quantifying correlations between objective outcome measures and PROMs for the sinonasal conditions using a narrative synthesis.

Results

In total, 59 studies met inclusion criteria. For nasal obstruction, rhinomanometry shows a lack of correlation whereas peak nasal inspiratory flow (PNIF) shows the strongest correlation with PROMs (r > 0.5). The Sniffin’ Stick test shows a stronger correlation with PROMs (r > 0.5) than the University of Pennsylvania Smell Identification Test (UPSIT) (r < 0.5). Computed tomography (CT) sinus scores show little evidence of correlation with PROMs and nasal endoscopic ratings (weak correlation, r < 0.5).

Conclusion

Overall, objective outcome measures and PROMs assessing sinonasal symptoms are poorly correlated, and we recommend that objective outcome measures be used with validated PROMs depending on the setting. PNIF should be used in routine clinical practice for nasal obstruction; rhinomanometry and acoustic rhinometry may be useful in research. The Sniffin’ Sticks test is recommended for olfactory dysfunction with UPSIT as an alternative. CT scores should be excluded as a routine CRS outcome measure, and endoscopic scores should be used in combination with PROMs until further research is conducted.

Sinonasal disorders are common conditions with the potential to cause a significant impact to the quality of life (QoL) of sufferers. ¹ , ² The most prevalent of these include chronic rhinosinusitis (CRS), allergic rhinitis (AR), and a deviated nasal septum (DNS). These 3 conditions share some common symptoms (nasal obstruction, discharge, facial pain/pressure, hyposmia/anosmia) that may be assessed by various patient‐reported outcome measures (PROMs) and objective outcome measures.

PROMs are disease‐specific symptom scores used to measure QoL complaints and, because the goal of treatment typically aims to improve patient QoL, they are used to assess disease severity and guide management plans. Common PROMs include the following: Rhinosinusitis Outcome Measurement (RSOM); Chronic Sinusitis Survey (CSS); Questionnaire of Olfactory Disorders (QOD); Sinusitis Control Test; Nasal Obstruction and Septoplasty Effectiveness Scale (NOSE); visual analogue scale (VAS) of specific symptoms, such as smell and nasal obstruction, or for global symptoms; and the 22‐item Sino‐Nasal Outcome Test (SNOT‐22).

Objective outcome measures are tests performed to assess physiological parameters and response to treatment, independently of patient response. The commonly used tests assess nasal endoscopy, computed tomography (CT) scans, and physiological measurements. For example, nasal endoscopy interpretation can be standardized to allow for comparison between studies by scores including the Lund‐Kennedy Score (LKS),³ Lildholdt,⁴ Kupferberg, ⁵ and Philpott‐Javer scores.⁶ CT scans are often staged using the Lund‐Mackay score (LMS).⁷ Nasal physiological measurements include peak nasal inspiratory flow (PNIF), acoustic rhinometry (AcR), rhinomanometry, rhinostereometry, and the measurement of mucociliary clearance (MCC) using rhinoscintigraphy and the saccharin test. Olfaction is most assessed using the Sniffin’ Sticks test and the University of Pennsylvania Smell Identification Test (UPSIT), although other test kits are available.

The European position paper on rhinosinusitis and nasal polyps 2012 and 2020 (EPOS 2012 and EPOPS 2020) both emphasize the lack of literature investigating the correlation between patient‐reported and objective outcome measures for CRS. ⁸ , ⁹ Elsewhere, it is noted that the relationship between subjective and objective assessments of nasal patency has also remained highly controversial. ¹⁰ , ¹¹ , ¹² Another review highlights that, despite the existence of numerous measurement tools for assessing nasal patency in DNS, no “gold standard” objective tools are in routine widespread use.¹³ Overall, the lack of depth in the evidence for the association between objective outcome measures and PROMs in the sinonasal disorders (CRS, AR, DNS) highlights the need to better evaluate the relationship between them.

Aims And Objectives

This systematic review aims to examine the correlation between objective outcome measures and disease severity (as characterized by PROMs) in sinonasal disorders and make recommendations for both research and clinical settings. Identifying suitable tools for rhinologists will help to select the appropriate investigations to improve clinical assessment and research efficiency.

Methodology

A literature search was undertaken using the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) checklist ¹⁴ with search terms including and relating to “sinonasal disorders,” “objective outcome measures,” “patient rated outcome measures,” and “correlation.” The specific search terms for the commonly used outcome measures were obtained from an Otolaryngology book chapter.⁷ All search terms used can be found in Appendix A. Synonyms and related terms within these fields were connected with the Boolean operator “OR.” These 3 search domains were then combined with the Boolean operator “AND.” The titles and abstracts of papers were screened. Both Medline (OVID) and EMBASE were searched for medical literature. An example of the search strategy for Medline (OVID) can be found in Appendix A.

Eligibility criteria

Inclusion criteria

Studies quantifying the correlation between objective outcome measures and PROMs in the sinonasal disorders.

Exclusion criteria

• Studies with either objective outcome measures or PROMs alone, but not both.

• Studies with mixed populations of the sinonasal disorders; eg, AR and CRS, etc.

• Studies where systemic disorders were the underlying cause of sinonasal disease; eg, cystic fibrosis, granulomatosis with polyangiitis, etc.

• Non peer‐reviewed publications.

• Non‐English publications.

Study selection and data extraction

Three reviewers were involved in the review. Two independent reviewers conducted the selection of studies and extracting data. All disagreements between the reviewers were discussed with the senior reviewer to reach a consensus. All data were managed using EndNote x72.1 (Clarivate, Philadelphia, PA). A Microsoft Excel spreadsheet (Microsoft Corp., Redmond, WA) was used to extract relevant data from the included studies. Fields included authors, year of publication, aim, number of participants, sinonasal disorder, PROMs, and objective outcome measures utilized.

Critical assessment of the included studies

To assess the quality of the included studies, a bespoke tool (Table 1) was devised from The Joanna Briggs Institute Critical Appraisal tools for cross‐sectional studies ¹⁵ and the NICE Quality appraisal checklist – quantitative studies reporting correlations and associations.¹⁶ An overall percentage score was given to each study to demonstrate the overall quality of methodology.

TABLE 1.

Bespoke critical analysis checklist^*

Checklist	None (0)	Reasonable (1)	Good (2)	Not applicable
1. Were there nationally/internationally accepted criteria used to diagnose the sinonasal disorders of interest (CRS, rhinitis, septal deviation)?	□	□	□	□
2. Were there clear criteria for inclusion/exclusion of participants?	□	□	□	□
3. Was defining correlation between objective and patient rated outcome measures the main aim of the study?	□	□	□	□
4. How sound was the theoretical basis for selecting the variables of interest (objective and patient rated outcome measure)? (appropriate for use of (objective and patient rated outcome measure)	□	□	□	□
5. Were the objective and patient rated outcome measure of interest validated for their use?	□	□	□	□
6. How was selection bias minimized?	□	□	□	□
7. How were confounding factors identified and dealt with, if necessary?	□	□	□	□
8. Was the follow‐up completed, if not any reasons described?	□	□	□	□
9. Was appropriate statistical analysis used?	□	□	□	□
10. Was the study sufficiently powered to detect a correlation (if one exists)?	□	□	□	□
11. Was the precision of correlation given or calculable? Is correlation meaningful?	□	□	□	□

^*The tool includes 12 criteria assessing different aspects of study methodology. Each criteria is scored between 0 and 2 (0 = criteria is completely unsatisfied, 1 = criteria is moderately unsatisfied, 2 = criteria is well satisfied). An overall percentage score was given to each study by dividing the total number of points scored by the total of scores available. Each study was given a score of 0 to 2 for each of the criteria. 0 points were awarded if the criteria were completely unfulfilled, 1 for reasonable fulfillment, and 2 for good fulfillment. An overall percentage score was calculated by dividing the number of points achieved by the total number available (only points available for criteria relevant to the study design were included in the total number of available points). Adapted from JBI checklists for observational studies and randomized controlled trial and quality appraisal checklist—quantitative studies reporting correlations and associations.

CRS = chronic rhinosinusitis.

Results

Literature search

The databases were all searched on April 30, 2020, revealing 433 publications (after removal of duplicates). Abstracts were reviewed and any irrelevant publications were removed, leaving 70 papers to be read as full texts. A total of 59 studies were included in the narrative synthesis (see PRISMA chart, Fig. 1 for details).

FIGURE 1.

PRISMA flowchart. PRISMA = Preferred Reporting Items for Systematic Reviews and Meta‐Analyses.

Methodological critical appraisal of the included studies

Overall, the average critical appraisal score of the 59 included studies is 88% (standard deviation [SD] 6.78%), with only 1 study obtaining a perfect score.¹⁷ All included studies are observational studies with intrinsic limitations including bias and confounding that could distort correlation findings.

Almost all studies used nationally or internationally accepted criteria to diagnose the sinonasal disorders. In total, 97% of the included studies precisely outlined their inclusion and exclusion criteria, with the exception of 2 studies. ¹⁸ , ¹⁹ After assessing these 2 studies in full‐text, it was decided to include them because they met the inclusion criteria and their statistical analyses provided useful information for our review.

Defining the correlation between objective outcome measures and PROMs was the main objective of 78% of the included studies. A total of 13 studies ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³² coincidentally provided information about the correlation between PROMs and objective outcome measures as a part of their exploratory statistical analyses.

The most common deficiency of the included papers was the use of non‐validated outcome measures. One third of the included studies ¹⁹ , ²⁰ , ²⁶ , ²⁸ , ²⁹ , ³⁰ , ³² , ³³ , ³⁴ , ³⁵ , ³⁶ , ³⁷ , ³⁸ , ³⁹ , ⁴⁰ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ failed to use validated outcome measures. All included studies used appropriate statistical tests for quantifying correlations including bivariate correlation tests and regression analysis. A total of 15 studies were statistically underpowered ²³ , ²⁸ , ³¹ , ³³ , ³⁵ , ³⁸ , ³⁹ , ⁴¹ , ⁴³ , ⁴⁵ , ⁴⁶ , ⁴⁷ , ⁴⁸ , ⁴⁹ , ⁵⁰ to detect a correlation because their sample sizes were less than 50 according to our medical statistician's advice.

Results synthesis

Meta‐analysis was considered inappropriate due to the heterogeneity of the outcome measures utilized in the included studies, determined by Cochran's Q test. Therefore, the Cochrane Consumers and Communication Review Group guidance on data synthesis and analysis was consulted for the narrative synthesis process of our review.⁵¹ Table 2 illustrates a summary of correlation status and strength between different pairs of outcome measures according to 4 main sinonasal symptom groups: nasal obstruction, olfactory dysfunction, global CRS symptoms, and MCC dysfunction symptoms/rhinorrhea.

TABLE 2.

A summary of correlation status and strength between different pairs of outcome measures according to sinonasal symptoms^*

Sinonasal symptom	Objective outcome measures	PROMS	Correlations status	Strength of correlations
Nasal obstruction	Rhinomanometry	NOSE; SNOT‐22; VAS Nasal Blockage	Present in all main sinonasal disorders CRS, AR, DNS	Conflicting in all main sinonasal disorders with majority studies weak correlation r <0.5) Strongest in AR (r >0.7 in 2 studies) Moderate in DNS (r >0.5 in 2 studies) Very weak in CRS and not clinically significant
	Acoustic rhinometry	VAS Nasal Blockage; NOSE; SNOT‐22	Mostly NOT present in all main sinonasal disorders CRS, AR, DNS	Majority ‐ No correlation in the sinonasal disorders Weak to Moderate in AR (3 studies r from −0.32 to −0.62) Moderate in CRS (1 study with r = 0.5)
	PNIF	VAS Nasal Blockage; NOSE; SNOT‐22; BNS	Present in CRS, AR; Not present in DNS	Majority – weak correlations (most studies r <0.5) Moderate correlations in CRS (2 studies with some r >0.5; 1 with r = −0.48)
	Other uncommon tools (rhinospirometry, LKS, nasal endoscopy polyps size, CT the increase in angle degree, PEFI, PIFI, NPEF, saccharine)	VAS Nasal Blockage; BNS	Inconclusive due to a very small number of studies	Inconclusive due to very small number of studies
Olfactory dysfunction	The Sniffin’ Sticks test	SNOT‐22; VAS Smell; QOD; SF‐36	Present in CRS only, not other sinonasal disorders (AR, DNS)	Majority ‐ Strong correlations (most studies r >0.7) Strongest correlation in CRS in both types (r >0.7 in 1 study); in CRSwNP and severe CRSwNP (r >0.7 in 2 studies), AFRS (r >0.7 in 1 study)
	The UPSIT	SNOT‐22; VAS Smell; QOD‐NS	Present in CRS only (2/3 studies), not other sinonasal disorders (AR, DNS)	Majority ‐ Weak correlations (r <0.35 in 2 studies)
	Other rare/unusual tools LMS, LKS, CT total olfactory cleft opacification, OCES	Olfactory scores – COT; QOD‐NS; SNOT‐22; VAS Smell	Inconclusive due to very small number of studies	Inconclusive due to very small number of studies
CRS global symptoms	LMS	SNOT‐22; SNOT‐16; VAS overall symptoms; CSS	Mostly NOT present in both CRS types	Majority ‐ No correlation in both CRS types Very weak correlation (r <0.35 in 6 studies) + 1 big study showed no clinically significant correlations with r = 0.058 Week correlation (r <0.5 in 1 study with only modified LMS scores)
	LKS	SNOT‐22; SNOT‐16; VAS overall symptoms; CSS, Rhinosinusitis Disability Index	Mostly present with VAS overall symptoms (an invalidated PROM) in both CRS types and AFRS (3/5 studies) Mostly NOT present with SNOT‐22 (an validated PROM) in both CRS types	Majority: weak correlations with VAS overall symptoms (r <0.35 in 3 studies) and No correlation with SNOT‐22 Moderate correlation (r = −.51) in 1 study with Philpott‐Javer12 scoring system with VAS in AFRS
Rhinorrhea MCC dysfunction	Rhinoscintigraphy	SNOT‐22	NOT present	No correlation

*Any correlation coefficients ≤0.5 are considered weak, between >0.5 and ≤0.7 moderate, >0.7 strong in bivariate correlation analysis.

AFRS = allergic fungal rhinosinusitis; AR = allergic rhinitis; BNS = blocked nose score; COT = composite olfactory test; CRS = chronic rhinosinusitis; CRSwNP = chronic rhinosinusitis with nasal polyps; CSS = Chronic Sinusitis Survey; CT = computed tomography; DNS = deviated nasal septum; LKS = Lund‐Kennedy score; LMS = Lund‐Mackay score; MCC = mucociliary clearance; NOSE = nasal obstruction and septoplasty effectiveness scale; NPEF = nasal peak expiratory flow; OCES = olfactory cleft endoscopy scale; PEFI = peak expiratory flow index; PIFI = peak inspiratory flow index; PNIF = peak nasal inspiratory flow; PROM = patient‐reported outcome measure; QOD = questionnaire of olfactory disorders; QOD‐NS = questionnaire of olfactory disorders–negative statements; SF36 = Short Form 36; SNOT‐16 = 16‐item Sino‐Nasal Outcome Test; SNOT‐22 = 22‐item Sino‐Nasal Outcome Test; UPSIT = University of Pennsylvania Smell Identification Test; VAS = visual analogue scale.

PROMs

For nasal obstruction, the most commonly studied PROM was the visual analogue scale (VAS) for nasal obstruction which was used in 45% of the included studies. ¹⁷ , ³⁶ , ⁴⁰ , ⁴¹ , ⁴⁴ , ⁴⁸ , ⁴⁹ , ⁵⁰ , ⁵² , ⁵³ , ⁵⁴ , ⁵⁵ , ⁵⁶ , ⁵⁷ , ⁵⁸ , ⁵⁹ , ⁶⁰ Only 25% of the included papers used the validated NOSE questionnaire for nasal obstruction. ³⁰ , ³¹ , ³⁵ , ⁴⁶ , ⁴⁸ , ⁵⁰ , ⁶¹ , ⁶² Because the NOSE questionnaire was only validated as a PROM for nasal obstruction due to a deviated septum for patients undergoing septoplasty in 2004,⁶³ studies conducted prior to 2004 would not have been able to use the questionnaire. Nevertheless, a large number of studies conducted after 2004 opted not to utilize the NOSE questionnaire. ¹⁷ , ¹⁹ , ²⁹ , ³⁵ , ⁴⁰ , ⁴¹ , ⁴² , ⁴⁴ , ⁴⁸ , ⁵⁰ , ⁵³ , ⁵⁴ , ⁵⁵ , ⁵⁶ , ⁵⁷ , ⁵⁸ , ⁵⁹ , ⁶⁰

For olfactory dysfunction, the 2 most popular choices of PROMs used were the SNOT‐22 ¹⁷ , ³⁹ , ⁴⁰ , ⁶⁴ and a VAS for the sense of smell ¹⁷ , ⁴⁰ , ⁶⁴ , ⁶⁵ ; only 3 studies utilized the Questionnaire of Olfactory Disorders (QOD). ³⁷ , ³⁹ , ⁶⁶

The SNOT‐22 ⁶⁷ was used in 83% of the included papers, ²⁰ , ²¹ , ²³ , ²⁴ , ²⁶ , ³⁴ , ⁴⁰ , ⁴⁵ , ⁶⁰ , ⁶⁸ , ⁶⁹ , ⁷⁰ , ⁷¹ , ⁷² , ⁷³ with VAS used in 10 out of 18 papers, ²⁰ , ²⁶ , ⁴⁰ , ⁵⁹ , ⁶⁰ , ⁶⁵ , ⁷¹ , ⁷² , ⁷⁴ and SNOT‐22 was used to correlate with rhinoscintigraphy results. ³³ , ³⁴

Objective outcome measures/tools

Nasal obstruction tools

There were 36 included studies evaluating the correlation for nasal obstruction in CRS, AR, and DNS, covering 3 main objective measures: rhinomanometry, ¹⁹ , ³⁰ , ³¹ , ³⁵ , ⁴⁸ , ⁴⁹ , ⁵² , ⁵³ , ⁵⁴ , ⁵⁵ , ⁵⁷ , ⁶¹ AcR, ¹⁹ , ²⁸ , ²⁹ , ³⁵ , ³⁶ , ⁴⁶ , ⁵⁴ , ⁵⁶ , ⁵⁸ and PNIF ¹⁷ , ⁴² , ⁴⁴ , ⁵⁰ , ⁵⁸ , ⁶² (12, 9, and 8 studies, respectively). A small number of publications (19%) utilized various uncommon and non‐validated measures for nasal obstruction. ¹⁷ , ³⁶ , ⁴⁰ , ⁴¹ , ⁴² , ⁴³ , ⁴⁶ , ⁵⁸

Rhinomanometry

Rhinomanometry is a well‐established and standardized quantitative tool to objectively assess nasal obstruction. ⁷⁵ , ⁷⁶ It measures nasal airway resistance, which is calculated from air flow and pressure readings taken from the nasal cavity. It has been considered the gold standard by some researchers, ⁷⁷ but is an expensive and time‐consuming test, often reserved for research. Statistically significant correlations between rhinomanometry and PROMs were found in all included studies in all of the included sinonasal disorders; however, the majority of the included publications showed a weak correlation (r < 0.5) and 14% of the included papers reported a moderate correlation (r > 0.5). ³⁰ , ⁴⁹ , ⁵³ , ⁵⁴ , ⁶¹ The strongest correlation was identified in only 1 paper with AR patients (r > 0.7).⁵³ There was a moderation correlation shown in 2 studies with DNS patients (r > 0.5). ³⁰ , ⁷⁸ The relationship between rhinomanometry and PROMs is not well‐studied in CRS patients, with only 1 study demonstrating a very weak and clinically insignificant correlation. ⁴⁹

AcR

AcR is an alternative tool to assess nasal obstruction. It measures the geometry of the nose by the means of acoustic reflection.⁷⁹ It estimates the volume of the nasal cavity through quantifying the cross‐sectional area of the nose. This test relies on the reflection of an audible sound wave (150 to 10,000 Hz) which is propagated from a click in a tube under the nostril.⁸⁰ Although AcR provides useful information, it is largely a research tool due to the complexity of performing the test in routine clinical practice. Out of the 9 studies quantifying the correlation between AcR and PROMs, the majority showed no correlation, ¹⁹ , ²⁸ , ³⁵ , ⁴⁶ , ⁵⁴ whereas the studies demonstrating significant correlation only revealed a variable strength of correlation. ²⁹ , ³⁶ , ⁵⁶ , ⁵⁸ In AR patients, there was a weak to moderate correlation between AcR and PROMs. ²⁹ , ³⁶ , ⁵⁶ A stronger correlation between AcR and PROMs (r = 0.5) was demonstrated in 1 study with CRS patients.⁵⁸

PNIF

PNIF is another tool to assess nasal obstruction objectively.⁸¹ A peak nasal inspiratory flowmeter includes a mouthpiece placed over the nose and the mouth to measure nasal flow with the mouth closed. When using the device, subjects are asked to inhale sharply through the nose 3 times and the best of 3 readings is taken as the final result.⁸² It has been considered to be the simplest, reliable, reproducible, and most widely available objective tool in measuring nasal patency. ⁸¹ , ⁸³

There were only 8 included studies evaluating the correlation between PNIF and PROMs, suggesting more research is needed in evaluating this inexpensive, easily applied, fast, and portable tool. A total 5 out of 8 studies identified a correlation between PNIF and PROMs in AR and CRS. The strength of correlation was weak in the majority of the included papers (0.35 < r < 0.5) ¹⁷ , ⁴² , ⁴⁴ , ⁵⁹ , ⁶⁰ ; however, stronger correlation was noted in CRS patients (r > 0.5). ¹⁷ , ⁵⁰ , ⁵⁸ The only publication in the DNS population found no statistically significant correlation between PNIF and PROMs, post‐septorhinoplasty.⁶²

Olfactory dysfunction tools

Although olfactory dysfunction can exist in all of the considered sinonasal disorders, all 11 included studies were conducted exclusively within the CRS population. The most commonly utilized objective outcome measures used to assess olfactory dysfunction were the Sniffin’ Sticks test and the UPSIT. The rest of the included papers were considered uncommon, non‐validated tools for assessing olfaction. ³⁸ , ³⁹ , ⁴⁰ , ⁴⁷

Sniffin’ Sticks test

The Sniffin’ Sticks test assesses 3 components of olfaction including threshold (T), discrimination (D) and identification (I) testing to give a composite TDI score.⁸⁴ The correlation between TDI scores and PROMs was considered strong (r > 0.7) in the all of the included publications. ¹⁷ , ³⁷ , ⁶⁵ , ⁸⁵ The strongest correlation was found with the QOD,³⁷ especially in patients with severe CRS with nasal polyps (CRSwNP) with an LMS of >15 and in allergic fungal rhinosinusitis (AFRS).^{65 , 85}

UPSIT

The UPSIT is another common smell test that has also been validated for use in different parts of the world, ⁸⁶ , ⁸⁷ , ⁸⁸ but only tests olfactory identification. Three papers evaluated its relationship with PROMs, ⁶⁴ , ⁶⁶ , ⁸⁹ 2 of which demonstrated a weak correlation. ⁶⁴ , ⁶⁶ The largest sized study with 367 patients showed no statistically significant correlation.⁸⁹

Clinical staging tools for CRS

The most studied objective outcome measure for CRS clinical findings was the LMS in 18 out of 23 publications. ¹⁸ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ³⁴ , ⁴⁰ , ⁴⁵ , ⁶⁸ , ⁶⁹ , ⁷⁰ , ⁷¹ , ⁷² , ⁷⁴ LKS was the next most commonly utilized tool with 5 out of 23 studies. ¹⁸ , ²⁰ , ⁴⁰ , ⁶⁵ , ⁷⁰ The majority of the included papers considered both main phenotypes of CRS with the exception of 1 paper studying AFRS. ⁶⁵

LMS

The LMS ⁹⁰ is based on points (0 = normal, 1 = partial opacification, 2 = total opacification) given for the degree of opacification of the various sinus groups; with the ostiomeatal complex scored as 0 or 2. The majority of the included publications (75%) demonstrated a lack of correlation with PROMs. Very weak correlations (r < 0.35) were found in the remaining 25% publications. ¹⁸ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ³⁴ , ⁴⁰ , ⁴⁵ , ⁶⁸ , ⁶⁹ , ⁷⁰ , ⁷¹ , ⁷² , ⁷⁴ , ⁹¹ , ⁹² Of note, the largest‐scale study with 1840 CRS patients showed a clinically insignificant correlation (r = 0.058).⁹³

LKS

In 1995, the LKS was designed as an endoscopic staging system to describe endoscopic findings in CRS patients who have already undergone endoscopic sinus surgery (ESS). It is based on the degree of scarring, crusting, edema, polyps, and discharge.³ The system scores for the following signs: scarring (0 = absent; 1 = mild; 2 = severe), crusting (0 = absent; 1 = mild; 2 = severe), edema (0 = absent; 1 = mild; 2 = severe), polyps (0 = none; 1 = middle meatus; 2 = beyond middle meatus), and discharge (0 = none; 1 = clear and thin; 2 = thick and purulent). Overall, LKS correlated poorly with PROMs for global CRS symptoms in the included studies. Although statistically significant correlations were found between LKS and the VAS of CRS symptoms, they were shown to be weak (r < 0.35). ²⁰ , ⁴⁰ Only 1 paper demonstrated a slightly stronger correlation (r = −0.51) where the VAS was correlated with a modified Kupferberg endoscopic staging score (Philpott‐Javer),⁶ which was modified for AFRS.⁶⁵ No statistically significant correlation was found between LKS and the CSS, Rhinosinusitis Disability Index, ¹⁸ or the SNOT‐22. ²⁰ , ⁴⁰ , ⁷⁰

MCC dysfunction tools

MCC is a key first‐line defense mechanism in the upper and lower respiratory tract and has shown to be significantly decreased in CRS patients, ⁹⁴ , ⁹⁵ , ⁹⁶ , ⁹⁷ suggesting that defective MCC may have a role in the pathogenesis of CRS ⁸ and AR.⁹⁸ Rhinorrhea may be considered the closest sinonasal symptom to correlate with MCC dysfunction in CRS and AR. ⁸ , ⁹⁹

Rhinoscintigraphy is an imaging test to assess nasal MCC in CRS patients with the use of technetium‐99m macro‐aggregated albumin. The only 2 included studies quantified the relationship between rhinoscintigraphy and SNOT‐22 in CRS populations; however, neither of these publications showed a statistically significant correlation. ^34, , ¹⁰⁰ No study was performed in AR patients. The saccharin test is an alternative, simpler, reliable, and cheaper option to measure MCC.¹⁰¹ A normal result (less than 35 minutes) is useful in excluding early MCC dysfunction.⁸ No included publications quantified the correlation between MCC measured by the saccharin test and PROMs for rhinorrhea. Overall, there is a paucity of literature concerning the relationship between objective measures of MCC and PROMs.

Discussion

Summary of the key findings

The key findings of the correlation between the outcome measures are outlined in Table 2.

Strengths

This systematic review with recommendations provides the first comprehensive overview of the literature on the correlation between objective outcome measures and PROMs for the main sinonasal disorders (CRS, DNS, AR) to date. This systematic review incorporates a methodological critical appraisal of the included studies using a bespoke checklist developed from 2 validated checklists. ¹⁵ , ¹⁶ Because the systematic review was conducted by 3 reviewers, this minimized any potential errors and selection bias commonly occurring in systematic reviews conducted by 1 reviewer.

Limitations

Although a systematic approach was conducted during our literature search, publication bias is always present. Limitations on resource availability meant that only English language publications were included in the review. Where possible, included studies were checked to ensure adherence to the diagnostic criteria for the sinonasal disorders (CRS, DNS, AR) present at the time of the studies. An inevitable limitation of this review is that older criteria utilized in some of the included papers might have not conformed to current diagnostic criteria. There were a limited number of studies identified for review in the case of objective outcome measures that have been poorly studied. Only Medline OVID and Embase OVID databases were searched for high‐quality peer‐reviewed studies and, consequently, other studies in the gray literature ¹⁰² will have been overlooked. The included papers were analyzed using a narrative approach instead of conducting a meta‐analysis, precluding the assessment of publication bias. Although a bespoke critical appraisal tool was generated from 2 other well‐known checklists to accommodate the included studies, it is a self‐designed and non‐validated tool.

Implications for future research

Our recommendations for the use of the objective outcome measures

With a focus on patient‐centered care, outcome measures should not only be able to detect the difference between the effect of study interventions but also reflect aspects of the disease felt to be important to participants in the study. ¹⁰³ , ¹⁰⁴ Therefore, one could argue that objective outcome measures are not always helpful due to the lack of correlation with PROMs which are crucial to inform improvements in health care delivery. On the other hand, it is well known in other medical specialties that objective outcome measures and PROMs often do not correlate. ¹⁰⁵ , ¹⁰⁶ This discrepancy can be explained by considering that PROM results are multifactorial whereas objective outcome measures only measure biological/physiological aspects of the puzzle. Thus, researchers can dispute that although certain objective outcome measures may not correlate with PROMs, the objective outcome measures can be useful if they provide different information and are appropriate to conduct in their study.

We aim to make recommendations for the use of the objective outcome measures in both routine clinical practice and in research settings. This section outlines our recommendations for the objective tools used to assess nasal obstruction, olfactory dysfunction, global CRS symptoms, and MCC dysfunction. A full consideration of the correlation findings from our review and other factors including time, costs, complexity, and efficiency has been made before reaching each recommendation. Table 3 summarizes our recommendations.

TABLE 3.

A summary of the recommendations for the use of objective tools for sinonasal disorders in research and routine clinical practice

Objective outcome measures	Research	Routine clinical practice
Nasal obstruction tools
PNIF	Highly recommend, with a validated PROM	Highly recommend, with a validated PROM
Rhinomanometry	Recommend with cautions, with a validated PROM	Not recommended
Acoustic rhinometry	Recommend with cautions, with a validated PROM	Not recommended
Other tools	Inconclusive	Inconclusive
CRS clinical staging tools
CT staging score LMS	Not recommended	Not recommended
Nasal endoscopic score LKS	Highly recommend, with a validated PROM	Encouraged, with a validated PROM, but depending on surgeon's reference
Olfactory dysfunction tools
Sniffin’ Sticks test	Highly recommend, with a validated PROM	Recommended, with a validated PROM for all patients reporting olfactory dysfunction at baseline assessments Recommended, with a validated PROM for further follow‐ups depending on other factors
UPSIT	Recommend a second choice after the Sniffin’ Sticks test, with a validated PROM (Consider other factors including costs, settings, patients etc.)	Recommend as a quick smell test, with a validated PROM in private settings Recommended as a second choice after the Sniffin’ Sticks test in public healthcare settings and if full olfactory assessment required
Other tools	Inconclusive	Inconclusive
Rhinorrhea/MCC dysfunction tools
Rhinoscintigraphy	Inconclusive	Inconclusive

CRS = chronic rhinosinusitis; CT = computed tomography; LKS = Lund‐Kennedy score; MCC = mucociliary clearance; PNIF = peak nasal inspiratory flow; PROM = patient‐reported outcome measure; UPSIT = University of Pennsylvania Smell Identification Test.

Nasal obstruction tools

PNIF

Most of the included papers reported a correlation between PNIF readings and PROMs, which may suggest redundancy in performing the investigation. However, all correlations found were considered weak to moderate, and PNIF is a simple, easy and cheap test to perform,⁸¹ being the most widely utilized tool for assessing nasal patency in ear, nose, and throat (ENT) clinics. Additionally, it has been demonstrated to be as valid and reliable as the “gold standard” measures for assessing nasal patency including rhinomanometry and AcR.⁸³ Overall, we recommend PNIF be used in routine clinical practice as an objective tool in conjunction with a validated PROM to assess changes in nasal patency in the main sinonasal disorders. We also support the use of PNIF in research settings due to its low cost and simplicity to reduce patient burden and improve research efficiency. More work is needed to confirm that PNIF is a useful objective tool for structural nasal obstruction such as DNS, so we make a more reserved recommendation of PNIF to be used alone as an objective tool for DNS, but always in conjunction with a validated PROM.

Rhinomanometry

Most of the included papers showed a weak correlation between rhinomanometry and a self‐rating score of nasal obstruction. Although rhinomanometry provides “gold standard” objective measurements of nasal patency,⁷⁶ it has only been generally performed in research settings due to the time, financial constraints, and the scarcity of the tool in routine ENT clinics. Additionally, undergoing this lengthy test can be inconvenient for most people, especially if repeated over time as an outcome measure in research or routine clinical practice. We recommend that the use of rhinomanometry should be limited to research scenarios only and that in routine clinical practice it is replaced by the use of a more cost‐ and time‐effective investigation, such as PNIF.

AcR

Most of the included publications showed no statistically significant correlation between AcR and the SNOT‐22 questionnaire. AcR is a complicated tool to perform requiring the users to be trained to obtain meaningful readings and it is often unavailable in ENT clinics.⁷ Therefore, we recommend that AcR should remain a research tool and, even in research settings, we propose that PNIF may represent a better alternative to AcR to reduce the burden of many investigations on patients and researchers.

Other tools

There is insufficient high‐quality evidence, in terms of the number of studies and the quality of the publications, to make any recommendations for the use of the various non‐validated tools for assessing nasal obstruction, such as rhinostereometry. More research should be conducted to examine these tests further as potential objective tools for assessing nasal patency.

Olfactory dysfunction tools

Sniffin’ Sticks test

A strong correlation between the Sniffin’ Sticks TDI scores and PROMs was noted in the majority of included studies, suggesting that the use of PROMs alone to assess olfaction is sufficient. However, a position paper on olfactory dysfunction has reported that self‐rating for the sense of smell is often unreliable and subjective olfactory assessment should not be undertaken in isolation, given its poor accuracy. ¹⁰⁷ In addition to being the most validated olfactory test globally, it has been used extensively in research and is more cost‐effective than single‐use smell tests, as the Sniffin’ Sticks can be reused numerous times during their shelf life (12 to 18 months).

We recommend the Sniffin’ Sticks test in research settings, as it provides a more comprehensive assessment of olfaction compared to UPSIT. On the contrary, the UPSIT has the advantage that it can be sent by post to patients for testing at home and does not require someone to administer the test. We advise researchers to consider the availability of testing resources and the research personnel required to perform the Sniffin’ Sticks tests at multiple follow‐ups, balanced against the overall cost of the kits, compared to the UPSIT.

In routine clinical practice, we recommend the Sniffin’ Sticks test in conjunction with a validated PROM for all patients presenting with olfactory dysfunction at the first clinical visit, thereby establishing a baseline assessment. The decision to employ the smell test in routine clinical follow‐up should depend on local factors such as cost, time, and the availability of clinical staff.

UPSIT

The correlation between the UPSIT and PROMs ranges from no statistically significant correlation to a weak correlation in all the included studies. Although UPSIT is a reliable, well‐validated, standardized smell test, the major drawback is that it assesses odor identification alone, which may explain the above issue with varying correlations. Because UPSIT is a single‐use test, it may be more costly (approximately £15 per kit) if used in a study with large patient numbers; however, it requires no clinician supervision and poses negligible burden on patients. For research settings, we would recommend this as a second‐choice test with the caveat of it being advantageous where participants are unable to travel to the study sites. In routine clinical practice, the UPSIT can be the first choice for a simple and quick smell test with a validated PROM in non‐universal health care systems where patients purchase their own tests.

Clinical staging tools for CRS symptoms

LMS

LMS has become a popular staging score for CT scans in CRS due to its simplicity, clarity, and its amenability for use by non‐specialist radiologists. Because the relationship between measurable effects of biological variables and symptoms is complex, the absence of the correlation between LMS and PROMs found in the majority of the included studies is unsurprising. LMS was never expected to correlate with symptom severity, merely created to evaluate quantitatively the burden of inflammation in CRS.⁹⁰ Although a CT scan may be required as a diagnostic test for CRS (where endoscopy findings are inconclusive) ⁸ and is mandatory for all patients undergoing any form of sinus surgery, LMS is not ideally placed as an outcome measure to be repeated over time in all scenarios, especially given the radiation exposure. This latter concern may change with the improved availability of cone beam CT scanning. Overall, we recommend that LMS should only be used as a clinical staging tool for CRS at the current time.

LKS

LKS remains the most widely used endoscopic scoring system in rhinological research and was created as a standardized objective outcome measure by providing a numeric score summarizing nasal endoscopic appearances, and also aiding assessment after ESS.³ LKS was found to not correlate with the SNOT‐22 ²⁰ , ⁴⁰ , ⁷⁰ and poorly correlated with other non‐validated PROMs including the VAS of CRS symptoms. ²⁰ , ⁴⁰ This suggests that LKS provides different information on the CRS disease extent and we recommend that it should always be performed with a validated PROM for CRS (eg, SNOT‐22) in both research and routine clinical practice settings. Our suggestion is supported by the Chronic Rhinosinusitis Outcome Measures (CHROME) paper's findings, ¹⁰⁸ stating that the 15 most highly rated outcomes in the study could be measured by the SNOT‐22 repeated over time, in conjunction with LKS.

The standardization of endoscopic appearances provided by LKS helps to reduce the heterogeneity of outcome measures in studies and hence facilitate systematic reviews and meta‐analyses. Hence, we recommend LKS as an objective outcome measure at all visits throughout studies, to compare their results with other studies. For routine clinical practice, the documentation of LKS depends on the clinician's preference. A set of LKS obtained pre‐ESS and post‐ESS or a record of repeated LKS over time could be useful in a personal audit of the ENT surgeon's own practice.

MCC dysfunction symptoms/rhinorrhea tools

Although rhinoscintigraphy has been demonstrated to be reliable in detecting a deduction in MCC,⁹⁷ it is poorly correlated to self‐rated symptomatic scores in our review. Due to the little evidence available, we were unable to make any recommendations for the use of any objective tools assessing MCC dysfunction.

Our recommendations for PROMS

Table 4 provides a summary of the recommendations for the use of PROMs for each sinonasal disorder. PROMs should be validated for a specific sinonasal disorder with a specific patient population due to cultural influences on their suitability. For CRS, a systematic review ¹⁰⁹ found that the highest‐quality validated PROM is the SNOT‐22. For nasal obstruction caused by DNS, the NOSE questionnaire is only currently validated PROM.⁶³ For olfactory dysfunction, the QOD has been validated ³⁷ in CRS. For AR, the Rhinoconjunctivitis Quality of Life Questionnaire (RQLQ) is a widely validated PROM for AR worldwide ¹¹⁰ and is more disease‐specific.

TABLE 4.

A summary of the recommendations for the use of PROMs for each sinonasal disorder

Diagnosis	Recommended validated PROMs to be used
Deviated nasal septum	NOSE score
CRS CRS‐related olfactory dysfunction	SNOT‐22 QOD
Allergic rhinitis	RQLQ

CRS = chronic rhinosinusitis; NOSE = nasal obstruction and septoplasty effectiveness scale; PROM = patient‐reported outcome measure; QOD = questionnaire of olfactory disorders; RQLQ = Rhinoconjunctivitis Quality of Life Questionnaire; SNOT‐22 = 22‐item Sino‐Nasal Outcome Test.

Although the VAS of different sinonasal symptoms has been used extensively in the included papers and advocated by EPOS 2020 for measuring symptom control due to its simplicity and availability, it is a general and non‐validated PROM for specific sinonasal disorders. Overall, we recommend the use of the validated PROMS for sinonasal disorders including the SNOT‐22 questionnaire, the NOSE questionnaire, and the RQLQ. If it is not feasible to use a validated PROM, a recognized quantitative scale such as a VAS of symptoms should be used as a substitution for measuring symptom control.

Recommendations for future research

Further research is required to facilitate the selection of the most suitable objective outcome measures in rhinology to minimize redundancy and reduce patient and research burden. Specifically, more emphasis should be paid to the poorly studied objective tools such as PNIF, the Sniffin’ Sticks test, and the UPSIT to strengthen recommendations for their use in the sinonasal disorders.

Further research should aim to produce an international guideline on “gold standard” objective outcome measures for the sinonasal disorders.

Conclusion

There is limited literature on the correlation between objective outcome measures and PROMs in the sinonasal disorders. Overall, we recommend that the setting‐appropriate objective outcome measures are used with validated PROMs for sinonasal symptoms. More work on the relationship between the outcome measures in sinonasal disorders is needed to confirm our recommendations.

OVID MEDLINE SEARCH

1 Rhinometry, Acoustic/ (555)

2 RHINOMANOMETRY/ (664)

3 Rhinostereometry.mp. (53)

4 SPIROMETRY/ (21122)

5 Peak inspiratory nasal flow.mp. (10)

6 PNIF.mp. (222)

7 Peak Expiratory Flow.mp. (9549)

8 PEF.mp. (4948)

9 Mucociliary Clearance/ (2515)

10 saccharine test.mp. (56)

11 Sniffin Sticks test.mp. (280)

12 smell test*.mp. (279)

13 Olfactory test*.mp. (848)

14 Sniffin Sticks.mp. (708)

15 University of Pennsylvania Smell Identification Test.mp. (429)

16 UPSIT.mp. (350)

17 Tomography, X‐Ray Computed/ (380121)

18 CT sinus.mp. (25)

19 Lund Mackay scores.mp. (124)

20 Lund Kennedy scores.mp. (43)

21 Nasal Endoscopy.mp. (1641)

22 Endoscopic score.mp. (551)

23 1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 (33194)

24 9 or 10 (2538)

25 11 or 12 or 13 or 14 or 15 or 16 (1880)

26 17 or 18 or 19 (380209)

27 20 or 21 or 22 (2213)

28 Sino‐nasal Outcome Test‐22.mp. (127)

29 SNOT‐22.mp. (634)

30 SNOT 22.mp. (634)

31 (SNOT‐16 or SNOT 16).mp. (15)

32 (SNOT‐20 or SNOT 20).mp. (277)

33 (SNOT‐23 or SNOT 23).mp. (4)

34 (31‐Item Rhinosinusitis Outcome Measurement or RSOM‐31).mp. (20)

35 Questionnaire of Olfactory Disorders.mp. (25)

36 Sinusitis Control Test.mp. (1)

37 (EuroQoL five‐dimensional questionnaire or EQ‐5D).mp. (8343)

38 Chronic Sinusitis Survey.mp. (78)

39 (Nasal Obstruction and Septoplasty Effectiveness Scale).mp. (4)

40 NOSE score.mp. (138)

41 (visual analogue scale or visual analog scale).mp. (51284)

42 28 or 29 or 31 or 32 or 33 or 34 or 35 or 36 or 37 or 38 (9371)

43 39 or 40 (140)

44 23 and 42 (51)

45 23 and 43 (22)

46 23 and 41 (354)

47 25 and 42 (53)

48 25 and 41 (78)

49 24 and 42 (19)

50 27 and 42 (113)

51 27 and 43 (7)

52 27 and 41 (118)

53 26 and 42 (144)

54 26 and 43 (2)

55 26 and 41 (849)

56 44 or 45 or 46 or 47 or 48 or 49 or 50 or 51 or 52 or 53 or 54 or 55 (1626)

57 nose diseases/ or nasal obstruction/ or nose deformities, acquired/ or paranasal sinus diseases/ or rhinitis/ (30128)

58 septal deviation.mp. (986)

59 rhinosinusitis.mp. (9313)

60 57 or 58 or 59 (34664)

61 56 and 60 (465)

62 limit 61 to English language (433)

How to Cite this Article:Ta NH, Gao J, Philpott C. A systematic review to examine the relationship between objective and patient‐reported outcome measures in sinonasal disorders: recommendations for use in research and clinical practice. Int Forum Allergy Rhinol. 2021;11:910–923.