Validation of the applicability of the traditional Chinese version of the University of Pennsylvania Smell Identification Test in patients with chronic rhinosinusitis

Abstract

The 40-item University of Pennsylvania Smell Identification Test (UPSIT) is the most widely used smell test in the world. Presently, culturally modified versions of this test are available in multiple languages. A traditional Chinese version of the UPSIT (UPSIT-TC) has been developed for administration in Taiwan. The purpose of this study was to investigate the validity and reliability of the UPSIT-TC in Taiwanese patients with chronic rhinosinusitis (CRS). The phenylethyl alcohol (PEA) odor detection threshold test, the North American version of UPSIT (UPSIT-NA), and the UPSIT-TC were administered to 40 healthy subjects and to 100 CRS patients before and after functional endoscopic sinus surgery (FESS). The UPSIT-TC showed good internal consistency (Cronbach's alpha = 0.887, 0.886, and 0.870 at three test occasions) and test–retest reliability (p < 0.001). The scores of UPSIT-TC were significantly correlated to the PEA thresholds (p < 0.001). The UPSIT-TC scores were significantly higher than those of the UPSIT-NA (p = 0.028) when analysis was performed with logistic regression with independent variables including test occasions (before or after FESS), test methods (UPSIT-NA or UPSIT-TC), status of polyp (with or without), and PEA thresholds (improved or did not improve). In addition, there were significant between-group differences in UPSIT-TC scores including healthy versus CRS, CRS with polyps versus CRS without polyps, and PEA thresholds improved versus PEA thresholds which did not improve. The UPSIT-TC is reliable and valid for measuring olfactory function in Taiwanese patients with rhinosinusitis. In addition, the UPSIT-TC clearly resulted in better performance than that of UPSIT-NA.

Despite the fact that a number of clinical olfactory tests, most notably tests of odor identification, detection, discrimination, and memory, have been described in the literature, only a few have achieved widespread acceptance and are available commercially.¹ The most widely used of these tests is the University of Pennsylvania Smell Identification Test (UPSIT).² UPSIT has been administered to nearly 500,000 patients.¹ The development of this test has allowed for accurate and convenient testing of olfactory function without the use of complex olfactometric equipment, cumbersome bottles, or pen-like devices.³ However, in rare instances parosmia may mask the true ability of a patient to identify odors, in common with other odor identification tests.¹ Moreover, most odor identification tests have cultural biases and the same test may not be applicable for persons of all ages and settings.⁴

Because of its wide applicability, the North American version UPSIT (UPSIT-NA) has been translated into multiple language versions. For example, there are British English, Dutch, French, Italian, German, Japanese, Korean, Portuguese, and Spanish versions of the UPSIT.¹ In most versions, a few UPSIT items or response alternatives have been modified to take into account cultural differences and to allow for the use of common norms. In our previous UPSIT-NA study of healthy Taiwanese subjects, 10 odorants were correctly identified by less than two-thirds of 40 young nursing students.⁵ Based on this result, 10 odorants from the UPSIT-NA were planned to be replaced to form a traditional Chinese language version of the UPSIT (UPSIT-TC); however, only 7 of these 10 odorants could be replaced because of the limitation of available odorants. “Pizza” in item 1, “smoke” in item 33, and “lemon” in item 36 were not replaced. In addition, “dill pickle” in item 25 of the UPSIT-NA was replaced because it was unfamiliar to Taiwanese subjects. Except for replacement of some odorants, some odor descriptors that were also considered unfamiliar by Taiwanese subjects were changed in the UPSIT-TC. In a pilot study of the UPSIT-TC, we compared the UPSIT-NA and the UPSIT-TC administered to a healthy group of Taiwanese subjects. In accord with the modifications, the scores on the UPSIT-TC were significantly higher than those on UPSIT-NA.⁶

Chronic rhinosinusitis (CRS) is a common etiology that affects olfactory function.^7,8 This study assessed the applicability of the UPSIT-TC in evaluating the olfactory function in CRS patients before and after functional endoscopic sinus surgery (FESS) and compared its applicability in patients with and without nasal polyps and in those whose olfactory function improved or did not improve after surgery.

MATERIALS AND METHODS

Study Subjects

Forty healthy subjects and 100 patients with CRS who underwent FESS between April of 2009 and January of 2012 were enrolled in this study. The healthy subjects considered their olfactory function as normal, and their olfactory thresholds were all below −6.00 log v/v when tested by the phenylethyl alcohol (PEA) odor detection threshold test.¹ The diagnosis of CRS was based on a history of rhinosinusitis, the findings of nasal endoscopy, and an examination of computed tomography scans.⁷ Duration of disease was qualified by continuous symptoms for at least 12 consecutive weeks. Nasal endoscopy identified discolored nasal drainage in the nasal cavities, nasal polyps, polypoid swelling, or edema of the middle meatus or ethmoid bulla. Computed tomography scans revealed mucosal thickening and complete opacification or air–fluid level of one or more sinuses. Any patient who had a history of immunodeficiency or a previous sinus surgery, whose age was <20 years, or whose olfactory dysfunction was suspected to result from head injury or upper respiratory infection was excluded from the study. The patients were divided into those with or without nasal polyps based on endoscopic examination. This study was approved by the Institutional Review Board of Taichung Veterans General Hospital. All participants provided written informed consent.

The 40 healthy subjects (9 male and 31 female subjects) had a mean age of 33.1 years (ranging from 22 to 48 years). The 100 CRS patients' (69 male and 31 female subjects) ages ranged from 20 to 84 years, with a mean of 46.0 years. Among CRS patients, nasal polyps were present in 52 patients (37 male and 15 female subjects). Their ages ranged from 20 to 78 years, with a mean of 45.6 years. Forty-eight patients (32 male and 16 female subjects) were without nasal polyps. Their ages ranged from 20 to 84 years, with a mean of 46.5 years.

Olfactory Testing

The PEA odor detection threshold test is a forced-choice measure of mono-odor olfactory detection.¹ The odorant concentrations are presented to tester via polypropylene squeeze bottles. A staircase testing procedure begins at the −6.00 log concentration with a step of a half-log (v/v) dilution series extending from −9.00 log concentration to −1.00 log concentration. The diluent is light United States Pharmacopeia grade mineral oil. The odorant concentration is increased until correct detection occurs on five sets of consecutive trials at a given concentration. If an incorrect response occurs on any trial, the staircase is moved upward one full log step. When a correct response is made on all five trials, the staircase is reversed and subsequently moved up or down in 0.50 log increments or decrements, depending on the subject's performance on two pairs of trials at each concentration step. The geometric mean of the last four of seven staircase reversal points serves as the threshold measure.

The UPSIT-NA and UPSIT-TC (Sensonics, Inc., Haddon Heights, NJ) are comprised of four 10-odorant booklets that can be self-administered in 10–15 minutes. Each of the 40 “scratch-and-sniff” odorants are embedded in 10- to 50-μm microcapsules fixed in a propriety binder and positioned on brown strips located at the bottom of the pages of each test booklet.⁹ When the examinee takes the UPSIT-NA or UPSIT-TC, he/she releases each of the 40 odorants by scratching the strip with a pencil tip in a standardized manner. The identity of the released odorant is signified by choosing a name from a set of four odor descriptors.¹ The test is scored as the number of odors identified correctly. A response is required for each test item even if no smell is perceived (i.e., the test is forced choice), allowing for the detection of malingering based on improbable responses. The UPSIT-TC differs from the UPSIT-NA in having eight odorant changes.⁶ “Clove” was replaced by “sandalwood” in item 8, “cheddar cheese” by “fish” in item 14, “cinnamon” by “coffee” in item 15, “gingerbread” by “rubber tire” in item 20, “dill pickle” by “jasmine” in item 25, “lime” by “grapefruit” in item 27, “wintergreen” by “magnolia” in item 29, and “grass” by “body powder” in item 32. Additionally, one descriptor, “skunk,” was replaced by “dog” at various places, and another descriptor, “pumpkin pie,” was removed from the UPSIT-TC.

The healthy subjects received assessment of UPSIT-TC and reassessment 2 weeks later. None of the health subjects reported having upper airway infection during the test–retest interval. All CRS patients took olfactory tests including the PEA odor detection threshold test, UPSIT-NA, and UPSIT-TC before surgery. After surgery, the patients only received local nasal treatment. Neither antihistamines nor oral or nasal steroids were prescribed. At 2 and 3 months after FESS, they took all of the three olfactory tests again. When the PEA thresholds were compared before and 3 months after FESS, patients were divided into those whose PEA thresholds improved after surgery (decreased PEA thresholds) and those whose PEA thresholds did not improve (unchanged or increased PEA thresholds).

Assessment of the Reliability and Validity of UPSIT-TC

The reliability of UPSIT-TC was assessed with internal consistency and test–retest reproducibility. The internal consistency was evaluated with Cronbach's α coefficient. The generally acceptable Cronbach's α is ≥0.7. The test–retest reliability was assessed by the correlation between the scores of the two UPSIT-TC tests taken 2 weeks apart by 40 healthy subjects without olfactory changes during the period. In general, an acceptable test–retest reliability coefficient score is ≥0.7.¹⁰ A correlation assessment between UPSIT-TC scores and PEA thresholds was obtained with the Pearson's correlation coefficient and was used to analyze the concurrent validity of the study instrument. Logistic regression through a generalized estimating equation model was used to predict the UPSIT scores with independent variables including test occasions (before or after FESS), test methods (UPSIT-NA or UPSIT-TC), status of polyp (with or without), and PEA thresholds (improved or did not improve). In addition, known group validity was used to assess whether the UPSIT-TC was able to discriminate between known groups, e.g., healthy subjects with normal olfaction and CRS patients, CRS patients with or without polyps, and CRS patients whose PEA thresholds improved or not. The repeated measures ANOVA analysis was used for comparing the scores of UPSIT-TC or UPSIT-NA before and after FESS. The independent t-test was used to analyze the between-group differences. The frequency of correct identification of the eight replacement items in the UPSIT-TC was compared with the frequency of correct identification of the eight original items in the UPSIT using chi-square analysis. All computations were performed using SPSS Version 17.0 (SPSS, Inc., Chicago, IL). Two-tailed values of p < 0.05 were considered statistically significant.

RESULTS

Reliability

Cronbach's α was 0.887, 0.886, and 0.870 at three test occasions of CRS patients. Pearson's correlation analysis computed across the two test occasions of healthy subjects without an olfactory change was 0.664 (p < 0.001).

Validity

Concurrent Validity.

Patients' response to the PEA odor detection threshold test and UPSIT-TC was compared at each of three time points: before surgery and 2 and 3 months after surgery. Pearson's correlation coefficients of the PEA thresholds and UPSIT-TC scores indicate a good correlation between two instruments (before surgery, correlation coefficient = −0.700 and p < 0.001; 2 months after FESS, correlation coefficient = −0.738 and p < 9.001; 3 months after FESS, correlation coefficient = −0.581 and p < 0.001).

Responsiveness

Sensitivity to Clinical Changes.

The mean UPSIT-NA and UPSIT-TC scores before and after FESS are listed in Table 1. Logistic regression through a generalized estimating equation model was used to predict the UPSIT scores with independent variables including test occasions (before or after FESS), test methods (UPSIT-NA or UPSIT-TC), status of polyp (with or without), and PEA thresholds (improved or did not improve). The UPSIT-TC scores were significantly higher than those of English version (p = 0.028, adjusted for test occasions, polyp status, and PEA thresholds).

Table 1.

Comparison of UPSIT-TC and UPSIT-NA scores before and after FESS

*p < 0.05.

#Mean ± SD.

§Logistic regression using a generalized estimate equation model.

¶Repeated measures ANOVA analysis.

UPSIT-TC = traditional Chinese version of the University of Pennsylvania Smell Identification Test; UPSIT-NA = North American version of the University of Pennsylvania Smell Identification Test; FESS = functional endoscopic sinus surgery.

When patients were further divided into patients with CRS with or without polyps, there were significant differences between pre- and post-FESS scores of UPSIT-TC and UPSIT-NA in patients with polyps (p = 0.01 and 0.03, respectively). However, there were no significant differences between pre- and post-FESS scores of UPSIT-TC or UPSIT-NA in patients without polyps (p = 0.89 and 0.372, respectively).

Rates of Correct Identification.

The rates of correct identification of the eight replaced odorants in UPSIT-TC before and after surgery are listed in Table 2. The rates of the correctly identified odorants in all 100 patients were significantly higher for items 8, 15, 25, 27, and 32 in the UPSIT-TC than in the UPSIT-NA before FESS and 2 and 3 months after FESS. The rate of correct identification of item 20 was also significantly higher in the UPSIT-TC 3 months after FESS.

Table 2.

Item analyses of the UPSIT-TC and the UPSIT-NA before and after FESS (n = 100)

*p < 0.05; **p < 0.01.

#Rate of correct identification.

UPSIT-TC = traditional Chinese version of the University of Pennsylvania Smell Identification Test; UPSIT = University of Pennsylvania Smell Identification Test; UPSIT-NA = North American version of the University of Pennsylvania Smell Identification Test.

Patients were categorized into CRS with or without nasal polyps with the rates of the correctly identified odorants for each group listed in Table 3. The rates of the correctly identified odorants in patients with nasal polyps were significantly higher for items 8, 15, 27, and 32 in the UPSIT-TC than in the UPSIT-NA before FESS and 2 and 3 months after FESS. The rate of correct identification was also significantly higher for item 25 before FESS and for item 20 3 months after FESS. The rates of the correctly identified odorants in patients without nasal polyps were significantly higher for items 8, 25, 27, and 32 in the UPSIT-TC the in UPSIT- NA before FESS and 2 and 3 months after FESS. The rate of correct identification was also significantly higher for item 15 before FESS and at 3 months after FESS.

Table 3.

Item analyses of the UPSIT-TC and the UPSIT-NA in patients with or without nasal polyps

*p < 0.05; **p < 0.01.

#Rate of correct identification.

UPSIT-TC = traditional Chinese version of the University of Pennsylvania Smell Identification Test; UPSIT-NA = North American version of the University of Pennsylvania Smell Identification Test; FESS = functional endoscopic sinus surgery.

The rates of correctly identified odorants for subgroups of patients whose PEA thresholds improved or did not improve after surgery are listed in Table 4. The rates of the correctly identified odorants for patients whose PEA thresholds improved after FESS were significantly higher for items 8, 15, 25, 27, and 32 in the UPSIT-TC than in the UPSIT-NA before FESS and 2 and 3 months after FESS. The rates of the correctly identified odorants in patients whose PEA thresholds did not improve after FESS were significantly higher for items 8, 27, and 32 in the UPSIT-TC than in the UPSIT-NA before FESS and 2 and 3 months after FESS. The rate of correct identification was also significantly higher in items 15 and 25 before FESS and 2 months after FESS.

Table 4.

Item analyses of the UPSIT-TC and the UPSIT-NA in patients whose PEA test thresholds improved or did not improve after FESS

*p < 0.05; *p < 0.01.

#Rate of correct identification.

UPSIT-TC = traditional Chinese version of the University of Pennsylvania Smell Identification Test; UPSIT = University of Pennsylvania Smell Identification Test; UPSIT-NA = North American version of the University of Pennsylvania Smell Identification Test; FESS = functional endoscopic sinus surgery; PEA = phenylethyl alcohol test.

Known Groups Validity.

The independent t-tests were used to compare the UPSIT-TC scores of healthy subjects with normal olfaction to the scores of the CRS patients before surgery. The mean UPSIT-TC score of healthy subjects was 33.18 ± 2.85 and was 20.81 ± 8.40 for CRS patients. There were significant differences in the UPSIT-TC scores between the two groups (p < 0.001).

When UPSIT-TC scores were used as the dependent variable of logistic regression analysis using a generalized estimating equation model, the UPSIT-TC scores of the nonpolyp group were significantly higher than those of the polyp group (p = 0.0013, adjusted for test occasions and PEA thresholds). In addition, the UPSIT-TC scores of patients whose PEA threshold improved were significantly higher than those of patients whose PEA thresholds were unchanged or decreased (p < 0.0001, adjusted for test occasions and polyp status; Table 5).

Table 5.

Comparison scores of UPSIT-TC between subgroups

*p < 0.01.

#Mean ± SD.

§Logistic regression using a generalized estimate equation model.

UPSIT-TC = traditional Chinese version of the University of Pennsylvania Smell Identification Test; FESS = functional endoscopic sinus surgery; PEA = phenylethyl alcohol test.

DISCUSSION

A number of clinical olfactory tests have been administered to Taiwanese subjects, including the UPSIT; detection threshold tests; Sniffin' Sticks; and a range of odor identification, discrimination, and memory tests.^4,7,11–15 The UPSIT, the most commonly used smell test in the world, has the advantages of self-administration and good validation. However, like other odor identification tests, it is influent by culture factors, which has led to the development of a number of culture-specific versions of the test.^16–18 CRS is one of the most common etiologies for olfactory dysfunction.^7,8 Therefore, we chose the CRS cohort for validation of UPSIT-TC. It is believed that CRS patients with nasal polyps tend to have more olfactory loss than those without nasal polyps.¹⁹ Perry et al.¹⁹ reported that patients with polyps had a higher preoperative olfactory dysfunction than those without polyps, and they had a higher improvement rate in olfaction 1 year after FESS. The short-term postoperative olfactory function showed significant improvement in our patients with polyps. However, our previous studies found that there were no significant differences in postoperative olfaction function between patients with or without polyps 6 months after FESS.⁷ The prognosis of olfaction after FESS depends on many factors including patients' olfactory reserve, surgical correction of the blocked olfactory cleft, and ongoing medical treatment.²⁰ The discrepancies in the results of olfactory function changes after FESS in the literature are likely reflecting differences in severity of enrolled CRS patients as well as different methods and time of olfactory testing.

Reliability is defined as the consistency or stability of the measuring instrument. The internal consistency indicates the homogeneity of the instrument. We obtained the internal consistency of UPSIT-TC through calculation the Cronbach's α value. The generally acceptable Cronbach's α is ≥0.7. Cronbach's α of UPSIT-TC was 0.887, 0.886, and 0.870 at three test occasions in CRS patients. Test–retest reproducibility measures the stability of an instrument tested on different occasions. We found that the UPSIT-TC scores significantly correlated to the retest results. Our results indicated good internal item homogeneity and test–retest correlation, indicating good reliability of the UPSIT-TC.

Validity involves the accuracy of the instrument, showing whether the instrument measures what it is supposed to measure. Concurrent validity is often used to validate a new test. Two concurrent measures are used to test each examinee, and the correlation of the results of two tests is then analyzed. In this study, we found the scores of UPSIT-TC had a strong correlation to PEA thresholds. Discriminant validity is the ability to distinguish among disease-affected patient groups and nonaffected groups. We found that the scores of UPSIT-TC were effective in distinguishing healthy volunteers with normal olfaction and CRS patients with olfactory dysfunction. When patients were divided into those with or without nasal polyps or those whose PEA thresholds improved or did not improve after FESS, the mean UPSIT-TC scores of patients who had CRS without polyps were significantly higher than those of patients who had CRS with polyps. In addition, the UPSIT-TC scores of patients whose PEA thresholds improved were significantly higher than those of patients whose PEA thresholds did not improve. Furthermore, the mean UPSIT-TC scores were significantly higher than those of the English version UPSIT after adjusting for test occasions, polyp status, and PEA thresholds. The UPSIT-TC clearly detected improved performance of CRS patients from a Taiwanese population.

In a pilot study of the UPSIT-TC administered to a healthy group of Taiwanese subjects, the rate of correct identification was higher than 70% for seven items among the eight items in which odorants had been replaced in the UPSIT-TC.⁶ In this study, we evaluated the olfactory function of CRS patients and found that the UPSIT-TC scores were significantly higher than those of the UPSIT-NA. In addition, the rate of correct identification was clearly higher in five items (items 8, 15, 25, 27, and 32). Future versions of this test will focus on further improving the test scores so that they will be comparable with those obtained from North American subjects, thereby allowing for the use of the same normative tables. In conclusion, the UPSIT-TC clearly resulted in improved performance by individuals from a Taiwanese population.