Comparison of performance of the 2016 ACR-EULAR classification criteria for primary Sjögren's syndrome with other sets of criteria in Japanese patients

Abstract

Objectives

To compare the performance of the new 2016 American College of Rheumatology (ACR)-European League Against Rheumatism (EULAR) classification criteria for primary Sjögren's syndrome (SS) with 1999 revised Japanese Ministry of Health criteria for diagnosis of SS (JPN), 2002 American-European Consensus Group classification criteria for SS (AECG) and 2012 ACR classification criteria for SS (ACR) in Japanese patients.

Methods

The study subjects were 499 patients with primary SS (pSS) or suspected pSS who were followed up in June 2012 at 10 hospitals in Japan. All patients had been assessed for all four criteria of JPN (pathology, oral, ocular, anti-SS-A/SS-B antibodies). The clinical diagnosis by the physician in charge was set as the ‘gold standard’.

Results

pSS was diagnosed in 302 patients and ruled out in 197 patients by the physician in charge. The sensitivity of the ACR-EULAR criteria in the diagnosis of pSS (95.4%) was higher than those of the JPN, AECG and ACR (82.1%, 89.4% and 79.1%, respectively), while the specificity of the ACR-EULAR (72.1%) was lower than those of the three sets (90.9%, 84.3% and 84.8%, respectively). The differences of sensitivities and specificities between the ACR-EULAR and other three sets of criteria were statistically significant (p<0.001). Eight out of 302 patients with pSS and 11 cases out of 197 non-pSS cases satisfied only the ACR-EULAR criteria, compared with none of the other three sets.

Conclusions

The ACR-EULAR criteria had significantly higher sensitivity and lower specificity in diagnosis of pSS, compared with the currently available three sets of criteria.

Introduction

Sjögren's syndrome (SS) is an autoimmune disease that affects mainly exocrine glands including the salivary and lacrimal glands, and is often associated with extraglandular manifestations, such as interstitial lung and kidney diseases, and neurological, haematological and musculoskeletal involvements.¹ It is characterised by lymphocytic infiltration into the exocrine glands and other organs, leading to dry mouth, dry eyes and various extraglandular symptoms. SS is subcategorised into primary SS (pSS) which is not associated with other well defined connective tissue diseases (CTDs), and secondary SS which is associated with other well defined CTDs.²

In Japan, the revised criteria for the diagnosis of SS proposed by the Japanese Ministry of Health (JPN) (1999),³ as well as the American-European Consensus Group classification criteria for SS (AECG) (2002)² have been used commonly in both daily clinical practice and clinical studies in this decade. In 2012, the American College of Rheumatology (ACR) published the 2012 ACR classification criteria for SS, which were proposed by the Sjögren's International Collaborative Clinical Alliance (SICCA).⁴ These three sets of criteria have also been applied for the diagnosis or classification of SS in Japan in the last 3 years. We previously analysed 694 Japanese patients with SS or suspected SS, and showed that the sensitivities of JPN, AECG and ACR in the diagnosis of SS were 79.6%, 78.6% and 77.5%, respectively, with respective specificities of 90.4%, 90.4% and 83.5%, when considering the clinical diagnosis as the ‘gold standard’.⁵ We concluded in that study the superiority of the JPN criteria in the diagnosis of SS in Japanese patients compared with the ACR and AECG criteria.⁵

Recently the 2016 new ACR-European League Against Rheumatism (EULAR) classification criteria for pSS (ACR-EULAR) were published.^{6 7} Investigators from the SICCA team and the EULAR Sjögren's Task Force formed the International SS Criteria Working Group to develop this single set of classification criteria that combined features of the ACR and AECG criteria, based on methodology consistent with the current ACR and EULAR guidelines.^{6 7} The working group adopted the methodology based on both data and expert clinical judgement, and finally defined the new classification criteria comprising five objective tests or items, and a total score of ≥4 as the cut-off for the diagnosis of pSS. The total score is derived from the sum of the weights assigned to each positive test or item as follows: focal lymphocytic sialadenitis in labial salivary gland with Focus Score (FS) of ≥1 (based on number of foci/4 mm²) and positive anti-SS-A/Ro serology with the highest weights (3 for each positive test), and Ocular Staining Score (OSS) of ≥5 (or Van Bijsterveld Score of ≥4) on at least one eye, Schirmer's test of ≤5 mm/5 min on at least one eye, and unstimulated whole saliva (UWS) flow rate of ≤0.1 mL/min with a weight of 1 for each positive test.^{6 7}

Comparison of the above four sets of criteria (table 1) shows certain differences in the adopted items. In addition to the adopted items, the purpose of these criteria sets also differs. Importantly, the JPN criteria were formulated for the diagnosis of SS as the diagnostic criteria, while other three sets of criteria (the ACR-EULAR, AECG and ACR criteria) were formulated for research purposes as the classification criteria. The purpose of the present study was to compare the performance of the new ACR-EULAR criteria with the former sets of criteria, such as the JPN, AECG and ACR criteria in Japanese patients.

Table 1.

Comparison of items adopted in the 2016 ACR-EULAR, JPN, AECG and ACR criteria

ACR, American College of Rheumatology ACR criteria for SS; ACR-EULAR, American College of Rheumatology (ACR)-European League Against Rheumatism (EULAR) classification criteria for pSS; AECG, The American-European Consensus Group classification criteria for SS; ANA, antinuclear antibody; JPN, The revised Japanese Ministry of Health criteria for the diagnosis of SS; pSS, primary SS; RF, rheumatoid factor; SS, Sjögren's syndrome; SS-A/Ro, anti-SS-A/Ro antibody; SS-B/La, anti-SS-B/La antibody.

Patients and methods

Study population

The study subjects were 499 patients (38 men and 461 women) with the diagnosis of pSS or suspected pSS, who had been checked for all four criteria of the JPN (pathology, oral, ocular, anti SS-A/Ro and SS-B/La antibody), and were followed up in June 2012 at 10 hospitals across Japan (Kanazawa Medical University Hospital, Nagasaki University Hospital, Hyogo Medical University Hospital, Keio University Hospital, Tokyo Women's Medical University Hospital, Tsurumi University Hospital, Kyushu University Hospital, University of Occupational and Environmental Health Hospital, Kyoto University Hospital, and University of Tsukuba Hospital), which form parts of the Research Team for Autoimmune Diseases, The Research Program for Intractable Disease of the Japan Ministry of Health, Labor and Welfare (MHLW).

Data collection and analysis

We collected clinical data through a questionnaire from the above 10 hospitals. We retrospectively examined the clinical diagnosis by the physician in charge, satisfaction of ACR-EULAR, JPN, AECG and ACR criteria. Because the OSS adopted in the ACR-EULAR and ACR criteria is not commonly performed in Japan, we regarded patients with Van Bijsterveld Score ≥4 in the Rose Bengal test, lissamine green test or fluorescein staining test to have satisfied OSS in the ACR-EULAR criteria, and patients who had positive Rose Bengal or lissamine green test (Van Bijsterveld Score ≥3) or fluorescein staining test to have satisfied OSS in the ACR criteria. Similarly, because numerous cases (116/499 cases) lacked results of the UWS, which was not adopted in the JPN criteria,³ we regarded patients who had UWS ≤0.1 mL/min, gum test ≤10 mL/10 min or Saxon test ≤2 g/2 min to have satisfied low salivary volume in the ACR-EULAR and AECG criteria.

Moreover, we performed the subanalysis using 383 cases who were examined for UWS, excluding 116 cases who lacked results of the UWS mentioned above. We examined satisfaction for each criteria set more strictly in this subanalysis than in whole analysis of 499 cases. For salivary volume, we regarded patients who had gum test ≤10 mL/10 min or Saxon test ≤2 g/2 min to have satisfied decreased salivary volume in JPN criteria, while UWS ≤0.1 mL/min in the ACR-EULAR and AECG criteria. For ocular staining, we regarded patients with Van Bijsterveld Score ≥3 in the Rose Bengal test, lissamine green test or fluorescein staining test, and/or positive fluorescein staining test to have satisfied positive ocular staining in JPN and ACR criteria, while Van Bijsterveld Score ≥4 in AECG and ACR-EULAR criteria.

We considered the clinical diagnosis by the physician in charge as the ‘gold standard’ for the diagnosis of pSS in this study. In all cases, the diagnosis established by the physician in charge was based on clinical findings, laboratory and serological tests of blood and saliva samples, sialography, scintigraphy, and histopathological examination of biopsy material. We regarded the clinical diagnosis by the physician in charge to be appropriate for the ‘gold standard’, because the clinical diagnosis was decided by senior and experienced clinicians belonging to 10 hospitals which form parts of the Research Team for Autoimmune Diseases, The Research Program for Intractable Disease of the Japan MHLW described above. We compared the sensitivity and specificity between the ACR-EULAR, JPN, AECG and ACR criteria in the diagnosis of pSS.

Statistical analysis

The differences of sensitivities and specificities between all possible pairs of the four sets of criteria were evaluated using the McNemar's test and the Newcombe's square-and-add method. A p value <0.05 denoted the presence of a statistically significant difference.

Results

Diagnosis of pSS and denial of pSS

None of the 499 patients had other well defined CTDs. pSS was diagnosed in 302 patients, whereas pSS was excluded in 197 patients by the physician in charge, and these judgements were considered as the ‘gold standard’ in the present study. For subanalysis using 383 cases that were examined for UWS, pSS was diagnosed in 203 patients, while pSS was excluded in 180 patients based on clinical judgements.

Sensitivity and specificity in diagnosis of pSS by different sets of criteria

For all the 499 patients, the sensitivities of the ACR-EULAR, JPN, AECG and ACR criteria in the diagnosis of pSS were 95.4% (95% CI 93.0% to 97.1%), 82.1% (79.6% to 84.1%), 89.4% (86.8% to 91.6%) and 79.1% (76.2% to 81.6%), respectively, considering the diagnosis by the physician in charge as the ‘gold standard’ (table 2). The respective specificities were 72.1% (68.4% to 74.7%), 90.9% (87.0% to 93.8%), 84.3% (80.2% to 87.6%) and 84.8% (80.3% to 88.5%) (table 2).

Table 2.

Sensitivity and specificity for the diagnosis of pSS by the four sets of criteria

Of the 499 enrolled patients, pSS was diagnosed in 302 patients, whereas pSS was excluded in 197 patients by the physician in charge. For subanalysis using 383 cases that were examined for unstimulated whole saliva (UWS), pSS was diagnosed in 203 patients, while pSS was excluded in 180 patients based on clinical judgements.

ACR-EULAR, American College of Rheumatology (ACR)-European League Against Rheumatism (EULAR) classification criteria for pSS; AECG, American-European Consensus Group classification criteria for SS; JPN, The revised Japanese Ministry of Health criteria for the diagnosis of SS; pSS, primary SS; SS, Sjögren's syndrome.

For subanalysis using 383 cases that were examined for UWS, the sensitivities of the ACR-EULAR, JPN, AECG and ACR criteria in the diagnosis of pSS were 94.1% (90.8% to 96.4%), 74.9% (71.3% to 77.6%), 85.7% (82.1% to 88.7%) and 79.8% (75.8% to 83.2%), respectively. The respective specificities were 76.7% (73.0% to 79.3%), 90.6% (86.5% to 93.7%), 86.1% (82.0% to 89.4%) and 81.1% (76.6% to 85.0%) (table 2).

The sensitivity of the ACR-EULAR criteria was statistically significantly higher than those of other three sets of criteria in both whole analysis and subanalysis (p<0.001) (table 3). The specificity of the ACR-EULAR criteria was statistically significantly lower than those of other three sets of criteria in whole analysis (p<0.001) (table 3). Although the specificity of the ACR-EULAR criteria was statistically significantly lower than those of the JPN and AECG criteria in subanalysis (p<0.001), the difference between the ACR-EULAR and ACR criteria was not statistically significant (p=0.117) (table 3).

Table 3.

Differences of the sensitivities and specificities (with 95% CI) for the diagnosis of pSS among the four sets of criteria

Of the 499 enrolled patients, pSS was diagnosed in 302 patients, whereas pSS was excluded in 197 patients by the physician in charge. For subanalysis using 383 cases that were examined for unstimulated whole saliva (UWS), pSS was diagnosed in 203 patients, while pSS was excluded in 180 patients based on clinical judgements.

The p values and confidence limits were computed by the McNemar's test and the Newcombe's square-and-add method.

–, not examined; ACR-EULAR, American College of Rheumatology (ACR)-European League Against Rheumatism (EULAR) classification criteria for pSS; AECG, American-European Consensus Group classification criteria for SS; JPN, The revised Japanese Ministry of Health criteria for the diagnosis of SS; pSS, primary SS; SS, Sjögren's syndrome.

These findings indicate that the ACR-EULAR criteria have higher sensitivity and lower specificity in the diagnosis of pSS, compared with the JPN, AECG and ACR in both whole analysis and subanalysis.

Agreement of ACR-EULAR criteria with the other three sets of criteria

Table 4 shows the satisfaction for these four sets of criteria in each case. The data showed that the ACR-EULAR criteria were satisfied by much more cases than the other three sets of criteria for both pSS (288 cases) and non-pSS (55 cases) groups. Many pSS (220/302 cases, 72.8%) satisfied all four sets of criteria, while many non-pSS (135/197 cases, 68.5%) satisfied none of four sets of criteria (table 4). Although 8 non-pSS cases satisfied all four sets of criteria, 11 patients with pSS did not satisfy any set of criteria (table 4). There was no case that satisfied all the other three sets of criteria except for the ACR-EULAR criteria among both pSS and non-pSS groups (table 4).

Table 4.

Satisfaction of the ACR-EULAR, JPN, AECG and ACR criteria in clinically diagnosed pSS and non-pSS cases

Of the 499 enrolled patients, pSS was diagnosed in 302 patients, whereas pSS was excluded in 197 patients by the physician in charge.

ACR-EULAR, American College of Rheumatology (ACR)-European League Against Rheumatism (EULAR) classification criteria for pSS; AECG, American-European Consensus Group classification criteria for SS; JPN, The revised Japanese Ministry of Health criteria for the diagnosis of SS; O, satisfaction; pSS, primary SS; SS, Sjögren's syndrome; X, non-satisfaction.

Importantly, 8 out of the 302 patients with pSS diagnosed by the physician in charge and 11 cases out of the 197 clinically non-pSS cases satisfied only the ACR-EULAR criteria, compared with none of the other three sets of criteria. These 19 cases explained the low agreement between the ACR-EULAR criteria and the other three sets of criteria. Further analysis of positivity for each item adopted in the ACR-EULAR criteria among the 8 patients with pSS who satisfied only the ACR-EULAR criteria indicated that they had positive FS (62.5%, 5/8 cases) or positive SS-A/Ro (37.5%, 3/8 cases), together with decreased salivary (87.5%, 7/8 cases) or lacrimal (12.5%, 1/8 cases) secretion, resulting in a total score of 4 in these 8 patients (figure 1A). The 11 non-pSS cases who satisfied only the ACR-EULAR criteria had positive FS (54.5%, 6/11 cases) or positive SS-A/Ro (45.5%, 5/11 cases), together with decreased salivary secretion (100%, 11/11 cases), resulting in a total score of 4 in these 11 cases (figure 1B).

Figure 1.

Satisfaction for each item adopted in the American College of Rheumatology (ACR)-European League Against Rheumatism (EULAR) classification criteria for pSS (ACR-EULAR) among 19 cases that satisfied only the ACR-EULAR criteria. (A) Eight patients with primary Sjögren's syndrome (pSS) who only satisfied the ACR-EULAR criteria had positive Focus Score (FS) (62.5%, 5/8 cases) or pSS-A/Ro (37.5%, 3/8 cases), together with decreased salivary (87.5%, 7/8 cases) or lacrimal (12.5%, 1/8 cases) secretion, resulting in Total Score 4. (B) Eleven non-pSS cases who only satisfied the ACR-EULAR criteria had positive FS (54.5%, 6/11 cases) or positive SS-A/Ro (45.5%, 5/11 cases), together with decreased salivary secretion (100%, 11/11 cases), resulting in Total Score 4. White boxes: non-satisfaction, shadowed boxes: satisfaction (dark shadow: 3 points, light shadow: 1 point). FS-LSG, FS ≥1 foci/4 mm² in labial salivary gland; SS-A/Ro, positive anti-SS-A/Ro antibody; Ocular staining, Van Bijsterveld Score ≥4 in Rose Bengal test, lissamine green test or fluorescein staining test; Schirmer, Schirmer's test ≤5 mm/5 min; Saliva, unstimulated whole saliva (UWS) ≤0.1 mL/min, gum test ≤10 mL/10 min or Saxon test ≤2 g/2 min.

Considered together, the above analyses suggest that judgement by the ACR-EULAR criteria for both diagnosis and exclusion of pSS was different from those by the JPN, AECG and ACR criteria (table 4).

Discussion

In the present study, we compared the sensitivity and specificity of the 2016 ACR-EULAR criteria for pSS with those of the JPN, AECG and ACR criteria, for the diagnosis of pSS using clinical data of 499 Japanese patients with pSS or suspected pSS. The results showed clearly that the 2016 ACR-EULAR criteria had higher sensitivity and lower specificity in the diagnosis of pSS, compared with the other three sets of criteria in both whole analysis (n=499) and subanalysis (n=383) using cases that were examined for UWS. Moreover, the degree of agreement of the ACR-EULAR criteria with the three sets of criteria for both diagnosis and exclusion of pSS was low. These results are different from those reported in a recent study by Shiboski et al,^{6 7} which showed high sensitivity (96%, 95% CI 92% to 98%) and specificity (95%, 95% CI 92% to 97%) for the ACR-EULAR criteria and high agreement rate with both AECG (κ coefficient: 0.91) and ACR criteria (κ coefficient: 0.82). These disagreements seem to be somewhat related to our 8 patients with pSS and 11 non-pSS cases who did not satisfy any of the JPN, AECG and ACR criteria, while they satisfied only the ACR-EULAR criteria. Among these 19 cases, 11 non-pSS cases had positive FS (6/11 cases) or positive anti-SS-A/Ro (5/11 cases), and decreased salivary volume (11/11 cases). These 11 non-pSS cases might cause the low specificity of the ACR-EULAR criteria. However, one has to pay enough attention to the adopted methods for assessment of salivary volume in this study. As mentioned in the Patients and methods section, we regarded patients with UWS ≤0.1 mL/min, gum test ≤10 mL/10 min or Saxon test ≤2 g/2 min to have satisfied decreased salivary volume in the ACR-EULAR and AECG criteria, because numerous cases (116/499 cases) lacked the results of UWS, which was not adopted in the JPN criteria.³ Importantly, 6 out of 11 non-pSS cases who satisfied only the ACR-EULAR criteria, were considered to have decreased salivary volume based on the gum and/or Saxon test instead of UWS (data not shown). These six cases might explain the low specificity of the ACR-EULAR criteria in the present study. Actually, in the subanalysis of 383 cases that were examined for UWS, the specificity (76.7%) of the ACR-EULAR criteria was higher than in the whole analysis (72.1%). On the other hand, eight patients with pSS who satisfied only the ACR-EULAR criteria but none of other three sets of criteria might cause the high sensitivity of the ACR-EULAR criteria. These eight patients with pSS had positive FS (5/8 cases) or positive anti-SS-A/Ro (3/8 cases), accompanied with decreased salivary (7/8 cases) or lacrimal (1/8 cases) volume. Collectively, these 19 cases (8 patients with pSS and 11 non-pSS cases) that satisfied only the ACR-EULAR criteria but none of other three sets of criteria seem to lead to the disagreement between the ACR-EULAR criteria and other three sets of criteria. All of these 19 cases had positive FS (11/19 cases) or positive anti-SS-A/Ro (8/19 cases) accompanied by decreased salivary (18/19 cases) or lacrimal (1/19 cases) volume. Thus these 19 patients, even including clinically judged 11 non-pSS cases, seem to have a high probability of suffering from ‘true pSS’. This means that the ACR-EULAR criteria might allow to correctly select patients with pSS who are misclassified by other criteria sets. Importantly, the highest sensitivity of the ACR-EULAR criteria might offer some advantages considering that more patients with true pSS could be selected for clinical and therapeutic trials. However, on the other hand, the lowest specificity of the ACR-EULAR criteria for Japanese patients with pSS was confirmed in both whole analysis and subanalysis using 383 cases that were examined for UWS, whereas the specificity was higher (76.7%) in subanalysis than in whole analysis (72.1%). A high specificity is the most critical aspect because this prevents subjects without pSS from entering clinical studies or trials. Therefore, if we apply the ACR-EULAR criteria to Japanese clinical studies targeted on pSS, we have to pay enough attention to this low specificity.

The 2016 ACR-EULAR criteria focused on pSS only but not on secondary SS, which is associated with other well defined CTDs.^{6 7} However, the International SS Criteria Working Group indicated that these criteria could also be applicable for secondary SS, and recommended further studies for secondary SS to confirm this.^{6 7} Thus, we also need to compare the performance of these four sets of criteria, including the ACR-EULAR, in the diagnosis of secondary SS because we targeted only patients with pSS or suspected pSS who were free of other CTDs.

The present study has certain limitations. First, we applied the diagnosis by the physician in charge as the ‘gold standard’. However, it is better to decide disease case or non-case status by expert clinical judgement based on clinical vignettes for the ‘gold standard’ diagnosis, which has been employed by newly developed classification criteria, such as the 2016 ACR-EULAR criteria for pSS^{6 7} and the proposed new classification criteria for systemic lupus erythematosus (SLE) by the Systemic Lupus International Collaborating Clinics.⁸ Moreover, since this study was entirely carried out in Japan, it might be predicted that the physicians who commonly used JPN criteria could be strongly influenced in their mind by this conceptual habit when they clinically defined cases as having pSS or not. Second, the methods adopted for ocular staining and salivary tests varied among the participating institutions, forcing us to modify certain items in some criteria as described in the Patients and methods section. For these reasons, we need a more sophisticated validation cohort study, using expert clinical judgement based on clinical vignettes as the ‘gold standard’ and integrated methodology for ocular staining and salivary measurement that fits completely with the items adopted in the criteria.

In addition to the classification criteria for clinical studies, we also need diagnostic criteria for daily clinical practice. The JPN criteria were established for the diagnosis of SS including both pSS and secondary SS, and are always used in daily clinical practice in Japan. Moreover, we reported previously that the JPN criteria had the highest sensitivity (79.6%) and specificity (90.4%) for all SS, including both primary and secondary SS, among the JPN, AECG and ACR criteria in Japan.⁵ However, the JPN criteria adopt sialography, scintigraphy, and gum and Saxon tests for assessment of salivary volume, which are neither adopted in the 2016 ACR-EULAR nor ACR criteria.^{3 4 6 7} Furthermore, the JPN criteria use a different cut-off value for ocular staining (Van Bijsterveld Score ≥3) from those used by the ACR-EULAR (Van Bijsterveld Score ≥4 or OSS ≥5), AECG (Van Bijsterveld Score ≥4) and ACR criteria (OSS ≥3).^{2–4 6 7} Thus, we need to examine the performance of the JPN criteria using alternative items that are adopted in the ACR-EULAR criteria, such as UWS for gum and Saxon tests, and Van Bijsterveld Score ≥4 or OSS ≥5 for Van Bijsterveld Score ≥3 by a prospective validation cohort study in the near future.

In conclusion, although this study has certain limitations, the results showed that the ACR-EULAR criteria have higher sensitivity and lower specificity in the diagnosis of pSS, compared with the JPN, AECG and ACR criteria. Furthermore, the degree of agreement of the ACR-EULAR criteria with the other three sets of criteria for the diagnosis and denial of pSS was low.