The diagnostic accuracies of the 2012 SLICC criteria and the proposed EULAR/ACR criteria for systemic lupus erythematosus classification are comparable

Abstract

In a joint effort, the European League Against Rheumatism (EULAR) and the American College of Rheumatology (ACR) recently proposed new criteria for the classification of systemic lupus erythematosus (SLE) with the overarching goal to identify potential participants for clinical studies. Herein, we present the first independent evaluation of these criteria in comparison with older classification grounds using an adult Scandinavian study population of confirmed SLE cases and individuals with SLE-mimicking conditions. We included 56 confirmed SLE cases meeting the 1982 ACR criteria (ACR-82) and/or the Fries “diagnostic principle” (antinuclear antibodies on at least one occasion plus involvement of at least two defined organ systems) and 55 controls with possible systemic autoimmune disease, including the presence of any SLE-related autoantibody. The proposed EULAR/ACR criteria showed a diagnostic sensitivity of 93% (95% confidence interval (CI), 0.83–0.98) compared with 83% (95% CI, 0.72–0.91) for the updated ACR criteria from 1997. The diagnostic accuracy of all tested classification grounds was fairly similar, achieving approximately 85%. However, the disease specificity of the EULAR/ACR criteria reached only 73% (95% CI, 0.59–0.83), which was comparable with the 2012 Systemic Lupus International Collaborating Clinics (SLICC) criteria, 75% (95% CI, 0.61–0.85), but clearly lower than for ACR-82, 94% (95% CI, 0.83–0.99). In this first independent evaluation of a limited number of cases, we found comparable results with respect to diagnostic sensitivity, specificity and accuracy regarding the SLICC-12 and the proposed EULAR/ACR classification criteria. However, their specificity for SLE appeared to be lower compared with ACR-82.

Introduction

The thorough heterogeneity among patients with systemic lupus erythematosus (SLE) causes problems regarding diagnostic accuracy in clinical practice, and particularly in clinical research. The recent attempts to create modern criteria for the classification of SLE have aimed at being more sensitive, clinically relevant and meaningful than the older American College of Rheumatology (ACR) classification criteria.¹

In 2012, the Systemic Lupus International Collaborating Clinics (SLICC) network presented a set of criteria based on almost 1400 patient scenarios, including a derivation and a validation of the new criteria as well as of the 1997 ACR classification.^2,3 Compared with the 1982 and 1997 ACR criteria, SLICC-12 contains additional clinical and immunological items and accepts cases with “renal lupus only” provided they are biopsy proven in combination with detectable antinuclear antibodies (ANA) and/or anti-double-stranded (ds)DNA antibodies.^2–4 In addition, SLICC-12 requires the presence both of clinical and immunological items. Several recent evaluations indicate that the SLICC-12 criteria have advantages regarding diagnostic sensitivity in comparison with other classification grounds, particularly in recent-onset disease.^5–9

In 2017, a committee appointed by the European League Against Rheumatism (EULAR) and ACR proposed new SLE classification criteria with the overarching goal to develop a system to identify potential participants for clinical studies.¹⁰ In the recently proposed criteria set presented at international meetings as well as in abstract, a positive level of ANA at a serum dilution of ≥1:80 is a required entry criterion, followed by weighted items including seven clinical (constitutional, mucocutaneous, arthritis, neurologic, serositis, hematological and renal) and three immunological (antiphospholipid, complement and highly specific antibody) domains.¹⁰ An accumulated score of 10 is considered as a cut-off for SLE classification.

In this brief report, we evaluated the performance of the proposed EULAR/ACR criteria in comparison with the Fries diagnostic principle (FDP), ACR-82, ACR-97 and SLICC-12, using two separate cohorts of patients with either potential or confirmed SLE.^2–4,11 FDP constitutes a clinical approach to diagnose SLE that is defined by the presence of ANA (at any time) above the local lab's cut-off (i.e. not a stipulated ANA titer as in the proposed EULAR/ACR criteria) on at least one occasion combined with the involvement of at least two defined organ systems: skin, joints, renal, lungs, serosa, nervous system and blood.¹¹

Materials and methods

Patients

In total, 111 cases were included. We previously performed an independent evaluation of SLICC-12 using one regional cohort of 243 patients with established SLE and one control cohort containing 55 individuals, referred to the Rheumatology Unit at Linköping University Hospital, with a fair suspicion of systemic autoimmune disease, including presence of ≥1 SLE-related autoantibody.⁶ Herein, the 55 patients who now have a follow-up time of ≥5 years were included in a control cohort.

Apart from the 55 control individuals with serology and symptoms compatible with SLE, we evaluated 56 new cases enrolled in our regional register KLURING (a Swedish acronym for Clinical Lupus Register in Northeastern Gothia); all had an SLE diagnosis confirmed by one single senior rheumatologist and fulfilled FDP and/or ACR-82.¹² These 56 confirmed SLE cases had entered the KLURING cohort from 2015 onward. Thus, none of them had been part of our former criteria evaluation.⁶ Characteristics of the 111 cases are detailed in Table 1.

Table 1.

Patient characteristics, including clinical manifestations and immunological findings of the included 111 cases

	Mean (range)
Characteristics	Control cohort (n = 55)	SLE cohort (n = 56)
Fulfilled ACR-82 criteria (n)	2.4 (1–3)	4.7 (3–8)
Age (years)	53.5 (23–88)	49.7 (22–81)
Female gender (n and %)	48 (87.3)	44 (78.6)
Clinical manifestations (n and %)
Acute cutaneous lupus	18 (32.7)	15 (26.8)
Chronic cutaneous lupus	2 (3.6)	11 (19.6)
Photosensitivity	17 (30.9)	25 (44.6)
Nonscarring alopecia	6 (10.9)	2 (3.6)
Oral ulcers	2 (3.6)	9 (16.1)
Arthritis	21 (38.2)	43 (76.8)
Pleuritis	8 (14.5)	14 (25.0)
Pericarditis	5 (9.1)	7 (12.5)
Renal disorder	2 (3.6)	17 (30.4)
Biopsy-proven lupus nephritis	2 (3.6)	15 (26.8)
Neurologic disorder (ACR-82)	0	2 (3.6)
Neurologic disorder (SLICC-12)	2 (3.6)	3 (5.4)
Hemolytic anemia	1 (1.8)	2 (3.6)
Leukopenia	13 (23.6)	27 (48.2)
Lymphopenia	6 (10.9)	29 (51.8)
Thrombocytopenia	4 (7.3)	5 (8.9)
Unexplained fever > 38.3℃	2 (3.6)	2 (3.6)
Immunologic criteria (n and %)
ANA (immunofluorescence microscopy)	49 (89.1)	55 (98.2)
Anti-dsDNA (Crithidia luciliae test)	8 (14.5)	29 (51.8)
Anti-Smith (line-blot confirmed by radial immunodiffusion)	0	2 (3.6)
Lupus anticoagulant (dilute Russell viper venom time)	19 (34.5)	10 (17.9)
Anticardiolipin; IgG, IgA, IgM (fluoroenzyme-immunoassay)	14 (25.5)	6 (10.7)
Anti-β2-glycoprotein-I; IgG, IgA, IgM (fluoroenzyme-immunoassay)	11 (27.5)a	9 (16.1)
Low complement; C3, C4 (nephelometry)	8 (14.5)	30 (53.6)
Direct Coombs test (hemolysis in gel)	8 (18.2)b	25 (50.0)c

ACR: American College of Rheumatology; ANA: antinuclear antibodies; Anti-dsDNA: anti-double-stranded DNA; Ig: immunoglobulin; SLE: systemic lupus erythematosus; SLICC: Systemic Lupus International Collaborating Clinics.

Tested on any occasion in:

^aForty of 55 cases.

^bForty-four of 55 cases.

^cFifty of 56 cases.

Oral and written informed consent was obtained from all participants. The research protocol was approved by the regional ethics review board in Linköping, Sweden (decision no. M75-08/2008).

Statistics

Classification grounds of participants based on FDP, ACR-82, ACR-97, SLICC-12 and proposed EULAR/ACR criteria were examined with analyses of sensitivity (proportion SLE cases correctly classified), specificity (proportion of non-SLE cases correctly specified), accuracy (proportion of cases correctly classified), positive predictive value (PPV; proportion of SLE-classified cases that are true SLE cases) and negative predictive value (NPV; proportion of non–SLE-classified cases that are true non-SLE cases), including 95% confidence intervals (CIs). Differences between groups were calculated using Mann–Whitney U test, chi-squared or Fisher exact test, where appropriate.

Results

The two cohorts had a similar distribution of age and gender (Table 1). The controls eventually received the following diagnoses: primary Sjögren's syndrome (n = 12), undifferentiated connective tissue disease (n = 8), antiphospholipid syndrome (n = 7), rheumatoid arthritis (n = 4), SLE (n = 4), fibromyalgia (n = 4), arthralgia (n = 3), psoriatic arthritis (n = 2), unspecified arthritis (n = 2), adult-onset Still disease (n = 1), polymyositis (n = 1), systemic sclerosis with primary biliary cirrhosis (n = 1), mixed connective tissue disease (n = 1), pyogenic arthritis, pyoderma gangrenosum and acne syndrome (n = 1), renal infarction (n = 1), multiple sclerosis (n = 1), palindromic rheumatism (n = 1) and recurrent pleuritis (n = 1).

Among clinical manifestations, arthritis (p < 0.0001), renal disorder (p = 0.0002) and lymphopenia (p = 0.001) were more common in confirmed SLE cases compared with controls without SLE (n = 51). Regarding immunologic criteria, anti-dsDNA detected by Crithidia luciliae immunofluorescence test (p < 0.0001), low complement (p < 0.0001) and a positive direct Coombs test (p = 0.002) were also more common in SLE cases than in control individuals. On the contrary, a positive lupus anticoagulant test was slightly more common among the controls (p = 0.05).

As indicated in Table 2, FDP, SLICC-12 and proposed EULAR/ACR criteria performed best with regard to diagnostic sensitivity and achieved results of at least 93%. However, ACR-82 achieved superior figures concerning specificity with 94%. The accuracy of all five classification grounds was fairly similar, with SLICC-12 reaching the numerically highest result of 88%.

Table 2.

Sensitivity, specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV), including 95% confidence intervals (in parentheses) given for each separate classification ground

	FDP	ACR-82	ACR-97	SLICC-12	Proposed EULAR/ACR
Sensitivity	0.95 (0.85–0.99)	0.80 (0.68–0.88)	0.83 (0.72–0.91)	1.0 (0.92–1.0)	0.93 (0.83–0.98)
Specificity	0.73 (0.59–0.83)	0.94 (0.83–0.99)	0.82 (0.69–0.91)	0.75 (0.61–0.85)	0.73 (0.59–0.83)
Accuracy	0.85 (0.77–0.90)	0.86 (0.79–0.92)	0.83 (0.75–0.89)	0.88 (0.81–0.93)	0.84 (0.76–0.90)
PPV	0.80 (0.69–0.88)	0.94 (0.83–0.99)	0.85 (0.73–0.92)	0.82 (0.72–0.90)	0.80 (0.69–0.88)
NPV	0.93 (0.79–0.99)	0.80 (0.68–0.88)	0.81 (0.68–0.90)	1.0 (0.88–1.0)	0.90 (0.76–0.97)

ACR: American College of Rheumatology; EULAR: European League Against Rheumatism; FDP: Fries' diagnostic principle; SLICC: Systemic Lupus International Collaborating Clinics.

The proposed EULAR/ACR weighted scores for all 111 included cases are demonstrated in Figure 1. The mean score of cases correctly classified as SLE were significantly higher compared with cases correctly classified as non-SLE (p < 0.0001), and cases correctly classified as SLE had significantly higher scores than cases incorrectly classified as SLE (p = 0.0005).

Figure 1.

Weighted scores according to the proposed EULAR/ACR criteria¹⁰ for each of the 111 cases with regard to their classification and final diagnosis.

ACR: American College of Rheumatology; EULAR: European League Against Rheumatism; SLE: systemic lupus erythematosus.

**p = 0.0005.

***p < 0.0001.

Discussion

Although SLE classification criteria are compiled for clinical research, many clinicians use them as a diagnostic compass. Thus, it is important that new SLE criteria sets perform well with regard to diagnostic sensitivity and specificity.¹³ The proposed set of criteria from EULAR and ACR has been presented at international meetings (EULAR and ACR) as well as in abstract.¹⁰ To our best knowledge, we here report the first independent real-life evaluation of the proposed criteria for classification of SLE from EULAR and ACR. Although the proposed criteria utilize a weighting system in which only the highest weighted item in each domain should be counted, we did not detect any significant differences between the SLICC-12 and the proposed EULAR/ACR criteria with respect to diagnostic sensitivity, specificity or accuracy. Their specificity, however, appeared to be lower compared with ACR-82. The latter may have been biased by the inclusion criteria of the SLE cohort. It is conceivable that high sensitivity combined with high specificity figures are not possible to achieve simultaneously with the use of one set of criteria; it may require a combination of criteria sets.^6,12 The given situation can also be decisive for whether sensitivity or specificity is of the highest importance. We do admit, however, that the number of included cases (n = 111) was fairly low and the lack of statistical power to detect small differences in performance between different classification grounds impeded firm conclusions. In addition, the limited number of included cases prevented us from evaluating the performance of the proposed EULAR/ACR criteria in subgroups comparing different numerical score intervals or by using other cut-offs than 10. On the contrary, cases with confirmed SLE and cases with lupus-imitating conditions were enrolled from one single university unit using the same accredited laboratory for all immunologic criteria. This constitutes an obvious strength since both SLICC-12 and the proposed EULAR/ACR criteria include additional laboratory items (complement and direct Coombs test, which were not included in older criteria sets), highlighting the need for reliable methods and similar antibody assays when fusing different SLE cohorts to increase statistical power.^2,10,14 In addition, our study population contained individuals with a reasonable suspicion of systemic autoimmune disease (including ≥ 1 SLE-related autoantibody) referred to a rheumatology specialist combined with confirmed SLE cases, which challenges the new criteria in a way resembling everyday clinical practice.

Based on our results, SLICC-12 and prosed EULAR/ACR criteria performed equally well with respect to sensitivity, specificity or accuracy. However, we would like to sound a note of warning concerning the choice of using a fixed ANA titer of ≥1:80 as an entry criterion.¹⁰ Cut-off titers for immunofluorescence (IF) ANA should be based on the 95th percentile among healthy blood donors. Tan et al. stated that an “abnormal titer of ANA” by IF-microscopy (or an equivalent assay) is required to qualify as a criterion according to ACR-82.⁴ However, a serum dilution (titer) corresponding to the 95th percentile among healthy referents differs among laboratories, depending on a number of variables, e.g. the microscope equipment, the antigen-source/fluorochrome density/antigen-specificity/dilution of the secondary antibodies, and on the subjective evaluation at ocular inspection under the microscope.¹⁵ Thus, each laboratory has to define its own cut-off titer for IF-ANA and, consequently, cut-off titers will vary from lab to lab, making it impossible to use the same cut-off worldwide. For instance, a cut-off for IF-ANA of 1:80 at our accredited laboratory in Linköping would have classified 14% of female and 4% of male blood donors as ANA positive.¹⁵ Based on this notion, we claim that the 95th percentile in a healthy blood donor population should form the basis for the cut-off limit of a positive ANA test. This is practiced in all Swedish medical laboratories performing routine ANA analyses, and is required to become an accredited “ANA laboratory”.

To conclude, the recently proposed SLE classification criteria from EULAR and ACR and the 2012 SLICC criteria showed comparable results regarding diagnostic sensitivity, specificity and accuracy using a Scandinavian study population of confirmed SLE cases and individuals with lupus-mimicking conditions.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by grants from the Swedish Rheumatism Association, the County Council of Östergötland, the Swedish Society of Medicine, the King Gustaf V’s 80-Year Anniversary Foundation and the King Gustaf V and Queen Victoria’s Freemasons Foundation.