Abstract
Objectives
To develop widely acceptable preliminary criteria of global flare for childhood-onset SLE (cSLE).
Methods
Pediatric rheumatologists (n=138) rated a total of 358 unique patient profiles (PP) with information about the cSLE flare descriptors (cSLE-FD) from two consecutive visits: patient global assessment of well-being, physician global assessment of disease activity (MD-global), health-related quality of life, anti-dsDNA antibodies, disease activity index score, protein/creatinine (P/C) ratio, complement levels and ESR. Based on 2996 rater responses about the course of cSLE (baseline vs. follow-up) the accuracy (sensitivity, specificity, area under the receiver operating characteristic curve) of candidate flare criteria was assessed. An international consensus conference was held to rank these candidate flare criteria as per the ACR-recommendations for the development and validation of criteria sets.
Results
The highest ranked candidate criteria considered absolute changes (Δ) of the SLEDAI or BILAG, MD-global, P/C ratio, and ESR; Flare scores can be calculated [0.5 × ΔSLEDAI + 0.45 × ΔP/C ratio + 0.5 × ΔMD-global + 0.02 × ΔESR], where values ≥ 1.04 are reflective of a flare. Similarly, BILAG-based flare scores [0.4 × ΔBILAG + 0.65 × ΔP/C ratio + 0.5 × ΔMD-global + 0.02 × ΔESR] of ≥ 1.15 were diagnostic of a flare. Flare scores increase with flare severity.
Conclusions
Consensus has been reached on preliminary criteria for global flares in cSLE. Further validation studies are needed to confirm the usefulness of the cSLE flare criteria in research and for clinical care.
Keywords: lupus, childhood-onset SLE, SLE, pediatric SLE, juvenile SLE, flare, criteria, children, cSLE
INTRODUCTION
Systemic lupus erythematosus is a complex, chronic multi-system autoimmune inflammatory disease that primarily targets young women of non-Caucasian ancestry (1–2). Up to 20% of patients are diagnosed during childhood, i.e. prior to age 16 years (cSLE), and their disease has a less favorable prognosis, particularly with respect to multi-organ and kidney involvement, when onset occurs early in life (3–5). The course of cSLE is characterized by episodes of clinically relevant worsening or disease flares; followed by periods of improvement which are generally the result of more intensive drug therapy. A flare of cSLE has been defined as “a measurable worsening of SLE disease activity in at least one organ system, involving new or worse signs of disease that may be accompanied by new or worse SLE symptoms; depending on the severity of the flare, more intensive therapy may be required” (6). However, at present, there are no generally accepted criteria or algorithms to determine whether a patient has experienced a flare of global disease in cSLE.
In an earlier phase of this project, an international consensus was reached about a set of cSLE flare descriptors (cSLE-FD) for identifying flares in cSLE patients. Previous research demonstrated that the scores of a disease activity measure alone are inadequate for identifying flares (7). Moreover, there was consensus around the need to discriminate three levels of flare severity: mild or minor, moderate, and major or severe flares (6).
The objectives of this phase of the project were to apply consensus formation methodology to develop preliminary criteria of global flare of cSLE under consideration of the cSLE-FD (a) using patient profile (PP) ratings that were completed by an international group of pediatric rheumatologists; and (b) ranking these candidate flare criteria under consideration of the American College of Rheumatology (ACR) suggested recommendations for development and validation of criteria sets.
PATIENTS AND METHODS
The overall approach to this phase of the project (Figure 1) was based on the methodological framework successfully employed in pediatric rheumatology in the past and as has been detailed by the Classification & Response Criteria Subcommittee of the ACR Committee on Quality Measures (8). The initial results of the consensus formation with respect to the domains and parameters to be considered in future cSLE flare criteria are described elsewhere (6). Briefly, pediatric rheumatologists who were members of the Childhood Arthritis & Rheumatology Research Alliance, the Pediatric Rheumatology European Society Juvenile Lupus Working Group, the Pan-American League of Arthritis & Rheumatology, or the ACR were invited to answer Delphi surveys. Subsequently, responses to two Delphi surveys resulted in consensus around a common definition of cSLE global flares, the cSLE-FD, followed by a data-driven exploration of candidate flare criteria (6). As opposed to previous criteria for flare in other pediatric rheumatic diseases, the latter suggested that uniform percentage changes are unlikely sufficient to capture cSLE flares with high sensitivity, and that other statistical techniques may yield cSLE flare criteria with higher accuracy.
Figure 1.
Study Design
We now present the phase of the project aimed at the development of preliminary criteria of global flare of cSLE. This encompassed patient profile (PP) ratings by pediatric rheumatologists from Australia, Africa, Asia, Europe and the Americas [Step 1]. The interpretation of the `true' disease course of a given PP was done using two approaches, which resulted in two distinct datasets for the subsequent validation exercises [Step 2]. Various candidate flare algorithms were generated and their ability to discriminate patients with flare was tested using the PP ratings [Step 3]; subsequently, during a consensus conference (CC), these candidate flare criteria were ranked under consideration of information from the medical literature, statistical performance, as well as reliability, feasibility, and face validity as per the ACR guidance document and the OMERACT filter (8) [Step 4].
Step 1: Patient Profiles & Ratings of Disease Course of a Patient Profile
Six of the authors conducted a pilot study to test the format of the PP. Built on this pilot study, we generated 358 unique PP, using prospectively collected cohort data of patients with cSLE (6, 9–11). Information for 137 PP were used previously to explore various statistical methods that might be utilized when developing cSLE flare criteria (6).
Patients whose disease course is reflected in the PP had a history of constitutional symptoms (89%), cSLE features pertaining to the mucocutaneous (90%), musculoskeletal (86%), hematologic (86%), renal (69%), neurologic (35%), vascular (31%), cardiac (29%), and gastrointestinal (21%) systems. Thus the relative frequency of organ involvement was comparable to that reported in the literature (12).
Data selected for the PP included all available visit pairs (baseline – follow-up) of cSLE patients considered to have had a flare as per the treating physician. Using a random-number approach, we selected 50 visit pairs representing “stable disease” as per the treating pediatric rheumatologist.
Each PP provided data about a cSLE patient at the time of a baseline visit and a follow-up visit approximately 3 to 6 months later. For each PP visit, the cSLE-FD were provided (6): [1] physician assessment of overall disease activity as measured on a visual analog scale (VAS) with a range from 0 to 10 (MD-global; 0 = inactive disease; 10 = very active disease); [2] parent assessment of patient overall well-being as measured on a VAS with a range from 0 to 10 (0 = very poor; 10 = very well); [3] health-related quality of life as measured by the Child Health Questionnaire physical summary score (CHQ-PHS); [4] proteinuria as measured by timed urine collection or spot protein/creatinine ratio (P/C ratio); [5] ESR; [6] levels of the complements C3 and C4; [7] summary score of a validated disease activity index; here: SLEDAI (13) and the BILAG (14–15); and [9] levels of anti-dsDNA antibodies, where changes between visits were categorized as follows: abnormal/newly normal; normal/normal; abnormal/abnormal, and normal/newly abnormal. Consensus was reached (Delphi surveys) that medication use should not be considered as a variable in future criteria sets of cSLE flare. Details on the format of the PP are provided in Appendix 1.
Disease Course
PP raters were randomly assigned to assess the disease course of a maximum of 40 PP. The disease course between visits was categorized by the PP raters as follows: major flare; moderate flare; minor flare; unchanged; or improved. Thus, a global flare was considered as “present” whenever the disease course was rated as minor, moderate, or major flare. The PP raters also provided feedback about which of the information provided was most important for their disease course assignment.
Step 2: Adjudication of Disease Course of the PP
A randomization scheme was pre-planned to ensure that each PP was sent to about 10 raters, with the ratio of American and international raters matching that of the PP raters' pool (about 1:1). PP with fewer than 4 ratings were regarded as “invalid” or “unqualified” and excluded from further consideration.
Adjudication of the (true) disease course
Only “qualified” PP with successful adjudication were considered in Step 3. Given that PP raters may not necessarily agree on the disease course, the “true” overall course of cSLE for a given PP was adjudicated using two approaches; (a) 67%-Rule: at least 2/3 of the raters agreed on a given disease course, (b) Majority-Rule: the majority of the raters of a PP agreed on a given disease course.
Step 3: Generation of Candidate Flare Criteria & Assessment of Performance
Three principal strategies were employed to develop a series of candidate flare algorithms: (a) consideration of relative changes of core disease descriptors, a method utilized in other response criteria used in pediatric rheumatology. Thus we generated candidate flare criteria, using uniform percentage changes of the cSLE-FD between 20 – 70%, in 5% increments. Furthermore, we developed algorithms that considered absolute baseline-to-follow-up changes of the cSLE-FD using two statistical techniques: (b) multinomial logistic regression, which yields a numeric “flare score” (or log odds of flare) calculated from the changes of several or all cSLE-FD predictors; (c) Classification Tree Analysis (CART) models which use Boolean rules to identify global flares of cSLE (16) and also features a “CART-score”.
As was deemed important based on previous consensus, both strategies (b) and (c) also allow for the estimation of flare severity (minor vs. moderate vs. major) using the “flare score” or “CART-score”, respectively (6).
Statistical analysis in preparation of the testing of candidate flare criteria
Using the 67%-Rule and the Majority-Rule dataset respectively, each cSLE-FD was assessed for its association with another cSLE-FD by multiple logistic regression analysis. Given the intended widespread use of the cSLE flare criteria, we tested whether there were systematic differences in the ratings provided by raters (a) from different geographic regions, or (b) with varying professional experience as measured by the duration of medical practice. Agreement among raters was assessed using intra-class correlation coefficients (ICC) and/or Kappa (κ) statistics. An ICC or a κ value can be interpreted as follows: poor agreement: ICC or κ < 0.4; fair to good agreement: ICC or κ ≥ 0.4 – 0.75; substantial to excellent agreement: ICC or κ ≥ 0.75 (17).
We also examined the cSLE-FD for internal redundancy using partial Pearson correlation coefficients (rp), which allow for the pair-wise assessment of changes of the cSLE-FD, while adjusting for the remaining cSLE-FD. High (rp ≥ 0.6) or very high (rp ≥ 0.8) values support redundancy or indicate that algorithms containing both cSLE-FD predictors could have the potential problem of collinearity which may cause unstable estimates, i.e. whether the patient truly has experienced a flare or not.
Performance & Accuracy
Each candidate flare criterion was assessed for diagnostic accuracy using receiver's operating characteristic (ROC) curve analysis. Specifically, the area under the ROC curve (AUC) was calculated, and the diagnostic accuracy was considered outstanding, excellent, good, fair, and poor if AUC was in the range of 0.9 – 1.0, 0.81 – 0.90, 0.71 – 0.80, 0.61 – 0.70, and < 0.60, respectively (18).
Each candidate flare algorithm derived by multinomial logistic regression consisted of several cSLE-FD predictors, and provides multiple “flare scores” (or log odds of flare). Considering all possible flare scores, the overall diagnostic accuracy of an algorithm can be estimated by means of the AUC.
For each algorithm, a “flare score threshold” was defined for clinical use and for comparison of the statistical performance of the pool of candidate flare algorithms. When using a certain flare algorithm, the assignment of a patient's disease course (here: major/moderate/minor flare vs. no flare) can be made by comparing the patient's “flare score” to the “flare score threshold”. CC participants concurred that the “flare score threshold” for a given algorithm should reflect the highest conditional AUC among all candidate thresholds on a ROC curve. The performance of the algorithm under this “flare score threshold” was then judged by its sensitivity and specificity.
Similar to algorithms derived by multinomial logistic regression, CART-based criteria yield `CART-scores' that can be used to decide on the presence of a flare, including its severity. Different from disease course criteria derived by multinomial regression, CART-based flare algorithms result in a single discrete value for sensitivity and specificity, respectively.
Step 4: Ranking of Candidate Flare Criteria
To support decision making when ranking the candidate flare criteria in terms of feasibility, reliability, and validity, CC participants reviewed a syllabus that provided the results of the preceding Delphi surveys (6) as well as relevant published medical literature. Additionally, the results of the statistical analyses (see Step 3) were available. CC participants were 11 attending experienced pediatric rheumatologists from South America, North America, and Europe with substantial clinical and research experience in cSLE (HB, CP, MB, ARe, DL, LT, AE, ARa, LS, CS-M, MP). A priori, the consensus level was set at 75%, i.e. comparable or even somewhat higher than that chosen for similar studies in cSLE and SLE in the past (15–19). Using nominal group technique guided by an experienced moderator (EHG), the expert panel assessed each of the candidate flare algorithms according to [1] feasibility, i.e. practicability: can the items be measured easily?; [2] reliability, i.e. reproducibility: can the items be measured precisely?; [3] redundancy: are there two or more items included in the candidate criteria measuring the same aspect of the disease?); [4] face validity, i.e. credibility (Are the criteria sensible?; [5] content validity, i.e. comprehensiveness: do the criteria sample all of the domains of the disease?; [6] criterion validity: do the criteria accurately (AUC) approximate the “gold standard”, i.e. the adjudicated disease course as per 67%-Rule or Majority-Rule?; [7] sensitivity and specificity: do the criteria effectively identify patients with cSLE flares and distinguish them from patients who do not have a flare of their cSLE?; and [8] discriminant validity: do the criteria detect the smallest clinically important change? (here: discriminate patients with minor flares from those with stable disease course). Based on the above considerations, the CC participants were asked to rank the candidate flare criteria from 1 (lowest) to 5 (highest validity).
The survey source data was batch-processed, and open source online survey software, Limesurvey, was used for responses management and as a presentation layer (see http://www.limesurvey.org/). All analyses were done using SAS 9.2 (SAS, Cary, NC) software and SYSTAT 12 (Systat Software, Inc, Chicago, IL) software. P-values < 0.05 were considered statistically significant.
Ethics Review
The study was approved by the institutional review boards of the participating pediatric rheumatology centers. Informed consent was obtained from all parents and, as appropriate, assent was given by the participants prior to the study procedures.
RESULTS
A total of 2,996 ratings were provided by 96 pediatric rheumatologists and used for Step 2. The response-rate of the pediatric rheumatologists to the PP was 70% (47% from the U.S. and Canada; 53% from Australia, Africa, South and Middle America, Asia and Europe). Among the total of 358 PP, 352 PP (98%) were rated by at least 4 raters, hence considered “qualified” for inclusion in Step 3. There were no significant differences of distribution of flares between qualified and unqualified PP (Fisher's exact test, p=0.62).
When the Majority-Rule was applied to the 352 “qualified” PP, there were 156 PP representing global flares (103 minor flares, 44 moderate flares and nine major flares) and 196 PP without cSLE flare (stable or improved cSLE). A total of 231 PP (66% of 352 PP) fit the criteria of the 67%-Rule; among them, 63 representing with flare (44 minor flares, 15 moderate flares and 4 major flares) and 168 PP without cSLE flare.
PP raters from different geographic locations did not differ systematically in the disease course assignment for a given PP (North America vs. other countries: ICC = 0.88). Similarly, PP raters with different duration of medical experience agreed very well with respect to the interpretation of the disease course (ICC = 0.9).
The cSLE-FD most commonly cited by the PP raters as important for the assignment of the (overall) disease course included the MD-global, the summary scores of the SLEDAI or BILAG, and the P/C ratio (Table 1). The absolute baseline-to-follow-up changes of the cSLE-FD per disease course are provided in Table 2. Irrespective of the dataset (67%-Rule; Majority-Rule), the MD-global, and the scores of the BILAG and SLEDAI, changed systematically and often significantly with cSLE global flares and their severity.
Table 1.
Most important descriptors to determine the presence of a global cSLE flare
| Descriptor | Percentage of PP where descriptor was named among the most important data provided to decide about disease course |
|---|---|
| Physician global assessment of disease activity (visual analog scale; 0=inactive disease; range: 0 –10) | 48% |
| SLEDAI summary score (range : 0 –105) | 48% |
| BILAG summary score (range: 0 – 72)† | 28% |
| Urine protein/creatinine ratio‡ | 25% |
| Anti-dsDNA antibodies | 21% |
| Complement C3 | 19% |
| Erythrocyte sedimentation rate [mm/hour] | 15% |
| Complement C4 | 13% |
| Patient global assessment of well-being (visual analog scale; 0= very poor; range: 0 –10) | 12% |
| BILAG renal domain score (range: 0 = 9]† | 8% |
| Absolute lymphocyte count | 7% |
| Health-related quality of life (Child Health Questionnaire physical summary score; CHQ-PHS) | 6% |
| European Consensus Lupus Assessment Measure (ECLAM ;range: 0–10) | 6% |
| Urine Dipstick of Protein | 6% |
A= 9; B= 3; C= 1; D or E = 0;
random spot urine sample [urine protein [mg/dL] / urine creatinine [mg/dL]
Table 2.
Change of Descriptors in Relationship to cSLE Disease Course†
| PP raters assessment Of disease course | N | Physician global assessment of disease | Patient global assessment of well-being | Urine protein/creatinine ratio | SLEDAI | BILAG | CHQ-PhS‡ | Complement C3 | Complement C4 | ESR |
|---|---|---|---|---|---|---|---|---|---|---|
| 66.7% Rule | ||||||||||
| No change/ improved | 168 | − 0.02 (0.11) | 0.04 (0.15) | − 0.16 (0.11) | − 0.36 (0.24) | − 0.82 (0.26) | 0.05 (0.87) | − 1.47 (1.67) | − 0.72 (0.76) | − 1.24 (1.24) |
| Any global cSLE flare | 63 | 1.49 (0.18) ** | − 0.48 (0.25) | 0.57 (0.18) ** | 4.00 (0.41) ** | 4.29 (0.44) ** | 1.28 (1.50) | − 9.28 (2.74) ** | − 2.05 (1.26) | 6.28 (1.96) ** |
|
| ||||||||||
| Majority Rule | ||||||||||
| No change/ improved | 196 | 0.11 (0.11) | − 0.02 (0.15) | − 0.11 (0.10) | − 0.23 (0.26) | − 0.73 (0.32) | − 0.51 (0.81) | − 0.53 (1.69) | 0.63 (1.84) | − 0.29 (1.44) |
| Any global cSLE flare * | 156 | 1.60 (0.13) ** | − 0.47 (0.17) ** | 0.59 (0.12) | 3.81 (0.31) ** | 4.14 (0.37) ** | 0.07 (0.98) | − 9.26 (1.92) ** | − 0.72 (2.11) | 7.83 (1.58) ** |
Indicates that the mean is different from that of “No change/improved” with a p<0.05;
Indicates that the mean in the rows of “Minor/mild flare”, “Moderate flare” and “Major/severe flare” indicates the mean is different from that of “No change/improved”, “Minor/mild flare”, and “Moderate flare” respectively with a p<0.05.
Values presented are changes in means (standard deviation)
Child Health Questionnaire, Physical Function Summary Score
The only cSLE-FD with moderate to high correlation to each other were the patient assessments of well-being and the CHQ-PHS, suggesting that these variables are complementary in the setting of cSLE flare measurement (Table 3).
Table 3.
Relationship of the changes in the cSLE Flare Descriptors
| Partial correlation coefficient† | Patient global assessment of well-being | Urine protein/crea tinine ratio | SLEDAI | BILAG | CHQ-PhS‡ | Complement C3 | Complement C4 | ESR |
|---|---|---|---|---|---|---|---|---|
| Physician global assessment of disease | −0.10 | 0.07 | 0.27** | 0.32** | −0.04 | −0.13* | 0.19** | 0.23** |
| Patient global assessment of well-being | 0.00 | 0.03 | −0.02 | 0.59** | 0.05 | −0.05 | −0.04 | |
| Urine protein/creatinine ratio | 0.15* | −0.03 | 0.02 | 0.02 | 0.01 | 0.00 | ||
| SLEDAI | 0.25** | 0.02 | −0.18** | −0.08 | 0.11 | |||
| BILAG | 0.07 | −0.04 | −0.06 | 0.19** | ||||
| CHQ-PHS | −0.11 | 0.05 | 0.12 | |||||
| Complement C3 | 0.24** | 0.12 | ||||||
| Complement C4 | −0.03 |
Correlation of two CV's after adjusting for other CV's.
Indicate significance of partial correlation coefficient at p-value < 0.05.
Indicate significance of partial correlation coefficient at p-value < 0.01.
Child Health Questionnaire, Physical Function Summary Score
CC participants concurred that flare algorithms which allowed for the inclusion of either the BILAG or the SLEDAI-2K were most suitable for use in clinical practice and research (consensus 91%).
Delphi respondents and CC participants alike regarded complement levels and anti-dsDNA antibodies as important cSLE-FD. However, none of these variables importantly improved the accuracy (AUC), sensitivity, or specificity when considered in candidate flare algorithms.
Candidate Criteria considering percentage changes of the cSLE-FD
Candidate criteria based on relative (%) changes of all or some of the cSLE-FD were generated. Despite often high specificity (maximum: 94%), none of these algorithms' sensitivity exceeded 63%. The CC participants refuted the usefulness of these algorithms, given their overall poor accuracy as measured by the AUC (consensus 100%).
Candidate Criteria considering absolute changes of the cSLE-FD
Candidate flare algorithms derived by multinomial regression that showed superior statistical performance (AUC) are summarized in Table 4.
Table 4.
Candidate flare algorithms based on multinomial logistic regression with the best overall performance to identify patients with flare as measured by the area under the receiver operating characteristic curve (AUC)
| Candidate Criterion | Absolute change of flare descriptors considered | 67% Rule | Majority Rule | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| AUC | Sensitivity | Specificity | AUC | Sensitivity | Specificity | |||||
| 1 | SLEDAI | P/C ratio* | ESR | MD-global# | 0.90 | 80% | 89% | 0.87 | 74% | 85% |
| 2 | BILAG | P/C ratio | C3 | MD-global | 0.89 | 83% | 83% | 0.85 | 73% | 85% |
| 3 | BILAG | P/C ratio | ESR | 0.89 | 83% | 84% | 0.84 | 72% | 85% | |
| 4 | SLEDAI | ESR | MD-VAS | 0.89 | 80% | 86% | 0.84 | 70% | 85% | |
| 5 | BILAG | P/C ratio | ESR | MD-global | 0.89 | 75% | 90% | 0.86 | 77% | 85% |
| 6 | BILAG | P/C ratio | CHQ-PHS | MD-global | 0.89 | 75% | 91% | 0.85 | 76% | 85% |
| 7 | SLEDAI | P/C ratio | ESR | 0.89 | 75% | 95% | 0.83 | 68% | 85% | |
| 8 | BILAG | P/C ratio | CHQ-PHS | 0.88 | 84% | 86% | 0.83 | 71% | 85% | |
| 9 | BILAG | P/C ratio | C3 | 0.88 | 80% | 89% | 0.85 | 68% | 85% | |
| 10 | BILAG | P/C ratio | Patient global$ | MD-global | 0.88 | 80% | 81% | 0.85 | 71% | 85% |
| 11 | BILAG | P/C ratio | Anti-dsDNA antibodies | MD-global | 0.88 | 80% | 82% | 0.85 | 72% | 85% |
| 12 | BILAG | P/C ratio | Patient global | 0.88 | 80% | 87% | 0.85 | 71% | 85% | |
| 13 | BILAG | P/C ratio | Anti-dsDNA antibodies | 0.88 | 80% | 90% | 0.84 | 72% | 86% | |
| 14 | BILAG | P/C ratio | 0.88 | 80% | 88% | 0.83 | 73% | 84% | ||
| 15 | BILAG | P/C ratio | MD-global | 0.88 | 74% | 89% | 0.85 | 71% | 85% | |
| 16 | SLEDAI | ESR | 0.88 | 70% | 94% | 0.85 | 65% | 85% | ||
| 17 | SLEDAI | P/C ratio | Patient global | MD-global | 0.87 | 79% | 87% | 0.84 | 76% | 85% |
| 18 | SLEDAI | P/C ratio | Anti-dsDNA antibodies | MD-global | 0.87 | 79% | 85% | 0.83 | 74% | 85% |
| 19 | SLEDAI | P/C ratio | MD-global | 0.87 | 79% | 85% | 0.83 | 75% | 85% | |
| 20 | SLEDAI | P/C ratio | C3 | MD-global | 0.87 | 77% | 86% | 0.83 | 71% | 85% |
| 21 | BILAG | P/C ratio | C4 | MD-global | 0.87 | 71% | 88% | 0.84 | 71% | 85% |
P/C ratio: Urine protein/ creatinine ratio from random urine sample
Patient global assessment of well-being measured on a visual analog scale (range: 0–10; 0= inactive disease)
Physician global assessment of disease measured on a visual analog scale (range: 0–10; 0= inactive disease)
The highest ranked algorithms as per the CC experts, under consideration of content validation and feasibility, are shown in Table 5 (67%-Rule data). Of note, analysis of the Majority-Rule dataset yielded comparable thresholds.
Table 5.
Highest ranked Candidate Flare Criteria†
| Rank | Algorithms | Area under the Receiver Operating Characteristic Curve§ | Flare score threshold‡ |
|---|---|---|---|
| 1 $ | 0.5 × SLEDAI + 0.45 × P/C ratio* + 0.5 × MD-global#+ 0.02 × ESR | 0.90 | 1.04 |
| 2 $ | 0.4 × BILAG + 0.65 × P/C ratio + 0.5 × MD-global + 0.02 × ESR | 0.89 | 1.15 |
| 3$ | 0.4 × SLEDAI + 0.33 × P/C ratio + 0.6 × MD-global | 0.87 | 0.88 |
| 4$ | 0.4 × BILAG + 0.55 × P/C ratio + 0.5 × MD-global | 0.88 | 1.26 |
| 5 (CART) | 3 ≤ SLEDAI OR 2 ≤ MD-global OR 0.7 < P/C ratio | 0.89 | Not applicable |
| 6 (CART) | 2 ≤ BILAG OR 2 ≤ MD-global OR 0.7<P/C ratio | 0.88 | Not applicable |
Algorithm considers for the change (baseline – follow-up) of each of the flare descriptors included
Values presented represent the area under the ROC curve considering PP with consensus as defined by the 67%-Rule
P/C ratio: Urine protein/ creatinine ratio form random urine sample
MD-global: Physician global assessment of disease measured on a visual analog scale (range: 0–10; 0= inactive disease)
Numeric values larger than or equal to the flare score signify a flare; higher scores are seen with more severe flare.
Consensus was achieved that criteria based on CART-analysis are particularly useful for daily clinical care where any arithmetic manipulations may appear prohibitive due to limited time. CART-models with superior statistical performance (AUC) included changes of the MD-global, ESR, P/C ratio, and the BILAG or SLEDAI. CART-based candidate flare criteria that considered changes of the SLEDAI as disease activity measure had high sensitivity, specificity and accuracy (AUC) at 89%, 85%, and 0.89 (67%-Rule data); CART-based candidate flare criteria that included changes of the BILAG (instead of the SLEDAI) had similar sensitivity, specificity and accuracy at 87%, 82% and 0.88. When using the dataset derived by Majority-Rule comparable measurement properties were noted.
Severity of Flares
The logistic models yield scores that can be used to define flare severity. More pronounced worsening of cSLE can be quantified. For the highest ranking flare criterion (Rank 1, Table 5), flare scores > 2.7 and >3.2 are associated with moderate and severe flares, respectively. For the second ranked criterion (Rank 2, Table 5),flare thresholds for moderate and severe flares are at 3 and 3.5
DISCUSSION
A set of preliminary criteria to measure global flares of cSLE has been delineated, using consensus formation methodology. The highest ranking criteria allow the use of either the BILAG or the SLEDAI and are based on a flare score that can easily be determined. The need of developing internationally acceptable criteria for disease flares has become more urgent since the introduction of randomized withdrawal trials in pediatric rheumatology, where time to flare or the proportion of patients who experience a flare are used as primary efficacy measures (19). Universally accepted criteria for flare are clinically desirable since flares have been associated with poor prognosis of cSLE.
Because of their popularity in pediatric rheumatology, we re-examined whether algorithms considering uniform percentage changes of the cSLE-FD can be used to accurately recognize cSLE flares. However, this approach did not yield criteria with sufficient sensitivity, confirming our previous research (6). As has been suggested by our earlier studies (6), flare algorithms based on CART or regression models, both approaches account for the differential importance of changes in individual cSLE-FD for recognizing cSLE flares, proved most suitable from a statistical point of view.
cSLE flare algorithms derived by multinomial regression are reminiscent of the disease activity score (DAS) used in rheumatoid arthritis (20). However, the DAS score considers the natural logarithm of the ESR and square roots of the number of swollen or tender joints, while the preliminary cSLE flare criteria require at most simple arithmetic maneuvers to calculate a cSLE flare score, supporting their ease of use.
Given the simplicity of CART-based criteria, they appear particularly suited for clinical care but a potential short-coming of CART-based criteria is the so-called `over-fitting of the mathematical model' which can make them prone to less favorable statistical performance in subsequent validation studies. This is supported by our previous work where we employed CART-analysis to explore cSLE flare criteria based solely on statistical considerations and described a CART-based algorithm with different parameters (6).
Although a non-specific marker of inflammation, the ESR is included in the criteria set for cSLE flare. The ESR is considered in selected SLE disease activity indices (24), and some previous studies in adults support the association between ESR and disease flares and damage accrual in SLE (25), supporting the relevance of ESR changes in the setting of cSLE flare.
Other criteria for measuring flare have been proposed for use in adult SLE. We tested the SELENA Flare Tool, using the PP ratings and found their sensitivity for cSLE global flares to be < 50%. In contrast, the BILAG flare tool [major flare: ≥ 1 new A domain score or 2 new B domain scores; moderate flare: 1 new B domain score; mild flare: 1 new C score [previous domain score: D or E]) (21) appeared to be more useful (sensitivity: 78%; specificity: 81%; data not shown). However, CC participants and Delphi respondents alike rejected the solitary use of disease activity indices to measure cSLE flare.
We would like to stress the observation that PP raters from different parts of the world and those with different degrees of experience demonstrated excellent concordance (inter-rater agreement) in their assessment of the cSLE course, demonstrating the robustness of the preliminary criteria in different settings.
This study must be seen in the light of certain limitations. Our datasets contained a limited number of moderate flares or severe flares, making the estimation of flare severity less reliable. However, our principal goal was to develop preliminary criteria for cSLE global flares and only as a secondary goal we aimed at classifying their severity.
We chose two approaches to adjudicate the disease course (67%-Rule, Majority-Rule) presented in the various PP, which might have introduced bias. However, both approaches yielded comparable results. Additionally, we explored other selection criteria (50%-Rule, 75% Rule) and found no systematic differences [data not shown].
Recently, response criteria for SLE considered both the SLEDAI and the BILAG (22). In exploratory analyses, we found evidence that consideration of both indices might improve the sensitivity of diagnosing minor flare without improving the overall accuracy of the highest-ranked criteria [data not shown].
The ACR has outlined a series of validation steps necessary before new criteria are to be widely used for clinical care or research (8). Among others, one step is to use data from clinical trials for developing response criteria. However, clinical trial data from interventions that impact cSLE disease activity are unavailable at present. In our study, the presence of a flare was based on the PP raters' perception of the course of cSLE rather than using data from a clinical trial. Given its prospective character and the expertise of the PP raters (over 10-year pediatric rheumatology experience: 65%; average number of cSLE patients treated/ month: 15 ± 18), we consider the quality of our data to be as high as that collected for clinical trials.
Besides criteria for global flare, criteria that help determine clinically relevant worsening of cSLE in specific organ systems are needed. As is clearly stated by the ACR, a single study can never suffice to adequately examine the measurement properties of a response criteria set. To confirm the accuracy of the preliminary criteria of flare, we are planning additional validation studies using other data sets and other criterion standards, such as changes in medication requirements.
Acknowledgements
Other: CCHMC: Shannen Nelson (overall study coordination), Carmela Sagcal (consensus conference support), Joshua Pendl and Jamie Meyers-Eaton (site coordination and database management); CCHMC Biomedical Informatics (Web-based data management application development).
Texas Scottish Rite Hospital: Shirley Henry (site coordination).
University of Chicago Comer Children's Hospital: Becky Pupluva (site coordination).
Children's Memorial Hospital: Dina Blair (site coordination).
British Columbia Children's Hospital: America Uribe (consensus conference support)
Morgan Stanley Children's Hospital: Candido Batres (site coordination), Lisa Imundo and Andrew Eichenfield (data collection).
MetroHealth Medical Center and Case Western Reserve University: Elizabeth Brooks, Kabita Nanda (data collection).
External Scientific Advisory Committee: We are indebted to the members of the External Scientific Advisory Committee of this study for their advice in the study implementation, conduction and its statistical analysis: Drs. Carol Wallace, Susan Bowyer, Vern Farewell, Rosalind Ramsey-Goldman, Nicola Ruperto, Carlos Rose and James Witter.
Investigators (data collection): Adam Huber, IWK Health Centre and Dalhousie University, Halifax, Nova Scotia, Canada; Alan Rosenberg, Royal University Hosp., Saskatoon, Saskatchewan, Canada; Alberto Martini, University of Genoa, Genoa, Italy; Alejandra Pringe, Hosp. Pedro de Elizalde, Buenos Aires, Argentina; Ali Yalcindag, Rhode Island Hosp., Providence, RI, USA; Amita Aggarwal, SGPGI, Lucknow, India; Anders Fasth, Göteborg University, Göteborg, Sweden; Andrew Lasky, Children's Mercy Hosp., Kansas City, KS, USA; Andrew Zeft, University of Utah, Salt Lake City, UT, USA; Ann Reed, Mayo Clinic, Rochester, MN, USA; Anne Stevens, Seattle Children's Hosp., Seattle, WA, USA; AnneMarie Brescia, DuPont Hosp. for Children, Wilmington, DE, USA; Annet Van Royen, Wilhelmina Children's Hosp., Utrecht, Netherlands; Basil Fathalla, Detroit Medical Center, Detroit, MI, USA; Berit Flatø, RiksHosp.et University Hosp., Oslo, Norway; Blanca Elena Rios Gomes Bica, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil; Brigitte Bader-Meunier, Hôpital Robert Debré, Paris, France; Carmen Laura De Cunto, Hosp. Italiano, Buenos Aires, Argentina; Carol Wallace, Childrens Hosp. & Regional Medical Ctr, Seattle, WA, USA; Carolina Duarte Salazar, Instituto Nacional de Rehabilitación, Mexico City, Mexico; Cassia M. P L. Barbosa, UNIFESP, Sao Paulo, Brazil; Cecilia Coto Hermosilla, Hosp. Pedro Borras Astorga, Habana, Cuba; Christian Hümer, Abteilung fur Pädiatrie, Vorarlberg, Austria; Christina Boros, University of Adelaide, Adelaide, Australia; Claire LeBlanc, University of Alberta, Alberta, Canada; Claudio A. Len, UNIFESP, Sao Paulo, Brazil; Clovis Artur Almeida da Silva, University of Sao Paulo, Sao Paulo, Brazil; Consuelo Modesto, Hosp. Vall d'Hebron, Barcelona, Spain; Deborah McCurdy, Mattel Children, Los Angeles, CA, USA; Daniel Kingsbury, Legacy Emanuel Children's Hosp., Portland, OR, USA; Daniel Lovell, Children's Hosp. of Cincinnati, Cincinnati, OH, USA; David Cabral, BC Children's Hosp., Vancouver, BC, Canada; David Sherry, Children's Hosp. of Philadelphia, Philadelphia, PA, USA; Deborah Rothman, Shriner's Hosp. for Children, Springfield, MA, USA;Delfor Alberto Giacomone, Hosp. de Niños Sor María Ludovica, La Plata, Argentina; Diana Milojevic, The University of California, San Francisco, CA, USA; Dorothee Stichweh, Children's Medical Center, Dallas, Texas, USA; Dowain Wright, Children's Hosp. Central California, Madera, CA, USA; Dolores Teresa Cantera Oceguera, Hosp. Pedro Borras Astorga, Habana, Cuba; Edsel Arce, Children's Hosp. Central California, Madera, CA, USA; Elizabeth Chalom, Saint Barnabas Medical Center, Livingston, NJ, USA; Emilia Spangenberg, Sociedad Uruguaya de Reumatología, Montevideo, Uruguay; Esi Morgan DeWitt, Children's Hosp. of Cincinnati, Cincinnati, OH, USA; Eyal Muscal, Baylor College of Medicine, Houston, TX, USA; Frank Dressler, Medizinische Hochschule Hannover, Hannover, Germany; Gaelle Chedeville, Montreal Children's Hosp., Quebec, Canada;Gail Faller, Chris Hani Baragwanath Hosp., Johannesburg, South Africa; Gary Sterba, Hosp. J. M. de los Rios, Caracas, Venezuela; Gerd Horneff, Asklepios Klinik für Kinder, Sankt Augustin, Germany; Giovany Beltrán Avendaño, Bogotá, Columbia; Graciela Espada, Hosp. de Niños Ricardo Gutierrez, Buenos Aires, Argentina;Harry L Gewanter, Pediatric & Adolescent Health Cntr, Midlothian, VA, USA; Hartmut Michels, Mengel E. Rheumatic Children's Hosp., Garmisch-Partenkirchen, Germany; Hermann Girschick, University of Würzburg, Würzburg, Germany; Irama Maldonado, Centro Nacional de Enfermedades Reumáticas, Caracas, Venezuela;Ivan Foeldvari, Kinder- & Jugendrheumatologie, Hamburg, Germany; Jaime de Inocencio, Centro de Salud Jazmín, Madrid, Spain; Janis Dionne, BC Children's Hosp., Vancouver, BC, Canada; Jelena Vojinovic, University of Niš, Niš, Serbia; Jennifer Huggins, Children's Hosp. of Cincinnati, Cincinnati, OH, USA; Jennifer Weiss, Joseph M. Sanzari Children's Hosp., Hackensack, NJ, USA; Jenny Soep, The Children's Hosp., Aurora, CO, USA; Jim Jarvis, Oklahoma University Health Sciences Center, Oklahoma City, OK, USA; Jing-Long Huang, Chang Gung Children's Hosp., Taoyuang Hsien, Taiwan; Johannes Roth, University Hosp. Münster, Münster, Germany; Jordi Antón, Hosp. Sant. Joan de Déu, Barcelona, Spain; Jose Goldenberg, Hosp. Israelita, Sao Paulo, Brazil; Joyce Hsu, Lucile Packard Children's Hosp., Palo Alto, CA, USA; Judy Olson, Medical College of Wisconsin, Milwaukee, WI, USA; Juliana Sato, Istituto G. Gaslini, Genoa, Italy; Kathleen A. Haines , New York University, New York, NY, USA; Kathleen O'Neil, Children's Hosp. of Oklahoma, Oklahoma City, OK, USA; Kelly Rouster Stevens, Wake Forest University Baptist Medical Center, Winston-Salem, NC, USA; Ken Schikler, University of Louisville, Louisville, KY, USA; Isabelle Kone-Paut, Hôpital de Bicêtre, Le Kremlin Bicêtre, France; L. Nandini Moorthy, Robert Wood Johnson Medical School, New Brunswick, NJ, USA; Lawrence Jung, Creighton School of Medicine, Omaha, NE, USA; Lenore Buckley, Virginia Commonwealth University School of Medicine, Richmond, VA, USA; Leonard H Sigal, Robert Wood Johnson Medical Grp, New Brunswick, NJ, USA; Leslie Abramson, Vermont Children's Hosp., Burlington, VT, USA; Linda Wagner-Weiner, University of Chicago, Chicago, IL, USA; Liora Harel, Schneider Children's Medical Center, Petah Tikva, Israel; Lisa Rider, National Institutes of Health, Bethesda, MD, USA; Manuel Ferrandiz, Instituto de Salud del Niño, Lima, Peru; Mara Becker, The Children's Mercy Hosp., Kansas City, MO, USA; María Martha Katsicas, Hosp. de Pediatría “Prof. Dr. Juan P. Garrahan”, Buenos Aires, Argentina; Maria Odete Esteves Hilario, Universidade Federal de São Paulo, São Paulo, Brazil; Maria Teresa Apaz, Universidad Católica Clinica Reina Fabiola, Cordoba, Argentina; Maria Teresa Terreri, Universidade Federal de São Paulo, São Paulo, Brazil; Mario J. Moreno, Hurtado 202 y Machala Edificio Crespo 1er, Ceunca, Ecuador; Mathew Adams, Children's Hosp. of Michigan, Detroit, MI, USA; Matthew Stoll, Children's Medical Center - Dallas, Dallas, TX, USA; Mauricio Alegria Mendoza, Colonia Médica, San Salvadore, El Salvadore; Melissa Elder, University of Florida, Gainesville, FL, USA; Michael Henrickson, Children's Hosp. of Cincinnati, Cincinnati, OH, USA; Michael S. Borzy, Ohio State University, Columbus, OH, USA; Monica Patricia Velasquez Mendez, Universidad Nacional De Colombia, Bogotá, Columbia;Nico Wulffraat, UMC Utrecht, Utrecht, Netherlands; Nicola Ruperto, University of Genoa, Genoa, Italy; Norm Ilowite, Children's Hosp. at Montefiore, New York City, NY, USA; Patricia Woo, Great Ormond Street Children Hosp., London, England; Pavla Dolezalova, Charles University in Prague, Prague, Czech Republic; Peter Blier, Baystate Children's Hospital, Springfield, MA, USA; Peter Chira, Stanford University, Stanford, CN, USA; Pilar Guarnizo, Universidad del Rosario Bogotá, Bogotá, Columbia; Polly Ferguson, University of Iowa, Iowa City, IA, USA; Prieur Anne-Marie, Hôpital Necker-Enfants-Malades, Paris, France; Hans-Iko Huppertz, Hess-Kinderklinik, Bremen, Germany; Pierre Quartier, Hôpital Necker-Enfants Malades, Paris, France; Raju Khubchandani, Jaslok Hosp. & Research Center, Mumbai, India; Randy Cron, University of Alabama, Birmingham, AL, USA; Raphael Hirsch, Children's Hosp. Pittsburgh, Pittsburgh, PA, USA; Raúl Gutiérrez Suárez Hosp. General de México, Mexico City, Mexico; Ricardo Russo, Hosp. de Pediatría “Prof. Dr. Juan P. Garrahan”, Buenos Aires, Argentina; Richard Vehe, University of Minnesota, Minneapolis-Saint Paul, MN, USA; Richard Vesely, Detska Fakultna Nemocnica, Košice, Slovakia; Rik Joos, Jan Palfijn Ziekenhuis, Merksem, Belgium; Rita Jerath, Medical College of Georgia, Augusta, GA, USA; Riva Brik, Rambam Medical Center, Haifa, Israel; Rob Nickeson, University of South Florida, Tampa, FL, USA; Robert Sundel, Boston Children's Hosp., Boston, MA, USA; Roberto Carreño Manjarrez, Hosp. Infantil De Mexico, Mexico City, Mexico; Rolando Cimaz, AOU Meyer and University of Florence, Florence, Italy; Rosario Jurado, Sanitorio Americano, Montevideo, Uruguay;Rotraud K. Saurenmann, University Children's Hosp., Zurich, Switzerland; Ruben Burgos-Vargas, Hosp. General de México, Mexico City, Mexico; Ruben Cuttica, Hosp. General de Ninos Pedro de Elizalde, Buenos Aires, Argentina; Ruth Eraso, Universidad De Antioquia, Medellín, Colombia; Sheila Knupp Feitosa de Oliveira, Instituto de Puericultura e Pediatria Martagão Gesteira, Rio de Janeiro, Brazil; Shirley Tse, The Hosp. for Sick Children, Toronto, Ontario, Canada; Silvia Magni-Manzoni, Fondazione IRCCS Policlinico San Matteo, Pavia, Italy; Silvia Meiorin, Hosp. de Niños Ricardo Gutierrez, Buenos Aires, Argentina; Stacy Ardoin, Ohio State University, Columbus, OH, USA; Stefan Hagelberg, Karolinska University Hosp., Stockholm, Sweden; Stella M. Garay, Hosp. Sor Maria Ludovica, La Plata, Argentina; Susa Benseler, The Hosp. for Sick Children, Toronto, Ontario, Canada; Susan Nielsen, Juliane Marie Centret, RigsHosp.et, Copenhagen, Denmark; Tadej Avcin, University Medical Center, Ljubljana, Slovenia; Terry L. Moore, Saint Louis University School of Medicine, Saint Louis, MO, USA; Thomas Griffin, Children's Hosp. of Cincinnati, Cincinnati, OH, USA; Tim Beukelman, University of Alabama - Birmingham, Birmingham, AL, USA; Tracy Ting, Children's Hosp. of Cincinnati, Cincinnati, OH, USA; Witske Kuis, Wilhelmina Children's Hosp., Utrecht, Netherlands; Wineke Armbrust, University Medical Center Groningen, Groningen, Netherlands; Yosef Uziel, Sapir Medical Center, Kfar Saba, Israel; Yuki Kimura, Hackensack University Medical Center, Hackensack, NJ, USA.
Grant Support: The study is supported by NIH grants 5U01-AR51868, P30-AR AR47363, P60-AR047884 and 2UL1RR026314.
Dr. Mina is supported by a NIAMS training grant T32100291.
Dr. Levy is supported by a NIAMS grant K23AR053202.
Appendix 1
LITERATURE
- 1.Klein-Gitelman M, Reiff A, Silverman ED. Systemic lupus erythematosus in childhood. Rheum Dis Clin North Am. 2002;28(3):561–77. vi–vii. doi: 10.1016/s0889-857x(02)00015-7. [DOI] [PubMed] [Google Scholar]
- 2.Hochberg MC. Updating the American College of Rheumatology revised criteria for the classification of systemic lupus erythematosus. Arthritis Rheum. 1997;40(9):1725. doi: 10.1002/art.1780400928. [DOI] [PubMed] [Google Scholar]
- 3.Brunner HI, Gladman DD, Ibanez D, et al. Difference in disease features between childhood-onset and adult-onset systemic lupus erythematosus. Arthritis Rheum. 2008;58(2):556–62. doi: 10.1002/art.23204. [DOI] [PubMed] [Google Scholar]
- 4.Hiraki LT, Benseler SM, Tyrrell PN, et al. Clinical and laboratory characteristics and long-term outcome of pediatric systemic lupus erythematosus: a longitudinal study. J Pediatr. 2008;152(4):550–6. doi: 10.1016/j.jpeds.2007.09.019. [DOI] [PubMed] [Google Scholar]
- 5.Kasitanon N, Magder LS, Petri M. Predictors of survival in systemic lupus erythematosus. Medicine (Baltimore) 2006;85(3):147–56. doi: 10.1097/01.md.0000224709.70133.f7. [DOI] [PubMed] [Google Scholar]
- 6.Brunner HI, Klein-Gitelman MS, Higgins GC, et al. Toward the development of criteria for global flares in juvenile systemic lupus erythematosus. Arthritis Care Res (Hoboken) 2010;62(6):811–20. doi: 10.1002/acr.20126. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Brunner HI, Higgins GC, Klein-Gitelman MS, et al. Minimal clinically important differences of disease activity indices in childhood-onset systemic lupus erythematosus. Arthritis Care Res (Hoboken) 2010;62(7):950–9. doi: 10.1002/acr.20154. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Singh JA, Solomon DH, Dougados M, et al. Development of classification and response criteria for rheumatic diseases. Arthritis Rheum. 2006;55(3):348–52. doi: 10.1002/art.22003. [DOI] [PubMed] [Google Scholar]
- 9.Hinze CH, Suzuki M, Klein-Gitelman M, et al. Neutrophil gelatinase-associated lipocalin is a predictor of the course of global and renal childhood-onset systemic lupus erythematosus disease activity. Arthritis Rheum. 2009;60(9):2772–81. doi: 10.1002/art.24751. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Suzuki M, Wiers K, Brooks EB, et al. Initial validation of a novel protein biomarker panel for active pediatric lupus nephritis. Pediatr Res. 2009;65(5):530–6. doi: 10.1203/PDR.0b013e31819e4305. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Suzuki M, Wiers KM, Klein-Gitelman MS, et al. Neutrophil gelatinase-associated lipocalin as a biomarker of disease activity in pediatric lupus nephritis. Pediatr Nephrol. 2008;23(3):403–12. doi: 10.1007/s00467-007-0685-x. [DOI] [PubMed] [Google Scholar]
- 12.Mina R, Brunner HI. Pediatric lupus--are there differences in presentation, genetics, response to therapy, and damage accrual compared with adult lupus? Rheum Dis Clin North Am. 2010;36(1):53–80. vii–viii. doi: 10.1016/j.rdc.2009.12.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Gladman DD, Ibanez D, Urowitz MB. Systemic lupus erythematosus disease activity index 2000. J Rheumatol. 2002;29(2):288–91. [PubMed] [Google Scholar]
- 14.Hay EM, Bacon PA, Gordon C, et al. The BILAG index: a reliable and valid instrument for measuring clinical disease activity in systemic lupus erythematosus. Q J Med. 1993;86(7):447–58. [PubMed] [Google Scholar]
- 15.Brunner HI, Feldman BM, Bombardier C, et al. Sensitivity of the Systemic Lupus Erythematosus Disease Activity Index, British Isles Lupus Assessment Group Index, and Systemic Lupus Activity Measure in the evaluation of clinical change in childhood-onset systemic lupus erythematosus. Arthritis Rheum. 1999;42(7):1354–60. doi: 10.1002/1529-0131(199907)42:7<1354::AID-ANR8>3.0.CO;2-4. [DOI] [PubMed] [Google Scholar]
- 16.Tafeit E, Moller R, Sudi K, et al. ROC and CART analysis of subcutaneous adipose tissue topography (SAT-Top) in type-2 diabetic women and healthy females. Am J Human Biol. 2000;12(3):388–94. doi: 10.1002/(SICI)1520-6300(200005/06)12:3<388::AID-AJHB9>3.0.CO;2-1. [DOI] [PubMed] [Google Scholar]
- 17.Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420–8. doi: 10.1037//0033-2909.86.2.420. [DOI] [PubMed] [Google Scholar]
- 18.Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36. doi: 10.1148/radiology.143.1.7063747. [DOI] [PubMed] [Google Scholar]
- 19.Caldwell JR, Furst DE, Smith AL, et al. Flare during drug withdrawal as a method to support efficacy in rheumatoid arthritis: amiprilose hydrochloride as an example in a double blind, randomized study. J Rheumatol. 1998;25(1):30–5. [PubMed] [Google Scholar]
- 20.Balsa A, Carmona L, Gonzalez-Alvaro I, et al. Value of Disease Activity Score 28 (DAS28) and DAS28-3 compared to American College of Rheumatology-defined remission in rheumatoid arthritis. J Rheumatol. 2004;31(1):40–6. [PubMed] [Google Scholar]
- 21.Gordon C, Sutcliffe N, Skan J, et al. Definition and treatment of lupus flares measured by the BILAG index. Rheumatology (Oxford) 2003;42(11):1372–9. doi: 10.1093/rheumatology/keg382. [DOI] [PubMed] [Google Scholar]
- 22.Belimumab: Anti-BLyS Monoclonal Antibody; Benlysta; BmAb; LymphoStat-B. Drugs R D. 2010;10(1):55–65. doi: 10.2165/11538300-000000000-00000. [DOI] [PMC free article] [PubMed] [Google Scholar]