Validation of the Auto-Inflammatory Diseases Activity Index (AIDAI) for hereditary recurrent fever syndromes

Abstract

Objectives

To validate the Auto-Inflammatory Diseases Activity Index (AIDAI) in the four major hereditary recurrent fever syndromes (HRFs): familial Mediterranean fever (FMF), mevalonate kinase deficiency (MKD), tumour necrosis factor receptor-associated periodic syndrome (TRAPS) and cryopyrin-associated periodic syndromes (CAPS).

Methods

In 2010, an international collaboration established the content of a disease activity tool for HRFs. Patients completed a 1-month prospective diary with 12 yes/no items before a clinical appointment during which their physician assessed their disease activity by a questionnaire. Eight international experts in auto-inflammatory diseases evaluated the patient's disease activity by a blinded web evaluation and a nominal group technique consensus conference, with their consensus judgement considered the gold standard. Sensitivity/specificity/accuracy measures and the ability of the score to discriminate active from inactive patients via the best cut-off score were calculated by a receiver operating characteristic analysis.

Results

Consensus was achieved for 98/106 (92%) cases (39 FMF, 35 CAPS, 14 TRAPS and 10 MKD), with 26 patients declared as having inactive disease and 72 as having active disease. The median total AIDAI score was 14 (range=0–175). An AIDAI cut-off score ≥9 discriminated active from inactive patients, with sensitivity/specificity/accuracy of 89%/92%/90%, respectively, and an area under the curve of 98% (95% CI 96% to 100%).

Conclusions

The AIDAI score is a valid and simple tool for assessing disease activity in FMF/MKD/TRAPS/CAPS. This tool is easy to use in clinical practice and has the potential to be used as the standard efficacy measure in future clinical trials.

The hereditary recurrent fever syndromes (HRFs) are rare Mendelian autoinflammatory diseases (AIDs) characterised by flares of fevers associated with acute inflammation affecting various tissues without evidence of an underlying cause. With the exception of familial Mediterranean fever (FMF), these diseases are very rare, with an estimated prevalence of less than two per million. The four main diseases by date of description and frequency are: FMF, mevalonate kinase deficiency (MKD), tumour necrosis factor receptor-associated periodic syndrome (TRAPS) and cryopyrin-associated periodic syndromes (CAPS). Recent advances in the molecular pathogenesis of HRFs have led to a better understanding of the common pathways and mediators of apoptosis, inflammation and cytokine signalling involved in their pathogenesis and have radically improved diagnosis and therapies.^{1 2} With the increasing potential for targeted therapies in AIDs, there is the need for validated assessment tools which can be used to evaluate the level of disease activity and response to therapy and thus to assess drug efficacy in standardised assessments across trials.^{3–6 7} The lack of such standardised and validated measures for assessing disease activity for either adults or children with AIDs has seriously hampered assessment of current treatments and comparison of treatment responses in the different HRFs. An international collaboration, initiated by Assistance Publique-Hôpitaux de Paris (APHP) in association with the Paediatric Rheumatology International Trials Organisation (PRINTO at http://www.printo.it)⁸ and supported by the EUROFEVER and EUROTRAPS networks,^9–11 has already designed the content and preliminary scoring of this Auto-Inflammatory Disease Activity Index (AIDAI)¹² using a single-format disease-adapted patient diary for the four major HRFs.

Here, we report the results of the second step of the project, which was to formally validate the AIDAI and its scoring system for the assessment of disease activity in HRFs through an international prospective data-collection process, a final consensus conference and statistical analysis.

Patients and methods

The overall methodology of this project phase was based on a framework used successfully in previous work in rheumatoid arthritis,¹³ juvenile idiopathic arthritis,^14–16 juvenile systemic lupus erythematosus^17–19 and inflammatory myopathies.²⁰ We also used the nominal group technique (NGT) originally defined as a ‘creative decision-making tool, facilitated by decision technique, in which group members must pool their judgments to invent or discover a satisfactory course of action’.^21–23

Study design

Patient enrolment began in November 2010 and ended in April 2011 in eight centres belonging to the PRINTO/EUROFEVER networks in six countries (France, Italy, The Netherlands, Germany, UK and Turkey). Ethics committee approval was obtained in countries as required by local regulations. In each centre, written or verbal informed consent was obtained from a parent or legal guardian, according to the local requirements.

A sample of consecutive patients (children and adults) attending the clinic of the participating centres and with a genetically confirmed diagnosis of FMF, MKD, TRAPS or CAPS and in various states of disease activity, ranging from active to inactive were enrolled. Patients were evaluated by at least one physician with expertise in the care of AIDs. All patients (or their parents for minors as appropriate) were asked to complete a 1-month prospective diary before the scheduled clinical appointment.

Content and scoring of the AIDAI tool

The original AIDAI diary contains 13 items as follows: (a) fever ≥38°C (100.4°F); (b) overall symptoms; (c) abdominal pain; (d) nausea/vomiting; (e) diarrhea; (f) headaches; (g) chest pain; (h) painful nodes; (i) arthralgia or myalgia; (j) swelling of the joints; (k) eye manifestations; (l) skin rash; (m) pain relief taken. Pain relief used was added in anticipation of a future composite score, but was not used in the calculation of the activity score and has not been considered for the remainder of this paper.¹² In the original version of the AIDAI, 11 out of 12 items were scored by the patients/parents as 0=absent, 1=minor, 2=mild, 3=severe, while fever was scored as 0=absent or 1=present for a total score in a single day of 0–34 and in a month of 31 days of 0–1054. For this paper, it was hypothesised that a simplified scoring system would produce a similar statistical performance while allowing greater simplicity in AIDAI completion. For this reason, patients/parents filled out the original diary, but, during analysis, the items of this diary were also dichotomised as no (0)=absence of symptom or yes (1)=presence of symptom (which included the 1, 2 and 3 of the original diary), yielding a total score in a single day of 0–12. In a month of 31 days, the cumulative score ranges from 0 to 372. In addition to the diary completed by the patient and/or parents, physicians and patients completed together during the visit a questionnaire retrospectively assessing disease activity during the preceding 30 days. For this purpose the tool documented: (1) the numbers of days with each symptom; (2) all treatments received; (3) the number of days of pain relief drugs; (4) the regularity with which disease-modifying antirheumatic drugs were taken (colchicine or biological agents); (5) the number of days out of school or work; (6) disturbance of patient's social life by the disease with a three-level categorical scale (none; a little; a lot); (7) patient's assessment of fatigue on a 21-point visual analogue scale (where 0=no fatigue and 100=maximum fatigue); (8) patient's global assessment of overall well-being and physician's global assessment of disease activity on two separate 21-point visual analogue scale^{24 25}; (9) the need to consult an external doctor; (10) the feasibility of the activity score (patient's opinion and comments).

Validation procedures

Data from the patients’/parents’ and physicians’ assessment were centrally entered in the EUROFEVER database at the PRINTO coordinating centre in Genoa, Italy. A three-step validation process was used.

Step 1: physician's blinded web evaluation of patient's disease activity level

In June 2011, eight international adult and paediatric physicians with expertise in AIDs (coauthors of this paper) were asked to evaluate the level of activity for each patient in the database as no activity (0), low activity (1), moderate activity (2) or severe activity (3). Evaluation was carried out by seven (one physician unavailable) of them via the internet by logging in the member area of the PRINTO website (http://www.printo.it). Patients were anonymised, total AIDAI score was not provided, and the physicians were blinded to the diagnosis so that they could not recognise the patients followed in their own hospital. As per the NGT, the physicians worked independently of each other. At each session, a minimum consensus of 6/7 (85%) for the inactive/active (low, moderate and severe activity combined) was required to consider a patient as having reached a final consensus. When consensus was not achieved, a new session was held for a total of three iterative independent evaluation sessions; at each subsequent session, physicians were provided with their previous score as well as with the blinded evaluation of the other participants.

Step 2: NGT consensus conference of patient's disease activity level

Eight physicians gathered in an NGT consensus conference held in Cambridge, UK on 27−28 June 2011. The conference was facilitated by one coauthor (NR) with expertise in NGT, who did not participate in the patient's evaluation in step 1. Participants were provided with a laptop with internet connection to connect to the PRINTO web database, and were asked to re-evaluate individually the level of disease activity for the patients in whom consensus was not achieved in the previous step. Then, for all patients, each of the eight participants was asked, in ‘round robin’ fashion, to provide a verbal explanation as to why he/she had assigned a certain level of disease activity to that particular patient. The individual evaluation of each physician was shown on a screen to all participants. After each ‘round robin’ discussion, a web-based electronic vote was taken, and consensus was achieved if a minimum of 6/8 (75%) participants provided the same level of disease activity. Patients for which consensus could not be achieved after a second vote were omitted from further considerations.

Step 3: statistical analysis of the AIDAI scoring system

According to the preliminary work previously published, the score calculation differs according to the specific disease as follows¹²: FMF could be scored by adding the variables a+c+g+i+j+l, MKD the variables a+c+d+e+h+i, TRAPS the variables a+b+c+i+k+l and CAPS the variables a+f+i+k+l; a graded value (0 to 3) was used except for fever (0 or 1). For a 31-day month, cumulative scores range from 0 to 496 for FMF, MKD and TRAPS and 0 to 403 for CAPS (figure 1). We hypothesised that the sum of the positive items from the diary as yes/no would have the same statistical performance as the disease-specific scoring system, and therefore a simplified score was calculated for each subject by summing all the items in the diary as total score=a+b+c+d+e+f+g+h+i+j+k+l both for the items scored 0–3 (range 0–1054) and for the simplified items scored 0–1 (range 0–372). Average values of the score were compared between activity groups (inactive vs the three levels of activity combined) through Spearman's correlation coefficients. The ability of the score to discriminate active from inactive patients and the best AIDAI total cut-off score for classifying the patients as active/inactive were evaluated by a receiver operating characteristic (ROC) analysis.²⁶ Sensitivity, specificity and accuracy of the scores were calculated using the best cut-off value of the total AIDAI score.

Figure 1.

Final Auto-Inflammatory Diseases Activity Index diary.

Data were entered in a web-based Access XP database and analysed with Excel XP (Microsoft), SPSS V18 by two of the authors (MPS and NR).

Results

A total of 106 patients were enrolled and available for the analysis: 42 FMF, 39 CAPS, 14 TRAPS and 11 MKD. Demographic data were available for 100 patients; there were 58 children and 42 adults, and 37 were female and 48 were male (gender missing for 15 patients). Median age was 1.6 years at onset, 8.1 years at diagnosis, 10.5 years at first visit to centre, and 15.5 years at AIDAI completion, with disease duration of 13 years.

Step 1: physician's blinded web evaluation of patient's disease activity level

During the three iterative web-based blinded evaluation sessions, consensus was achieved on 63/106 cases (59.4%), with 13 patients declared as inactive, 49 as active (26 with low activity, 13 with moderate activity, 10 with high activity), and one patient was not evaluable (wrong diagnosis) by consensus. For the remaining 43 patients, consensus was not achieved and patients were therefore considered for the consensus NGT discussion as per step 2.

Step 2: NGT consensus conference on patient's disease activity level

During the NGT meeting, consensus was achieved for an additional 36 patients for a total of 98/106 cases (92%). For the remaining eight cases, consensus was not achieved, and the patients were therefore omitted from further evaluation. Table 1 reports the disease activity grading for the four HRFs diagnosed in 98 patients with consensus and for all patients combined. Of the 98 subjects, 26 (27%) were declared to have inactive disease, while the remaining 72 (63%) were classified as having active disease (33 low, 28 moderate, 11 high disease activity). The descriptive statistics for each item of the AIDAI tool for the 98 patients is reported in table 2.

Table 1.

Consensus classification of the 98 patients with four hereditary recurrent fever syndromes according to the level of disease activity

Classification	FMF (N=39)	CAPS (N=35)	TRAPS (N=14)	MKD (N=10)	Total (N=98)
Inactive	8	12	6	0	26
Active
Low activity	13	14	2	4	33
Mild activity	11	8	3	6	28
Severe activity	7	1	3	0	11

CAPS, cryopyrin-associated periodic syndromes; FMF, familial Mediterranean fever; MKD, mevalonate kinase deficiency; TRAPS, tumour necrosis factor receptor-associated periodic syndrome.

Table 2.

Patient/parent completed 1 months’ diary score with items dichotomised as 0/1 in 98 patients with hereditary recurrent fever syndromes

Diary item	FMF (N=39)	CAPS (N=35)	TRAPS (N=14)	MKD (N=10)	Total (N=98)
(a) Fever ≥38°C (100.4°F)	2.77±4.27	0.94±2.42	5.36±9.15	2.80±2.04	2.35±4.61
(b) Overall symptoms	5.56±6.43	3.97±9.05	13.86±20.16	7.70±8.21	6.86±11.22
(c) Abdominal pain	6.72±6.66	1.57±5.56	4.14±7.49	3.40±7.66	3.90±6.58
(d) Nausea/vomiting	1.97±4.14	0.80±2.84	0.00±0.00	1.60±1.58	1.17±3.10
(e) Diarrhoea	2.31±5.11	0.31±1.69	0.14±0.53	2.30±4.06	1.44±4.04
(f) Headaches	2.62±3.85	5.29±13.81	3.79±5.16	5.70±6.02	3.93±8.66
(g) Chest pain	0.85±1.81	0.14±0.55	1.14±3.03	0.00±0.00	0.52±1.62
(h) Painful nodes	1.05±3.91	0.77±2.21	0.93±2.56	6.20±5.33	1.40±3.58
(i) Arthralgia or myalgia	3.77±6.08	6.11±13.13	11.36±18.11	3.20±4.71	5.37±10.89
(j) Swelling of the joints	1.23±3.00	0.94±2.90	2.14±5.57	0.00±0.00	1.06±3.18
(k) Eye manifestations	0.26±1.02	3.57±7.76	3.43±9.20	0.30±0.67	2.15±6.98
(l) Skin rash	1.44±6.73	3.43±9.37	5.00±9.16	1.40±2.67	3.15±8.87
Total AIDAI score	19.10±16.41	20.49±26.00	34.93±49.13	22.50±15.93	22.20±26.79

Data are means±SD, maximum 31 points per item.

AIDAI, Auto-Inflammatory Diseases Activity Index; CAPS, cryopyrin-associated periodic syndromes; FMF, familial Mediterranean fever; MKD, mevalonate kinase deficiency; TRAPS, tumour necrosis factor receptor-associated periodic syndrome.

Step 3: statistical analysis of the AIDAI scoring system

Since the analyses gave the same results whether the 0–3 score or the simplified no(0)/yes(1) version was used (Spearman's correlation coefficient 0.96), only the latter is reported here (see figure 1).

In the final analytical step, in order to properly calculate sensitivity, specificity and ROC cut-off, the 98 patients were dichotomised as having active (low, moderate and high activity combined) or inactive disease. Figure 2 shows the distribution of the AIDAI total score with each AIDAI item dichotomised as yes/no: 14 subjects (12%) had a score=0; the score had a mean value of 22.2±26.8 and a median of 14 (range=0–175); values are skewed toward lower values of the AIDAI total score. According to the ROC curve (figure 3), an AIDAI score ≥9 points identifies active patients, while an AIDAI total score <9 points identifies patients as inactive; sensitivity was 89% (95% CI 80% to 94%), specificity 92% (95% CI 76% to 98%), area under the curve (AUC) 98% (95% CI 96% to 100%) and accuracy 90% (95% CI 84% to 96%) (table 3). Similar performances were obtained when the original 0–3 score for each item of the AIDAI was applied as follows: sensitivity was 92% (95% CI 86% to 98%), specificity 96% (95% CI 88% to 100%), AUC 99% (95% CI 97% to 100%) and accuracy 93% (95% CI 87% to 98%) (data not shown).

Figure 2.

Distribution of the Auto-Inflammatory Diseases Activity Index total score for 98 patients with a hereditary recurrent fever syndromes diagnosis (0/1 scoring).

Figure 3.

Receiver operating characteristics curve with binary items of disease activity (0/1 scoring) showing an area under the curve of 0.98 (95% CI 0.96 to 1) in 98 patients with a hereditary recurrent fever syndromes diagnosis.

Table 3.

Accuracy measures of the AIDAI total score (0/1 scoring) for 98 patients with a hereditary recurrent fever syndromes diagnosis

Activity as defined by the AIDAI total score	Activity as defined by consensus		Total
	Active	Inactive
≥9 active	64 (89%)	2 (8%)	66
<9 inactive	8 (11%)	24 (92%)	32
Total	72	26	98

A total AIDAI score <9 separates inactive from active patients (total score ≥9).

AIDAI, Auto-Inflammatory Diseases Activity Index.

Discussion

Based on the results of the consensus conference, the AIDAI proved to be a valid and reliable tool for the assessment of presence or absence of disease activity in the four main HRFs. The diverse group of clinical experts caring for adults and children and the multicentre international enrolment setting served to enhance its validity and applicability in broad clinical and research settings. Accurate and reproducible evaluation of disease activity is of major importance in assessing treatment efficacy and appreciating the effect of disease activity on quality of life and, although this is currently unproven, it may predict the development of serious long-term complications. AIDAI is the first validated instrument designed to standardise assessment of AIDs activity across trials, and to facilitate comparison and meta-analysis of clinical trials in the future. AIDAI will also allow patient's self-reported disease activity evaluation in daily practice.

In its current form, the AIDAI score is very easy to use. A unique patient/parents’ diary gathering all variables for the four main autoinflammatory diseases is convenient for routine clinical use. The statistical analyses showed that a simple sum of the positive items from the diary with binary value (yes/no) has the same statistical performance as the disease-specific scoring system with graded values reported in the preliminary publication.¹² Both patients and physicians will benefit from this simplified scoring system: binary values are easier for patients, and the single format with a straightforward sum of all positive items simplifies its use by physicians.

In this validation phase, patients were asked to complete a 1-month prospective diary before their scheduled clinical appointment. However, the diary could be completed for periods of other lengths as clinically appropriate. A 3-month period survey is probably more suitable for episodic diseases such as FMF and MKD, but the period could be longer in TRAPS and shorter in CAPS. As detailed in the Patients and methods section, the calculation of the score is straightforward, consisting of the sum of all 12 variables divided by the number of months over which the diary was completed (0–372 in a month of 31 days).

An AIDAI cut-off score of 9 accurately differentiated patients with active versus inactive disease with an area under the ROC curve of 0.98, with sensitivity and specificity exceeding 0.8. This cut-off will help physicians in daily practice to decide if a patient has active or inactive disease. This is moreover important for clinical trials in which an index discriminating patients achieving remission from those who did not might increase the power of the statistical comparisons, thus permitting a smaller sample size,²⁷ a crucial factor in orphan and ultra-orphan diseases such as HRFs.

The involvement of adult and paediatric centres in the study allowed the inclusion of a proportionate number of adults and paediatric patients. Similarly, the collection of consecutive patients in each centre reflects a distribution of active and inactive patients in line with what is observed in everyday clinical practice.

Our study has some limitations particularly because of the relatively small number of patients with these rare diseases. The limited number of patients did not allow separate analyses of the different diseases or discrimination with sufficient power between the different levels of disease activity (low, moderate and high).

The AIDAI score was validated for the four main AIDs, but it seems likely that it could be applied to other AIDs that share some of their clinical features, such as periodic fever, aphthous, pharyngitis and cervical adenitis syndrome (PFAPA) and Schnitzler's syndrome, although further studies will be required to validate this hypothesis. While disease activity is important, it does not encompass the total effect of disease on patients with AIDs. A further step would be the creation of a composite score including additional disease characteristics such as use of rescue treatments, biological measurements, fatigue, quality of life and disease-associated damage.¹² Future longitudinal studies should also be implemented to allow testing of other psychometric characteristics of the tool such as the sensitivity to change.

In conclusion, this paper reports a simplified AIDAI score built and validated by international experienced clinicians of AIDs belonging to the PRINTO, EUROFEVER and EUROTRAPS networks. This tool will be useful in clinical practice, as well as clinical trials, in the assessment of activity of these rare diseases, permitting more reliable analysis of the efficacy of new treatments.