Connective tissue diseases: Predicting death in SSc: planning and cooperation are needed

Abstract

Systemic sclerosis is associated with a high level of patient mortality. A promising prognostic model that could enable more effective management and improve survival was recently validated; however, the results demonstrate that choosing the best cohorts for development and validation of predictors of mortality is essential.

Systemic sclerosis (SSc) is generally accepted as having the highest frequency of case-specific mortality of the connective tissue diseases. The causes of death in patients with SSc have changed over time. Before the development of angiotensin-converting enzyme (ACE) inhibitors, renal crisis was the most common reason for mortality; however, more recent analyses performed in different countries around the world have suggested that pulmonary fibrosis and pulmonary hypertension have now become the most frequent causes of SSc-related death.^{1, 2} The reasons for this shift are numerous: more frequent and effective screening; improved technology to detect internal organ involvement at earlier stages of disease;³ and the development of novel efficacious therapies. Such therapies include ACE inhibitors in renal crisis, immunosuppressive medications, and drugs for pulmonary hypertension, as well as the increased use and success of transplantation procedures. Despite these advances, we remain unable to accurately and reliably predict the risk of mortality in an individual patient with newly diagnosed SSc. In their recent publication, Fransen et al.⁴ performed a validation study for a simple predictive model of 5-year survival in patients with SSc, originally developed in 1999 by Bryan and colleagues.⁵ In this commentary we discuss their results in the context of the specific challenges in developing and validating prognostic tools for use in SSc, which need to be considered before applying such instruments in the clinic, or when developing new predictive models.

Risk prediction rules enable clinicians to interpret medical data for diagnostic, therapeutic and prognostic assessment, by providing the probability of a particular outcome, such as 5-year mortality. These prognostic models are developed by applying statistical techniques to identify combinations of predictor variables that categorize a group of patients with a specific disease into risk subgroups. Over the past 20 years, changes in healthcare systems and the advent of evidence-based medicine have forced increased attention on to the development of prediction rules that are simple to use, but which are also accurate and robust. The most well-known example of a patient assessment tool that uses clinical risk prediction rules is the Framingham Coronary Heart Disease Prediction Score.⁶

Methodological standards for the development of prediction rules were proposed by Wasson et al.⁷ and were later modified by Laupacis and colleagues.⁸ Furthermore, several publications have discussed the factors that are important for effective validation of prediction rules.^{9, 10} Despite these solid methodological foundations, the identification of clinical features that predict mortality in patients with SSc has been difficult, primarily because individuals with the disease only represent a small percentage of the general population. Only a few investigators from a small number of clinics have followed enough patients with SSc over time to enable robust estimation of the prognostic value of clinical factors in relation to disease outcomes. For this reason, many of the published studies examining risk factors for mortality in SSc have used prevalent populations, as opposed to an inception cohort of newly presenting cases. This type of selection means that a study might not include patients who died early in the course of the disease, resulting in a population disproportionately composed of patients with longstanding SSc. Underrepresentation of patients with early SSc-associated fatality could result in failure to identify risk factors predictive of this outcome, and the resultant model would fail to provide an accurate prognosis when applied to patients in earlier stages of disease.

In their 1999 publication,⁵ Bryan et al. described a simple tool for prediction of 5-year mortality in patients with SSc on the basis of three clinical factors. The model was developed in a prospective cohort of 280 patients with SSc who were referred to a single center for evaluation upon disease onset (defined as the first demonstration of cutaneous symptoms) between 1982 and 1991. A minimum of 5 years of follow-up data were required for a patient to be included in the study. The prediction rules were developed using logistic regression followed by a Monte Carlo simulation method. In addition to age and gender, significant predictors of mortality included an erythrocyte sedimentation rate (ESR) elevated to ≥25 mm/h, proteinuria above trace, and a lung carbon monoxide diffusing capacity (DLCO) <70% of the predicted value. The presence of all three clinical risk factors was found to predict 100% mortality at 5 years. This model is appealing owing to its simplicity, but had not been externally validated in another population of patients with SSc; external validation of this prediction model was the purpose of the study by Fransen and co-workers.⁴

When applied to external validation cohorts, the performance of prediction rules is frequently disappointing, for three main reasons. Firstly, the patient populations included in the development and validation cohorts might differ in their clinical composition. This can have an effect on both discrimination (the ability of the risk prediction rules to distinguish those patients who will or will not die at the specified time point) and calibration (the degree of similarity between observed and predicted risks) of the model. Clinical variables in populations of patients with SSc that could affect the performance of the model include disease duration (as discussed above), and the proportion of patients with diffuse versus limited cutaneous SSc. These two subsets of the disease have different clinical features and natural history, with multiple published studies confirming worse survival in diffuse SSc. Secondly, the definitions of the predictor variables, or the techniques used to measure them, might differ between the development and validation cohorts. Finally, validation cohorts are often smaller than development cohorts, which will decrease the power and accuracy of any statistical analyses.

The validation cohort in the study by Fransen and colleagues⁴ was a European multicenter population of patients with SSc, diagnosed before 2002, with follow-up for at least 5 years or until death. Centers from the European League against Rheumatism Scleroderma Trials and Research (EUSTAR) group were invited to enroll patients. The authors examined the predictor variables (presence of urine protein, elevated ESR and low DLCO) originally proposed by Bryan et al.⁵ to evaluate their power of discrimination. These variables were assessed by chart review and medical correspondence, and then electronically transferred to the research center. A total of 1,049 patients were included in the analysis, and thus small sample size was not a limitation of the study. Any missing data for variables were replaced by single imputation, after which no bias was observed between patients who lived and those who died. The discrimination of the prediction rules was acceptable, with an area under the curve (AUC) for the original model of 0.78 (95% CI 0.74–0.82). The authors also recalibrated the model using the regression coefficients obtained from their multivariable model, and included disease subset as an additional variable (diffuse versus limited), which only resulted in a slight improvement in the AUC to 0.81 (95% CI 0.78–0.85).

The observed mortality was only 31% among the patients presenting with all three risk factors in the validation cohort,⁴ suggesting a major overestimation of death at 5 years by the original model (predicted mortality of 100%). Thus, in this validation cohort, the model showed good discriminatory performance but considerable overestimation and, hence, poor calibration, limiting its potential clinical use. This variation could, in part, be attributable to the validation cohort being enrolled one decade later, during which time therapeutic regimens have changed and might have improved survival. Another contributor is that the validation cohort had more patients with limited cutaneous SSc, and also comprised a prevalent population that is more likely to have included patients with a longer disease duration before enrollment (average disease duration was not reported), compared with the development cohort. These findings demonstrate, as pointed out by Fransen et al.,⁴ that testing of prediction models in the appropriate clinical context in which they were developed is of vital importance.

The factors discussed above represent considerable challenges for the development and validation of clinical risk prediction rules in patients with SSc. The proper use of the many national and international registries of patients with SSc that have been established will be beneficial for the future understanding of mortality in this disease. Collaboration and planning between these centers as well as consideration of separate models for diffuse and limited cutaneous SSc will be vital in the development and validation of risk stratification tools in SSc. Accurate predictor variables, ultimately incorporating serologic, genetic and biomarker information, can then be applied to patient care and clinical trial design.