Derivation and validation of a national multicenter mortality risk stratification model – the ExCare model: a study protocol

Abstract

Introduction

Surgical care is essential for proper management of various diseases. However, it can result in unfavorable outcomes. In order to identify patients at higher risk of complications, several risk stratification models have been developed. Ideally, these tools should be simple, reproducible, accurate, and externally validated. Unfortunately, none of the best-known risk stratification instruments have been validated in Brazil. In this sense, the Ex-Care model was developed by retrospective data analysis of surgical patients in a major Brazilian university hospital. It consists of four independent predictors easily collected in the preoperative evaluation, showing high accuracy in predicting death within 30 days after surgery.

Objectives

To update and validate a Brazilian national-based model of postoperative death probability within 30 days based on the Ex-Care model. Also, to develop an application for smartphones that allows preoperative risk stratification by Ex-Care model.

Methods

Ten participating centers will collect retrospective data from digital databases. Variables age, American Society of Anesthesiologists (ASA) physical status, surgical severity (major or non-major) and nature (elective or urgent) will be evaluated as predictors for in-hospital mortality within 30 postoperative days, considered the primary outcome.

Expected results

We believe that the Ex-Care model will present discriminative capacity similar to other classically used scores validated for surgical mortality prediction. Furthermore, the mobile application to be developed will provide a practical and easy-to-use tool to the professionals enrolled in perioperative care.

Introduction

Surgical care is essential for proper management of several clinical conditions and the number of procedures performed over the years has been increasing in Brazil and worldwide.1, 2 However, it can lead to relevant negative outcomes. The global perioperative risk is multifactorial, depending on the interaction between anesthesia, patient’s clinical status, surgical trauma, and hospital care. In high income countries, the risk of complications and postoperative morbidity is estimated at 3% to 17%.3, 4

In this sense, perioperative risk stratification as part of safety and quality of care policies facilitates informed consent and assists professionals involved in the perioperative care to plan assistance.⁵ Therefore, different assessment tools have been implemented to identify patients at high surgical risk. Most of these models, however, have been developed and validated in high income countries.6, 7

In this scenario, our research group developed the SAMPE model. Derived from the analysis of 13,524 surgical patients at Hospital de Clínicas de Porto Alegre (HCPA), it incorporates 4 variables easily collected in the preoperative period: age, American Society of Anesthesiologists physical status (ASA-PS) classification, nature of the procedure (urgency or elective) and surgical severity (major, intermediate, or minor). The resulting model showed good discriminative capacity for 30-day in-hospital mortality, but it still had some inconsistencies.⁸ To correct these instabilities, we developed the Ex-Care model, evaluating a different sample of 17,791 patients at our institution. After statistical refinement, the Ex-Care model performed better than its predecessor⁹ using the same predictors. Nevertheless, this result is validated for HCPA population, requiring an assessment of its accuracy in other national institutions.

Thus, considering the recent recommendations in perioperative care, which guide the creation of a national system to identify patients with a higher risk of post-surgical morbidity and mortality,¹⁰ we aim to build a robust model based on data from patients operated in Brazil. Information regarding surgeries performed in hospitals in different regions of the country will be collected. The Ex-Care model will have its performance evaluated based on the data collected at participating centers. Thereafter, an update of the Ex-Care model will be built and validated. Finally, as previously done for Ex-Care (available on iOS platforms, https://apps.apple.com/br/app/excare/id1515296910?l=en, and Android, https://play.google.com/store/apps/details?id=excare.model), a mobile application for smartphones contemplating the data of the updated model will be developed.

Methods

Study design

Retrospective, multicenter, cohort study which aims to build a national preoperative risk model based on Ex-Care model of probability of postoperative death within 30 days, with hospital as a random effect. The updated Ex-Care model will be validated in a different sample from that of the derivation. The study will be conducted and reported according to Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines.¹¹

Source of data and participants

The study will be carried out in ten national reference hospitals located in different Brazilian regions and will be coordinated by the Anesthesia and Perioperative Medicine Service of HCPA – called the Executive Committee (EC). To analyze the performance of the Ex-Care model and to update it, all consecutive hospitalized patients who underwent procedures from January 1st, 2017 to April 30th, 2018 in the participating centers that meet the inclusion criteria will be enrolled. For its validation, another sample containing data from patients who underwent surgery from May 1st to December 31st, 2018 will be evaluated. All subjects above 16 years of age who are candidates for elective or emergency surgery will be considered eligible. Diagnostic procedures and those under sedation or local anesthesia will be excluded; in addition, if a patient was submitted to more than one surgical intervention during hospitalization, only the major one will be considered for outcome analysis. Liver, lung, and heart transplantation and patients diagnosed with brain death who were candidates for organ donation will be excluded.

Outcome

In-hospital mortality within 30 days after surgery will be the primary outcome. Every patient included in both derivation and validation cohort will be followed until this period. Those who remain hospitalized after 30 days of surgery, or who are discharged before that period, will not be followed-up from that point on. All participating centers must have a digital database capable of determining the status of patients (hospital discharge, hospitalization, or death) on the 30th postoperative day.

Model Development

Selection of predictors for derivation cohort

When building surgical risk stratification instruments, variables related to patient’s clinical condition and procedure to be performed must be considered. These variables can be measured at different times of perioperative care.¹² Ideally, a risk score should be simple, reproducible, objective, accurate, applicable to most surgeries (including emergency ones), and validated for all populations.⁷ In the construction of the original SAMPE model, four variables with proven accuracy in existing perioperative risk models were considered: ASA-PS classification, age, surgical severity (minor, intermediate, and major), and nature (urgent or elective). The classification of surgical severity in three categories was carried out through the opinion of specialists and literature review, taking into account surgical time, magnitude of the trauma and predicted bleeding.¹³ For its development, urgent surgical procedures were considered those in which there is risk of death and/or limb loss within 24 hours.

While building the SAMPE model, 13,524 patients were evaluated, a logistic regression model was adjusted for the four variables analyzed and all correlated significantly with in-hospital death within 30 days. C-statistic demonstrated excellent predictive capacity for probability of death within 30 days with area under receiver operating characteristic (AUROC) of 0.9137, a high-performance value. The Hosmer-Lemeshow statistic of 13.28 (p = 0.125) in the derivation data set proved to be an acceptable calibration model. Subsequently, the model was validated with a new sample of 7254 patients, with a sensitivity of 86.4% and specificity of 81.4%. The c-statistic of the validation sample was 0.922.⁸

When evaluating SAMPE model statistics, however, there was no difference in the odds ratio (OR) of the variable surgical size for small and medium-sized procedures (OR 0.691; CI 0.467–1.022). Thus, this variable was adjusted unifying small and medium-sized surgeries into a single group, “non-major procedures”. This adjustment did not show a good fit by the Hosmer-Lemeshow test and, from a later verification, it was noticed that the variable age did not meet the assumption of linearity, showing that the risk of death did not increase steadily throughout life. In order to deal with this lack of linearity, a new logistic regression model was adjusted using polynomial regression splines technique for the independent variable age. It consists of the inclusion of continuous exposure coded by using splines functions. The junction between two intervals is called “knot”.¹⁵ We have modeled age using splines with five knots to allow for non-linearity. The final tool developed was given the name Ex-Care model. It presented an AUROC of 0.926 (CI 0.91–0.93) for the derivation sample, with goodness-of-fit by the Hosmer-Lemeshow Test of 9.26 (p = 0.41), indicating good calibration.⁹ The update of the Ex-Care model at the national level will maintain the adopted statistical refinements and will have patients operated at hospitals in different country regions as a sample.

Sample size

A robust derivation model requires at least 100 events per candidate variable and a validation study requires at least 10 events.14, 15 Therefore, a minimum of 440 events were needed to achieve stable estimates from the regression model. Although there are regional differences, the mean mortality rate across all surgical procedures in Brazil was 1,6% in the period of 2008 to 2016,¹⁶ requiring a minimum sample size of 27,500.

We aim to recruit as many hospitals as possible. Considering data from previous studies at HCPA that involved similar profile of participants and the same time interval, we believe that approximately 6500 patients/year will be enrolled at HCPA.⁸ Furthermore, based on a survey carried out with the coordinating researchers from the other centers, we estimate to collect data from 77,000 patients (Table 1).

Table 1.

Estimated sample size per participating center per year.

Participating centers	Estimated subjects (per center/ annually)
Hospital de Clínicas de Porto Alegre	6500
City: Porto Alegre - RS
Hospital Ernesto Dornelles	4000
City: Porto Alegre - RS
Hospital Tacchini	3500
City: Bento Gonçalves - RS
Hospital Moinhos de Vento	3500
City: Porto Alegre – RS
Hospital São Paulo	4000
City: São Paulo - SP
Hospital São Carlos	3500
City: Fortaleza - CE
Hospital Brasília	3000
City: Brasília - DF
Campus de Botucatu Faculdade de Medicina UNESP – SP	4000
City: São Paulo - SP
Instituto do Câncer do Estado de São Paulo	3500
City: São Paulo - SP
Hospital Universitário Prof. João Cardoso Nascimento	3000
City: Aracajú - SE

Data management

Participating centers will be asked to send a query to the EC reporting the following characteristics of each eligible patient: age, sex, medical record number, ASA-PS classification, surgery performed, nature of the procedure (elective or urgency/ emergency), date of surgery, date of discharge or death (if it occurs until before the 30th day), origin of the patient (public health system or private sector). Queries will be built by the Information Technology (IT) Services of each participating center. IT, using Structured Query Language (SQL), creates programs that interact with the relational database and generate information organized in rows and columns – for our purpose, a spreadsheet will be developed in Microsoft Excel. In order to ensure data confidentiality, patients name will not be recorded. Additionally, the database will be shared on a drive exclusively available for the project researchers. Each participating center has a coordinator responsible for adequate data collection at the institution. EC will be responsible for calculating the risk of death in 30 days based on the Ex-Care model. Following stratification, patients will be divided into 4 classes of probability of death within 30 days (Table 2).

Table 2.

Color-coded risk classification based on the probability of death within 30 days.

This risk stratification will be compared with the clinical evolution of the patients, with mortality within 30 days after surgery as primary outcome to be analyzed. The full data collection and record linkage strategy are depicted in Figure 1.

Figure 1.

Model development methodology.

Missing data

Most prediction models, including those based on logistic regression, are not able to deal with missing data, requiring care when developing, validating, and implementing.¹¹ In order to reduce this inconvenience, we chose to only select centers that have electronic medical records and in which all the predictors necessary for the calculation of surgical risk by the Ex-Care model are routinely registered in clinical practice. Missing data will be handled in complete-case analysis.

Statistical analysis methods

Categorical data will be summarized in absolute and relative frequencies. Continuous data will be presented as mean and confidence interval (95%). Each patient will have their probability of death quantified using the Ex-Care model for temporal validation. Logistic regression model adjusted for the independent predictors of Ex-Care model (age, ASA-PS score, surgical severity, and nature of the procedure) will assess the association with primary endpoint mortality within 30 days; the magnitude with which each variable is related to the outcome will be demonstrated with odds ratios and 95% confidence intervals. In this model, the linearity assessment of the age predictor will be performed using splines.¹⁷

Model calibration will be evaluated by plotting the proportion of expected events and the predicted probabilities for the defined risk groups. The Hosmer-Lemeshow test will be used to establish this calibration, comparing the expected and observed number of deaths by group risk and performing the AUROC measurement for discrimination. AUROC represents the overall accuracy or performance of the test, as it takes into account all sensitivity and specificity values for each variable value.

For Ex-Care model update, a mixed logistic regression model with random effect for hospitals will be adjusted, aiming to control the heterogeneity among different centers. This will consist of the same independent predictors previously mentioned and will be validated in another sample (patients operated from May to December, 2018), using the methods above described. All analysis will be performed using the R-3.5.1 and SAS, version 9.4 programs.

Discussion

Perioperative risk stratification is a fundamental principle of care for surgical patients. Adopting an easy-to-use tool in the preoperative period for different specialties and capable of identifying patients most likely to have complications is part of the risk management strategy to be implemented by hospitals worldwide. In theory, the concept that high-risk patients need more care seems clear and easy to understand; in daily practice, however, there is a systematic failure in the individual clinical judgment of physicians to identify and manage these patients.¹⁸

In an ideal scenario, every patient who undergoes a procedure should have their surgical risk predicted in the preoperative period. The use of surgical risk models, in addition to assisting decision-making by health professionals, facilitates dialogue with patients and family members, as well as allows monitoring and comparison of outcomes between centers.

Finally, in view of the great heterogeneity between surgical mortality rates among countries, it is not viable to generalize parameters validated for specific populations to all centers. Furthermore, when assessing the socioeconomic, cultural, and health service structure between developed and developing countries, the difference in outcomes becomes evident.1, 18, 19 In a cohort published in 2016 for elective surgeries, similar mortality rates were evidenced when comparing high-income to middle- and low-income countries. However, patients operated in high-income countries were older, had more comorbidities and, once developed complications, were better treated, as determined by a minor failure to rescue rate.²⁰ When evaluating non-elective surgeries, outcome differences become more evident.²¹ Hence, creating a risk model developed and validated for the Brazilian reality must take priority.

In order to fill this gap, based on the Ex-Care model, the update proposed by this protocol, through the use of an application to be developed, will provide an easy-to-use digital tool, with the possibility of interactive improvements, capable of incorporate a robust surgical risk prediction score, validated in a Brazilian surgical sample. It is important to highlight that when using the application to calculate the risk by the Ex-Care model the professional will have, in addition to stratification in the classes described (Table 2), the percentage value of the individual probability of in-hospital death within 30 days.

Limitations

Although the Ex-Care model has been validated with good accuracy in the population assisted by Hospital de Clínicas de Porto Alegre, we cannot assertively guarantee that it will have the same performance at national level. Brazil is a country of continental dimensions with great socioeconomic disparity and availability of resources among its regions.16, 22 This feature may make it difficult to create a single model representative of the assistance provided throughout its territory. In fact, before applying it more widely in Brazil, we must test its performance in different scenarios and populations. In the literature, there are examples of risk prediction models that, when used without adjustments in samples other than the one originally used for their construction, did not show good calibration capacity, requiring corrections in the original model.23, 24 Perhaps different models will be necessary for distinct groups of hospitals (for example, regional models). That being said, it is important to note that the external validation of the Ex-Care model is the next step in the strategy of consolidating and disseminating the use of this new tool in perioperative setting.

Secondly, this is a retrospective study, with data collected from information present in queries sent by participating centers. In order to reduce the missing data, we chose to select only hospitals that had a digital system and where the requested data are recorded in routine clinical care.

Finally, although the Ex-Care model aims to objectively estimate the probability of death within 30 days post-surgery, for its composition, two subjective variables were used (ASA-PS and surgical severity). Nevertheless, the ASA-PS is a classification widely used in the clinical setting and, for the variable surgery severity, a comprehensive review of the literature and consultation with specialists was carried out.

Implications

We believe that an updated Ex-Care model will be a helpful tool to accurately stratify the risk of death to which patients operated in Brazil are subjected, supporting professionals involved in perioperative care to identify high-risk surgical patients and to better plan therapeutic strategies. Also, in order to encourage the use of perioperative risk scores and facilitate access to the Ex-Care model, we will create and make available to professionals engaged in the care of surgical patients a smartphone application that contemplates the new model.

Conclusion

To date, there is no surgical risk model developed for the Brazilian population. This study protocol outlines the methodology that will be used to national derivation and validation of the Ex-Care Model, a tool that aims to accurately estimate the probability of in-hospital death within 30 postoperative days.

Ethics approval

This study is performed in accordance with the declaration of Helsinki and follows the resolutions of the Brazilian National Health Council. Ethical approval was granted by the HCPA Postgraduate Research Group Ethics Committee (Project Number: 2019.0192). Moreover, all other participating centers obtained approval from their own ethics committees. As this is a retrospective observational study and follows standard care for surgical patients, the HCPA Ethics Committee exempted the need for a consent form for participants. A Data Usage Commitment Term was prepared requesting all researchers to ensure data confidentiality.

Conflicts of interest

The authors declare no conflicts of interest.