A multicentre Spanish study for multivariate prediction of perioperative in-hospital cerebrovascular accident after coronary bypass surgery: the PACK₂ score

Abstract

OBJECTIVES

To develop a multivariate predictive risk score of perioperative in-hospital stroke after coronary artery bypass grafting (CABG) surgery.

METHOD

A total of 26 347 patients were enrolled from 21 Spanish hospital databases. Logistic regression analysis was used to predict the risk of perioperative stroke (ictus or transient ischaemic attack). The predictive scale was developed from a training set of data and validated by an independent test set, both selected randomly. The assessment of the accuracy of prediction was related to the area under the ROC curve. The variables considered were: preoperative (age, gender, diabetes mellitus, arterial hypertension, previous stroke, cardiac failure and/or left ventricular ejection fraction <40%, non-elective priority of surgery, extracardiac arteriopathy, chronic kidney failure and/or creatininemia ≥2 mg/dl and atrial fibrillation) and intraoperative (on/off-pump).

RESULTS

Global perioperative stroke incidence was 1.38%. Non-elective priority of surgery (priority; OR = 2.32), vascular disease (arteriopathy; OR = 1.37), cardiac failure (cardiac; OR = 3.64) and chronic kidney failure (kidney; OR = 6.78) were found to be independent risk factors for perioperative stroke in uni- and multivariate models in the training set of data; P < 0.0001; AUC = 0.77, 95% CI 0.73–0.82. The PACK₂ stroke CABG score was established with 1 point for each item, except for chronic kidney failure with 2 points (range 0–5 points); AUC = 0.76, 95% CI 0.72–0.80. In patients with PACK₂ score ≥2 points, off-pump reduced perioperative stoke incidence by 2.3% when compared with on-pump CABG.

CONCLUSIONS

PACK₂ risk scale shows good predictive accuracy in the data analysed and could be useful in clinical practice for decision making and patient selection.

INTRODUCTION

Although mortality during cardiac surgery has decreased in recent decades [1] morbidity has increased, mainly because patients undergoing cardiac surgery are older and have an increased vulnerable comorbid condition [2]. Stroke remains a severe complication after coronary artery bypass grafting (CABG) procedures, and its prevention is particularly important as the patients affected spend prolonged stays in intensive care and have a markedly higher mortality rate. The risk of stroke in CABG surgery is not homogeneous, and different clinical and echocardiographic features have been identified to help in stratifying it. Several prediction models have been developed to preoperatively identify this subgroup of patients with an increased risk of perioperative stroke; however, none has been extensively applied in routine clinical practice [3]. The NNECDSG stroke risk scale [4] was based on 33 062 consecutive patients undergoing isolated CABG surgery in Northern New England between 1992 and 2001, and showed good predictive accuracy with an area under the receiver operating characteristic curve (c-statistic) of 0.70 [95% confidence interval (CI), 0.67–0.72]. However, it is clear that an ideal, generally applicable, simple and widely accepted stroke risk assessment tool does not exist and that each one has its own limitations.

The purpose of this study was to define a clinical grading scale to assess perioperative stroke risk in the CABG population, using the predictive preoperative criteria of stroke outcome in a rapid and efficient procedure carried out at the time of surgery.

METHODS

Study setting, patient cohort and data collection

We designed a retrospective, multicentre, observational study on consecutive adult (≥18 years of age) patients undergoing CABG surgery as a single procedure, recruited from 21 Spanish National Health System hospitals (see Appendix 1). The patients were obtained from hospital administrative databases in reverse chronological order, starting on 31 December 2011. In order to meet the inclusion criteria, we only considered CABG as a single surgical procedure. Patients undergoing >1 relevant procedures during the study period were only considered for analysis in the first surgical procedure. All patients who underwent CABG with associated surgical procedures were excluded. Using standardized case report forms, we collected selective preoperative data, which were entered in a computer database. All the clinical variables collected had previously been shown to have a significant impact on perioperative stroke risk, in accordance with the NNECDSG prediction model [4].

Variables

Perioperative stroke was defined as any new temporary or permanent, focal or global neurological defect, within 30 days after surgery, or later if still in hospital, in accordance with the published guidelines [5]. Temporary stroke included transient ischaemic attack, defined as a fully reversible neurological defect lasting <24 h. Prolonged reversible ischaemic neurological deficits were defined as events lasting >24 h and <3 weeks. All the stroke outcomes included in this study were diagnosed by a neurologist, and in most cases, brain CT-scan or MRI was used for lesion assessment. The study excluded patients with diffuse postoperative brain encephalopathy presented as delirium, confusion, prolonged alteration in mental status and agitation in the immediate postoperative period, which could be related to circulatory bypass time and might also reflect longer exposure to anaesthesia [6].

The perioperative stroke risk variables included the 13 variables proposed by the NNECDSG prediction model, clustered into nine major factors: age, female gender, diabetes mellitus, arterial hypertension, vascular disease (including all extracardiac vascular disease manifestations such as intermittent claudication, amputation, absent pedal pulse and/or lower extremity ulcers; previous surgery and/or percutaneous intervention on the abdominal, thoracic aorta, lower extremity and supra-aortic vessels), previous stroke events (ictus with/without residual neurological defect or transient ischaemic attack), chronic renal failure (known glomerular filtration impairment <60 ml/min/1.73 m², patients requiring dialysis and/or preoperative creatininemia ≥2 mg/dl), preoperative cardiac failure (left ventricular ejection fraction <40% by echocardiography or ventriculography and/or preoperative NYHA functional class III–IV), and non-elective priority of surgery (urgency: operation required within 24 h to minimize the chance of further clinical deterioration; or emergency: there should be no delay in providing surgical intervention). The presence of preoperative atrial fibrillation was also considered. The NNECDSG score was calculated for each of the patients and was used to validate our multicentre study. In its assessment, the vascular disease variable was computed by the combination of the above definition plus the presence of a previous stroke event as defined in [4]. The intraoperative variable collected was the performance of CABG surgery under the off-pump beating heart (OPCAB) or cardioplegic cardiopulmonary bypass (CPBCAB) technique.

Statistical analysis

The continuous variables were presented as mean ± standard deviation (SD) and categorical variables were shown as a percentage (%). All group comparisons were unpaired. Continuous variables were compared using the Student's t-test, and categorical variables were compared using χ² analysis or Fisher's exact test, as appropriate. Age was considered both as a categorical clustered and continuous variable. A two-tailed value of P < 0.05 was considered as statistically significant.

The entire data set was divided randomly into a ‘training’ set used to develop the clinical prediction rule and a ‘test’ set to assess the performance of the rule when applied to data different from that from which it was developed. We first performed a univariate analysis, including preoperative variables and OPCAB/CPBCAB performance. Considering perioperative neurological outcome, variables with a significant level of P < 0.05 in the univariate analysis were included in a stepwise multivariate logistic regression model, to determine the independent perioperative stroke predictors. A predictive score model was developed including former independent risk factors, weighting their strength of association with neurological outcome to produce a simple and intuitive model. The predictive accuracy of the multivariate model, NNECDSG and the score scheme developed was assessed by means of the area under the receiver operating curve (c-statistic). The Hosmer–Lemeshow (H–L) goodness-of-fit test was also applied to calibrate these models.

CHAIDS (Chi² Automatic Interaction Detector) analysis, a statistical multi-way tree algorithm, was used to build profiles to help us better identify groups of perioperative stroke risk, integrating the relationships between the predictive score system and OPCAB/CPBCAB performance.

A Statistical Stressing Test to test the predictive score system was developed by randomly selecting 50 groups of 1000 patients from the entire data set. The c-statistic was calculated for each individual analysis of the groups.

The Statistical Package for the Social Sciences (IBM^® SPSS^® Statistic version 19.0; SPSS, Inc., Chicago, IL, USA) was used for the statistical analysis. The study's protocol was approved by the Human Subjects Review Committee of each of the participating institutions.

RESULTS

Patient cohort

A total of 28 296 patients were included in the study. Single-institution accrual was variable (range 157–3060 patients). Of these, 1949 (6.89%) patients had missing preoperative variables and were excluded from the analysis. Complete information was available for 26 347 (93.11%) patients. The data set was divided randomly into a ‘training’ set (n = 13 096, 49.7%) and a ‘test’ set (n = 13 246, 50.27%) to develop and check the predictive score, respectively.

The overall incidence of perioperative stroke was 1.38%, with values ranging from 0.3 to 2.5% among the participating centres. The characteristics of the patients studied are displayed in Table 1. The NNECDSG score showed very good discriminatory accuracy in predicting perioperative stroke in our cohort (c-statistic = 0.69 (0.67–0.72), P < 0.0001; H–L test: χ² = 8.32, P = 0.40).

Table 1:

Baseline characteristics of global ‘training’ and ‘check’ groups

	Group	‘Training’ group	‘Check’ group	P
	(n = 26 347)	(n = 13 096)	(n = 13 246)
Age	64.88 ± 9.92	64.92 ± 9.82	64.83 ± 10.01	0.46
55–59	3230 (12.26%)	1615 (12.33%)	1615 (12.19%)	0.74
60–64	4228 (16.05%)	2131 (16.27%)	2097 (15.83%)	0.34
65–69	4938 (18.74%)	2492 (19.03%)	2446 (18.47%)	0.25
70–74	5106 (19.38%)	2536 (19.36%)	2570 (19.40%)	0.94
75–79	3605 (13.68%)	1818 (13.88%)	1787 (13.49%)	0.36
≥80	995 (3.78%)	458 (3.49%)	537 (4.05%)	0.08
Female	4689 (17.79%)	2323 (17.74%)	2366 (17.86%)	0.79
Diabetes mellitus	9654 (36.64%)	4856 (37.08%)	4798 (36.22%)	0.15
Arterial hypertension	15233 (57.81%)	7553 (57.67%)	7680 (57.98%)	0.61
Previous stroke	1282 (4.87%)	630 (4.81%)	652 (4.92%)	0.69
Cardiac failure	4000 (15.18%)	1996 (15.24)	2004 (15.13%)	0.59
Non-elective surgery	3461 (13.14%)	1698 (12.97%)	1763 (13.31%)	0.44
Vascular disease	4148 (15.74%)	2100 (16.04%)	2048 (15.46%)	0.20
Chronic kidney failure	2277 (8.64%)	1126 (8.59%)	1151 (8.69%)	0.79
Preoperative atrial fibrillation	1098 (4.17%)	544 (4.15%)	554 (4.18%)	0.93
Off-pump coronary bypass surgery	10634 (40.36%)	5265 (40.20%)	5369 (40.53%)	0.59
Postoperative stroke	365 (1.39%)	174 (1.33%)	191 (1.44%)	0.46

Statistical significance of differences between ‘training’ and ‘check’ groups.

Identification of independent preoperative risk factors of stroke

Univariate analysis was carried out on the ‘training’ group (Table 2). Off-pump surgery was not related to a lower rate of perioperative stroke (1.31% CPBCAB vs 1.38% OPCAB, P > 0.05). The multivariate analysis showed cardiac failure (OR = 3.64, P < 0.0001), non-elective surgery (OR = 2.32, P < 0.0001), vascular disease (OR = 1.37, P = 0.007) and chronic kidney failure (OR = 6.78, P < 0.0001) as independent risk factors for perioperative development of a stroke after CABG surgery (Table 3). The H–L goodness-of-fit test of the multivariate model showed good adjustment (χ² = 1.93, P = 0.93) and c-statistic predictive accuracy of 0.77 (0.73–0.82), P < 0.0001.

Table 2:

Univariate analysis in terms of perioperative neurological outcome performed in ‘training’ group

	No postoperative stroke	Postoperative stroke	P
	(n = 12 925)	(n = 174)
Age	64.92 ± 9.82	65.16 ± 9.66	0.75
55–59	1594 (12.33%)	21 (12.07%)	0.92
60–64	2096 (16.22%)	35 (20.11%)	0.18
65–69	2462 (19.05%)	30 (17.24%)	0.62
70–74	2509 (19.41%)	27 (15.52%)	0.21
75–79	1792 (13.86%)	26 (14.94%)	0.67
≥80	449 (3.47%)	9 (5.17%)	0.22
Female	2297 (17.77%)	26 (14.94%)	0.37
Diabetes mellitus	4774 (36.94%)	82 (47.13%)	0.007
Arterial hypertension	7433 (57.51%)	120 (68.97%)	0.003
Previous stroke	614 (4.75%)	16 (9.19%)	0.01
Cardiac failure	1919 (14.85%)	77 (44.25%)	<0.0001
Non-elective surgery	1648 (12.75%)	50 (28.74%)	<0.0001
Vascular disease	2046 (15.83%)	54 (31.03%)	<0.0001
Chronic kidney failure	1052 (8.14%)	74 (42.53%)	<0.0001
Preoperative atrial fibrillation	534 (4.13%)	10 (5.75%)	0.25
Off-pump coronary bypass surgery	5193 (40.18%)	72 (41.38%)	0.76

Table 3:

Multivariate analysis performed in ‘training’ group in terms of perioperative neurological outcome

	P	OR (95% CI)
Diabetes mellitus	0.28	1.19 (0.87–1.63)
Arterial hypertension	0.43	1.15 (0.82–1.61)
Previous stroke	0.21	1.42 (0.82–2.45)
Cardiac failure	<0.0001	3.64 (2.66–4.98)
Non-elective surgery	<0.0001	2.32 (1.64–3.29)
Vascular disease	0.007	1.37 (1.07–1.94)
Chronic kidney failure	<0.0001	6.78 (4.91–9.38)

OR: odds ratio; CI: confidence interval.

Development and validation of predictive PACK₂ score

An outcome risk-stratification scale was developed from the logistic regression model. Table 4 indicates the specific punctuation assignments used in calculating the PACK₂ score.

Table 4:

Determination of the PACK₂ score

Variable	PACK2 score
Priority of surgery
Non-elective	1
Elective	0
Peripheral arteriopathy
Yes	1
No	0
Preoperative cardiac failure and/or LVEF < 40%
Yes	1
No	0
Chronic kidney failure and/or preoperative creatininemia >2 mg/dl
Yes	2
No	0
Total PACK2 score	0–5

In the ‘training’ group, predictive accuracy for the NNECDSG scheme was c-statistic = 0.70 (95% CI 0.67–0.74), P < 0.0001; H–L test: χ² = 8.11, P = 0.42; and for PACK₂ score c-statistic = 0.77 (95% CI 0.73–0.81), P < 0.0001; H–L test: χ² = 0.16, P = 0.69.

In the ‘test’ group, predictive accuracy for the NNECDSG scheme was c-statistic = 0.69 (95% CI 0.65–0.73), P < 0.0001; H–L test: χ² = 4.13, P = 0.85; and for PACK₂ score c-statistic = 0.77 (95% CI 0.73–0.80), P < 0.0001; H–L test: χ² = 1.39, P = 0.24.

The Statistical Stressing Test confirmed the good predictive accuracy of the PACK₂ score, being c-statistic >0.8 in 36%, >0.7 in 84% and non-significant in 8% of the random sets of 1000 patients selected.

Clinical application of PACK₂ score

It is noted that 59.66% of patients were categorized as score 0 (n = 15719), 25.54% (n = 6728) as 1 point, 9.99% (n = 2631) as 2 points, 3.61% (n = 950) as 3 points, 1.03% (n = 272) as 4 points and 0.18% (n = 47) as 5 points. The stroke rates were: 0.48, 1.20, 3.57, 6.53, 15.44 and 21.28%, respectively, P < 0.0001 (Fig. 1).

Figure 1:

Distribution of percentage of perioperative stroke according to range of PACK₂ score punctuations. The black fine line represents exponential tendency of increasing incidence of the perioperative stroke (y = 0.266 e^0.78; R² = 0.978).

Decision tree CHAID analysis (Fig. 2) was performed by applying the PACK₂ score to stratify the preoperative risk of developing a perioperative stroke, in order to assess the benefit of OPCAB vs CPBCAB surgery. Zero and one points were analysed independently, showing no statistical difference in the perioperative stroke rate between off-pump and on-pump surgery (0 points: 0.6 vs 0.4%, P = 0.311; 1 point 1.3 vs 1.1%, P = 0.505, respectively). However, off-pump significantly reduced the perioperative stroke rate for patients with a PACK₂ score ≥2, (4.1 vs 6.4%, P = 0.002).

Figure 2:

Decision tree to stratify perioperative stroke risk in the overall CABG series (n = 26 347), attending PACK₂ score and off-pump technique. OPCAB: off-pump coronary bypass surgery, CPBCAB: coronary bypass surgery in cardiopulmonary bypass, df = degree of freedom.

DISCUSSION

This multicentre study developed a clinical grading scale, the PACK₂ score, to predict perioperative stroke risk in isolated CABG surgery. Our results suggest the PACK₂ score 1) has good accuracy and discriminatory power, with a c-statistic = 0.77 (95% CI 0.74–0.79), P < 0.0001, comparable with other specific scales such as the NNECDSG score; and 2) is simple enough to be used with only four preoperative variables.

Several prognostic perioperative stroke models had previously been developed and validated [2], and as these models have been shown to be highly accurate in predicting outcomes, we opted to use their clinical predictive variables as enrolment criteria for this study. However, several of them use complex scales, or complex algebraic equations in outcome prediction, and none has been simplified into a standard clinical grading scale analogous to the PACK₂ score. This scale contains only four preoperative variables and therefore balances a compromise between simplicity and the accuracy of its outcome predictions. To strike an appropriate balance between these two factors, the main purpose of the grading scale must be considered. The PACK₂ score was simply developed to provide a standard assessment tool that can be easily and rapidly applied during preoperative clinical assessment of CABG patients.

How might the PACK₂ score be used? Perioperative stroke risk in CABG patients is often a fundamental question, and reported scales are often used to provide initial information regarding this issue [7, 8]. While prognostication is undoubtedly important to assess surgical benefits and risks, any attempt to predict a precise outcome could lead to mistakes. A scale such as the PACK₂ score should therefore be incorporated as part of risk stratification for perioperative CABG stroke studies and patient decision management, but not as a precise risk predictor of outcomes. Our aim was, therefore, to show its utility as a clinical tool in assessing stroke risk in CPBCAB surgery. In this study, cases with a score of ≥2 had a reduced incidence of perioperative stroke when undergoing OPCAB vs CPBCAB (P = 0.002).

Several limitations involved in this study should be noted. First, it is limited by its dependence on a retrospective observational study, and the conclusions derived from it are necessarily limited in application. However, we addressed this limitation by enrolling patients from multiple institutions, thereby minimizing any distortions due to the effects of different surgical, anaesthetic, perfusion-related and medical specific practices at the different centres. Secondly, OPCAB performance was not assessed in this study under ‘touch’ or ‘non-touch’ aorta schemes. As ‘touch’ schemes have been reported to limit the benefits of OPCAB over CPBCAB in terms of perioperative stroke incidence, the statistical differences found in high-risk patients may support the thesis of the potential reduction of neurological events with OPCAB surgery. Thirdly, the neurological findings were assessed at each site of origin and not by a single neurologist performing all the preoperative and postoperative examinations, so there could be significant variations in clinical practice, and thus diagnosis, among the 21 centres. Finally, as perioperative diffuse brain encephalopathy was excluded, this may potentially have underestimated the real prevalence of small perioperative strokes; however, other studies have reported diffuse brain encephalopathy, mainly related to intraoperative factors such as cardiopulmonary bypass and general anaesthesia times [6].

CONCLUSIONS

On the basis of data from 21 Spanish medical centres with 26 347 patients enrolled, we conclude that the PACK₂ score is an easy-to-use classification scheme that stratifies the risk of perioperative stroke in CABG patients. Further validation from other centres will be necessary before this scale can be considered as a useful tool for planning prognostic and therapeutic strategies to prevent perioperative stroke.

APPENDIX. CONFERENCE DISCUSSION

Dr A. Badreldin (Koblenz, Germany): First of all, can you tell us what the difference is between your PACK2 score and the STS score, which tells us in a logistic calculation and percentage, the incidence or the possibility of incidence of stroke postoperatively?

Secondly, regarding your variables, the four variables are already included in the STS scoring system together with other variables; accordingly, the STS score should have more accuracy. Thirdly, we see in the c-statistics of the STS score that were validated externally in independent patient subsets, that the STS score had a better performance than the 0.6 or 0.7 results of your ROC analysis. Can you comment on these points?

Dr Martín: We based our score on an excellent predictive model which is the Northern New England scale. And the four variables are also included in the EuroSCORE, the STS and many other predictive scores. However, the main point of our score is that it shows an excellent predictive accuracy which you can calculate from memory at the moment when you have a knife in the hand and you have not done it before. So at that moment, when you are in a hurry or have not calculated it before, it allows a very simple system of calculation, so you can decide at a glance if you are going to cannulate the patient or if you are going to ask for the stabilizer to perform the coronary surgery.

Dr Badreldin: Actually, your answer was my second comment. I think the main advantage of your score is the simplicity.

Dr Martín: That is it.

Dr Badreldin: And I am very fascinated with simplicity in scoring systems. That is why my idea was that we should work on the existing preoperative scores in cardiac surgery as, for example, Dr Nashef did last year with the EuroSCORE and developed EuroSCORE II. Your score might be a wonderful bedside test, yes, I agree with you totally, but I think we could draw our attention a little bit more to improving the existing scores rather than creating more new scores. Then we will lose interest. This is my point.

Dr R. Osnabrugge (Rotterdam, Netherlands): I only have a minor question. In your early slides, you showed that you excluded some patients because of missing data in some variables. Do you think that this has introduced bias, and have you considered multiple imputation techniques, for instance, to correct for that?

Dr Martín: We do not think so. They represented a small percentage of the global set, and we preferred to have good quality of data. We did not consider it important to exclude 2000 patients out of more than 28 000 because of missing data on them.

Dr Osnabrugge: Alright. So it was a really small portion?

Dr Martín: Yes.

Dr Osnabrugge: Fine.