Limited utility of algorithms predicting blood transfusions

Abstract

Background

Prediction of transfusion is presumed to reduce wastage rates in pre-operative autologus blood donation (PABD) and unnecessary providing and cross-matching in allogeneic transfusion. The clinical utility of published algorithms in predicting transfusions was analysed.

Materials and methods

In a cohort of 195 patients undergoing total hip arthroplasty, after PABD, expected transfusion needs were predicted with two published algorithms (A and B). The algorithms were then compared to actual transfusions. Assumptions and formulae of these algorithms were varied in an attempt to improve their prognostic utility.

Results

The optimal variation of A resulted in allogeneic transfusions (PABD setting) or uncross-matched transfusions (allogeneic setting) of 27.3%, and a wastage rate of autologous units or unnecessary cross-matching of 73.8%, compared to 33.3% and 76.6%, respectively, for the original algorithm. The original version of algorithm B resulted in (allogeneic) transfusions of 78.8%, and a wastage rate or unnecessary cross-matching of 46.2%. The former could be improved by a variation of the algorithm to 69.7%. Comparing the optimal variations of both algorithms, the more elaborate algorithm A reduced overall transfusion risk significantly better (P =0.001). The two algorithms were not statistically different in reducing resource consumption (P =0.09).

Discussion

Although the prognostic utility of algorithm A was significantly better for reducing overall transfusion risk, both algorithms were unable to meaningfully identify patients who would benefit from PABD or cross-matching. The algorithms could not increase the percentage of PABD patients transfused, or the percentage of cross-matched patients transfused in the allogeneic setting. Furthermore, they could neither reduce transfusion risk nor resource consumption.

Introduction

In the wake of the human immunodeficiency virus epidemic, pre-operative autologous blood donation (PABD) became a popular way to reduce allogeneic transfusions and consequently the risk of transfusion-transmitted viral infections. At our hospital, in the 1990s, the number of PABD rose steadily to a peak of about 1,050 collections in 1996. However, from the outset there was considerable waste of autologous red cell units that were not transfused back into the patient, with the wastage rate being approximately 55%¹. In the period from 2002 to 2004 this rate had increased to 65%. Given that the cost-effectiveness of PABD programmes relies heavily on the number of discarded units² we were very concerned about the cost-effectiveness of our programme, and measures to reduce the wastage rate were urgently needed. It was necessary to predict better which patients actually needed transfusions upon surgery and to restrict PABD to those patients.

Several algorithms to predict a given patient’s need for transfusion upon surgery have been published. Two algorithms have been particularly developed for PABD programmes^3,4. However, these algorithms have never been thoroughly validated in different settings. It was, therefore, unclear whether implementation of any of the algorithms would actually serve our goal of reducing the wastage rate of our PABD programme. The surgical blood order equation (SBOE) approach⁴ has only recently been analysed in the setting of femoral fracture surgery and was found to reduce inventory management costs⁵.

The aim of this study was to validate the above mentioned algorithms in the context of our PABD programme. We chose a well-defined and well-documented cohort of our PABD patients who had been entered into a multicentre, randomised trial⁶.

It should be acknowledged that the results of this study can also be interpreted in the framework of pre-transfusion provision and cross-matching of allogeneic blood because they can show whether the algorithms analysed are able to determine which patient might need pre-operative provision and possible cross-matching of blood and which might not.

Materials and methods

Patients

Patients undergoing primary total hip arthroplasty and those eligible for autologous blood donation were entered into a multicentre, randomised trial from January 2002 until June 2005⁶. Usually on two occasions, 3 to 5 weeks prior to surgery, 500 mL of blood were collected on each visit. The blood was stored as whole blood, either leucocyte-depleted or not which was the intervention in the trial. The subgroup of patients recruited in the Marburg centre was analysed for this study. After the trial had finished, two algorithms were retrospectively applied to these patients based on available pre-donation and pre-operative data. These data were complete in the sense that the algorithms could be applied as intended by their authors without having to make assumptions for any of the parameters needed. The algorithms predicted whether a patient would have needed a blood transfusion upon surgery. This prediction was compared with the actual transfusion requirements of the given patient. The peri-operative period we analysed was defined as the time interval from the day of surgery until discharge (after an average stay of 14 days in hospital⁶). Blood samples for haemoglobin measurements were taken at regular time points post-operatively: in the recovery room, on post-operative days 1, 3, and 7, and on post-operative day 15 (or day of discharge if this was earlier). Additional haemoglobin measurements were carried out if considered necessary on clinical grounds.

Methods

Two published algorithms, referred to as algorithm A³ and B⁴ were evaluated. Details of the algorithms, in particular the formulae for the different calculation steps, are given in the original publications of the articles^3,4. In short, algorithm A works as follows:

• - calculate the average blood loss (as volume of red cells lost) that can be expected for the surgical procedure the algorithm is to be applied to;

• - calculate the patient’s blood volume;

• - calculate the tolerable blood loss (as volume of red cells lost) of the patient based on the minimally acceptable haematocrit of this patient;

• - calculate the probable red cell demand of this patient as the difference of the expected minus the tolerable blood loss; if this difference is positive the patient is expected to need a transfusion.

Expected blood loss (step 1) was calculated as the average of the calculated blood loss of all patients included in the study with necessary data. The patient’s blood volume (step 2) was calculated using the formula⁷ proposed by the authors of the algorithm. In an attempt to improve algorithm A we further applied several other published formulae for determining blood volume⁸. For calculation of the tolerable blood loss (step 3) we applied two different transfusion triggers, i.e. minimal acceptable haematocrit values, simulating a more liberal transfusion strategy (haematocrit <0.27 L/L) and stricter one (haematocrit <0.24 L/L). Probable red cell need was then calculated (step 4) and divided by the average red cell content of a red cell concentrate (i.e. 162 mL in our case, based on recent quality control data), resulting in the number of red cell concentrates the patient would have needed.

Algorithm B is much easier to perform. It is based on the so-called “surgical blood order equation” (SBOE). The transfusion need expressed as number of red cell concentrates needed is calculated as the blood loss (expressed as haemoglobin loss measured in g/L) minus the difference of pre-operative minus the minimally tolerable haemoglobin value. This algorithm assumes that the transfusion of one red cell concentrate increases the haemoglobin level of the patient by 1 g/L. However, this simplifying assumption roughly applies only to patients with a body weight of about 70 kg. Another simplifying assumption concerns the calculation of the average blood loss through surgery. We tried to “optimise” this algorithm by applying the more accurate formula from algorithm A for the calculation of the average blood loss.

Both algorithms were carried out as originally described. For both algorithms we decided that if the predicted transfusion need was <0.5 red cell concentrates the patient would not have needed a transfusion and was categorised accordingly.

One major adjustment had to be made in order to satisfy the assumptions that underlie both algorithms. The algorithms are to be applied to patients before PABD is carried out. However, in this study they were applied after PABD had been performed. This could have, in principal, distorted the results. Patients undergoing PABD are more likely to be transfused than patients not undergoing PABD^9,10. This is mainly due to the fact that many patients who donate autologous blood cannot completely restore their pre-donation haemoglobin level until surgery. This means that those patients might be transfused only because they had donated autologous blood in the pre-operative phase, leading to a lower pre-operative haemoglobin than they would have had without PABD. This sort of bias can, though, only play a role in patients who had actually been transfused. In order to eliminate this potential source of bias we checked all transfused patients to see whether their pre-operative haemoglobin was lower than their pre-donation haemoglobin. If this was the case, we corrected, i.e. increased, the haemoglobin value that triggered the transfusion, by the difference between the pre-operative and pre-donation haemoglobin. If the resulting haemoglobin was still below the transfusion trigger, the patient was categorised as “transfusion needed”, otherwise he would have been categorised as “no transfusion needed”. For example, assume a patient had a pre-donation haemoglobin of 144 g/L and a preoperative haemoglobin of 136 g/L, this indicates a loss of red cell mass of 8 g/L due to PABD. Assume further that the transfusion trigger for this patient was 90 g/L (i.e. a haematocrit of 0.27 L/L) and the lowest haemoglobin (during surgery or in the post-operative phase) was 86 g/L, resulting in transfusion. We then added the lost 8 g/L to the 86 g/L, arriving at a lowest haemoglobin of 94 g/L, which means that the patient would not have been transfused had he not donated autologous blood because his lowest haemoglobin would now have been above his transfusion trigger. This patient was then categorised as “no transfusion needed” even though he was transfused. By this procedure we could eliminate any bias potentially introduced by PABD and ensure that all patients categorised as “transfusion needed” would still have been transfused even if they had not donated autologous blood.

A patient was considered “transfused” if he or she received any kind of red cells, whether autologous or allogeneic. However, since autologous blood was available for all patients and was transfused first, the transfusion of the first autologous unit resulted in the categorisation of the patient as “transfusion needed” (once a potential haemoglobin loss due to PABD had been taken into account, see above).

Statistical analysis

We constructed 2×2 tables and calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for the different variations of the algorithms. An exemplary 2×2 table is presented in Table I.

Table I.

Exemplary 2×2 table for calculating the performance characteristics of the transfusion prediction algorithms.

		Clinical outcome: transfusion needed
		Yes	No
Algorithm: transfusion predicted	Yes	a	b
		Allogeneic/uncross-matched transfusion prevented	Unnecessary PABD/unnecessary cross-matching performed	a/(a + b) = PPV
	No	c	d
		Allogeneic/uncross-matched transfusion performed	Unnecessary PABD/unnecessary cross-matching prevented	d/(c + d) = NPV
(a + c)/(a + b + c + d) = baseline transfusion risk		a/(a + c) = sensitivity	d/(b + d) = specificity

Several useful pieces of information can be obtained from such a table: the parameter (1 – sensitivity) (cf. cell “c” in Table I) reflects the risk of allogeneic transfusion in the PABD setting and the risk of uncross-matched transfusion in the allogeneic setting, thus representing overall transfusion risk. In addition, the parameter (1 – PPV) (cf. cell “b” in Table I) reflects the wastage rate in the PABD setting and unnecessary cross-matching in the allogeneic setting, thus representing (financial) resource consumption.

Since it is known that PPV and NPV are dependent on the prevalence, i.e. the baseline risk in a prediction model, we recalculated these values for algorithm A in a hypothetical scenario assuming a transfusion risk of 80%, based on the already established sensitivity and specificity for the algorithm at a liberal transfusion threshold of a haematocrit <0.27 L/L. The rationale behind this analysis was that it was expected that a higher transfusion risk (i.e. a higher prevalence in epidemiological terms) might result in a higher and more clinically useful PPV.

A χ² test was used for the comparison of the performance characteristics of the two algorithms.

Results

The Marburg trial centre contributed 225 patients to the randomised trial although 12 patients were later excluded from the analysis⁶. For this study 195 data sets had complete data allowing application of the algorithms. Baseline data of the 195 patients are presented in Table II. The average reduction of the pre-operative haemoglobin level due to PABD was 13 g/L.

Table II.

Baseline characteristics of study patients.

Gender	97 men/98 women
Age [years]*	58.5 (9.7)
Height [cm]*	171 (9.0)
Weight [kg]*	80.3 (14.2)
Body mass index	27.5 (4.2)
Pre-donation haemoglobin [g/L]*	147 (11.2)
Pre-operative haemoglobin [g/L]*	134 (12.6)
ASA status
1	36
2	137
3	18

Legend

^*Data presented as mean (standard deviation).

The data from 144 patients could be used to calculate the average blood loss for hip arthroplasty. The calculated average blood loss was 710 mL. This value was necessary for algorithm A. For algorithm B this number needed to be transformed into the amount of haemoglobin lost which was done as described in an earlier paper¹¹. The average amount of haemoglobin lost was determined in this way to be 50 g/L. When a minimal tolerable haematocrit of 0.27 L/L was applied, 33 patients (16.9%) were classified as “transfusion needed”, this number therefore represents the baseline risk of being transfused. When the stricter threshold of 0.24 L/L was applied, only 23 patients (11.8%) were classified as “transfusion needed”.

The results for algorithm A are listed in Table III. How these figures were calculated can be discerned from an example of a 2×2 table shown in Table IV. The formula used in the original article³ to calculate blood volume is designated as “SA2”. The optimal variation of the algorithm resulted in allogeneic transfusions (PABD setting) or uncross-matched transfusions (allogeneic setting) in 27.3% of patients, and a wastage rate of autologous units (PABD setting) or unnecessary cross-matching (allogeneic setting) of 73.8%, compared to 33.3% and 76.6%, respectively, for the original algorithm (blood volume formula “SA2” in Table III). As expected, with a more liberal transfusion strategy, the PPV increased, thereby reducing wastage rate in the PABD setting and unnecessary cross-matching in the allogeneic setting (columns 2 and 4 of Table III). However, the absolute values do not meaningfully inform the decision on whether to offer PABD to a certain patient or not. The performance of the algorithm can only be slightly improved by using different formulae for the calculation of blood volume not proposed in the original article³. The percentage of allogeneic transfusions or uncross-matched transfusions is high for both transfusion triggers and does not inform clinical decision making either. The fact that it is higher for the stricter transfusion trigger may be counterintuitive but is due to the fact that the denominator for this number is the baseline risk of being transfused which is quite different for the two transfusion triggers (see above).

Table III.

Performance characteristics of algorithm A³. As an example, for the figures in bold the underlying 2×2 table is shown as Table V.

Transfusion trigger (haematocrit)	0.24 L/L		0.27 L/L
Blood volume formula* (referenced in 8)	Unnecessary PABD (= wastage rate)/unnecessary cross-match [%]	Allogeneic/uncross-matched transfusion [%]	Unnecessary PABD (= wastage rate)/unnecessary cross-match [%]	Allogeneic/uncross-matched transfusion [%]
W1	85.0	60.9	77.3	33.3
W2	81.8	65.2	75.7	48.5
W3a	83.6	60.9	75.3	39.4
W3b	81.4	65.2	73.8	48.5
SA1a	84.6	56.5	77.1	27.3
SA1b	85.1	56.5	78.1	30.3
SA2	84.1	56.5	76.6	33.3
SA3	83.3	56.5	75.9	36.4
SA4	85.3	56.5	78.5	30.3

Legend

^*W: body weight-based, SA: body surface area-based blood volume calculation.

Table IV.

A 2×2 table for the calculation of the performance characteristics of the original version of algorithm A³ for the 0.24 L/L transfusion trigger.

		Clinical outcome: transfusion needed
		Yes	No
Algorithm: transfusion predicted	Yes	10	53	PPV = 10/63 = 15.9%
	No	13	119	NPV = 119/132 = 90.2%
Baseline transfusion risk = 23/195 = 11.8%		Sensitivity = 10/23 = 43.5%	Specificity = 119/172 = 69.2%	N=195

Legend 13 out of 23 patients (or 1 – sensitivity = 1 – 0.435) = 0.565 is the proportion of patients transfused with allogeneic blood because of wrongly withheld PABD or with uncross-matched transfusions. This value (as a percentage) is presented and highlighted in column 3 of Table III. Likewise, 53 out of 63 patients (or 1 – PPV = 1 – 0.159) = 0.841 is the proportion of patients without the need for transfusions for whom the algorithm predicted transfusion. Thus, this figure represents unnecessary PABD (= wastage rate) or unnecessary cross-matching. This value (as a percentage) is presented and highlighted in column 2 of Table III.

In the hypothetical scenario with an 80% transfusion rate (i. e. classification of 156 patients as “transfusion needed”) the PPV of algorithm A for a transfusion threshold of haematocrit <0.27 L/L would increase to 90%. However, the NPV would fall to 34%, and 52 patients (26.7% of all patients) of the total 79 patients for whom no transfusion was predicted would need transfusions and would thus have benefited from PABD.

The results of the much less sophisticated algorithm B are shown in Table V. Again, how these figures were calculated can be discerned from an example of a 2×2 table shown in Table VI. The original version of the algorithm resulted in allogeneic transfusions or uncross-matched transfusions in 78.8% of patients and a wastage or unnecessary cross-matching rate of 46.2%. Only the former could be improved by the variation of the algorithm to 69.7%. These performance characteristics could only be reached with a transfusion threshold of a haematocrit <0.27 L/L; a more stringent transfusion threshold of a haematocrit <0.24 L/L led to even more unfavourable results.

Table V.

Performance characteristics of algorithm B⁴. As an example, for the figures in bold the underlying 2×2 table is shown as Table VI.

Transfusion trigger (haematocrit)	0.24 L/L		0.27 L/L
	Unnecessary PABD (= wastage rate)/unnecessary cross-match [%]	Allogeneic/uncross-matched transfusion [%]	Unnecessary PABD (= wastage rate)/unnecessary cross-match [%]	Allogeneic/uncross-matched transfusion [%]
Algorithm variation
Original	0	95.7	46.2	78.8
“Optimised”	50	91.6	60.0	69.7

Table VI.

A 2×2 table for calculating the performance characteristics of the original version of algorithm B⁴ for the 0.27 L/L transfusion trigger.

		Clinical outcome: transfusion needed
		Yes	No
Algorithm: transfusion predicted	Yes	7	6	PPV = 7/13 = 53.8%
	No	26	156	NPV = 156/182 = 85.7%
Baseline transfusion risk = 33/195 = 16.9%		Sensitivity = 7/33 = 21.2%	Specificity = 156/162 = 96.3%	N=195

Legend 26 out of 33 patients (or 1 – sensitivity = 1 – 0.212) = 0.788 is the proportion of patients transfused with allogeneic blood because of wrongly withheld PABD or with uncross-matched transfusions. This value (as a percentage) is presented and highlighted in column 5 of Table IV. Likewise, 6 out of 13 patients (or 1 – PPV = 1 – 0.538) = 0.462 is the proportion of patients without the need for transfusions for whom the algorithm predicted transfusion. Thus, this figure represents unnecessary PABD (= wastage rate) or unnecessary cross-matching. This value (as a percentage) is presented and highlighted in column 4 of Table IV.

Comparing the optimal variations of both algorithms, the more elaborate algorithm A reduced the risk of allogeneic transfusions or uncross-matched transfusions significantly better (P =0.001). The two algorithms were not statistically different in reducing resource consumption (P =0.09).

Discussion

The results of this study demonstrate that published algorithms predicting transfusion need cannot be easily adopted in different clinical settings even if these settings include the surgical procedure the algorithms were developed for.

Although the prognostic utility of algorithm A was significantly better for reducing the risk attributed to transfusion, both algorithms were unable to meaningfully identify patients who were to benefit from PABD or cross-matching. Furthermore, the algorithms could not increase the percentage of PABD patients transfused or the percentage of transfused patients with cross-matching in the allogeneic setting. In summary, the two algorithms could neither reduce transfusion risk nor resource consumption.

The reason why the prognostic utility of algorithm A was better than that of algorithm B might not only be that the former was more elaborate. Algorithm A takes into account the first 5 post-operative days, whereas algorithm B can strictly only be applied to the first 24 hours following surgery. Given the fact that a lot of patients reach their nadir haemoglobin value a couple of days after an operation, it is clear that transfusions not only take place in the 24 hours following surgery, but also several days later. Since these transfusions are only accounted for by the formulae of algorithm A, it seems reasonable to infer that this characteristic of algorithm A might explain its better prognostic performance.

Several attempts have been made in different clinical settings to predict transfusion requirements of patients undergoing surgery. However, the development of a clinical prediction rule requires several steps^12–14 before the clinical usefulness¹⁵ of the model can be considered proven. This final proof, also called impact analysis¹⁶, has not actually been demonstrated for any of the models predicting transfusion which we came across.

In a model developed for head and neck surgery¹⁷, pre-operative haemoglobin, surgical technique, and tumour stage were included in the model. Validation of the model required adjustment (“recalibration”) of the original model, which exemplifies that validation is an inevitable step before a model can be introduced into clinical practice. However, to our knowledge, no impact analysis for this model has subsequently been performed. In another study¹⁸, a model was developed for knee and hip arthroplasty patients; only pre-operative haemoglobin level was found to be predictive of transfusion. To our knowledge, no independent validation of this model has been published. Since the transfusion rate in this study was approximately 25% and the model variable is also part of the prediction algorithms we studied, a prospective validation would most likely have produced results comparable to those we determined for the liberal transfusion triggers in our study.

Other approaches to reduce unnecessary provision and cross-matching of blood in patients undergoing elective surgery by including patient-specific factors in the models¹⁹ have also not been convincing because they cannot satisfactorily identify patients who do not need a transfusion²⁰.

Clinical prediction rules usually have the best performance in clinical settings in which the probability of the events to be predicted lies around 50%²¹. In one study in cardiac surgery patients²² with a transfusion rate of around 40% a prediction model proved quite useful in a validation group by reducing the number of cross-matches by 30% without leading to under-provision of red cell units. However, the model was only internally and retrospectively validated which might impair its external validity, i.e. its generalizability to other hospitals²³.

Another problem which was not adequately addressed in any of the discussed studies concerns the question of what the model is intended to achieve. If the goal is to reduce the risk that accompanies transfusion, one has to focus on the sensitivity of the model because the aim is to reduce the number of allogeneic transfusions (in the PABD setting) or the number of uncross-matched transfusions (in the allogeneic setting) in the patients who are to be transfused anyway, i.e. one wants to reduce the number of patients in cell “c” of Table I in relation to cells “a + c” (all transfused patients). This goal might be referred to as medical. One should bear in mind that the baseline risk of being transfused is “fixed” and is primarily dependent on the surgical procedure and its circumstances. If one wants to reduce resource consumption, i.e. reduce the wastage rate in the PABD setting or the number of unnecessary cross-matches in the allogeneic setting, one needs to focus on the PPV of the model, because one wants to reduce the number of patients in cell “b” of Table I in relation to cells “a + b”, i.e. all patients having donated autologous blood or all patients having red cells cross-matched. This goal might be referred to as economic or financial. The results of this study clearly demonstrate that these goals cannot be reached concurrently: they are usually conflicting and one has to decide beforehand which goal to follow before any attempts to improve a model are made. Future studies developing or analysing transfusion prediction models need to address these issues explicitly.

Although this was not a primary objective of this study, the results also cast much doubt on the cost-effectiveness of PABD in a clinical setting with a baseline transfusion probability of around 10%, which is currently still the threshold above which, in Germany, patients need to be informed about autologous transfusion techniques²⁴. The analysis showed that the average haemoglobin loss caused by donation amounted to the equivalent of more than one red cell concentrate and more than 50% of the actually transfused patients would not have needed any transfusion at all if they had completely restored their pre-donation haemoglobin level by the time of surgery. This is also the reason why actual wastage rates are lower than the wastage rates predicted by the algorithms. These results are also in accordance with a recent meta-analysis of the efficacy of PABD programmes¹⁰. If one wanted to increase the cost-effectiveness of PABD programmes by accepting a certain proportion of allogeneic transfusions, the two algorithms would have their merits, because they would have avoided PABD in a large part of patients in our study cohort (69% for the original version of algortihm A, cf. Table IV; 96% for the original version of algorithm B, cf. Table VI). Only 10–15% of these patients would have been transfused.

Conclusion

Published clinical prediction rules need to be validated in the clinical settings they are supposed to be applied to. In surgery with a low peri-operative transfusion risk, for example primary total hip arthroplasty, the two algorithms analysed do not inform clinical decision-making with respect to which patients should undergo PABD, or, if no PABD programme is established, which patients need red cell cross-matching prior to surgery.