Performance of predictive models of survival in horses undergoing emergency exploratory laparotomy for colic

Abstract

Objective

To evaluate previously published predictive survival models in a population of horses undergoing colic surgery in the midwestern United States.

Study design

Retrospective cohort study; single referral hospital.

Animals

A total of 260 horses met the inclusion criteria.

Methods

Medical records of horses undergoing surgical treatment for colic were reviewed. Previously published models were applied to cohort data to predict outcome. Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy for prediction of short‐term survival were calculated.

Results

Single‐variable and multivariable models performed similarly for prediction of survival, with a mean 79% sensitivity (range: 44%–94%), 48% specificity (range: 22%–83%), 63% PPV (range: 56%–72%), 73% NPV (range: 60%–83%), and 64% accuracy (range: 59%–72%). Blood lactate ≤6 mmol/l and the colic severity score (CSS) were highly sensitive for prediction of survival; however, both had poor specificity.

Conclusion

Single‐variable and multivariable predictive models did not perform as well for prediction of survival in the study cohort compared to original reports, suggesting that population‐specific factors contribute to patient survival.

Clinical significance

Predictive models of survival developed in one population may be less reliable when used to predict outcome in horses undergoing colic surgery from an independent population. Additional model testing and refinement using data from multiple surgical centers could be considered to improve prediction of outcome for horses undergoing laparotomy for treatment of colic.

1. INTRODUCTION

The decision to treat or euthanize a horse with severe colic is significantly influenced by the clinical judgment of the attending clinician. This subjective evaluation could result in pre‐ or intraoperative euthanasia of horses that may have survived if surgical intervention was pursued. ¹ Possible reasons for premature destruction may include misconceptions regarding (1) lesion‐specific prognosis; (2) the impact of surgery on future athletic performance; or (3) the effect of age or concurrent disease on recovery. Applying predictive models could augment clinical judgment by providing more objective criteria to help decide if a positive outcome is likely with surgical treatment. However, a predictive model developed in one horse population may not perform well in another horse population and the use of unvalidated predictive models in clinical practice is not recommended. ²

Single variables used as predictors of prognosis in clinical practice are extrapolated from reports of similar case series or retrospective clinical studies. Abnormal values for multiple cardiovascular parameters (heart rate, capillary refill time, packed cell volume) have consistently been associated with perioperative survival. ³ , ⁴ , ⁵ , ⁶ , ⁷ , ⁸ Hematological parameters used to evaluate the risk of mortality include anion gap, L‐lactate, ionized calcium, ionized magnesium, creatinine, and glucose. ⁹ , ¹⁰ , ¹¹ , ¹² , ¹³ , ¹⁴ , ¹⁵ , ¹⁶ Evaluation of peritoneal fluid lactate concentrations and appearance are also suggested to be useful to predict survival of horses with specific types of colic. ⁶ , ⁸ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ The advantage of single variable predictors is that they can be rapidly assessed in the course of clinical evaluation and do not require reference to algorithms or complex calculations. However, evaluation of a single variable cannot capture the entire clinical picture of a case.

Several prognostic models have been developed based on multivariable regression modeling of specific colic populations. ²² , ²³ The colic severity score (CSS) uses numerical scores (0 to 4) to each of four categories (pulse, peritoneal total protein, blood L‐lactate, and mucous membrane appearance) based on semi‐quantitative cutoffs with a total score >7 strongly predictive of death. ²⁴ Grulke et al. developed a shock score that also uses numerical scores (1 to 4) and quantitative cutoffs for several categories, including heart rate, respiratory rate, packed cell volume, among other variables. ²⁵ More recently, McConachie et al. developed the equine multiorgan disfunction syndrome for surgical gastrointestinal disease (MODS SGI) scoring system to predict survival in horses with acute surgical colic. ²⁶ This model uses scores (0 to 3) based on qualitative descriptions or quantitative cut‐offs for parameters of cardiovascular, renal, hepatic, respiratory, musculoskeletal, coagulation, and gastrointestinal status that may not readily available in the emergency clinical setting. ²⁶

Although some prognostic models were developed for specific populations, ² , ⁷ , ¹¹ , ¹² most have been developed using data from horses with a wide variety of underlying conditions that were treated either surgically or medically. Additionally, few models have been validated by testing in an independent population of horses. The authors have observed that in practice, clinicians often apply predictive models or single variable cutoff points without consideration of whether the patient being evaluated fits the specific inclusion criteria used for development of the relevant model. This is particularly true for patients that may require surgery, as there are few predictive models developed specifically for horses requiring surgical intervention for treatment of colic.

The objective of this study, therefore, was to evaluate previously published predictive survival models in a population of horses undergoing colic surgery in the midwestern United States. We hypothesized that the previously reported survival models would not perform as well in this study cohort as in the original populations. We further hypothesized that multivariable models would perform better than single variable models for predicting survival.

2. MATERIALS AND METHODS

2.1. Study design

Medical records for all horses that underwent exploratory laparotomy at the University of Illinois Veterinary Teaching Hospital between January 1, 2009 and December 31, 2020 were reviewed. Horses that underwent exploratory laparotomy for reasons unrelated to abdominal pain (e.g., traumatic body wall rupture), foals <6 months of age, and cases in which euthanasia was elected by owner despite recommendation for treatment were excluded. For horses undergoing multiple laparotomies during the same hospitalization, only the initial surgery was included in the analysis.

Admission data included signalment, duration of colic signs prior to presentation, physical examination findings (heart rate, respiratory rate, rectal temperature), and time between admission and surgery. Mucous membrane color and capillary refill time (CRT) were recorded. Mucous membrane appearance was considered abnormal if CRT >3 seconds or if color was described as hyperemic, muddy, or petechiated. Time of admission was recorded; admission between 5:00 p.m. and 8:00 a.m., or on a weekend or holiday was considered outside of normal working hours. Preoperative hematological values included packed cell volume (PCV) and total solids (TS). Venous blood gas analysis was performed with a point of care blood analyzer (pHOx Ultra, Nova Biomedical, Waltham, Massachusetts) and included pH, pCO₂, pO₂, oxygen saturation, hematocrit, hemoglobin, bicarbonate, base excess, oxygen content, sodium, potassium, chloride, ionized calcium, ionized magnesium, glucose, lactate, blood urea nitrogen (BUN), creatinine, total CO₂, anion gap, and osmolality. Results of abdominocentesis were recorded when available and scored as normal appearance, serosanguinous, turbid, or no fluid obtained. Analyses performed on abdominal fluid included lactate (Lactate Scout, EKF Diagnostics, Cardiff, Wales, UK) and TS.

Intraoperative data retrieved from the surgery reports and anesthesia records included duration of general anesthesia and surgery, surgical diagnosis, and location of the lesion. Hematological variables from the first intraoperative blood sample were recorded including arterial blood gas values as for venous blood gas (pHOx Ultra, Nova Biomedical, Waltham, Massachusetts), PCV and TS. Arterial blood was collected from an indwelling catheter (facial or transverse facial artery) and maintained in a closed system until analysis. Surgery performed between 5:00 p.m. and 8:00 a.m., or on a weekend or holiday was considered outside of normal working hours.

Postoperative colic was defined as an episode of abdominal pain requiring sedation or other therapeutic intervention (e.g., nasogastric intubation or additional analgesic medication) after the horse had recovered and been returned to its stall. Presence of postoperative colic (yes/no), treatment, diagnosis, and relaparotomy were recorded as appropriate. Short‐term survival was defined as survival to hospital discharge, and horses were classified as survivors or nonsurvivors (euthanized or died). The number of days of hospitalization following surgery was recorded. For nonsurvivors, the reason for euthanasia was recorded and categorized as: recommended due to grave prognosis, postoperative complications, or natural death.

2.2. Predictive models

PubMed was searched for studies containing predictive models for horses with colic (using the following terms: “equine” “colic” and “prediction OR prognostic”) published between 1986 and 2020. This search yielded 137 publications, 100 of which had full text available for review. Models were included if the study population was clearly described and limited to referral practice, and if the variables and models were described in sufficient detail to successfully reproduce the model. Eight single variable cutoffs ¹⁷ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ and four multivariable models ¹³ , ²⁴ , ²⁵ , ³² met these criteria (Table 1). Models were applied to the study cohort to determine predicted outcome (survival or nonsurvival to hospital discharge). Individuals with missing data relevant to each model were excluded from model analysis. Two multivariable models included measurements that were not available in the retrospective medical records. First, systolic arterial pressure is assessed in determination of the Shock Score but was not measured in our cohort before induction of anesthesia. ²⁵ However, because the Shock Score is determined by the highest score out of six parameters, the predicted outcome in the study cohort was calculated on the basis of the five available parameters. ²⁵ Second, the linear model by Thoefner et al. includes a categorical pain coefficient (none/mild, moderate, and severe), ³² which was not specifically recorded in our cohort. To calculate predicted outcome for this model in the study cohort, heart rate was used to approximate pain (moderate if HR < 80 bpm, severe if HR ≥80 bpm; it was assumed that colic surgery would not be performed in horses displaying no or mild pain).

TABLE 1.

Summary of diagnostic criteria and thresholds for single variable and multivariable predictive models as applied to the study cohort

Model	Criteria considered for study cohort	Performance and population in original report
Single variable models
Johnston et al. 17	Peripheral lactate at admission <6 mmol/l predicted to survive	Se 84%, Sp 83% (large colon volvulus)	Large colon volvulus
Radcliffe et al. 27	Peripheral lactate at admission <8 mmol/l predicted to survive	No predictive value reported	Surgical colic
Orsini et al. 28	PCV at admission ≤43% predicted to survive	Serum lactate and PCV combined predictive value of 94%	Medical and surgical colic
Orsini et al. 28	PCV at admission ≤50% predicted to survive
Puotunen‐Reinert et al. 29	Heart rate at admission ≤60 bpm predicted to survive	No predictive values reported	Medical and surgical colic
Puotunen‐Reinert et al. 29	Heart rate at admission ≤80 bpm predicted to survive	No predictive values reported	Medical and surgical colic
Delesalle et al. 30	Peritoneal lactate at admission ≤6 mmol/l predicted to survive	Probability of death if lactate >6 is 29% without strangulating obstruction, 52% with strangulating obstruction	Medical and surgical colic
McCoy et al. 31	Intraoperative lactate <5 mmol/l predicted to survive	2.5x higher relative risk of complications if lactate >5 mmol/l (arterial lactate in recovery)	Surgical colic
Multivariable Models
Grulke et al. 25	Shock Score: Horses with shock score of 3 (meeting one or more of the following criteria) predicted to die: heart rate ≥ 80 bpm; respiratory rate ≥ 35 bpm; PCV ≥55%; blood lactate ≥8.3 mmol/dl; blood urea nitrogen ≥55 mEq/dl	Odds ratio of death 10.8 for shock score 3 vs. shock score 1	Surgical colic
Thoefner et al. 32	Linear model including pain coefficient, PCV, capillary refill time, and rectal temperature. Cutoff for survival set at probability of death ≤0.86	PPV and NPV 87%	Medical and surgical colic
Reeves et al. 23	Linear model including age, sex, capillary refill time, PCV and heart rate. Cutoff for survival set at probability of death ≤0.6	Se 70, Sp 89, PPV 80, NPV 82	Medical and surgical colic
Furr et al. 24	Colic severity score: Summed scores based on four criteria: heart rate, peritoneal total protein, blood lactate, and mucous membrane appearance. Horses with total score <7 are predicted to survive	Development: Se 52.9, Sp 90.1, PPV 58.1, NPV 88.1 Validation: Se 66.7, Sp 100, PPV 100, NPV 91.8	Medical and surgical colic

Abbreviations: NPV, negative predictive value; PPV, positive predictive value; Se, sensitivity; Sp, specificity.

2.3. Statistical analysis

All analyses were conducted in R version 4.1.2 ³³ within the RStudio environment, version 1.3.1093 ³⁴ with two‐sided tests of hypotheses; p < .05 was the criterion for statistical significance. For categorical variables, descriptive statistics were reported as frequency count (percentage of total). For continuous variables, normality was assessed by Shapiro–Wilk test and visual evaluation of histogram and QQ plots. Descriptive statistics for continuous variables were reported as mean and standard deviation if normally distributed, and median and interquartile range (IQR; lower quartile, upper quartile) if they were not normally distributed.

Diagnostic test performance indices were calculated for all cases in the study cohort with each single‐variable and multivariable model using the confusionMatrix function of R package caret, version 6.0.90. ³⁵ These included sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy, calculated by comparing the predicted to actual outcome with survival considered the “positive” result and nonsurvival the “negative”. These performance indices were compared between single and multivariable models, as well as between models developed from a mixed population (medical and surgical) or surgical population, by Student's t‐test using the R package stats, version 4.1.2. ³³ All plots were generated using the R package ggpubr, version 0.4.0. ³⁶

3. RESULTS

3.1. Study population

A total of 260 horses (age range: 6 months to 32 years, mean: 10.8 ± 6.9 years) met the inclusion criteria. The cohort included 103 (39.6%) mares, 34 (13.1%) stallions, and 123 (47.3%) geldings. Breed distribution was representative of the patient population seen for emergency care at the University of Illinois, with 35 unique breeds (Table S1) categorized into eleven groups: stock horses (104 [40%]), gaited horses (23 [8.9%]), Thoroughbred (22 [8.5%]), Standardbred (20 [7.8%]), Arabian (20 [7.7%]), warmblood (20 [7.7%]), draft (14 [5.4%]), mixed breed (13 [5%]), Morgan (8 [3.1%]), pony (8 [3.1%]), and miniature horse (8 [3.1%]). Mean bodyweight was 1063 ± 263 pounds. Admissions occurred outside of normal working hours for 159/260 (61.2%) cases.

3.2. Admission data

Quantitative admission data are summarized in Table 2. Mucous membrane appearance was abnormal in 42/260 (16.2%) horses. Abdominocentesis was performed in 167/260 (64.2%) horses, and fluid was obtained in 151 horses. Peritoneal fluid was described in all 151 horses when it was obtained: 86 (57%) serosanguinous, 46 (30.5%) normal, 16 (10.6%) turbid or cloudy, two (1.3%) bloody, and one enterocentesis (0.7%). Venous blood gas results are summarized in Table S2.

TABLE 2.

Summary of admission data as mean ± SD or median (quartile 1, quartile 3) with number of observations (N) for each variable, grouped by survivors (N = 156) and nonsurvivors (N = 104)

Variable	N	Survivors	N	Nonsurvivors
Rectal temperature (°F)	124	99.5 ± 1.4	77	99.5 ± 2.1
Heart rate (bpm)	148	57 ± 17.3	99	72 ± 20.5
Respiratory rate (bpm)	128	24 (19.5, 32)	89	28 (20, 40)
Peritoneal lactate (mmol/l)	45	3.2 (2.1, 7.1)	41	9.9 (6.4, 16)
Peritoneal total solids (g/dl)	51	2.6 ± 1.2	36	3.6 ± 1.6

3.3. Surgical and anesthetic data

Median anesthesia time was 148 min (IQR 110, 195) and median surgical time was 105 min (IQR 70, 155). Surgical diagnoses are summarized in Table 3. No definitive diagnosis was recorded in the medical records of one horse which was euthanized for intractable colic 10 days postoperatively.

TABLE 3.

Intraoperative diagnoses, organized by primary intestinal segment affected. Percentage column refers to total cases (260) for intestinal segment, and number of cases within the intestinal segment for specific diagnosis (i.e., pedunculated lipoma, 34 of 112 small intestine cases)

Diagnosis	Number	Percent
Small intestine	112	43.1%
Pedunculated lipoma	34	30.4%
Anterior enteritis	17	15.2%
Mesenteric rent	11	9.8%
Epiploic foramen entrapment	10	8.9%
Volvulus	7	6.3%
Small intestinal strangulation of unidentified cause	6	5.4%
Adhesions	6	5.4%
Omental strangulation	5	4.5%
Gastrosplenic ligament entrapment	3	2.7%
Intussusception	3	2.7%
Two jejunojeunal, 1 ileocecal
Impaction	3	2.7%
Diaphragmatic hernia	2	1.8%
Indirect scrotal hernia	2	1.8%
Infarction	2	1.8%
Pyloric thickening	1	0.9%
Large colon	104	40%
Volvulus	40	38.5%
Displacement	30	28.8%
Seventeen right dorsal, 10 left dorsal, 3 not specified
Twelve of 30 had concurrent impaction
Impaction	25	24%
Enterolith	6	5.8%
Colitis	3	2.9%
Small colon	31	11.9%
Fecalith/enterolith	17	54.8%
Impaction	11	35.5%
Pedunculated lipoma	1	3.2%
Nephrosplenic entrapment	1	3.2%
Volvulus	1	3.2%
Cecum	4	1.5%
Impaction	2	50%
Displacement	1	25%
Incarceration in diaphragmatic hernia	1	25%
Stomach	4	1.5%
Gastric rupture	4	100%
Extra‐gastrointestinal	4	1.5%
Suspected renal failure and disseminated intravascular coagulation	1
Enlargement of kidney and multiple lymph nodes	1
Hemoabdomen secondary to unidentified mass	1
Bile duct rupture secondary to cholelithiasis	1

3.4. Postoperative colic

Sixty‐five of the 188 (34.5%) horses that recovered from anesthesia exhibited colic signs postoperatively. Of these, 33/65 (50.8%) responded to medical treatment, 16/65 (24.6%) underwent repeat exploratory laparotomy, and 16/65 (24.6%) were euthanized without a second exploratory. Ten of the 16 horses that underwent repeat laparotomy (62.5%) were euthanized; diagnoses included ileus (n = 10), ischemia–reperfusion injury (n = 8), anastomotic stricture (n = 2), severe adhesions (n = 2), and rupture of the right dorsal colon. The remaining 6/16 (37.5%) horses survived to hospital discharge. Surgical diagnoses in horses that survived after repeat laparotomy were anastomosis site complications (n = 3), gas distension, ileus, and devitalization/reperfusion injury. One of the six horses had a third laparotomy during the same hospitalization at which time adhesions were identified as the cause of colic signs. That individual ultimately survived to discharge.

3.5. Outcomes

Duration of follow up was limited to the time of discharge from the hospital (median 8 days [IQR 6,10]). Of the 260 horses undergoing general anesthesia, 188 (72.3%) survived to anesthetic recovery. Of those, 156 (83%) survived to hospital discharge. Overall short‐term survival rate was 60% (156/260).

Of 104 nonsurviving horses, 72 horses were euthanized intraoperatively based upon recommendation of the attending veterinarian due to a grave prognosis. Thirty‐two horses were euthanized due to complications that developed after surgery that were unresponsive to medical and/or surgical therapy. Reasons for euthanasia are summarized in Table 4. Lesions were categorized as unresectable due to the length involved in 24 horses, and due to the location of affected viscus in 20 horses. Irreparable lesions included: extensive mesenteric rent, irreducible ileocecal intussusception, irreducible jejunal intussusception, and a large mass adhered to multiple organs. Two horses were euthanized in the recovery box as a result of inability to stand; one due to hypovolemic shock and secondary renal failure, and one in which a cause was not identified.

TABLE 4.

Case outcomes and reasons for euthanasia

Outcome	Number	Percent
Survival to discharge	156	60%
Intraoperative euthanasia	72	27.7%
Unresectable lesion	44	61.1%
Rupture of intestinal viscus	18	25%
Irreparable lesion	4	5.6%
Excessive adhesions	3	4.2%
Peritonitis	3	4.2%
Postoperative euthanasia	32	12.3%
Postoperative colic	23	71.9%
Endotoxemia and deterioration in face of treatment	3	9.4%
Inability to stand in recovery	2	6.3%
Laminitis	2	6.3%
Severe intra‐abdominal hemorrhage	1	3.1%
Disseminated intravascular coagulation	1	3.1%

3.6. Performance of predictive models

A summary of the evaluated models, the populations from which they were generated and their performance (as reported in the original publication) is found Table 1. The Shock Score ²⁵ and the linear model by Thoefner et al. ³² included measurements that were unavailable in the retrospective medical records. Predicted outcome for these two models were calculated with minor modifications as described in the methods.

All the evaluated single variable cutoffs and multivariable models performed similarly for prediction of outcome in the study cohort, with a mean 79% sensitivity (range 44%–94%), 48% specificity (20%–82%), 70% PPV (63%–78%), 65% NPV (51%–78%), and 67% accuracy (60%–73%) (Table 5). There was no difference between single variable cutoffs and multivariable models (all p > .3), or between models derived from a mixed medical and surgical population versus those derived from a surgical population, (all p > .1) for any measures of diagnostic performance in the study cohort (Figure 1, Table 6). Single variable cutoffs of blood lactate (6 or 8 mmol/l), PCV (50%), and heart rate (80 bpm) were highly sensitive for prediction of survival, but poorly specific, with better NPV than PPV. The most accurate single variable predictor was abdominal lactate (cutoff of lactate ≤6 mmol/l predictive of survival), at 73% accuracy. The CSS ²⁴ was the most sensitive of the multivariable predictive models (sensitivity 92%, NPV 78%), but had poor specificity (40%).

TABLE 5.

Diagnostic performance of single variable and multivariable predictive models for predicting outcome (survival to hospital discharge) in the study cohort. Unless otherwise specified, variables were assessed at hospital admission

Model	Variable (S)	Sensitivity	Specificity	PPV	NPV	Accuracy	N
Single variable models
Johnston et al. 17	Venous lactate <6 mmol/l	90%	30%	65%	67%	65%	249
Radcliffe et al. 27	Venous lactate <8 mmol/l	94%	20%	63%	69%	64%	249
Orsini et al. 28	PCV < 43%	71%	63%	74%	60%	68%	227
Orsini et al. 28	PCV < 50%	93%	33%	67%	78%	69%	227
Puotunen‐Reinert et al. 29	HR < 60 bpm	70%	67%	76%	59%	68%	247
Puotunen‐Reinert et al. 29	HR < 80 bpm	93%	25%	65%	69%	66%	247
Delesalle et al. 30	Peritoneal lactate <6 mmol/l	69%	78%	78%	70%	73%	86
McCoy et al. 31	Intraoperative lactate <5 mmol/l	89%	39%	71%	69%	70%	210
Multivariable models
Grulke et al. 25 (Shock Score)	HR, PCV, lactate, BUN	66%	56%	69%	52%	62%	260
Thoefner et al. 32	Pain, PCV, CRT, temperature	82%	43%	69%	61%	66%	176
Reeves et al. 23	Age, sex, CRT, PCV, HR	44%	82%	77%	51%	60%	187
Furr et al. 24 (colic severity score)	HR, peritoneal TP, lactate, mm	92%	40%	69%	78%	71%	110

Abbreviations: BUN, blood urea nitrogen; CRT, capillary refill time; HR, heart rate; mm, mucous membrane appearance; N, number of cases included; NPV, negative predictive value; PCV, packed cell volume; PPV, positive predictive value.

FIGURE 1.

Diagnostic performance of all tested models plotted by sensitivity and specificity (A) and positive predictive value (PPV)/negative predictive value (NPV) (B). Dot size reflects model accuracy, color is model type (single variable in orange, multivariable in blue), and shape is original study population (circle is mixed surgical and medical colic, triangle is surgical colic)

TABLE 6.

Diagnostic performance of predictive models grouped by model type (single variable vs. multivariable) and original study population (mixed surgical and medical colic vs. surgical colic only). Presented as mean ± standard deviation (SD); p‐value represents result of Student's t‐test between the two groups

	SensitivIty		Specificity		PPV		NPV		Accuracy
	Mean ± SD	p‐value	Mean ± SD	p‐value	Mean ± SD	p‐value	Mean ± SD	p‐value	Mean ± SD	p‐value
Model type
Single variable	84 ± 12%	0.333	44 ± 22%	0.413	70 ± 5%	0.674	68 ± 6%	0.362	68 ± 3%	0.302
Multivariable	71 ± 21%		55 ± 19%		71 ± 4%		61 ± 13%		65 ± 5%
Original study population
Mixed	77 ± 17%	0.382	54 ± 21%	0.134	72 ± 5%	0.102	66 ± 9%	0.759	68 ± 4%	0.372
Surgical	85 ± 13%		36 ± 15%		67 ± 4%		64 ± 8%		65 ± 4%

Abbreviations: NPV, negative predictive value; PPV, positive predictive value; SD, standard deviation.

Measures of diagnostic performance were only found in the original reports for four models. Lactate cutoff of 6 mmol/l had higher sensitivity in our study cohort (90%) compared to the original report (84%), but much lower specificity (30% vs. 83%). ¹⁷ This model was originally described in horses specifically treated for large colon volvulus. When applied to the cases of the study cohort with a diagnosis of large colon volvulus (n = 40), the model still did not perform well (sensitivity 75%, specificity 46%, PPV 48%, NPV 73%, accuracy 60%). For multivariable models, all measures of diagnostic performance were lower in the study cohort than reported in the original publications (Figure 2).

FIGURE 2.

Diagnostic performance of tested models in the study population (cohort, blue) compared to reported values from original publication when available (publication, orange). Blank space indicates that the parameter was not included in the original publication. PPV, positive predictive value; NPV, negative predictive value

4. DISCUSSION

All tested models underperformed the original reports when used to predict outcome in this independent study cohort, supporting the hypothesis that the models would not perform as well when applied to a different population. Interestingly, diagnostic performance for the previously reported models was similar between single variable cutoffs and multivariable models when applied to the study cohort, therefore we reject the hypothesis that multivariable models would be more accurate. These findings suggest that while general associations between clinical variables and outcome may be broadly applicable across populations, population‐ and facility‐specific factors still play an important role in predicting survival in horses undergoing colic surgery.

The great variety of lesions underlying in signs of colic and the associated pathological processes cause a wide spectrum of patient morbidity, presenting an inherent challenge in prediction of survival for this heterogeneous horse population. A primary goal of this analysis was to determine the prognostic accuracy for survival using information available prior to placing a horse under general anesthesia and making a definitive intraoperative diagnosis. Additional intraoperative variables, such as performance of a resection and anastomosis, have been associated with survival to discharge in other selected populations. ²¹ We elected to exclude these variables because of their bias towards the portion of the cohort that recovered from anesthesia, while we were interested in prognostication for the entire group.

Performance indices for the models were calculated considering survival as the positive outcome. In this context, “sensitivity” is the proportion of horses correctly identified as survivors, and “specificity” is the proportion of horses correctly identified as nonsurvivors. Most of the single variable cutoff values assessed were moderately to highly sensitive and poorly specific. Clinically, this means that single value cutoffs are best at predicting nonsurvival when the horse has a high value (that is, above the cutoff). A PCV threshold of 50% ²⁸ and the CSS (a summed score based on heart rate, peritoneal total protein, blood lactate, and mucous membrane appearance) ²⁴ had the highest NPV of all models considered, indicating they were best for predicting nonsurvival in the study population. The multivariable model by Reeves et al. (a linear model incorporating age, sex, capillary refill time, PCV and heart rate) ²³ and peritoneal lactate cutoff of 6 mmol/l (≤6 mmol/l predictive of survival) had the highest PPV. Overall accuracy was similar for single variable and multivariable models in the study cohort. Peritoneal lactate cutoff of 6 mmol/l, ³⁰ was the most accurate predictor of outcome at 73%. This means that, at best, nearly a third of patients would be incorrectly predicted to either survive or not survive according to the models.

The roles of pre‐emptive euthanasia and economic factors in colic survival present an additional complication when evaluating outcomes. It can be argued that all horses euthanized intraoperatively should be excluded from analysis, as those horses were not given the opportunity to survive, and a subset may have survived if further treatment was pursued. However, this approach does not account for cases in which the lesion identified at surgery was incompatible with life. These inclusion criteria are important to consider in evaluation of predictive models, as the predictive value and accuracy of a diagnostic test are affected by disease prevalence, in addition to the test's sensitivity and specificity. The overall survival rate in the study population was lower than some contemporary reports which focus only on horses that recovered from general anesthesia. ²¹ , ³⁷ , ³⁸ However, it is critical to note that the study cohort also includes horses that were euthanized intraoperatively due to the nature of their lesion, and survival rate is comparable to other reports where all surgical cases are included. ³⁷ , ³⁹ This highlights that inclusion criteria of a study can alter the consequent survival rate and offer a more (or less) optimistic outlook than is appropriate for the broader surgical population.

The cutoff value for a predictive model can be selected to optimize the sensitivity or specificity of the model. The cutoffs for mathematical predictive models were selected to maximize the sum of specificity and sensitivity. ²³ , ³² However, the severity of false negative and false positive errors is not equivalent in the case of life‐or‐death clinical decisions. If a false negative error (predicting a horse will die that could otherwise live) is considered the more grievous error (because the owner would be prompted to euthanize the horse), it may be more desirable to select a cutoff value that maximizes sensitivity, while accepting lower specificity. While peritoneal lactate cutoff of 6 mmol/l ³⁰ had the greatest overall accuracy for prediction of outcome, the CSS, ²⁴ venous lactate cutoff of 8 mmol/l, ²⁷ and admission heart rate cutoff of 80 bpm ²⁹ may be the most applicable for a clinical setting due to their high sensitivity, meaning that horses with values below the cutoff are correctly identified as survivors.

Of the models tested, measures of diagnostic accuracy were reported in the original publications for three multivariable models and one single variable predictor (Table 1). The models performed better in the original populations than in the current study population, apart from venous lactate cutoff of 6 mmol/l, which had a sensitivity of 90% in the current study, compared to 84% originally reported (Figure 2). ¹⁷ This is unsurprising, as it is generally expected that a model will perform best in the population from which it is developed. Validation of a predictive model in multiple populations is recommended before making assumptions about its general applicability for use in clinical decision‐making. ² Without such external validation, clinicians should exercise caution when applying predictive models to clinical cases.

Three of the multivariable models tested here were originally developed in general colic populations. ²³ , ²⁴ , ³² As our study population included only colic patients requiring surgical intervention, this probably explains why these multivariable models did not perform as well in our study cohort when compared to the respective original populations. Further, some single variable cutoffs were derived from studies focused on specific lesions (i.e., large colon volvulus) ¹⁷ or based on specific time points (intraoperative and recovery/postoperative arterial lactate) ³¹ and may not be applicable beyond those specific situations. Notably, when applied to horses in the study cohort with large colon volvulus, the model developed specifically for that lesion still did not perform as well as in the original report. Further, there was no statistically significant difference identified when measures of diagnostic performance were compared between the models from a mixed colic population and those from surgical colic cases. This suggests that even the models developed from a more similar patient population did not perform well when applied to a different cohort.

Models originally designed to predict nonsurvival, rather than survival, such as those reported by Reeves et al. ²³ and Thoefner et al., ³² could have had improved performance if a different cutoff value was used for survival. However, clinical utility of these two specific models is limited by the required computations. In contrast, the CSS, ²⁴ which is the most accurate multivariable model tested, is easily applied using information rapidly acquired from a routine colic evaluation.

4.1. Limitations

Inherent limitations of a retrospective study include the quality of available medical record data, lack of randomly assigned treatment groups, and clinician‐dependent differences in case management. The effect of clinician bias is impossible to eliminate, as case management and euthanasia recommendations are affected by surgeon assessment, experience, and judgment. Duration of follow‐up in this study was limited to the time of discharge from hospital. Inclusion of longer‐term follow up may have captured additional post‐operative complications and mortality, such as incisional infections and recurrent colic. Prediction of long‐term survival is arguably of more interest to the horse owner; however, this was beyond the scope of the current study, which focused on short‐term outcomes. Horses that were euthanized rather than undergo exploratory laparotomy or were euthanized intraoperatively due to financial considerations were excluded. Results of this study reflect cases from one institution, and (as demonstrated by this study) the performance of these models may differ when applied to other populations.

4.2. Conclusions

Single‐variable and multivariable predictive models did not perform as well for prediction of outcome in the study cohort compared to their original reports. These findings suggest that while general associations between clinical variables and outcome may be broadly applicable across populations, population‐specific factors still play an important role in survival. Not surprisingly, patient outcomes and performance of prognostic models in this cohort consisting only of surgical cases appears to be less favorable than previous reports that include both of medical and surgical colic cases. The best‐performing single variable cutoffs and multivariable models were highly sensitive for prediction of survival, but poorly specific.

4.3. Clinical significance

While no single variable was sufficient to accurately predict the outcome of a horse requiring exploratory laparotomy for colic, our findings support the continued use of parameters commonly considered in clinical assessment of patient status, such as admission heart rate, packed cell volume, and blood lactate. Predictive models of survival developed in one population are highly likely to be less reliable when used to predict outcome in horses undergoing colic surgery in an independent population. Additional model testing and refinement using data from multiple surgical centers could be considered to improve prediction of outcome for horses undergoing laparotomy for treatment of colic.

AUTHOR CONTRIBUTIONS

R.C. Bishop: Study initiation and design, data collection, data analysis and interpretation, drafting of the manuscript, approved the final version of the manuscript. S.D. Gutierrez‐Nibeyro: Study design, manuscript revision, approved the final version of the manuscript. M.C. Stewart: Study design, manuscript revision, approved the final version of the manuscript. A.M. McCoy: Study design, data analysis and interpretation, manuscript revision, approved the final version of the manuscript.

FUNDING INFORMATION

There was no funding specific to this work.

CONFLICT OF INTEREST

There was no funding specific to this work. The authors declare no conflict of interest related to this report.

Supporting information

Table S1. Breeds of horses in study cohort. Thirty‐five unique breeds were represented, categorized into 11 types for the purpose of data analysis.

Table S2. Blood gas data from admission (venous) and early intraoperative period (arterial) with number of observations for each variable (N = number of horses with a value reported for each variable). Central tendency reported as mean and standard deviation if normally distributed, or median (quartile 1, quartile 3) if not normally distributed.

Bishop RC, Gutierrez‐Nibeyro SD, Stewart MC, McCoy AM. Performance of predictive models of survival in horses undergoing emergency exploratory laparotomy for colic. Veterinary Surgery. 2022;51(6):891‐902. doi: 10.1111/vsu.13839