Validity of Pediatric Index of Mortality 2 score as an Outcome Predictor in Pediatric ICU of a Public Sector Tertiary Care Hospital in Pakistan

Abstract

Pediatric Index of Mortality 2 (PIM-2) is one of the leading mortality scores used in intensive care units all around the world. We assessed its validity as an outcome predictor in a pediatric intensive care unit (PICU) of Mayo Hospital/King Edward Medical University Lahore, Pakistan. We enrolled 154 consecutive admissions, aged 1 month to 13 years, requiring intensive care from January to June of 2019. Patient demographics along with PIM-2 data were collected; PIM-2 score and mortality risk was calculated; and the outcome recorded as death or survival. The median age at admission was 0.50 years (interquartile range [IQR]: 0.24–1.78) and the median weight was 5.0 kg (IQR: 3.08–10.0) with females constituting 54%; malnutrition was also common (66%). Observed mortality was 29.9% (46 out of 154) and expected mortality (cut-off ≥ 99.8%) was 27.9% with a standardized mortality ratio of 1.07 (95% confidence interval [CI]: 0.79–1.41). Sepsis was the most common diagnosis at admission (27.9%) with the highest mortality (52.2%). Chi-square analysis revealed a sensitivity of 54.3% and a specificity of 83.3% ( p -value 0.00). PIM-2 score showed acceptable discrimination between survivors and nonsurvivors with an area under the receiver operating characteristic curve of 0.75 (95% CI: 0.67–0.84) ( p -value = 0.00); however, poor calibration according to Hosmer–Lemeshow goodness of fit test (Chi-square = 15.80, df = 7, and p -value of 0.027 [< 0.1]), thus requiring recalibration according to local population characteristics.

Introduction

Throughout the world, the quality of intensive care is not equitable because of the variability of resources available and the investments made by the health care systems in that place. ¹ Even in the same country, units designated for this purpose do not provide an equal level of care, which is usually attributable to patient characteristics, late referrals, case mixes, etc. Thus, before comparing the quality of these intensive care units, the results should be objectively quantified for them to be acceptable. ¹ Reducing mortality is the primary objective of a pediatric intensive care unit (PICU), ² whereby their efficacy is assessed and all the efforts are concentrated on this primary goal. For this purpose, mortality prediction scores have been in use in their updated forms in both pediatric and adult intensive care unit (ICUs) (APACHE, SNAP, SNAPPE, MPM, PRISM, PIM, etc.). The equations for these scores, obtained by logistic regression models, describe the relationship between predictor variables and the probability of death. ^{3 4 5} These scoring systems cannot provide an individual's risk of mortality very accurately. However, these do aid in comparing severity in patients with similar disease presentations and also to compare the working capacity of different ICUs. Pediatric index of mortality 2 (PIM-2) is one of the most commonly used scores in pediatric intensive care settings throughout the world to gauge the efficacy of critical care and its outcome. ²

PIM-2 score introduced by Slater et al in 2003 is an upgrade of the Pediatric PIM score. It uses 10 variables to predict the probability of death and has shown better outcome predictability than other scoring systems. ⁶ Data are collected at the first encounter at the ICU, usually within the first hour of admission, unlike other scores that have variable timelines. PIM-2 scoring system has been validated in many countries, both developed and developing, with variable results. Studies from India (Gandhi et al) and China (Qiu et al) have shown acceptable discriminatory performance with good calibration. ^{4 5} However, reports about implementation and outcome of these scoring systems are limited from Pakistan, such as the one from Qureshi et al done in 2007. ⁷ A few centers, however, have reported poor results and thus calibrated the scores according to their population characteristics. A study from India showed a lower sensitivity of 70.6% for PIM-2. ⁸

In Pakistan, we have peculiar circumstances which are different from modern PICUs viz-a-viz late referral, high frequency of malnutrition, the paucity of critical care services, less advanced technology and expertise, as well as health care worker understaffing, to take care of patients. Hence, results from modern setups may not hold true in our setting. Therefore, before implementation, these scoring systems should be validated and calibrated to give reliable results that may help in improving the quality of care. With this background and rationale, the study was done to assess the validity of the PIM-2 score as an outcome predictor in the PICU of Mayo Hospital, Lahore, Pakistan.

Materials and Methods

It was a cross-sectional study done at the Pediatric ICU of Mayo Hospital/King Edward Medical University Lahore, which is a 14-bed unit including a 4-bed High Dependency Unit. Mayo Hospital is one of the largest tertiary care hospitals in Pakistan with the second largest public sector PICU in the city. This study was done for a period of 6 months from January 1, 2019 to June 30, 2019. The sample size of 154 consecutive ICU admissions was calculated by using 95% confidence interval, expected mortality to be 46.21%, ⁴ and sensitivity of PIM-2 to be 70.6% (10% margin of error) ⁸ and specificity of 65.6% (10% margin of error). ⁴ The patients were enrolled via nonprobability consecutive sampling after getting approval from the institutional review board/ethical committee of King Edward Medical University. The patients who required ICU care from the age of 1 month to 13 years were included in the study. Exclusion criteria included death within the first 8 hours of ICU admission, discharge within the first 24 hours of ICU admission, leave against medical advice, transfer to other ICU settings or refusal of resuscitation, patients with complex congenital disorders, and end-stage diseases (end-stage renal disease, cirrhosis, etc.).

Written informed consent was taken from parents/guardians. Demographic data were collected to characterize the study population, including age at admission, gender, and nutritional status (Z-score of 2 or below for weight-for-age according to World Health Organization). The actual outcome for all cases was documented as survival or death. A senior pediatric resident recorded all 10 parameters of the PIM-2 score within first hour of admission. These included elective admission to PICU, recovery postprocedure, cardiac bypass, high-risk diagnosis (cardiac arrest preceding ICU admission, severe combined immune deficiency, leukemia or lymphoma after the first induction, spontaneous cerebral hemorrhage, cardiomyopathy or myocarditis, hypoplastic left heart syndrome, HIV infection, liver failure as the main reason for ICU admission, neurodegenerative disorder), low-risk diagnosis (asthma, bronchiolitis, croup, diabetic ketoacidosis, obstructive sleep apnea), no response of pupils to bright light (>3 mm and both fixed), mechanical ventilation (at any time during the first hour in PICU), systolic blood pressure (mmHg; patients were labeled as hypotensive according to Pediatric Advanced Life Support 2018 guidelines), base excess (meq/L; arterial blood), and FiO ₂ and PaO ₂ (mmHg). Diagnosis at admission, length of mechanical ventilation required, and length of hospital and ICU stay at the unit were also recorded. The score was calculated by the predesigned formulae of PIM-2 and the mortality percentage was recorded. Patients having a percentage score greater than or equal to 99.8% ⁴ were labeled as “yes” for predicted mortality. Patients were followed for a period of 7 days in ICU to record mortality.

Data were analyzed by using Statistical Program for Social Science (SPSS Inc., Chicago, Illinois, United States) version 20.0. Median and interquartile range (IQR) was calculated for age, weight, length of ICU, and hospital stay as well as the length of ventilation due to skewness of data. Frequency and percentages were calculated for gender and malnutrition. Distribution of various diagnoses was expressed in the form of frequencies and percentages both for ICU admissions as well as for mortality. A 2 × 2 contingency table was used to calculate diagnostic accuracy with Chi-square analysis taking p -value ≤0.05 as significant. Sensitivity and specificity were calculated for the score as shown in Table 1 . Data were stratified for age (5 years, 3 years, 1 year, and 6 months), gender, weight (5 and 10 kg), and malnutrition. The poststratification Chi-square test was used by taking p -value ≤0.05 as significant to check the impact of the above-mentioned factors with mortality. Differences between survivors and nonsurvivors were assessed with the help of the Mann–Whitney U test for their significance. The association of diagnostic categories (high-risk, low-risk, and neither) with mortality was analyzed by the Pearson's Chi-square test and Fisher's exact test was used to check the association of pupillary response with mortality.

Table 1. Association of actual mortality to predicted by Pediatric Index of Mortality 2 score ( n  = 154) .

		Actual mortality		Total
		Yes	No
Predicted mortality by PIM-2	Yes % within predicted mortality	25 (a) 58.1%	18 (b) 41.8%	43 100%
	No % within predicted mortality	21 (c) 18.9%	90 (d) 81.1%	111 100%
Total		46 29.9%	108 70.1%	154 100%

Abbreviations: PIM-2, pediatric index of mortality 2 score.

Note: Cut-off for actual mortality is ≥ 99.8%.

Sensitivity: (a/a + c) = 54.3%.

Specificity: (d/d + b) = 83.3%.

p -value: 0.00 (≤0.05).

Receiver operating characteristic (ROC) curve was generated to calculate the area under the ROC (AUC) to compute the discriminatory power of the test with 95% CI and taking p -value to be ≤0.05 as significant. Hosmer–Lemeshow goodness-of-fit test was used to check the calibration of the score in our ICU population taking p -value >0.05 showing a good fit of the model.

Results

A total of 172 patients were enrolled in the study. Data were analyzed for the 154 patients that fulfilled the selection criteria. All the patients had nonsurgical reasons for admission. Of these, sepsis (43, 27.9%) was the most common admission diagnosis followed by respiratory diseases (42, 27.27%), neurologic diseases (21, 13.6%), cardiac cases (15, 9.7%), and gastrointestinal (13, 8.4%) as depicted in Fig. 1 .

Fig. 1.

Admissions ( n  = 154) and mortality ( n  = 46) according to diagnosis (percentages).

The median age at admission was 0.50 years (IQR of 0.24–1.78) and the median weight was 5.0 kg (IQR of 3.08–10.00). The median (IQR) values of length of mechanical ventilation for those requiring it in the first hour of admission, ICU stay, and hospital stay were 72 hours (36.0–120.0), 2.0 days (1.20–3.50), and 5.0 days (3.0–10.0), respectively. There were 83 (53.9%) females and 71 (46.1%) males, while malnutrition was noted in 102 (66.2%) and hypotension in 44 (28.6%) patients.

The observed mortality was 29.87% (46 of 154) and that expected by PIM-2 score (mortality percentage ≥ 99.8%) was 27.92% (43 of 154) with an SMR of 1.07 (95% CI: 0.79–1.41). Sepsis contributed to the maximum percentage of mortality (52.2%, 24 out of 46). The analysis revealed a sensitivity of 54.3% and a specificity of 83.3%, with a p -value of <0.01. This is shown in Table 1 .

Poststratification analysis of data variables showed the following results. Age stratification for mortality was found significant only for age cut-off of 6 months. The mortality rate in children less than 6 months of age was 36.9% (31 out of 84) while it was 21.4% (15 out of 70) in those above 6 months of age with a p -value of 0.037. Patients with a weight of less than 5 kg had a mortality of 38.5% (30 out of 78) while those weighing more than 5 kg had a mortality of 21.1% (16 out of 76; p  = 0.018). In female patients, mortality was 33.7% (28 out of 83) while it was 25.3% in males ( p  = 0.257). Higher mortality was seen in patients who were malnourished, 33.3% (34 out of 102) as compared with 23.1% (12 out of 52) in well-nourished patients ( p =  0.188).

It was found that the absence of pupillary response to light at admission in the two patients was associated with mortality in both. Moreover, only 28.9% (44 out of 152) of those having positive pupil response died during the specified period of PICU admission ( p  = 0.08).

The association of mechanical ventilation (either CPAP, BiPAP, or definitive airway) required in the first hour of ICU admission was also analyzed with the mortality. In total, 70% (21 out of 30) of patients who required mechanical ventilation in the first hour died, while only 20.2% (25 out of 124) patients died in the other group ( p  < 0.001). Among the level of diagnosis, 50% mortality (9 out of 18) was observed in the high-risk diagnosis group, while only 10.5% (2 out of 19) mortality was observed in the low-risk mortality group, and approximately 30% (35 out of 117) mortality was observed in neither high-risk nor low-risk diagnostic group. Of these, sepsis was the most common diagnosis ( p  = 0.03).

The data variables were also compared between survivors and nonsurvivors, revealing statistically significant differences in both groups, illustrated in Table 2 .

Table 2. Comparison of survivors and nonsurvivors ( n  = 154) .

	Survivors ( n  = 108)		Nonsurvivors ( n  = 46)		p -Value
	Mean	SD	Mean	SD
Age (y)	2.07	3.09	1.51	2.83	0.15
Ventilation length (h)	61.33 ( n  = 9)	14.00	78.77 ( n  = 22)	50.64	0.44
ICU stay (d)	2.63	3.36	2.79	1.92	0.30
Hospital stay (d)	9.76	9.02	3.56	3.68	0.00 a
Hypotension (PALS 2018)	21 ( n )	19.4 (%)	23 ( n )	50.0 (%)	0.00 a
Base excess (meq/L or mmol/L)	9.55	7.50	12.27	8.58	0.07
Percentage of inspired oxygen (%)	33.95	11.64	59.83	32.79	0.00 a
PaO 2 (mmHg)	136.50	48.76	141.58	56.35	0.42
Risk of mortality (percentage)	50.56	44.44	81.81	32.57	0.00 a

Abbreviations: ICU, intensive care unit; PaO ₂ , partial pressure of oxygen; PALS, Pediatric Advanced Life Support; SD, standard deviation.

^ap -value ≤0.05 is significant as determined by Mann–Whitney U test.

Area under the receiver operating characteristics curve (AUC) for the current PIM-2 model was 0.75 (95% CI: 0.67–0.84) with a p -value of <0.001 ( Fig. 2 ), which showed acceptable discrimination between survivors and nonsurvivors. Hosmer–Lemeshow goodness-of-fit test along the risk deciles of mortality produced a Chi-square value of 15.80 (df = 7, p  = 0.027; Table 3 ).

Fig. 2.

Receiver operating characteristic curve.

Table 3. Contingency table for Hosmer and Lemeshow test ( n  = 154) .

Groups	Actual mortality = no		Actual mortality = yes		Total
	Observed	Expected	Observed	Expected
1	14	13.452	1	1.548	15
2	14	13.424	1	1.576	15
3	13	13.320	2	1.680	15
4	12	12.718	3	2.282	15
5	9	10.437	6	4.563	15
6	13	8.933	2	6.067	15
7	11	9.029	5	6.971	16
8	10	7.239	3	5.761	13
9	12	19.448	23	15.552	35

Note: Statistical test for goodness of fit for logistic regression models. Chi-square value = 15.80; df =7; p -value = 0.027 (<0.05).

Discussion

Mortality risk assessment models can be applied to PICU populations for grading standard of care, which is gauged in the form of risk-adjusted mortality rates. Albeit, some believed that using an erratic event such as mortality in an ICU may not be the most appropriate approach for this purpose, ⁹ considering that morbidity issues and ICU utilization in terms of length of stay are not managed in this way. However, these models do indicate how well PICUs function at their core purpose of prevention of death in critically ill children. They provide a standardized way of comparing the performance of ICUs with themselves over time and with national and international benchmarks, and are useful tools to track the level of care across diagnostic groups, although careful consideration should be given to calibration of these tools.

PIM-2 has been found to have good discrimination and calibration in the units in which it was derived. ¹⁰ However, mixed reviews have been described in the validation studies that were done in developing countries because of the different population demographics, admission protocols, case mix, and level of care provided in these ICUs. These mostly included single-center studies. One study from India ¹¹ compared three mortality scores over a case-mix of patients, concluding an acceptable performance (AUC = 0.728) and good calibration. Calibration was assessed by using the Hosmer–Lemeshow goodness-of-fit test. Another study conducted in Kings College Hospital, United Kingdom ¹² enrolled patients of pediatric acute liver failure and demonstrated poor discrimination. A study from Trinidad and Tobago including 217 patients in an ICU which cared for both adults and children found that among children, the PIM-2 did not show good discrimination, with an AUC of 0.62, a Hosmer–Lemeshow p -value of 0.69, which claimed a good fit of the model for calibration and a lower observed mortality rate (30%) than expected (34%). ¹³

We observed mortality of 29.9% in our ICU population which was similar to some other ICUs of developing countries like India (30% in 2007, 46.2% in 2011, 28% in 2014, and 34% in 2018) ^{4 13 14 15} and Pakistan (28.7% in 2007), ⁷ albeit higher than ICUs of the western world with greater medical personnel, nursing staff, and specialized equipment. The expected mortality computed by PIM-2 score was 27.9% giving an SMR of 1.07 (95% CI: 0.79–1.41), which was almost equal to 1 but showed that PIM-2 score just slightly underpredicted mortality which was quite contrary to well-established centers which depicted overprediction by the model with SMR <1. ^{10 13 16} Most of the developing countries have reported an SMR above 1, ^{2 15} suggesting under prediction by the model including one study in Pakistan which reported an SMR of 1.47. ⁷

Our results also revealed a significantly increased mortality in infants below 6 months of age and children with weight less than 5 kg. Malnourishment was a common problem associated with the majority of admissions (66.2%), although it was not significant.

The sensitivity and specificity of the PIM-2 score were 54.3 and 83.3%, respectively, which suggests that it cannot be used as a screening tool for mortality, but the high specificity denotes that the score predicts survivors accurately. Various risk factors were also analyzed for their association with mortality including pupillary response to light at admission, an early requirement of ventilation within first hour of ICU admission, and the severity of the diagnosis. There was 100% mortality with an absence of pupillary response to light. Others have reported similar results. ^{10 14} A higher mortality in the high-risk diagnostic group and those with ventilation requirement in the first hour has also been reported by Gandhi et al (61.7% mortality in patients requiring mechanical ventilation, p  = 0.00) ⁴ and Sankar et al (44% mortality associated with ventilation requirement, p  < 0.001) and 13% in high-risk diagnostic groups ( p  = 0.01). ¹⁴

The comparison of survivors and nonsurvivors was also significant with a shorter hospital stay, greater percentage of hypotension, greater FiO ₂ requirement during ventilation, and greater risk of mortality among the nonsurvivor group. Other researchers have reported variable results. These include a greater median length of stay among nonsurvivors (5 days) than the survivors (4 days), ¹⁴ a greater mean duration of stay among nonsurvivors (15.6 ± 17.9 days) compared with survivors (7.3 ± 6.4), and a greater PIM-2 score value among nonsurvivors. ² Hariharan et al also reported similar findings with a greater mean duration of stay and greater PIM-2 score value among the nonsurvivor group. ¹⁶

The performance of PIM-2 was assessed by evaluating its discrimination power and calibration. Discrimination is the ability of the model to categorize patients into two outcome groups such as survivors and nonsurvivors, and it is assessed by measuring the AUC. ¹⁷ Acceptable discrimination is represented by the AUC of >0.7, the value of >0.8 suggests good discrimination, and >0.9 is labeled as excellent. ¹⁶ Our study reported an AUC of 0.75 (95% CI: 0.67–0.84; p  = 0.00), which showed an acceptable performance. Other studies from developing countries reported an AUC of 0.82 from Barbados, ¹⁶ 0.62 from Trinidad, ¹³ 0.74 and 0.69 from India, ^{14 15} and 0.75 and 0.79 from Egypt. ^{2 18} Some of the studies from developed countries also reported an AUC of 0.726 (China), ⁵ and 0.79 in Italy. ³ However, most of the well-developed centers had higher values of AUC including 0.9 from Australia and New Zealand ¹⁰ and 0.87 from Spain. ¹⁹ A previous study done in Lahore, Pakistan reported an AUC of 0.88 with a good calibration of the model, ⁷ as it was done in an ICU with a higher influx of cardiac and postsurgical patients and patient to nurse ratio of 1:1.

The calibration is assessed by the Hosmer–Lemeshow goodness-of-fit test, which measures the correlation between the predicted outcome and actual outcomes over the entire range of risk prediction. ²⁰ A good calibration is demonstrated by a p -value of >0.1. ²¹ Our study established a poor calibration of the model in our ICU population with Hosmer–Lemeshow goodness-of-fit test results of Chi-square = 15.80, df = 7, and p -value of 0.027 ( Table 3 ). These results were consistent with findings of an Italian study that stated that the PIM-2 model overpredicted deaths in their population, ³ which was akin to an Indian study. ¹⁵ Similar findings were reported by other authors. ^{12 22 23 24} They variably attributed this to the differences in the characteristics of the study population, small sample sizes, and better quality of intensive care in the units where PIM-2 was established. Perhaps similar factors are responsible for the results of our study as well.

No missing data suggested easy application of the score because the included variables are usually available at admission to intensive care. Also, the score is calculated within 1 hour of ICU admissions so these variables are less likely to be affected by any intervention.

The main issues in our ICU were poor patient to nursing and medical staff ratio, limited and outdated equipment, delayed laboratory investigations, and a majority of nonsurgical patients. None of the admissions were elective or postoperative as pediatric surgery has an ICU of their own for that purpose and only patients with medical conditions are admitted to our ICU. It was a single unit study with limited sample size, having many patients with delayed presentation and referral to the emergency department, and later some even refusing appropriate investigations and therapies because of the social norms of the families and nonavailability of advanced treatments, which might have adversely affected the outcome. Also, the diagnostic categorization was done by the initial diagnosis at admission and was not reviewed in the score calculation with the final diagnosis at discharge and the cardiac diagnosis was not confirmed due to the nonavailability of pediatric echocardiography facility in the hospital. A multicenter study including different hospitals as well as a combination of both medical and surgical ICUs of the same hospital should be performed to validate this score, which might help in screening patients with a high risk of mortality to cater to their management at the earliest with the maximum possible resources.

Conclusion

Our study shows an acceptable discriminatory function of PIM-2 score in differentiating survivors from those who died. It showed good specificity but low sensitivity and had poor calibration for our population. A multicenter large-scale study is required including surgical patients to recalibrate the score according to case-mix and local population characteristics and look at the confounders affecting the mortality.

Funding Statement

Funding None.