Performance of Pediatric Index of Mortality PIM-3 in a Tertiary Care PICU in India

Abstract

Pediatric index of mortality-3 (PIM-3) is the latest update of one of the commonly used scoring systems in pediatric intensive care. It has free accessibility and is easy to use. However, there are some skepticisms regarding its practical usefulness in resource-limited settings. Hence, there is a need to generate region-specific data to evaluate its performance in different case mixes and resource constraints. The aim of the study is to evaluate the performance of the PIM-3 score in predicting mortality in a tertiary care PICU of a developing country. This is a retrospective cohort study. All children aged 1 month to 18 years admitted to the PICU during the study period from July 2016 to December 2018 were included. We reviewed the patient admission details and the case records of the enrolled. patients. Patient demographics, disease profile, co-morbidities, and PIM-3 scores were recorded along with the outcome. Area under receiver operating characteristics (AUROC) curves was used to determine discrimination. Standardized mortality ratio (SMR) and Hosmer Lemeshow goodness of fit were used to assess the calibration. Out of 282 children enrolled, 62 (21.9%) died. 58.5% of the patients were males, and 60% were less than 5 years of age. The principal diagnoses included respiratory and neurological conditions. The AUROC for PIM-3 was 0.961 (95% CI [0.93, 0.98]) and overall SMR was 1.28 (95% CI [0.96, 1.59]). Hosmer-Lemeshow goodness-of-fit was suggestive of poor calibration ( χ ²  = 11.7, p  < 0.05). We concluded that PIM-3 had good discrimination but poor calibration in our PICU setting.

Introduction

Mortality reduction is one of the fundamental aims of any intensive care unit (ICU). Critical care physicians often use a set of clinical signs at admission to determine the likelihood of mortality. Pediatric ICUs are high stake areas constantly grappling with several sick children at any given point with limited workforce and resources, especially in developing nations. Scoring systems can thus be valuable in sickness estimation and mortality projections in the ICU. It may also facilitate inter-mural comparisons and guide quality improvement drives. ¹

Predictive models use statistical equations to identify high-risk patients based on a combination of clinical variables, laboratory parameters, and risk factors to predict mortality. The last three decades have witnessed the advent of several predictive models in pediatric critical care. However, most of the well-known predictive models are based on data from high-income countries. These scoring systems can be broadly classified as predictive and descriptive scores. ² Predictive scores predict the risk of death at the time of admission. These include pediatric risk of mortality (PRISM) and pediatric index of mortality (PIM). ^{3 4} In contrast, descriptive scores are used to assess the magnitude and extent of multiorgan dysfunction. These include pediatric logistic organ dysfunction (PELOD) ⁵ and pediatric multiple organ dysfunction scores (PEMOD). ⁶

Worldwide, the most commonly used mortality predictive scores are PRISM and PIM. Both these models have been developed and validated in high-income countries. To date, no predictive model has been explicitly designed to account for the differences in disease epidemiology, resources, and demographics of developing regions.

PRISM score is a validated scoring system developed by the United States of America (USA). It has undergone three updates, with the latest version issued in 2015 known as PRISM-IV. ⁷ The new algorithm includes separate scores for the age of the patient, source of admission, any cardiopulmonary arrest within 24 hours before admission, cancer, and low-risk systems of primary dysfunction in addition to the neurological and nonneurological PRISM-III scores. However, the time frame for clinical measurements has been reduced to the first 4 hours of PICU care, and the laboratory variables are measured in the period from 2 hours before PICU admission through the first 4 hours. Advantages of its use are its ability to discriminate nonsurvivors from survivors along with good calibration reported in numerous studies. However, it is time consuming and quite complex. Studies that evaluated PRISM in developing nations have shown favorable results, with Qureshi et al reporting good discrimination and calibration of PRISM-III (area under receiver operating characteristics [AUROC] 0.78, [95% CI: 0.67, 0.89]; χ ²  = 7.49, p  = 0.49) in their PICU. ⁸ In another study by Choi et al, PRISM-III accurately predicted mortality (AUROC 0.79, [95% CI 0.65, 0.98]; p  = 0.39). ⁹ However, Patki et al reported an underprediction of mortality with PRISM-III (5.68%) and PIM-2 (8.84%) in their cohort. ¹⁰ Nevertheless, both models had good calibration as well as discrimination in their study. Recently, Garcia et al also reported a similar performance of PRISM-IV in their cohort. ¹¹

Another popular predictive score is PIM. It is a much simplified scoring system using fewer variables. It was based on the data from seven PICUs in Australia and United Kingdom. The second version of this score, known as PIM-2, is the updated version with a larger contemporary dataset of 20,787 children in Australia, the United Kingdom, and New Zealand. It has been validated across the globe in several studies. ⁴ Developing regions have also reported good discrimination and calibration in several studies. ^{12 13 14}

The latest version of this model, PIM-3, was developed in 2010-2011 as an advancement over the previous version. ¹⁵ A significant limitation of PIM is that it does not take into account the effect of treatments received prior to admission to the PICU. In countries like India, prehospital management is unavailable to a large section of the population and thus may not influence the performance of the PIM model. However, very few studies from resource-constraint regions have evaluated the performance of PIM-3. ^{16 17} Jung et al studied the performance of PRISM III, PIM-2, and PIM-3 in Korean PICU and found AUROC of 0.77, 0.79, and 0.82, respectively. ¹⁶ There were no significant differences in the Hosmer-Lemeshow test results for PRISM III ( p  = 0.49), PIM-2 ( p  = 0.24), and PIM-3 ( p  = 0.33). The standardized mortality rate (SMR) calculated using PIM-3 (1.11) was closer to 1 than PIM-2 (0.84). Abdelkader et al from Egypt reported the overall expected mortality using the PRISM-III as 6.7% and PIM-3 as 7.4%, with the observed mortality of 17.6%. ¹⁸ There was an underprediction of mortality at all probability levels with both the tests. Though both these tests demonstrated excellent discrimination with an AUROC of 0.9 each, only PRISM-III had good calibration ( χ ²  = 4.57, p  = 0.148) while PIM-3 had poor calibration ( χ ²  = 8.66, p  = 0.01).

In the Indian subcontinent, notable is the work of Tyagi et al who compared PRISM-III, PIM-2, and PIM-3 in 350 patients admitted to the PICU. They concluded that PIM-3 had a good discrimination ability with reported AUROC for PRISM III, PIM-2, and PIM-3 as 0.667, 0.728, and 0.726, respectively. There was also good calibration of all three scoring systems in their study cohort. ¹⁹ On the other hand, Poddar et al reported better discrimination with PIM-2 (AUC = 0.83, [95% CI: 0.73, 0.92]) compared with PIM-3 (AUC = 0.70, [95% CI: 0.58, 0.82]). However, the Hosmer–Lemeshow goodness-of-fit test showed poor calibration for PIM-2 ( χ ²  = 21.6, p  = 0.006) and PIM-3 scores ( χ ²  = 23.6, p  = 0.003). ²⁰ Sankar et al made similar conclusions in their research which indicated good discrimination but poor calibration of PIM-3 in their study cohort. ²¹ With such limited data available on PIM-3, we conceptualized this study to evaluate the performance of PIM-3 in our region.

Materials and Methods

This retrospective cohort study was conducted in a recently established tertiary care PICU in Western Rajasthan. All patients aged 1 month to 18 years admitted to PICU during the study period of July 2016 to December 2018 were included. Admission details were reviewed from the health management information system (HMIS), and further records were obtained to extract PIM-3 score variables within the first hour of admission for each child. The primary routes of admission in the PICU were from pediatric emergency, operation theaters, and pediatric ward. Readmissions were regarded as new admissions, and the probability of death was estimated based on the PIM-3 variables at the time of the current admission. Exclusion criteria included children who either died or were transferred out within 1 hour of admission and patients with incomplete data. The baseline demographic data, dataset pertaining to PIM-3 variables, morbidity profile, and their outcomes as either death or discharge from PICU were compiled. PIM-3 variables included pupillary reaction to bright light, systolic blood pressure, partial oxygen tension (PaO ₂ ), fraction of inspired oxygen (FiO ₂ ), base excess in arterial blood gas analysis; mechanical ventilation at any time during the 1 hour of PICU admission; elective admission to PICU; recovery from surgery or the procedure that was the main reason for ICU admission. It also included predefined diagnoses according to the developer's guidance as a low-risk, high-risk, or very high-risk diagnosis. The scores were generated using an online PIM-3 calculator. ¹⁵

Statistical Analyses

Data were analyzed using IBM SPSS version 23.0 (IBM Corp., Armonk, New York, United States). The performance of PIM-3 was assessed by discrimination and calibration. The ability of the scoring system to distinguish survivors and nonsurvivors accurately is termed as discrimination and was assessed by calculating the AUROC. AUROC values are interpreted as fail 0.50 to 0.60, poor 0.60 to 0.70, fair 0.70 to 0.80, good 0.80 to 0.90, and excellent >0.90. ²² The correlation of actual outcomes and predicted outcomes is known as calibration. For this, the mortality across the deciles of risk and Hosmer-Lemeshow goodness-of-fit was done in addition to SMR. All the children were classified by the probability of death in three groups. <5%, 5 to 14.99%, and ≥15% probability of death. The average probability of death and predicted probability were estimated and further used to calculate the SMR. SMR is defined as the actual prevalence of death divided by the average predicted probability of death. ROC was prepared by considering the probability of death in the two groups (nonsurvivors and survivors). A threshold cut-off point of 21.5 was estimated as it had the maximum sensitivity and specificity. In addition, sensitivity, specificity, positive predictive value, and negative predictive value were also calculated.

Results

A total of 362 records were screened during the study period, of which 75 were excluded as per the preset criteria. The flow of participants in the study is given in Fig. 1 . The final dataset comprised 282 children with a median age of 24 months (range: 1–216 months), among which 62 children had died (21.9%). Males constituted 58.5% of the cohort. The common admitting diagnoses included respiratory conditions in 48 (17%) followed by neurological illness in 43 participants (15.2%), while sepsis accounted for 9.6% of admissions ( Table 1 ). Severe acute malnutrition was documented in 26 cases (8.9%). Eleven (3.9%) patients had a history of cardiac arrest at the time of admission, which qualified as a very high-risk diagnostic variable in PIM-3 scores. The median PIM-3 scores and probability scores with their interquartile range in each group are listed in Table 1 . A higher PIM-3 score indicated a higher probability of mortality. The SMR calibration of the PIM-3 model was based on the PIM-3 probability intervals of <5%, 5 to 14.99%, and ≥15%. Table 2 shows the PIM-3 probability intervals and the subject's outcomes. The majority of patients belonged to the <5% mean probability group. The overall observed mortality was 22% compared with the cumulative expected mortality of 17%, with an SMR of 1.28. The Chi-square goodness of fit was ( χ ² for goodness of fit = 11.7, p  < 0.05), suggesting poor calibration. The AUROC analysis showed an excellent discriminatory ability of the PIM-3 to distinguish between survivors and nonsurvivors, with an AUROC of 0.96 (95% CI: [0.93, 0.98]; Table 3 ; Fig. 2 ). PIM-3 had a sensitivity of 87% and specificity of 90%, and a positive predictive value of 72% ( Supplementary Table 1 , available in the online version only). The SMRs in the overall demographic and the clinical course group were in the range of 1 to 1.5, except for predicting mortality in sepsis and malaria, where it was 2 and 2.78, respectively depicted in the Forest plot ( Supplementary Fig. 1 , available in the online version only). Additional calibration assessment done by plotting the observed versus the predicted mortality across deciles of risk and performing the Hosmer-Lemeshow goodness-of-fit ( p  < 0.05) indicated poor calibration model ( Supplementary Fig. 2 , available in the online version only).

Fig. 1.

Flow of participants in the study.

Table 1. Demographic and clinical characteristics of the study population.

Variables ( n  = 282)	Survivors n (%)	Nonsurvivors n (%)	Total n (%)
Males	132 (80.0)	33 (20.0)	165 (58.5)
Age (y)
< 1	66 (79.5)	17 (20.5)	83 (29.4)
1 to <5	68 (76.4)	21 (23.6)	89 (31.6)
5 to <10	32 (84.2)	6 (15.8)	38 (13.5)
≥10 y	54 (75.0)	18 (25.0)	72 (25.5)
Diagnoses
Sepsis	12 (44.4)	15 (55.6)	27 (9.6)
Respiratory	41 (85.4)	4 (14.6)	48 (17)
Neurological	34 (79.1)	9 (20.9)	43 (15.2)
Cardiac	16 (72.7)	6 (27.3)	22 (7.8)
Postoperative	34 (97.1)	1 (2.9)	35 (12.4)
Liver failure	17 (60.7)	11 (39.3)	28 (9.9)
Renal disease	13 (72.2)	5 (27.8)	18 (6.4)
Metabolic and endocrine	11 (91.7)	1 (8.3)	12 (4.3)
Dengue	8 (88.9)	1 (11.1)	9 (3.2)
Malaria	1 (50.0)	1 (50.0)	2 (0.7)
Others	33 (86.8)	5 (13.2)	38 (13.5)
Outcomes	220 (78)	62 (22)	282
PIM-3 score a	−3.13 [−5.31, −2.33]	0.08 [−0.97, 1.27]	−2.7 [−4.68, -1.21]
Probability a	4 [0,9]	51.5 [27,78.5]	6 [1,23]

^aMedian [IQR].

Table 2. Observed vs. predicted mortality as per PIM-3 scores.

Probability	Mean probability	N	Observed n (%)		Predicted n (%)		SMR (95% CI)
			survivors	Death	Survivors	Death
<5%	1.35	117	117 (100)	0	115.43 (98.6)	1.58 (1.4)	0
5–14%	7.74	69	66 (95.7)	3 (4.3)	63.66 (92.3)	5.34 (7.7)	0.56 a (0.11, 1.64)
≥15%	43.17	96	37 (38.5)	59 (61.5)	54.56 (56.8)	41.44 (43.2)	1.42 b (1.08, 1.84)
Total	17.15	282	220 (78.0)	62 (22.0)	233.64 (82.85)	48.48 (17.2)	1.28 (0.98, 1.63)

Note: χ ² for goodness of fit = 11.7, p <0.05; Standardized mortality rate (SMR) = observed death/predicted death.

^aExact approximation.

^bPoisson approximation.

Table 3. PIM-3 probability based on AUC-derived threshold cut off.

Probability	Mean probability of death	n	Observed n		Predicted n		AUROC (95% CI)	SMR (95% CI)
			Survivors	Death	Survivors	Death
<21.5	5.14	207	199	8	196.37	10.63	0.89 (0.82, 0.95)	0.75 a (0.32, 1.48)
≥21.5	50.28	75	21	54	37.29	37.71	0.84 (0.75, 0.93)	1.43 a (1.07, 1.86)
Total	17.15	282	220	62	232.65	49.35	0.96 (0.94, 0.98)	1.25 (0.96,1.60)

Note: χ ² for goodness of fit = 8.42, p <0.05; Standardized mortality ratio (SMR) = observed death/predicted death.

^aPoisson approximation.

Fig. 2.

ROC curve analysis for PIM 3 scores with area under the curve (AUC).

Discussion

Mortality risk prediction models in PICUs are essential means to assess intensive care quality and performance. However, they need to be validated in a locally representative sample before routine implementation. In low- and middle-income countries (LMIC), most deaths occur soon after admission and are primarily caused by a high burden of tropical diseases such as malaria, pneumonia, and diarrheal diseases. ²³ Most of the predictive models have been developed in regions with different disease epidemiology and better resources. Various health care facilities may also have significant differences in admission thresholds, case mix variety, and available resources, which can influence the performance of a predictive model. Hence, it is essential to regularly assess the accuracy of available prognostic scores in different case-treatment milieus.

In this study, the patient profile included a good mix of medical and surgical cases. The clinical profile of our patient population predominantly had infections, vector-borne diseases, and malnutrition, while genetic disorders and trauma are the predominant causes of mortality in developed nations. We found an AUROC of 0.96, which indicates an excellent ability of PIM-3 to discriminate survivors from non-survivors. Calibration of PIM-3 was assessed by Hosmer–Lemeshow goodness-of-fit test and SMR. The former test showed that the PIM-3 model has poor calibration in our PICU. The SMR in this study was 1.28, which indicates under prediction of deaths by the PIM-3 scores.

The prevalence of mortality in our PICU in the 2.5 years was 22%, lower than other studies from our region. Reported mortality by Honna et al ²⁴ from Indonesia is 45.7%, Gandhi et al from India ²⁵ is 46.2%, Tyagi et al from India is 39.4%, ¹⁹ and Qureshi et al from Pakistan ⁸ is 28.7%. However, the median mortality risk was much higher in our study cohort than in the studies from which PIM-3 was derived. ²⁶ This indicates that the patient cohort in our study was sicker at admission. Also, it may be worthwhile to note, delayed presentation of patients in a resource-constrained setting like ours could perhaps account for this discrepancy in mortality estimates.

However, despite the regional differences in disease prevalence and severity, the discrimination of PIM-3 in our study was excellent, as indicated by an AUROC of 0.96. Our results align well with the PIM-3 development study, which reported an AUROC >0.80 for PIM-3. ¹⁵ Other researchers from developed countries have made similar inferences with AUROCs ranging from 0.91 in Australia, 0.90 to 0.93 in New Zealand, 0.85 in the UK, and 0.84 to 0.86 in Scotland. ¹⁵ Emerging data from some developing regions have also confirmed the good discriminatory ability of PIM-3. ²⁷ However, some studies have reported poor discrimination in specific subsets of patients, such as haemato-oncological cases. ¹⁷ While a multicenter study in Italy confirmed a significant improvement in PIM-3 compared with PIM-2 ³³ , Tyagi et al did not find any significant improvement between PIM-2 and PIM-3. ¹⁹

Another performance measure is the calibration of the model. It is the comparison of a given unit's performance as against that of the original study. It is assessed by SMR and Hosmer–Lemeshow goodness-of-fit. In this study, overall SMR was >1, indicating that PIM-3 underpredicted mortality in our cohort though it was not statistically significant. This was most pronounced in the ≥15% risk probability subgroup with an SMR of 1.42. Disease-specific SMR indicated a value of 2 in sepsis and 2.78 in malaria patients, i.e., these two groups had twice the predicted mortality in the study. While the cohort in the present work is small, the high SMR noted with malaria seems to be a statistically sensible finding given the original PIM models were likely trained in cohorts with essentially zero malaria. However, the minimal number of malaria cases in this cohort is unlikely to have affected the model's performance in the present study. Nonetheless, the lack of “high risk” diseases encountered in resource-limited settings is a substantial limitation of the PIM-3 score, which may explain the higher SMR in tropical conditions. Other contributing factors for higher SMR include malnutrition, poor referral system, delayed diagnosis, complications of ICU treatments, and lack of advanced treatment modalities such as ECMO, transplant surgeries, etc. In the present study, the goodness-of-fit test was statistically significant, indicating poor calibration of PIM-3 in our setting. Some of the possible reasons may be considerable heterogeneity of disease prevalence and patient characteristics. Second, there can be a temporal drift in patient demographics due to changing referral patterns, health care policies, and treatment protocols with time. Lastly, there can be methodological problems with the algorithm itself. Statistical overfitting is a common occurrence in a complex model, which tends to capture random noise in the data. Similar to our findings, Poddar et al have also reported poor calibration of PIM-3 in their study. ²⁰

Emerging data reiterates this finding in the works of other researchers from the developing world, indicating that PIM-3 might underpredict deaths in these resource constraint settings. ^{16 27} Most studies from developing countries have reported underprediction of mortality by the PIM scoring model. ^{13 28 29 30 31} By contrast, developed countries have noted a tendency of overprediction with PIM. ^{32 33} Sankar et al conducted an observational study to compare PIM-2 and PIM-3 in a cohort of 202 children, with mortality of 34%. ²¹ They found that sepsis and pneumonia were the common admitting diagnoses and concluded that PIM-3 had poor calibration across risk deciles similar to our study. Likewise, a study in Egypt concluded that PIM-3 had poor calibration in their region, and there was significant underprediction of mortality at all probability levels. ¹⁸

By contrast, studies from the developed world paint a different picture, with several centers observing good calibration. A large-scale retrospective multicentric study was conducted in 17 PICUs as a part of the Pediatric Intensive Therapy Network (TIPNet) from 2010 to 2014 across Italy. It included 17,109 children and concluded that PIM-3 had good discrimination as well as calibration. ²⁶ Similarly, Malhotra et al from UAE reported good calibration in their PICU patients. ³⁴

A recent study by Niederwanger et al, compared several scoring systems such as (PRISM, PRISM-III, PRISM-IV, PIM, PIM-2, PIM-3, PELOD, PELOD-2) to determine which is the most useful score for pediatric sepsis patients. They reported fair discriminating ability (AUROC, 95% CI) of PIM (0.76 [0.68–0.76]), PIM-2 (0.78 [0.72–0.78]), PIM-3 (0.76 [0.68–0.76)]), PRSIM-III (0.75 [0.68–0.75]), and PELOD-2 (0.75 [0.66–0.75]), while PELOD-2 (AUROC-0.84[0.77–0.91]) and PRISM-IV (AUROC 0.8[0.72–0.88]) had good discriminative ability. ³⁵

Some limitations of our study must be addressed. Since this is a single-center retrospective study with a small sample size, the results may not generalize to other centers or populations. Prospective, more extensive multicentric studies may help in further refinement of this model and can pave the way for achieving better quality standards amongst PICUs of the region. Furthermore, there is a need to design composite scores for developing nations like India, which include variables like malnutrition, malaria, resources, etc.

Conclusion

PIM-3 may be a promising option for mortality prediction in resource-constrained settings with good discrimination. However, a model updating exercise in resource-constrained settings is required whereby modifications are made to improve the calibration.

Funding Statement

Funding None.