Development of a severity of illness scoring system (ITAT) for resource-constrained hospitals in developing countries

Abstract

Objective

To develop a new pediatric illness severity score, called Inpatient Triage, Assessment, and Treatment (ITAT), for resource-limited settings to identify hospitalized patients at highest risk of death and facilitate urgent clinical re-evaluation.

Methods

We performed a nested case-control study at a Malawian referral hospital. The ITAT score was derived from 4 equally-weighted variables, yielding a cumulative score between 0 and 8. Variables included oxygen saturation, temperature, and age-adjusted heart and respiratory rates. We compared the ITAT score between cases (deaths) and controls (discharges) in predicting death within 2 days. Our analysis includes predictive statistics, bivariable and multivariable logistic regression, and calculation of data-driven scores.

Results

A total of 54 cases and 161 controls were included in the analysis. The area under the receiver operating characteristic curve was 0.76. At an ITAT cutoff of 4, the sensitivity, specificity, and likelihood ratio were 0.44, 0.86, and 1.70, respectively. A cumulative ITAT score of 4 or higher was associated with increased odds of death (OR: 4.80; 95% CI: 2.39 – 9.64). A score of 2 for all individual vital signs was a statistically significant independent predictor of death.

Conclusions

We developed an inpatient triage tool (ITAT) appropriate for resource-constrained hospitals that identifies high-risk children after hospital admission. Further research is needed to study how best to operationalize ITAT in developing countries.

Introduction

Similar to many resource-limited settings, Malawi struggles with both shortages and inadequate training of medical staff (Liese 2004, WHO 2006, Mueller et al. 2011). In a recent survey, only 42% of expected hospital staff days were adequately filled, just 63% of health care staff felt they were adequately trained, and merely 57% felt they were adequately supervised (Mueller et al. 2011). The country has taken steps to alleviate some of these inadequacies by training providers in Emergency Triage, Assessment, and Treatment (ETAT) (Gove et al. 1999), a validated component of the World Health Organization (WHO) Integrated Management of Childhood Illness (IMCI) program that provides simplified algorithms for triaging and treating patients on hospital admission. Although ETAT has led to dramatic improvements in mortality when used at initial triage (Robinson 2011, Robertson & Molyneux 2001, Molyneux et al. 2006), once patients are admitted to the inpatient ward, no further formalized surveillance system exists to provide continued monitoring and timely clinical intervention. As a result, students, nurses, and other junior health care workers are tasked with identifying deteriorating ward patients in often overcrowded facilities despite limited expertise. A recent government audit in Malawi found that inpatient pediatric mortality is often due to inadequate monitoring and delays in instituting emergency treatment (MMoH 2010). Additional inpatient tools similar to ETAT are needed to help undertrained staff recognize and intervene on deteriorating pediatric ward patients after admission.

In developed countries, severity-of-illness scores, often referred to as Pediatric Early Warning System (PEWS) scores, have gained popularity as inpatient tools to objectively identify and triage patients prior to their need for urgent resuscitation. Numerous scoring systems have been introduced with varying levels of complexity (Duncan 2007, Duncan et al. 2006, Monaghan 2005, Tibballs & Kinney 2006, Haines et al. 2006, Parshuram et al. 2009, Egdell et al. 2008, Edwards et al. 2009, Tucker et al. 2009, Tibballs et al. 2005), leading to substantial reductions in emergent resuscitation (Brilli et al. 2007), clinical deterioration events (Parshuram et al. 2011), and ward respiratory arrests (Hunt et al. 2008), as well as increased hospital staff confidence (Parshuram et al. 2011, Monaghan 2005) when combined with a rapid response team. Though adult severity of illness scores have been successfully implemented in South Africa (Rosedale et al. 2011) and Tanzania (Rylance et al. 2009), no pediatric scoring systems have been reported in similar developing settings.

Building on the successful implementation of the ETAT program during hospital admission, we aimed to develop a simplified pediatric early warning score that could provide continued monitoring and triage of patients throughout their hospitalization. Called Inpatient Triage, Assessment, and Treatment (ITAT), this tool was designed for resource-limited settings in which staff may have limited time or training to adequately identify deteriorating patients prior to death. Once identified, these high-risk patients could then be clinically reassessed with changes in treatment or possible referral to a higher level of supervision.

Methods

Study Population

Kamuzu Central Hospital (KCH) is a large referral hospital in Lilongwe, Malawi. Non-surgical patients are initially triaged on admission, then admitted to one of several wards, with the most critically ill sent to the acute care, malnutrition, or high-dependency wards. In 2010, KCH had a patient-to-nurse ratio of around 25 to 50:1 during the day and often exceeding 100:1 at night (unpublished data), an inpatient child mortality rate of 7%, and a pediatric HIV prevalence of 8.8% (McCollum et al. 2011). Although 3–4 physicians technically supervise the children’s wards, in reality, they often have many other obligations and the majority of clinical decisions are made by junior doctors, medical students, and clinical officers (COs). COs are a cadre of non-physician clinicians that provide the majority of clinical care in Malawi after receiving three years of theoretical and practical training followed by one year of clinical internship.

All children (age < 15 years) on the acute care and malnutrition wards at KCH from December 2010 to April 2011 (rainy season) were eligible for the study, which included the majority of children admitted to the hospital. Children on the surgical, high-dependency unit, and non-acute wards were given ITAT scoring sheets in their charts but were not included in the study because they were either less ill or were already receiving closely supervised care and the goal of the program was to identify pediatric patients on the general ward in need of higher levels of care.

ITAT Score

Vital signs are collected at KCH using automated vital sign poles, while respiratory rate is determined by observation. The ITAT score is derived from 4 vital signs: heart rate, respiratory rate, oxygen saturation, and temperature, with ranges based on a previously validated PEWS score (Duncan et al. 2006). A score from 0 to 2 was given for each vital sign based on the age-associated degree of abnormality, with higher scores resulting from more abnormal vital signs (Panel 1). The cumulative ITAT score had a range from 0–8. The previously validated individual parameters including heart rate (Parshuram et al. 2009, Haines et al. 2006), respiratory rate (Parshuram et al. 2009, Haines et al. 2006), oxygen saturation (Parshuram et al. 2009, Haines et al. 2006), and temperature (Parshuram et al. 2009) are all objective and important components of the pediatric advanced life support guidelines (Kleinman et al. 2010). Blood pressure was not included as it is technically more difficult, time consuming, and is generally considered a late sign of pediatric shock (Ceneviva et al. 1998). An assessment of mental status was also excluded as this was felt to be a more complicated and subjective evaluation. Clinicians were notified to review all patients with ITAT scores > 4, which we hypothesized would be the optimal threshold score, as it offered a balance of sensitivity in identifying high risk patients, but also enough specificity that it would not overburden the clinicians in this very resource-limited setting with “false positive” notifications.

Study Design

The ITAT analysis occurred within a 3-phase parent study evaluating two new programs: 1) the provision of vital sign equipment and the implementation of our ITAT score, and 2) the introduction of a new cadre of health care workers, called “Vital Sign Assistants,” who were to perform routine vital signs including the ITAT score with appropriate clinician notification (Olson et al. 2013).

We used a prospective nested case-control design to retain an appropriate distribution of time in the hospital among controls. Children were eligible for inclusion in the analysis if they had non-missing data for final outcome (death or discharge from ward), date of outcome, date of vital sign collection, and at least three of the four ITAT vital signs (oxygen saturation, temperature, heart rate, and respiratory rate). In subjects with only 3 vital signs, the missing vital sign was imputed using single imputation (‘proc mi’ procedure in SAS) to obtain maximum likelihood estimates of the mean vector and covariance matrix. The mean, standard deviation, minimum, maximum, and covariance matrix of all vital sign measures remained similar before and after the imputation. Descriptive statistics, including demographic data and diagnoses, were reviewed using SAS.

Children who died on the acute care or malnutrition wards during the study period were considered cases. Each case’s last set of vital signs taken before death was used. If the last set of vital signs was taken more than two days before death, the case was excluded. For each case, three controls were selected among a randomly sorted list of non-case vital sign sets, yielding a 1:3 case to control ratio. Sample size was limited to the number of subjects enrolled in the parent study during the same time period. Only vital sign sets that fell within 2 days of the case’s death were used as controls. As is typical and appropriate in nested case-control studies, controls could become a case at a later point in time; a control could serve as a control more than once, as sets of vital sign collections were the unit of analysis (Rothman et al. 2008). However, the same set of vital signs for an individual patient was not used more than once. Cumulative ITAT scores (ITAT score of 0–8) were calculated based on individual vital sign values, and used for analysis.

ITAT Score Development

To determine predictors of mortality, “high” scores, defined as having a cumulative ITAT score equal to or greater than the pre-determined cutoff, were compared to “low” scores, defined as having a cumulative ITAT score lower than the pre-determined cutoff, for all possible cumulative scores (0 to 8). Sensitivity (probability of having a high cumulative ITAT score among children that died), specificity (probability of having a low cumulative ITAT score among children that did not die), and stratum-specific likelihood ratio were calculated for each ITAT score cutoff. Bayes Theorem was used to calculate positive predictive value (PPV) and negative predictive value (NPV) for a cutoff score of 4, using the prevalence of death in our study population. Receiver operating characteristic (ROC) curves were then plotted using the sensitivity and specificity (1-specificity) for each ITAT score cutoff, and the area under the ROC curve was calculated using the trapezoidal rule. Microsoft Excel 2010 (Microsoft, Redmond, Washington, USA) was used for the above calculations.

Bivariable and multivariable logistic regression with robust variance was used to estimate odds ratios and 95% confidence intervals for the association between each predictor variable (oxygen saturation, temperature, heart rate, and respiratory rate) and final outcome (death or discharged from ward). Odds ratios approximated rate ratios due to density sampling. Predictor variables were assessed in the following ways: continuous form using various transformations, including splines; categories using disjoint indicator variables; and by their given score. We used disjoint indicator variables based on the Akaike’s Information Criterion (AIC). Variables were assessed for collinearity and extreme observations. All modeling was done using SAS version 9.2 (SAS Institute, Cary, North Carolina, USA).

Data driven ITAT scores were estimated in order to evaluate the utility of the assigned ITAT scores. Beta coefficients, which provide a natural linear interpretation for classification of the odds of disease, were derived from each of the predictor variables in the multivariable logistic regression model and were used in two ways: 1) as their unrounded value; and 2) rounded to the nearest integer to obtain the data driven ITAT score. The individual ITAT scores were then summed to obtain the cumulative data driven ITAT score. Sensitivity, specificity, stratum-specific likelihood ratio, and ROC curves were repeated using the data driven ITAT scores.

Ethical review

The study was approved by the University of North Carolina Internal Review Board and the Malawi National Health Sciences Research Committee. As vital signs are a routine component of patient care, individual patient consent was not required, although information pamphlets regarding the study were still made available throughout the children’s wards.

Results

In total, 1615 children were eligible for study inclusion while 377 were excluded for insufficient vital signs or outcome status, and 1084 had no vital signs taken during their hospitalization (Table 1). Imputation was used on 630 of the 3,983 eligible sets of vital signs with 1 missing vital sign. The remaining vital sign sets included all 4 vital signs. Missing data were due to incomplete documentation in patient charts. Fifty-four cases met eligibility criteria; with 161 controls included in the analysis (1 case only had 2 controls instead of 3 because only 2 controls met eligibility criteria). Case and control patients were unevenly distributed within the parent study, with the majority (41) of cases (and therefore controls) in the final third of the rainy (malaria) season. For both cases and controls, malaria, diarrhea/dehydration, and pneumonia were the top three diagnoses.

Table 1.

Characteristics of study cohort

	Died (N=54)*		Discharged (N=1,552)
	N	%	N	%
Gender
Male	21	39.62	792	51.13
Female	32	60.38	757	48.87
Missing	1		3
Age
0–1 months	2	3.70	102	6.57
2–6 months	5	9.26	159	10.24
7–<12 months	11	20.37	179	11.53
1–2 years	17	31.48	532	34.28
3–5 years	10	18.52	351	22.62
6+ years	9	16.67	229	14.76
Missing	0		0
Ward
Admission	51	94.44	1522	98.07
Malnutrition	3	5.56	30	1.93
Missing	0		0
HIV status
HIV exposed	3	15.00	56	7.73
HIV infected	4	20.00	57	7.87
HIV uninfected	13	65.00	611	84.39
Missing	34		828
Primary diagnosis
Convulsions	0	0.00	26	1.68
Diarrhea/Dehydration	6	11.11	120	7.75
Malaria	37	68.52	1117	72.11
Malnutrition	2	3.70	25	1.61
Pneumonia	5	9.26	184	11.88
Sepsis	1	1.85	50	3.23
Other	3	5.56	27	1.74
Missing	0		3
Day of week admitted
Monday - Friday	50	92.59	1189	76.61
Saturday - Sunday	4	7.41	363	23.39
Missing	0		0

^*9 children not represented who died but had vital signs collected

> 2 days before death, so eligible for control group only.

HIV: Human Immunodeficiency Virus

Table 2 shows the sensitivity, specificity, and stratum-specific likelihood ratio (LR) of each cumulative ITAT score in predicting mortality within 2 days. At a score of 4, which was the pre-determined threshold in which clinicians were notified to evaluate a patient, the sensitivity, specificity, and likelihood ratio were 0.44, 0.86, and 1.70, respectively. The area under the ROC curve was 0.76 (Figure 1). The positive predictive value (PPV) and negative predictive value (NPV) for a cutoff of 4 was 0.18 and 0.96 respectively, using the prevalence of death in our study population (6.7%, 206/3066).

Table 2.

ITAT score sensitivity, specificity, and likelihood ratios (LR)

Cutoff	Sensitivity	Specificity	LR
8	0.00	1.00	inf
7	0.06	1.00	inf
6	0.13	0.99	11.93
5	0.30	0.94	3.35
4	0.44	0.86	1.70
3	0.63	0.73	1.42
2	0.80	0.57	1.07
1	0.96	0.30	0.62
0	1.00	0.00	0.12

inf=infinity

Figure 1.

The receiver operating characteristics (ROC) curve for the Inpatient Triage, Assessment, and Treatment (ITAT) score. For an ITAT score of >=4, the sensitivity, specificity, and likelihood ratios were 0.44, 0.86, and 1.70, respectively.

Individual and composite predictors of mortality are shown in Table 3. Having a cumulative ITAT score of 4 or higher was independently associated with an increased odds of death, compared to having an ITAT score less than 4 (crude OR 4.80, 95% CI 2.39–9.64). For all vital signs except respiratory rate, a score of 1 (compared to a score of 0) did not predict death. A score of 2 (compared to a score of 0) was a significant independent predictor of death for all vital signs. Beta and data driven ITAT scores based on multivariable regression are also included but did not suggest that any individual vital sign should have greater weight in determining the ITAT score.

Table 3.

Predictors of mortality, using disjoint indicator coding for scores.

Individual Vital Signs				All Vital Signs
	Beta	SE	Odds Ratio (95% CI)	Beta	SE	Odds Ratio (95% CI)	Data driven score
Overall risk score
>= 4 vs <4	1.5686	0.3557	4.80 (2.39 – 9.64)
Oxygen saturation
1 vs 0	0.5631	0.4713	1.76 (0.70 – 4.42)	0.0937	0.4797	1.10 (0.43 – 2.81)	0.00
2 vs 0	2.8657	1.1096	17.56 (2.00 – 154.53)	2.1610	1.2186	8.68 (0.80 – 94.59)	2.00
Temperature
1 vs 0	0.6078	0.3647	1.84 (0.90 – 3.75)	−0.0194	0.4405	0.98 (0.41 – 2.33)	0.00
2 vs 0	2.8338	0.5877	17.01 (5.38 – 53.83)	2.0289	0.6843	7.61 (1.99 – 29.08)	2.00
Heart rate
1 vs 0	0.1518	0.4088	1.16 (0.52 – 2.59)	0.1105	0.4546	1.12 (0.46 – 2.72)	0.00
2 vs 0	1.3654	0.3957	3.92 (1.80 – 8.51)	0.5657	0.5238	1.76 (0.63 – 4.92)	1.00
Respiratory rate
1 vs 0	1.5612	0.3627	4.76 (2.34 – 9.70)	1.1582	0.4364	3.18 (1.35 – 7.49)	1.00
2 vs 0	2.0083	0.5264	7.45 (2.66 – 20.91)	1.0124	0.6138	2.75 (0.83 – 9.17)	1.00

SE = Standard Error

The sensitivity, specificity, and likelihood ratio for data driven ITAT scores are described in table 4 and again show little improvement compared to the original ITAT score. The data driven area under the ROC curve was 0.78.

Table 4.

Data driven score sensitivity, specificity, and likelihood ratios (LR)

Cutoff	Sensitivity	Specificity	LR
7	0.00	1.00	inf
6	0.00	1.00	inf
5	0.04	1.00	inf
4	0.22	0.98	9.94
3	0.31	0.97	7.45
2	0.50	0.85	1.57
1	0.83	0.65	1.68
0	1.00	0.00	0.26

inf=infinity

Discussion

In clinical settings lacking sufficient personnel and training such as KCH, improved inpatient monitoring and triage systems are necessary to identify and treat critically ill pediatric patients prior to deterioration and death. The ITAT scoring system includes a simple objective algorithm that builds on an already existing program to continue monitoring and triage after a patient has been admitted to the ward. Using this scoring system, health care personnel can quickly and efficiently identify children at high risk of death on the ward, allowing clinicians the opportunity to review and intervene on those patients appropriately.

Previously described severity of illness scores have gained widespread acceptance in the developed world, showing improved outcomes following their implementation (Parshuram et al. 2011, Brilli et al. 2007, Hunt et al. 2008). Several limitations, however, inhibit their potential effectiveness in resource limited settings. One such limitation is that greater performance comes with the cost of greater complexity. The best performing scores such as those developed by Duncan (Duncan et al. 2006) (sensitivity 78%, specificity 95%) and Parshuram (Parshuram et al. 2009) (82% sensitivity, 93% specificity), include many items (up to 16) or are time intensive and clinically subjective (level of consciousness, respiratory effort, airway threat). In addition, several scoring systems include items that are dependent on treatment already provided such as oxygen (Duncan et al. 2006, Edwards et al. 2009, Monaghan 2005, Parshuram et al. 2009, Egdell et al. 2008) intravenous fluids (Duncan et al. 2006, Haines et al. 2006), and nebulized adrenaline (Haines et al. 2006). These items have less utility in settings where their associated treatments are underutilized or not available. Previously reported scoring systems are designed to identify acutely deteriorating patients, often within hours of their arrest or transfer to intensive care. In settings lacking resources for rapid response teams, intensive care, or even routine vital signs, scoring systems may be more appropriately used to guide overall management than to provide triggers for urgent resuscitation.

The greatest strengths of the ITAT score are that it can be done quickly and that it requires only a minimal amount of clinical expertise. With only 4 easily measured items, all of which can be objectively measured using vital sign equipment or direct observation, a single ITAT score assessment can be completed more quickly than previously reported scores. Furthermore, the ITAT score does not include items requiring high levels of clinical knowledge such as evaluations of mental status and respiratory effort, or items that are based on responses to clinical interventions such as IV boluses and oxygen. The ITAT score shows similar characteristics compared to the data-driven ITAT score, suggesting our algorithm would not benefit from more complicated weight adjustment. This overall simplicity allows for the possibility of task-shifting this tool to less costly, capable workers who would require less training, allowing personnel with greater experience to focus on more specialized tasks. The ITAT score may be used as a continuous surveillance system with multiple assessments performed each day, allowing clinicians an objective measure of a patient’s status over time and opportunities to review and change management plans where appropriate. In settings with high staff and patient turnover, this tool provides some level of continuity between multiple assessments from different clinicians. The threshold score for clinician notification can also be adjusted depending on human resource constraints. Using our data, decreasing the ITAT threshold score to 3 or greater would more than double the number of clinician notifications (unpublished data).

The ITAT score’s simplified design, however, also leads to several limitations. One such limitation is that it lacks the accuracy of more complicated scoring systems. Previously validated scores report an area under the ROC curve of 0.86–0.91 (Duncan et al. 2006, Edwards et al. 2009, Egdell et al. 2008, Parshuram et al. 2011, Tucker et al. 2009) compared with the ITAT score ROC of 0.76, suggesting the more complex scoring systems are better able to discriminate patients with poor outcomes. Similarly, sensitivity and specificity are also slightly decreased with our scoring system. Our use of imputation as well as the use of clinician interventions for ITAT scores greater than or equal to 4 may have decreased our area under the ROC curve, sensitivity, and specificity. The use of single imputation may also slightly overestimate precision. Unlike previous scores that aim to recognize deteriorating patients requiring intervention within hours, the ITAT score is designed to predict pediatric death within 2 days. Thus, while children with an elevated score should be urgently evaluated, it should be emphasized that the score is designed as a surveillance tool on the general ward to provide continuous patient monitoring throughout their hospitalization. As is common at KCH, a substantial amount of missing data resulted from either incomplete or no vital signs being collected by health care staff, which may have introduced selection bias into our study. The absence of vital signs was not always at random. Children that died were more likely to have zero vital sign assessments, which was expected since the most critical children were more likely to die immediately following admission to the ward, prior to any vital sign collection. These children may have received vital signs on initial triage in the “under five” clinic prior to admission, though we chose not to include those vital signs in our analysis since they are often incomplete, poorly documented, and would not add value in evaluating our inpatient triage tool. Missing vital sign data was also associated with time of day the outcome occurred, with afternoon or night having higher odds of being missing compared with morning (data not shown). The association between ‘time of day’ and missing vital sign data was likely due to fewer on-duty staff at night. Influential observations, or single observations that have a significant impact on the regression function, were numerous in our study, which may be indicative of our relatively small sample size. It is possible that missing data and/or influential observations biased our effect estimates and standard errors. A prospective cohort design was not used in this study because while individuals who died most often did so within 48 hours of admission, individuals who survived most often had multiple vital sign sets, thus disproportionately affecting score performance. A case control design avoided this concern.

In conclusion, we have designed and implemented a simplified pediatric severity of illness score (ITAT) capable of recognizing children at high risk of death. Once identified as high risk, patients can be triaged for clinician re-evaluation, possible changes in management, or increased levels of supervision. Future studies should investigate the best way to operationalize the ITAT program in similar settings, including significant changes in the ITAT score as trigger for clinician review, patient outcome measures such as mortality, and the evaluation of task-shifting the ITAT score to health workers with less training.