Table 3. Calibration of Observed Versus Predicted Mortality in the Validation Dataset, by Score.

Severity Strata	Observed Mortality,No. (%)		Crude OR (95% CI)	Mean Predicted Mortality, %	C Statistic (Adjusted for Optimisma)	PPVb	NPVb
PERCH score stratac
-1 to 1	0/275	(0)	...	1.0	0.76 (0.76)	23.6%	95.8%
2	5/139	(3.6)	(ref)	1.9
3-4	13/560	(2.3)	0.64 (.22-1.82)	2.8
5-6	29/408	(7.1)	2.05 (.78-5.41)	6.8
7-17	88/373	(23.6)	8.28 (3.28-20.9)	23.1
Total	135/1755	...	...	...	...	...	...
WHO 2005 classification01
Severe	31/1145	(2.7)	(ref)	2.7	0.73 (0.73)	0%	92.3%
Very severe	104/610	(170)	7.39 (4.88-11.2)	17.0
Total	135/1755	...	...	...	...	...	...
South Africa RISC stratae
-2 to 1	9/567	(1.6)	(ref)	1.6	0.76 (0.76)	19.4%	95.6%
2	15/440	(3.4)	2.19 (.95-5.05)	3.3
3	13/227	(5.7)	3.77 (1.59-8.94)	6.9
4	24/141	(17)	12.7 (5.76-28.1)	15
5-8	73/376	(19.4)	14.9 (7.37-30.3)	19.6
Total	134/1751	...	...	...	...	...	...
WHO danger signs, No.f,g
0	14/916	(1.5)	(ref)	1.5	0.82 (0.82)	32.5%	96.1%
1	37/404	(9.2)	6.50 (3.47-12.2)	9.2
≥2	67/206	(32.5)	31.1 (17.0-56.8)	32.5
Total	118/1526	...	...	...	...	...	...

Abbreviations: CI, confidence interval; NPV, negative predictive value; OR, odds ratio; PERCH, Pneumonia Etiology Research for Child Health; PPV, positive predictive value; ref, reference category; RISC, Respiratory Index of Severity in Children; WHO, World Health Organization.

^aOptimism is where the C statistic overestimates the score’s predictive ability due to overfitting of the model to the data, eg, when using a small dataset.

^bAs the predicted probabilities of death assigned to cases using the score alone were all <0.5 (range, 0.02-0.24), a cutoff of >0.2 was used to define a predicted death to calculate PPV and NPV.

^cThe decision to split the score into quintiles was set out in the statistical analysis plan and the split by frequency was computed using Stata software; groups are not exactly the same size given that we could not split groups of children assigned the same integer score. A total of 79 children were assigned a PERCH score of ≥10; observed mortality in this group was 42%. Of the 5 children who died with a PERCH score of 2, 3 were female, 2 were classified as very severe pneumonia, 4 were 12-59 months old, all were normal-high weight for height; 1 was lethargic, and 2 had <92% oxygen saturation at admission.

^dSevere pneumonia is defined as cough/difficulty breathing and lower chest wall indrawing (LCWI); very severe pneumonia is cough/any difficulty breathing plus any one of the following danger signs: central cyanosis, inability to feed, vomiting everything, convulsions, lethargy, or severe respiratory distress (head nodding or grunting). Note that oxygen concentration was not used in the definition of very severe pneumonia.

^eCharacteristics included in the RISC score included oxygen saturation <90%, LCWI, low weight for age, refusal to feed, and wheeze. The validation dataset exhibited higher observed mortality than seen in the South African data from which the score was developed [1].

^fWHO danger signs were defined as central cyanosis or oxygen saturation <90% on pulse oximetry, inability to drink/feed, vomiting everything, convulsions, lethargy/unresponsiveness or impaired consciousness, and severe respiratory distress (head nodding). The severity of chest wall indrawing was not noted in the dataset and therefore “severe chest wall indrawing” could not be included as a danger sign; all of those cases categorized with no danger signs displayed some LCWI (as it was a requirement for study eligibility [severe or very severe pneumonia as per WHO 2005 guidelines]). The sample size used to assess the performance of the PERCH, RISC, and WHO scores was larger than that possible to use for the danger signs score due to missing data on some of the danger signs. When restricted to only the population with a score based on danger signs, the C statistics were as follows: PERCH score, 0.79; WHO 2005 score, 0.75; RISC score, 0.81.

^gOf the 14 children with no danger signs who died, 9 (65%) were female; 11 (79%) were aged 1-11 months; 10 (72%) were low/very low weight for height; 11 (79%) had illness duration for >3 days before presenting to hospital; none exhibited lethargy or unresponsiveness; only 3 were hypoxic (<92% oxygen); 9 (64%) had a cough. A total of 6 (43%) were in the highest PERCH risk stratum, and the remainder were equally distributed between the 3 other PERCH strata (not the lowest risk stratum with 0% mortality).