Development and validation of a novel clinical risk score to predict hypoxaemia in children with pneumonia using the WHO PREPARE dataset

Abstract

Background

Hypoxaemia predicts mortality at all levels of care, and appropriate management can reduce preventable deaths. However, pulse oximetry and oxygen therapy remain inaccessible in many primary care health facilities. We aimed to develop and validate a simple risk score comprising commonly evaluated clinical features to predict hypoxaemia in 2–59-month-old children with pneumonia.

Methods

Data from seven studies conducted in five countries from the Pneumonia Research Partnership to Assess WHO Recommendations (PREPARE) dataset were included. Readily available clinical features and demographic variables were used to develop a multivariable logistic regression model to predict hypoxemia (oxygen saturation <90%) at presentation to care. The adjusted log coefficients were transformed to derive the PREPARE hypoxemia risk score and its diagnostic value was assessed in a held-out, temporal validation dataset. The model and risk score were analysed by evaluating the area under the receiver operating characteristic curve (AUC), sensitivity and specificity.

Results

We included 14 509 children in the analysis; 9.8% (n=2515) were hypoxemic at presentation. The multivariable regression model to predict hypoxemia included age, sex, respiratory distress (nasal flaring, grunting and/or head nodding), lower chest indrawing, respiratory rate, body temperature and weight-for-age z-score. The model showed fair discrimination (AUC 0.70, 95% CI 0.67 to 0.73) and calibration in the validation dataset. The simplified PREPARE hypoxaemia risk score includes five variables: age, respiratory distress, lower chest indrawing, respiratory rate and weight-for-age z-score.

Conclusion

The PREPARE hypoxemia risk score, comprising five easily available characteristics, has the potential to be used to identify hypoxemia in children with pneumonia with a fair degree of certainty for use in health facilities without pulse oximetry. Its implementation would require careful consideration to limit the burden of inappropriate referrals on patients and the health system. Further external validation in community settings in low- and middle-income countries is required.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Pulse oximetry is unavailable or underutilised in many resource-limited settings in low-income and middle-income countries.

• Hypoxaemia is a good predictor of mortality and its early identification and further management can reduce mortality.

WHAT THIS STUDY ADDS

• The Pneumonia Research Partnership to Assess WHO Recommendations (PREPARE) hypoxemia risk score was developed using one of the largest and most geographically diverse datasets on childhood pneumonia to date.

• Using age, lower chest indrawing, respiratory rate, respiratory distress and weight-for-age z-score to calculate the PREPARE hypoxemia risk score could help identify children with hypoxemia in settings without pulse oximeters.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• This study contributes to the important discussion on how best to identify hypoxaemic children in the absence of pulse oximetry.

• Further research is warranted to validate the findings in community settings.

• Operationalising and integrating the score within existing clinical management pathways must be tailored to the setting of implementation.

Introduction

Hypoxemia—low blood oxygen levels, defined as an oxygen saturation (SpO₂) of <90%, is common in children with WHO classified pneumonia in low-income and middle-income countries (LMICs).¹ It is a good predictor of preventable deaths,²,⁴ and early identification of hypoxemia and appropriate management can reduce mortality.⁵,⁸ The availability of pulse oximetry to measure SpO₂ impacts clinicians’ assessment of illness severity, diagnosis, treatment and need for referral.^{6 9} As such, there is increasing rationale to support universal access to pulse oximetry in primary care settings to address the high burden of pneumonia-related mortality.¹⁰,¹³

Although the availability of pulse oximetry is increasing, in part due to increased focus during the COVID-19 pandemic,¹⁰ access is limited and inequitable, and pulse oximetry remains underutilised in many LMIC settings, particularly in the community and primary care health facilities.¹⁴,¹⁷ Barriers to the implementation of pulse oximetry in LMICs include high cost of quality paediatric devices with appropriately sized probes, maintenance, requisite training, battery charging and patient-related factors (eg, crying and movement), which can be particularly challenging among young children.¹⁸,²⁰ Furthermore, performing pulse oximetry can take time, especially in agitated children; an important consideration for implementation in health facilities with high case loads or insufficient staff-to-patient ratios.

In primary care-level health facilities without pulse oximetry, where the WHO’s Integrated Management of Childhood Illnesses (IMCI) chartbook²¹ is used to guide the management of sick children, many hypoxemic children are not identified.^{6 19} In health facilities where a pulse oximeter is not available, identifying children at the highest risk of being hypoxemic using clinical signs alone could improve the sensitivity of the IMCI chartbook for identification of children in need of oxygen therapy, hospitalisation or referral. Previous clinical scores to predict hypoxemia were limited in sample size, geographical representation or included patients already identified as requiring referral based on other clinical features.²²,²⁶ While we acknowledge that hypoxaemia occurs in children outside of the 2–59 months age group, and in patients without pneumonia, this analysis is restricted to this study population as a starting point for future work in other groups.

We aimed to develop and validate a clinical risk score to predict hypoxaemia (SpO₂<90%) at presentation to a health facility in children aged 2–59 months with pneumonia according to the 2014 WHO IMCI definition (ie, cough and/or difficulty breathing with fast breathing and/or lower chest indrawing) using the Pneumonia REsearch Partnership to Assess WHO REcommendations (PREPARE) dataset.^{21 27} The goal is not to replace the pulse oximeter, but to guide referral to health facilities where children with suspected hypoxaemia can be appropriately managed.

Methods

Study design

We used the WHO PREPARE dataset, to derive and validate a clinical risk score to identify hypoxemia in children 2–59 months with pneumonia. The PREPARE dataset included primary, patient-level data for children 0–59 months with pneumonia, from 45 separate studies conducted from 1994 to 2018, in over 20 countries in Asia, Africa, Latin America, Oceania and North America.²⁷ Detailed description of this dataset is found elsewhere.²⁷ We adhered to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis guidelines.²⁸

Patient and public involvement

The development of the research question was informed by the high rates of pneumonia-related mortality, specifically in children with hypoxemia. Patients were neither advisers in this study nor were they involved in the design, recruitment or conduct of the study. Results of this study will be made publicly available through open-access publication.

Study population

For the present analysis, we included studies that reported SpO₂ measured at the same time as assessment of clinical symptoms and signs (within 2 hours) and for which pulse oximetry was performed in ≥80% of cases. Studies in which fewer than 75% of patients had data for each candidate predictor were excluded. These inclusion criteria were specified in order to reduce bias by excluding studies that measured outcomes and predictors less systematically. Patients aged 2–59 months with IMCI defined pneumonia (irrespective of SpO₂ reading) presenting to health facilities were included. Patients under 2 months and above 5 years were excluded due to differences in the definition of pneumonia and differences in the clinical presentation of hypoxemia. Patients with severe pneumonia (defined as cough and/or difficulty breathing with a general danger sign including stridor in a calm child) or complicated severe acute malnutrition were excluded, since they would already have met IMCI criteria for referral.^{21 29}

Outcome

The primary outcome was hypoxemia, defined within the context of this study as a pulse oximetry measurement of SpO₂ <90%,^{21 27} assessed at the same time as the clinical predictors (symptoms and signs). A sensitivity analysis assessed the development of a model for the prediction of blood oxygen saturation at a SpO₂ value of <92%. This threshold was selected acknowledging the high risk of mortality in children at this threshold¹³ and the use of this threshold by various clinical guidelines.^{30 31}

Candidate predictors

Candidate predictors were selected a priori based on expert knowledge and previous diagnostic value for predicting hypoxemia, identified in the literature,²⁶ and considering reliability, feasibility and resources available at typical primary care health facilities in LMICs. The seven candidate predictors were sex, age, chest indrawing, respiratory rate, axillary temperature, weight-for-age z-score and respiratory distress (composite predictor defined as the presence of either nasal flaring, grunting and/or head nodding). A sensitivity analysis assessed the performance of the model without respiratory distress, with and without wheezing. Wheezing was included in a sensitivity analysis because it is difficult to identify without the use of a stethoscope, often absent or infrequently used in primary care health facilities in LMICs, however included within IMCI.^{21 32} In the analyses without respiratory distress as a candidate predictor, we excluded all patients with signs of respiratory distress. Given the variety of aims and methods among studies in the PREPARE dataset, some clinical predictors may have been measured with knowledge of the outcome status (peripheral oxygen saturation). Duration of cough or difficulty breathing, history of fever, duration of fever, mid-upper arm circumference, weight-for-height z-score, tracheal tugging, heart rate, cyanosis and capillary refill time were considered as candidate predictors based on potential clinical relevance but were not included due to high levels of missingness in the PREPARE dataset.

Sample size

We used 75% of the dataset for the development of the score and 25% for validation. Each study dataset was split by reserving the last 25% of enrolled patients for the validation dataset. Such a temporal validation strategy is considered intermediate between internal and external validation.²⁸ Using the approach outlined by Riley et al,³³ 659 cases of hypoxemia would be required in order to have a sufficient sample size to include the 12 prespecified parameters (from the seven candidate predictors) to develop the score (54.9 events per parameter), considering a conservative R² Nagelkerke of 0.15, shrinkage factor of 0.9 and outcome prevalence of 9.8% in the development dataset. The development dataset of 10 884 children contained 1146 cases of hypoxemia (ie, 95.5 outcome events per parameter). The temporal validation dataset included 275 outcome events, well above the recommended 100 events often recommended for a robust validation.^{34 35}

Analysis

Pulse oximetry measurements were not adjusted for altitude because no sites were greater than 2500 m above sea level.³⁶ Continuous predictors were categorised a priori based on currently accepted thresholds in order to maximise interpretability, encourage uptake among healthcare providers and policy-makers and align with current clinical practices: age: 2–5 months, 6–11 months, 12–59 months; temperature: <35.5, 35.5–37.4, ≥37.5°C; respiratory rate: 0–9, 10–19, ≥20 bpm above IMCI age-specific cut-offs (age 2–11 months ≥50 bpm; age 12–59 months ≥40 bpm) and weight-for-age z-score: ≥−2, <−2 to −3, <−3. Patients with missing data were excluded as multiple imputation was not feasible as it was determined that data were unlikely to be missing at random.³⁷

The model was developed using multivariable logistic regression with the least absolute shrinkage and selection operator (LASSO) with L1 regularisation. The adjusted model included all parameters and was further adjusted for study. Associations with 95% CIs for adjusted OR (aOR) that did not cross 1 were considered significant. Discrimination was evaluated by calculating the area under the receiver operating characteristic (AUROC) curve of the model in both the development and validation datasets. Calibration was evaluated using calibration plots (observed risk on Y-axis, predicted risk on the X-axis) in the validation dataset.

In order to convert the model into a simple clinical risk score, the regression coefficient of each retained parameter with a statistically significant aOR (CI did not overlap with 1.0) was rounded to the nearest 0.5 and then doubled.^{38 39} Sensitivity, specificity, likelihood ratios, miss rate (proportion of patients with hypoxemia not identified), false discovery rate (proportion of patients inappropriately identified as having hypoxemia among all patients identified as having hypoxemia) and proportion of children identified as being hypoxemic at each of the score’s cut-offs were evaluated in the temporal validation dataset.

Recognising that the appropriate threshold for referring a child with suspected hypoxaemia would be context dependent, a decision curve analysis compared the clinical utility (net benefit) of the model and score across a range of clinically-plausible referral thresholds. The net benefit quantifies the trade-off between true positives (hypoxemic children predicted to have hypoxemia) and false positives (non-hypoxemic children incorrectly predicted to have hypoxemia), weighted according to the relative cost of a false positive (threshold probability), which would vary in different contexts. The threshold probability reflects the cut-off (predicted probability of hypoxemia) above which a given intervention (in this case, referral) might be considered.

All analyses were performed using STATA V.16.⁴⁰

Results

Baseline characteristics

The PREPARE dataset included 294 968 children from 45 studies, of which 14 509 children from seven studies met inclusion criteria and were included in this analysis (figure 1). Four studies were randomised controlled trials, and three were prospective cohort studies (online supplemental table S1). All studies included outpatients presenting to hospitals, and one study also included patients presenting to health centres.⁴¹ Among the seven studies included in this analysis, patients were recruited from five countries: Australia, Gambia, India, Malawi and Nepal (online supplemental table S1). Of the 14 509 patients with pneumonia, 9.8% had an SpO₂ less than 90% at presentation (online supplemental table S2).

Figure 1. Flow diagram of PREPARE study dataset describing which datasets were excluded/included, the split between development and validation datasets and proportion of patients meeting the primary outcome (SpO₂<90%). Danger signs were defined by the IMCI⁸ chartbook (ie, inability to drink, lethargy or unconsciousness, convulsions, vomiting everything or stridor in a calm child). IMCI, Integrated Management of Childhood Illnesses; PREPARE, Pneumonia Research Partnership to Assess WHO Recommendations.

Baseline characteristics between the development (n=10 884, 75%) and validation dataset (n=3625, 25%) were similar (online supplemental table S2). Within the total dataset (n=14 509), 54.9% of children were aged less than 1 year, there were more male patients than female patients, 62.0% of children had lower chest indrawing, 34.1% had a temperature of equal to or more than 37.5°C and 6.7% had a very low weight-for-age z-score (<−3).

Model development

The predictors most strongly associated with hypoxemia (SpO₂<90%) were respiratory rate ≥20 bpm above the IMCI age-adjusted cut-off (aOR 2.41, 95% CI 1.25 to 4.64), any sign of respiratory distress (aOR 2.21, 95% CI 1.90 to 2.58), weight for age z-score <−3 (aOR 1.96, 95% CI 1.56 to 2.42) and lower chest indrawing (aOR 1.62, 95% CI 1.35 to 1.95) (table 1). None of the 12 candidate parameters were eliminated by the LASSO penalty during model development and; hence, all seven candidate predictors were retained in the final model.

Table 1. Multivariable regression model for predicting hypoxemia in the development dataset (n=10 884).

Characteristic	No hypoxemia (n=9738)	Hypoxemia (n=1146)	OR (95% CI)	Adjusted OR (95% CI)
Age
2–5 months	3000 (30.8%)	400 (34.9%)	1.34 (1.16 to 1.55)	1.29 (1.20 to 1.66)
6–11 months	2506 (25.7%)	324 (28.3%)	1.30 (1.11 to 1.52)	1.41 (1.11 to 1.50)
12–59 months	4232 (43.5%)	422 (36.8%)	(reference)	(reference)
Sex
Male	5455 (56.0%)	670 (58.5%)	(reference)	(reference)
Female	4283 (44.0%)	476 (41.5%)	0.90 (0.80 to 1.02)	0.91 (0.80 to 1.03)
Pneumonia classification
Fast breathing only	3836 (39.4%)	216 (18.8%)	(reference)	(reference)
Lower chest indrawing	5902 (60.6%)	930 (81.2%)	2.80 (2.40 to 3.26)	1.62 (1.35 to 1.95)
Any sign of respiratory distress (nasal flaring, grunting or head nodding)
No	5559 (57.1%)	371 (32.4%)	(reference)	(reference)
Yes	4179 (42.9%)	775 (67.6%)	2.78 (2.44 to 3.16)	2.21 (1.90 to 2.58)
Respiratory rate
< Age-adjusted tachypnoea threshold*	114 (1.2%)	10 (0.9%)	(reference)	(reference)
0–9 breaths/min above cut-off	4317 (44.3%)	284 (24.8%)	0.75 (0.39 to 1.45)	0.86 (0.44 to 1.66)
10–19 breaths/min above cut-off	3556 (36.5%)	437 (38.1%)	1.40 (0.73 to 2.69)	1.36 (0.71 to 2.62)
≥20 breaths/min above cut-off	1751 (18.0%)	415 (36.2%)	2.70 (1.40 to 5.20)	2.41 (1.25 to 4.64)
Body temperature
Normal (≥35.5°C–37.5°C)	6226 (63.9%)	682 (59.5%)	(reference)	(reference)
≥37.5°C	3401 (34.9%)	453 (39.5%)	1.21 (1.07 to 1.38)	1.04 (0.91 to 1.19)
<35.5°C	111 (1.1%)	11 (1.0%)	0.90 (0.48 to 1.69)	0.92 (0.48 to 1.75)
Weight for age z-score
≥−2	7939 (81.5%)	838 (73.1%)	(reference)	(reference)
<−2 to −3	1181 (12.1%)	183 (16.0%)	1.47 (1.24 to 1.74)	1.38 (1.15 to 1.64)
<−3	618 (6.4%)	125 (10.9%)	1.92 (1.56 to 2.35)	1.95 (1.56 to 2.42)

^*IMCI chartbook threshold for tachypnoea: age 2–11 months ≥50 breaths per minute; age 12–59 months ≥40 breaths per minute.

IMCI, Integrated Management of Childhood Illnesses.

Model performance

The PREPARE Hypoxaemia clinical prediction model had an AUROC of 0.70 (95% CI 0.69 to 0.72) in the development dataset and an AUROC of 0.70 (95% CI 0.68 to 0.73) in the temporal validation dataset (figure 2). Calibration of the PREPARE hypoxemia prediction model in the validation dataset is illustrated in online supplemental figure S1.

Figure 2. Receiver operating characteristic curve for the PREPARE hypoxemia clinical prediction model for children 2–59 months of age with pneumonia: A. Development dataset (n=10,884), B Validation dataset (n=3,625). AUROC, area under the receiver operating characteristic curve; PREPARE, Pneumonia Research Partnership to Assess WHO Recommendations.

When converted into the PREPARE hypoxemia risk score (table 2), the retained predictors included age 2–11 months (+1 point), lower chest indrawing (+1 points), respiratory rate ≥20 bpm above age-adjusted tachypnoea threshold (+2 points), any sign of respiratory distress (+2 points) and weight for age z-score <−2 (+1 point), with the score ranging from 0 to 7 points (table 3). Sensitivity, specificity, positive and negative likelihood ratios, miss rate, false discovery rate and the proportion of children identified as hypoxemic are included for each score cut-off (table 3).

Table 2. Components of the PREPARE hypoxemia risk score in the development dataset (n=10 884).

Factor	Adjusted log coefficient	PREPARE hypoxemia risk score*
Age
2–5 months	0.26	+1
6–11 months	0.34	+1
12–59 months	--	---
Pneumonia classification
Fast breathing only	--	---
Lower chest indrawing	0.49	+1
Any sign of respiratory distress (nasal flaring, grunting or head nodding)
No	--	--
Yes	0.79	+2
Respiratory rate
< age-adjusted tachypnoea threshold†	--	---
0–9 breaths/min above cut-off	−0.15	---
10–19 breaths/min above cut-off	0.31	---
≥20 breaths/min above cut-off	0.88	+2
Weight for age z-score
≥−2	--	---
<−2 to −3	0.32	+1
<−3	0.66	+1

^*Regression coefficient of each retained parameter with a statistically significant adjusted OR (CI did not overlap with 1.0) was rounded to the nearest 0.5 and then doubled to derive the points contributing to the PREPARE hypoxemia risk score.

^†IMCI chartbook threshold for tachypnoea: age 2–11 months≥50 breaths per minute; age 12–59 months≥40 breaths per minute.

IMCI, Integrated Management of Childhood Illnesses; PREPARE, Pneumonia Research Partnership to Assess WHO Recommendations.

Table 3. Performance of the PREPARE hypoxemia risk score at each cut-off in the validation dataset (n=3625).

Score	Hypoxemia, n/N (%)	Sensitivity (95% CI)*	Specificity (95 CI)*	Positive likelihood ratio (95% CI)*	Negative likelihood ratio (95% CI)*	Miss rate† % (n/N)	False discovery rate‡ % (n/N)	Proportion referred§ % (n/N)
0	6/450 (1.3%)	100.0 (98.7 to 100.0)	0.0 (0.0 to 0.1)	1.00	---	0% (0/275)	92.4% (3350/3625)	100% (3625/3625)
1	21/781 (2.7%)	97.8 (95.3 to 99.2)	13.3 (12.1 to 14.5)	1.13 (1.10 to 1.15)	0.16 (0.07 to 0.36)	2.2% (6/275)	91.5% (2906/3175)	87.6% (3175/3625)
2	44/545 (8.1%)	90.2 (86.0 to 93.4)	35.9 (34.3 to 37.6)	1.41 (1.34 to 1.47)	0.27 (0.19 to 0.39)	9.8% (27/275)	89.6% (2146/2394)	66.0% (2394/3625)
3	56/737 (7.6%)	74.2 (68.6 to 79.3)	50.9 (49.2 to 52.6)	1.51 (1.40 to 1.63)	0.51 (0.41 to 0.62)	25.8% (71/275)	89.0% (1645/1849)	51.0% (1849/3625)
4	67/689 (9.7%)	53.8 (47.7 to 59.8)	71.2 (69.7 to 72.8)	1.87 (1.66 to 2.11)	0.65 (0.57 to 0.74)	46.2% (127/275)	86.7% (964/1112)	30.7% (1112/3625)
5	45/284 (15.8%)	29.5 (24.1 to 35.2)	89.8 (88.7 to 90.8)	2.89 (2.34 to 3.55)	0.79 (0.73 to 0.85)	70.5% (194/275)	80.9% (342/423)	11.7% (423/3625)
6	28/119 (23.5%)	13.1 (9.3 to 17.7)	96.9 (96.3 to 97.5)	4.26 (2.96 to 6.10)	0.90 (0.86 to 0.94)	86.9% (239/275)	74.1% (103/139)	3.8% (139/3625)
7	8/20 (40.0%)	2.9 (1.3 to 5.7)	99.6 (99.4 to 99.8)	8.12 (3.35 to 19.70)	0.97 (0.95 to 0.99)	97.1% (267/275)	60.0% (12/20)	0.6% (20/3625)

^*Calculated at>each respective cut-off.

^†Miss rate (also known as false negative rate=false negative/(true positive+false negative).

^‡False discovery rate=false positive/(false positive+true positive).

^§Referral used as a proxy for identification of hypoxemia, assuming that all children identified as at risk of hypoxemia would be referred.

PREPARE, Pneumonia Research Partnership to Assess WHO Recommendations.

In the absence of pulse oximetry, the decision curve analysis suggests that in contexts where it might be acceptable to identify (and refer) up to six non-hypoxemic children for each hypoxemic child (up to a threshold probability of ~17%), it would be optimal to use a cut-off of ≥5 to guide referral decision for further management, whereas in contexts in which greater than 1-in-7 true positives are required, a cut-off of ≥6 would be preferable (online supplemental figure S2). The PREPARE hypoxemia model had an equivalent or higher net benefit than the PREPARE hypoxemia risk score (at any cut-off) across all threshold probabilities.

Prediction of hypoxaemia at SpO₂ threshold of < 92%

In our sensitivity analysis using the same dataset, 15.7% (2281/14 509) children had SpO2 <92%. The proportion with SpO₂<92% was slightly lower in the validation dataset (14.2%, 514/3625) compared with the development dataset (16.2%, 1767/10 884). The predictive value of predictors in predicting SpO₂<92% compared with SpO₂<90% was similar (online supplemental table S3, table 1).

The PREPARE SpO₂<92% clinical prediction model had a similar AUROC (0.67, 95% CI 0.65 to 0.70) compared with the PREPARE hypoxemia (SpO₂<90%) prediction model (AUROC 0.70, 95% CI 0.68 to 0.73) (online supplemental figure S3). The calibration plot for this model is shown in online supplemental figure S4.

The PREPARE SpO₂<92% risk score integrates 5 clinical predictors for a total score of 6 points: Age 6–11 months (+1 point), lower chest indrawing (+1 point), respiratory rate ≥20 bpm above age-adjusted tachypnoea threshold (+2 points), any sign of respiratory distress (+1 point) and weight-for-age z-score <−3 (+1 point) (online supplemental table S4). The performance of the risk score at each cut-off is presented in online supplemental table S5.

PREPARE hypoxaemia risk score without respiratory distress

When excluding respiratory distress as a candidate predictor, the dataset was larger with 15 592 children from 14 studies. Five studies were prospective cohorts, seven were randomised controlled trials, one was a retrospective cohort and one was a prospective case series (online supplemental table S6). Among the 14 studies included in this analysis, patients were recruited from 23 countries from North, Central and South America, Africa, Asia and Oceania (online supplemental table S6). Of the 15 592 patients with pneumonia, 16.1% had a SpO₂ <90% at presentation (online supplemental table S7). Baseline characteristics were similar in the development and validation datasets (online supplemental table S7) and did not differ from the primary dataset including respiratory distress. Without respiratory distress, lower chest indrawing (aOR 4.44, 95% CI 3.83 to 5.15) contributed more towards the prediction of hypoxemia (online supplemental table S8). None of the 11 candidate parameters was eliminated by the LASSO penalty during model development.

The PREPARE hypoxaemia clinical prediction model excluding respiratory distress had a higher AUROC than the model that included respiratory distress as a candidate predictor (AUROC 0.78 vs 0.70) (online supplemental figure S5). The calibration plot of this model is shown in online supplemental figure S6.

The final PREPARE hypoxemia risk score when excluding respiratory distress integrated four clinical predictors: age 2–11 months (+1 point), lower chest indrawing (+3 points), respiratory rate >=20 bpm above age-adjusted tachypnoea cut-off (+1 point) and weight for age z-score <−2 (+1 point) for a maximum total score of 6 points (online supplemental table S9). The performance of the risk score at each cut-off is presented in online supplemental table S10.

PREPARE hypoxemia risk score with wheezing

Using the same methodology as the primary analysis (ie, with respiratory distress), including wheezing as a candidate predictor, resulted in a slightly smaller dataset of 13 792 children from seven studies (online supplemental table S11). In this dataset, 9.9% had a SpO₂ of less than 90% at presentation (online supplemental table S12). Baseline characteristics in these development and validation datasets were similar (online supplemental table S12) and did not differ from the primary dataset, which excluded wheezing. Wheezing had an adjusted OR of 1.28 (95% CI 1.11 to 1.47), sensitivity of 37.6% and specificity of 73.2% (online supplemental table S13) for identification of hypoxemia. None of the 12 candidate parameters was eliminated by the LASSO penalty during model development.

The PREPARE hypoxaemia clinical prediction model that included wheezing performed similarly to the model without wheezing in the validation datasets (AUROC 0.72 vs 0.70) (online supplemental figure S7). The calibration plot of the PREPARE hypoxemia clinical prediction model, including wheezing, is shown in online supplemental figure S8.

The final PREPARE hypoxemia risk score when including wheezing integrated five clinical parameters: age 2–11 months (+1 points), chest indrawing (+1 points), respiratory distress (+2 points), respiratory rate ≥20 bpm above cut-off (+2 points) and weight for age z-score <−2 (+1 point) for a maximum total score of 7 points (online supplemental table S14). Wheezing was not retained in the risk score as the adjusted log coefficient was 0.24. A comparison of all risk score weights from the main analysis and sensitivity analyses is found in online supplemental table S15.

Discussion

We developed the PREPARE hypoxemia risk score from one of the largest and most diverse pneumonia datasets, including over 14 500 childhood pneumonia cases (of which 1421 had hypoxemia) from 7 studies in 5 countries and in 14 studies in 23 countries within a sensitivity analysis. The components of the score include age, respiratory distress, respiratory rate, lower chest-indrawing and weight-for-age z-score. The risk score demonstrated fair discrimination in predicting hypoxemia (SpO₂<90%) in children with pneumonia without danger signs or other indications for referral.

Previous studies predicting hypoxaemia included fewer patients from more restricted geographical settings.⁸²²,^{26 42} The use of very large and diverse datasets reduces instability of individual predictions, limits overfitting and provides the opportunity for generalisability to a wide range of settings.^{46 47} A model developed from a dataset with a similar number of patients, though exclusively evaluated in outpatient clinics from Bangladesh and Malawi, had a higher AUROC (0.79) compared with the PREPARE hypoxemia risk score (0.70) but had a similar AUROC to the PREPARE hypoxemia risk score excluding respiratory distress (0.78).²² However, this study and many others included patients with IMCI danger signs in whom hospitalisation or referral would already be recommended by existing guidelines.^{21 22 29} IMCI danger signs were retained in these prior analyses to evaluate their performance in hypoxemia prediction models alongside other candidate parameters, and these studies showed IMCI danger signs added little to no predictive value for hypoxemia.²¹ While excluding such patients in our study limits, our ability to comment on the value of IMCI danger signs in hypoxemia prediction models it aligns more closely with the needs of healthcare providers currently using the IMCI framework, helping them to decide which patients with pneumonia not already identified for hospitalisation or referral may benefit from higher level care. As found in other studies,²¹ this approach also underscores the high number of patients with pneumonia with hypoxaemia (1421/14 509; 9.8%) who would not have been identified as requiring referral or hospitalisation using the current IMCI algorithm in the absence of a pulse oximeter. In line with previous studies, greater tachypnoea,^{23 24 43 45} chest indrawing,^{22 23 45} younger age,^{8 22} wheezing,²² respiratory distress^{8 22 24 43 45} and low and very-low weight-for-age z-score^{8 22} were identified as clinical signs predictive of hypoxemia in children with pneumonia. In the present study and previous studies, individual signs performed poorly in identifying hypoxaemia, emphasising the benefits of a multivariable clinical risk score.

The use of the PREPARE hypoxemia risk score and selection of cut-off to guide referral of children with suspected hypoxemia would need to be tailored to the setting of implementation. Selecting a cut-off with the highest balance between sensitivity and specificity may not be the most appropriate referral threshold in many clinical contexts.^{48 49} Care must be taken when deciding to use such a score in order to limit referral of children without hypoxemia or danger signs, which could potentially come at great cost to individual patients and/or the health system. Considering that all patients with hypoxemia identified using this score would have otherwise been misidentified as not requiring referral by IMCI without pulse oximetry, using cut-offs with lower sensitivity (and higher specificity) could still make a helpful contribution, limiting unnecessary referrals, while still identifying many children with hypoxemia who would have been missed by the IMCI algorithm in settings where pulse oximetry is not available. Nevertheless, even utilising cut-offs of 5 or 6, where between 4% and 12% of all children would be referred, the majority of referrals (74%–81%) would still be for children who were not hypoxaemic. It may be that depending on the setting of implementation, a cut-off with relatively low specificity should consider alternative approaches to referral, such as short admission or follow-up within the community, to observe clinical evolution as proposed by Graham et al.¹³ Growing accessibility to ‘smartphones’ might increase the feasibility of a digital tool, which would allow more precise predictions and enable referral thresholds to be better tailored to the specific context of implementation. More pragmatically, elements from the PREPARE hypoxemia risk score could be combined with data on their prognostic value to predict death,⁵⁰ to update the IMCI criteria for severe pneumonia.

The sensitivity analysis excluding respiratory distress had a higher AUROC value compared with the primary analysis including respiratory distress as a candidate predictor. This, however, should not be interpreted to mean that respiratory distress does not add value in identifying hypoxemia. In fact respiratory distress had similar predictive value to identifying hypoxemia compared with lower chest indrawing and very fast breathing for age (ie, a respiratory rate of 20 breaths or more above age cut-off values). Differences in AUROC values between our analyses and compared with other studies are influenced by differences in the underlying datasets.⁴⁶ Indeed, this sensitivity analysis included two times the number of studies and thus the higher discrimination may reflect better generalisability of a score developed from a more diverse range of countries and settings. The value of respiratory distress should thus still be explored to improve the detection of children with hypoxemia and/or severe pneumonia within IMCI consultations, all the while considering the training, mentorship and resources required to reliably assess these signs.

Limitations

There are several limitations to this study. First, we intended to derive a clinical risk score using data from primary level health facilities since this is where pulse oximetry is often inaccessible and thus the clinical risk score would be most useful. Unfortunately, due to missing data on the outcome of interest and important clinical predictors, almost all studies from primary level health facilities were excluded, leaving a cohort derived predominantly from patients presenting to hospital outpatient departments. While the exclusion of children with danger signs may reduce this spectrum bias, the high proportion of pneumonia patients with chest indrawing (62%) and hypoxaemia (9.8%) compared with what is expected at primary level health facilities demonstrates the more severe presentation of included patients. Nonetheless, a similar study that was limited to ambulatory clinics found that the sensitivity and specificity of individual variables for predicting hypoxaemia were similar to hospital-based studies.²² Second, it has been reported that pulse oximetry is less accurate in patients with darker skin,^{51 52} and this was not considered in the present clinical risk score. Third, only 9% of children within the target age range with pneumonia were included (14 509/157 433). The vast majority of children and studies were excluded due to missing data (predictors or outcome). While this limited the diversity of the dataset, this decision was taken in order to prioritise better quality data, limiting bias in order to retain studies that systematically assessed the predictors and outcome of interest. Further work is needed to confirm that the score is generalisable to the whole pneumonia population more widely—assuming that patients with pulse oximetry readings are likely to be more severe, the prevalence of hypoxaemia in the pneumonia population is likely lower,⁵³ and the score may, therefore, benefit from updating. Fourth, our risk score was developed only for children aged 2–59 months with pneumonia. Indeed, hypoxaemia occurs in children without pneumonia, and in children younger than 2 months, and older than 5 years. While the needs for a similar risk score would be helpful for other age groups, given the differences in the definition of pneumonia for infants less than 2 months, the lack of WHO definition of pneumonia in children 5 years and older and differences in clinical presentation of hypoxemia, a separate analysis is required.

Conclusion

The PREPARE hypoxemia risk score could improve recognition of children with pneumonia who are at high risk of hypoxemia in health facilities where pulse oximetry is not available. Such children would benefit from further management and may reduce morbidity and mortality, even when other well-recognised indications of referral are absent. These benefits must be weighed against the burden of increased referrals on patients and health systems. Further validation and model updating in community settings are both a research and policy priority.

YBN is staff member of the World Health Organization. The authors alone are responsible for the views expressed in this article and they do not necessarily represent the views, decisions or policies of the institutions with which they are affiliated.

Data availability statement

Data are available upon reasonable request. All data relevant to the study are included in the article or uploaded as supplementary information.