Abstract
Background
There are no validated clinical decision aids to identify neonates and young children at risk of hospital readmission or postdischarge mortality in sub-Saharan Africa, leaving the decision to discharge a child to a clinician’s impression. Our objective was to determine the precision of clinician impression to identify neonates and young children at risk for readmission and postdischarge mortality.
Methods
We conducted a survey study nested in a prospective observational cohort of neonates and children aged 1–59 months followed 60 days after hospital discharge from Muhimbili National Hospital in Dar es Salaam, Tanzania or John F. Kennedy Medical Center in Monrovia, Liberia. Clinicians who discharged each enrolled patient were surveyed to determine their perceived probability of the patient’s risk of 60-day hospital readmission or postdischarge mortality. We calculated the area under the precision-recall curve (AUPRC) to determine the precision of clinician impression for both outcomes.
Results
Of 4247 discharged patients, 3896 (91.7%) had available clinician surveys and 3847 (98.7%) had 60-day outcomes available: 187 (4.8%) were readmitted and 120 (3.1%) died within 60 days of hospital discharge. Clinician impression had poor precision in identifying neonates and young children at risk of hospital readmission (AUPRC: 0.06, 95% CI: 0.04 to 0.08) and postdischarge mortality (AUPRC: 0.05, 95% CI: 0.03 to 0.08). Patients for whom clinicians attributed inability to pay for future medical treatment as the reason for risk for unplanned hospital readmission had 4.76 times the odds hospital readmission (95% CI: 1.31 to 17.25, p=0.02).
Conclusions
Given the poor precision of clinician impression alone to identify neonates and young children at risk of hospital readmission and postdischarge mortality, validated clinical decision aids are needed to aid in the identification of young children at risk for these outcomes.
Keywords: mortality, health services research, neonatology
What is already known on this topic?
In parts of sub-Saharan Africa, hospital readmissions and postdischarge mortality rates are estimated to range from 1% to 18% within months of hospital discharge.
There are no validated clinical decision aids to accurately identify neonates, infants and young children at risk of hospital readmission or postdischarge mortality in sub-Saharan Africa.
What this study adds?
Clinician impression alone had poor precision in identifying neonates, infants and children at risk of hospital readmission within 60 days of hospital discharge.
Clinician impression also did not accurately identify neonates, infants and children at risk of 60-day postdischarge mortality.
How this study might affect research, practice or policy?
Clinician impression alone is not sufficient to accurately identify neonates, infants and children at risk of hospital readmission or postdischarge mortality.
Validated and objective clinical decision aids are urgently needed to better identify neonates, infants and children at risk of hospital readmission and postdischarge mortality.
Introduction
Mortality rates among children aged <5 years in sub-Saharan Africa are 74 per 1000 live births, which is the highest in the world and is 14 times higher than rates in Europe and North America.1 The time after an inpatient hospital admission is particularly vulnerable in the life of a child in sub-Saharan Africa. Recent studies suggest that readmissions occur in 8%–18% of young children and as much as 1%–20% of young children die within 6 months after hospital discharge.2–5 Childhood mortality rates in the period immediately after hospitalisation for an illness (ie, the postdischarge period) approximate and may even outpace rates of mortality during hospitalisation.5 6
Although clinical prediction rules for all-cause hospital readmissions among children in settings like the USA have been developed that include variables such as prior healthcare utilisation and markers of illness severity,7 to our knowledge, there are currently no clinical prediction rules for hospital readmissions among children in sub-Saharan Africa, where readmissions are common. Differences in healthcare access and disease prevalence in the USA and sub-Saharan Africa necessitate the creation of clinical prediction rules catered to settings in sub-Saharan Africa. Clinical prediction rules have been developed to identify both young children and those aged <15 years at risk of postdischarge mortality in some settings in sub-Saharan Africa.8 9 However, these clinical prediction rules lack external validation and thus are not widely used in clinical practice. Given the absence of validated risk assessment tools to identify young children at risk of readmission and postdischarge mortality in sub-Saharan Africa, the decision to safely discharge a young child from a hospital is often driven by clinical judgement.
Clinician impression relies on a clinician’s ability to recognise patterns that may be associated with severe disease or an adverse outcome.10 However, the accuracy of clinician impression to predict outcomes, such as severe disease from infections, among children has varied in previous studies conducted in high-income settings.11–13 In a survey of 39 providers in Kenya, clinicians underestimated the overall incidence of postdischarge mortality among children.14 However, that study did not assess clinician impression of postdischarge mortality for individual patients and, to our knowledge, that has not been studied previously.
Given the absence of validated prognostic tools for hospital readmission and postdischarge mortality among children in sub-Saharan Africa, our primary objective was to determine the precision of treating clinicians’ clinical impression to identify neonates and young children at risk for hospital readmission and postdischarge mortality in Dar es Salaam, Tanzania and Monrovia, Liberia. Our secondary objective was to evaluate factors associated with accuracy of treating clinicians’ clinical impression to identify neonates and young children at risk for hospital readmission and postdischarge mortality.
Methods
Study design
We conducted a survey nested in a prospective observational cohort study of paediatric patients discharged from Muhimbili National Hospital in Dar es Salaam, Tanzania and John F. Kennedy Medical Center in Monrovia, Liberia from October 2019 to January 2022. Details of our study protocol have been published previously.15 Besides differences in estimated and actual enrolment, there were no deviations from that protocol. Neonates and young children aged 1–59 months were enrolled at discharge from the neonatal or paediatric wards at each facility. Follow-up consisted of caregivers receiving phone calls up to 60 days after hospital discharge. Caregivers provided written consent for participation in Tanzania and oral consent in Liberia because of cultural preference and low rates of caregiver literacy.
Patient and public involvement
The development of the research question was informed by the disease burden of readmission and postdischarge mortality among children in sub-Saharan Africa. Patients were not involved in the design, recruitment or conduct of the study, nor were they advisers in this study. Results of this study will be made publicly available through publication.
Study setting
This study was conducted at two large, national referral hospitals supported by each country’s Ministry of Health. They are in urban areas in their respective countries. Muhimbili National Hospital has a catchment of approximately 6 million people and John F. Kennedy Medical Center has a catchment of approximately 1.5 million people. Both hospitals are training hospitals for paediatric residents who are completing their specialty training.
Study populations
Neonates and young children discharged from the wards were consecutively enrolled. Neonates and young children who died during initial hospitalisation were excluded. Neonates and young children whose caregivers did not have telephones for follow-up or those who declined enrolment were excluded.
Surveyed clinicians included consultants/specialists, interns/residents or medical officers. Consultants/specialists were certified paediatricians or paediatric specialists who completed medical school, residency and subspecialty training (for specialists). Interns/residents had completed medical school and were completing residency training in paediatrics. Medical officers received 3 years of clinical training prior to providing clinical care to patients.
Study procedures
After obtaining informed consent from caregivers, trained research coordinators at each site approached the clinician who discharged each enrolled patient, obtained consent and asked them to complete a survey near the time of the patient’s hospital discharge. This survey modelled previous surveys that assessed clinician impression12 and was developed through an iterative process by the research team with multiple opportunities to each investigator to refine the content. The survey was also reviewed by an expert in survey design (ie, a survey methodologist) to ensure question clarity and appropriate response types (online supplemental appendix survey). This survey was designed to allow clinician respondents to describe their perceived probability that each patient would experience both outcomes. Responses were recorded on standardised, electronic case report forms in electronic tablets using SQL (Tanzania) and KoboToolbox (Liberia).
bmjpo-2023-001972supp001.pdf (79.6KB, pdf)
Measurement and outcomes
The exposure variable was the impression of the discharging clinician of the patient’s risk of: (1) unplanned hospital readmission within 60 days of hospital discharge or (2) all-cause, 60-day postdischarge mortality. Aligned with previous studies,12 probabilities of perceived risk of readmission or postdischarge mortality included categorical options of 0%, 1%–5%, 6%–20%, 21%–40%, 41%–60%, 61%–80%, 81%–99% and 100%. This survey also assessed discharging clinicians’ perceptions of why readmission or postdischarge mortality were possible for those who were identified as at-risk for each outcome. Surveyed clinicians were familiar with the patients’ clinical history and laboratory results during hospital admission. To assess for outcomes, phone calls to patients’ caregivers were made by research staff at 7, 14, 30, 45 and 60 days after hospital discharge. Outcomes were determined as reported by caregivers to research staff.
Statistical analyses
The association of the discharging clinicians’ predicted probability of readmission or postdischarge mortality and proportion of patients at each clinician-estimated risk threshold (eg, 0%, 1%–5%, etc) who were readmitted or died was compared using χ2 or Fisher’s exact testing (p<0.05 for significance).12 We calculated sensitivity, specificity, positive predictive value and negative predictive value of treating clinician’s impression at each per cent risk threshold using caregiver-reported readmission or postdischarge mortality as the reference standards. We determined the precision of clinician impression for identifying patients at risk of readmission or postdischarge mortality by calculating the area under the precision-recall curve (AUPRC) which is useful for evaluating binary classifiers in imbalanced datasets.16 95% CI for the AUPRC were calculated through fivefold cross-validation. We compared AUPRC and corresponding 95% CIs to baseline chance of the outcome occurring in that group. A 95% CI higher than baseline chance indicated better precision than random chance.
We conducted subanalyses by the discharged patient’s age group (ie, neonate or young child), clinician experience level (ie, consultant/specialist, intern/resident or medical officer), site and time to outcome. We conducted binary logistic regression analyses to assess whether the perceived reason for risk for each outcome was associated with the patient’s likelihood of each outcome. Additionally, we conducted binary logistic regression analyses to assess whether the clinician probability for each outcome was associated with the patient’s likelihood of each outcome after adjusting for clinician type, patient age at discharge (months), whether the discharge diagnosis was infectious or not and duration of hospitalisation (days). Due to small sample sizes in the non-0% clinician probability categories, we reduced the categorisation of perceived risk to 0%, 1%–20%, 21%–60% and 61%–99%. The clinician cited reason for the outcome was also considered in the model but was removed due to collinearity as assessed by the variance inflation factor. All tests were two-sided tests and used a 0.05 significance level. AUPRC analyses were conducted through the precrec package in R.17 All analyses were performed in R V.4.1.3 (R Foundation for Statistical Computing, Vienna, Austria) and SAS V.9.4 (SAS Institute, Cary, North Carolina, USA).
Results
There were 4460 discharged patients, 4247 (95.2%) enrolled and 3896 (91.7%) had complete clinician surveys (figure 1). Enrolment was approximately equal between the two sites (Tanzania n=1997, 51.3%, Liberia n=1899, 48.7%) (table 1). There were 2173 (55.8%) neonates and 1723 (44.2%) young children who had clinician surveys available.
Figure 1.
Flow diagram for included neonates and children aged 1–59 months discharged from the neonatal wards and paediatric wards in Dar es Salaam, Tanzania and Monrovia, Liberia.
Table 1.
Characteristics of neonates and young children included in the evaluation of clinician impression on predicting 60-day hospital readmission or postdischarge mortality
| Patient characteristics | Overall population (n=3896) n (%) | Readmitted to hospital 60 days after hospital discharge (n=187) n (%) | Died 60 days after hospital discharge (n=120) n (%) |
| Discharged from neonatal ward* | 2173 (55.8) | 80 (3.6) | 61 (2.8) |
| Age in days at discharge, median (IQR) | 2 (1, 7) | 4 (2, 13) | 2 (0, 10) |
| Discharged from paediatrics ward | 1723 (44.2) | 107 (6.2) | 59 (3.4) |
| Age in months for paediatric patients aged 1–59 months, median (IQR)† | 12 (5, 24) | 8 (4, 18) | 8 (4, 21) |
| Sex‡ | |||
| Male | 2198 (56.4) | 109 (5.0) | 74 (3.4) |
| Female | 1691 (43.4) | 78 (4.6) | 46 (2.7) |
| Country | |||
| Tanzania | 1997 (51.3) | 140 (7.0) | 63 (3.2) |
| Liberia | 1899 (48.7) | 47 (2.5) | 57 (3.0) |
| Disposition from the hospital | |||
| Discharge | 3775 (96.8) | 186 (4.9) | 106 (2.8) |
| Left against medical advice | 119 (3.1) | 1 (0.8) | 14 (11.8) |
| Transfer to another facility | 2 (0.05) | 0 (0.0) | 0 (0.0) |
*Twenty-four neonates had missing age.
†Thirty-one young children had missing age.
‡Seven participants did not have a documented sex.
Sixty-day outcomes were available for 3847 (98.7%) enrolled patients. The median age of enrolled neonates was 2 days (IQR: 1–7) and 12 months (IQR: 5–24) for infants and children. The most common discharge diagnoses among neonates were sepsis (29.7%, n=609), prematurity (28.8%, n=591) and birth asphyxia (15.8%, n=323). Among infants and children, pneumonia (12.1%, n=223), diarrhoeal disease (10.1%, n=186), and malaria (7.2%, n=133) were most common.
There were 187 (4.8%) patients readmitted and 120 (3.1%) died within 60 days of discharge. There were 80 (3.6%) neonates who were readmitted and 61 (2.8%) died within 60 days of hospital discharge. Among infants and children, 107 (6.2%) were readmitted and 59 (3.4%) died after hospital discharge. The median time from hospital discharge to readmission was 30 days (IQR: 7–45). The median time from hospital discharge to mortality was 30 days (IQR 14–45).
Clinician impression and hospital readmission
Nearly three quarters of patients were perceived to have 0% risk of readmission within 60 days (table 2). Patients who were readmitted were more likely to have a perceived risk of readmission of 0% than those who were not readmitted (81% vs 72%, p<0.001; table 2). Among the 187 neonates and young children who were readmitted, 80.7% (n=151) were perceived to have 0% risk of readmission.
Table 2.
Association of discharging clinicians’ predicted probability and unplanned 60-day hospital readmission among neonates and young children
| Total, n (%) (n=3896) | Not readmitted 60 days after hospital discharge, n (%) (n=3709) | Readmitted 60 days after hospital discharge, n (%) (n=187) | P value* | |
| Clinician predicted probability | <0.001 | |||
| 0% | 2838 (72.8) | 2687 (72.4) | 151 (80.7) | |
| 1%–5% | 9 (0.2) | 9 (0.2) | 0 (0.0) | |
| 6%–20% | 26 (0.7) | 23 (0.6) | 3 (1.6) | |
| 21%–40% | 736 (18.9) | 720 (19.4) | 16 (8.6) | |
| 41%–60% | 38 (1.0) | 35 (0.9) | 3 (1.6) | |
| 61%–80% | 237 (6.1) | 227 (6.1) | 10 (5.3) | |
| 81%–99% | 11 (0.3) | 7 (0.2) | 4 (2.1) |
There was one respondent who estimated that the discharged child was at risk of hospital readmission but did not assign a proportion.
*By Fisher’s exact test to assess independence of clinician predicted probability and likelihood of hospital readmission.
Overall, clinician impression had poor precision in identifying neonates and young children at risk of readmission (AUPRC: 0.06, 95% CI: 0.04 to 0.08) (table 3). Among medical officers, clinician impression had greater precision in identifying children at risk of readmission (AUPRC: 0.23, 95% CI: 0.17 to 0.34); this group was marginally better at identifying patients at risk of readmission than interns/residents and consultants/specialists.
Table 3.
Test characteristics for clinician predicted probability of unplanned, 60-day hospital readmission among all enrolled neonates and young children aged 1–59 months overall and by clinician type
| Threshold % | Sensitivity (95% CI) | Specificity (95% CI) | Positive predictive value (95% CI) | Negative predictive value (95% CI) | Area under precision-recall curve (95% CI) |
| All clinicians (n=3895)* | 0.06 (0.04, 0.08), chance=0.05 | ||||
| ≤5 | 19.3 (13.9, 25.1) | 72.5 (71.0, 73.8) | 3.4 (2.5, 4.4) | 94.7 (94.3, 95.1) | |
| ≤20 | 19.3 (13.9, 25.1) | 72.7 (71.3, 74.1) | 3.4 (2.5, 4.5) | 94.7 (94.4, 95.1) | |
| ≤40 | 17.7 (12.3, 23.5) | 73.7 (72.3, 75.1) | 3.3 (2.4, 4.3) | 94.7 (94.3, 95.0) | |
| ≤60 | 16.0 (11.2, 21.9) | 74.3 (72.9, 75.7) | 3.0 (2.1, 4.1) | 94.6 (94.3, 94.9) | |
| ≤80 | 7.5 (4.2, 11.8) | 93.7 (92.9, 94.4) | 5.6 (3.1, 8.7) | 95.3 (95.1, 95.5) | |
| ≤99 | 2.1 (0.5, 4.3) | 99.8 (99.7, 99.9) | 36.4 (9.1, 66.7) | 95.3 (95.2, 95.4) | |
| Consultant/specialist (n=175) | 0.05 (0.02, 0.08), chance=0.03 | ||||
| ≤5 | 16.7 (0, 50.0) | 69.2 (62.1, 75.8) | 1.9 (0.0, 5.7) | 95.8 (94.7, 97.6) | |
| ≤20 | 16.7 (0, 50.0) | 69.8 (62.7, 76.3) | 1.9 (0, 5.9) | 95.9 (94.7, 97.6) | |
| ≤40 | 16.7 (0, 50.0) | 73.4 (66.3, 79.9) | 2.1 (0, 6.7) | 96.1 (95.0, 97.7) | |
| ≤60 | 0.0 (0.0, 0.0) | 75.2 (68.6, 81.1) | 0.0 (0.0, 0.0) | 95.5 (95.1, 95.8) | |
| ≤80 | 0.0 (0.0, 0.0) | 85.2 (79.9, 90.5) | 0.0 (0.0, 0.0) | 96.0 (95.7, 96.2) | |
| ≤99 | 0.0 (0.0, 0.0) | 100 (100, 100) | NA | 96.6 (96.6, 96.6) | |
| Intern/resident (n=3502) | 0.06 (0.03, 0.08), chance=0.05 | ||||
| ≤5 | 14.9 (9.9, 20.5) | 72.4 (70.8, 73.9) | 2.5 (1.7, 3.5) | 94.6 (94.3, 95.0) | |
| ≤20 | 14.9 (9.9, 20.5) | 72.5 (70.9, 74.0) | 2.5 (1.7, 3.5) | 94.6 (94.3, 95.0) | |
| ≤40 | 14.9 (9.9, 20.5) | 73.2 (71.6, 74.7) | 2.6 (1.7, 3.6) | 94.7 (94.4, 95.1) | |
| ≤60 | 13.6 (8.7, 19.3) | 73.5 (71.9, 75.0) | 2.4 (1.6, 3.4) | 94.6 (94.3, 94.9) | |
| ≤80 | 5.6 (2.5, 9.3) | 94.1 (93.3, 94.9) | 4.3 (1.9, 7.3) | 95.4 (95.2, 95.6) | |
| ≤99 | 1.2 (0.0, 3.1) | 99.9 (99.8, 100) | 40.0 (0.0, 100) | 95.5 (95.4, 95.5) | |
| Medical officer (n=217) | 0.23 (0.17, 0.34), chance=0.09 | ||||
| ≤5 | 55.0 (30.0, 75.0) | 76.1 (70.1, 81.7) | 18.9 (11.6, 26.5) | 94.3 (91.6, 96.9) | |
| ≤20 | 55.0 (30.0, 75.0) | 78.7 (72.6, 84.3) | 20.7 (12.7, 29.4) | 94.5 (91.9, 96.9) | |
| ≤40 | 40.0 (20.0, 60.0) | 82.2 (76.6, 87.3) | 18.4 (9.4, 28.9) | 93.1 (90.8, 95.5) | |
| ≤60 | 40.0 (20.0, 60.0) | 86.8 (81.7, 91.4) | 23.5 (12.1, 36.4) | 93.4 (91.3, 95.7) | |
| ≤80 | 25.0 (9.8, 45.0) | 93.9 (90.4, 96.9) | 29.4 (10.5, 50.0) | 92.5 (90.9, 94.4) | |
| ≤99 | 10.0 (0.0, 25.0) | 97.9 (95.9, 99.5) | 33.3 (0.0, 80.0) | 91.5 (90.6, 92.8) |
*There was one respondent who estimated that the discharged child was at risk of hospital readmission but did not assign a proportion.
By clinician type, medical officer clinician impression had poor precision in identifying neonates (online supplemental table 1) but greater precision when identifying infants and children at risk for readmission (online supplemental table 2). In site-specific analyses, clinician impression was imprecise when identifying neonates and young children at risk of readmission in Tanzania (AUPRC: 0.11, 95% CI: 0.07 to 0.15, chance: 0.07) and Liberia (AUPRC: 0.03, 95% CI: 0.02 to 0.04, chance: 0.02). Regardless of the time from hospital discharge to readmission, clinician impression had poor precision in identifying neonates and young children at risk for readmission (online supplemental table 3).
bmjpo-2023-001972supp002.pdf (89.7KB, pdf)
Clinician impression and postdischarge mortality
Most (96.1%, n=3746) patients were assigned 0% risk of postdischarge mortality (table 4). Patients who died within 60 days of discharge were more likely to have a perceived risk of 0% than patients who survived (96% vs 90.8%, p=0.002; table 4). Among the 120 neonates and young children who died within 60 days of hospital discharge, 90.8% (n=109) were estimated to have a 0% probability of postdischarge mortality.
Table 4.
Association of discharging clinicians’ predicted probability and all-cause, 60-day postdischarge mortality among neonates and young children
| Total, n (%) (n=3896) | Did not die 60 days after hospital discharge, n (%) (n=3776) | Died 60 days after hospital discharge, n (%) (n=120) | P value* | |
| Clinician predicted probability | 0.002 | |||
| 0% | 3746 (96.1) | 3637 (96.3) | 109 (90.8) | |
| 1%–5% | 10 (0.3) | 10 (0.3) | 0 (0.0) | |
| 6%–20% | 9 (0.2) | 7 (0.2) | 2 (1.7) | |
| 21%–40% | 13 (0.3) | 11 (0.3) | 2 (1.7) | |
| 41%–60% | 95 (2.4) | 90 (2.4) | 5 (4.2) | |
| 61%–80% | 21 (0.5) | 20 (0.5) | 1 (0.8) | |
| 81%–99% | 2 (0.1) | 1 (0.03) | 1 (0.8) |
*By Fisher’s exact test to assess independence of clinician predicted probability and likelihood of hospital readmission.
Overall, clinician impression had poor precision in identifying neonates and young children at risk for postdischarge mortality (AUPRC: 0.05, 95% CI: 0.03 to 0.08) and did not vary substantially among interns/residents, specialist/consultants or medical officers (table 5). Clinician impression had poor precision in identifying postdischarge mortality among neonates (AUPRC: 0.04, 95% CI: 0.03 to 0.06, chance: 0.03) and infants and children (AUPRC: 0.06, 95 % CI: 0.03 to 0.10, chance: 0.03). When analysed by site, clinician impression in both Tanzania (AUPRC: 0.08, 95% CI: 0.03 to 0.13, chance: 0.03) and Liberia (AUPRC: 0.04, 95% CI: 0.03 to 0.06, chance: 0.04) had poor precision in identifying neonates and young children at risk of postdischarge mortality. Clinician impression had poor precision regardless of the time to postdischarge mortality (online supplemental table 4).
Table 5.
Test characteristics for clinician predicted probability of all-cause, 60-day postdischarge mortality among all enrolled neonates and young children aged 1–59 months by clinician type
| Threshold % | Sensitivity (95% CI) | Specificity (95% CI) | Positive predictive value (95% CI) | Negative predictive value (95% CI) | Area under precision-recall curve (95% CI) |
| All clinicians (n=3895)* | 0.05 (0.03, 0.08), chance=0.03 | ||||
| ≤5 | 9.2 (4.2, 15.0) | 96.3 (95.7, 96.9) | 7.3 (3.5, 11.9) | 97.1 (96.9, 97.3) | |
| ≤20 | 9.2 (4.2, 15.0) | 96.6 (95.9, 97.2) | 7.7 (3.8, 12.7) | 97.1 (96.9, 97.3) | |
| ≤40 | 7.5 (3.3, 12.5) | 96.7 (96.2, 97.3) | 6.8 (2.9, 11.6) | 97.1 (96.9, 97.2) | |
| ≤60 | 5.8 (1.7, 10.8) | 97.1 (96.5, 97.6) | 5.8 (1.9, 10.5) | 97.0 (96.9, 97.2) | |
| ≤80 | 1.7 (0.0, 4.2) | 99.4 (99.2, 99.7) | 8.3 (0.0, 22.7) | 96.9 (96.9, 97.0) | |
| ≤99 | 0.8 (0.0, 2.5) | 99.9 (99.9, 100) | 50.0 (0.0, 100) | 96.9 (96.9, 96.9) | |
| Specialist or consultant (n=175) | 0.05 (0.02, 0.07), chance=0.04 | ||||
| ≤5 | 0.0 (0.0, 0.0) | 97.6 (95.2, 99.4) | 0.0 (0.0, 0.0) | 95.9 (95.8, 95.9) | |
| ≤20 | 0.0 (0.0, 0.0) | 97.6 (95.2, 99.4) | 0.0 (0.0, 0.0) | 95.9 (95.8, 95.9) | |
| ≤40 | 0.0 (0.0, 0.0) | 97.6 (95.2, 99.4) | 0.0 (0.0, 0.0) | 95.9 (95.8, 95.9) | |
| ≤60 | 0.0 (0.0, 0.0) | 97.6 (95.2, 99.4) | 0.0 (0.0, 0.0) | 95.9 (95.8, 95.9) | |
| ≤80 | 0.0 (0.0, 0.0) | 98.8 (97.0, 100) | 0.0 (0.0, 0.0) | 95.9 (95.9, 96.0) | |
| ≤99 | 0.0 (0.0, 0.0) | 100 (100, 100) | – | 96.0 (96.0, 96.0) | |
| Intern/resident (n=3502) | 0.05 (0.02, 0.08), chance=0.03 | ||||
| ≤5 | 8.8 (3.9, 14.7) | 97.1 (96.5, 97.6) | 8.3 (3.7, 13.8) | 97.3 (97.1, 97.4) | |
| ≤20 | 8.8 (3.9, 14.7) | 97.2 (96.6, 97.7) | 8.5 (3.8, 14.2) | 97.3 (97.1, 97.4) | |
| ≤40 | 6.9 (2.9, 12.7) | 97.2 (96.6, 97.7) | 6.8 (2.7, 12.0) | 97.2 (97.1, 97.4) | |
| ≤60 | 5.8 (1.9, 10.8) | 97.2 (96.7, 97.8) | 5.9 (1.9, 11.0) | 97.2 (97.1, 97.3) | |
| ≤80 | 1.9 (0.0, 4.9) | 99.6 (99.4, 99.8) | 11.8 (0.0, 30.8) | 97.1 (97.1, 97.2) | |
| ≤99 | 0.9 (0.0, 2.9) | 99.9 (99.9, 100.0) | 50.0 (0.0, 100.8) | 97.1 (97.1, 97.2) | |
| Medical officer (n=218) | 0.05 (0.03, 0.08), chance=0.05 | ||||
| ≤5 | 18.2 (0.0, 45.5) | 82.6 (77.8, 87.9) | 5.0 (0, 12.2) | 94.9 (93.8, 96.5) | |
| ≤20 | 18.2 (0.0, 45.5) | 85.9 (81.2, 90.3) | 6.1 (0, 14.8) | 95.1 (94.1, 96.7) | |
| ≤40 | 18.2 (0.0, 45.5) | 88.9 (84.5, 92.8) | 7.7 (0, 18.7) | 95.3 (94.2, 96.8) | |
| ≤60 | 9.1 (0.0, 27.3) | 93.7 (90.3, 96.6) | 6.7 (0, 23.1) | 95.1 (94.5, 96.1) | |
| ≤80 | 0.0 (0.0, 0.0) | 97.6 (95.2, 99.5) | 0.0 (0.0, 0.0) | 94.8 (94.7, 94.9) | |
| ≤99 | 0.0 (0.0, 0.0) | 100 (100, 100) | – | 94.9 (94.9, 94.9) |
*There was one respondent who estimated that the discharged child was at risk of hospital readmission but did not assign a proportion.
Reason for perceived risk of hospital readmission and postdischarge mortality
Patients for whom clinicians attributed inability to pay for treatment as the reason for readmission had 4.76 times the odds readmission (95% CI: 1.31 to 17.25, p=0.02) compared with those with no perceived risk (table 6). Patients whose clinician cited ‘other’ reasons to be at risk had lower odds of readmission compared with those whose clinician did not believe they were at risk for readmission (OR: 0.24, 95% CI: 0.09 to 0.66, p=0.005). Patients for whom clinicians attributed inability to pay for treatment as the reason for potential postdischarge mortality had 5.53 times the odds of postdischarge mortality (95% CI 1.22 to 25.10, p=0.03).
Table 6.
Reasons for perceived risk of hospital readmission and postdischarge mortality
| Clinician cited reason for outcome | No hospital readmission, n (%) | Hospital readmission, n (%) | OR (95% CI) | P value |
| No risk | 2584 (69.7) | 148 (79.1) | Referent | – |
| Clinician perceived inability to pay for treatment | 11 (0.3) | 3 (1.6) | 4.76 (1.31, 17.25) | 0.02 |
| Clinician perceived social concerns | 70 (1.9) | 3 (1.6) | 0.75 (0.23, 2.41) | 0.63 |
| Clinician perceived progression of illness | 754 (20.3) | 29 (15.5) | 0.67 (0.45, 1.01) | 0.05 |
| Other* | 290 (7.8) | 4 (2.1) | 0.24 (0.09, 0.66) | 0.005 |
| Clinician cited reason for outcome | Did not die within 60 days, n (%) | Died within 60 days, n (%) | OR (95% CI) | P value |
| No risk | 2652 (70.2) | 80 (66.7) | Referent | – |
| Clinician perceived inability to pay for treatment | 12 (0.3) | 2 (1.7) | 5.53 (1.22, 25.10) | 0.03 |
| Clinician perceived social concerns | 72 (1.9) | 1 (0.8) | 0.46 (0.06, 3.36) | 0.44 |
| Clinician perceived progression of illness | 754 (20.0) | 29 (24.2) | 1.28 (0.83, 1.97) | 0.27 |
| Other* | 286 (7.6) | 8 (6.7) | 0.93 (0.44, 1.94) | 0.84 |
*Included concerns about caregiver understanding, general clinician impression and patient with history of recurrent illness.
In multivariable analyses, patients whom clinicians estimated to be at moderate risk for hospital readmission (ie, 21%–60%) were at decreased odds of hospital readmission (adjusted OR: 0.45, 95% CI: 0.26 to 0.74, p=0.003) (table 7). No other factors were independently associated with either hospital readmission or postdischarge mortality.
Table 7.
Multivariable regression model for all-cause 60-day hospital readmission and postdischarge mortality among young children discharged in Dar es Salaam, Tanzania and Monrovia, Liberia
| Characteristics | No hospital readmission, n (%), n=3709 | Hospital readmission, n (%), n=187 | Adjusted OR (95% CI), n=3426 | P value |
| Perceived risk | ||||
| 0% | 2687 (72%) | 151 (81%) | Referent | – |
| 1%–20% | 44 (1.2%) | 3 (1.6%) | 0.96 (0.23, 2.77) | 0.95 |
| 21%–60% | 743 (20%) | 19 (10%) | 0.45 (0.26, 0.74) | 0.003 |
| 61%–99% | 234 (6.3%) | 14 (7.5%) | 1.08 (0.57, 1.90) | 0.79 |
| Discharge provider type | ||||
| Intern/resident | 3341 (90%) | 161 (86%) | Referent | – |
| Specialist or consultant | 169 (4.6%) | 6 (3.2%) | 0.78 (0.30, 1.65) | 0.55 |
| Medical officer | 198 (5.3%) | 20 (11%) | 2.07 (1.21, 3.39) | 0.01 |
| Patient age at discharge, months | 1 (0, 10) | 3 (1, 11) | 1.00 (0.98, 1.01) | 0.45 |
| Discharge diagnosis | ||||
| Non-infectious | 1492 (40%) | 87 (47%) | Referent | – |
| Infectious | 2217 (60%) | 100 (53%) | 0.86 (0.62, 1.19) | 0.35 |
| Duration of hospitalisation, days | 7 (3, 13) | 8 (3, 16) | 1.002 (1.0001, 1.005) | 0.04 |
| Characteristics | Did not die within 60 days, n (%), n=3776 | Died within 60 days, n (%), n=120 | Adjusted OR (95% CI), n=3426 | P value |
| Perceived risk | ||||
| 0% | 3637 (96%) | 109 (91%) | Referent | – |
| 1%–20% | 17 (0.5%) | 2 (1.7%) | 2.95 (0.44, 11.8) | 0.18 |
| 21%–60% | 101 (2.7%) | 7 (5.8%) | 1.92 (0.73, 4.20) | 0.14 |
| 61%–99% | 21 (0.6%) | 2 (1.7%) | 2.94 (0.46, 10.4) | 0.15 |
| Discharge provider type | ||||
| Intern/resident | 3400 (90%) | 102 (85%) | Referent | – |
| Specialist or consultant | 168 (4.5%) | 7 (5.8%) | 1.53 (0.63, 3.15) | 0.29 |
| Medical officer | 207 (5.5%) | 11 (9.2%) | 1.76 (0.82, 3.45) | 0.12 |
| Patient age at discharge, months | 1 (0, 11) | 1 (0, 7) | 0.99 (0.97, 1.00) | 0.13 |
| Discharge diagnosis | ||||
| Non-infectious | 1531 (41%) | 48 (40%) | Referent | – |
| Infectious | 2245 (59%) | 72 (60%) | 0.95 (0.64, 1.42) | 0.80 |
| Duration of hospitalisation, days | 7 (3, 13) | 8 (3, 14) | 1.00 (0.99, 1.00) | 0.88 |
Discussion
Among nearly 3900 neonates and young children discharged from referral hospitals in Dar es Salaam, Tanzania and Monrovia, Liberia, clinician impression had poor precision for identifying those at risk of unplanned hospital readmissions and postdischarge mortality. Medical officer clinician impression at both sites had slightly greater precision in identifying young children at risk of readmission. Clinician perception of inability to pay for treatment was associated with readmission and postdischarge mortality.
The poor precision of clinician impression in identifying neonates and young children at risk of readmission and postdischarge mortality differs from findings in studies in high-income settings that assessed the diagnosis of acute coronary syndrome and sinusitis in adults,18 19 the presence of pneumonia20 or the development of severe pneumonia among children12 or the presence of appendicitis among children.21 This difference is likely multifactorial in nature and may consist of differences in available diagnostic and prognostic resources. Prior studies suggest that laboratory capabilities in resource-limited settings are inadequate,22–24 leading to dependence on clinical examination findings to make diagnoses and determine prognosis,25 which may hinder the accuracy of clinician impression to identify neonates and young children at risk of untoward postdischarge outcomes. Moreover, clinicians may not consider key factors in the home (eg, access to healthcare facilities and maternal health) that may contribute to postdischarge outcomes. Additionally, clinician impression had poor precision in identifying neonates and young children at risk for readmission or postdischarge mortality regardless of the time from discharge to either event. Prior studies of clinician impression assessed outcomes within hours or days12 18 19 and not up to 60 days, which may contribute to the difference in our results compared with prior studies assessing clinician impression in prognostication of outcomes.
Clinician impression among medical officers had fair precision in identifying young children at risk of readmission. This may be due to the combination of more clinical experience than interns/residents and more time spent with patients than consultants/specialists who often spend less time directly with patients and more time supervising clinical care. Prior studies conducted in high-income settings demonstrate that clinician impression of less experienced clinicians may have less discriminatory value than that of more experienced clinicians.12
linical prediction rules for postdischarge mortality have been developed among young children aged 6 months–5 years in Uganda and aged <15 years in Mozambique.8 9 These clinical prediction rules include variables such as clinical diagnoses and anthropometry to assign weighted points to included variables to assess an individual patient’s risk for postdischarge mortality up to 6 months after discharge. However, none of these have focused specifically on neonates and none have been externally validated, which is a necessary step to assess discriminatory value prior to clinical use. Thus, the current state of prognostic determination for young children after discharge in sub-Saharan Africa depends on clinician impression and, given its poor precision demonstrated in our study, validated clinical prediction rules to identify neonates and young children at risk of postdischarge mortality are urgently needed. Such clinical decision aids should include commonly collected variables and may include biomarkers to add precision to risk stratification to identify neonates and young children at risk of postdischarge morbidity and mortality.26–28
Our examination of reasons for estimated outcomes suggested that clinician perception of inability to pay for future treatment was associated with higher risk of readmission and postdischarge mortality. Young children from lower socioeconomic status have poorer overall health outcomes compared with young children from higher socioeconomic households in sub-Saharan Africa.29–31 This is particularly relevant in the postdischarge period during which the financial burden of care seeking may influence the ability for a family to seek additional clinical care after a potentially costly hospital admission.
Limitations
Clinician impression is multifactorial and depends on clinical training as well as available laboratory, radiological and clinical data that may not have been available to all discharging clinicians. We did not assess the availability of these in our analysis. We did not include nurses or clinical officers in our study, which is a limitation as these groups may have good insight into potential adverse outcomes after discharge. We could not account for the potential role that variations in the quality of clinical care provided to patients may have had in readmissions or postdischarge mortality.
Conclusions
Clinician impression had poor precision in identifying neonates and young children at risk of unplanned hospital readmission and postdischarge mortality at two referral hospitals in Dar es Salaam, Tanzania and Monrovia, Liberia. Validated and objective clinical decision aids to assist clinicians in the identification of young children at risk of readmission and postdischarge mortality may facilitate the identification of those at greatest risk.
Supplementary Material
Acknowledgments
We would like to thank the patients and their caregivers who enrolled in this study. We also thank the clinicians who volunteered their time to respond to the surveys.
Footnotes
Contributors: CAR, RK, RCI, AS, EG, HKM, CRS, MN, KM and CD conceptualised and designed the study. CAR, RCI, JK, AS, EG, CRS, KM and CD oversaw data collection and verified the underlying data. CAR and ALW verified the underlying data. ALW conducted the statistical analyses. CAR wrote the first draft of the manuscript. CAR, RK, RCI, JK, Y-JGC-N, AS, EG, HKM, CRS, ALW, MN, CRM, CGW, RFB, KM and CD interpreted the data, reviewed, and provided input to the final draft. CAR had final responsibility for the decision to submit for publication and is responsible for the overall content as the guarantor.
Funding: The authors wish to acknowledge funding for this work from the National Institutes of Health (K24 DK104676 and P30 DK040561 to CPD), the Boston Children’s Hospital Global Health Program to CAR, the Palfrey Fund for Child Health Advocacy to CAR, and the Emory Pediatric Research Alliance Junior Faculty Focused Award to CAR.
Competing interests: None declared.
Patient and public involvement: Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.
Provenance and peer review: Not commissioned; externally peer reviewed.
Supplemental material: This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
Data availability statement
Data are available upon reasonable request. Data Sharing Agreement: Data may be made available upon reasonable request to the corresponding author.
Ethics statements
Patient consent for publication
Consent obtained from parent(s)/guardian(s).
Ethics approval
The study received ethical clearance from the Tanzania National Institute of Medical Research (#NIMR/HQ/R8a/Vol.IX/3494), the Muhimbili University of Health and Allied Sciences Research and Ethics Committee (#307/323/01), the John F. Kennedy Medical Center Institutional Review Board (#08062019), the Boston Children’s Hospital Institutional Review Board (#P00033242), and the use of deidentified data was exempted from review by the Emory University Institutional Review Board (no number provided for exempted studies). Participants gave informed consent to participate in the study before taking part.
References
- 1.World Health Organization . Child mortality (under 5 years). 2022. Available: https://www.who.int/news-room/fact-sheets/detail/levels-and-trends-in-child-under-5-mortality-in-2020 [Accessed May 2023].
- 2.Pavlinac PB, Singa BO, Tickell KD, et al. Azithromycin for the prevention of Rehospitalisation and death among Kenyan children being discharged from hospital: a double-blind, placebo-controlled, randomised controlled trial. Lancet Glob Health 2021;9:e1569–78. 10.1016/S2214-109X(21)00347-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Connon R, George EC, Olupot-Olupot P, et al. Incidence and predictors of hospital readmission in children presenting with severe anaemia in Uganda and Malawi: a secondary analysis of TRACT trial data. BMC Public Health 2021;21:1480. 10.1186/s12889-021-11481-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Nemetchek B, English L, Kissoon N, et al. Paediatric Postdischarge mortality in developing countries: a systematic review. BMJ Open 2018;8:e023445. 10.1136/bmjopen-2018-023445 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Childhood Acute Illness and Nutrition (CHAIN) Network . Childhood mortality during and after acute illness in Africa and South Asia: a prospective cohort study. Lancet Glob Health 2022;10:e673–84. 10.1016/S2214-109X(22)00118-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Wiens MO, Pawluk S, Kissoon N, et al. Pediatric post-discharge mortality in resource poor countries: a systematic review. PLoS One 2013;8:e66698. 10.1371/journal.pone.0066698 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Goodman DM, Casale MT, Rychlik K, et al. Development and validation of an integrated suite of prediction models for all-cause 30-day Readmissions of children and adolescents aged 0 to 18 years. JAMA Netw Open 2022;5:e2241513. 10.1001/jamanetworkopen.2022.41513 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Wiens MO, Kumbakumba E, Larson CP, et al. Postdischarge mortality in children with acute infectious diseases: derivation of Postdischarge mortality prediction models. BMJ Open 2015;5:e009449. 10.1136/bmjopen-2015-009449 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Madrid L, Casellas A, Sacoor C, et al. Postdischarge mortality prediction in sub-Saharan Africa. Pediatrics 2019;143:e20180606. 10.1542/peds.2018-0606 [DOI] [PubMed] [Google Scholar]
- 10.Horwood J, Cabral C, Hay AD, et al. Primary care clinician antibiotic prescribing decisions in consultations for children with Rtis: a qualitative interview study. Br J Gen Pract 2016;66:e207–13. 10.3399/bjgp16X683821 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Van den Bruel A, Thompson M, Buntinx F, et al. Clinicians' gut feeling about serious infections in children: observational study. BMJ 2012;345:e6144. 10.1136/bmj.e6144 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Gao HM, Ambroggio L, Shah SS, et al. Predictive value of clinician "Gestalt" in pediatric community-acquired pneumonia. Pediatrics 2021;147:e2020041582. 10.1542/peds.2020-041582 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Turnbull S, Lucas PJ, Redmond NM, et al. What gives rise to clinician gut feeling, its influence on management decisions and its Prognostic value for children with RTI in primary care: a prospective cohort study. BMC Fam Pract 2018;19:25. 10.1186/s12875-018-0716-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Paul S, Tickell KD, Ojee E, et al. Knowledge, attitudes, and perceptions of Kenyan Healthcare workers regarding pediatric discharge from hospital. PLoS One 2021;16:e0249569. 10.1371/journal.pone.0249569 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Rees CA, Kisenge R, Ideh RC, et al. A prospective, observational cohort study to identify neonates and children at risk of Postdischarge mortality in Dar es Salaam, Tanzania and Monrovia, Liberia: the PPDM study protocol. BMJ Paediatr Open 2022;6:e001379. 10.1136/bmjpo-2021-001379 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Saito T, Rehmsmeier M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced Datasets. PLoS One 2015;10:e0118432. 10.1371/journal.pone.0118432 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Saito T, Rehmsmeier M. Precrec: fast and accurate precision-recall and ROC curve calculations in R. Bioinformatics 2017;33:145–7. 10.1093/bioinformatics/btw570 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Williams JW, Simel DL, Roberts L, et al. Clinical evaluation for sinusitis. making the diagnosis by history and physical examination. Ann Intern Med 1992;117:705–10. 10.7326/0003-4819-117-9-705 [DOI] [PubMed] [Google Scholar]
- 19.Oliver G, Reynard C, Morris N, et al. Can emergency physician Gestalt "rule in" or "rule out" acute coronary syndrome: validation in a multicenter prospective diagnostic cohort study. Acad Emerg Med 2020;27:24–30. 10.1111/acem.13836 [DOI] [PubMed] [Google Scholar]
- 20.Neuman MI, Scully KJ, Kim D, et al. Physician assessment of the likelihood of pneumonia in a pediatric emergency Department. Pediatr Emerg Care 2010;26:817–22. 10.1097/PEC.0b013e3181fb0d95 [DOI] [PubMed] [Google Scholar]
- 21.Lee WH, O’Brien S, Skarin D, et al. Accuracy of clinician Gestalt in diagnosing Appendicitis in children presenting to the emergency Department. Emerg Med Australas 2019;31:612–8. 10.1111/1742-6723.13220 [DOI] [PubMed] [Google Scholar]
- 22.Yadav H, Shah D, Sayed S, et al. Availability of essential diagnostics in ten low-income and middle-income countries: results from national health facility surveys. Lancet Glob Health 2021;9:e1553–60. 10.1016/S2214-109X(21)00442-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Peeling RW, Mabey D. Point-of-care tests for diagnosing infections in the developing world. Clin Microbiol Infect 2010;16:1062–9. 10.1111/j.1469-0691.2010.03279.x [DOI] [PubMed] [Google Scholar]
- 24.Singh H, Schiff GD, Graber ML, et al. The global burden of diagnostic errors in primary care. BMJ Qual Saf 2017;26:484–94. 10.1136/bmjqs-2016-005401 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.World Health Organization . Integrated management of childhood illness: management of the sick young infant aged up to 2 months: IMCI chart booklet. 2019. Available: https://www.who.int/maternal_child_adolescent/documents/management-sick-young-infant-0-2-months/en [Accessed 24 Sep 2021].
- 26.Njunge JM, Tickell K, Diallo AH, et al. The childhood acute illness and nutrition (CHAIN) network nested case-cohort study protocol: a multi-Omics approach to understanding mortality among children in sub-Saharan Africa and South Asia. Gates Open Res 2022;6:77. 10.12688/gatesopenres.13635.2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Njunge JM, Gwela A, Kibinge NK, et al. Biomarkers of post-discharge mortality among children with complicated severe acute malnutrition. Sci Rep 2019;9:5981. 10.1038/s41598-019-42436-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Sarangam ML, Namazzi R, Datta D, et al. Intestinal injury biomarkers predict mortality in pediatric severe malaria. MBio 2022;13:e01325-22. 10.1128/mbio.01325-22 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Ogbo FA, Ezeh OK, Awosemo AO, et al. Determinants of trends in neonatal, post-neonatal, infant, child and under-five mortalities in Tanzania from 2004 to 2016. BMC Public Health 2019;19:1243. 10.1186/s12889-019-7547-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Burkey MD, Weiser SD, Fehmie D, et al. Socioeconomic determinants of mortality in HIV: evidence from a clinical cohort in Uganda. J Acquir Immune Defic Syndr 2014;66:41–7. 10.1097/QAI.0000000000000094 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Ezeh OK, Agho KE, Dibley MJ, et al. Risk factors for Postneonatal, infant, child and Under-5 mortality in Nigeria: a pooled cross-sectional analysis. BMJ Open 2015;5:e006779. 10.1136/bmjopen-2014-006779 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
bmjpo-2023-001972supp001.pdf (79.6KB, pdf)
bmjpo-2023-001972supp002.pdf (89.7KB, pdf)
Data Availability Statement
Data are available upon reasonable request. Data Sharing Agreement: Data may be made available upon reasonable request to the corresponding author.