Abstract
Background
Post-discharge mortality is a frequent but poorly recognized contributor to child mortality in resource limited countries. The identification of children at high risk for post-discharge mortality is a critically important first step in addressing this problem.
Objectives
The objective of this project was to determine the variables most likely to be associated with post-discharge mortality which are to be included in a prediction modelling study.
Methods
A two-round modified Delphi process was completed for the review of a priori selected variables and selection of new variables. Variables were evaluated on relevance according to (1) prediction (2) availability (3) cost and (4) time required for measurement. Participants included experts in a variety of relevant fields.
Results
During the first round of the modified Delphi process, 23 experts evaluated 17 variables. Forty further variables were suggested and were reviewed during the second round by 12 experts. During the second round 16 additional variables were evaluated. Thirty unique variables were compiled for use in the prediction modelling study.
Conclusion
A systematic approach was utilized to generate an optimal list of candidate predictor variables for the incorporation into a study on prediction of pediatric post-discharge mortality in a resource poor setting.
Keywords: Candidate predictor variables, pediatrics, prediction, post-discharge mortality, sepsis
Introduction
Acute infectious diseases account for most childhood deaths in resource limited countries, particularly on the African continent.1 Sepsis, according to the Pediatric Consensus Conference definition, is the systemic response to any infectious disease which can progress from a mild inflammatory derangement (systemic inflammatory response syndrome) to multi-organ failure, shock and death2, and therefore, can be broadly considered to be the primary cause of acute infectious disease death.3 While the period of acute illness is well known to account for much of this burden, relatively little is known about the epidemiology of the post-acute period and its contribution to overall infectious disease associated mortality. A recent systematic review has found that the months following hospital discharge account for at least as many deaths as the hospital period.4 Despite the apparently high burden of post discharge mortality in children, little currently exists to address this important issue. The identification of vulnerable children is an important first step in addressing this important public health problem.3 Research aimed at creating tools to aid in such identification must ensure that the most important and relevant predictors are included for analysis in order to optimize model development.5,6
The method by which candidate predictors are selected vary widely among studies and many include informal discussions among the research team and colleagues or more formal methods such as focus group discussions, questionnaires and surveys. A well-recognized method by which groups can reach consensus on a subject of interest is the Delphi method.7 The Delphi method is a structured communication technique in which a facilitator solicits experts to answer a questionnaire in two or more rounds. A summary of each round is reviewed by the experts providing opportunity to modify previously selected answers, thus converging, in theory, towards the most correct answer. This method has been used successfully in prediction of modelling research as a means to generate a comprehensive set of candidate predictor variables to be used in statistical modeling.8
The purpose of this protocol development project was to generate a comprehensive list of candidate predictor variables, using a modified Delphi process, to be used for pediatric post-discharge mortality research in South-Western, Uganda. Ultimately, the predictors derived will be used in the creation of post-discharge mortality prediction models.
Methods
Design
A modified two-round Delphi process was undertaken to determine an optimal set of candidate predictor variables to be collected for the prediction modelling portion of the study. The primary modification was the elimination of the participants to modify their responses based on the responses of others. As the primary aim of this project was to aid in protocol development, this did not satisfy the definition of research requiring review by the research ethics boards of the University of British Columbia and the Mbarara University of Science and Technology. The study team adhered to best research practices throughout the project.
Participants
Participants in the Delphi process were recruited by the solicitation of the primary research team, and included both those solicited by the research team as well as the research team itself. The research team included experts (as defined subjectively by the research team) in global health, critical care, pediatrics, statistics, methodology, epidemiology, computer engineering and infectious diseases. Expertise was sought in at least one area potentially relevant to post-discharge mortality. The required areas of expertise of participants (research team and non-research team participants) included (1) pediatrics, (2) sepsis, (3) infectious diseases, (4) microbiology/laboratory medicine, (5) international health, (6) epidemiology or (7) social sciences. The participants were invited to participate in both rounds of the Delphi process. Our target sample size was 20 individuals that covered all seven areas of expertise and included participants from the proposed research country (Uganda). Experts also were asked to self-identify their area of expertise based on a list provided at the end of the survey.
Process
The modified Delphi process occurred between November 2010 and January 2011 and was completed during two rounds of emailed surveys using SurveyMonkey ® software (Palo Alto, CA). The invitational email is included in Appendix 1. Fourteen days was granted to participants during each round of the process. After each round the research team members involved in candidate variable selection discussed the survey responses and determined whether the existing or suggested variable should be added, modified or eliminated. A final consensus of candidate predictors was made by the research team following the results of the second survey. This process was considered a modified Delphi process as no direct interaction was possible between candidates. However, candidates did have the opportunity to review and critique each other's suggestions.
Round 1
Round 1 of the Delphi process was initiated in November, 2010. An initial list of 17 candidate variables was generated by the research team prior to this first round of the Delphi process, following a review of existing literature as well as the clinical and research experiences of co-investigators (Table 1). This list of 17 variables included multi-part variables (e.g. vital signs labelled as a single variable although it included respiratory rate, temperature, heart rate and blood pressure). Therefore, the actual list of candidate variables for statistical modelling would be higher than the list of variables reviewed. Critical domains of the initial variable selection included (1) its potential as a predictor and likely inter- and intra-rater reliability of its measurement, (2) its general availability in most resource limited contexts, (3) its cost, and (4) the time and resources required collecting the variable. These four domains, therefore, were the primary component of the Delphi evaluation.
Table 1.
Initial list of candidate predictor variables, N=17
| Clinical | Laboratory | Social/Demographic |
| Vital signs (HR, RR, BP, T) | Hemoglobin | Age |
| Oxygen saturation | Blood culture | Sex |
| Height | Zinc level | Maternal education |
| Weight | Wealth | |
| Hib, Pneumococcal vaccination | Distance from health facility | |
| Length of stay | ||
| Co-morbidities | ||
| Admission diagnosis | ||
| Discharge AMA |
HR = heart rate, RR = respiratory rate, BP = blood pressure, T = temperature, Hib = haemophilus influenza type b, AMA = against medical advice
In addition to the rating of each of the 17 variables according to the four domains, participants were given the opportunity to make comments and suggest additional variables for consideration during the second round of the Delphi process. The primary research team considered each proposed variable and eliminated those variables deemed to be unsuitable based on any of the domains or if redundant (e.g. Nutrition status was available as weight/height for age z-score). The new set of potential candidate predictor variables were then incorporated into the second round of the Delphi process. Using the survey results from this initial round, the primary research team also modified or removed variables from the initial list of 17 candidate variables to be used in the final list of candidate predictor variables.
Round 2
The second round of the Delphi process was conducted in December, 2010. During this round 16 new variables were assessed using the same domains and scoring system used during the first round. The ability to make comments was preserved. This round did not include a re-review of the initial variables used during the first round. Using the results from this second round of evaluation, the primary research team retained, modified or removed the additional candidate predictor variables and incorporated these into the final list of candidate predictor variables.
Analysis
Evaluation of each variable was considered for applicability on 4 domains previously described. Respondents scored each predictor as having either (1) high, (2) moderate, (3) unlikely or (4) no applicability for each of the four domains. Responses were tabulated and reported using descriptive analyses only (SAS 9.3, Carey, NC). The proportion of respondents indicating a high level of applicability was of primary interest. The proportion of respondents indicating unlikely or no applicability of the candidate variables to each domain was also of interest. A final list of variables for inclusion as candidate predictors was determined by the study team using both the results of the distributed survey results as well as other considerations such as budgetary, research staff availability and equipment availability.
Results
Twenty three participants from both high resource rich and resource limited environments (approximately equal distribution) participated during the first round of the Delphi process. During this round, 17 initially proposed candidate predictors (Table 1) were evaluated and a further 40 were proposed.
Of the 40 proposed variables, 16 were selected for inclusion in the second round. During the second round, 12 participants (52%) of the initial 23 completed the survey (Figure 1).
Figure 1.
Flow diagram of Delphi process.
* Variables included some with multiple predictors listed as single variables (eg. height/weight and vital signs). Total predictors are therefore not a sum of final variables from Round 1 and Round 2 of Delphi process.
Round 1
Self-identified areas of expertise among the participants include pediatrics (70%), sepsis (57%), infectious diseases (30%), international health (26%) epidemiology (22%), critical care (17%), microbiology or laboratory medicine (9%) and the social sciences (9%). Participants included individuals from both high resource and resource limited countries.
Applicability of proposed candidate predictors as potential predictors
Of the 17 variables evaluated during this first round of the Delphi process, oxygen saturation, height/weight, co-morbidities, age, maternal education and wealth received scores of “high applicability” for over 80% of participants who evaluated these indicators. Vital signs, vaccination status, length of stay, admission diagnosis, discharge against medical advice, and distance from hospital received a moderate amount of “high” ratings (60% — 80%). Sex, zinc level, hemoglobin level, and prior antibiotic use received low levels (<60%) of high ratings relative to the other variables. Zinc level in particular received few high ratings (29%). Although wealth received a high proportion of high ratings, only 13 participants rated this. Many respondents commented on their lack of knowledge of wealth scoring tools. Variables exhibiting the highest proportion of “unlikely” or “not at all” applicable included zinc level and sex, with 21% and 24% or participants providing these scores, respectively.
Applicability of proposed candidate predictors for availability
Variables scoring over 80% on availability included length of stay, age and sex. Vital signs, height and weight, discharge against medical advice, co-morbidities, maternal education and distance were considered highly available by 60 — 80% of participants. Prior antibiotics, hemoglobin, zinc level, hemoglobin, zinc level and wealth were considered to be highly available in 40 − 60% of participants. Oxygen saturation, blood culture and immunization status were considered highly available in fewer than 40% of subjects. Only blood culture and vaccination status were considered unlikely or not available for more than 20% of participants.
Applicability of proposed candidate predictors for cost
Age and sex were noted to be highly applicable in terms of cost for greater than 80% of participants. Height and weight, length of stay, co-morbidities, admitting diagnosis, discharge against medical advice, maternal education, wealth and distance were all considered to be highly applicable in terms of cost in 60% – 80% of participants. Vaccination status, prior antibiotic use and hemoglobin were considered highly applicable in terms of cost in 40 — 60% of participants and oxygen saturation, blood culture and zinc level were all considered highly applicable in terms of cost for fewer than 40% of subjects. Twenty nine percent of respondents believed blood culture to be unlikely or not applicable in terms of cost and 78% of participants believed that zinc level was unlikely or not applicable in terms of costs.
Applicability of proposed candidate predictors for time and resources required
The applicability of time and resources required to measure each potential candidate predictor was assessed and closely mirrored the cost domain results. The primary difference is that discharge against medical advice and the assessment of co-morbidities were rated highly applicable in 40% – 60% of participants in this domain rather than over 60% in the cost domain. Only zinc was associated with a high proportion of participants selecting it as being unlikely or not applicable (71%).
Proposed new variables
Forty additional variables were proposed as potential candidate predictors, many of which were similar or overlapping (Table 2). Sixteen of these were included in the second round of the Delphi process.
Table 2.
Proposed new variables and final round 2 variables.
| Clinical | Laboratory | Social/Demographic |
| Proposed new variables for prediction, N=40 | ||
| Eating & drinking status | Blood glucose | Number of siblings |
| Mental status | C-reactive protein | Number of parents present |
| Weight gain during admission | CBC with differential | Type of dwelling |
| Inputs and outputs | Chest X-ray | Water source |
| Breastfeeding success | HIV serology | Hygiene indicators |
| Urination in 12h prior to admit | CD4 count | Bed net use |
| Nutritional status | Blood gases | Cooking habits |
| HIV status | Ferritin | Maternal co morbidities |
| Antibiotic appropriateness | INR/PTT | Ethnicity |
| Time since last hospitalization | Lactate, pH, or serum bicarbonate | Social economic status |
| Time from referral to admission | Blood urea nitrogen | Sibling deaths |
| Length of illness | Serum creatinine | Maternal birth spacing |
| Mid-upper arm circumference | Albumin | |
| Previous hospitalizations | CNS culture | |
| Final round 2 variables for prediction, N=16 | ||
| Mental status | Glucose | Number of siblings |
| Pre-admit urine freq. | Acidosis | Maternal co-morbidities |
| Prior hospitalizations | Coagulation (INR/PTT) | Maternal age |
| Time since last admit | Renal (BUN/SCr) | Parents live at home |
| Duration of illness | Sibling deaths | |
| MUAC | Water source | |
| Prematurity | Bed net use | |
CBC = complete blood count, INR = International normalized ratio, PTT = Partial thromboplastin time, BUN = Blood urea nitrogen, SCr = Serum creatinine, MUAC = Mid-upper arm circumference
Round 2
Twelve participants completed this final round of the Delphi process. During this process 16 potential candidate predictor variables (or groups of predictor variables) were evaluated using the previously used domains and scoring system.
Applicability of proposed candidate predictors as potential predictors
Mental status was deemed to be highly applicable as a potential candidate predictor variable in 80% of participant responses, and was the highest of all 16 candidate predictors. Urinary frequency prior to admission, duration of illness prior to admission, prematurity, glucose level, and acidosis measurement were considered highly applicable in 60% – 80% of responses. Previous hospitalizations in the past year, maternal co-morbidities and the number of parents living at home were associated with a high level of applicability in 40% – 60% of participants, while the time since last hospitalization, mid-upper arm circumference, presence of sibling deaths, blood coagulation profile, renal function profile and maternal age were all associated with fewer than 40% of participants considering them as highly likely to be associated with post-discharge mortality. Only maternal age and the number of parents living at home was considered to be unlikely or not at all applicable as predictors by more than 20% of participants.
Applicability of proposed candidate predictors for availability
Duration of illness prior to hospitalization and mental status were considered highly available by over 80% of participants. Previous hospitalizations, maternal age, the number of parents living at home, bednet use and water source were considered highly available in 60% – 80% of subjects. Urination in 12 hours prior to admission, prematurity, sibling deaths, and glucose were considered highly available by 40% – 60% of participants, while the number of siblings, blood coagulation profile, acidosis measurement, maternal co-morbidities, renal function and mid-upper arm circumference were considered highly available in fewer than 40% of participants. Several candidate predictors were considered unlikely or not to be available by more than 20% of participants, including the number of parents living at home, maternal age, number of siblings, renal function, blood coagulation and acidosis measurement.
Applicability of proposed candidate predictors for cost
The proportion of participants rating the candidate predictor variables as highly cost-effective was over 80% for bednet use only. Urination in the 12 hours prior to admission, previous hospitalizations, sibling deaths, number of siblings, maternal age, number of parents living at home and water source were considered highly cost-effective for 60% – 80% of participants, while time since last hospitalization, prematurity, maternal co-morbidities and glucose were considered cost-applicable by 40% – 60% of participants. Duration of illness prior to admission, mid-upper arm circumference, acidosis measurement, blood coagulation profile and renal function received a high applicability rating in terms of cost by fewer than 40% of participants. The laboratory variables including renal function, blood coagulation profile, and acidosis measures were considered unlikely or not applicable in terms of cost in over 20% of participants, as was collection of maternal age and the number of parents living at home.
Applicability of proposed candidate predictors for time and resources required
The responses of the participants for the applicability of the time and resources required to collect the proposed candidate predictor variables mirrored the results of the applicability of the cost of collection of these variables on all descriptive terms.
Final list of candidate predictor variables
The final list of candidate predictor variables was determined through discussions among the primary research team (Table 3). Budget considerations and actual availability at the proposed research site played the most significant role following a review of the survey results. The final list of 30 candidate predictors was approved by the primary research team and incorporated into the research proposal.
Table 3.
Final list of candidate predictors for predictive model, N = 30
| Clinical | Laboratory | Social/demographic |
| Height | Hemoglobin | Bed net use |
| Weight | HIV status | Maternal age |
| Mid-upper arm circumference | Malaria blood smear | Maternal education |
| Heart rate | Maternal HIV status | |
| Respiratory rate | Maternal death | |
| Systolic blood pressure | Number of siblings | |
| Diastolic blood pressure | Sibling deaths | |
| Temperature (axillary) | Distance (time) | |
| Oxygen saturation | Distance (cost) | |
| Blantyre coma scale | Availability of latrine | |
| Immunization status | Water source | |
| Prior antibiotic use | Boiling of drinking water | |
| Prior antimalarial use | ||
| Time since last hospitalization | ||
| Duration of illness |
Discussion
A list of candidate predictor variables was derived using a modified, 2-round, Delphi approach. This approach, utilizing the expertise of individuals not part of the primary research team, added considerable depth to the evaluation of candidate predictor variables, as evidenced by contrasting the initial list of predictors with the final list of predictors (Tables 1 and 3).
The final list of candidate predictors was chosen based on both subjective considerations by the research team and objective results of the survey. Some proposed variables that scored relatively poorly during the second round of the Delphi process were chosen based on very practical considerations at the research site. For example, mid-upper arm circumference was not highly rated on any of the domains. However, this variable was chosen as a simpler method of nutritional status than a measure based on the variables of height, weight and age. The true cost considerations from a research perspective included the cost of scales and length boards for height and weight, compared to a mid-upper arm circumference tape, which is less expensive. The research team, did however, agree with the results of newly proposed laboratory variables, which tended to score poorly in all domains, particularly in the cost and resources domains.
This protocol development project actively utilized experts from resource limited countries, beyond those on the research team, who had in-depth knowledge of current practices and standards at the proposed research areas. This broad expertise ensured that the more pragmatic indicators were appropriately evaluated and further ensured that the overall context was broadly considered throughout the two phases of the Delphi process.9
This process was subject to several limitations. A primary limitation to this process was the inability to revise responses based on a summary of group responses. Because of this we were not able to evaluate more the effect of further consideration of individual responses. Furthermore, no opportunities were available for specific discussion on any candidate predictor variables. Although provisions for comments were incorporated into the survey, these were only seen by the primary research team. The reason for this Delphi process simplification was to facilitate participation, rather than due to an accidental omission of this process. A further limitation of this survey was a relatively high proportion of individuals not completing the second round (10 out of 23). No identifying information or participant characteristics were collected during the second round of the survey and therefore the breakdown of self-identified expertise was not considered in the responses, as it was during the first round. However, each section of the survey (clinical, laboratory and social/demographic) was initiated by a question about the participants ability to contribute meaningfully and they were provided with an option to skip the section if they desired.
Conclusion
This modified Delphi process was an effective method to add both objectivity and to broaden the perspective for the selection of candidate predictor variables. This increases the likelihood that a robust set of candidate variables will be included in the proposed post-discharge mortality modelling research project.
References
- 1.Liu L, Oza S, Hogan D, Perin J, Rudan I, Lawn JE, et al. Global, regional, and national causes of child mortality in 2000–13, with projections to inform post-2015 priorities: an updated systematic analysis. Lancet [Internet] 2015;385(9966):430–440. doi: 10.1016/S0140-6736(14)61698-6. Available from: http://linkinghub.elsevier.com/retrieve/pii/S0140673614616986. [DOI] [PubMed] [Google Scholar]
- 2.Goldstein B, Giroir B, Randolph A. International pediatric sepsis consensus conference: definitions for sepsis and organ dysfunction in pediatrics. Pediatr Crit Care Med [Internet] 2005 Jan;6(1):2–8. doi: 10.1097/01.PCC.0000149131.72248.E6. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15636651. [DOI] [PubMed] [Google Scholar]
- 3.Wiens MO, Kumbakumba E, Kissoon N, Ansermino JM, Ndamira A, Larson CP. Pediatric sepsis in the developing world: challenges in defining sepsis and issues in post-discharge mortality. Clin Epidemiol [Internet] 2012 Jan;4:319–325. doi: 10.2147/CLEP.S35693. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?.artid=3514048&tool=pmcentrez&rendertype=abstract. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Wiens M, Pawluk S, Kissoon N. Pediatric Post-Discharge Mortality in Resource PoorCountries: A Systematic Review. PLoS One [Internet] 2013 Jan;8(6):e66698. doi: 10.1371/journal.pone.0066698. [cited 2014 Jun 19] Available from: http://dx.plos.org/10.1371/journal.pone.0066698. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Royston P, Moons KGM, Altman DG, Vergouwe Y. Prognosis and prognostic research: Developing a prognostic model. Br Med J [Internet] 2009 Mar;338:b604. doi: 10.1136/bmj.b604. [cited 2010 Dec 13] Available from: http://www.bmj.com/cgi/doi/10.1136/bmj.b604. [DOI] [PubMed] [Google Scholar]
- 6.Hemingway H, Riley RD, Altman DG. Ten steps towards improving prognosis research. Br Med J [Internet] 2009 Dec;339:b4184. doi: 10.1136/bmj.b4184. [cited 2011 Jan 27] Available from: http://www.bmj.com/cgi/doi/10.1136/bmj.b4184. [DOI] [PubMed] [Google Scholar]
- 7.Hsu C, Ohio T. The Delphi technique: Making sense of consensus. Pract Assessment, Res Eval [Internet] 2007;12(10):1–8. Available from: http://pareonline.net/getvn.asp?.v=12&n=10. [Google Scholar]
- 8.Von Dadelszen P, Payne B, Li J, Ansermino JM, Broughton Pipkin F, Côté A-M, et al. Lancet [Internet] 9761. Vol. 377. Elsevier Ltd; 2011. Jan 15, Prediction of adverse maternal outcomes in pre-eclampsia: development and validation of the fullPIERS model; pp. 219–227. [cited 2015 Feb 4] Available from: http://www.ncbi.nlm.nih.gov/pubmed/21185591. [DOI] [PubMed] [Google Scholar]
- 9.Molyneux E, Mathanga D, Witte D, Molyneux M. Practical issues in relation to clinical trials in children in low-income countries: experience from the front line. Arch Dis Child. 2012;97:848–851. doi: 10.1136/archdischild-2011-301476. [DOI] [PubMed] [Google Scholar]