Standardized Outcome Measures for Preterm and Hospitalized Neonates: An ICHOM Standard Set

Abstract

Introduction

Approximately, one in ten infants is born preterm or requires hospitalization at birth. These complications at birth have long-term consequences that can extend into childhood and adulthood. Timely detection of developmental delay through surveillance could enable tailored support for these babies and their families. However, the possibilities for follow-up are limited, especially in middle- and low-income countries, and the tools to do so are either not available or too expensive. A standardized and core set of outcomes for neonates, with feasible tools for evaluation and follow-up, could result in improving quality, enhance shared decision-making, and enable global benchmarking.

Methods

The International Consortium for Health Outcomes Measurement (ICHOM) convened an international working group, which was comprised of 14 health-care professionals (HCP) and 6 patient representatives in the field of neonatal care. An outcome set was developed using a three-round modified Delphi process, and it was endorsed through a patient representative-validation survey and an HCP survey.

Results

A literature review revealed 1,076 articles and 26 registries which were screened for meaningful outcomes, patient-reported outcome measures, clinical measures, and case mix variables. This resulted in a neonatal set with 21 core outcomes covering three domains (physical, social, and mental functioning) and 14 tools to assess these outcomes at three timepoints.

Discussion

This set can be implemented globally and it will allow comparison of outcomes across different settings and countries. The transparent consensus-driven development process which involved stakeholders and professionals from all over the world ensures global relevance.

Introduction

Approximately one in ten infants are born preterm (before 37 completed weeks of gestation) or require hospitalization at birth. Complications at birth have long-term consequences that can extend into childhood and adulthood. Some of these infants develop disabilities that affect every aspect of day-to-day life and result in learning, social, or motor difficulties. Besides the direct physical or psychological consequences for the child, parents experience stress during the neonatal intensive care unit stay, and the frequent subsequent hospital admissions have an impact on the whole family [1, 2, 3, 4].

Currently, the neonatal outcomes that are assessed are largely acute (e.g., survival rate or discharge on oxygen) or intermediate (neurodevelopment at the age of 2 years). However, there is a debate on which outcomes to target during long-term follow-up in order to detect important complications [5]. Guidance on how to follow-up and what measures to use varies among institutions and guidelines [6, 7]. It is known that developmental surveillance for the timely detection of developmental delay by health-care professionals (HCPs) could enable tailored support for former neonates at risk and their families.

If parents also received education and support, with early intervention, former preterm infants might more often reach their full potential [8, 9, 10]. Nevertheless, meaningful outcomes for parents and children with lived experience may be different from those traditionally viewed as important by HCPs and, therefore, may not have been measured in clinical care or research [11, 12]. Differing views on meaningful outcomes between stakeholders and HCPs are likely universal. Such divergences are probably amplified by disparities of income and health-care accessibility in different countries.

However, the possibilities to follow-up and support former neonates at risk are limited, especially in middle- and low-income countries, and the tools to do so are either not available or too expensive [13]. A standardized minimal set of meaningful outcomes for former preterm and hospitalized neonates, agreed upon by stakeholders and neonatal HCPs representing high-, middle-, and low-income countries, with feasible tools for evaluation and follow-up, could result in improving quality, enhance decision-making between HCPs and patients, and enable global benchmarking.

Materials and Methods

ICHOM Background and Goals

The International Consortium for Health Outcomes Measurement (ICHOM) is a non-profit organization that aims to facilitate the adoption of value-based health care worldwide by developing standardized outcome sets created by an international Working Group (WG) representing high-, middle-, and low-income countries all over the globe, consisting of patient representatives and HCPs. A standard set also provides recommendations on timepoints of administration and tools to evaluate the specific outcomes. The tool selection criteria are based on the tools' reliability, content validity, construct validity, internal consistency, test-retest reliability, and clinical utility as previously described for other ICHOM outcome sets. Preferential tools are available in multiple languages, and single tools cover multiple outcomes. Uncomplicated implementation within diverse, international, clinical settings is deemed important and the tools should not be too time consuming or burdensome to complete.

WG Assembly and Composition

We assembled a WG of 20 international experts from 20 organizations across 15 different countries in the fields of perinatal and neonatal care; 14 HCPs (neonatology, general pediatrics, nursing, psychology, ophthalmology, public health, and epidemiology) and a patient advisory group (PAG) recruited through the European Foundation for the Care of Newborn Infants (EFCNI) network was convened (see online suppl. Table 1; for all online suppl. material, see). These 6 patient representatives in the PAG were chosen because they either work very closely with preterm infants and their families, or they had preterm or severely ill newborns themselves. A project team (E.S., J.H., A.F., N.S., J.R., C.N., A.C., L.M.) consisting of project managers, research associates, a chair, and two research fellows coordinated the development process.

Work Plan and Decision-Making

An initial comprehensive literature review of the last 10 years was conducted to identify potentially relevant outcomes. Studies were identified by searching Medical Literature Analysis and Retrieval System Online (MEDLINE), Cochrane Central Register of Controlled Trials (CENTRAL), and PubPsych using terms encompassing “neonate, preterm, outcome, mortality, morbidity, neurodevelopment, and health-related quality of life.” The full syntax is available on demand. Outcomes for infants admitted to the hospital beyond the neonatal period (first 28 days of life), those with genetic disorders, severe malformations, or rare conditions (prevalence of ≤1/2,000 population) were excluded [14]. For tool selection, the literature, key international surveys, and clinical practice guidelines were reviewed. Factors influencing follow-up, such as low maternal educational level, socioeconomic status, stress of daily living, and neurodevelopmental outcome, were considered in choosing suitable tools [4, 15, 16, 17].

Between March 2019 and June 2020, there were seven teleconferences in which the WG discussed the scope, outcomes, tools, case-mix variables, and timepoints for the set that took place. The outcomes, tools, and case mix variables were subject to voting through a three-round modified Delphi process (see online suppl. Fig. 1). The voice of the patient representatives was instrumental in the development of the standard set. In addition to the teleconferences, members of the PAG were interviewed to elicit outcomes and case-mix variables of importance.

The final step was an open review period to elicit feedback on the set from HCPs and parents with lived experience. This was done via two anonymous online surveys. Parent responses were sourced from high-, low-, and middle-income countries to ensure that recommendations were applicable worldwide. They were asked to rate the importance of the selected outcomes and for any outcome suggestions or critical concepts they felt were missing. Professionals from around the world (different from the WG) were polled on the set.

Results

The final set was composed of 21 key outcomes and 14 tools encompassing three domains: physical, social, and mental functioning.

Outcome Domains

The literature review process, as shown in Figure 1, revealed 1,076 articles and 26 registries to be included for identifying outcomes. 46 outcomes were voted on and 21 outcomes were included in the set. The outcomes could be grouped into five mental outcomes, six social outcomes, and ten physical functioning outcomes (Fig. 2). To ensure universal understanding, every outcome was defined (Table 1).

Fig. 1.

RCT, randomized controlled trial.

Fig. 2.

Outcome wheel. The numbers next to the outcomes refer to recommended tools to evaluate each outcome.

Table 1.

Included outcomes in the set with definition, data source, and agreement in the WG

Outcome	Outcome definition	Data source	Agreement, %
Mental functioning
Anxiety	An emotion characterized by an unpleasant state of inner turmoil, often accompanied by nervous behavior and feeling physically unwell	P	80
Depression	A medical illness that negatively affects how a child feels, the way we think, and act. It is characterized by feelings of sadness and/or loss of interest in activities once enjoyed	P	87
Behavior	Behavioral actions that influence functioning in society	P	80
Cognition	The ability to learn, think, remember, reason, and make decisions	P	93
Neurodevelopment	Functioning of the brain that affects a child's behavior, vision, hearing, and movement	P	87
Social functioning
Schooling	The ability to participate in educational activity in any kind of structured way	P	80
Social functioning	The ability to interact easily and successfully with others	P	80
Relationship with others	The ability to establish a connection with others which provides security, happiness, and meaning to life	P	80
Health-related quality of life	A person's perception of his/her ability to achieve the goals, expectations, and standards he or she values of importance; being able to fulfil the duties required to be self-sufficient for the activities of daily life	P	87
Communication	The ability to express thoughts, feelings, and wishes to others and being understood	P/C	80
Impact on family	The effect of the medical circumstances of caring for a preterm or hospitalized newborn patient that influences family life of any kind	P	80
Physical functioning
Vision	The ability to see	C	93
Hearing	The ability to perceive sounds by the ears sufficiently to cope in daily life/communicate	C	93
Sleep	The primary activity of the brain during early development, which impacts mental and physical development	P	80
Pain	An unpleasant sensory and emotional experience associated with actual or potential tissue damage or described in terms of such damage	P	71
Readmission	Readmission to the hospital after being discharged from the primary hospital stay	C	80
Survival	Being alive after experiencing a serious medical problem	C	93
Disability	Refers to a problem in body function which results in a difficulty encountered by an individual in executing a task or action	P	80
Motor function	The ability to learn, execute and control specific movements	P	93
Pulmonary function	Describes the extent to which somebody can breathe without any support	C	80
Feeding, nutrition, and growth	The ability to receive a proper intake of food in order to develop, metabolize, and repair	C	80

P, parent-reported outcomes measure (PROM); C, clinical measure

The timepoints for evaluating each outcome were chosen based on clinical appropriateness, feasibility, and relevance. The WG and the PAG agreed on three timepoints for outcome collection: during hospitalization, at 2 years, and at 5 years of age.

Tools to Evaluate Selected Outcomes

There were 44 patient-reported outcome measures (PROMs) and 50 clinical measures that were identified in the literature that could measure the 21 final outcomes. After assessing validity, reliability, feasibility, and clinical utility, six PROMs and eight clinical measures were included in the set (Table 2). Not all selected outcomes could be measured at all timepoints, either due to irrelevance (for example: the outcome “schooling” at the timepoint “during hospitalization”) or because no suitable tool was available. This resulted in specific tool packages for each timepoint. The tool packages consisted of two parts: a parent-reported part that can either be completed in advance or following the appointment and a clinician-administered part that is completed during clinical check-ups. For some outcomes (e.g., breastfeeding, schooling, pulmonary function), there were neither standard nor practical tools available. To assess these outcomes, specific questions were devised. In Figure 3a/b/c, the outcome wheel shows the specific tools for the outcomes evaluated for each timepoint.

Table 2.

Included tools in the neo standard set

Clinical measures	Outcomes covered
Survival: Yes/no?	Survival
If yes: cause of death?
Discharge on oxygen?	Pulmonary function
Use of bronchodilators/steroids?
Readmission due to pulmonary problems?
Breastfeeding? Formula? Fortification?	Feeding, nutrition
Number of readmissions	Readmission
Hearing screening test	Hearing
Kay picture test	Vision
WHO growth charts early years	Growth
Current education of the child?	Schooling
Parent-reported outcome measures	Outcomes covered
Parental stressor scale: NICU (PSS: NICU)	Impact on family
Parent report of children's abilities-revised (PARCA-R)	Cognition, neurodevelopment, communication, motor function
Strength and difficulties questionnaire 2–4 years (SDQ)	Behavior, social functioning, relationship with others, anxiety, depression, impact on family
Modified checklist for autism in toddlers-revised version (M-CHAT R/T)	Disability
CDC milestone checklist	Communication, social functioning, relationships with others, cognition, motor function, behavior
TNO-AZL questionnaire for preschool children's health-related quality of life (TAPQOL)	Sleep, feeding, pulmonary function, health-related quality of life, pain, motor function, relationship with others, social functioning

Fig. 3.

a Outcomes and tools during hospitalization. Only the outcomes in the green spokes are measured during hospitalization. The specific tools to measure these outcomes as indicated by the numbers next to the outcomes are depicted in bold in the box. PARCA-R: parent report of children's abilities-revised, SDQ: strengths and difficulties questionnaire, M-CHAT-R: modified checklist for autism in toddlers-revised, TAPQOL: TNO-AZL preschool children quality of life questionnaire, WHO growth charts: World Health Organization growth chart, CDC-milestone checklist: Center for Disease Control and Prevention milestone checklist, PSS; NICU: parental stressor scale: neonatal intensive care unit, Kay picture test: vision test. b Outcomes and tools at 2 years of age. Only the outcomes in the green spokes are measured at 2 years of age. The specific tools to measure these outcomes as indicated by the numbers next to the outcomes are depicted in bold in the box. PARCA-R: parent report of children's abilities −revised, SDQ: strengths and difficulties questionnaire, M-CHAT-R: modified checklist for autism in toddlers-revised, TAPQOL: TNO-AZL preschool children quality of life questionnaire, WHO growth charts: World Health Organization growth chart, CDC-milestone checklist: Center for Disease Control and Prevention milestone checklist, PSS; NICU: parental stressor scale: neonatal intensive care unit, Kay picture test: vision test. c Outcomes and tools at 5 years of age. Only the outcomes in the green spokes are measured at 5 years of age. The specific tools to measure these outcomes as indicated by the numbers next to the outcomes are depicted in bold in the box. PARCA-R: parent report of children's abilities-revised, SDQ: strengths and difficulties questionnaire, M-CHAT-R: modified checklist for autism in toddlers-revised, TAPQOL: TNO-AZL preschool children quality of life questionnaire, WHO growth charts: World Health Organization growth chart, CDC-milestone checklist: Center for Disease Control and Prevention milestone checklist, PSS; NICU: parental stressor scale: neonatal intensive care unit, Kay picture test: vision test.

Case-Mix Variables

To allow for risk adjustment and comparison across cultures and health systems, a set of 26 case-mix variables were voted in by the WG. All case mix variables and definitions are presented in Table 3.

Table 3.

Case-mix variables

Category and case-mix variable	Variable definition	Data source	Agreement, %
Demographic factors
Multiples	Indicate if the child is a multiple	P/A	100
Parent education level	Highest level of education achieved by either parent	P	100
Sex of child	The patient's sex at birth	P/A	94
Booking status inborn or outborn	The setting where the child was born	C/A	94
Country of birth	The country the child was born in	P/A	94
Number of children	Number of children of the parent	P	78
Parental relationship status	Indicate the parent's current relationship status	P	72
Parent support	If sociopsychological support is available to the parent and if the parent has used this support	P	72
Lifestyle factors
Maternal drug use	History of drug use by the mother during pregnancy	P/C	84
Maternal body Mass index	Body mass index of the mother during the first trimester pregnancy	C	78
Clinical factors
Gestational age	Gestational age of the child	C	94
Postpartum parental distress or depression	Diagnosis of postpartum depression, perinatal depression, or PTSD	C	94
Birth weight	Weight of the child at birth	C	94
Respiratory distress syndrome	Diagnosis of respiratory distress syndrome	C	89
Mode of delivery	Vaginal, instrumental, or cesarean delivery	C	89
Complications during delivery	Shoulder dystocia or asphyxia during delivery	C	89
Hypoxic-ischemic encephalopathy	Diagnosis of hypoxic-ischemic encephalopathy and its severity	C	83
Complications during pregnancy	Gestational diabetes, hypertension, or pre-eclampsia during pregnancy		78
Nosocomial infections following birth	Number of nosocomial infections during initial hospitalization	C	78
Previous history of neonatal	Has there been a previous history of neonatal death for either parent?	P/A	78
Maternal diabetes	Diagnosis of maternal diabetes	C	72
Treatment-related factors
Ventilation	Use of respiratory support	C	89
Neonatal resuscitation	If the mother received antenatal steroids	C	83
Surgical exposure	If the child has required surgical intervention	C	83
Parenteral nutrition	If the child required parenteral nutrition and its duration	C	72
Neonatal resuscitation	If the child required chest compressions	C	72

P, parent-reported outcome measure (PROM); A, administrative measure; C, clinical measure.

Professional Open Review Survey and Patient Validation Survey

Professionals from 15 different countries (n = 49) participated in the open review survey. 57% were physicians, 18% nurses, 2% healthcare administrators, 14% researchers, and 9% others. The majority (94%) agreed with the scope and the timeline proposed for the set, and over 88% agreed with the outcomes and case-mix factors recommended by the WG. The question on schooling at 5 years of age was added based on the suggestion from the survey results.

Parents from four different countries (n = 50, Mexico, South Africa, the United Kingdom, and the USA) participated in the patient validation survey. 92% of the respondents agreed that all important outcomes were covered in the set and that no critical concepts were missing. Every single outcome that was chosen by the WG was deemed “most important” by more than 70% of the parents.

Discussion

This Preterm and Hospitalized Newborn Health (NEO) standard set defines 21 meaningful outcomes covering three domains: mental, social, and physical functioning. These are based on the expert recommendation of an international WG consisting of patient representatives and HCPs.

The outcomes that matter most to patients and their families focus on independence, quality of life, social integration, and the impact on family. Yet, they often continue to remain secondary to conventional research initiatives [11, 12]. In this NEO standard set, besides the commonly reported outcomes concerning physical functioning, more than 50% focus on mental and social functioning.

The loss of follow-ups and reduced adherence to check-ups is a common occurrence in routine clinical care. It is important to ensure that the patient representative's voice is at the center of defining outcomes that matter most to them, and it promotes ownership and built-in accountability to participate in outcomes measurement.

Besides intrinsic motivation, creating a comprehensive set was of great importance. Time consuming appointments and traveling long distances for follow-up appointments may decrease participation [18]. In order to provide a set that can be used across differently resourced settings, it is paramount to recommend tools that are available free of charge (or for a minimal fee). Furthermore, by using PROMs that can be performed by parents at home, it reduces time and costly travel to medical facilities.

There are several guidelines on the follow-up of neonates at risk [6, 7]. These guidelines often recommend tools that require some training by HCPs in order to reliably perform testing. The clinical measures recommended in the NEO standard set can be assessed by any HCP without special training. Unlike costly tools like the Bayley screening test, patient-reported outcome measures like the PARCA-R or the TAPQOL can be easily performed by parents at home. Another advantage of assessing children at home by their parents (an environment that is familiar to them) is that there will be a more reliable outcome assessment, unlike assessment in a clinical setting.

Although this NEO standard set covers the first 5 years after birth, the consequences of preterm birth or hospitalization in the newborn period may extend beyond this period. Neurodevelopmental delay may become more apparent over the course of time [19]. Similarly, learning or sleeping disorders are likely to be detected in later childhood [20, 21].

The NEO standard set aims to ensure a smoother transition of care into general pediatric care, which is covered by ICHOM's Overall Pediatric Health (OPH) standard set. The OPH standard set covers outcomes from birth to 24 years of age, irrespective of the medical condition. After 5 years of age, children initially followed by this outcomes set can transition to the OPH set to measure the relevant outcomes into adolescence [22]. Furthermore, there are data that shows that the consequences of premature birth can extend far into adulthood, like predisposition to metabolic syndrome, leading to earlier an onset of cardiovascular disease, or reduced pulmonary capacity, leading to limited respiratory capacity compared to healthy-born peers. This underlines the importance of long-term follow-up of these infants even beyond adolescence [23].

Without feasible tools, the set's relevance for clinical practice would be at stake. Selecting appropriate tools proved difficult. For example, a satisfactory method of assessing “pulmonary function” that could be feasibly applied across healthcare settings was not available. This was solved by using administrative questions evaluating discharge on oxygen or readmission due to pulmonary issues, as these were the surrogate markers for the consequences of pulmonary function on daily life. “Schooling” or being able to attend any form of education is of major importance for cognitive and social development [24, 25]. Therefore, not being able to attend any form of education may give HCPs an idea about neurodevelopment and cognition [26]. By assessing whether a child can participate in any form of educational activity, the “schooling” question can be answered independently of global variation in schooling systems.

Cognition was deemed to be very important by both stakeholders and professionals; however, measuring IQ with commercially available tools is often expensive and complex. The NEO standard set recommends the PARCA-R and the CDC-milestone checklist as screening alternatives, as they are both freely available and their results indicate whether a more thorough assessment is required.

Strengths and Limitations

This study has several strengths, as it allows the ability to compare quality of care among health-care institutions, regions, or different countries by recommending a standardized collection of outcome data. WG members representing different parts of the globe (six continents) contributed to important insider knowledge about the local infrastructure and health-care processes. This ensured that the set could be implemented across rural and urban regions globally. Furthermore, the WG members represented eight different professions, and patient representatives originated from four different continents. An extensive review of the literature on this topic was performed and it was followed by a thorough discussion by the whole WG. In order to validate the set, a global survey among professionals (15 countries) and parents (four countries) ensured the clinical relevance and completeness of the set.

This study also has some limitations, as although six continents were represented by members of the WG, and the survey was performed by professionals and parents from various different countries, they are not representative of the global pediatric population. The countries represented by the WG cover around 12% of the global pediatric population below the age of five. Despite our great effort, we were not able to acquire WG members from China, Russia, or the eastern/southern African continent, which cover around 25% of the world's pediatric population below the age of five [27]. Another limitation of this study is that non-English literature was excluded from the literature search. Furthermore, WG members mainly originated from countries, apart from India, with Christianity as predominant religious background and countries with Islamic population are underrepresented.

Conclusion

The NEO standard set provides parents and HCPs with a core set of meaningful outcomes for neonates at risk. It offers clinical measures or easy-access tools and recommends three timepoints to evaluate outcomes. It is comprehensive and focuses on PROMs enabling implementation in various settings; therefore, it does not depend on available financial resources or existing follow-up infrastructure. The transparent and consensus-driven development process by an international WG ensures global relevance and, using this set, allows comparison of outcomes across different settings and countries.

The next steps are the implementation of the set in the clinical workflow and closing the loop by getting the data back to parents and HCPs. This will require making the tools for all timepoints easily available in all areas. Thus, a collaborative effort will be necessary to implement the outcome set into easy-to-use computer and smartphone applications. With this, the set can be formed as the starting point for long-lasting quality improvement for neonates at risk around the globe.

Statement of Ethics

This study is exempt from ethics committee approval since it only uses previously published literature and does not have any collected patient data.

Conflict of Interest Statement

D.W. is the cocreator and author for the PARCA-R assessment and validation. S.M. is supported by an EFCNI sponsorship. I.B.G receives an Abbvie-In kind, a support to make an awareness campaign on prematurity and RSV. All other authors have no conflicts of interests to declare.

Funding Sources

A.M. is supported by the Brazilian National Research Council. J.C. is supported by the Australian Medical Research Future Fund Career Development Fellowship (#1141354). E.S. and J.H. received a research fellowship funding from ICHOM.

Author Contributions

E.S. and J.H., performed the literature search and analysis, prepared materials for the consensus process, drafted the manuscript, and finalized it in collaboration with all the authors. J.R. and N.S. performed analysis and prepared materials for the consensus process. A.F., as WG Chair led the WG by promoting the vision for the project and guiding the consensus building process. The following WG members participated in at least 50% of the WG calls and surveys: C.N., J.B., J.C., M.D., H.K., S.M., A.M., A.Ma., R.M., K.M.-K., E.V., J.W., D.W., and J.Z.

Data Availability Statement

The data that support the findings of this study are extracted from previously published literature. Further inquiries can be directed to the corresponding author.

Supplementary Material

Supplementary data

Supplementary data

Funding Statement

A.M. is supported by the Brazilian National Research Council. J.C. is supported by the Australian Medical Research Future Fund Career Development Fellowship (#1141354). E.S. and J.H. received a research fellowship funding from ICHOM.