Face and Content Validity of Variables Associated with the Difficult-to-Sedate Child in the Pediatric Intensive Care Unit: A Survey of Pediatric Critical Care Clinicians

Abstract

Background

Clinicians recognize that some critically ill children are difficult-to-sedate. It may be possible to identify this clinical phenotype for sedation response using statistical modeling techniques adopted from machine learning. This requires identification of a finite number of variables to include in the statistical model.

Objective

To establish face and content validity for 17 candidate variables identified in the international literature as characteristic of the difficult-to-sedate child phenotype.

Methods

Pediatric critical care clinicians rated the relevance of 17 variables characterizing the difficult-to-sedate child using a four-point scale ranging from not (1) to highly relevant (4). Face and content validity of these variables were assessed by calculating a mean score for each item and computing an item-level content validity index. Items with a mean score >1 were rated as having adequate face validity. An item-level content validity index ≥0.70 indicated good to excellent content validity.

Setting and Participants

Web-based survey emailed to members of the Pediatric Acute Lung Injury & Sepsis Investigators Network or the Society of Critical Care Medicine Pediatric Sedation Study Group.

Results

Of 411 possible respondents, 121 useable surveys were returned for a response rate of 29%. All items had a mean score >1, indicating adequate face validity. Ten of 17 items scored an item-level content validity index ≥0.70. The highest scoring items were requiring three or more sedation classes simultaneously, daily modal sedation score indicating agitation, sedation score indicating agitation for 2 consecutive hours, receiving sedatives at a dose >90th percentile of the usual starting dose and receiving intermittent paralytic doses for sedation.

Conclusions

Computation of an item-level content validity index validated variables to include in statistical modeling of the difficult-to-sedate phenotype. The results indicate consensus among pediatric critical care clinicians that the majority of candidate variables identified through literature review are characteristic of the difficult-to-sedate child.

Introduction

Each year, more than 115,000 critically ill children receive sedation to help them tolerate intubation and mechanical ventilation.¹ A substantial number of these children do not respond as expected to appropriately dosed sedation and remain agitated for some period of time, leading to iatrogenic injury and increased stress.^2–5 These children, who remain agitated despite receiving usual doses of sedation, or are eventually adequately sedated but require much larger amounts of sedative drugs, are described by the clinical team as treatment failures, suboptimally sedated, or difficult-to-sedate.^6–8 Little is known about the reasons contributing to this phenomenon in these children, preventing early identification of the child who will be difficult-to-sedate. The child is often identified as difficult-to-sedate at the time care providers are actively administering sedative drugs, resulting in a delay in the attainment of therapeutic concentrations and the desired clinical effect.^8–10 This experience causes excessive and potentially avoidable burden on the child and family, and increases the chances that the child’s safety has been compromised, and injury may have occurred.^2–5 Developing a mechanism to identify the difficult-to-sedate child could allow for early identification, and prepare the care provider with a priori knowledge that the child may require more than the typical sedation needs. However, the first step towards the goal of early identification is consensus on the characteristics defining the difficult-to-sedate child.

Background

Many factors hamper identification of the difficult-to-sedate child. Sedation in the pediatric intensive care unit (PICU) is a complex phenomenon, impacted by multiple variables. Easily implemented, valid and reliable instruments that describe sedation levels in children have only become available and widely used in the last decade.^11,12 Patients cared for in the PICU vary widely in age and encompass enormous physiological and psychosocial differences.⁷ Although well-studied in adults, there are limited data on the metabolism and elimination of drugs commonly used for sedation in critically ill children.^13,14 Organ maturation and critical illness affect the rate at which sedation medications are distributed, metabolized and eliminated from the body.⁷ The influence of psychosocial development in response to sedation is not thoroughly described. There may be a genetic basis for the difficult-to-sedate child, due to polymorphisms in the genes that encode drug metabolizing enzymes as well as pertinent receptors.^3,15 Finally, each PICU’s individual sedation management plan dictates how and when sedation is delivered, the specific agents and doses used to provide sedation, as well as the definition of optimal levels of sedation.⁷ These factors contribute to the challenge of studying sedation in critically ill children.

Defining sedation-related clinical phenotypes in critically ill children would facilitate better clinical management of these patients while decreasing potential harm. Specifically, insight into an individual child’s response to sedation would allow the selection of personalized therapy and potentially contribute to improved clinical outcomes. Phenotype identification supports treatments geared to the needs of individual patients by considering each individual’s unique genetic, biomarker, phenotypic or psychosocial characteristics that distinguish them from other patients with similar presentations.¹⁶ High doses of sedatives and the simultaneous use of multiple sedative agents, as typically occurs in the difficult-to-sedate child, generally results in adverse effects such as hypotension, bradycardia, propofol infusion syndrome and iatrogenic withdrawal syndrome.¹⁷ Based on recent evidence that prolonged or repeated use of sedative and anesthetic drugs may negatively affect the developing brain by causing brain cell death, the United States Food and Drug Administration has required a warning be added to drug labels indicating that brain development in children three and under may be affected by exposure to these drugs. Included in this group are some of the most commonly used pediatric sedation drugs including midazolam, lorazepam, pentobarbital, ketamine and propofol.¹⁸ Identifying and providing targeted sedation strategies most effective for the difficult-to-sedate child could minimize these effects.

An operational definition of the difficult-to-sedate child clinical phenotype does not exist. In other populations, advanced statistical methods including cluster, classification and regression tree, and latent class analysis have been used to analyze large datasets and create an operational definition of specific phenotypes within a disease process such as childhood asthma, pediatric sepsis or acute respiratory distress syndrome.^19–21 These statistical methods require identification of candidate variables likely to be associated with the concept under investigation. In the case of intubated and sedated children, the difficult-to-sedate child clinical phenotype might include a combination of demographic, physiologic, genetic and developmental factors.^22–24

Using a three step process, we sought to create an operational definition of the difficult-to-sedate clinical phenotype using a large data set from the Randomized Evaluation of Sedation Titration for Respiratory Failure (RESTORE) study (clinicaltrials.gov identifier: NCT00814099).⁹ The RESTORE data set includes 2,449 children with acute respiratory failure and contains hundreds of variables and thousands of data points, requiring a thoughtful approach to identifying candidate variables. Step one in identifying candidate variables from those available in the RESTORE database was the completion of a concept analysis using the methodology described by Walker and Avant.²⁵ The concept analysis was based on a review of the international literature related to sedation in pediatric critical care. Thirty three studies (16 from Europe, 13 from the United States and Canada and 4 from South America) were reviewed and are listed in Table 1.

Table 1.

Studies included in Concept Analysis

Study	Setting/Location
Alexander, E., Carnevale, F. A., & Razack, S. (2002). Evaluation of a sedation protocol for intubated critically ill children. Intensive & Critical Care Nursing, 18, 292–301. doi: 10.1016/S0964339702000502.	PICU Canada
Ambrose, C., Sale, S., Howells, R., Bevan, C., Jenkins, I., Weir, P., … Wolf, A. (2000). Intravenous clonidine infusion in critically ill children: Dose-dependent sedative effects and cardiovascular stability. British Journal of Anaesthesia, 84, 794–796. doi: 10.1093/oxfordjournals.bja.a013594.	PICU UK
Amigoni, A., Mozzo E, Brugnaro L, Gentilomo C, Stritoni V, Michelin E, & Pettenazzo A. (2012). Assessing sedation in a pediatric intensive care unit using comfort behavioural scale and bispectral index: These tools are different. Minerva Anestesiologica, 78, 322–329.	PICU Italy
Aneja, R., Heard, A.M., Fletcher, J.E., & Heard, C.M. (2003). Sedation monitoring of children by the bispectral index in the pediatric intensive care unit. Pediatric Critical Care Medicine, 4, 60–64. doi: 10.1097/01.PCC.0000031464.58239.75.	PICU US
Arenas-Lopez, S., Riphagen, S., Tibby, S. M., Durward, A., Tomlin, S., Davies, G., & Murdoch, I. A. (2004). Use of oral clonidine for sedation in ventilated paediatric intensive care patients. Intensive Care Medicine, 30, 1625–1629. doi: 10.1007/s00134-004-2319-0.	PICU UK
Arnold, J.H., Truog, R.D., Rice, S.A. (1993). Prolonged administration of isoflurane to pediatric patients during mechanical ventilation. Anesthia & Analgesia, 76, 520–526.	PICU US
Berkenbosch, J.W., Fichter, C.R., & Tobias, J.D. (2002). The correlation of the bispectral index monitor with clinical sedation scores during mechanical ventilation in the pediatric intensive care unit. Anesthesia & Analgesia, 94, 506–511.	PICU US
Beytut, D., Basbakkal, Z., & Karapinar, B. (2016). Validity and reliability study of sedation diagnosis method comfort scale. Agri Dergisi, 28, 89–97. doi: 10.5505/agri.2015.24471.	PICU Turkey
Brunow de Carvalho, W., Lucas da Silva, P. S., Paulo, C. S., Fonseca, M. M., & Belli, L. A. (1999). Comparison between the comfort and hartwig sedation scales in pediatric patients undergoing mechanical lung ventilation. Sao Paulo Medical Journal = 54. 54. Revista Paulista De Medicina, 117, 192–196.	PICU Brazil
Bustos Bu, R., & Fuentes S, C. (2007). Bispectral index and COMFORT scale melation in the evaluation of sedation on paediatrics intensive care units. Revista Chilena De Pediatria, 78, 592–598. doi:dx.doi.org/10.4067/S0370-41062007000700004.	PICU Chile
Courtman, S. P., Wardurgh, A., & Petros, A. J. (2003). Comparison of the bispectral index monitor with the comfort score in assessing level of sedation of critically ill children. Intensive Care Medicine, 29, 2239–2246. Doi: 10.1007/s00134-003-1997-3.	PICU UK
Crain, N., Slonim, A., & Pollack, M. M. (2002). Assessing sedation in the pediatric intensive care unit by using BIS and the COMFORT scale. Pediatric Critical Care Medicine, 3, 11–14.	PICU US
Curley, M.A.Q,, Wypij, D., Watson, R.S., Grant, M.J.C., Asaro, L.A., Cheifetz, I.M., … the RESTORE Study Investigators and the Pediatric Acute Lung Injury and Sepsis Investigators (PALISI) Network. (2015). Protocolized sedation vs usual care in pediatric patients mechanically ventilated for acute respiratory failure: A randomized clinical trial. JAMA, 313, 379–389. doi:10.1001/jama.2014.18399.	PICU US
Curley, M.A., Harris, S.K., Fraser, K.A., Johnson, R.A., & Arnold, J.H. (2006). State behavioral scale: A sedation assessment instrument for infants and young children supported on mechanical ventilation. Pediatric Critical Care Medicine, 7, 107–114. doi: 10.1097/01.PCC.0000200955.40962.38.	PICU US
da Silva, P. S. L., Reis, M. E., de Aguiar, V. E., & Fonseca, M. C. M. (2016). Use of fentanyl and midazolam in mechanically ventilated children--does the method of infusion matter?. Journal of Critical Care, 32, 108–113. doi: 10.1016/j.jcrc.2015.12.003.	PICU Brazil
Darnell, C.M., Thompson, J., Stromberg, D., Roy, L., & Sheeran, P. (2008). Effect of low-dose naloxone infusion on fentanyl requirements in critically ill children. Pediatrics, 121, e1363–71. doi: 10.1542/peds.2007-1468.	PICU US
de Wildt, S. N., de Hoog, M., Vinks, A. A., Joosten, K. F. M., van Dijk, M., & van den Anker, J. N. (2005). Pharmacodynamics of midazolam in pediatric intensive care patients. Therapeutic Drug Monitoring, 27, 98–102.	PICU The Netherlands
Dreyfus, L., Javouhey, E., Denis, A., Touzet, S., & Bordet, F. (2017). Implementation and evaluation of a paediatric nurse-driven sedation protocol in a paediatric intensive care unit. Annals of Intensive Care, 7, 36. doi:10.1186/s13613-017-0256-7.	PICU France
Gaillard-Le Roux, B. Liet, J., Bourgoin, P., Legrand, A., Roze, J., & Joram, N. (2017). Implementation of a nurse-driven sedation protocol in a picu decreases daily doses of midazolam. Pediatric Critical Care Medicine. 18:e9–e17. doi: 10.1097/PCC.0000000000000998.	PICU France
Ista, E., de Hoog, M., Tibboel, D., & van Dijk, M. (2009). Implementation of standard sedation management in paediatric intensive care: Effective and feasible? Journal of Clinical Nursing, 18, 2511–2520. doi: 10.1111/j.1365-2702.2009.02836.x.	PICU The Netherlands
Ista, E., van Dijk, M., Tibboel, D., & de Hoog, M. (2005). Assessment of sedation levels in pediatric intensive care patients can be improved by using the COMFORT “behavior” scale. Pediatric Critical Care Medicine, 6, 58–63. doi: 10.1097/01.PCC.0000149318.40279.1A.	PICU The Netherlands
Marx, C.M., Smith, P.G., Lowrie, L.H., Hamlett, K.W., Ambuel, B., Yamashita, T.S., & Blumer, JL. (1994). Optimal sedation of mechanically ventilated pediatric critical care patients. Critical Care Medicine, 22, 163–170.	PICU US
Neunhoeffer, F., Kumpf, M., Renk, H., Hanelt, M., Berneck, N., Bosk, A., … Hofbeck, M. (2015). Nurse-driven pediatric analgesia and sedation protocol reduces withdrawal symptoms in critically ill medical pediatric patients. Paediatric Anaesthesia, 25, 786–794. doi: 10.1111/pan.12649.	PICU Germany
Parkinson, L., Hughes, J., Gill, A., Billingham, I., Ratcliffe, J., & Choonara, I. (1997). A randomized controlled trial of sedation in the critically ill. Paediatric Anaesthesia, 7, 405–410. doi: 10.1046/j.1460-9592.1997.d01-109.x.	PICU UK
Playfor SD, Thomas DA, Choonara I, & Jarvis A. (2000). Quality of sedation during mechanical ventilation. Paediatric Anaesthesia, 10, 195–199.	PICU UK
Reed, M. D., Yamashita, T. S., Marx, C. M., Myers, C. M., & Blumer, J. L. (1996). A pharmacokinetically based propofol dosing strategy for sedation of the critically ill, mechanically ventilated pediatric patient. Critical Care Medicine, 24, 1473–1481.	PICU US
Rosen, D. A., & Rosen, K. R. (1991). Midazolam for sedation in the paediatric intensive care unit. Intensive Care Medicine, 17(Suppl 1), S15–9.	PICU US
Silva, C. da C., Alves, M.M., El Halal, M.G., Pinheiro Sdos, S., & Carvalho, P.R. (2013). A comparison of gradual sedation levels using the comfort-B scale and bispectral index in children on mechanical ventilation in the pediatric intensive care unit. Revista Brasileira De Terapia Intensiva, 25, 306–311. doi: 10.5935/0103-507X.20130052.	PICU Brazil
Tobias, J.D., & Berkenbosch, J.W. (2004). Sedation during mechanical ventilation in infants and children: Dexmedetomidine versus midazolam. Southern Medical Journal, 97, 451–455.	PICU US
Triltsch, A.E., Nestmann, G., Orawa, H., Moshirzadeh, M., Sander, M., Grosse, J, … Spies, C.D. (2005). Bispectral index versus COMFORT score to determine the level of sedation in paediatric intensive care unit patients: A prospective study. Critical Care, 9(1), R9–17. doi: 10.1186/cc2977.	PICU Germany
Twite, M. D., Zuk, J., Gralla, J., & Friesen, R. H. (2005). Correlation of the bispectral index monitor with the COMFORT scale in the pediatric intensive care unit. Pediatric Critical Care Medicine, 6, 648–653. doi: 10.1097/01.PCC.0000185482.76715.D2.	PICU US
Valkenburg, A.J., Boerlage, A.A., Ista, E., Duivenvoorden, H.J., Tibboel, D., & van Dijk, M. (2011). The COMFORT-behavior scale is useful to assess pain and distress in 0- to 3-year-old children with Down syndrome. Pain, 152, 2059–2064. doi: 10.1016/j.pain.2011.05.001.	PICU The Netherlands
Wolf, A., McKay, A., Spowart, C., Granville, H., Boland, A., Petrou, S., … Gamble, C. (2014). Prospective multicentre randomised, doubleblind, equivalence study comparing clonidine and midazolam as intravenous sedative agents in critically ill children: The SLEEPS (safety profiLe, efficacy and equivalence in paediatric intensive care sedation) study. Health Technology Assessment, 18(71), 1–212. doi: 10.3310/hta18710.	PICUs UK (10)

The second step in creating our operational definition, described in this article, involved assessing face and content validity of the candidate variables identified in the concept analysis. We did this in order to substantiate their appropriateness and ensure all possible candidate variables were included in step three, a planned latent class analysis. Although face and content validity are generally used in instrument assessment we used them here to obtain expert opinion on which of the variables identified through the literature review and concept analysis were consistent characteristics of the difficult-to-sedate child. Face validity assesses whether an instrument seems to measure what it purports to measure. It assesses the relevance of an item to a construct in the opinion of experts.²⁶ Content validity is generally used to assess whether the content of an instrument is inclusive and representative of the domain of interest; i.e., do the items completely measure the domain.^27–29 Polit and Beck²⁹ note that content validity assesses if the items in the tool, when considered as a group, provide a reasonably complete operational definition of the construct being measured. Although not intended to be a formal instrument for repeated use, our survey was constructed to include what we had identified as the characteristics of the difficult-to-sedate child phenotype and used to seek expert opinion as to their relevance and completeness. Here we report on the face and content validity of candidate variables potentially characteristic of the difficult-to-sedate phenotype in children based on our survey of expert pediatric critical care clinicians.

Methods

Design and Data Collection

This study consisted of a web-based survey sent to a purposive sample of experts, practicing pediatric critical care providers, and is described here using the Checklist for Reporting Results of Internet E-Surveys (CHERRIES).³⁰ The survey link was sent via e-mail to all members of the Pediatric Acute Lung Injury & Sepsis Investigators (PALISI) network and to all members of the Society of Critical Care Medicine (SCCM) Pediatric Sedation Study Group. These groups were chosen because members are practicing critical care clinicians with extensive experience in pediatric critical care and sedation. PALISI members are clinical researchers from PICUs across North America who collaborate to conduct multi-center research studies concerning pediatric critical illness, with a focus on interventions and outcomes.³¹ Members of the SCCM Pediatric Sedation Study Group are critical care clinicians from the United States with a strong interest in pediatric sedation and knowledge of best practices. The group’s primary charge is to develop guidelines related to pediatric sedation. The Institutional Review Board of the University of Pennsylvania reviewed the study and determined it to be exempt from full board review. No personal information was collected and data was stored on a password-protected drive, to which only the investigators had access. As described above, we developed the list of candidate variables included in the survey through a literature review and concept analysis.

A pediatric critical care nurse scientist and a pediatric intensivist reviewed an initial draft of the survey for clarity and completeness. Prior to deployment, the research team tested the technical functionality of the survey, which used the Qualtrics (Washington, DC, USA) survey platform. In order to avoid coercion and ensure anonymity, the PALISI Network Coordinator and SCCM Quality and Guidelines Specialist forwarded an email containing an introduction, instructions and the survey link (Appendix A) to their membership. A unique survey link was set up for each group in order to better describe participants. The survey link was sent to 389 PALISI members, representing 78 centers on April 6, 2015, with reminder emails sent one and two weeks later. SCCM task force members (24 members representing 14 centers) received the initial email on April 22, 2015, with a reminder sent one week later. Two individuals were members of both groups and received both sets of emails. The survey was closed to responses on May 8, 2015.

The survey (Appendix B) was a voluntary, self-administered web-based survey consisting of five screens in total, including an introductory page, a page displaying questions concerning 17 candidate variables and an “Other (please list)” free-text question, a respondent demographics page, a page with a single free-text question, and a final thank you page. The first question in the SCCM survey asked if the respondent had previously completed the survey. A “yes” response closed the survey. There were no mandatory items. To encourage initial participation, an estimate of the time required for survey completion (“a few minutes”) was included in the introductory text. To encourage continued participation once started, a progress bar at the bottom of the screen displayed the participant’s progress, along with text indicating percent completed. Forward and back buttons allowed respondents to review and change their answers prior to survey submission. The survey platform captured all responses entered, even if the full survey was not completed. To provide context, respondents were instructed to answer each question in relation to a patient’s first four days of endotracheal intubation, assuming that the patient’s pain was adequately controlled and that sedation medication doses were appropriate. Each item related to the candidate variables was scored as not (1), somewhat (2), quite (3) or highly (4) relevant in identifying the difficult-to-sedate phenotype.

Data Analysis

Data was downloaded from the survey management site to a password-protected drive as an Excel spreadsheet and was analyzed using IBM SPSS Statistics 24 (IBM, Armonk, NY, USA). Thirteen surveys (11 from PALISI and 2 from SCCM respondents) which were opened but had no data entered were deleted during data cleaning. All surveys with any data concerning characteristics of the difficult-to-sedate child were included in the analysis, even if they were incomplete. Descriptive analysis of the two respondent demographic questions consisted of calculation of frequencies and percentages. In order to determine face validity, we calculated the mean score for each item. A mean score greater than 1 (not relevant) was considered an indicator of acceptable face validity, in order to ensure that items considered by the majority of experts to have a degree of relevance to the concept were retained. We also calculated an item-level content validity index (I-CVI) for each candidate variable. The I-CVI is a way to measure interrater agreement of each item in an instrument, and to identify items that should be retained or deleted from the instrument. In order to calculate the I-CVI, the number of experts who ranked an item as quite or highly relevant is divided by the total number of experts.²⁵ Using the guidelines for good content validity identified by Polit, Beck & Owen,²⁸ a threshold of 0.70 (at least 70% of respondents rated these items as quite (3) or highly (4) relevant) was considered an indication of good to excellent content validity for the item. In order to ensure accuracy and account for missing data, we used the number of complete responses to the item as the denominator in our calculations.

Results

One hundred twenty-one clinicians, 113 (95%) physicians, 3 (2%) advanced practice nurses, 4 (3%) nurse scientists and 1 (<1%) respiratory therapist responded to the survey sent to 411 individuals for a response rate of 29%. Table 2 provides further detail about response rates and sample demographics. Of the 89 clinical sites represented by PALISI and SCCM groups, members from 61 sites (69%) responded, with a mean of 1.6 individuals per site (range of 1 to 4) completing the survey. Twelve of 17 items related to candidate variables had a 100% response rate, four had 99%, and one item had a 98% response rate. Six of 2,040 data points related to the candidate variables were missing, resulting in a missing data rate of 0.3%. All variables had a mean score >1, ranging from 1.5, midway between not and slightly relevant, to 3.5, midway between quite and highly relevant. Table 3 summarizes the I-CVI for each item. Ten of seventeen items met the threshold of 0.70. Those items include requiring three or more sedation classes simultaneously, a daily modal State Behavioral Scale (SBS) score indicating agitation (SBS +1/+2), an SBS score indicating agitation for 2 consecutive hours, receiving sedatives at a dose >90th percentile of the usual starting dose, receiving intermittent paralytic doses for sedation, suspected delirium, unplanned endotracheal extubation, unplanned removal of an invasive device, paradoxical response to sedation, and Trisomy 21. At 0.65, the I-CVI for the item previous sedation exposure did not quite meet the threshold. The six items which had a low I-CVI were all demographic or diagnostic characteristics, including not able to verbally communicate, body mass index >90th percentile, an oncologic diagnosis, moderate or severe cerebral disability, moderate or severe overall disability, and bronchiolitis.

Table 2.

Survey Response Details

	Total n (%)	PALISI n (%)	SCCM n (%)
Center Representation1	61/89 (69)2	59/78 (76)	7/14 (50)
Respondents	121/411 (29)3	112/389 (29)	9/24 (39)
Role4
Attending Physician	115 (95)	106 (95)	9 (100)
Advanced Practice Nurse	3 (3)	3 (3)	0
Nurse Scientist	2 (2)	2 (2)	0
Respiratory Therapist	1 (<1)	1 (<1)	0

PALISI, Pediatric Acute Lung Injury & Sepsis Investigators; SCCM, Society for Critical Care Medicine.

¹Nineteen of 121 respondents (16%) did not indicate organizational affiliation.

²Due to overlap in organizations represented by PALISI and SCCM members, total center representation does not equal the sum of PALISI plus SCCM center representation.

³Two potential respondents were members of both the PALISI and SCCM groups, so number of total possible respondents does not equal the sum of PALISI plus SCCM respondents.

⁴Pediatric ICU Fellow (physician-in-training), Research Assistant, and Pharmacist were other options but none participated.

Table 3.

Difficult-to-Sedate Criteria: Mean Score and Item-level Content Validity Index (I-CVI)

	Mean Score	I-CVI
	Total (n=121)	Total (n=121)	PALISI (n=112)	SCCM (n=9)
Sedation Characteristics
Requires 3 or more sedation classes simultaneously	3.51	0.93 (112)	0.92 (103)	1.00 (9)
Daily modal SBS +1/+2	3.211	0.82 (98)1	0.83 (92)2	0.75 (6)3
SBS +1/+2 for 2 consecutive hours	3.094	0.79 (95)4	0.80 (89)	0.75 (6)3
Doses >90th percentile of usual starting dose	3.244	0.78 (93)4	0.78 (87)2	0.67 (6)
Intermittent paralytic doses for sedation	3.13	0.74 (90)	0.74 (83)	0.78 (7)
Sedation-related Events
Suspected delirium	3.15	0.79 (95)	0.78 (87)	0.89 (8)
Unplanned endotracheal extubation	3.13	0.72 (87)	0.71 (79)	0.89 (8)
Unplanned removal of an invasive device	3.13	0.71 (86)	0.71 (79)	0.78 (7)
Paradoxical response to sedation	2.94	0.70 (85)	0.71 (79)	0.67 (6)
Demographic/Diagnostic Characteristics
Trisomy 21	2.98	0.71 (86)	0.71 (79)	0.78 (7)
Previous sedation exposure	2.82	0.65 (79)	0.65 (73)	0.67 (6)
Not able to verbally communicate	2.36	0.43 (52)	0.41 (46)	0.67 (6)
>90th percentile for BMI	1.96	0.24 (29)	0.23 (26)	0.33 (3)
Oncologic diagnosis	1.914	0.23 (28)4	0.23 (26)2	0.22 (2)
Moderate or severe cerebral disability	2.05	0.22 (27)	0.22 (25)	0.22 (2)
Moderate or severe overall disability	1.93	0.17 (20)	0.16 (18)	0.22 (2)
Bronchiolitis	1.554	0.09 (11)4	0.10 (11)2	0

PALISI, Pediatric Acute Lung Injury & Sepsis Investigators; SCCM, Society for Critical Care Medicine; SBS, State Behavioral Scale; BMI, Body mass index.

Data presented as mean or I-CVI (n of respondents who ranked item as quite or highly relevant).

¹Total n=119.

²Total n=111.

³Total n=8.

⁴Total n=120.

Responses from the PALISI and SCCM groups were similar. This would be expected as the members of both groups are practicing clinicians with experience in pediatric sedation, and the SCCM group was added to increase the pool of experts. Two items which met the I-CVI threshold in the PALISI group were just under 0.70 in the SCCM group (both 0.67), paradoxical response to sedation and sedation doses >90^th percentile of the usual starting dose. The highest-rated item for both groups was requiring three or more sedation classes simultaneously. The SCCM group ranked suspected delirium and unplanned endotracheal extubation second and third. The PALISI group ranked daily modal SBS indicative of agitation and SBS indicative of agitation for two consecutive hours second and third. The results for the six items which clearly did not meet the I-CVI threshold were ranked in the same order by both groups, and each of these items had an I-CVI <0.50.

Several respondents identified characteristics not listed. Table 4 summarizes the 17 responses provided when the “Other (please list)” option was selected for the question “Typically has the following demographics/diagnoses/characteristics”. Young age was listed as a characteristic by 5 respondents, the remaining characteristics were identified by single respondents. Table 4 also summarizes the 61 free-text responses listing other criteria characterizing the difficult-to-sedate child. Twelve respondents identified age ≤4 years, 8 identified multiple drugs/bolus doses, 5 identified medical diagnosis, sleep/day-night cycling issues or psychiatric diagnosis, and anxious parents and rapid change in sedation level were each identified by 4 respondents.

Table 4.

Summary of Free-Text Responses

	n
“Other” demographics/diagnoses/characteristics that the difficult-to-sedate child typically has (n=17)
Infant/toddler	5
Lengthy PICU stay	1
Prior history of delirium	1
Airway repair	1
Intoxicated	1
Parents’ expectations	1
>5 days of sedation	1
Autism Spectrum Disorder	1
ECMO/ECLS or CRRT	1
ADHD, anxiety disorder, other psychiatric diagnosis	1
Transplant recipient	1
Multi-organ dysfunction	1
Prematurity	1
Other criteria that characterize the difficult-to-sedate child phenotype (n=61)
Age ≤4 years	12
Multiple drugs/bolus doses	8
Diagnosis	5
Psychiatric diagnosis (e.g., anxiety, autism, ADHD)	5
Sleep/day-night cycling issues	5
Anxious parents	4
Rapid change in sedation level	4
Nursing factors (e.g., experience, nurse/patient ratio)	3
Patient instability limits sedation doses	3
Activity limited by technology or instability	2
Adolescent	2
History of negative sedation experience	2
Other	6

PICU, pediatric intensive care unit; ECMO, extracorporeal membrane oxygenation; ECLS, extracorporeal life support; CRRT, continuous renal replacement therapy; ADHD, attention-deficit/hyperactivity disorder.

Discussion

Although the international pediatric critical care literature has identified that there is a population of difficult-to-sedate children, there is not consensus on an operational definition. Creating this operational definition will facilitate research needed to help critical care clinicians understand how best to care for this challenging population. This study assessed face and content validity of candidate characteristics, derived from a literature review and concept analysis, to be used in constructing an operational definition of the difficult-to-sedate child phenotype. The majority of items met the threshold we set for good to excellent content validity. The items that did not were all related to demographic or diagnostic characteristics, and the mean scores for these items were in the somewhat relevant range. The results support including all candidate variables evaluated in this survey when developing the model in the next phase of our project, a latent class analysis, as all variables had a mean score >1, indicating some degree of relevance to the concept.

Because content validity also considers whether all important elements of a domain are represented,^28,29 we were particularly interested in the number of additional characteristics identified in the free-text responses. A few characteristics were consistently identified, including young age, sleep or day/night cycling issues, requiring multiple bolus doses, a psychiatric diagnosis or parental anxiety. Although not all of these variables were measured in the RESTORE data set, those that were measured such as age, received medication to facilitate sleep, received medication to treat delirium, received multiple bolus doses and medical diagnosis will be added to the list of candidate variables to be evaluated in the next phase of this project.

In general, there was remarkable consistency between the PALISI and SCCM responders, despite the small number of respondents and small population of the SCCM group. Aside from organizational affiliation, there was minimal missing data, which further supports the consistency of the findings. No single center was over-represented in the sample, so it is unlikely that responses were skewed by regional differences such as differing patient populations or local sedation practices.

As with any survey, several factors may have introduced bias. The survey was voluntary and participants self-selected, so the results may represent the viewpoint of clinicians who have a specific point of view related to this topic. The sample population was drawn from a research network and an expert sedation workgroup in North America, and may not be representative of the international PICU clinician population in general. It is also possible that a respondent from either of the groups, who only received one invitation to complete the survey, may have taken the survey multiple times or that the two individuals who were members of both groups and received two invitations may have taken the survey more than once. We collected minimal respondent demographic data and respondents were not assigned any type of identifier, so there was no way to identify multiple surveys from a single individual. Because the survey link was sent via email, it is also possible that the link may have been provided to an individual not included in the original sample frame. We attempted to prevent this by including a request that the survey link not be forwarded in the email sent to the SCCM group, but did not include this in the PALISI email request. Although no respondent listed an organization not included in the PALISI or SCCM lists provided by those organizations, 16% of respondents did not identify their organization. Finally, 95% of the respondents were physicians. Nurses, who are consistently at the bedside, may have a different perception of the characteristics of the difficult-to-sedate child. It would be interesting to solicit the expert opinion of this group of providers, to see if any additional characteristics are identified.

Conclusions

This survey asked practicing clinicians to assess whether the items identified through a theoretical concept analysis agreed with their practice experience. The results of this survey indicate consensus among expert PICU clinicians, primarily physicians, that the items included in this survey are consistent characteristics exhibited by the child who is difficult-to-sedate. They will be used in phase three of this project to create a statistical model of the difficult-to-sedate child phenotype using latent class analysis. Additional characteristics identified by the expert panel will also be added to the list of candidate variables to be included in the model. Developing a mechanism to prospectively identify the difficult-to-sedate child would allow sedation tailored to the individual child, avoiding the burden placed on the child and family and decreasing the potential for injury.

Appendix A. Text of PALISI and SCCM Survey Participation Request Emails

Dear PALISI Network Colleagues

We are interested in establishing face validity for criteria that will be used in a study identifying the “difficult to sedate child” phenotype, and would appreciate your expert opinion. Please take a few minutes to answer 8 questions on our Qualtrics survey https://upenn.co1.qualtrics.com/XXXXXXXX

Data are encrypted and results will be reported in aggregate. You will have an opportunity to add a comment at the end of the survey. Thank you for taking the time to review and respond.

Dear SCCM Pediatric Sedation Study Group Colleagues

We are interested in establishing face validity for criteria that will be used in a study identifying the “difficult to sedate child” phenotype, and would appreciate your expert opinion. Please take a few minutes to answer the following 8 questions. NOTE: If you have previously taken this survey, distributed to you as a PALISI member, thank you for your participation. We request that you please answer question 1.

To start the survey please clink on this link: https://upenn.co1.qualtrics.com/XXXX

Please do not forward to your colleagues, as we would like to know your opinion as a member of the SCCM Task Force.

Data are encrypted and results will be reported in aggregate. You will have an opportunity to add a comment at the end of the survey. Thank you for taking the time to review and respond.

Appendix B. Difficult to Sedate Survey (SCCM Version)

Have you completed this survey as a PALISI member?

• ❍1- Yes
• ❍2- No

  NOTE: If “Yes” was selected, the survey skipped all remaining questions and the text “Thank you for your previous participation” was displayed

The “difficult to sedate child”... (assuming pain is adequately controlled and sedation medication doses are appropriate)

1. Exhibits a State Behavioral Scale (SBS) score of +1 (restless and difficult to calm) or +2 (agitated) for two consecutive hours

   • ❍1- Not Relevant
   • ❍2- Somewhat Relevant
   • ❍3- Quite Relevant
   • ❍4- Highly Relevant

2. Exhibits a consistent pattern of agitation, demonstrated by a daily modal (most frequently occurring) SBS score of +1 (restless and difficult to calm) or +2 (agitated)

   • ❍1- Not Relevant
   • ❍2- Somewhat Relevant
   • ❍3- Quite Relevant
   • ❍4- Highly Relevant

3. Requires intermittent paralytic doses for sedation management

   • ❍1- Not Relevant
   • ❍2- Somewhat Relevant
   • ❍3- Quite Relevant
   • ❍4- Highly Relevant

4. Requires sedation medication doses above the 90th percentile of usual starting doses (e.g. >0.2 mg/kg/hour for morphine or midazolam)

   • ❍1- Not Relevant
   • ❍2- Somewhat Relevant
   • ❍3- Quite Relevant
   • ❍4- Highly Relevant

5. Requires three or more sedative classes simultaneously to achieve target sedation

   • ❍1- Not Relevant
   • ❍2- Somewhat Relevant
   • ❍3- Quite Relevant
   • ❍4- Highly Relevant

6. Experiences sedation-related events that include the following:

   	1- Not Relevant	2- Somewhat Relevant	3- Quite Relevant	4- Highly Relevant
   Unplanned endotracheal extubation Unplanned removal of any invasive device Reports of a paradoxical response to sedation Suspected of having delirium

7. Typically has the following demographics/diagnoses/characteristics:

   	1- Not Relevant	2- Somewhat Relevant	3- Quite Relevant	4- Highly Relevant
   >90th percentile for BMI History of previous sedative exposure Bronchiolitis Oncologic diagnosis Trisomy 21 Moderate or severe cerebral disability Moderate or severe overall disability NOT able to verbally communicate Other (please list):

8. Please list any other criteria that you feel characterizes the “difficult to sedate child” phenotype.

9. Please select what best describes your role:

   • ❍Attending Physician
   • ❍Pediatric ICU Fellow
   • ❍Advanced Practice Nurse
   • ❍Nurse Scientist
   • ❍Respiratory Therapist
   • ❍Research Assistant
   • ❍Pharmacist
   • ❍Other (please list) ____________________

10. Please use the drop down menu to provide the name of your organization. If your organization is not listed please select “other” (last item in the drop down box).

11. Please provide any closing thoughts that you think may be important to consider with regard to the “difficult to sedate child” phenotype.

    Thank you … The END!! Please click the Next (≫) button to submit your survey.