Association Between Child Psychiatric Emergency Room Outcomes and Dimensions of Psychopathology

Abstract

Objective:

To determine the degree to which dimensional psychopathology predicts length of stay in an emergency department (ED) and need for hospital admission among children with psychiatric complaints.

Method:

Electronic health records of children age 4-17 years who presented to the ED of a large academic medical center were analyzed using a natural language processing tool to estimate Research Domain Criteria (RDoC) symptom scores. These scores’ association with length of stay and probability of admission versus discharge to home were evaluated.

Results:

We identified 3,061 children and adolescents who presented to the ED and were evaluated by the psychiatry service between November 2008 and March 2015. Median length of stay was 7.8 hours (interquartile range 5.2-14.3 hours) and 1,696 (55.4%) were admitted to the hospital. Higher estimated RDoC arousal, cognitive, positive, and social domain scores were associated with increased length of stay in multiple regression models, adjusted for age, sex, race, private insurance, voluntary admission, and diagnostic categories. In similarly adjusted models, odds of hospital admission were increased by higher RDoC arousal and cognitive domain scores and decreased by higher negative domain scores.

Conclusions:

A natural language processing tool to characterize dimensional psychopathology identified features associated with differential outcomes in children in the psychiatric ED, most notably symptoms reflecting arousal and cognitive function. Methodologically, this in silico approach to risk stratification should facilitate precision psychiatry in children within the emergency setting.

1. Introduction

Children and adolescents with psychiatric symptoms are increasingly evaluated in emergency departments (ED).[1-3] However, relatively little is known about the clinical features that impact ED utilization, including length of stay and probability of subsequent hospitalization. Such features are important to identify, as they could serve as targets for early intervention and prevention efforts. To date, the small prior literature has focused on individual diagnoses, chief complaints, or sociodemographic features to stratify utilization.[2,4,5]

The National Institute of Mental Health (NIMH) has encouraged investigators to consider psychopathology in terms of dimensions that cut across and dissect within traditional diagnostic categories and likely bear a closer relationship to underlying neurobiology than traditional categorical diagnoses.[6,7] One proposed set of dimensions is the Research Domain Criteria (RDoC) framework, which includes five domains: negative valence, positive valence, cognition, social process, and arousal and regulation.[8] These five domains are subdivided into constructs and have been mapped across units of analysis from genes through molecules, cells, circuits, and physiology, up to behaviors, self-reports and paradigms.[9] To this end, we previously demonstrated that a validated natural language processing (NLP) tool applied to narrative clinical notes could identify the five RDoC-informed dimensions of psychopathology. [10,11] Among adult psychiatric inpatients, estimated RDoC domain scores were associated with length of stay and readmission risk - for example, higher positive and negative valence as measures of affective symptomatology both predicted shorter length of hospital stay.

In an effort to better understand the predictive validity of dimensional psychopathology among children and adolescents, this NLP-based estimation approach was applied to psychiatric ED notes for a large cohort of children and adolescents. We evaluated the extent to which the RDoC-informed transdiagnostic features capture clinically-relevant aspects of these patients beyond those captured in coded clinical data and hypothesized that both structured clinical features and RDoC-based transdiagnostic features would be associated with clinical utilization as has been observed in adult populations.

2. Methods

Based on the presence of a narrative clinical note in the electronic health record (EHR), we identified children younger than 18 years who presented to the ED of a large tertiary-care academic medical center and were seen by the continuously staffed, ED-based acute psychiatry service between November 1, 2008 and March 31, 2015. This EHR captures the comprehensive clinical care including that of triage nursing, bedside nursing, emergency medicine doctors, and psychiatrists. This was a period of stability in terms of both deployed hospital information systems and diagnostic coding as it precedes ICD10 implementation. These visits included either evaluation in a 6-bed locked emergency psychiatry unit or consultation in another area of the ED (e.g., pediatrics) by the continuously available psychiatric staff of the locked ED psychiatry unit. Coded clinical data including ICD9 codes, and narrative text clinical notes, were used to generate an i2b2 data mart (i2b2 server software version 1.6, tranSMART Foundation).[12] The study was approved by the Institutional Review Board of Partners Healthcare with a waiver of informed consent under 45 CFR 46.116.

Length of stay was defined as the difference between registration time and discharge time as recorded in the EHR. Discharge destination was also determined from EHR disposition codes. Additional coded features included age at visit, sex, race (collapsed to form a single binary variable reflecting racial/ethnic minority status, i.e., white/non-white), public versus private insurance type, and psychiatric diagnostic history.

Chief complaint was also categorized from the EHR, extracted from the visit note and manually curated into descriptive categories selected a priori and operationalized as a set of natural language processing rules for categorizing stereotyped clinical language used within the health system. Multiple matches were allowed when multiple chief complaints were provided. These included suicidal ideation/attempts, behavioral dysregulation, involuntary transfer for evaluation (Massachusetts General Law c. 123, Section 12), symptoms of mania/psychosis, and symptoms of depression/anxiety.

We applied an NLP tool previously described and validated (through clinical prediction, formal neuropsychiatric testing, and agreement with expert opinion) to score narrative notes for synonyms of each of the five RDoC domains: positive and negative valence, cognitive systems, arousal and regulatory systems, and social processes.[10] In prior application of this tool relating it to adult inpatient psychiatry length of stay, a one-unit increase in RDoC score was associated with a 1.2 day reduction in length of stay for positive domain and a 0.9 day increase in length of stay for social domain.[10] An implementation of this tool was made freely available for download and inspection by the original authors.[13] In brief, the tool works by identifying the words and phrases present in a clinical document from an expertly curated lexicon, amplified through machine learning, of words and phrases with relevance to an RDoC domain. The score is the percentage of domain-relevant words or phrases present in a given document without regard to context in which the word is used. The freely available implementation of the tool in Python version 2.7.14 is open for inspection, including the relevant term lists, and use.[14] Whereas the RDoC NLP portion-of-tokens-present algorithm is intuitive, the particular tokens driving a result are non-obvious but clinically interesting. Although interpretability of machine learning models is an active area of research, [15-17] a supplemental analysis was completed to give insight into the particular way in which the RDoC NLP phenotype is working in the context of this clinical cohort and when used for disposition prognostication. For this supplemental analysis, a random forest classifier of ED disposition (home versus inpatient admission) was trained using the raw token list for each RDoC domain.[18] The Gini coefficients (measures of importance of individual words to the correct classification of a patient’s disposition) were then log transformed and used as weights to create word clouds, one per RDoC domain, of the domain token list weighted by relevance to this particular classification task (see Supplemental Materials). This approach to interpreting the algorithmic phenotypes attempts to balance competing concerns, namely a desire to understand why a computed phenotype works as it does balanced by the reality that meaning arises from the combination of terms, rather than the individual terms per se.[10] The Gini coefficient thus provides a disposition prediction task-specific heuristic view into the phenotypes based on how frequently an individual term would correctly classify patients’ disposition. We also provide a relevant example phrase with domain-relevant token selected from the document with the highest cognitive domain score, although we emphasize that the algorithm itself is not context sensitive whereas human reading is likely to be.

Analysis was limited to each child’s first admission during the follow-up period (November 1, 2008 through March 31, 2015). Primary analysis first used multiple regression models to examine the five RDoC domains simultaneously, and then expanded to adjustment for sociodemographic and clinical features. For length of stay, models used linear regression; for hospitalization, logistic regression. As length of stay is significantly skewed, it was log transformed for analysis. All analysis used R version 3.4.3.[19]

3. Results

A total of 3,061 index evaluations of children and adolescents by psychiatry during the follow-up period were identified. In this cohort, median length of stay was 7.8 hours (interquartile range 5.2-14.3 hours). The demographic characteristics of the cohort are summarized in Table 1. Among these 3,061 visits, 1,696 (55.4%) resulted in hospital admission. The majority of children were white (60.9%) and had private insurance (58.9%). A large minority (44.5%) had a documented psychiatric diagnosis prior to presentation. In multivariable logistic regression including only RDoC domain estimates (Table 2, left), arousal, cognitive, and positive domains were associated with greater admission likelihood, whereas negative domain symptoms were associated with discharge. In multivariable logistic regression (Table 2, right), adding conventional structured demographic and coded diagnostic variables to the RDoC domain estimates, arousal, cognitive, and negative domains remained significantly associated with admission likelihood. Among the conventional structured demographic and coded diagnostic variables, children who were older, who arrived in the emergency department involuntarily, and with chief complaints of self-harm/suicidality were significantly more likely to be admitted. Conversely, those with private insurance were less likely to be admitted.

Table 1:

Sociodemographic, clinical, and natural language processing-derived features for the full cohort and stratified by emergency department disposition

	Discharged	Admitted	Total
n	1365	1696	3061
Research Domain Criteria
Arousal/Regulatory Domain (mean (SD))	10.99 (4.40)	12.21 (4.27)	11.67 (4.37)
Cognitive Domain (mean (SD))	16.25 (6.04)	19.15 (5.59)	17.86 (5.97)
Negative Domain (mean (SD))	14.40 (6.01)	15.48 (5.60)	15.00 (5.81)
Positive Domain (mean (SD))	16.37 (6.56)	18.27 (5.90)	17.42 (6.28)
Social Domain (mean (SD))	15.01 (6.84)	17.79 (6.59)	16.55 (6.84)
Sociodemographic Features
Sex (male %)	678 (49.7)	802 (47.3)	1480 (48.4)
Age (years, mean (SD))	14.94 (3.62)	15.43 (3.12)	15.21 (3.36)
Race (white %)	865 (63.4)	1000 (59.0)	1865 (60.9)
Private Insurance (%)	854 (62.6)	950 (56.0)	1804 (58.9)
Clinical Features
Has Prior Psychiatric Diagnosis (%)	611 (44.8)	750 (44.2)	1361 (44.5)
Self-harm (%)	389 (28.5)	741 (43.7)	1130 (36.9)
Behavioral Dysregulation (%)	204 (14.9)	261 (15.4)	465 (15.2)
Psychosis Symptoms (%)	33 (2.4)	84 (5.0)	117 (3.8)
Involuntary Transfer (%)	38 (2.8)	121 (7.1)	159 (5.2)
Depression Symptoms (%)	289 (21.2)	215 (12.7)	504 (16.5)

Table 2:

Multivariate models of admission, RDoC only (left) versus RDoC plus additional sociodemographic and clinical data (right)

	RDoC only			RDoC + structured data
	Odds Ratio	CI	p	Odds Ratio	CI	p
Research Domain Criteria
Arousal/Regulatory Domain	1.13	1.02 – 1.25	.017	1.18	1.07 – 1.31	.002
Cognitive Domain	1.51	1.34 – 1.71	<.001	1.53	1.35 – 1.74	<.001
Negative Domain	0.90	0.81 – 0.99	.038	0.87	0.79 – 0.97	.009
Positive Domain	1.13	1.03 – 1.24	.009	1.05	0.95 – 1.16	.345
Social Domain	1.08	0.95 – 1.21	.233	1.10	0.98 – 1.25	.116
Sociodemographic Features
Sex (male)				0.96	0.82 – 1.12	.593
Age (years)				1.04	1.02 – 1.07	<.001
Race (white)				1.00	0.85 – 1.18	.979
Private Insurance				0.77	0.66 – 0.91	.002
Clinical Features
Has Prior Psychiatric Diagnosis				0.99	0.85 – 1.16	.936
CC Self-harm				2.11	1.79 – 2.49	<.001
CC Behavioral Dysregulation				1.33	1.06 – 1.67	.015
CC Involuntary Transfer				2.83	1.93 – 4.23	<.001

RDoC = Research Domain Criteria

CI = confidence interval

CC = chief complaint

In multivariable linear regression including only RDoC domains (Table 3, left), ED length of stay was increased by higher cognitive, social, and positive domains, with cognitive domain making the largest contribution. When conventional coded variables are added to the multivariable linear regression, the arousal domain also became significantly associated with longer length of stay (Table 3, right). Among the conventional structured and coded variables, private insurance and white race were associated with shorter length of stay, and presentation involuntarily, for self-harm, or behavioral dysregulation were all associated with longer length of stay (Table 3). Figure 1 shows the Kaplan-Meier curve for time-to-ED-discharge by quartile of RDoC cognitive domain score, the one domain which was significantly associated in all four models - that is, those of both length of stay (Table 3) and disposition (Table 2) outcomes including either RDoC NLP and structured data (right side of Tables 2 and 3) or only RDoC NLP (left side of Tables 2 and 3) – as a means of illustrating strength of association. An example of the cognitive domain estimation taken from the document with the highest cognitive burden score is: “She was laughing inappropriately and not answering questions clearly … She has been more cooperative and calm since initiation of psychotropic, but continued to display some disinhibited behavior and very poor attention.”

Table 3:

Multivariate models of log(length of stay) using RDoC only (left) versus RDoC plus additional sociodemographic and clinical data (right)

	RDoC only			RDoC + structured data
	B	CI	p	B	CI	p
Research Domain Criteria
Arousal/Regulatory Domain	0.03	−0.00 – 0.07	.053	0.04	0.01 – 0.07	.017
Cognitive Domain	0.17	0.13 – 0.21	<.001	0.17	0.13 – 0.21	<.001
Negative Domain	0.03	−0.01 – 0.06	.105	0.02	−0.01 – 0.06	.155
Positive Domain	0.07	0.04 – 0.10	<.001	0.07	0.04 – 0.10	<.001
Social Domain	0.06	0.03 – 0.10	.001	0.06	0.02 – 0.10	.001
Sociodemographic Features
Sex (male)				−0.05	−0.09 – 0.00	.072
Age (years)				0.00	−0.00 – 0.01	.341
Race (white)				−0.05	−0.10 – −0.00	.049
Private Insurance				−0.20	−0.25 – −0.15	<.001
Clinical Features
Has Prior Psychiatric Diagnosis				−0.01	−0.06 – 0.03	.588
CC Self-harm				0.13	0.08 – 0.18	<.001
CC Behavioral Dysregulation				0.11	0.04 – 0.18	.002
CC Involuntary Transfer				0.20	0.09 – 0.31	<.001

RDoC = Research Domain Criteria

CI = confidence interval

CC = chief complaint

Figure 1:

Kaplan-Meier curve showing time to discharge from the emergency department stratified by quartile (color) of National Institute of Mental Health Research Domain Criteria cognitive domain scores estimated by natural language processing of clinical documentation.

4. Conclusions

Across more than 3,000 psychiatric ED visits among children and adolescents, we identified previously recognized as well as novel factors reflected in narrative notes that were associated with greater risk of hospital admission and prolonged ED stay. Specifically, in addition to chief complaints and public insurance, multiple estimated RDoC domains captured from narrative notes were significantly associated with ED outcomes, with the most prominent risk observed for the cognitive systems domain.

In light of the increase in utilization of psychiatric emergency rooms by children,[1-3] better understanding of the reasons children require hospital admission via the ED, or remain in the ED for long periods, has become critical. While prior efforts relied primarily on health claims data, the present study sought to determine whether narrative clinical notes could help identify additional features associated with risk for admission or prolonged stay. Beyond corroborating previously identified predictors, indicating construct validity, these results extend the literature by highlighting the value of dimensional traits relevant to the RDoC framework.

In particular, this study highlights the critical role played by cognitive systems among children at increased risk for admission or extended length of stay in the ED. Although we use a novel methodology, this result is consistent with existing literature reporting variation in directly assessed cognition relates to length of hospital stay among child psychiatric inpatients.[20] It also adds to the substantial literature that associates cognitive dysfunction with a range of poor functional outcomes in both children[21,22] and adults with neuropsychiatric illness.[23-25] As cognitive variation also relates to social and academic outcomes in neurotypically developing children,[26,27] these data extend the rationale for considering cognition in efforts to stratify youth at risk for a range of poor public health outcomes. The present data do not provide insight into a causal pathway between cognition and disposition or length of stay, but possible explanations may include structural barriers to admission of children with cognitive symptoms such that beds are disproportionately unavailable to this group; disposition preferences or additional evaluative complexity arising from interested third-parties such as schools or social service agencies; and decreased clinical certainty among providers evaluating patients with greater cognitive symptom burdens.

More generally, these findings provide further validation for the ability of a simple NLP tool to capture meaningful clinical features from narrative notes: these features improve the prediction of outcomes in this cohort. The RDoC NLP tool was developed using adult clinical notes across multiple hospitals, so the generalization to a child population in a different treatment setting is both notable and encouraging as an example of transfer learning from more available to less available clinical data types. Although NLP may be unfamiliar, it is not inherently complex, and the system used here could be implemented in an emergency department should adequate evidence for its utility arise. Whereas implementing many predictive models requires extensive data harmonization efforts to line up the data that exist in a clinical record with the data that a given model expects, NLP does not assume carefully structured inputs; as such, NLP may be easier to implement. That said, NLP assumes the existence of documentation and providers may document well after the fact, such that data availability, rather than harmonization, is the core challenge to implementation. Whereas the ability to utilize unharmonized pre-existing data is a strength of this approach, a challenge is the inability to readily attribute particular meaning to any given component of the phenotype. While the Supplemental Materials provide one view of individual terms contributing to these phenotypes, we note that formal approaches to this goal are a topic of active research.[28] In particular, while it is tempting to attribute meaning to the context in which phenotypic topics occur, the algorithm we apply is not sensitive to context.

Multiple additional limitations should be noted. The goal of this analysis was not to develop a predictor for clinical application, so it does not focus on metrics of discrimination or calibration; further investigation in other cohorts would be required to develop and validate such predictors. These results simply indicate the utility of capturing dimensional psychopathology in future prediction models. In addition, the scores extracted by this approach are not substitutes, or even proxies, for a more direct evaluation of the RDoC constructs which were identified, but rather aggregated estimates based on historical notes. Indeed, the results suggest that further investigation of these RDoC-related constructs with more targeted assessment batteries may be useful in refining predictions of ED outcomes in children and adolescents. As this study was limited to the NIMH-specified RDoC domains as previously operationalized in the NLP tool used, it is possible that other, wholly different, dimensions may better capture symptoms associated with utilization. It is important to note that these results presume dedicated psychiatric documentation which may not be promptly available in all settings. Further work expanding these approaches to non-psychiatric documentation is needed, although early results are encouraging.[29] Finally, this documentation was completed by a dedicated ED-based team, staffed continuously by psychiatrists; as such, while length of stay is not meaningfully impacted by time to evaluation, it still reflects a complex set of inputs (e.g., clinical uncertainty, bed availability) which cannot be distinguished. More broadly, if the aim is merely clinical prediction, rather than alternative phenotyping, we note that the sort of coded data available in most health records and categorization of chief complaints are also associated with both outcomes studied.

In addition to highlighting a potential role for cognitive systems in the stratification of youth at risk for psychiatric hospitalization, these results represent a promising first step toward development of large in silico child and adolescent cohort studies by demonstrating that clinical features can be extracted and characterized by NLP. As a complement to more standard means of assessment,[30] they may facilitate investigations of a range of longitudinal outcomes in these understudied populations. Although the transfer of phenotypes learned in adult populations to children is encouraging and the potential for clinical application is appealing, much more work is needed. The computed phenotypes used here were developed in adults and thus it is possible that better results could be obtained if an adequate dataset could be created to reapply the original phenotype derivation method to develop, and compare, child- and emergency-specific domain scores. We hope that further work in this vein will lead to translatable transdiagnostic phenotypes that can help recognize the potential of precision medicine in neuropsychiatric illness.[31]

Abbreviations:

ED

emergency department

NIMH

National Institute of Mental Health

RDoC

Research Domain Criteria

NLP

natural language processing

EHR

electronic health record