Dimensional measures of psychopathology in children and adolescents using large language models

Abstract

Background:

To enable greater use of NIMH Research Domain Criteria (RDoC) in real-world settings, we applied large language models to estimate dimensional psychopathology from narrative clinical notes.

Methods:

We conducted a cohort study using health records from individuals age 18 years or younger evaluated in the psychiatric emergency department of a large academic medical center between November 2008 and March 2015. Outcomes were hospital admission and length of emergency department stay. RDoC domains were estimated using a HIPAA-compliant large language model (gpt-4–1106-preview), and compared to a previously-validated token-based approach.

Results:

The cohort included 3,059 individuals (median age 16 (25%−75% 13–18); 1580 (52%) female, 1479 (48%) male; 105 (3.4%) identified as Asian, 329 (11%) as Black, 288 (9.4%) Hispanic, 474 (15%) as another race, and 1863 (61%) as white), of whom 1695 (55%) were admitted. Correlation between LLM-extracted RDoC scores and the token-based scores ranged from small to medium by Kendall’s Tau (0.14–0.22). In logistic regression models adjusted for sociodemographic and clinical features, admission likelihood was associated with greater scores on all domains, with the exception of sensorimotor, which was inversely associated (p<.001 for all adjusted associations). Tests for bias suggested modest but statistically significant differences in positive valence scores by race (p<.05 for Asian, Hispanic, and Black individuals).

Conclusion:

A large language model extracted estimates of 6 RDoC domains in an explainable manner, which were associated with clinical outcomes. This approach can contribute to a new generation of prediction models or biological investigations based on dimensional psychopathology.

Trial registration:

n/a

INTRODUCTION

The traditional categorical diagnostic system that emerged in DSM-3(1) has been consistently criticized for ignoring dimensional features of psychopathology, as well as the extent to which individual diagnoses overlap with each other, as well as with normal variation(2). To address these limitations, and to better approximate neurobiological systems, the NIMH Research Domain framework was developed(3). This framework posited 5, and later 6, dimensions on which neuropsychiatric systems could be mapped across scale: Negative Valence Systems, Positive Valence Systems, Cognitive Systems, Social Processes, Arousal and Regulatory Systems, and Sensorimotor Systems(4).

Substantial effort has been expended to measure functions in individual domains(2). One challenge to the more widespread application of the RDoC framework is the lack of documentation of dimensional measurements in clinical settings(5). In an effort to enable dimensional analysis of narrative clinical documentation, we previously developed and validated a simple natural language processing approach that employs lists of terms and synonyms corresponding to each domain, and measures the extent to which such terms are reflected in notes(6). We showed that this approach had utility in a range of contexts – predicting longitudinal clinical outcomes including suicide(7), correlating with postmortem neuropathology(8), and enabling the first genomewide association studies of RDoC domains(9).

The development of large language models enables far more powerful approaches to document summarization and concept extraction, with a range of applications in medicine(10). We hypothesized that these models could estimate RDoC-like dimensional measures of psychopathology at scale from electronic health records, overcoming some of the limitations of prior natural language processing strategies. This approach would enable large-scale investigation of RDoC-like phenotypes across health systems, biobanks, and other large longitudinal data sets. To test this possibility, we drew on a large cohort of children and adolescents evaluated in a psychiatric emergency room(11). We did not aim to develop clinically-deployable predictors, but rather to use clinical outcomes to examine predictive validity for the resulting scores as compared to previous methods.

METHOD AND MATERIALS

The cohort included all psychiatric ED visits for children younger than 18 years drawn from the electronic health record (EHR) of a large tertiary-care medical center between November 1, 2008 and March 31, 2015. Visits could occur in a 6-bed locked emergency psychiatry unit, or consultation by psychiatric staff in another area of the ED. For individuals with more than one visit, we used the index visit to avoid correlated observations.

Sociodemographic data were drawn from the EHR. Race and ethnicity are defined according to past US Census categories, and were collected to enable consideration of bias in subsequent scoring given the known limitations of large language models. Insurance type (private versus public) was used as a proxy for socioeconomic status as in our prior work. Additional data included ICD9 codes (as the end-date was selected to precede ICD10 implementation) and narrative clinical notes. The study was approved by the Institutional Review Board of Partners Healthcare, and subsequently Mass General-Brigham, with a waiver of informed consent under 45 CFR 46.116.

The two outcome measures were defined using EHR metadata. Length of stay was defined as the discharge date and time minus the registration date and time; it was log-transformed as in our prior work for analysis to account for skewness. Discharge destination was categorized on the basis of disposition codes, here dichotomized to hospitalization or discharge to home.

We drew on previously-curated chief complaint data manually curated by the authors(11). More than one category could be indicated; they included suicidality and behavioral dysregulation, as well as involuntary transfer for evaluation (defined by Massachusetts General Law c. 123, Section 12).

To estimate RDoC domain scores, we applied two different approaches. First, as in our prior publication, we applied an NLP tool(6) that has been validated via clinical prediction, concordance with expert opinion, and formal neurocognitive testing, and was made available for open access(12). That method uses an expert-curated lexicon of terms (‘tokens’) associated with each RDoC domain, and then counts the proportion of those terms present in any given document. It does not consider context (in a sentence, or in a note section) nor take into account negation (e.g., “Not depressed” and “Depressed” would be scored identically).

Second, we used a secure, HIPAA-compliant cloud-based instance of GPT4 (GPT-4 turbo, gpt-4–1106-preview) to score each note, yielding estimated RDoC scores. Notes were sequentially presented via the Microsoft Azure OpenAI API, using Python to permute note order and parse resulting JSON-formatted scores). As initial queries of GPT-4 indicated that its training included references encompassing RDoC (Supplemental Text A), we used the GPT-4 API to score each note with the following prompt: “You are a skilled psychiatrist scoring an emergency room clinical note in terms of how the patient symptoms over the past 24 hours reflect the 6 NIMH Research Domain Criteria (RDoC): Negative Valence Systems, Positive Valence Systems, Cognitive Systems, Social Processes, Arousal and Regulatory Systems, and Sensorimotor Systems. Remember that substance use can be reflected as Positive Valence symptom. Notes are scored on a 0–10 scale to capture the magnitude of documented symptoms relevant in a given domain. Score 0 if no symptoms are present and functioning is normal, 1–3 mild symptoms, 4–6 moderate, 7–9 severe, 10 extremely severe. Generally moderate requires treatment and severe requires hospitalization. Return only a JSON object with each score, Negative, Positive, Cognitive, Social, Arousal, and Sensorimotor.” Temperature (a measure of randomness in choosing the next word in a series) was set at 0 to yield the most deterministic results possible, and the model was re-initiated without persisting prior context for each note.

We first examined correlations between the 2 sets of RDoC scores for the 5 original domains (Sensorimotor represents a more recent addition to the RDoC framework(4)). To understand calibration of these scores beyond the anchor points provided in the prompt, we asked the model to identify examples of each anchor point for cognition, as an example domain. We also provided a single narrative note to the model and asked that it alter as little as possible to increase or decrease the cognition domain score, recording the difference between the two notes.

We then used multiple regression to examine the association between the GPT4-generated domain scores and the two outcomes – logistic regression for likelihood of hospitalization, and linear regression for log-transformed length of stay. We compared nested models (sociodemographic alone, sociodemographic plus LLM-derived RDoC, sociodemographic plus LLM-derived RDoC plus token-derived RDoC), examining whether these measures meaningfully improved model fit.

We also examined individual sociodemographic features for association with RDoC scores in multivariable linear regression models, to investigate the possibility of bias. As a follow-up, we explored the extent to which bias might be introduced by the LLM by presenting a subset of 200 notes permuted to reflect either white, Black, or Asian race, and compared scores on each domain using paired t-tests. That is, for a given note, we generated three versions, one describing the patient as white, another as Black, and a third as Asian, to see if the estimated RDoC scores differed.

All analyses used R version 4.3.2.

RESULTS

The cohort included 3,059 individuals of median age 16 years (25%−75% 13–18 years); 1580 (52%) reported gender as female, 1479 (48%) male; 105 (3.4%) identified as Asian, 329 (11%) as Black, 288 (9.4%) Hispanic, 474 (15%) as another race, and 1863 (61%) as white. Additional characteristics are summarized in Table 1. Of the full cohort, 1695 (55%) were admitted, and the remainder discharged. Median length of stay was 8 hours (25%−75% 5–14 hours).

Table 1.

Characteristics of children and adolescents evaluated in a psychiatric emergency room, by discharge status

Feature	Overall N=3,059 (a)	Admit N=1,695	Discharge N=1,364	p-value (b)
Gender				0.2
Female	1,479 (48%)	802 (47%)	677 (50%)
Male	1580 (52%)	893 (53%)	1287 (50%)
Age (Years)	16 (13, 18)	16 (14, 18)	16 (13, 18)	0.01
Race and Ethnicity				<0.001
Asian	105 (3.4%)	74 (4.4%)	31 (2.3%)
Black	329 (11%)	166 (9.8%)	163 (12%)
Hispanic	288 (9.4%)	166 (9.8%)	122 (8.9%)
Other (c)	474 (15%)	290 (17%)	184 (13%)
White	1,863 (61%)	999 (59%)	864 (63%)
Private insurance	1,802 (59%)	949 (56%)	853 (63%)	<0.001
Involuntary transport	159 (5.2%)	121 (7.1%)	38 (2.8%)	<0.001
Chief complaint
self-harm	1,130 (37%)	741 (44%)	389 (29%)	<0.001
behavioral dyscontrol	465 (15%)	261 (15%)	204 (15%)	0.7
Any prior psychiatric diagnosis	1,361 (44%)	750 (44%)	611 (45%)	0.8
Token-based RDoC estimates
RDoC negative	14.6 (11.2, 19.1)	15.7 (12.4, 19.1)	14.6 (10.1, 18.0)	<0.001
RDoC positive	16.4 (13.1, 21.3)	18.0 (14.8, 21.3)	16.4 (11.5, 19.7)	<0.001
RDoC cognitive	18 (15, 22)	18 (16, 22)	16 (13, 20)	<0.001
RDoC social	17 (11, 21)	17 (13, 21)	15 (11, 19)	<0.001
RDoC arousal-regulatory	11.4 (8.6, 14.3)	12.4 (9.5, 15.2)	10.5 (8.6, 13.3)	<0.001
LLM-based RDoC estimates
RDoC negative	7.00 (7.00, 7.00)	7.00 (7.00, 9.00)	7.00 (6.00, 7.00)	<0.001
RDoC positive	2.00 (1.00, 4.00)	2.00 (2.00, 4.00)	2.00 (1.00, 3.00)	<0.001
RDoC cognitive	4.00 (3.00, 5.00)	4.00 (3.00, 6.00)	4.00 (3.00, 5.00)	<0.001
RDoC social	6.00 (5.00, 7.00)	7.00 (6.00, 7.00)	6.00 (4.00, 6.00)	<0.001
RDoC arousal-regulatory	6.00 (5.00, 7.00)	6.00 (5.00, 7.00)	5.00 (4.00, 6.00)	<0.001
RDoC sensorimotor	2.00 (1.00, 3.00)	2.00 (1.00, 3.00)	1.00 (1.00, 2.00)	<0.001

RDoC, NIMH Research Domain Criteria score

^(a)n (%); Median (25%−75% IQR)

^(b)Pearson’s Chi-squared test; Wilcoxon rank sum test

^(c)‘Other’ refers to Pacific Islander, Native American, or Other reported by patient or parent/guardian at registration

We first correlated RDoC domain estimates using the large language model with those derived using the prior token-based method (Figure 1). All domains exhibited statistically significant (p<.001), small to medium correlations by Kendall’s Tau (between 0.14 and 0.22). Using Pearson’s correlation, medium to large correlations were observed (r=0.24–0.32). LLM RDoC domains also exhibited correlation with each other, ranging from r of 0.14 to 0.66 (Supplemental Figure 1). The newly-derived sensorimotor domain was most correlated with arousal (0.39) and cognition (0.44).

Figure 1.

Correlation between token-based RDoC scores (x axis) and those derived from a large language model (y-axis)

Token, token-based estimates; GPT, large language model-based estimates. Red line indicates linear regression best fit. Kendall’s rank correlation tau = 0.14, 0.22, 0.17, 0.21, and 0.15 for arousal, cognitive, negative, positive, and social domains, respectively; all p<.001. Pearson correlations r=0.24, 0.34, 0.30, 0.32, and 0.28, respectively; all p<.001.

We next sought to increase the transparency of LLM-based scores. The LLM-based approach also allows explanation of the predictions, as well as answering additional questions. Supplemental Text A reflects the response to a request to explain the RDoC matrix; Supplemental Text C illustrates a typical explanation of RDoC scores and a response to a request for clarification. We specifically assessed calibration by requesting that the model modify an input note, using as few changes as possible, to alter its score. Supplemental Text D illustrates only the resulting changes.

We then examined the extent to which individual LLM RDoC domains were associated with two outcomes, likelihood of hospital admission and length of emergency department stay, as a means of understanding predictive validity. In logistic regression models with or without inclusion of sociodemographic and clinical features, greater scores on all domains were significantly associated with likelihood of admission (Figure 2, Supplemental Table 1), with the exception of sensorimotor, which was inversely associated with risk. Effects were of similar magnitude in adjusted and unadjusted models (Supplemental Table 1). In a direct comparison of prediction associated with the token based approach versus the LLM-based approach, discrimination was significantly better for the LLM-based model (Figure 3).

Figure 2.

Logistic regression model for hospital admission, with sociodemographic and clinical features as well as Research Domain Criteria score estimates from a large language model

Other race or ethnicity refers to Native American, Pacific Islander, or any other category identified by individual or guardian at registration

Figure 3.

Decision curve analysis of regression models based on RDoC scores from large language model, token model, or both

Greater scores on LLM-RDoC scores were similarly statistically significantly but modestly associated with greater length of ED stay, with the exception of the sensorimotor domain, for which no significant association was identified (Supplemental Table 2). Once again, effect sizes were similar with or without adjustment for sociodemographic and clinical features.

We also examined the possibility of bias in RDoC score estimates derived from the LLM method (Table 2). Male patients on average scored higher on positive, cognitive, and sensorimotor domains, and lower on the negative valence domain. Patients without private insurance scored higher on cognitive, social, and sensorimotor domains. Conversely, patients identifying as Black, Hispanic, or Asian scored lower on the positive valence domain, and those identifying as Black and Hispanic also scored lower on the arousal domain. As these differences could reflect multiple confounding variables (e.g., differential utilization of the psychiatric emergency room, or differential documentation), we also compared to the simpler token-based method; see Supplemental Table 3.

Table 2.

Associations between RDoC estimates from large language model and sociodemographic features

	RDoC negative			RDoC positive
Predictors	Estimates	CI	p	Estimates	CI	p
Gender (male)	−0.13	−0.23–−0.02	0.017	0.53	0.41–0.65	<0.001
Age (years)	0.04	0.02–0.05	<0.001	0.13	0.11–0.15	<0.001
Private insurance	−0.09	−0.20–0.02	0.112	−0.07	−0.20–0.06	0.262
Race or ethnicity [Other * ]	0.16	0.01–0.31	0.04	−0.17	−0.34–0.01	0.065
Race or ethnicity [Black]	−0.13	−0.30–0.04	0.145	−0.21	−0.42–−0.01	0.041
Race or ethnicity [Hispanic]	−0.11	−0.30–0.08	0.251	−0.3	−0.52–−0.08	0.007
Race or ethnicity [Asian]	−0.19	−0.47–0.10	0.203	−0.37	−0.70–−0.03	0.031
	RDoC cognitive			RDoC social
Predictors	Estimates	CI	p	Estimates	CI	p
Gender (male)	0.53	0.42–0.64	<0.001	0.10	−0.01–0.21	0.079
Age (years)	0	−0.02–0.02	0.917	0.01	−0.01–0.02	0.39
Private insurance	−0.17	−0.29–−0.05	0.005	−0.40	−0.52–−0.28	<0.001
Race or ethnicity [Other * ]	−0.05	−0.21–0.11	0.539	0.22	0.06–0.38	0.008
Race or ethnicity [Black]	−0.03	−0.21–0.16	0.778	0.01	−0.17–0.20	0.886
Race or ethnicity [Hispanic]	−0.17	−0.36–0.03	0.103	0.10	−0.10–0.30	0.322
Race or ethnicity [Asian]	0.05	−0.26–0.35	0.757	−0.07	−0.37–0.24	0.66
	RDoC arousal			RDoC sensorimotor
Predictors	Estimates	CI	p	Estimates	CI	p
Gender (male)	0.03	−0.08–0.14	0.566	0.30	0.21–0.40	<0.001
Age (years)	−0.02	−0.03–0.00	0.058	−0.09	−0.10–−0.07	<0.001
Private insurance	−0.11	−0.23–0.01	0.082	−0.18	−0.28–−0.08	<0.001
Race or ethnicity [Other * ]	−0.07	−0.23–0.10	0.428	−0.08	−0.21–0.06	0.268
Race or ethnicity [Black]	−0.21	−0.39–−0.02	0.031	0.00	−0.15–0.16	0.965
Race or ethnicity [Hispanic]	−0.3	−0.50–−0.10	0.004	−0.21	−0.38–−0.05	0.012
Race or ethnicity [Asian]	−0.31	−0.61–0.00	0.051	−0.01	−0.27–0.24	0.929

^*Other race or ethnicity refers to individuals registered based on self-report, or guardian report, of Native American, Pacific Islander, or any other race not listed among these categories

As a more direct test of the potential impact of race, we permuted a random subset of 200 notes to reflect patients of either white, Black, or Asian race. We then compared scores on each domain between pairs of notes. Mean differences comparing scores from Black to white permuted patients, and associated p-values, were −0.05 (p=.13), 0 (p=1.0), −0.05 (p=.25), −0.10 (p=.06), −0.06 (p=.58), and −0.05 (p=.32) for negative, positive, cognitive, arousal, social, and sensorimotor domains, respectively. Mean differences comparing scores from Asian to white permuted patients, and associated p-values, were 0.02 (p=.51), −0.06 (p=0.05), −0.02 (p=0.75), −0.03 (p=0.61), −0.07 (p=.13), and 0.03 (p=.54).

DISCUSSION

In this analysis of psychiatric emergency room visits for more than 3,000 children and adolescents, we found that estimates of RDoC domains derived from a large language model were correlated with estimates from a previously-validated natural language processing method, but explained additional variance in clinical outcomes.

These results represent an example of zero-shot learning – that is, the capacity of LLM’s to perform well on tasks they were not specifically trained to solve, without examples. We initially determined that GPT4 was able to describe the NIMH Research Domain Criteria, asking it to summarize them or apply them to particular scenarios as described in the Supplemental Materials. The prompt itself simply provided a scale to apply in order to generate scores, mirroring those commonly used in (for example) reports of pain symptomatology(13).

There is no true gold standard for RDoC domain estimates, although particular behavioral paradigms have been endorsed by committees constituted to operationalize RDoC principles(4). Here we relied on presenting anchor points tied to need for treatment, and validating against a previously-validated token-based approach. The goal of the present effort was not to replace more detailed assessments using rating scales and tests, but rather to enable legacy data sets – particularly large-scale electronic health records – to be employed for dimensional psychopathology investigations. While our prior work enabled a range of studies drawing on electronic health records, including prediction of clinical outcomes(7), genomic studies(9), and postmortem investigation(8), its ability to capture the full range of psychopathology was limited because it only considered individual terms, rather than context.

In addition to convergent validity supported by correlation with prior scores, we found strong support for predictive validity. Incorporating the LLM-derived scores significantly improved model fit (compared to sociodemographic features alone, and sociodemographic features plus estimated RDoC scores), indicating that they captured additional variance useful in explaining the outcomes. These effects were observed for both likelihood of hospitalization, and length of emergency room stay. The fact that associations are modest in magnitude for the latter likely reflects the reality that many determinants of length of stay are reflect system rather than patient-level factors – e.g., availability of pediatric hospital beds. We emphasize that these analyses were not intended to develop prediction tools, which would entail a different approach(14), but rather to understand the improvement afforded by these new scores.

As in our prior work, we observed substantial correlation between individual RDoC domains(15). Despite efforts to identify functionally distinct dimensions, given that disorders frequently span such dimensions, it is not surprising that they are not truly orthogonal. In cognition, the value of investigating individual domains compared to general intelligence, or g factor, has been debated(16). By analogy, efforts to derive and quantify a p factor for general psychopathology may merit further consideration(17).

A key strength of this approach is its transparency and interpretability. While we focused on score generation, the examples we provide illustrate the capacity of these models to justify scores on the basis of a given note. They also suggest the overlap between RDoC domains more broadly and traditional categorical diagnostic systems, at least in narrative notes: as expected, clinicians document symptoms that correspond to diagnostic features. Notably, the models are also able to indicate the changes in symptoms that might warrant an increase or decrease in score (Supplement D). Their flexibility is also a potential benefit: prompts may be readily adapted to reflect shifts in definitions or domains. For example, in pilot investigation, we noted that the model did not distinguish substance use as reflecting positive valence; we therefore included a reminder in the prompt that substance use reflects reward pathways. Similarly, adding the 6^th (sensorimotor) RDoC domain, not present when we reported our prior method, required no additional curation. Further efforts to calibrate individual domains against gold standards, when they emerge, should require only modest modification of prompt language, or incorporation of a small number of examples to achieve few-shot learning(18). Future calibration should also consider whether allowing a broader range of scores (e.g., between 0 and 100, or incorporating decimal values) would capture more variability.

On the other hand, we also identified possible bias by race or ethnicity and gender in estimated RDoC scores – that is, we see modest but statistically significant differences in scores from individual groups. These differences are likely to be multifactorial, and do not necessarily represent bias in the language model per se. For example, if the acuity of the population using the emergency room differed by race, it could induce association between mean scores and race. In other words, if a particular group of individuals was less likely to access care except in more extreme states of illness, that could explain the observed effects. Biases in clinician documentation – that is, a tendency to apply different terms to symptoms in one race or ethnicity versus another, or one gender versus another – could also explain our results. Along these lines, we previously reported race-associated differences in sentiment reflected in hospital discharge notes(19). On the other hand, in another set of experiments, we found that permuting notes to change race between white, Black, and Asian did not yield meaningfully different scores, making LLM-specific biases a less likely explanation for the differences we observed. Existing studies suggest that LLM output bias may vary by context. For example, one prior study found minimal difference in GPT-generated patient instructions by race(20), while another found associations of race with particular diagnoses in vignette generation by LLM(21).

Limitations

This study has multiple limitations. First, it reflects a specific clinical population, region, and time period – the language used to describe symptoms or diagnoses is likely to vary with age and by geographic area, as each health system has its own culture around documentation. Changes in documentation, including more use of templated notes and mandated open note access for patients(22), are likely to alter their information content. As such, fully understanding the performance of LLM’s in other contexts will therefore require application in additional populations, and the present study could encourage such efforts. Second, while we have previously validated the original token-based approach to estimating psychopathology, it would also be valuable to compare the scores generated by the LLM to those from behavioral paradigms or other RDoC symptom measures in future work, if and when those forms of data become available applying the full range of RDoC phenotyping in large cohorts. Indeed, a potential application of our approach could be to identify individuals for further investigation using the many paradigms developed for specific RDoC subdomains. Finally, as with any naturalistic cohort, we cannot exclude confounding effects – for example, the possibility that associations between individual RDoC domains and outcome reflect some unmeasured feature reflected in nearly every domain as well as outcome measurement.

Conclusion

Taken together, our results support the utility of applying a large language model to estimate dimensional psychopathology in electronic health records. The widespread availability of these tools should facilitate these efforts, unlocking concepts in narrative clinical notes for a range of applications from prediction to phenotyping for biological investigations. While caution is warranted – particularly to ensure protection of confidentiality and avoidance of bias – such models should substantially accelerate efforts to expand RDoC use in both research and clinical settings(2).