Analyzing Generative AI and Machine Learning in Auto-Assessing Schizophrenia’s Negative Symptoms

Abstract

Background and Hypothesis

Traditional assessments of schizophrenia’s negative symptoms rely on subjective and time-consuming psychiatric interviews. To provide more objective and efficient evaluations, this study examines the efficacy of an automated system utilizing generative AI (GenAI) and machine learning (ML) to assess negative symptoms of schizophrenia, including expression (EXP) and motivation and pleasure (MAP) domains.

Study Design

A semi-structured interview protocol based on the Clinical Assessment Interview for Negative Symptoms was used to conduct interviews with schizophrenia patients. An experienced senior psychiatrist carried out these interviews, which were audio- and video-recorded, at the National Taiwan University Hospital between July 2022 and August 2023. An ML-based system analyzed visual and audio data for EXP assessment, while GenAI analyzed interview transcripts for MAP assessment.

Study Results

The study cohort consisted of 69 males and 91 females with a mean age of 41.68 years (SD = 10.46). The ML-based EXP assessment showed moderate to substantial reliability, with an intraclass correlation coefficient (3, 1) (ICC_3,1) of 0.65 and a weighted kappa of 0.62. The GenAI-based MAP assessment demonstrated good reliability, with an ICC_3,1 of 0.82 and a weighted kappa of 0.77. The system achieved strong linear correlations with clinician ratings (Pearson’s correlation coefficient ≥ 0.54) and maintained low error rates (mean absolute error ≤ 0.81; root mean square error ≤ 1.16) for each assessment item.

Conclusions

The study demonstrates the efficacy of GenAI and ML in the automated assessment of schizophrenia’s negative symptoms, highlighting their potential to enhance the consistency and efficiency of clinical evaluations.

Introduction

Negative symptoms, characterized by a reduction in normal functions related to expression (EXP) as well as motivation and pleasure (MAP), pose significant challenges in the care of schizophrenia patients due to their resistance to current treatments.¹ Traditionally, these symptoms are assessed through psychiatric interviews using specialized instruments such as the Clinical Assessment Interview for Negative Symptoms (CAINS) and the Positive and Negative Syndrome Scale (PANSS).² However, this approach is subjective and time-consuming, highlighting the need for automated systems that offer a more objective, consistent, and efficient way to assess these symptoms.

Efforts to automate the assessment of negative symptoms generally fall into 2 main approaches. The first aims to replicate traditional clinical evaluations by training models on clinician-administered interviews and applying established rating scales, such as the CAINS or PANSS.^3-12 The second approach estimates symptom severity by analyzing behavior during simple tasks—such as reading aloud or answering brief questions—typically recorded using smartphones or video devices.^13-16 While the first approach is resource-intensive and requires close collaboration between clinicians and technologists, it closely aligns with standard clinical practices. It holds promise for a variety of applications, including serving as a reference for rater training, monitoring the quality of human ratings in clinical trials, and enabling remote assessments through automated machine-based raters. In contrast, the second approach is more scalable and less labor-intensive but relies on proxy indicators of symptoms and may lack the validity needed for use in formal clinical settings. Thus, this study focuses on the first approach: developing an automated system that assesses negative symptoms from recordings of psychiatric interviews.

In line with this focus, past research on automated systems for evaluating negative symptoms in schizophrenia has primarily focused on binary classifications, categorizing symptoms as either “low severity” or “high severity.”^3-8 These approaches provide only a coarse assessment and overlook subtle variations in symptom severity. Only 4 studies have recently attempted a more detailed and fine-grained evaluation.^9-12 Tron et al. proposed a machine learning (ML) algorithm that used the Facial Action Coding System (FACS) to analyze facial EXPs from videos captured by 3D structured light cameras and estimate PANSS symptom scores. However, this approach is impractical for wider clinical use due to its high cost and complexity of the equipment.⁹ Similarly, Bishay et al. developed SchiNet, a deep learning model that focuses on facial EXPs and uses FACS to estimate CAINS and PANSS scores. Yet they were limited to assessing EXP-related symptoms, neglecting other symptom areas.¹⁰ Huang et al. and Chuang et al. employed multimodal deep learning models incorporating linguistic, acoustic, and visual features for PANSS score estimation but validated their methods on very small datasets of only 26 patients.^11,12 Furthermore, only one of these studies⁹ provided information about their interview questions, hindering the replication of future research and comparison of methodologies. All 4 studies either did not report correlations or showed low correlations with clinicians’ ratings (0.27-0.53). These limitations underscore the need to develop an automated assessment system that is validated on a larger dataset, incorporates a detailed interview protocol, and uses a more accurate analytic method to assess negative symptoms’ EXP and MAP dimensions.

Large language models (LLMs), a type of generative AI (GenAI), excel in handling diverse natural language tasks, with proprietary models such as GPT, Claude, and Gemini often outperforming open-source alternatives.¹⁷ This capability has revolutionized healthcare by enhancing clinical decision-making, automating documentation, and personalizing patient care.¹⁸ Recent studies in mental health show their effectiveness in analyzing social media for mental health detection,¹⁹ interpreting electronic health records,²⁰ and assessing psychiatric symptoms from clinical interviews.²¹ However, their application in evaluating negative symptoms remains underexplored.

In ML, instead of training models from scratch, a more common approach is to pretrain them on large, diverse datasets. Pretraining allows models to gain a broad understanding of various patterns and features, making them adaptable to specific tasks with minimal additional training. Pretrained models can act as feature extractors, generating low-dimensional representations (embeddings) of input data, which can then be fine-tuned or directly applied to particular applications.²² Additionally, recent advancements have demonstrated the potential of zero-shot learning, where pretrained models can tackle new tasks without requiring task-specific training data. For instance, studies have demonstrated that LLMs can generate appropriate responses to various language tasks by simply being provided with a text prompt, a technique known as zero-shot prompting, which eliminates the need for further task-specific training.²³

Our objective is to develop an automated system capable of analyzing traditional psychiatric interview recordings to assess a broader range of negative symptoms than previous studies have addressed. This system is designed to be applicable in clinical settings and to address the limitations of earlier approaches. To achieve this, we designed a semi-structured interview protocol with questions exploring schizophrenia patients’ emotions, motivations, and pleasurable experiences according to CAINS items. We constructed a dataset of 160 patient interviews, which is larger than those in previous studies. We proposed a dual-strategy approach to evaluate negative symptoms’ EXP and MAP domains, as illustrated in Figure 1. For the EXP domain, we use ensemble learning, combining 2 types of classifiers to analyze visual and audio data. For the MAP domain, we pioneer using LLMs to analyze interview transcripts with zero-shot prompting based on CAINS guideline and our interview protocol. Our code is available in a GitHub repository at https://github.com/MartyJan/SchizoNegSympAI, released under an MIT license.

Figure 1.

Pipeline of Automated Negative Symptom Assessment in Our Study. Our automated system evaluates negative symptoms of schizophrenia by analyzing clinical interview recordings. For the EXP domain, it extracts features from video and audio data and applies ensemble learning with 2 types of classifiers. For the MAP domain, it uses LLMs with zero-shot prompting to analyze interview transcripts based on CAINS guidelines and our interview protocol.

Methods

Participants

We recruited participants from the Department of Psychiatry, National Taiwan University Hospital, between July 2022 and August 2023. The inclusion criteria required subjects to be between 20 and 65 years old, diagnosed with schizophrenia according to the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) criteria,²⁴ and either stable outpatients or individuals participating in rehabilitation programs at the hospital’s daycare center. We excluded individuals with diagnoses of bipolar disorder, schizoaffective disorder, or major depressive disorder, as well as those with a history of substance abuse, intellectual impairments, or severe neurological diseases such as Parkinson’s disease and tardive dyskinesia that could interfere with the assessment during the interview. Sex and gender were determined from hospital health records. All participants provided written informed consent before they participated in the study. Ethical approval for this research was granted by the Institutional Review Board at National Taiwan University Hospital with REC number 202204072RINB.

Interview Protocol

To standardize the interview process for assessing negative symptoms, we designed a semi-structured interview protocol called the Semi-Structured Interview for CAINS (si-CAINS) (Appendix S1) based on all 13 CAINS items. The si-CAINS is divided into 3 sections. Section 1 (S1) includes open-ended questions about the patient’s daily activities over the past week, which is the time frame for CAINS ratings. Patients are also asked to describe their family members and living situations. These initial queries give the interviewer a basic understanding of the patient’s life for follow-up questions. Section 2 (S2), consisting of 3 questions (Q2 to Q4), evaluates how patients articulate and process different emotions by asking them to recount personal experiences of happiness, sadness, and anger. The interviewer is guided to evaluate EXP items during S1 and S2 because these sections are relatively free-form and are designed to elicit the maximum levels of the patients’ emotional EXPs. Section 3 (S3) includes 9 semi-structured questions (Q5 to Q13) corresponding to items 1 to 9 in CAINS. This section investigates the patient’s MAP in social interaction, school or work, and leisure activities.

The si-CAINS closely follows the guidelines and instructions outlined in the CAINS manual²⁵ and CAINS validation study,²⁶ with some modifications to ensure the uniformity of the interview structure and refine the anchor points for greater accuracy. These modifications include: (1) the addition of S2 to elicit the maximum level of emotional EXP from patients, enabling the automated system to capture critical moments for assessing the EXP domain; (2) the inclusion of evaluation principles for various life situations (Appendix S1, Q9) to improve the precision of assessing “Motivation for Work and School Activities” (item 5); and (3) the provision of alternative anchor points for items related to the frequency of expected pleasurable social (item 4), work and school (item 6), and recreational (item 9) activities in the upcoming week. This adjustment was based on our preliminary interview findings, where some chronic schizophrenia patients, due to cognitive deficits, had difficulty accurately determining and expressing their expected frequency of these pleasurable activities.

Data Collection

Each participant was administered the si-CAINS to assess their negative symptoms, conducted primarily in Mandarin by an experienced senior psychiatrist (C.-M. Liu). The face-to-face interviews were recorded using a 2D camera (SONY ZV-1) at 1920 × 1080 pixels and 30 frames per second, capturing the patient’s upper body from above the knee and their front faces. The audio was recorded at a 48 kHz sampling rate using lavalier microphones (Hollyland LARK 150) attached to the patient and psychiatrist, synchronized with the video via a wireless receiver.

EXP Assessment

The EXP assessment takes interview segments S1 and S2 as input and leverages ensemble classifiers to analyze video and audio data, utilizing various feature types tailored to evaluate distinct symptoms. “Facial Expression” (item 10) is evaluated using facial features, “Vocal Expression” (item 11) via speech features, “Expressive Gestures” (item 12) through pose features, and “Quantity of Speech” (item 13) by speaking duration features.

To begin, the NeMo speaker diarization framework²⁷ is employed on audio data to separate the patient’s speech from that of the interviewer. Speaking duration features are derived by calculating the proportion of time patients speak in 1-second intervals. To focus exclusively on patients’ expressive behaviors, audio and video segments where the patient is actively speaking are concatenated separately for further feature extraction.

Video data are initially processed using MediaPipe,²⁸ which computes 52 facial blend shapes and 25 3D landmarks of the upper body to track facial EXPs and body posture for each frame. The mean and SD of these features are then calculated over 1-second sliding windows, summarizing the movements and generating a set of features that reflect the patient’s facial and body expressiveness. Audio data are handled using a pretrained XLS-R encoder,²⁹ which transforms raw audio waveforms into low-dimensional audio embeddings. These embeddings capture a range of prosodic and acoustic patterns—such as pitch, pitch variation, rhythm, and intensity—not through direct measurement but by automatically learning these characteristics from large-scale audio data. This approach enables a more comprehensive and data-driven representation of vocal expressivity without the need for manually defined or handcrafted acoustic features.³⁰ Following this, Hartley spectral pooling³¹ is applied to downsample the time frames, ensuring consistent feature length across subjects (model parameters are detailed in Appendix S2). Finally, the features are flattened into 1D vectors.

These vectors are input into 2 types of classifiers: ordinal random forest (Ord-RF)³² and nominal random forest (Nom-RF).³³ The ordinal classifier handles categories following a specific order, while the nominal classifier treats each category as distinct and unordered. In the training phase, Tomek link under-sampling³⁴ addresses data imbalance, and the ordinal and nominal classifiers are trained separately. During inference, outputs from both classifiers are ensembled, and the category with the highest probability determines the final estimation.

MAP Assessment

Audio recordings are transcribed into text using Whisper.³⁵ Large language models are then employed to analyze the text for MAP assessment. This process involves analyzing interview segments S1 and S3 transcripts according to evaluation criteria, similar to a writing assessment.³⁶ It employs a zero-shot prompting technique, with a Mandarin prompt that includes a task description, evaluation criteria, and conversation content (Appendix S3). The evaluation criteria comprise the guidelines in the “Rating Anchor Points” section of the CAINS manual²⁵ for each item as well as our alternative anchor points, which are described in the si-CAINS (Appendix S1). Various LLMs proficient in Mandarin language processing,¹⁷ such as Claude-3-Haiku,³⁷ Gemini-1.0-pro,³⁸ and GPT-3.5-turbo,³⁹ are utilized at a fixed temperature of 0.5. The final symptom severity score is calculated by averaging the modes of the output scores obtained by applying the same prompt to each model 3 times. To ensure data security and privacy, audio was transcribed locally, and only de-identified textual transcripts, with all personally identifiable information removed, were submitted to third-party LLMs via API. As a result, no sensitive patient information was shared externally.

Model Performance Evaluation

Following previous studies,^9-12 model performance is evaluated by calculating the root mean squared error (RMSE), mean absolute error (MAE), and Pearson’s correlation coefficient (PCC) between model-estimated scores and the ratings provided by the clinician. Additionally, the mean error (ME), which is derived by subtracting human rater scores from estimated scores, is used to assess directional bias. Reliability between model estimates and clinical ratings is measured using the intraclass correlation coefficient (3, 1) (ICC_3,1) and weighted kappa coefficient (κ_w). An ICC_3,1 value greater than 0.90 indicates “excellent” reliability; values between 0.75 and 0.9 denotes “good” reliability, values between 0.5 and 0.75 suggest “moderate” reliability, and values below 0.5 indicate “poor” reliability.⁴⁰ A kappa value from 0.61 to 0.80 indicates “substantial” reliability, whereas those from 0.41 to 0.60 suggest “moderate” reliability.⁴¹ Moreover, the confusion matrix is used to visualize the distribution of estimated scores vs human ratings. The vertical axis represents the actual scores, while the horizontal axis shows the estimated scores. A higher concentration of samples along the diagonal line indicates better model performance. For the EXP assessment, we utilized 5-fold cross-validation, randomly dividing the entire dataset into 5 partitions. In each iteration, 1 partition (20%, 32 subjects) was used for validation, and the remaining 4 partitions (80%, 128 subjects) were used for training. The metrics were averaged across all 5 folds. For MAP assessment, metrics were computed across all 160 subjects, as zero-shot prompting does not require training data.

Results

Dataset Statistics

This study enrolled 160 schizophrenia patients, ranging in age from 20 to 62 years, with a mean age of 41.68 years and a SD of 10.46 years. Participants were all Taiwanese. The cohort consists of 69 males and 91 females. The subjects have a MAP total score of 8.01 ± 7.33 and an EXP total score of 2.60 ± 3.25, with statistics for each item detailed in Table S1. The average duration of the interviews is 18.22 minutes, with a SD of 5.05 minutes.

EXP Assessment

First, focusing on the performance of the ensemble method in Table 1, it demonstrates moderate to substantial reliability, with an ICC_3,1 of 0.65 and a weighted kappa coefficient of 0.62 for the EXP total score. Furthermore, the PCC for each item exceeds 0.5, signifying a strong positive linear relationship between the model’s estimates and the clinician’s evaluations. The model also maintains a low error rate, with the RMSE and MAE for each item being less than 1.0. This suggests that the average error margin does not exceed 1 anchor point, a threshold generally accepted as sufficiently precise for clinical applications.⁴² The model exhibits a slight tendency to underestimate scores, as indicated by negative MEs for each item; however, the magnitude of these errors remains below 0.12 across all items.

Table 1.

Model Comparison for EXP Assessment

	Ensemble						Ord-RF						Nom-RF
	RMSE	MAE	ME	PCC	ICC 3,1	κ w	RMSE	MAE	ME	PCC	ICC 3,1	κ w	RMSE	MAE	ME	PCC	ICC 3,1	κ w
Facial expression	0.82	0.53	−0.12	0.56	0.55	0.54	0.86	0.57	−0.14	0.53	0.53	0.51	0.87	0.58	−0.10	0.50	0.50	0.48
Vocal expression	0.80	0.51	−0.03	0.54	0.52	0.51	0.81	0.53	−0.04	0.52	0.50	0.49	0.86	0.54	0.02	0.51	0.49	0.48
Expressive gestures	0.87	0.56	−0.12	0.59	0.58	0.57	0.89	0.58	−0.13	0.57	0.56	0.55	0.87	0.56	−0.14	0.58	0.57	0.56
Quantity of speech	0.65	0.31	−0.11	0.61	0.58	0.56	0.67	0.32	−0.11	0.59	0.57	0.56	0.63	0.31	−0.13	0.60	0.57	0.55
EXP total	2.48	1.76	−0.38	0.68	0.65	0.62	2.55	1.81	−0.41	0.66	0.64	0.61	2.50	1.78	−0.36	0.67	0.62	0.60

This table compares the performance of the ensemble method with 2 individual classifiers, ordinal random forest (Ord-RF) and nominal random forest (Nom-RF), for the EXP assessment. The ensemble method outperforms the individual classifiers in terms of the lowest RMSE and MAE, as well as the highest PCC, ICC_3,1, and weighted kappa. Notably, the ensemble model achieves moderate to substantial reliability with an ICC_3,1 of 0.65 and a weighted kappa of 0.62 for the EXP total score.

Abbreviations: ICC_3,1, intraclass correlation coefficient (3, 1); EXP, expression; MAE, mean absolute error; PCC, Pearson’s correlation coefficient; κ_w, kappa coefficient; RMSE, root mean squared error.

Next, comparing the ensemble method with the individual classifiers (Ord-RF and Nom-RF), we observe that it outperforms both by achieving the lowest RMSE and MAE, as well as the highest PCC, ICC_3,1, and weighted kappa (Table 1). Moreover, the confusion matrices further corroborate this superior performance, with the ensemble method demonstrating a stronger concentration along the diagonal compared to Ord-RF and Nom-RF (Figure S1). This enhanced performance may be attributed to the ensemble method’s ability to combine the strengths of both ordinal and nominal classifiers, enabling a more flexible and nuanced interpretation of symptom severity. Ordinal classifiers are well-suited for handling ordered categories, such as the rating anchors used in clinical scales, where categories like “mild,” “moderate,” and “severe” follow a logical progression. However, in real-world assessments, symptom presentations may fall between these discrete categories. For instance, a patient may present with an emotional EXP deficit that clearly exceeds the criteria for a score of 1 (“mild”) but partially meets the criteria for a score of 2 (“moderate”), creating ambiguity. Because ordinal classifiers assume a strict ordering of categories, they may have difficulty resolving such borderline cases. In contrast, nominal classifiers treat each category independently, allowing more flexibility in capturing symptoms that do not clearly align with a specific severity level. By integrating both approaches, the ensemble method benefits from the ordinal model’s capacity to preserve the rating hierarchy and the nominal model’s capacity to handle ambiguous or in-between symptom presentations.

To examine whether different interview segments influence model performance, we conducted experiments using 2 input settings: (1) segments S1 and S2, which include open-ended questions (S1) and emotion-eliciting questions (S2), and (2) the entire interview (S1-S3) into the model. The results (Table S2) demonstrate that the former approach yields more accurate evaluations for each item than the latter regarding all metrics. When using the full interviews, the ICC_3,1 and weighted kappa values for the EXP total score are lower than 0.4, indicating poor reliability in the model’s estimations. This finding aligns with the intentional design of the interview protocol, which guides an interviewer in evaluating EXP-related symptoms during S1 and S2.

MAP Assessment

We conducted a qualitative analysis by selecting a subject from our dataset and comparing LLMs’ outputs with excerpts from the conversation transcript between the psychiatrist (P) and the subject (S). Table 2 presents the results for evaluating “Motivation for Close Family/Spouse/Partner Relationships.” The transcript reveals that although the subject resides with his mother and older sister, he seldom initiates interaction with his family. Our analysis indicates that all 3 LLMs effectively extract this key information related to this item from the provided transcripts. In their assessment, Claude noted that “the subject interacts less with his mother and older sister, showing limited initiative.” Gemini mentioned that “the subject lives with his mother and older sister but interacts with them infrequently.” GPT reported a low initiative, responding only when his family members initiate questions. Furthermore, the models correctly assigned higher scores, correlating with more severe symptoms. In assessing “Frequency of Expected Pleasurable Social Activities—Next Week,” all models adhere to the evaluation criteria by basing their ratings on “Frequency of Pleasurable Social Activities—Past Week” (item 3). They assign the same score as item 3 because the subject expects similar levels of pleasure in the upcoming week compared to the previous week (Table S3).

Table 2.

Transcript Excerpt and LLM Responses for CAINS Item 1 Assessment

	Content
Transcript excerpt	P: Who lives with you? S: My mom and older sister. P: Do you go out on weekends? S: Sometimes I go shopping for groceries with my mom. P: You said you live with your mom and older sister, right? How is your relationship with them? S: It’s okay. P: So, do you actively talk to them every day? S: Rarely talk. P: With your mom? S: Only if my mom asks me something, then I will respond. P: Between your mom and your older sister, who are you closer to? S: My older sister. P: [...] Do you initiate conversations with her? S: Also rarely. P: [...] Your mom or older sister talks to you; does that make you happy? S: It’s okay.
Claude-3-Haiku	Item: Motivation for Close Family/Spouse/Partner Relationships Assessment: The subject interacts less with his mother and older sister, showing limited initiative; mostly his mother who initiates. Although he claims to have a good relationship, he rarely initiates conversations with family members and does not particularly look forward to interacting with them. Score: 2
Gemini-1.0-pro	Item: Motivation for Close Family/Spouse/Partner Relationships Assessment: The subject lives with his mother and older sister but interacts with them infrequently, usually with his mother or older sister initiating conversation. The subject stated that he has a better relationship with his older sister, although he does not interact much. Occasionally, the subject goes grocery shopping with his mother but does not frequently interact with her and shows little initiative to do so. Score: 2
GPT-3.5-turbo	Item: Motivation for Close Family/Spouse/Partner Relationships Assessment: The subject’s interaction frequency and quality with family members primarily involve his mother and older sister. However, there is a low initiative, responding only when they initiate questions, and the duration of interactions is also relatively short. The subject feels that interactions with his mother are average, with no particularly pleasant feelings. Score: 2

This table presents a qualitative analysis of the LLMs’ performance in evaluating the subject’s motivation for family/spouse/partner relationships. Based on the conversation transcript, all 3 LLMs accurately extracted key insights regarding the subject’s interactions with family members. Additionally, the models assigned higher scores for more severe symptoms, aligning with the subject’s responses.

Abbreviations: CAINS, Clinical Assessment Interview for Negative Symptoms; LLMs, large language models.

Subsequently, we conducted a quantitative analysis comparing scores extracted from these LLM outputs with clinical ratings to measure their ability to evaluate the severity of negative symptoms. The results (Table 3) demonstrate that our method achieves moderate to good reliability, with an ICC_3,1 above 0.72 and a weighted kappa coefficient greater than 0.64 for the MAP total score. Each CAINS item shows a PCC above 0.5, indicating a positive linear relationship between the model’s estimates and the clinical evaluations. The MAEs below 1.0 and the RMSEs around 1.0 also suggest an average error range of approximately 1 anchor point, indicating clinically acceptable precision in model performance.⁴²

Table 3.

Performance of MAP Assessment

	Claude-3-Haiku						Gemini-1.0-pro						GPT-3.5-turbo
	RMSE	MAE	ME	PCC	ICC 3,1	κ w	RMSE	MAE	ME	PCC	ICC 3,1	κ w	RMSE	MAE	ME	PCC	ICC 3,1	κ w
Motivation for close family/spouse/partner relationships	1.16	0.84	0.69	0.55	0.51	0.40	0.97	0.59	0.36	0.57	0.53	0.50	1.05	0.59	0.32	0.58	0.52	0.49
Motivation for close friendships and romantic relationships	1.47	1.23	1.19	0.71	0.7	0.44	1.15	0.89	0.74	0.72	0.7	0.58	1.12	0.76	0.11	0.59	0.56	0.56
Frequency of pleasurable social activities—past week	0.78	0.46	0.18	0.76	0.76	0.75	0.84	0.51	−0.05	0.74	0.73	0.73	0.93	0.61	−0.13	0.73	0.70	0.70
Frequency of expected pleasurable social activities—next week	0.99	0.67	0.46	0.68	0.68	0.62	0.91	0.58	0.19	0.71	0.70	0.69	1.01	0.64	0.11	0.65	0.63	0.63
Motivation for work and school activities	1.20	0.96	0.93	0.74	0.74	0.54	1.09	0.81	0.73	0.77	0.76	0.63	1.18	0.82	0.41	0.63	0.60	0.57
Frequency of expected pleasurable work and school activities—next week	1.16	0.83	0.73	0.74	0.74	0.63	1.23	0.81	0.58	0.72	0.69	0.63	1.30	0.83	0.14	0.52	0.51	0.51
Motivation for recreational activities	1.09	0.81	0.76	0.71	0.70	0.55	0.86	0.52	0.27	0.75	0.72	0.70	1.00	0.59	0.15	0.65	0.61	0.61
Frequency of pleasurable recreational activities—past week	0.87	0.56	0.19	0.69	0.69	0.68	0.84	0.53	−0.16	0.75	0.73	0.73	0.97	0.66	−0.09	0.69	0.67	0.66
Frequency of expected pleasurable recreational activities—next week	0.93	0.63	0.36	0.69	0.68	0.64	0.89	0.54	0.11	0.74	0.71	0.70	0.99	0.61	0.14	0.65	0.63	0.62
MAP total	7.44	6.10	5.48	0.81	0.80	0.64	5.49	4.18	2.76	0.83	0.82	0.77	6.78	4.91	1.16	0.76	0.72	0.72

This table provides a quantitative comparison of LLM-extracted scores and clinical ratings for MAP assessment. All 3 LLMs demonstrate moderate to good reliability, with an ICC_3,1 above 0.72 and a weighted kappa coefficient exceeding 0.64 for the MAP total score. Each CAINS item exhibits a strong positive correlation (PCC > 0.5) and low error rates (MAE < 1.0, RMSE ≈ 1.0) between the model’s estimates and clinical evaluations.

Abbreviations: CAINS, Clinical Assessment Interview for Negative Symptoms; ICC_3,1, intraclass correlation coefficient (3, 1); LLMs, large language models; MAE, mean absolute error; MAP, motivation and pleasure; ME, mean error; PCC, Pearson’s correlation coefficient; κ_w, kappa coefficient; RMSE, root mean squared error.

A closer examination of the performance of the 3 LLMs reveals that Claude exhibits a more pronounced upward bias across items, with the ME being positive and the highest among the 3 models. This suggests that Claude tends to assign higher scores than the clinical ratings. This pattern is exemplified in Table S4. In the case, the subject maintains a close bond with his mother, leading the clinician to assign a score of 0 for “Motivation for Close Family/Spouse/Partner Relationships,” despite the subject’s limited interaction with his father or brother. Although Claude correctly identifies this nuance in the transcript, it may interpret the reduced contact with other family members as overall diminished motivation, thus assigning a higher score of 1. Additionally, analysis of the confusion matrices (Figure S2) reveals that Claude generally overestimates symptoms. However, it does not deviate significantly from clinical ratings, resulting in a high error rate but also a high correlation. In contrast, GPT shows more noticeable deviations from the diagonal line in these matrices, corresponding to its lower ICC_{3, 1} and PCC values for most items compared to those of the other 2 models.

Discussion

Automated systems for assessing negative symptoms are vital, as they provide an objective standard for evaluation and can streamline the assessment process when integrated with “autonomous virtual therapists”⁴³ to conduct interviews automatically. This can improve assessment consistency, reduce clinician burden, and enable more frequent patient monitoring in clinical and drug trial settings. This study proposed a detailed semi-structured interview protocol (si-CAINS), collected a large dataset of 160 patient interviews, and utilized ML and GenAI to assess negative symptoms’ 2 major dimensions (EXP and MAP). Our approach addresses the limitations of previous research, which often focused solely on EXP-related symptoms, was validated on small sample sizes (<100 subjects), and lacked comprehensive interview protocols.^9-12

The clinical team in our research also participated in a reliability study of the PANSS.⁴² This study showed that the ICC for “Emotional Withdrawal,” which corresponds to item 7 in CAINS, was 0.69; for “Lack of Spontaneity and Flow of Conversation,” corresponding to item 13 in CAINS, it was 0.73; for “Blunted Affect,” a combination of items 10, 11, and 12 in CAINS, it was 0.44; and for “Passive/Apathetic Social Withdrawal,” a combination of items 1 and 2 in CAINS, it was 0.71. These results indicate moderate reliability. Similarly, our method achieves the same level of reliability, suggesting that our model’s performance is comparable to manual ratings. In the validation research conducted by the team that originally developed the CAINS, ICCs for human raters ranged from 0.69 to 0.94 (moderate to excellent reliability) for MAP items and from 0.64 to 0.71 (moderate reliability) for EXP items.²⁶ By comparison, our automated system achieved ICCs ranging from 0.53 to 0.76 for MAP items and from 0.52 to 0.58 for EXP items—slightly lower, but still within a similar level of reliability (moderate to good for MAP items, moderate for EXP items). Nonetheless, direct comparisons with human inter-rater reliability using the same CAINS interview videos will be essential to more robustly validate our system.

Compared to the method of Bishay et al.,¹⁰ which also used CAINS but targeted only EXP items, our approach outperforms in terms of RMSE, MAE, and PCC (Table 4). Moreover, our method demonstrates reliability reaching clinically acceptable levels. Additionally, as far as we know, we are the first to use LLMs in evaluating negative symptoms by analyzing interview transcripts. Our zero-shot prompting method provides scores with moderate to good reliability compared to clinical ratings and offers correct explanations for its evaluations. This makes it a valuable training resource for novice clinicians, helping them understand the reasoning behind the scores, and an insightful feedback system for experienced clinicians, assisting them in identifying possible errors in their ratings.

Table 4.

Comparison of State-of-the-Art Methods

	Ours					Bishay’s10
	RMSE	MAE	PCC	ICC 3,1	κ w	RMSE	MAE	PCC	ICC 3,1	κ w
Facial expression	0.82	0.53	0.56	0.55	0.54	0.93	0.66	0.46	NA	NA
Vocal expression	0.80	0.51	0.54	0.52	0.51	1.10	0.77	0.27	NA	NA
Expressive gestures	0.87	0.56	0.59	0.58	0.57	1.15	0.90	0.36	NA	NA
Quantity of speech	0.65	0.31	0.61	0.58	0.56	1.30	0.98	0.30	NA	NA
EXP total	2.48	1.76	0.68	0.65	0.62	3.34	2.67	0.45	NA	NA

This table shows that our method outperforms state-of-the-art approaches across all EXP items, achieving lower RMSE and MAE, along with higher PCC. Additionally, our approach exhibits clinically acceptable reliability, with ICC_3,1 and weighted kappa values consistently above 0.5, whereas previous studies do not report these reliability measures.

Abbreviations: ICC_3,1, intraclass correlation coefficient (3, 1); EXP, expression; MAE, mean absolute error; NA, not applicable; PCC, Pearson’s correlation coefficient; κ_w, kappa coefficient; RMSE, root mean squared error.

Many previous studies do not disclose their interview questions, making it difficult for future research to replicate or compare methodologies. In contrast, we meticulously designed an interview protocol that starts with questions about the subject’s life and continues with those aimed at eliciting emotions and understanding motivations and pleasures. The findings indicate that using only the segments (S1 and S2) of the interview specifically designed to elicit emotional responses results in better model performance for EXP assessment than the entire interview. This is consistent with the intentional design of si-CAINS and the guidance provided to human interviewers. S1 and S2 are intended to probe patients to actively elaborate on their emotional experiences, leading to richer and more expressive responses for analysis, whereas S3 is more structured and typically elicits more concise answers. This underscores the importance of careful interview design and protocol development in automating negative symptom assessment.

Our automated system can potentially assess negative symptoms in schizophrenia patients across different cultural, linguistic, and ethnic backgrounds. Firstly, the CAINS has been translated into multiple languages, with all versions demonstrating satisfactory reliability.^44-46 Therefore, the si-CAINS is ready for translation and can be easily adapted to evaluate patients from diverse ethnic groups. Secondly, the ML model for EXP assessment uses subtle expressive features, such as blend shapes and pose landmarks, less influenced by cultural differences than macro-EXPs.⁴⁷ Moreover, the XLS-R model’s encoder for extracting speech features is robust in capturing cross-lingual speech representations.²⁹ Thirdly, LLMs have demonstrated excellent reasoning performance in multiple languages,⁴⁸ making them well-suited for MAP assessment in various linguistic contexts. In addition, our methods exhibited stable reliability across sex. For EXP assessment, we observed moderate to substantial reliability (ICC_3,1 = 0.73 and κ_w = 0.71 for females; ICC_3,1 = 0.74 and κ_w = 0.75 for males). For MAP assessment, reliability was moderate to good (ICC_3,1 ≥ 0.71 and κ_w ≥ 0.61 for females; ICC_3,1 ≥ 0.71 and κ_w ≥ 0.65 for males), as detailed in Appendix S4 and Tables S5 and S6. These results highlight the robustness of our approach and its applicability across sexes.

Prior research⁴⁹ has demonstrated a strong association between verbal productivity and negative symptom severity, particularly for expressive domains such as flat affect and alogia. To determine whether this single measure could account for the performance of our multimodal EXP assessment framework, we examined the impact of verbal output, quantified by the total number of words spoken by patients, on model outcomes (see Appendix S5 for details). Table S7 reveals that features related to word count exhibit low importance scores within the model, except when assessing the “Quantity of Speech” item, where the word count feature has a greater contribution to its evaluation, as expected due to the direct relationship between word count and speech quantity. Furthermore, the removal of features most predictive of word count had a negligible impact on model performance, with changes in reliability metrics only 0.1 for the EXP total score (Table S8). These findings suggest that verbal productivity does not substantially drive our model, reinforcing the necessity of integrating diverse feature modalities—including non-verbal cues—for reliable and comprehensive assessment of negative symptoms.

Limitations

While our study achieves state-of-the-art results, it has several limitations. First, the data distribution is unbalanced, with patients ranging from no impairment (score 0) to moderately severe deficits (score 3). This imbalance may result from our recruitment settings, which included patients from outpatient clinics and daycare centers. The scarcity of cases with severe deficits (score 4) in the training dataset may reduce the model’s accuracy in assessing more severe symptoms. Future studies should include more data from patients with severe deficits, especially those in community or chronic care wards, to improve model training and evaluation. Another limitation is the exclusion of schizophrenia patients with common comorbidities, such as major depressive disorder and substance abuse. Including patients with these comorbidities in future studies would provide a more comprehensive understanding of symptom EXP and improve the model’s robustness in handling a broader range of clinical cases. In addition, our dataset was collected from a single medical center and includes only East Asian patients, limiting the generalizability of our findings to different clinical settings and ethnic groups. Despite the potential of our method to effectively assess patients from diverse cultural, linguistic, and ethnic backgrounds, it remains essential to validate our model on a more diverse, multi-center dataset to ensure its robustness across various ethnic groups and clinical environments. Lastly, the lack of direct comparisons between our model’s performance and the inter-rater reliability of human assessors using the same CAINS interview recordings represents another limitation.

Conclusion

In conclusion, we designed a semi-structured interview protocol (si-CAINS) to assess emotional responses, motivations, and pleasures in schizophrenia patients, aiming to automate the evaluation of both the EXP and MAP dimensions of negative symptoms based on CAINS. Our dual-strategy approach uses ML for EXP assessment and GenAI with zero-shot prompting for MAP assessment. The integration of LLMs enhances transparency by providing detailed reasoning alongside severity scores. Validated on a large dataset of 160 patients, our method demonstrated state-of-the-art performance with notable reliability, making it a promising tool for improving the consistency and efficiency of negative symptom assessments.

Author Contributions

C.-M. Liu and Y.-H. Chan contributed equally and are considered co-first authors of this work.

Funding

This work was supported by the National Science and Technology Council (NSTC 111-2320-B-002-043-MY2, NSTC 113-2119-M-033-001) and National Taiwan University Hospital (NTUH 112-S0148, 114-CTC0015).

Conflicts of Interest

None declared.