Ecological assessment of transdiagnostic clinical symptoms in serious mental illness with daily smartphone surveys

Abstract

Clinical symptoms in serious mental illness (SMI) fluctuate dynamically, yet standard interview-based assessments are limited in tracking these changes. Smartphone-based ecological surveys offer a scalable approach to monitoring symptoms in naturalistic settings. We analyzed intensive longitudinal data from 56 outpatients with psychotic or affective disorders who contributed 3901 daily surveys and 423 clinical assessments over one year or longer. The daily survey items assessed a broad range of general affective, behavioral, and functional experiences. Using participant-level leave-one-subject-out cross-validation, we trained machine learning models to estimate concurrent clinical symptom severity from smartphone daily surveys. Models showed significant predictive performance for the Montgomery–Åsberg Depression Rating Scale (repeated measures correlation [r_rm] = 0.57; p < 0.001) and Young Mania Rating Scale (r_rm = 0.39; p < 0.001). Estimating positive symptoms measured by the Positive and Negative Syndrome Scale also showed significant associations (r_rm = 0.24, p < 0.001), but with variable accuracy across participants. Factor modeling demonstrated the strongest convergent validity for negative affective domains, indicating that survey responses aligned most closely with depression, anxiety, and irritability. Importantly, machine learning estimated MADRS scores tracked within-person symptom fluctuations, demonstrating sensitivity to clinically meaningful changes over time. Adherence decreased over time but was not significantly associated with symptom severity. These findings demonstrate that smartphone-based daily surveys can generate concurrent estimates of negative affective symptom severity in real-world contexts, supporting a scalable framework for extending symptom assessment beyond in-person clinical settings in psychotic and affective disorders.

Introduction

Expressions of psychopathology in individuals with serious mental illness (SMI), including psychotic and mood disorders, are highly dynamic and vary substantially across individuals [1, 2]. Clinical episodes may emerge rapidly and often lead to significant distress and functional impairment. Tracking this temporal variability in naturalistic settings remains a significant challenge as traditional interview-based assessments require trained raters, are time-intensive, and cannot be administered frequently enough to capture dynamic symptom fluctuations. There is a growing interest for reliable digital phenotyping methods using personal mobile devices that can capture these symptom fluctuations with greater temporal resolution to support the goals of personalized psychiatry [3, 4]. Such tools may offer clinical value by providing valid, scalable contemporaneous estimates of symptom severity in naturalistic settings, thereby extending clinical assessments beyond periodic in-person visits [5]. Additionally, fine-grained symptom data collected via ecological assessments may ultimately support timely clinical decision-making and inform the delivery of digital mental health interventions in real-world settings [6, 7].

Structured clinical interviews remain the gold standard for psychiatric evaluation but rely on retrospective recall over weeks or months, which may introduce memory-related biases [8, 9]. This is particularly problematic for individuals with SMI, who often experience cognitive impairments [10, 11]. For instance, prior studies have shown that individuals with depressive or psychotic disorders are likely to exaggerate negative affective experiences when recalling them retrospectively, compared to how they rate their emotions in the moment [9, 12]. Additionally, individuals with psychosis may also show reduced insight into their mental health state and functioning, further compromising the reliability of retrospective reporting, particularly in unfamiliar clinical settings [13–16].

Ambulatory self-report assessments, including daily diaries and ecological momentary assessment (EMA), offer a way to capture symptoms and experiences as they unfold in everyday life [17, 18]. While EMA traditionally involves multiple within-day prompts to assess momentary states, daily diary surveys represent a lower-frequency but conceptually similar approach, asking participants to reflect on their experiences over the preceding day. Both methods yield dense longitudinal data that capture naturalistic symptom variability, reducing recall bias and enhancing ecological validity by prompting self-report that is proximal in time to the experience [17, 19]. In recent years, the widespread ownership and portability of smartphones have made them a scalable and cost-effective platform for delivering these assessments via push notifications [20]. Importantly, prior studies show that smartphone-based ambulatory surveys are feasible and acceptable even for individuals with SMI, including those experiencing severe symptoms or low motivation [21–26].

Despite these advantages, ambulatory assessments also have limitations. First, they rely on self-report without clinical interpretation, raising concerns that cognitive or insight impairments in SMI may compromise the accuracy of symptom ratings and introduce bias [13, 14, 16, 27]. Second, high-frequency EMA protocols can be burdensome over long periods, sometimes leading to fatigue, burnout, and attrition [28]. Although evidence is mixed, it is often assumed that cognitive challenges and illness severity may further impede compliance to experience sampling protocols [25, 28–30]. Missing data from these factors can substantially reduce the usefulness of ambulatory survey responses as a symptom monitoring tool.

Evaluating the clinical utility of ambulatory surveys for symptom assessment requires two key validation approaches. First, convergent validity must be established by testing whether survey responses correspond with established interview-based clinical instruments, which remain the gold standard in psychiatry despite their own limitations. Second, ambulatory surveys must demonstrate sensitivity to within-person symptom changes over time, not just between-person differences. A clinically useful ecological tool should capture symptom fluctuations as they occur rather than only reflecting stable trait-like characteristics. This requires data collection over sufficiently long periods to observe episodes of both symptom worsening and improvement within individuals, enabling tests of whether daily surveys can track meaningful clinical changes rather than measurement noise.

Prior studies have shown that ambulatory self-report ratings correlate with validated clinical scales in SMI, including schizophrenia, bipolar disorder, and major depressive disorder [24, 31–36]. However, most were brief (a few weeks) and included few clinical assessments, focusing primarily on between-person associations (e.g., group of individuals who report higher average EMA symptoms also present higher interview-based clinical scores in average). While informative, such approaches do not fully disentangle between- and within-person effects and are limited in demonstrating whether day-to-day fluctuations ecological survey ratings covary with temporal changes in clinical symptoms within the same individual. To establish within-person convergence and better characterize dynamic symptom processes, longer study durations with repeated clinical assessments are needed.

The overarching goal of this study was to evaluate whether daily smartphone-based ecological surveys can provide valid concurrent estimates of clinician-rated symptom severity in individuals with SMI. We trained machine learning models to estimate total scores of established clinician-rated scales—the Positive and Negative Syndrome Scale (PANSS) for psychotic symptoms, the Young Mania Rating Scale (YMRS) for manic symptoms, and the Montgomery–Åsberg Depression Rating Scale (MADRS) for depressive symptoms—using daily smartphone surveys assessing affect, symptoms, functioning, and everyday general experiences in a transdiagnostic SMI sample. The smartphone survey items were adapted from a prior study designed to capture a broad range of behaviors and internal states rather than symptoms specific to a single psychiatric disorder [37]. As such, the survey provides more comprehensive coverage of general affective domains than of symptoms specific to serious mental illness (e.g., psychosis, mania, negative symptoms). Based on this construct coverage, we hypothesized a priori that predictive performance would be strongest for negative affective domains and weaker for manic and psychotic symptoms. We further anticipated that survey non-completion would increase over time but would not relate to symptom severity at the within-person level, supporting the feasibility of smartphone-based surveys for tracking clinical symptoms in naturalistic settings.

Materials / subjects and methods

Overview of study design

A total of 56 outpatients with serious mental illness were enrolled in the Bipolar Longitudinal Study between 2015 and 2022, designed to follow participants for up to one year or longer. Participants were recruited through clinician referrals from McLean Hospital’s inpatient units (at the time of discharge) and outpatient clinics, including the McLean’s Schizophrenia and Bipolar Disorder Program, and community advertisements. Research staff subsequently contacted referred patients, conducted eligibility screenings, and obtained written informed consent prior to enrollment. Diagnoses included primary psychotic disorders (e.g., schizophrenia, schizoaffective disorder) and affective disorders (e.g., major depressive disorder, bipolar I & II). Participants completed screening, baseline assessments, and monthly follow-up clinical assessments in person or via video conference call. Trained raters conducted monthly semi-structured interviews using standardized instruments: the PANSS [38] for psychotic symptoms, YMRS [39] for manic symptoms, and the MADRS [40] for depressive symptoms.

Ethics approval and consent to participate

The study protocol was approved by the Mass General Brigham (MGB) Institutional Review Board (2015P002189) and informed consent was obtained from all subjects. All procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration.

Smartphone survey data collection

Participants installed the Beiwe [41] smartphone application to complete daily surveys originally designed to assess daily experiences in the general population and adapted from our prior study in college students [37]. Survey items were designed to assess a broad range of behaviors and internal states over the past 24 hours, including positive and negative affect, social activity, stress levels, sleep quality, substance use, physical health, and general functioning, along with selected psychosis-related experiences. The full list of survey items and their corresponding constructs is provided in the Supplementary Materials. Beiwe is an open-source, secure web-based platform designed for digital phenotyping, offering compatibility with both Android and iOS devices. Beiwe offers cloud-based storage compliant with HIPAA standards, and a comprehensive back-end for data processing and analysis [41]. For this study, Beiwe was programmed to deliver daily notifications at 5:00 PM Eastern Time, prompting participants to complete a 32-item, and to submit one survey per day.

Statistical analyses

Smartphone daily survey-derived features

To estimate clinical symptom scores from ecological smartphone survey data, we extracted survey responses from the 14-day window preceding each clinical interview, requiring a minimum of 3 completed surveys for inclusion. The 14-day window was selected a priori to align with standard clinical assessment timeframes, as the clinical instruments typically cover the preceding one to two weeks. The minimum threshold of three surveys was chosen to maximize the number of observations available for model training as sample size is critical for generalizable machine learning models; stricter thresholds substantially reduced the sample size. For each eligible clinical assessment, we derived summary features from the corresponding survey data, including the mean, standard deviation, minimum, and maximum values, to capture both average levels and within-window variability in symptom expression. We also conducted post hoc sensitivity analyses varying observation window duration (7 vs. 14 days) and minimum survey thresholds (3 vs. 5 surveys) to evaluate the robustness of our findings; results are reported in the Supplementary Materials.

Machine learning model development and evaluation

Machine learning model development was conducted using the validated PHOTON AI package [42]. We evaluated three estimators, elastic net regression, support vector regression (SVR), and random forest regression (RFR), to identify the optimal model for each outcome. Models were trained to minimize mean absolute error (MAE), using participant-level nested leave-one-subject-out cross-validation to prevent data leakage and ensure generalizability. Therefore, when making estimation for each participant with multiple observations, none of the person’s own observations at any point were used. Hyperparameters were optimized using random grid search with 25 iterations in the inner loops. Statistical significance of model performance was assessed using permutation testing as implemented in scikit-learn. For each target, the labels were permuted at the participant level. For feature importance, 1000 permutation was performed within the PHOTON-AI framework. Here, individual predictor variables were permuted while keeping the trained model fixed, and the resulting change in MAE was quantified. This procedure evaluates the contribution of each feature to prediction performance without altering the cross-validation structure.

We used repeated measures correlation (rmcorr) to assess the within-subject association between estimated and observed values across multiple time points [43]. Correlation coefficients were interpreted based on conventional thresholds in behavioral and psychological sciences: values between 0.00–0.10 were considered negligible, 0.10–0.39 weak, 0.40–0.69 moderate, 0.70–0.89 strong, and 0.90–1.00 very strong correlations [44].

To evaluate model performance while accounting for the repeated-measures structure of the data, we report the marginal R² values, which represent the proportion of variance in observed scores explained by the model estimates, independent of variance attributed to random effects. To further characterize within-person variability in model performance, we computed Pearson correlations between estimated and observed scores for participants with at least five clinical assessments.

Daily survey item associations

We used Bayesian multilevel modeling approach using the brms package in R to examine survey item-level associations between daily survey responses and clinical symptom ratings [45]. Univariate models were fit separately for each clinical total score using the mean value of each corresponding survey item over a specified time window. Both between-person and within-person effects were estimated simultaneously using standardized slopes. We adopted a univariate modeling framework to enhance interpretability, as feature importance derived from multivariate machine learning models can be difficult to interpret due to multicollinearity and regularization effects.

In the multilevel modeling framework, between-person effects reflected associations between each participant’s average survey responses across all timepoints and their overall symptom severity, whereas within-person effects captured associations between day-to-day deviations from each participant’s mean and concurrent symptom fluctuations. Models included fixed effects for age and sex and random intercepts and slopes to account for individual variability in baseline levels and trajectories. A skew-normal distribution was used to accommodate asymmetric outcome distributions, and weakly informative priors were defined based on observed means and standard deviations. Posterior estimates were summarized using 89% highest density intervals (HDIs), consistent with Bayesian reporting conventions [46]. A detailed description of the modeling framework is available in a prior publication [47], and model specifications are provided in the supplementary materials.

Estimating transdiagnostic symptom dimensions

As a secondary analysis, we trained machine learning models to estimate transdiagnostic symptom factor scores derived from structured clinical assessments. This approach enabled us to assess the extent to which specific latent symptom constructs could be tracked using daily surveys. Latent factor scores were based on a previously validated model that aggregates items from the PANSS, MADRS, and YMRS, developed in a diagnostically heterogeneous sample that closely mirrors the diagnostic composition of the current cohort [48]. Factor scores were computed for a correlated-factors model comprising seven symptom dimensions: positive symptoms, negative symptoms, depression, mania, disorganization, hostility, and anxiety. Individual-level factor scores were estimated using the Empirical Bayes method, implemented in the lavaan R package [49].

Model performance by participant characteristics

To evaluate whether predictive performance varied by diagnosis or sociodemographic characteristics, we compared MAE between participants with a primary psychotic disorder and those with an affective disorder. Primary psychotic disorders included schizophrenia and schizoaffective disorder, while affective disorders included bipolar I disorder, bipolar II disorder, and major depressive disorder, based on DSM-5 diagnoses [50]. Covariates included age, sex, education level, race, overall EMA missingness, and the mean total scores of each clinical rating scale across all study visits. False discovery rate correction was applied to account for multiple comparisons.

Ambulatory survey missingness

Adapting the Bayesian multilevel modeling approach described above, we examined whether daily survey completion rate (i.e., the percentage of daily surveys completed during the 14 days preceding each clinical visit) was associated with each clinical total scores. Number of days since study enrollment, age, and sex were included as covariates.

Results

Sociodemographic and BLS data summary

The included sample had a mean age of 30.1 years (SD = 8.8; range = 19–61), and 55.4% identified as female. Consistent with local demographics, the majority of participants were White (64.3%), followed by Asian (16.1%) and African American (14.3%). A total of 89.3% of participants had completed at least some college education. Regarding primary diagnosis, 30% (n = 17) were classified with a psychotic disorder (schizophrenia or schizoaffective disorder), and 70% (n = 39) with an affective disorder (bipolar I, bipolar II, or major depressive disorder).

Across the sample, 1028 clinician-administered interviews were completed, with a median of 12 sessions per participant (SD = 13.0; range = 1–63). A total of 12,984 daily smartphone surveys were submitted, with a median of 81 daily surveys per participant (SD = 245, range = 2-1253). Sociodemographic and study characteristics stratified by diagnostic group are summarized in Table 1.

Table 1.

Sociodemographic table of participants with primary psychosis and affective disorder.

Characteristic	Primary Psychosis	Affective Disorder
Count: n (%)	17 (30%)	39 (70%)
Study Duration: Median Weeks (SD; range)	33 (64; 1–304)	40 (54; 5–249)
Clinical Assessments per Participant: Median (SD; range)	12 (13; 2–46)	12 (12; 1–63)
Daily Survey Responses per participant: Median (SD; range)	111 (248; 14–1029)	86 (253; 3–1029)
Age: Mean (SD)	31 (8.4)	29.7 (9.1)
Sex: n (%) Female	4 (23.5%)	27 (69.2%)
Race
White: n (%)	9 (52.9%)	27 (69.2%)
African American: n (%)	4 (23.5%)	4 (10.3%)
Asian: n (%)	3 (17.6%)	6 (15.4%)
AI/AN: n (%)	1 (5.9%)	0 (0%)
Other: n (%)	0 (0%)	2 (5.1%)
Education
Unknown: n (%)	0 (0%)	1 (2.6%)
No high school Diploma: n (%)	1 (5.9%)	0 (0%)
High School Diploma or equivalent: n (%)	3 (17.6%)	1 (2.6%)
Some College: n (%)	7 (41.2%)	12 (30.8%)
2-Year College: n (%)	1 (5.9%)	3 (7.7%)
4-Year College: n (%)	3 (17.6%)	15 (38.5%)
Some Graduate School: n (%)	1 (5.9%)	1 (2.6%)
Graduate Degree: n (%)	1 (5.9%)	6 (15.4%)
Primary Diagnosis
Schizophrenia: n (%)	7 (41.2%)	0 (0%)
Schizoaffective Disorder: n (%)	10 (58.8%)	0 (0%)
Bipolar Disorder 1: n (%)	0 (0%)	25 (64.1%)
Bipolar Disorder 2: n (%)	0 (0%)	3 (7.7%)
Major Depressive Disorder: n (%)	0 (0%)	11 (28.2%)
Clinical Scale Total Scores
MADRS: Mean (SD)	10.8 (6.6)	15.4 (9)
YMRS: Mean (SD)	7.7 (5.4)	7.1 (4.8)
PANSS Positive: Mean (SD)	12.4 (4.6)	9.4 (2.4)
PANSS Negative: Mean (SD)	12.1 (3.6)	9.8 (1.9)

For each participant, the total number of completed clinical assessments and daily survey responses was calculated; table values report the median (SD) across participants within each diagnostic group.

AI/AN, American Indian or Alaskan Native.

Optimization of observation time windows

The 14-day window with a minimum of 3 completed surveys, selected a priori as described in the Methods, yielded the largest sample (n = 423 observations). Post hoc sensitivity analyses comparing alternative windows (7-day vs. 14-day) and availability thresholds (3 vs. 5 minimum surveys) demonstrated comparable model performance across conditions, supporting the robustness of these analytic parameters (see Supplementary Materials).

Estimating interview-based clinical assessment scores

Predictive model performance was comparable across the three machine learning algorithms, with elastic net regression achieving slightly lower MAE than SVR and RFR. Elastic net results were therefore used for all subsequent analyses. As shown in Fig. 1, prediction accuracy was highest for MADRS scores (marginal R² = 0.32; 95% CI [0.25, 0.40]; repeated-measure correlation (r_rm) = 0.57, p < 0.001), indicating moderate predictive performance. Performance for the YMRS was weak (marginal R² = 0.12; 95% CI [0.07, 0.20]; r_rm = 0.39, p < 0.001). Prediction of PANSS Positive scores yielded weak performance (marginal R² = 0.04; 95% CI [0.01, 0.07]; r_rm = 0.24, p < 0.001), while predictive accuracy for PANSS Negative was negligible (marginal R² = 0.002; 95% CI [0.00, 0.02]; r_rm = 0.02, p = 0.66).

Fig. 1. Daily smartphone surveys estimate interview-based clinical symptom severity.

Observed interview-based clinical rating scores are plotted against symptom severity estimates derived from daily smartphone surveys using elastic net regression (n = 56 participants; 423 observations). R² represents the marginal coefficient of determination from the mixed-effects model, and r_rm denotes the repeated-measures correlation.

We also evaluated whether temporal features, such as slope (linear trend) and root mean square of successive differences (RMSSD), improved model performances. However, these features did not contribute to model accuracy and were not included in the final models. This may be due to the sparse and irregular sampling of daily survey data, which limits the reliability of sequential trend estimation. Linear regression model fits were also stratified by diagnostic group (primary psychosis vs. affective disorder), and no significant group-by-slope interactions were observed. Additional model performance metrics are provided in Table 2 and the Supplementary Materials.

Table 2.

Model performance summary for estimating clinical scale scores and factor-based symptom scores.

Target	MAE	RMSE	rrm	Marginal R2	Within-Person Corr
Clinical Scales
MADRS	5.60	7.07	0.57***	0.32 [0.25, 0.40]	0.48 (0.29)
YMRS	4.60	6.00	0.39***	0.12 [0.07, 0.20]	0.42 (0.23)
PANSS Positive	2.82	4.19	0.24***	0.04 [0.01, 0.07]	0.17 (0.33)
PANSS Negative	2.79	3.63	0.02	0.00 [0.00, 0.02]	0.01 (0.33)
Factor Dimensions
Depression	3.37	4.22	0.43***	0.40 [0.29, 0.50]	0.38 (0.33)
Anxiety	3.84	4.74	0.41***	0.40 [0.27, 0.51]	0.30 (0.35)
Mania	4.22	5.27	0.21***	0.08 [0.04, 0.18]	0.05 (0.28)
Hostility	4.03	4.86	0.32***	0.18 [0.09, 0.28]	0.21 (0.29)
Positive Symptoms	3.40	4.76	0.31***	0.16 [0.06, 0.31]	0.23 (0.43)
Negative Symptoms	3.79	4.85	0.23***	0.07 [0.02, 0.15]	0.05 (0.30)
Disorganization	4.05	5.14	0.19***	0.06 [0.01, 0.13]	0.22 (0.35)

MADRS, Montgomery–Åsberg Depression Rating Scale; YMRS, Young Mania Rating Scale; PANSS, Positive and Negative Syndrome Scale; MAE, Mean Absolute Error; RMSE, Root Mean Squared Error. R_rm, Repeated measure correlation between observed and estimated clinical symptom scores.

Significance levels: ***p < 0.001, **p < 0.01, *p < 0.05. Factor symptoms dimensions are based on Chung et al., 2024. Within-Person Correlation represent the mean (SD) of within-person Pearson correlations across individuals.

To evaluate prospective prediction, we conducted additional time-lagged analyses using survey windows ending 3, 7, and 14 days prior to clinical assessment. MADRS and YMRS predictions remained statistically significant across all lags, with a general trend of decreasing accuracy at longer time lags (see Supplementary Materials for details).

Estimating transdiagnostic symptom dimensions

To examine which specific symptom dimensions were most closely tracked by daily survey data, we applied Elastic Net regression models to transdiagnostic factor scores derived from a previously validated dimensional model of SMI [48]. Among all symptom domains, the depression (marginal R² = 0.40; 95% CI [0.29, 0.50]; r_rm = 0.43, p < 0.001) and anxiety (marginal R² = 0.40; 95% CI [0.27, 0.51]; r_rm = 0.41, p < 0.001) factors demonstrated the highest predictive performance, both in the moderate range. Predictive performance for mania-related dimensions was weaker, with the mania factor showing limited accuracy (marginal R² = 0.08; 95% CI [0.04, 0.18]; r_rm = 0.21, p < 0.001) and the hostility factor showing better performance (marginal R² = 0.18; 95% CI [0.09, 0.28]; r_rm = 0.32, p < 0.001). Among psychosis-related domains, positive symptoms were predicted at a weak level (marginal R² = 0.16; 95% CI [0.06, 0.31]; r_rm = 0.31, p < 0.001), whereas predictive performance for negative symptoms (marginal R² = 0.07; 95% CI [0.02, 0.15]; r_rm = 0.23, p < 0.001) and disorganization (marginal R² = 0.06; 95% CI [0.01, 0.13]; r_rm = 0.19, p < 0.001) were limited. Model evaluation metrics for all transdiagnostic symptom dimensions are summarized in Table 2, with additional details provided in the Supplementary Materials.

Within-person linear associations

Given the robustness of the predictive model for MADRS, we next evaluated its ability to capture temporal variability in MADRS scores at the within-person level. Pearson correlations were calculated within each participant who had at least five clinical assessments (n = 28 out of 56) to characterize within-person linear associations between estimated and observed scores. As shown in Fig. 2, the distribution of within-person correlations for MADRS scores showed moderate-level associations (median r = 0.58, median SD = 0.29), suggesting that the machine learning model can estimate MADRS scores over time and track changes at the within-person level. To further illustrate this, we visualized estimated and observed MADRS scores over time for a subset of participants with more than 10 assessments spanning at least one year. A full summary of within-person correlation results for other clinical scales and factor scores is provided in Table 2.

Fig. 2. Smartphone survey–based estimates track interview-based clinical symptom severity and change.

A Distribution of within-person correlations between observed and estimated MADRS scores using elastic net, shown as a box plot. Example trajectory scatter plots from six participants with varying degrees of model accuracy are plotted. B Estimated change between successive clinical assessments (Δ) derived from daily smartphone surveys plotted against observed changes in interview-based clinical rating scores. Marginal distributions are added along each axis.

Tracking changes in depressive symptoms

We examined whether the model could detect changes in depressive symptoms between subsequent clinical interviews. Change scores (Δ) were computed by calculating the difference in each participant’s observed and predicted MADRS scores at a given clinical visit relative to their immediately preceding visit. As shown in Fig. 2B, predicted and observed change scores were moderately correlated (marginal R² = 0.23, r_rm = 0.49, p < 0.001).

Participant characteristics and model performance

No significant differences in MAE were observed between participants with primary psychosis and those with affective disorders across all symptom domains (all p > 0.05). Likewise, individual-level predictive performance, assessed by within-person correlations, did not differ significantly between diagnostic groups (p > 0.05). Other covariates, including age, sex, education, race, overall EMA missingness, and average clinical symptom severity (MADRS, YMRS, PANSS), were not associated with MAE.

Daily survey item associations

Based on Bayesian multi-level regression analyses, survey item-level associations for MADRS, YMRS, PANSS Positive, and PANSS Negative total scores are shown in Fig. 3. Higher average ratings of all negative valence items (e.g., anxious, irritable, upset) and feeling “stressed” were significantly associated with greater MADRS score at both the between-person and within-person levels. Conversely, lower ratings of positive valence (e.g., feeling energetic, happy) and lower perceived ability to manage stress were similarly associated with higher MADRS scores at both levels. At the between-person level, greater endorsement of physical or somatic symptoms (e.g., breathing difficulty, heart racing, pain) and lower levels of social engagement (e.g., in-person or digital interaction, feeling outgoing) were also associated with more severe depressive symptoms. Psychosis-related survey items (e.g., auditory or visual hallucinations, paranoid thoughts) were not significantly associated with MADRS scores at either level.

Fig. 3. Associations between smartphone-based daily survey items and interview-based clinical scale ratings at the between- and within-person levels using Bayesian multilevel modeling approach.

Points represent posterior medians, and horizontal bars indicate 89% HDIs. Red indicates significant positive associations; blue indicates significant negative associations.

Daily survey missingness

Based on the Bayesian multilevel regression framework described above, higher within-person MADRS scores were associated with greater survey completion rate (14 days prior to each clinical assessment; standardized slope: Median = –0.037, 95% CI [–0.080, –0.001], pd = 0.98). In contrast, between-person variation in MADRS scores was not related to survey completion rate (Median = 0.010, 95% CI [–0.133, 0.163], pd = 0.56). Among covariates, longer study days since onboarding was significantly associated with reduced daily survey completion (Median = 0.0003, 95% CI [0.0002, 0.0005], pd = 1.00). Age, sex, and primary diagnosis were not significant predictors of study completion rate. Similarly, within- and between-person effects of YMRS, PANSS Positive, and PANSS Negative scores were all non-significant, with study duration presenting as the only consistent predictor of daily survey missingness across models.

Discussion

A novel aspect of this study was the intensive longitudinal design in which individuals with SMI, recruited from a transdiagnostic sample, completed repeated multidimensional clinical assessments while in parallel submitting daily smartphone-based surveys over an extended period. This design provided an ecologically valid framework to characterize the natural course of fluctuating symptoms in real-world contexts. Leveraging this unique dataset, we evaluated whether daily smartphone surveys could reliably estimate clinical symptom severity, with focus to the capacity of predictive models to capture within-person changes over time.

Depressive symptoms

Among the clinical symptom scales evaluated, the machine learning model estimating MADRS total scores showed the strongest correspondence with observed clinical ratings. As expected, survey items reflecting negative affect (e.g., feeling lonely, upset, or irritable) and perceived stress were positively associated with MADRS scores at both between- and within-person levels. A 6–9 point change on the MADRS is considered clinically meaningful [51]. In our longitudinal dataset, we identified multiple instances in which participants’ depressive symptoms exceeded this threshold and confirmed that the model could track these changes at the within-person level, underscoring its potential utility for remote monitoring of clinically significant shifts in depressive symptom severity.

A key advantage of our predictive modeling approach is its ability to generate interpretable estimates of symptom severity on standardized clinical scales using daily survey data collected proximal to clinical assessments. Rather than directly translating items from clinical rating scales, the model leverages a broad range of smartphone-based self-report measures of affect and functioning, enabling scalable and low-burden assessment in naturalistic settings. Notably, model performance did not differ significantly between individuals with primary psychotic disorders and those with affective disorders, supporting the transdiagnostic applicability of this approach.

Manic symptoms

Machine learning model for estimating YMRS scores showed modest performance (r_{rm =} 0.39, p < 0.01). Manic presentations often involve comorbid symptoms, including elevated mood, hostility, and irritability, all of which are evaluated with the YMRS. When examining factor scores [48], the hostility factor showed substantially better predictive accuracy than the mania factor, suggesting that the model performance was primarily driven by hostility and irritability, rather than core manic symptoms (i.e., elevated mood and high energy). Consistent with this, daily survey items rating hostility and irritability were significantly associated with YMRS scores at both the between- and within-person levels, whereas items reflecting elevated mood (e.g., feeling inspired, determined, or energetic) were associated only at the within-person level. Similarly, behavioral indicators of mania, such as increased physical activity and reduced sleep, were also associated with YMRS at the within-person levels. These findings may reflect the transient nature of core manic symptoms, which merge actuely rather than as stable trait-like differences.

Our findings are consistent with prior work (e.g., Busk et al.) [52], demonstrating that smartphone surveys can estimate both depressive and manic symptoms, with stronger performance for depressive domains. In the present study, the relatively modest predictive performance for YMRS is likely attributable, in part, to limited variability in manic symptom severity. Although a substantial proportion of participants carried a bipolar I diagnosis, most outpatients were clinically stable over the extended study period and more frequently endorsed depressive rather than manic symptoms. Consequently, relatively few surveys and clinical assessments captured periods of elevated manic states, constraining the model’s ability to learn and predict manic symptom variation.

Additionally, the underrepresentation of severe manic episodes may reflect limitations in the temporal resolution of clinical assessments. Given that clinician-rated interviews were conducted at relatively infrequent intervals, transient or rapidly fluctuating manic states may not have been captured at the time of assessment, leading to underrepresentation of elevated manic symptom severity in the outcome measures. While our findings suggest that individuals with SMI remain engaged with smartphone-based assessments during periods of low mood, it remains an open question whether similar adherence is maintained during acute manic states. Future studies that include individuals with greater manic symptom severity and incorporate targeted assessment of manic features may improve model performance in this domain.

Psychotic symptoms

The machine learning model estimating PANSS Positive scores was statistically significant but explained only modest variance. Three survey items assessed positive symptoms (visual hallucinations, auditory hallucinations, and paranoid thoughts). Consistent with prior studies [24, 34], individuals with higher PANSS Positive scores tended to endorse these psychosis-related items with greater severity; however, within-person associations were highly variable. While some individuals showed reasonable correspondence between survey responses and PANSS Positive ratings over time, others did not. For example, several participants with elevated PANSS Positive scores did not endorse any positive symptom items. This misalignment may reflect reduced insight or differences in how individuals interpret and report psychosis-related experiences in self-report formats. For instance, participants may not recognize certain experiences as symptoms, may normalize or minimize them, or may interpret survey items differently from clinicians, leading to discrepancies between self-reported survey responses and clinician-rated assessments [53, 54].

The model for PANSS Negative symptoms explained negligible variance. This was expected, as the study did not include patients with prominent negative symptoms and the survey lacked coverage of core domains such as motivation and anhedonia. The only notable association was a between-person level association linking reduced social engagement (e.g., feeling less socially engaged or outgoing) to higher PANSS Negative scores. Prior studies have shown that EMA items targeting negative symptoms correlate well with validated clinical scales such as the PANSS Negative or Clinical Assessment Interview for Negative Symptoms, and achieved acceptable adherence [23, 24, 55]. Within-person changes, however, have not been systematically investigated, and future studies are needed to evaluate whether these associations hold over time.

Ecological survey missingness

As expected, adherence declined over time since enrollment. However, symptom severity or sociodemographic variables were not associated with survey compliance. Contrary to expectations, participants showed modestly higher compliance rate during periods of elevated depressive symptoms relative to their personal average. While further research needs to determine the generalizability of this finding, our results offer encouraging evidence that individuals with SMI can remain engaged with smartphone-based surveys during periods of worsened depressive mood. For manic and psychotic symptoms, no relationship between missing surveys and symptom severity was observed, although clinical rating in these domains showed limited variability and were skewed toward minimal to moderate levels. Additional observations during periods of heightened mania or psychosis are needed to clarify whether symptom exacerbation influences survey engagement. Future studies should evaluate strategies to enhance engagement, including incentives that provide direct value to participants.

Limitations

Several limitations should be noted. First, our models used standard machine learning algorithms that assume independent and identically distributed observations, which do not fully capture the repeated-measures nature of the data. Although this may limit the ability to model within-person dependencies and temporal dynamics, we prioritized these approaches to ensure generalizability to unseen observations, including new individuals without prior data. Future work should evaluate mixed-effects or hierarchical ML frameworks capable of estimating subject-specific random effects and updating predictions as additional within-person data accumulate. Second, model performance declined at the extremes of symptom severity, with a systematic pattern of underestimation at higher symptom levels and overestimation at lower levels. This compression toward the mean likely reflects regression-to-the-mean effects common in regression-based and machine learning models, as well as restricted variability and skewed distributions of clinical ratings toward lower symptom levels. Together, these factors limit the model’s ability to accurately capture less frequent extreme states, resulting in increased prediction error at both ends of the severity spectrum. Third, although we found no evidence that elevated symptom severity reduced survey compliance, our analyses could not account for missed assessments due to hospitalization or acute clinical episodes. Consequently, missingness analyses may have underestimated disengagement linked to clinical worsening, biasing results toward more stable participants and limiting generalizability to those with greater symptom fluctuations. Fourth, the number of observations per participant showed a right-skewed distribution, with most participants contributing multiple observations and a smaller subset contributing longer than a year follow-up. Although participant-level cross-validation guards for optimistic bias in model evaluation, participants with more repeated assessments may have had greater influence during model training. Finally, the stronger performance observed for MADRS scores likely reflects the greater number of survey items capturing negative affect, providing a richer predictor pool. This study did not determine the optimal number or content of survey items needed to estimate symptom severity across domains. Future research should identify validated, minimal item sets that preserve predictive accuracy while reducing participant burden and supporting long-term engagement.

Conclusions

This study demonstrates that daily smartphone-based surveys can generate valid concurrent estimates of negative affective symptom severity in individuals with serious mental illness. Predictive model for MADRS showed moderate performance with sensitivity to within-person changes over time, while predictive accuracy for manic and psychotic symptoms was more limited. As prior studies suggest [23, 26, 52], more targeted survey content and samples capturing greater symptom variability may improve model performance in these domains. Importantly, our participant-level cross-validation approach supports a scalable “cold-start” framework in which symptom estimates can be generated for new individuals without prior individual-level data. Such approach could address a key barrier to real-world deployment, where collecting individual-level training data is often costly or impractical. Future work should explore integrating passive sensor data with active self-report to improve precision across symptom domains and reduce participant burden [56, 57].

Author contributions

YC: Conceptualization, Methodology, Formal analysis, Visualization, Writing-original draft, BWG: Data curation, Formal analysis. HR: Conceptualization, Data curation, Writing-Reviewing and Editing; VLH: Conceptualization, Formal analysis, Software, Writing, Reviewing and Editing ; JMG: Methodology; DO: Conceptualization, Funding acquisition, Writing- Reviewing and Editing; JPO: Conceptualization, Methodology, Writing-Reviewing and Editing; SR: Conceptualization, Funding acquisition, Writing-review editing; EL: Conceptualization, Funding acquisition, Writing - review & editing, JB: Conceptualization, Data curation, Funding acquisition, Supervision, Project administration, Writing - review & editing.

Data availability

Mass General Brigham requires IRB approval and a signed Data Use Agreement for data sharing Please contact the corresponding author.

Code availability

Machine learning analyses were conducted using PHOTONAI (https://github.com/wwu-mmll/photonai).

Competing interests

DO has received honoraria from Boehringer-Ingelheim in the past 12 months for unrelated work.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41398-026-04218-9.