Evaluating Psychometric Validity of Ecological Momentary Assessment for Youth Irritability

Abstract

Background:

Irritability is a transdiagnostic symptom in youth, leading to long-term adverse consequences. Ecological momentary assessment (EMA), or naturalistic clinical phenotyping, can quantify real-world experiences and affective dynamics of irritability in vivo and may be less contaminated by biases that impact retrospective report measures. However, to date, no research examines the psychometric properties of EMA measures of irritability.

Methods:

The current study assesses EMA data from N = 49 youth receiving treatment for clinically impairing irritability (Mage = 10.63, 36.73% female). Analyses evaluate two irritability EMA items and one positive-affect EMA item for within and between person variability, internal consistency, test-retest reliability, and convergent and discriminant validity.

Results:

EMA items demonstrate acceptable psychometric properties including acceptable variability, consistency, reliability, and convergent and discriminant validity. Comparisons to previous work in anxious and healthy youth are discussed.

Limitations:

Study limitations include participants’ concurrent involvement in treatment and exclusion of outburst-related EMA measures from study analyses.

Conclusion:

These results may facilitate future research with irritability EMA items in clinical populations; future work should validate EMA items psychometrically before use in clinical trials.

Introduction

Irritability, or an increased proneness to anger relative to peers (Brotman et al., 2017), is a common reason that youth and families seek psychiatric care (Collishaw et al., 2010; Evans et al., 2023) and predicts long-term adverse clinical, academic, and socioeconomic outcomes (Stringaris et al., 2009). Irritability is a transdiagnostic clinical phenomenon, implicated in multiple psychiatric diagnoses (Leibenluft, 2017). Irritability involves elevated propensity to frustration and displays of anger following blocked goal attainment (Leibenluft, 2017). Ecological momentary assessment (EMA) (Shiffman et al., 2008) leverages smartphone technology to repeatedly assess experiences in the moments they occur; it can quantify youths’ real-world experiences of irritability (Naim et al., 2021) in ways that have advantages over retrospective report (Trull & Ebner-Priemer, 2013). However, EMA items for irritability have yet to be evaluated psychometrically to investigate internal validity and reliability for use within clinical research and trials (Byrne et al., 2023; Forkmann et al., 2018; Mestdagh & Dejonckheere, 2021).

EMA has grown in popularity in youth psychopathology research in the last decade (Henry et al., 2024) and offers many advantages compared to traditional self-report, parent-report, and clinician-report measures to assess youth irritability. Assessments occur within the participant’s natural environment, enhancing ecological validity (Verhagen et al., 2016) and decreasing the probability of retrospective recall bias (Trull & Ebner-Priemer, 2013). Hypothesized components of irritability include temporal and contextual components (e.g., pervasive mood states, triggered outbursts) that may be better characterized by EMA methods than retrospective report (Evans et al., 2023; Naim et al., 2021; Naim et al., 2022). Further, affective dynamics themselves (e.g., increased mood lability) may be clinically relevant in youth with irritability and are associated with functional impairment (Dejonckheere et al., 2018; Naim et al., 2022; Olthof et al., 2023). EMA can also capture within person change in irritable affect over time rather than only comparing irritability between participants (Verhagen et al., 2016). Thus, EMA may be particularly well suited to increase understanding of the phenomenology of youth irritability (Naim et al., 2021).

As irritability EMA research advances, it is crucial to assess the psychometric validity and reliability of new measures. Research assessing youth irritability via EMA is still nascent (Evans et al., 2023; Flynn et al., 2021; Naim et al., 2021). Past studies have adapted items from previously validated self and parent-reported questionnaires, including the Affective Reactivity Index (Stringaris et al., 2012) and the Positive and Negative Affect Scale (Thompson, 2007), into EMA items. Existing EMA items adapted from the parent- and child- Affective Reactivity Index metrics are necessarily altered to capture momentary affect (Naim et al., 2021). Changes in EMA items and context necessitate additional validation of adapted irritability EMA items to avoid threats to both the replicability and internal validity of irritability EMA studies (Byrne et al., 2023; Flake & Fried, 2020; Mestdagh & Dejonckheere, 2021). Past work has assessed irritability EMA items adopted from the Affective Reactivity Index (Stringaris et al., 2012) in relation to child, parent, and clinician-reported assessments and a Research Domain Criteria (RDoC)-informed behavioral task measuring frustrative nonreward (Naim et al., 2021), as well as items’ dynamics (Naim et al., 2022) and network structure (Tseng et al., 2023). In clinically irritable youth, recent work has demonstrated changes in EMA-assessed irritability symptoms during treatment for irritability (Naim et al., in press). However, to date no research has examined the between and within person psychometric properties of irritability EMA items in clinically irritable youth. Psychometric analysis will support the internal validity of irritability EMA measures for future use in clinical research and trials. Importantly, measures must be validated in clinically irritable populations, as well as compared to other clinical and non-clinical youth, to ensure their suitability across symptomatology.

A previous study by Byrne et al. (2023) examined within and between person psychometric properties of anxiety EMA items in anxious and healthy youth, and found evidence for satisfactory psychometrics in youth with anxiety but not in healthy youth, demonstrating the need to compare items across clinical groups. We use data from the same EMA study protocol as Byrne et al. (2023) to examine the between and within person psychometric properties of EMA items capturing irritability in an independent sample of youth with clinically significant irritability who completed EMA measures while completing treatment for irritability (Naim et al., 2024; Naim et al., in press). The current study leveraged the same approach as Byrne et al. (2023), and examined irritability items’ between and within person variability, internal consistency, test-retest reliability across two EMA intervals, and convergent and discriminant validity. To facilitate comparison across irritable, anxious, and healthy youth in our sample and that of Byrne et al. (2023), supplemental analyses examined the psychometric properties of anxiety and mood related items in youth with irritability. Consistent with prior work on anxiety EMA items (Byrne et al., 2023), we hypothesized that irritability EMA items would demonstrate acceptable psychometric properties in youth with irritability, including moderate to good test-retest reliability, good convergent and discriminant validity, and sufficient between- and within-person internal consistency as determined by accepted standards for each of our analyses.

Methods

Participants

Participants included 49 youth with chronic irritability ages 8–17 years recruited as part of a larger study protocol in the Washington, D.C. area (Naim et al., 2024; Naim et al., in press). All youth received twelve sessions of exposure-based cognitive behavioral therapy (CBT) for clinically impairing irritability. Sample size was based on availability of data from a completed study treatment cohort (Naim et al., 2024) and an ongoing follow-up treatment study (see Table 1 for demographic information). The sample also overlaps with a recently published study examining changes in EMA-assessed youth irritability symptoms and parent behaviors during CBT treatment for irritability (Naim et al., in press). Full recruitment and screening procedures are outlined in Naim et al. (2024). Eligible youth presented with at least one core symptom of disruptive mood dysregulation disorder (DMDD; i.e., chronically irritable mood and/or temper outbursts), with at least moderate impairment in two or more domains (home, school, and peers) (Naim et al., 2023). While youth frequently presented with multiple diagnoses, primary diagnosis was determined based on youths’ chief presenting complaint and clinical judgement of the diagnosis with greatest impairment (Naim et al., 2021) (see Table 1).

Table 1.

Demographic and clinical characteristics of N = 49 youth with clinically impairing irritability. Female = sex assigned at birth. Primary Diagnosis = diagnosis of greatest impairment. ARI = Affective Reactivity Index. DMDD = Disruptive Mood Dysregulation Disorder. ODD = Oppositional Defiant Disorder. ADHD = Attention Deficit Hyperactivity Disorder.

	Total Sample (N = 49)
Female, n (%)	18 (36.73)
Age (years), mean (SD)	10.63 (1.83)
Race, n (%)
Asian	1 (2.04)
American Indian or Alaskan Native	0 (0.00)
Black/African American	4 (8.16)
Multiple Races	3 (6.12)
White	38 (77.55)
Not reported	3 (6.12)
Ethnicity, n (%)
Latino or Hispanic	4 (8.16)
Not Latino or Hispanic	42 (85.71)
Not reported	3 (6.12)
Primary Diagnosis, n (%)
DMDD	24 (48.98)
ODD	12 (24.48)
ADHD	13 (26.53)
Interval 1 Questionnaires, mean (sd)
ARI-S 1 wk Total	4.29 (3.42)
ARI-P 1 wk Total	7.63 (2.67)
Clinician ARI Total	39.38 (14.13)
Interval 2 Questionnaires, mean (sd)
ARI-S 1 wk Total	4.45 (3.06)
ARI-P 1 wk Total	5.65 (2.61)
Clinician ARI Total	33.87 (15.88)

Procedures

All study procedures were approved by the National Institute of Mental Health Institutional Review Board. Participants completed a larger smartphone-based EMA protocol assessing multiple dimensions of irritability, mood, and anxiety symptoms and their situational contexts (Naim et al., 2021). Our study exclusively used iPhones; if participants did not have an iPhone available, participants received a study-provided iPhone. Prior to study participation, research assistants provided a standardized EMA training to ensure familiarity with the study protocol and smartphone, reviewing each EMA item, completing a practice prompt, and ensuring that EMA prompts worked on participants’ devices (Naim et al., 2021). EMA prompts used ReTAINE technology (Smith et al., 2019). At each prompt, participants received a text with a link to a website delivering the EMA items (see Figures 1–2). Study-provided iPhones were limited to accessing only the website that delivered the items. Participants had 60 minutes to respond to each prompt, after which it was coded as missing data. Data was collected directly on a secure server, and no data was saved on study phones. When participants opted to use their own phones, no data was stored on the phone and all surveys were accessed via text message.

Fig 1.

Example text messages from ReTAINE system

Fig 2.

Login page of Retaine website to complete EMA prompts.

Participants completed three seven-day EMA intervals before, during, and following treatment (Naim et al., in press), receiving three prompts daily; we report on the first two intervals in our analyses here. Interval 1 occurred the week prior to their first CBT session, and Interval 2 after session 6 of CBT. To enhance compliance and ease of use, participants selected 60-minute time frames during morning (6:00–9:00 AM), afternoon (3:00–6:00 PM), and evening (7:00–10:00 PM) periods to receive each prompt; prompt alerts were then randomized within these windows. Youth were compensated for participation and offered a monetary bonus if they completed 75% or more of prompts (Naim et al., 2021). Participants were excluded from analyses for each interval if they did not complete at least five prompts in that interval, following prior EMA research (Bylsma et al., 2011; Byrne et al., 2023; Kircanski et al., 2015; Naim et al., 2021). To assess convergent validity with established irritability measures, participants, their parents, and clinicians completed validated irritability assessments (Haller et al., 2020; Stringaris et al., 2012) as closely to each EMA interval as possible.

Two prompts designed to assess irritability symptoms’ chronometry (i.e., both pervasive mood states and momentary affect) are the focus of the current analysis. One prompt assessed frustration since the previous prompt to capture irritability throughout the day (“Since the last beep, I felt frustrated”), and another assessed momentary anger at the time of the prompt (“At the time of the beep, I felt annoyed or angry”) (Naim et al., 2021). Prompts were broadly modeled on items from the Affective Reactivity Index (e.g., “Angry most of the time”) (Stringaris et al., 2012). To assess discriminant validity, analyses included an EMA prompt capturing positive affect, “At the time of the beep, I felt happy.” The “Annoyance/Anger At,” “Frustration Since”, and “Happy” prompts were rated using a 5-point Likert scale, ranging from 1 = Not at all to 5 = Extremely. Of note, participants also completed additional items assessing temper outbursts: e.g., “Since the last beep, I felt really, really angry and out of control” (rated as a categorical “Yes”/”No” response). However, the temper outburst item’s endorsement was too low to include in analyses (<7.69% “Yes” at any prompt in Interval 1). Of the three continuous EMA items, analyses included those allowing for direct comparison with previous psychometric analyses in anxious and healthy youth (Byrne et al., 2023).

Materials

Diagnostic Status

The KSADS-PL is a semi structured diagnostic interview that assesses present and lifetime DSM-5 disorders (Kaufman et al., 1997). As outlined in Naim et al. (2023), clinicians assessed participants for eligibility and diagnostic categorization using the KSADS-PL, as well as a DMDD supplement (Wiggins et al., 2016).

Child and parent-rated irritability

Irritability symptoms were assessed by youth and their parents via the Affective Reactivity Index Self and Parent 1-Week version as closely to each EMA interval as possible (number of days from completing EMA: Affective Reactivity Index youth-report Mdn = 3.00; Affective Reactivity Index parent-report Mdn = 3.00) (Stringaris et al., 2012). The Affective Reactivity Index comprises six items pertaining to irritable feelings and behaviors that were summed to generate a total score, and one additional item assessing irritability-related impairment. Each is rated on a 3-point Likert scale from 0 = Not true to 2 = Certainly true. The Affective Reactivity Index has exhibited strong reliability and construct validity (DeSousa et al., 2013; Mulraney et al., 2014; Stringaris et al., 2012). In the current sample, the Affective Reactivity Index-Self exhibited excellent internal consistency in Interval 1 (α = 0.90) and good internal consistency in Interval 2 (α = 0.86). The Affective Reactivity Index-Parent showed good internal consistency for both Intervals 1 (α = 0.85) and 2 (α = 0.80).

Clinician-rated irritability

The Clinician Affectivity Reactivity Index is a 12-item semi-structured interview with good validity and sufficient reliability (Haller et al., 2020). As closely to each EMA interval as possible, clinicians assessed youth irritability via the Clinician Affectivity Reactivity Index (number of days from completing EMA: Mdn = 2.00) (Haller et al., 2020) (see Naim et al., 2023).

Analyses

Statistical analyses were completed using R version 4.3. Descriptive statistics were computed to assess participant demographics (see Table 1). EMA items’ psychometric properties were examined separately by interval, as appropriate. As outlined in Byrne et al. (2023), analyses assessed irritability and positive affect EMA items’ variability, internal consistency, test-retest reliability, and convergent and discriminant validity. Within person variability was evaluated by quantifying the degree of change in response from one prompt to the next, and between person variability by assessing the proportion of variability attributed to between subjects differences relative to total variability. Internal consistency was assessed as consistency between both EMA prompts within individuals, and across individuals at each prompt time. Test-retest reliability was evaluated via correlations between mean EMA responses at both timepoints. Convergent validity was assessed by comparing mean EMA responses to validated self, clinician, and parent report measures at each timepoint. Finally, discriminant validity was examined by comparing mean EMA responses to both irritable EMA items with the positive affect EMA item. To facilitate comparison across clinical groups assessed by Byrne et al. (2023) and the current study, supplemental analyses assessed the variability, test-retest reliability, and convergent and discriminant validity of anxiety and mood related EMA measures (see Supplemental Tables 1–2).

Variability

Each participant’s within person variability was assessed as moment-to-moment variability in EMA responses at each interval via mean adjusted squared successive differences (MASSD) (Byrne et al., 2023). Unanswered prompts were removed to assess MASSD between all available EMA responses; MASSD accounts for varying lengths between assessments (Funkhouser et al., 2021; Jahng et al., 2008; Trull et al., 2008). To generate a mean MASSD score for each EMA item at each interval, MASSD scores were aggregated across participants (Woyshville et al., 1999). While there are no standardized cutoffs for acceptable variability, higher MASSD scores indicate more within-person variability in participants’ responses over time regardless of their mean score, while lower MASSD scores indicate greater stability in responses over time.

Intraclass correlation coefficients (ICC) investigated between person variability in responses of each EMA item at each interval. ICCs were calculated with missing prompt data removed. Here, ICC describes the proportion of an EMA item’s inter-subject variability relative to total variability. A higher ICC score indicates more variability in responses between participants (Bolger & Laurenceau, 2013; Snijders & Bosker, 2011). We used the Wald test to evaluate whether the ICC value was significantly greater than zero (Snijders & Bosker, 2011).

Reliability

Internal consistency

Internal consistency was calculated using a procedure from previous EMA research to account for the multilevel nature of EMA data (Byrne et al., 2023; Viechtbauer, 2017). First, a multivariate multilevel model generated correlations between the two irritability EMA items at the prompt (within each observation) and person (within each participant) level. Prompt and person level correlations excluded missing prompt data. Building on previous approaches (Byrne et al., 2023; Forkmann et al., 2018), internal consistency was calculated (with >.70 indicating sufficient consistency) between irritability measurements at each observation and within each participant via the Spearman-Brown formula to predict reliabilities:

$${\rho }_{xx\prime }^{\ast }=\frac{n{\rho }_{xx\prime }}{1+\left(n-1\right){\rho }_{xx\prime }},$$

$n$, number of items; $\rho$, correlation of items; ${\rho }_{xx\prime }$, predicted reliability.

Test-retest reliability

Participants’ mean scores were calculated for each EMA item at Intervals 1 and 2. Mean EMA scores for each interval excluded missing prompt values. Each EMA item’s test-retest reliability across participants was quantified using the ICC between each item’s mean score at both intervals. For these analyses, ICC indicates the correlation between average EMA item responses at each interval. ICC values and 95% confidence intervals were quantified via a mean-rating (k = 2), consistency, two-way mixed effects model. An ICC score < 0.5 indicates poor reliability, an ICC between 0.5–0.75 moderate, 0.75–0.9 good, and >.90 excellent (Koo & Li, 2016; Portney & Watkins, 2009).

Validity

Convergent validity of the irritability EMA items was assessed via Pearson’s r correlations between participants’ mean scores and child, parent, and clinician-reported measures of irritability (Affective Reactivity Index-Self; Affective Reactivity Index-P; Clinician Affective Reactivity Index, respectively), with a correlation of <± 0.29 considered small, ±0.30 to ±0.49 = moderate, and >±0.50 = strong at the p < 0.05 level. Discriminant validity was evaluated via Pearson’s correlations between mean irritability EMA item scores and mean EMA positive affect scores at each interval.

Results

The current study initially included N = 50 youth receiving CBT treatment for clinically impairing irritability. Demographic data and clinical characteristics are presented in Table 1. One participant was excluded from all EMA analyses due to not responding to at least five EMA prompts in either interval (N = 49). At Interval 1, two participants were excluded from analyses due to not responding to at least five EMA prompts (N = 47 with sufficient data) but were included in Interval 2. Average compliance rates (percentage of prompts completed) were 80.77% at Interval 1 and 76.31% at Interval 2. There were no significant differences between compliance rates between morning, afternoon, and evening prompts in either interval (ps > 0.48). EMA ratings of frustration differed significantly between Interval 1 and Interval 2 (t(46) = 2.72, p <.01), while other items displayed no significant differences (ps < .24). Mean and standard deviation of each EMA item in Interval 1 was calculated by aggregating EMA responses across participants and timepoints (see Table 2). Missing completely at random (MCAR) analyses revealed that EMA data was not missing at random for either Interval 1 (X² = 18.73, p < .01) or Interval 2 (X² = 21.85, p < .01) (see Discussion).

Table 2.

Ecological momentary assessment item variability in youth with clinically impairing irritability at Intervals 1 and 2.

EMA Item	Interval	M	SD	Min	Max	M MASSD	SD MASSD	Min MASSD	Max MASSD	ICC
“Since the last beep, I felt frustrated”	Interval 1	1.90	1.21	1.00	5.00	1.31	1.19	0.00	4.67	0.42
	Interval 2	1.70	1.10	1.00	5.00	0.97	1.24	0.00	6.33	0.48
“At the time of the beep, I felt annoyed or angry”	Interval 1	1.66	1.09	1.00	5.00	1.23	1.25	0.00	5.07	0.36
	Interval 2	1.58	1.04	1.00	5.00	0.81	0.89	0.00	3.37	0.43
“At the time of the beep, I felt happy”	Interval 1	3.09	1.24	1.00	5.00	1.77	1.62	0.00	7.26	0.25
	Interval 2	2.99	1.28	1.00	5.00	1.45	1.23	0.00	5.63	0.36

Within and Between Person Variability

The descriptive statistics and variability of each EMA item at each interval is reported in Table 2. Mean MASSD scores for both irritability EMA items exhibited within person variability at both intervals (MASSD scores ranging 0.81–1.31). A small portion of youth demonstrated no variability within an EMA item with a MASSD score = 0.00 (Interval 1: n = 8 “Annoyance/Anger At,” n = 3 “Frustration Since”; Interval 2: n = 12 “Annoyance/Anger At,” n = 9 “Frustration Since”). The gap between mean and maximum MASSD scores suggests that response variability differed considerably between youth. We observed comparable variability for the “Happy” EMA item at both intervals, demonstrating within person variability across EMA items (1.45–1.77). ICC values demonstrated between-person variability at both intervals for both irritability and “Happy” EMA items (ICC scores ranging 0.25–0.48; see Table 2).

Reliability

Internal consistency of the two irritability EMA items at both person and prompt levels at each interval and test-retest reliability between Intervals 1 and 2 are presented in Table 3. At each interval, both irritability EMA items exhibited excellent internal consistency at the person level (Interval 1 = 0.97, Interval 2 = 0.91), and lower but acceptable consistency at the prompt level (Interval 1 = 0.62, Interval 2 = 0.53). This is consistent with analyses in anxious and healthy youth populations (Byrne et al., 2023). ICC values indicated good test-retest reliability for irritability EMA measures (ICC = 0.84 for the “Frustration Since” prompt and ICC = 0.78 for the “Annoyance/Anger At” prompt), and moderate reliability for the positive affect measure (ICC = 0.70).

Table 3.

Internal consistency and test-retest reliability in youth with clinically impairing irritability at Intervals 1 and 2.

	Reliability (person)	Reliability (prompt)
Interval 1 EMA Irritability Items	0.974	0.616
Interval 2 EMA Irritability Items	0.914	0.528
EMA Item	ICC	ICC 95% CI
“Since the last beep, I felt frustrated”	0.836	0.723 to 0.905
“At the time of the beep, I felt annoyed or angry”	0.782	0.64 to 0.873
“At the time of the beep, I felt happy”	0.699	0.518 to 0.821

Validity

Information on convergent and discriminant validity is presented for each EMA item at Interval 1 in Figure 3. Both irritability EMA items were strongly correlated with one another (r(45)=0.90, p<.01) and were moderately correlated with child reported irritability on the Affective Reactivity Index-Self (r(45)=0.56 for the “frustrated” prompt and r(45)=0.58 for the “angry or annoyed” prompt, ps<.01). Both EMA items demonstrated small to moderate correlations with clinician reported irritability on the Clinician Affective Reactivity Index (r(45)=0.38 for the “Frustration Since” prompt, p<.01 and r(45)=0.32 for the “Annoyance/Anger At” prompt, p<.05). Items were not significantly correlated with parent-reported irritability on the Affective Reactivity Index-Parent, with the “Annoyance/Anger At” EMA prompt and Affective Reactivity Index-Parent trending toward significance (p = .06). Irritability EMA measures were negatively correlated with the positive affect EMA measure, though this association was not significant, indicating overall good discriminant validity.

Fig 3.

Convergent and discriminant validity in youth with clinically impairing irritability at Interval 1. *p < .05; **p < .01.

Discussion

We examined the psychometric properties of irritability EMA items in a sample of treatment-seeking youth with irritability. Critically, the irritability EMA items demonstrated acceptable variability, consistency, reliability, and both convergent as well as discriminant validity in youth with irritability, and thus may be suitable for clinical populations and trials. These findings also extend previous work demonstrating convergence between these irritability EMA items and existing clinical and behavioral measures, facilitating studying irritability at multiple levels of analysis to refine conceptions of youth irritability and ultimately inform treatment (Joyner & Perkins, 2023; Naim et al., 2021). Below, these findings are discussed in context with psychometric analysis of EMA items by Byrne et al. (2023) in anxious and healthy youth.

As hypothesized, the EMA irritability items demonstrated within and between person variability. While there is no agreed-upon standard for variability in EMA research, results were similar to those reported by Byrne et al. (2023) and suggest that these items capture differences both within individual response patterns and between participants. This extends findings reported by Byrne et al. (2023) that anxiety EMA items in this study protocol sufficiently capture between and within person variability in anxious youth. In supplemental analyses, Byrne et al. (2023) examined irritability EMA items from this protocol in anxious and healthy youth. They found evidence for within and between person variability in youth with anxiety, but weaker within person variability in healthy youth, suggesting that these items may not sufficiently capture irritability fluctuations in healthy populations. These results are not totally unexpected given findings by (Naim et al., 2022) that youth with DMDD demonstrate significantly more overall negative affect lability compared to healthy youth; healthy youth may thus demonstrate fewer overall fluctuations in irritable affect. Therefore, while these irritability items may be especially useful to capture irritability symptoms within clinical trials, additional measures or changes may be necessary to be adequately sensitive to dynamic fluctuations in irritability in healthy comparison populations (Byrne et al., 2023). Additionally, similar to Byrne et al. (2023), the “Happy” EMA item exhibited within and between person variability in this clinically irritable sample and may be suited for future research on positive and negative affect in youth with irritability (Naim et al., 2022; Vogel et al., 2023).

The irritability EMA items exhibited excellent internal consistency at the person level, and lower but sufficient consistency at the prompt level. These findings were consistent with hypotheses, and parallel to findings by Byrne et al. (2023) with anxiety EMA items in anxious youth in this protocol. Indeed, as Byrne et al. (2023) note, lower prompt-level reliability may reveal meaningful fluctuations in affect across time and contexts (Mischel et al., 2002). Unlike approaches that aggregate EMA scores across timepoints, variation in prompt-level responses reflects both pervasive mood states and momentary affect. Such measures may be clinically meaningful in irritable youth and vary by symptom presentation (Dejonckheere et al., 2018; Naim et al., 2022; Olthof et al., 2023). These findings underscore these EMA items’ utility for clinical research. Further, despite involvement in CBT treatment and a significant change in EMA responses for the “frustration” prompt in these analyses, both EMA items exhibited good test-retest reliability across intervals, indicating they are measures of stable constructs.

EMA irritability items demonstrated good convergent validity. Both irritability items were strongly correlated with child-reported irritability and moderately correlated with clinician-rated irritability, in line with previous evidence of convergence between irritability EMA items and youth and clinician-reported irritability (Naim et al., 2021). However, items were not significantly correlated with parent-reported irritability. This contrasts with findings by Byrne et al. (2023) of a small correlation between EMA anxiety items and parent reported anxiety in anxious youth. However, current findings align with previous research demonstrating marked informant discrepancies between youth and parent reported irritability; particular clinical and demographic features may contribute to discrepancies (see Mallidi et al., 2023), which may also reflect different underlying irritability-related constructs (De Los Reyes et al., 2009; Zik et al., 2022). The lack of convergence between EMA irritability items and parent-reported irritability also diverges from Naim et al. (2021), whose findings demonstrate convergence between irritability EMA and parent-reported irritability in this protocol. However, while their analyses used a transdiagnostic sample, the current study’s focus on youth with clinical irritability undergoing CBT treatment may underscore the particular importance of obtaining multiple informant reports of youth irritability in clinical contexts. Finally, both irritability EMA items exhibited a negative but non-significant correlation with the positive affect EMA item, while they were highly correlated with one another (r = 0.90), indicating discriminant validity.

Our study has several potential implications for future research in clinically irritable youth. Irritability may involve more dynamic affect changes than other emotional symptoms (e.g., outbursts) (Naim et al., 2022). Future work could use robust EMA measures of irritability to understand dynamic changes in irritable affect, as well as influence of contextual factors on irritability symptoms (e.g., parental or peer interactions) (Naim et al., 2021). Further, EMA assessment of real time irritability symptoms during treatment may facilitate understanding of treatment response and tailoring of treatment to individual youths’ needs (Naim et al., 2021). Finally, examining changes in irritable affect over the course of treatment, such as the significant decrease in reported frustration in the current sample, may allow researchers to test hypotheses on putative mechanisms of treatment efficacy (Brotman et al., 2017).

Despite demonstration of the robustness of the psychometric properties of these irritability EMA items, this study had several important limitations. First, participants’ involvement in CBT treatment during the EMA intervals creates a potential confound in the interpretation of results, particularly reliability analyses, though good test-retest reliability was observed despite evidence of response change in one item. Second, similar to findings from Byrne et al. (2023) that anxiety-related EMA items from this study protocol were valid in participants with anxiety but not healthy volunteers, supplemental analyses demonstrated limited within person variability in for these anxiety items in participants with primary irritability, despite frequent clinical co-occurrence of irritability and anxiety symptoms (Stoddard et al., 2014). However, anxious EMA items significantly correlated with both parent and child reported anxiety in youth with irritability; diminished within person anxiety may thus reflect reduced fluctuations in anxiety in the current sample. Third, while analyses suggest these EMA items’ ability to capture dynamic fluctuations in affect, the binary nature and low endorsement of the outburst related EMA item prevented its inclusion in these analyses. These EMA items thus do not capture outbursts, which are highly salient clinically and a target of novel interventions (Naim et al., 2023). Future work should aim to combine these items with EMA measures better able to capture phasic dimensions of irritability, especially for use in clinical trials. Fourth, though overall EMA compliance was high, the EMA data in each study interval were revealed to be missing not at random. Though all included participants met the accepted cutoff for data completion, a few outlier participants with relatively low compliance rates may have driven this result. Additionally, while the study assessed child, parent, and clinician reports of irritability as closely as possible to each EMA interval, these did not always exactly map on to the EMA interval being studied, which may have impacted findings.

Lastly, the limitations of our particular sample should be considered. The accessibility of outpatient treatment at the National Institute of Mental Health (NIMH) and the unique nature of the clinical irritability phenotype may have impacted the generalizability of the current study. Specifically, our study had a relatively small sample size, was majority white and male, and all youth and parents were receiving treatment at the NIMH. Our sample had the time and ability to enroll in a time intensive treatment study and complete additional measures for research, which may create a selection bias relative to the general population (e.g., higher compliance rates). Finally, we provided free study phones and standardized training on EMA measures that may not be easily replicated in a typical clinical setting. Future research should replicate these findings using larger samples across diverse populations.

Conclusions

Analyses demonstrate that EMA items designed to capture irritability exhibit acceptable between and within person psychometric properties, including between and within person variability, internal consistency, test-retest reliability, and convergent and discriminant validity in youth with irritability, suggesting items are psychometrically sound and fit for use with clinically irritable populations in clinical trials. Results align with previous research demonstrating strong psychometric properties of anxiety EMA items in anxious participants (Byrne et al., 2023), as well as the need for comparison across clinical groups. As EMA increases in popularity in youth psychopathology research, it is essential to validate emerging measures to avoid threats to replication and internal validity. This study uniquely evaluates the psychometric properties of irritability EMA items while accounting for the dynamic nature of EMA data, facilitating their use in future clinical research.

Highlights.

• We assessed psychometrics of irritability ecological momentary assessment (EMA) items

• Items demonstrated within and between person variability

• Items demonstrated internal consistency and test-retest reliability

• Items demonstrated convergent and discriminant validity with clinical irritability measures

• Irritability EMA items are valid for use in clinical research and trials