Psychometric Evaluation of a Big Five Personality State Scale for Intensive Longitudinal Studies

Abstract

Despite enthusiasm for using intensive longitudinal designs to measure day-to-day manifestations of personality underlying differences between people, the validity of personality state scales has yet to be established. In this study, we evaluated the psychometrics of 20-item and 10-item daily, Big Five personality state scales in three independent samples (N = 1,041). We used multilevel models to separately examine the validity of the scales for assessing personality variation at the between- and within-person levels. Results showed that a five-factor structure at both levels fits the data well, the scales had good convergent and discriminative associations with external variables, and personality states captured similar nomological nets as established global, self-report personality inventories. Limitations of the scales were identified (e.g., low reliability, low correlations with external criterion) that point to a need for more, systematic psychometric work. Our findings provide initial support for the use of personality state scales in intensive longitudinal designs to study between-person traits, within-person processes, and their interrelationship.

A comprehensive model of personality must account for relatively stable traits that vary between people as well as trait-relevant states that vary within a person across time and situations (Baumert et al., 2017; Cervone, 2005). Although most personality research has focused on the between-person trait structure, contemporary personality theories emphasize within-person processes that give rise to differences between people (e.g., DeYoung, 2015; Fleeson & Jayawickreme, 2015; Pincus & Ansell, 2013; Read et al., 2017). Understanding how personality traits manifest in day-to-day life, including the precipitants and consequences of shifts in personality states (i.e., short-term expressions of personality traits), is also key to explaining the well-established links between traits and life outcomes (Hampson, 2012).

Measuring Personality States

The gap between measurement and theory has prompted growing interest in empirical methods capable of capturing variance in personality at both the individual differences (i.e., between-persons) and occasion differences (i.e., within-person) levels of analysis. One promising approach is the use of intensive longitudinal designs (ILDs), encompassing more specific techniques like ambulatory assessment, ecological momentary assessment, or daily diaries, which involve repeated measurement of personality states over the course of days or weeks. ILD research has shown there is considerable within-person variability of self-reported personality states and that the aggregation of an individual’s states approximates their personality trait standing (Fleeson, 2001; Fleeson & Gallagher, 2009; Jones et al., 2017). These methods have also been used to relate fluctuations in personality states to proximal contexts, affording insight into questions about the dynamics of the person and situation that have eluded cross-sectional methods (Bleidorn, 2009; Fleeson, 2007; Fleeson & Law, 2015; Matz & Harari, 2020; Rauthmann et al., 2016; Sherman et al., 2015).

Although ILDs are the vanguard of personality measurement, there has yet to be a thorough psychometric evaluation of personality state scales used in these studies. In their recent review of the ILD personality state literature, Horstmann and Ziegler (2020) identified major limitations in previous efforts to validate these measures. Notably, nearly all studies that tested scale psychometrics examined the reliability and validity of average state scores (i.e., between-person level), but not individual state scores (i.e., within-person level). Horstmann and Ziegler reported that the most common validity evidence has been indexing convergence between average states and personality traits measured by global, self-report inventories, and reliability has typically only been estimated for the average states. These studies only offer psychometric evidence for measuring trait-like, between-person variance—not within-person variance—despite the importance of each source of variability for understanding personality.

In one exception to this pattern, Zimmerman et al. (2019) examined both the within- and between-person levels in developing their ILD scales of personality states, with the primary goal of measuring maladaptive processes (see also Wright & Simms, 2016). The scales introduced by that study were constructed in an exploratory manner and were not intended to align with a particular model of personality. This approach was well-suited for the scale’s purpose of clinical assessment, but not for testing personality theories built on the Big Five model of personality, which is arguably the dominant framework for describing individual differences. Because of the Big Five’s eminence in personality research, it is unsurprising that many ILD personality state measures have been informed by this model (e.g., Aschwanden et al., 2019; Bleidorn, 2009; Borkenau & Ostendorf, 1998; Fleeson, 2001, 2007; Fleeson & Law, 2015; McCabe & Fleeson, 2016). Tethering personality state measures to the Big Five permits researchers to leverage the model’s robust structure and nomological net for conceptualizing trait expressions in daily life. For personality state measures to be validly used for studying the Big Five between-person traits, within-person processes, and their interrelationship, the psychometrics for both levels of analysis must be established.

The Need for Brief Measures

A unique advantage of ILD over traditional cross-sectional measures of personality is that it affords naturalistic, temporally sensitive data. The resolution of personality process models is limited, however, by the burden placed on participants of completing lengthy surveys in the stream of everyday life. Furthermore, the potential for participant reactivity to a tedious ILD protocol threatens the validity of their responses (Barta et al., 2012). Thus, the assessment demands of ILD protocols exacerbate the traditional tensions of the bandwidth-fidelity trade-off. That is, researchers are faced with a trade-off between content coverage or temporal granularity when both components are essential to modeling personality processes. Short scales offer one solution, and there are several such scales for personality traits with good psychometrics (e.g., Gosling et al., 2003; Thalmayer et al., 2011), but none have been systematically psychometrically validated for assessing personality states with ILDs. For these reasons, there is great potential value in validating brief measures of personality states that can be administered multiple times per day with minimal interference in participant’s daily activities.

Current Study

The purpose of this study was to assess the reliability and validity of a 20-item and 10-item personality state scale for measuring both between- and within-person variance. Our goal was not to construct a definitive instrument from scratch; rather, we aimed to evaluate a Big Five ILD scale akin to those used in a number of published studies. This study is an initial effort that can inform a broader, ongoing validation process needed to comprehensively test the psychometrics of personality state measures for ILD.

Our analyses were guided by expectations of what constitutes a valid measure at each, conceptually and statistically distinct level of personality (Horstmann & Ziegler, 2020). Because a personality state is the contextualized, short-term enactment of a personality trait, each personality state score is presumed to be influenced by the corresponding trait as well as the situation in which it manifests. In this study, we separated these sources of variance to examine the structure and correlates of personality traits and states.

At the between-person level, individual differences reflect variation in how people tend to behave, think, and feel across situations (i.e., personality traits). In ILDs, these characteristic patterns can be estimated by averaging each person’s personality state scores over the study period. For a Big Five personality state scale to be valid for studying between-person differences we expected that (a) covariation of average personality states would conform to the expected five-factor trait structure (i.e., structural validity); (b) there would be adequate variance in average states between people to justify studying average states as individual differences; (c) average states would correlate with corresponding traits measured by established global self-report inventories (i.e., convergent validity), and (d) average states and global self-report trait measures would share nomological nets evidenced by a similar set of associations with an individual’s typical behaviors, thoughts, and affect assessed in the ILD protocol (i.e., construct validity or nomological homomorphy; Rauthmann et al., 2019).

At the within-person level, deviation of personality states from an individual’s average level (i.e., trait) are elicited by and lead to other behavioral, cognitive, and affective states. States and situations that tend to covary with shifts in personality states over time suggest characteristic processes underlying between-person differences. For personality states and corresponding traits to be functionally related via these within-person processes, they would presumably share a nomological net. Thus, to be a valid measure at the within-person level, we expected that (a) the same personality states that covary between people would also covary from day-to-day and so would form five, within-person factors resembling the between-person structure; (b) there would be adequate within-person variance to justify studying within-person processes; and (c) personality states would co-occur with other behavioral, cognitive, and affective states measured in the ILD protocol that map onto the nomological net of corresponding traits assessed by global self-report inventories.

Method

We report how we determined our sample size, all data exclusions, all manipulations, and all measures in the study.

Participants

Sample 1 (S1).

S1 consisted of undergraduate students enrolled in introductory psychology courses at the University of Pittsburgh. The sample was mostly White (86%; 10% Asian, 6.5% Black or African American)¹ and mostly female (62%) with a mean age of 18.6 years (SD = 0.96). The total sample size was 330. Sample size selection was influenced by the goal of arriving at stable effect sizes for analyses based on covariance/correlation matrices (e.g., structural equation modeling, multilevel modeling). Based on recent empirical work suggesting correlation estimates stabilize at n = 250 (Schönbrodt & Perugini, 2013), we over-sampled with a target of n > 300 to ensure the minimum sample size was achieved. After removing outliers, the final sample size for S1 was 294. Criteria for outliers is described in the Analytic Plan section.

Sample 2 (S2).

Participants in S2 were community members recruited from posted flyers and online postings for a study on personality and daily life. For inclusion, participants had to be between 18 and 40 years of age. Participants also had to be users of a smartphone running iOS or Android software. To recruit a distinct community sample, individuals were not eligible if they were enrolled in a full-time undergraduate program. All participants in S2 were prescreened to ensure a gender-balanced sample as well as adequate representation of personality traits of interest for the parent study on narcissism. Namely, the modesty facet scale of the NEO Personality Inventory–Revised (NEO-PI-R; Costa & McCrae, 1992) was used to assess participants during the prescreening interview. Low modesty, a core feature of narcissism, was oversampled such that a 2:1:1 ratio of low, moderate, and high levels of modesty within each gender were recruited. S2 was mostly White (89%; 8.5% Asian, 4.4% Black or African American), balanced on gender (female = 52%), with a mean age of 27.6 years (SD = 4.9). The total sample size was 342 and was determined by the same considerations as S1. After removing outliers, the final sample size for S2 was 316.

Sample 3 (S3).

S3 was drawn from the University of Pittsburgh Adult Health and Behavior Project (AHAB) project, which comprises a registry of behavioral and biological measurements for the study of individual differences. Participants 30 to 54 years of age were recruited via mass-mail solicitation from communities of Southwestern Pennsylvania in two periods of data collection (2001–2005; 2008–2011). At enrollment, participants were in good general health, without reported history of atherosclerotic cardiovascular disease, chronic kidney or liver disease, recent treatment for cancer, major neurological disorders, or psychotic illness (e.g., Manuck et al., 2010; John-Henderson et al., 2016). Data used in the present analyses derive from an ongoing, second wave of AHAB data collection, occurring roughly 11 to 16 years following participants’ initial participation. The sample available for analysis is mostly White (85%, 14% African American, <.01% Asian, <.01% biracial), 54% females, with an average age of 59.5 years (SD = 7.2). Total projected enrollment for the second wave of the AHAB registry is N~800. However, due to the COVID-19 global pandemic, participant recruitment and data accrual was suspended in March of 2020. This study uses those data that were available at that time. The total sample size was 458. After removing outliers, the final sample size was 431.

Procedure

Sample 1 and Sample 2.

Study procedures were nearly identical for S1 and S2. For both samples, participants completed baseline self-report questionnaires then could elect to participate in the ILD portion of the study. Orientation to the protocol and participation were conducted entirely online without direct contact with study staff. Participants in S1 received course credit for completing the study. In S2, participants who completed baseline questionnaires were entered into prize drawings for $75 Amazon gift cards. For the ILD portion, they received a $100 Amazon gift card for completing 90% or greater of the total surveys administered during the study period. Gift cards of prorated value (e.g., $75 was given for 75% participation) were given to those who completed less than 90% of surveys.

Baseline assessments included self-report questionnaires on personality and psychological functioning. Baseline data were collected and managed using Research Electronic Data Capture (REDCap) hosted at the University of Pittsburgh (Harris et al., 2009; 2019). REDCap is a secure, web-based software platform designed to support data capture for research studies supported by the Department of Biomedical Informatics (Clinical and Translational Sciences Institute at the University of Pittsburgh Grant Number UL1-TR-001857).

When participants chose to participate in the ILD protocol, they viewed a video training presentation explaining the ILD procedures and instructions for downloading the MetricWire smartphone application (MetricWire, Inc., 2019) used to administer surveys. A short comprehension quiz was given following the training to check for understanding. Failure to show adequate comprehension lead to exclusion from further participation.

The ILD protocol involved randomly initiated surveys delivered via push notifications between 9 am and 9 pm. Participants in S1 received five surveys per day, and those in the S2 received seven surveys per day. For this study, only self-reported positive and negative affect from the random surveys were used to index average daily affect. A longer survey was administered at 9 pm each day asking about thoughts, feelings, and events occurring in the past 24 hours. Participants had 3 hours after the initial push notification to complete the end of day survey.

Sample 3.

The study procedures for S3 involved several in-person lab sessions and 8 days of ILD monitoring outside of the lab. Participants were compensated $150 to $250, depending on their participation in an ancillary brain imaging protocol. Other components of data collection included neurocognitive testing, assessments of social and demographic factors, and instrumented measurements of physical functioning and cardiometabolic risk factors. Only data from self-report questionnaires were used for this study, so the procedures from other sessions will not be detailed further.

For the ILD portion of the study, participants were provided electronic tablets to complete end-of-day surveys and received in-person training about the protocol. Daily surveys were completed in Qualtrics and included the same questions administered to S1 and S2.

Measures

Items used to assess personality states and other ILD variables are shown in Table 1. Descriptive statistics for every measure and omega coefficients for scales are included in the online supplementary material. Unless otherwise noted, items were administered to all three samples in the same format.

Table 1.

Items Used to Assess State Constructs.

State construct	Items
Personality
Extraversion	Lethargic/Energetica, Talkative/Silenta, Bold/Timid, Unassertive/Assertive
Neuroticism	Insecure/Securea, Relaxed/Tensea, Nervous/At Ease, Unexcitable/Excitable
Conscientiousness	Careless/Thorougha, Hardworking/Lazya, Serious/Frivolous, Inefficient/Efficient
Agreeableness	Stingy/Generousa, Distrustful/Trustfula, Warm/Cold, Rude/Polite
Openness	Unimaginative/Imaginativea, Uninquisitive/Curiousa, Conventional/Creative, Imperceptive/Perceptive
Behaviors
	Did you drink any alcoholic beverages today? Did you engage in any physical activity, long enough to work up a sweat, get your heart thumping, or get out of breath? Did you receive emotional support (e.g., listening to you, comforting you) from anyone? Did you have social contact with anyone, either at your home or away from your home? Did you engage in any leisure activities today (watching TV, listening to music, playing a game, going to a movie or sporting event, taking a walk, hike or run, etc.)? Did you do any work around the house or yard? Did you do any family or personal errands away from home (grocery shopping, doctor appointment, etc.)? Did you go to work? Did you engage in work-related activities that involved other people (e.g., attended a meeting; collaborated on a project, engage in a task-related discussion)? Did you socialize with anyone at work or during your commute? How often did you feel isolated? How often did you feel left out?
Affect
Positive affect	Happy, Excited, and Relaxed (S1 and S2) Happy, Calm, Full of Pep, Lively, Cheerful, and At Ease (S3)
Negative affect	Nervous, Sad, and Angry (S1 and S2) Sad, Hostile, On Edge, Angry, Tense, and Unhappy (S3)
Energy
	How often did you feel tired today? How often did you run out of energy today?
Loneliness
	How often did you feel lonely? How often did you feel isolated? How often did you feel left out?
Stress
	How often did you feel like you were unable to control the important things in your life? How often did you feel confident about your ability to handle your personal problems? How often did you feel like things were going your way? How often did you feel like difficulties were piling up so high that you could not overcome them?

Note. All items were assessed at the daily level except for affect in S1 and S2 which was measured multiple times per day and averaged for a daily score. S1 = Sample 1; S2 = Sample 2; S3 = Sample 3.

^aIndicates the item was used in the 10-item scale.

Personality States.

Personality states were measured at the end of the day with 20 bipolar adjective items informed by those used in Fleeson (2001). For each item, participants indicated the extent to which two adjectives best described them in the past 24 hours. Items were rated on a 7-point Likert-type scale. For example, to assess state Extraversion, participants would rate whether talkative describes extremely well (1) to silent describes extremely well (7), with the mid-point indicating both describe equally well. Two (for the brief scale) or four items (for the full scale) were averaged to calculate personality state subscale scores. Seven items were reverse scored.

Daily Behaviors.

Occurrence of personality-relevant behaviors were recorded at the end of each day. Participants used binary scales to indicate whether a behavior did or did not occur in the past 24 hours.

Daily Positive and Negative Affect.

Emotion adjectives from the Positive and Negative Affect Schedule (PANAS; Watson et al., 1988) were adapted to assess affect. In S1 and S2, affect was rated at each randomly initiated prompted time-point throughout the day. Items were reworded from the PANAS in the surveys to read “How ADJECTIVE do you feel right now?” Ratings were made on a slider scale from not at all (0) to extremely (100) for each adjective. Each individual’s mean affect reported across surveys within a day was used to measure daily affect. In S3, affect was rated at the end of day surveys. Items were reworded to read “During the past 24 hours to what extent did you feel . . . ” and adjectives were rated on a Likert-type scale from not at all (1) to very much (5). The mean of corresponding items indexed daily positive and negative affect.

Daily Energy.

Level of energy was assessed with two items in which participants rated how often they felt tired or out of energy in the past 24 hours on a Likert-type scale from almost never (1) to very often (4).

Daily Loneliness and Stress.

Daily loneliness and stress were each measured by three items. Loneliness items were adapted from the UCLA Loneliness Scale (Russell, 1996). Stress items were adapted from the Perceived Stress Scale (Cohen et al., 1983). Participants rated the extent to which they felt lonely and stressed in the past 24 hours on a 4-point (S1 and S2) or 5-point (S3) Likert-type scale from never to very often.

Global Personality Traits.

Big Five personality traits and facets were self-reported at baseline in S2 and S3. The 60-item Big Five Inventory–2 (BFI-2; Soto & John, 2017) was used in S2, and the 240-item NEO-PI-R was used in S3. For each item of the BFI-2 and NEO-PI-R, participants rated the extent to which a characteristic applies them (e.g., “I am someone who is outgoing”). In both instruments, items were rated on a Likert-type scale from disagree strongly (0) to agree strongly (4). Items from both instruments produce personality facet scores that are averaged to produce trait scores for Extraversion, Neuroticism (NEO-PI-R)/Negative Emotionality (BFI-2), Conscientiousness, Agreeableness, and Openness to Experience (NEO-PI-R)/Open-Mindedness (BFI-2).

Analytic Plan

All analyses were conducted with Mplus Version 8.4 (Muthén & Muthén, 2017) and RStudio Version 3.6.3 (Rstudio Team, 2020). We used multilevel models to disaggregate personality traits and states. In these models, each personality state score was decomposed into statistically independent sources of variance using latent decomposition. Variance at the between-person level reflects an individual’s average reported personality state expression across study days, indicating their relatively stable, cross-situational tendencies. Within-person variance represents how much an individual’s personality state deviates from their average level on a given day and suggests situational influences on personality expression. Thus, between-person variance approximates personality traits and variability at the within-person level are the day-to-day fluctuations in personality states.

Twenty-Item Scale Structural Validity.

To examine the within- and between-person structures of the personality states, we conducted a series of multilevel exploratory factor analyses (MEFAs) of the 20 items from our ILD scale. For these analyses, we pooled the three samples to maximize generalizability. In MEFA, a separate factor structure is estimated at each level. The structure at the between-person level represents how average personality states covary from person to person, whereas the within-person structure represents how personality states covary from day to day.

Factors were extracted using maximum likelihood estimation and oblique Geomin rotation. Models were evaluated in terms of factor interpretability and by using standard criteria for single-level factor analysis (i.e., comparative fit index [CFI] ≥ .95, root mean square error of approximation (RMSEA) < .06, and standardized root mean square residual (SRMR) < .08; Hu & Bentler, 1999). We considered factor loadings ≥ |.30| to be markers for a factor. Level-specific reliabilities for the subscales were estimated using omega coefficients, which has been shown to be preferable to other internal consistency measures in multilevel settings (Geldhof et al., 2014).

Twenty-Item Scale Construct Validity.

Construct validity of the subscales was assessed by examining the amount of variability in personality states subscales, convergent and discriminant relations with other trait and state variables, and by indexing their nomological homomorphy with commonly used global self-report personality trait measures. Because the samples had differences in criterion variables, we conducted these analyses separately for each sample. We used Bayes estimation with noninformative priors for all the following multilevel models. The Bayes estimator does not provide two-tailed p values, so we considered point estimates with 95% credibility intervals that did not contain zero to be statistically significant.

Ten-Item Scale Structural and Construct Validity.

Candidate items for a brief, 10-item scale were selected based on the MEFA results. We considered items for inclusion with primary factor loadings on the target factor at the between- and within-person levels, and items having the lowest cross-loadings on nontarget factors at both levels. We then fit models using the pooled sample with different combinations of the selected items with confirmatory factor analysis (CFA). Overall model fit was evaluated with the same criteria used for the MEFAs. The final model was chosen to balance model fit and breadth of item content. After identifying two items per personality state, we conducted the same construct validation analyses described above in each sample with the reduced subscales. Additionally, we compared the content coverage of the 10-item and 20-item scales by indexing their nomological homomorphy.

Results

Twenty-Item Scale Structural Validity

For this stage of analysis, we pooled the samples to maximize generalizability. Prior to evaluating structural validity, 1,438 observations (~16% of total observations) were removed due to unusually high or low personality state item variability within a day suggestive of invalid responding (i.e., flatline or flip-flopping between extreme values). To determine variance thresholds, we visually inspected distribution plots of within-day item standard deviations for outliers. Based on these distributions, we removed observations with item-level standard deviations ≤ 0.80 and ≥ 2.60. This resulted in removing 50 participants for a total pooled sample size of 1,041 participants and 7,664 days.

Mplus output and table of model fit indices for the MEFAs are available in the online supplementary materials. Although we expected five factors resembling the Big Five traits within- and between people, we estimated all permutations of one to six factors at each level as a basis of comparison for the hypothesized model. The model with five factors at the between- and within-person levels achieved good fit (CFI = .96; RMSEA = .03; SRMR_within/SRMR_between = .02/.03). Model fit indices improved slightly if a sixth factor was extracted at either level, but the added factors had uniformly low factor loadings (i.e., < |.30|) or were marked by secondary factor loadings, so we proceeded with analysis of the model with five factors between- and within-person.

Factor loadings for this model can be found in Table 2. At the between-person level, all but two items loaded most strongly on the target factor and only two items had cross loadings ≥ |.30| on nontarget factors. The factor structure was less consistent at the within-person level, but all factors except for Agreeableness had at least two primary loadings on the target factor.

Table 2.

Factor Loadings for Multilevel Exploratory Factor Analysis of Personality State Items.

Item	Extraversion	Neuroticism	Conscientiousness	Agreeableness	Openness
Between-person
Energetic	.13	−.35	.44	.05	.08
Bold	.73	−.12	.12	.01	.01
Talkative	.66	.01	.02	.29	−.09
Assertive	.62	−.09	.30	−.04	.10
Insecure	−.18	.56	−.31	−.05	.02
Nervous	.03	.99	−.07	.06	−.04
Tense	.01	.92	.11	−.08	.02
Excitable	.43	.18	−.03	.29	.19
Thorough	−.12	−.03	.85	.15	.04
Hardworking	.09	−.01	.83	.02	−.07
Serious	−.07	.09	.69	−.15	−.13
Efficient	.06	−.18	.72	.02	.08
Warm	.14	−.04	−.03	.58	.02
Polite	−.27	−.02	.12	.88	.00
Generous	.01	.00	.19	.54	.14
Trustful	.04	−.22	.00	.66	−.01
Imaginative	−.01	−.05	.13	.00	.90
Curious	.04	.00	.19	.07	.66
Creative	−.02	.01	−.12	−.04	.91
Perceptive	.09	.00	.38	.17	.37
Within-person
Energetic	.32	−.08	.26	.07	.07
Bold	.32	−.10	.10	−.02	.12
Talkative	.42	−.13	.01	.10	−.02
Assertive	.26	.04	.02	.21	.08
Insecure	.00	.31	−.17	.02	−.35
Nervous	.02	.60	.01	−.01	−.08
Tense	−.12	.68	.06	.00	.07
Excitable	.23	.02	−.05	.36	.03
Thorough	−.02	−.01	.48	.04	.26
Hardworking	.22	.08	.55	.00	−.01
Serious	−.08	.20	.28	−.08	−.01
Efficient	.07	−.01	.39	.37	−.04
Warm	.19	−.21	.12	−.04	.05
Polite	−.14	−.21	.20	.11	.18
Generous	.05	−.12	.00	.22	.08
Trustful	−.16	−.24	.03	.45	.00
Imaginative	.07	.00	−.01	.00	.61
Curious	.01	.00	.00	.21	.32
Creative	.08	.05	−.13	.12	.29
Perceptive	.02	.01	.05	.42	.08

Note. Factor loadings > |.30| are bolded and underlined. Item labels refer to the positively keyed pole of the scale. Comparative fit index = .96; root mean square error of approximation = .03; standardized root mean square residual_within/standardized root mean square residual_between = .02/.03.

After confirming good overall model fit and factor interpretability, we calculated personality state subscale scores. We then removed 39 participants with zero variance in every personality state subscale. Most participants removed for this reason were those who completed only one or two daily surveys, thus did not contribute enough data for reliable estimates, otherwise the responses were indicative of invalid responding (e.g., flatlining). Table 3 shows the means and standard deviations for the subscales in the pooled sample and sample specific descriptive statistics can be found in the online supplementary material. Intercorrelations among subscales at each level are presented in Table 4. The pattern of associations was similar across levels and ranged from modest to strong. The intercorrelations suggest a considerable amount of shared variance indicating a general factor in these scales.

Table 3.

Descriptive Statistics and Reliability Estimates for 20-Item and 10-Item Personality State Subscales.

Subscale	M	SD	ICC	ω Within-person	ω Between-person
Extraversion
Twenty-item	4.55	1.06	.59	.52	.83
Ten-item	4.59	1.18	.53	.49	.72
Neuroticism
Twenty-item	3.44	1.00	.58	.56	.83
Ten-item	3.05	1.45	.61	.45	.86
Conscientiousness
Twenty-item	4.89	1.06	.57	.54	.84
Ten-item	4.98	1.34	.52	.50	.83
Agreeableness
Twenty-item	5.27	0.98	.62	.44	.81
Ten-item	5.21	1.15	.61	.26	.70
Openness
Twenty-item	4.73	1.01	.67	.51	.87
Ten-item	4.81	1.18	.60	.46	.84
Average
Twenty-item	—	—	.61	.51	.84
Ten-item	—	—	.57	.43	.79

Note. Results are for the pooled sample. ICC = intraclass correlation coefficient, ω = omega coefficient.

Table 4.

Correlations Among Personality States at the Within- and Between-Person Levels for the 20-Item and 10-Item Scales.

	Extraversion	Neuroticism	Conscientiousness	Agreeableness	Openness
Twenty-item scale
Extraversion	—	−.57	.68	.63	.60
Neuroticism	−.22	—	−.55	−.54	−.35
Conscientiousness	.35	−.10	—	.65	.55
Agreeableness	.34	−.29	.27	—	.57
Openness	.39	−.18	.29	.33	—
Ten-item scale
Extraversion	—	−.61	.57	.57	.55
Neuroticism	−.25	—	−.65	−.64	−.50
Conscientiousness	.23	−.24	—	.62	.54
Agreeableness	.22	−.28	.18	—	.53
Openness	.26	−.31	.28	.25	—

Note. Results are from the pooled sample. Between-person correlations are shown above the diagonal; within-person correlations are shown below the diagonal. Bolded correlations were statistically significant (i.e., the 95% credibility intervals for the point estimates did not contain zero).

Reliability.

Omegas for the state subscales in the pooled sample are shown in Table 3, and sample-specific reliability estimates are provided in the online supplementary material. Average reliability of the subscale scores was higher at the between-person (ω = .86, range = .81–87) than within-person level (ω = .51, range = .44–56).

Twenty-Item Scale Construct Validity

Variability.

To determine whether there was sufficient within- and between-person variance in personality states, we calculated intraclass correlations (ICCs) using an unconditional multilevel model (i.e., no predictors). ICCs index the proportion of variance that is between-persons and 1—ICC is the proportion that is within-persons. ICCs for the personality state subscales in the pooled sample are shown in Table 3. The average ICC was .61 (range = .57-.67) indicating 61% of the variance in personality states is between people and 39% is within-persons from day-to-day.

Convergent and Discriminative Validity.

To determine the degree of convergence between average personality states and personality traits, we correlated personality states with the BFI-2 (S2) and NEO-PI-R (S3) trait scales at the between-person level (Table 5). Correlations with BFI-2/NEO-PI-R facets are available in the online supplementary material. In both samples, average personality states had the highest correlations with the corresponding trait with one exception: average daily Agreeableness was associated (negatively) just as strongly with NEO-PI-R Neuroticism as with Agreeableness. At the same time, with few exceptions, average personality states were also significantly associated with each other trait consistent with the sizeable amount of shared variance across scales observed in Table 4. However, after adjusting for their shared variance using multiple regression, associations among the same average personality states and traits remained relatively strong, but cross-trait associations were attenuated and became non-significant in most cases (see Table 6) consistent with a strong general factor.

Table 5.

Correlations Between Average Personality States and Baseline Traits for the 20-Item/10-Item Scales.

	Average personality state
Baseline traits	Extraversion	Neuroticism	Conscientiousness	Agreeableness	Openness
BFI-2 Trait subscale
Extraversion
Twenty-item	.74	−.25	.34	.29	.36
Ten-item	.74	−.45	.38	.27	.35
Negative emotionality
Twenty-item	−.32	.63	−.22	−.32	−.23
Ten-item	−.32	.63	−.26	−.29	−.22
Conscientiousness
Twenty-item	.36	−.19	.65	.31	.15
Ten-item	.33	−.31	.66	.26	.15
Agreeableness
Twenty-item	.13	−.21	.23	.57	.21
Ten-item	.09	−.25	.28	.51	.17
Open mindedness
Twenty-item	.31	−.06	.21	.24	.59
Ten-item	.25	−.14	.21	.26	.58
NEO-PI-R Trait subscale
Extraversion
Twenty-item	.47	−.15	.22	.28	.29
Ten-item	.47	−.26	.26	28	.25
Neuroticism
Twenty-item	−.41	.49	−.35	−.36	−.26
Ten-item	−.33	.49	−.35	−.37	−.26
Conscientiousness
Twenty-item	.30	−.23	.44	.27	.16
Ten-item	.26	−.28	.42	.28	.17
Agreeableness
Twenty-item	.13	−.22	.20	.36	.10
Ten-item	.11	−.24	.21	.36	.13
Openness
Twenty-item	.11	−.02	.03	.13	.35
Ten-item	.12	−.05	.06	.11	.33

Note. Average personality states were estimated using random intercepts in multilevel models. Values on the diagonal indicate convergent correlations between personality states and corresponding traits. Bolded values are statistically significant (i.e., the 95% credibility intervals for the point estimates do not contain zero). The BFI-2 was only administered in Sample 2, and the NEO-PI-R in Sample 3. BFI-2 = Big Five Inventory–2; NEO-PI-R = NEO Personality Inventory–Revised.

Table 6.

Associations Between Average Personality States and Baseline Traits Adjusted for Shared State Variance for the 20-Item/10-Item scales.

	Average personality state
Baseline traits	Extraversion	Neuroticism	Conscientiousness	Agreeableness	Openness
BFI-2 Trait subscale
Extraversion
Twenty-item	.74	.02	−.03	.01	.06
Ten-item	.69	−.12	.06	.02	.06
Negative emotionality
Twenty-item	−.02	.60	−.06	−.03	−.13
Ten-item	.14	.65	−.04	.02	−.07
Conscientiousness
Twenty-item	.07	−.05	.62	.08	−.14
Ten-item	.09	−.08	.63	.06	−.11
Agreeableness
Twenty-item	−.16	−.02	.07	.57	.03
Ten-item	−.13	−.05	.16	.46	.01
Open mindedness
Twenty-item	.07	.03	−.03	.04	.56
Ten-item	.05	.08	−.02	.11	.56
NEO-PI-R Trait subscale
Extraversion
Twenty-item	.40	.05	.04	.12	.11
Ten-item	.44	−.09	.10	.14	.07
Neuroticism
Twenty-item	−.27	.49	−.19	−.20	−.22
Ten-item	−.19	.45	−.17	−.20	.19
Conscientiousness
Twenty-item	.06	.01	.34	.04	.01
Ten-item	.04	−.03	.32	.05	.03
Agreeableness
Twenty-item	−.13	−.04	−.01	.23	−.16
Ten-item	−.12	.01	−.01	.23	−.10
Openness
Twenty-item	−.06	.04	−.05	−.03	.32
Ten-item	−.06	.08	−.07	−.06	.31

Note. Associations are standardized regression coefficients from multilevel multiple regression models including all personality state subscales as predictors of BFI-2/NEO-PI-R traits. Average personality states were estimated using random intercepts. Values on the diagonal indicate convergent correlations between personality states and corresponding traits. Bolded values are statistically significant (i.e., the 95% credibility intervals for the point estimates do not contain zero). The BFI-2 was only administered in Sample 2, and the NEO-PI-R in Sample 3. BFI-2 = Big Five Inventory–2; NEO-PI-R = NEO Personality Inventory–Revised.

Next, we examined correlations among personality state subscales and external state variables from the ILD protocol at the within- and between-person levels. Like the personality states, the other daily state variables were decomposed into between- and within-person variance allowing us to investigate the nomological nets at each level. Correlations at the between-person level show the associations among average personality states and average behavior, thoughts, and feelings, and correlations at the within-person level indicate how shifts in personality states relate to shifts in behavior, thoughts, and feelings across days. Of the 220 associations tested per sample (22 external variables × five traits at between- and within-person levels), ~64% replicated (i.e., the correlation’s statistical significance and sign were the same) in all three samples, and an additional ~15% replicated in two samples. Furthermore, these patterns of associations were in line with theory and previous research on personality correlates. We summarize the replicated effects here, and full correlation tables are in Table 7.

Table 7.

Correlations Between the Twenty-Item Personality State Subscales and External Variables.

	Sample 1					Sample 2					Sample 3
Behaviors	E	N	C	A	O	E	N	C	A	O	E	N	C	A	O
Between-person
Drank alcohol	.12	−.08	−.13	−.21	.01	.10	−.11	−.08	.04	.04	.19	−.11	.13	.14	.14
Exercised	.27	−.17	.29	.08	−.02	.14	−.06	.19	.08	.00	.23	−.10	.20	.21	.18
Received emotional support	.06	.27	.01	.11	.02	.00	.21	.03	−.1	.04	.12	.01	.07	.11	.17
Had an argument	.12	.15	−.25	−.21	−.07	.03	.22	.02	−.16	−.07	−.19	.39	−.21	−.24	−.16
Had social contact	−.15	.04	.00	.06	−.18	.08	.13	.11	.12	−.03	.25	−.19	.23	.31	.11
Engaged in leisure	−.19	−.16	−.01	.18	.00	.05	−.17	−.04	.14	.05	.07	−.21	.07	.14	.08
Did housework	.01	−.11	.17	−.01	.16	.11	−.05	.18	.07	.09	−.02	−.11	.02	.05	.08
Ran errands	−.04	−.14	.09	−.17	.13	.19	−.03	.12	.06	.02	.12	.04	.05	.10	.15
Worked/volunteered	.06	−.04	.15	.00	−.05	.19	−.24	.17	.15	.05	.11	.05	.15	.01	−.05
Worked on a team	.22	.01	.19	.13	.05	.26	−.10	.26	.04	.09	.17	−.07	.15	.12	.19
Socialized at work	.23	−.10	.21	.10	.03	.27	−.17	.17	.11	.06	.12	−.13	.16	.19	−.03
Affect
Positive affect	.52	−.47	.39	.27	.25	.41	−.29	.15	.26	.30	.67	−.67	.50	.57	.48
Negative affect	−.21	.47	−.24	−.45	−.18	−.27	.62	−.24	−.4	−.15	−.39	.61	−.31	−.34	−.18
Energy
Felt tired	−.41	.41	−.36	−.27	−.10	−.31	.46	−.18	−.28	−.20	−.44	.47	−.34	−.31	−.22
Ran out of energy	−.36	.39	−.4	−.39	−.20	−.32	.47	−.24	−.28	−.16	−.4	.42	−.3	−.24	−.18
Loneliness
Felt lonely	−.49	.49	−.35	−.29	−.10	−.48	.65	−.34	−.43	−.21	−.4	.49	−.28	−.3	−.14
Felt isolated	−.47	.48	−.31	−.35	−.13	−.47	.62	−.33	−.43	−.18	−.41	.48	−.26	−.29	−.13
Felt left out	−.43	.48	−.29	−.36	−.18	−.45	.61	−.34	−.47	−.19	−.39	.49	−.28	−.32	−.14
Stress
Unable to control things	−.41	.63	−.37	−.22	−.12	−.41	.68	−.32	−.37	−.22	−.45	.58	−.37	−.36	−.16
Confident to handle problems	.13	−.39	.26	.33	.09	.38	−.33	.27	.23	.20	.46	−.58	.40	.41	.23
Things were going your way	.28	−.51	.30	.39	.13	.48	−.51	.34	.37	.31	.58	−.64	.42	.47	.31
Difficulties piling high Within-person Drank alcohol	−.42 .09	.63 −.15	−.40 −.12	−.20 .09	−.16 .02	−.32 .16	.64 −.12	−.23 −.07	−.33 .07	−.20 .07	−.4 .14	.57 −.10	−.35 −.04	−.32 .05	−.18 .03
Exercised	.13	.02	.19	.09	.09	.14	−.06	.18	.10	.11	.12	−.03	.10	.05	.04
Received emotional support	−.04	.12	−.02	.02	.01	.08	.09	.02	.09	.00	.09	.03	.01	.02	.05
Had an argument	−.10	.23	.01	−.16	−.01	.01	.15	.03	−.24	−.02	.02	.21	.01	−.10	.01
Had social contact	.06	−.01	.02	.10	.10	.13	.02	.13	.06	.07	.19	.00	.04	.07	.12
Engaged in leisure	.12	−.04	.07	.14	.01	−.02	−.09	−.09	.11	.02	.04	−.10	−.07	.03	.00
Did housework	.04	.03	.06	.07	.05	−.01	−.08	.04	.06	.07	.03	−.02	.06	.02	.03
Ran errands	.11	−.06	.01	.06	.06	.07	−.08	−.04	.10	.03	.03	.03	.01	.03	.03
Worked/volunteered	.02	.10	.17	.00	.01	.09	.16	.32	.00	.05	.11	.06	.25	−.03	.07
Worked on a team	.04	.07	.20	.02	.03	.09	.12	.29	.01	.08	.16	−.05	.09	.09	.10
Socialized at work	.02	.11	.15	.05	.01	.11	.13	.31	.00	.06	.12	−.08	.03	.10	.06
Affect
Positive affect	.29	−.28	.03	.19	.18	.27	−.25	.10	.26	.21	.40	−.27	.15	.24	.23
Negative affect	−.26	.30	−.02	−.23	−.13	−.14	.31	.01	−.23	−.10	−.17	.38	−.05	−.22	−.14
Energy
Felt tired	−.21	.10	−.05	−.04	−.11	−.29	.06	−.13	−.10	−.13	−.24	.12	−.12	−.09	−.13
Ran out of energy	−.23	.08	−.07	−.06	−.12	−.29	.11	−.16	−.14	−.15	−.24	.12	−.13	−.10	−.13
Loneliness
Felt lonely	−.26	.26	−.04	−.16	−.10	−.30	.16	−.12	−.21	−.15	−.17	.14	−.09	−.10	−.13
Felt isolated	−.24	.24	−.05	−.19	−.08	−.25	.15	−.10	−.21	−.12	−.14	.14	−.06	−.09	−.08
Felt left out	−.22	.20	−.06	−.14	−.13	−.17	.13	−.06	−.20	−.11	−.12	.11	−.09	−.09	−.09
Stress
Unable to control things	−.20	.32	−.04	−.16	−.11	−.14	.21	−.04	−.17	−.11	−.10	.18	−.03	−.07	−.08
Confident to handle problems	.15	−.16	.06	.11	.08	.09	−.13	.02	.10	.08	.06	−.10	.05	.09	.06
Things were going your way	.23	−.28	.09	.21	.13	.18	−.18	.08	.17	.13	.14	−.15	.07	.14	.11
Difficulties piling high	−.13	.25	.02	−.11	−.06	−.20	.29	−.01	−.17	−.10	−.13	.17	−.06	−.11	−.08

Note. All items were assessed at the daily level except for affect in Sample 1 and Sample 2 which was measured multiple times per day and averaged for a daily score. Personality states and external variables were decomposed into between- and within-person variance and correlations were estimated separately at each level.

Bolded correlations were statistically significant (i.e., the credibility intervals do not contain zero). E = Extraversion; N = Neuroticism; C = Conscientiousness; A = Agreeableness; O = Openness.

Correlations at the between-person level indicated that individuals who reported more Extraversion on average were more likely to report exercising over the study period. Those who reported more Conscientiousness were more likely to report exercising, going to work, and doing housework and were less likely to report having an argument. Individuals who reported more Extraversion, Conscientiousness, Agreeableness, or Openness on average reported more positive affect and energy, and less negative affect, loneliness, and stress. In contrast to the positive valence of the other average personality states, individuals who reported more Neuroticism overall reported less positive affect and energy, and more negative affect, loneliness, and stress. Those who reported more Neuroticism were also more likely to report having an argument and receiving emotional support and were less likely to report engaging in leisure activities.

Turning to correlations at the within-person level, we found that on days people reported more Extraversion than was typical of them, they were more likely than their usual to report drinking alcohol, socializing, running errands, going to work, and exercising. When people reported more Agreeableness, they were less likely to report having an argument and more likely to report drinking alcohol, engaging in leisure activities, and exercising. On days people reported higher Conscientiousness, they were also more likely to report going to work and exercising but were less likely to report drinking alcohol than their usual. On days people reported more Neuroticism, they were more likely than their usual to report having an argument and working and were less likely to report engaging in leisure activities and drinking alcohol. When people reported more Openness, they were more likely to report exercising and socializing the same day.

Personality states were valenced at the within-person level like at the between-person level; on days when people reported more Extraversion, Agreeableness, or Openness than their usual, they reported more positive affect and energy, and less negative affect, loneliness, and stress whereas on days people reported higher Neuroticism they reported more negative affect, stress, and loneliness. In contrast to the between-person level, however, when people reported more Conscientiousness than is typical of them, they reported somewhat more positive affect, but not less negative affect or stress.

Nomological Homomorphy With the BFI-2/NEO-PI-R.

For the final construct validity analyses, we compared the nomological nets of our personality state subscales and BFI-2 and NEO-PI-R trait scales. These analyses provide distinct, psychometric information from convergent validity (i.e., correlations between personality state scales and global trait assessments) by indexing whether our personality state scale and two global trait measures have a comparable pattern of associations with another set of variables. This was done by first estimating a vector of correlations between a personality state subscale and all external variables, and a vector of correlations between the corresponding BFI-2/NEO-PI-R trait and those same external variables. Then, we correlated these resulting vector “profiles” to evaluate their nomological homomorphy. Higher correlations indicate that the state and trait measures of personality are tapping similar constructs.

Table 8 shows the vector correlations between BFI-2/NEO-PI-R traits and both levels of personality states. The nomological profiles between corresponding average personality states and traits had a mean correlation of .90 (range = .74-.98) with the BFI-2 in S2 and .81 (range = .47-.99) with the NEO-PI-R in S3. Within-person personality state variance had lower nomological homomorphy with the trait measures, but with the exception of state Agreeableness and Openness and their NEO-PI-R counterparts, all vector correlations were significant (ps < .001). The average vector correlation at the within-person level was .77 (range = .64-.88) with the BFI-2 traits and .66 (range = .20-.90) with the NEO-PI-R. At both levels, Extraversion, Neuroticism, and Conscientiousness showed the most nomological homomorphy, whereas Agreeableness and Openness showed relatively less.

Table 8.

Nomological Vector Correlations Among Between- and Within-Person Variance in Personality States and Baseline Traits for 20-Item and 10-Item scales.

	Between-person, rv	Within-person, rv
BFI-2
Extraversion
Twenty-item	.96	.89
Ten-item	.98	.89
Neuroticism
Twenty-item	.98	.76
Ten-item	.98	.87
Conscientiousness
Twenty-item	.95	.64
Ten-item	.95	.67
Agreeableness
Twenty-item	.87	.88
Ten-item	.89	.77
Openness
Twenty-item	.74	.68
Ten-item	.75	.64
Average
Twenty-item	.90	.77
Ten-item	.91	.77
NEO-PI-R
Extraversion
Twenty-item	.93	.88
Ten-item	.95	.85
Neuroticism
Twenty-item	.99	.90
Ten-item	.98	.91
Conscientiousness
Twenty-item	.95	.64
Ten-item	.95	.72
Agreeableness
Twenty-item	.73	.68
Ten-item	.71	.62
Openness
Twenty-item	.47	.20
Ten-item	.44	.24
Average
Twenty-item	.81	.66
Ten-item	.81	.67

Note. Vector correlations were calculated by taking the vector of correlations between each personality state subscale and external variables, and between each baseline trait and those same external variables, then correlating the vectors of corresponding personality constructs. Between-person variance reflects each person’s average personality state level and within-person variance reflects each person’s average daily deviations from their average level. Bolded values indicate correlation is statistically significant (p < .001). The BFI-2 was only administered in Sample 2, and the NEO-PI-R in Sample 3. r_v = vector correlations indexing nomological homomorphy; BFI-2 = Big Five Inventory–2; NEO-PI-R = NEO Personality Inventory–Revised.

Ten-Item Scale Structural and Construct Validity

Mplus output and a table of model fit indices for CFAs using different combinations of candidate items for the 10-item scale are available in the online supplementary files. Approximately half of the models tested were non-positive definitive, likely due to collinearity of certain items (e.g., Relaxed to Tense and At Ease to Nervous). Of models that were positive definite, all achieved adequate model fit, so the final items were selected to maximize content coverage. These items are marked with superscript in Table 1. Before calculating subscale scores, we removed outliers following the same criteria used for the 20-item scale, for a total of 1,039 participants and 7,486 days.² Intercorrelations between subscales were similar to the 20-item scale as shown in Table 4.

Descriptive statistics, omegas, and ICCs for the brief personality state subscales are shown in Table 2. The average reliabilities of these scales were lower than the 20-item scales (ω_between = .79, ω_within = .43). Within- and between-person variability of the 10-item scales was comparable to the 20-item scales with an average ICC of .57 (range = .43-.79).

Table 5 shows the correlations between the brief subscales and the BFI-2/NEO-PI-R. Convergence between methods was comparable to the 20-item scale with the highest correlations between daily personality subscales and corresponding traits. Like the 20-item scale, cross-trait correlations and multiple regression results suggested a strong general factor as seen in Table 6.

We compared the content coverage of 10- and 20-item scales by indexing their nomological homomorphy. This was done by applying the same vector correlation approach described previously; that is, we took the vector of correlations between each personality state subscale and external variables in the 10-item and 20-item versions separately then correlated the vector profiles. We found there were only negligible differences in correlations with the set of external variables. The average nomological vector correlation between the 10- and 20-item daily personality subscales was .99 between-persons and .97 within-persons indicating the reduced set of items captured a nearly identical nomological net at both levels. These vector correlations and the 10-item scale correlations with external variables are reported in the online supplementary material.

Nomological Vector Similarities With the BFI-2/NEO-PI-R.

Vector correlations between the 10-item scale and the BFI-2 and NEO-PI-R can be found in Table 8. The average vector correlations differed by .01 or less from the 20-item scale.

Discussion

Although there is much enthusiasm for using ILDs in naturalistic settings to test theories integrating personality trait structure with processes, the reliability and validity of available scales for studying both the between- and within-person levels of analysis has yet to be established. In this study, we evaluated the psychometrics of two versions of a personality state scale for daily assessment like those in active use. Across three samples, we found initial evidence that the ILD scales have the potential to validly measure Big Five traits and states, while also identifying several notable limitations. We found alignment between the Big Five traits and personality states, and the ILD scales captured a similar nomological net as two, widely used personality trait measures. However, we also found that the scales had low reliability, and the magnitude of correlations with criterion variables were small. Because we did not set out to construct an authoritative ILD instrument from the ground-up, these limitations should be addressed with future concerted efforts following psychological scale development best practices in future research (see, e.g., Clark & Watson, 2019, for a general overview on scale construction; Horstmann & Ziegler, 2020, for recommendations on personality state measurement; Wright & Zimmermann, 2019, for ILD scale development). In addition to the general recommendation of more psychometric work, we offer specific suggestions based on our results for the next steps needed to mitigate these limitations when using personality state measures.

Differences Between Trait and State Levels

Parity in the structures and nomological nets at the between- and within-person levels supports the possibility of using these ILD scales to investigate links between personality traits and state-level processes. Areas of divergence between levels may have theoretical importance as well by revealing the nature of functional trait–state relationships proposed by personality theories. For instance, in line with work showing robust associations between Conscientiousness and positive life outcomes (Roberts et al., 2007), we found that individuals who reported more Conscientiousness on average tended to report less negative affect and stress. However, on days people were more Conscientious than was typical of them, they did not report less negative affect and stress than usual. This suggests that being more Conscientious in general is adaptive, but instances of elevated Conscientiousness are not necessarily pleasant. These results are consistent with theories that propose a willingness/ability to sacrifice immediate rewards for long-term goal pursuit is a core mechanism underlying individual differences in Conscientiousness (Carver, 2016; DeYoung, 2015) and points to the rich opportunity for testing theoretical models like these with ILDs that allow separation of between- and within-person levels. Indeed, Beckmann et al. (2010) have illustrated this aspect of Conscientiousness in the within-person association with negative affect in managers.

Other discrepancies between levels have less apparent theoretical grounding and provoke more questions than they provide answers. In particular, Agreeableness items formed a clear between-person factor but did not cohere within-person. Is this result because instances of warmth, generosity, trust, and politeness follow separate, day-to-day processes despite strongly covarying between people? Or is the issue methodological? Items validated for assessing traits cannot a priori be assumed to validly measure states, so it is possible that the items we used, which were adapted from trait measures, are poor indicators of state Agreeableness. For example, it makes conceptual sense that warmth may be more context-dependent and thus a better state indicator, whereas generosity may be more trait-like and not as effective for measuring states (and indeed warmth had more within-person variance than generosity [ICC_warmth = .39, ICC_generosity = .53]; see online supplementary materials for all item-level ICCs). Underscoring the possibility that state Agreeableness is especially distinct from its trait counterpart, we found it had low nomological homomorphy with BFI-2/NEO-PI-R Agreeableness, consistent with previous work (Rauthmann et al., 2019). Concerted item construction efforts are needed to go beyond simply changing the instructions for trait measures to determine what items are best for assessing Agreeableness states.

Related to the role of context, our sampling schedule may not have been well-suited for assessing variance (or covariance) in Agreeableness, as shifts in this state may be somewhat limited to interpersonal situations. Although all personality states are thought to fluctuate in response to situational cues, other traits may have more manifestations that extend beyond the duration of an event (e.g., a social interaction) such as state Extraversion and Neuroticism reflected in day-to-day mood changes, and Conscientiousness in day-to-day motivation levels. Researchers could use event-contingent designs to sample social interactions and see whether homing in on a proximal context results in a more coherent within-person Agreeableness factor. If this were the case, it would suggest that within-person changes in Agreeableness occur primarily in interpersonal contexts and would help explain the weak factor structure we found sampling at the daily level. Alternatively, it may be that state Agreeableness does persist beyond social situations, but has more intrapersonal components (e.g., cognitive or affective components like thinking prosocial thoughts, reflecting warmly on relationships) that are not well captured by the current set of items.

Low Scale Reliability

Judging reliability of personality state scales is difficult without accepted guidelines for omega coefficients; but, according to conventional standards of internal consistency (i.e., alpha coefficients), reliability for the 20-item scale was good at the between-person level and poor at the within-person level, and reliability for the 10-item scale was moderate to poor at both levels. This shortcoming is not unique to the personality state scales we used, as low reliability is commonly reported in ILD research (e.g., Halvorson et al., 2021; Zimmerman et al., 2019; Zygar et al., 2018). One reason for generally low reliability of ILD instruments is the premium placed on scale brevity. Although ultra-brief scales with comparable reliabilities to what we found in our study have been regularly used in the published individual differences literature (e.g., Donnellan et al., 2006), a scale’s validity is inherently limited by its reliability. In addition to the impact of scale length, within-person scale variance tends to have particularly low reliability because substantive day-to-day variability of interest and unsystematic residual (i.e., error) variance are incorporated into this part of the model. The possible consequences of error variance on internal consistency becomes even greater for scales with a small number of items like those used in ILDs.

To maximize the potential of personality state measurement without sacrificing the defining advantage of sampling frequency in ILDs, researchers can improve reliability by assessing fewer constructs in each survey but with more items per construct. For instance, studies may need to focus on measuring one or two personality states over the ILD protocol or randomly assess a different trait at each survey. Another consideration is that the items we administered were not selected following best practices of scale construction. Future scale development efforts can increase reliability by following standard procedures of auditioning a large pool of items for each personality state scale and narrowing them down to those that achieve good internal consistency.

Do the Magnitude of Correlations Represent True Effects or Method Effects?

An unresolved question from our analyses is whether the effect sizes we observed, which could be considered small in magnitude (mean significant correlations r_{within-person} = |.15|, r_{between-person} = |.32|), reflect the true relationship between personality states and the external criterion or byproducts of methodological choices. Our results are not unprecedented; comparable effect sizes and the finding of relatively smaller associations at the within- versus between-level are often reported in ILD studies of other state constructs (e.g., Giacomin & Jordan, 2016; Halvorson et al., 2021; Ringwald & Wright, 2021), and the correlation between personality and external criterion, regardless of method, tends to be around the .10 to .30 range we found (Gignac & Szodorai, 2016; Richard et al., 2003). Conceptually, because of the multifinality of personality, we might expect correlations with a particular behavior, thought, or emotional state to be small. For example, at the between-person level, some people’s high level of Conscientiousness manifests in keeping a tidy house but for others it manifests in maintaining a strict exercise regimen. At the within-person level, there is even more variability as some days a person’s Conscientiousness manifests in doing housework, whereas on other days it manifests in exercising. This variability in state expression is also part of the reason the within-person structure of personality tends to be less well-defined than the between-person structure (e.g., Edershile et al., 2019; Wright & Simms, 2016). Thus, despite clear regularities in the types of behavior, thoughts, and emotions that covary with personality states and traits, the true correlation with any specific instantiation may be small in magnitude and the within-person factor loadings may be relatively weak.

However, there needs to be more systematic work comparing different sampling schedules and scale designs to determine whether the effect sizes could be stronger with other methods. We measured personality once per day, but personality states fluctuate within a day, so a more frequent sampling rate could draw a more direct link between personality and other state-level variables and result in stronger correlations. In support of this possibility, there is some evidence that stronger associations would be found with shorter lags between state measurements (Rauthmann et al., 2016). Another potential way to tighten the relationship between states would be to ask participants to rate their personality states in relation to a circumscribed situation (e.g., “how would you describe yourself during the social interaction”) rather than over a particular time interval (e.g., “how would you describe yourself in the past 24 hours”) as has been common practice in this literature. We suggested event-contingent sampling methods for testing the context-specificity of Agreeableness, but this approach could be broadly applied to strengthen observed correlations by assessing personality states during theoretically trait-specific situations (e.g., Conscientiousness when engaging in a difficult task, or Extraversion when drinking alcohol).

The effect sizes we found might also be attenuated due to the noted low reliability of the personality state scales and the low reliability of the external criteria, which were almost all single item measures in our study. In addition to designing studies to accommodate more items per construct as discussed earlier, using analytic approaches that reduce error variance, like latent variable modeling that isolates the reliable shared variance (e.g., multilevel structural equation modeling; Sadikaj et al., 2021), could help detect meaningful effects with brief ILD scales. As a proof-of-concept, we reanalyzed some of the correlations in our study using latent scores and found consistent increases in the size of significant effects (differences ranged from |.02| to |.11|). These supplementary analyses suggest that (a) low reliability indeed contributes to the small magnitude of correlations and (b) latent variable modeling could help address the impact of poor reliability inherent to brief scales.

An issue related to the size and meaningfulness of associations with personality state measures is the question of whether they provide unique information from trait measures. Establishing the incremental validity of state measures for theory-testing purposes requires showing they are better predictors of external criteria than trait measures (i.e., have stronger associations) because they are tapping the state construct. Determining if incremental associations are attributable to the target construct, in turn, means ruling out the possibility that extraneous aspects of the method are driving the associations. For example, if a person completed a trait assessment every day, the average of those reports could incrementally predict outcomes over a single trait measure because of the precision gained from repeated measures, not because it tapped a distinct construct. The approaches discussed previously to unconfound method and construct by isolating the effects of different instructions, sampling schedules, and items can all be used to ensure incremental validity is established for a personality state construct as measured by a given scale.

The General Personality Factor

Although we found a number of specific associations that distinguish personality states from one another and that align with the nomological nets of corresponding traits (e.g., Neuroticism with arguments, Conscientiousness with working), there was also evidence for a strong general factor. The ILD scales had high intercorrelations, and they had high correlations with nearly all BFI-2/NEO-PI-R traits, not just the personality state’s trait equivalent. After adjusting for their shared variance, however, personality states uniquely and almost exclusively associated with the trait equivalent. There is little precedence with which to compare these results as few Big Five personality state studies reported scale intercorrelations or correlations with global trait measures. Two studies that used comparable, adjective-based ILD personality state measures showed more discriminability with BFI traits (Matz & Harari, 2020; Sherman et al., 2015) and among state scales (Matz & Harari, 2020) than we found, but another study reported intercorrelations among state scales similar to ours (Bleidorn, 2009). Our personality state scales also had weak discriminability with some external variables, such that the “adaptive” traits (i.e., Extraversion, Conscientiousness, Agreeableness, and Openness) associated with nearly all positively valenced outcomes and Neuroticism with nearly all negatively valenced ones.

Taken together, the personality state scores seem to measure valid, trait-relevant variance as well as nonspecific but valenced variance. This valenced, general factor of personality is not unique to our ILD scales as a large body of research suggests it is a common feature of self-report personality trait measures (Irwing, 2013; Revelle & Wilt, 2013). There is disagreement about whether personality scales of similar valence tend to cluster together for substantive or artifactual reasons. Some researchers have speculated that the general factor is an indicator of social adjustment (van der Linden et al., 2016), but many studies suggest it is primarily the product of rater-specific variance (Anusic et al., 2009; DeYoung, 2006; Gnambs, 2013; McCrae et al., 2008) or social desirability/evaluative bias (Bäckström, 2007; Pettersson et al., 2012).

The contribution of these sources of bias found in trait assessments of personality has yet to be thoroughly investigated in state assessments. Validation of personality state scales would benefit from undergoing the same empirical scrutiny as trait measures such as collecting state-level informant reports of personality in dyadic ILD studies to isolate rater-specific variance or directly estimating a latent general factor from personality states and examining its correlates. It has been shown for trait assessments that changing the wording of items to be more neutral attenuates their common variance (Bäckström et al., 2009), so testing the structure of states using items that de-emphasize valence could be another way to achieve better differentiation. A challenge for state assessment in particular is that affect is a core component of some personality states, meaning the strong valence may accurately reflect the construct rather than being a nuisance to correct for.

The temporal dimension of ILDs also adds new information that cannot be gleaned from cross-sectional measures, but that could bear on interpretation of the general factor. In our study, personality state and baseline trait ratings of the middle-aged sample had a stronger general factor, but also less within-person variability than the other two samples (see online supplemental material for sample specific results). Previous work has similarly shown older adults are less variable in some personality states than younger adults (Noftle & Fleeson, 2010). This could be evidence that the general factor reflects social adjustment because, from a developmental view of personality maturation, older adults tend to be more emotionally and behaviorally stable (i.e., less variable from day-to-day) than younger adults (Roberts et al., 2006). On the other hand, there is research showing older adults rate their personality as more “normal” than younger adults (Wood et al., 2007), so the low variability and strong general factor could be evidence for acquiescent responding. The relationship between variability and structure of personality states has psychometric and theoretical implications that should be explored further with ILDs.

Conclusion

As personality assessment expands beyond descriptive, between-person trait structure to include explanatory, within-person processes it is important to ensure the methods being introduced for this purpose are psychometrically sound. Our study adds to this necessary foundation by showing initial support for the validity of two versions of a personality state scale that generalized across individuals from emerging adulthood to midlife in three independent samples. We examined the scale’s construct validity by testing associations with a wide array of behavioral, cognitive, and affective variables and found preliminary evidence that the 20-item and 10-item scales capture similar content as well-validated, global personality inventories. At the same time, our evaluation of these personality state scales for ILD identifies a pressing need for additional psychometric work using these methods to fulfill their promise of advancing personality science.