Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Apr 5.
Published in final edited form as: Qual Life Res. 2015 Nov 13;25(3):507–516. doi: 10.1007/s11136-015-1170-9

Advantages and Psychometric Validation of Proximal Intensive Assessments of Patient Reported Outcomes Collected in Daily Life

Eve B Carlson 1, Nigel P Field 2, Josef I Ruzek 3, Richard A Bryant 4, Constance J Dalenberg 5, Terence M Keane 6, David A Spain 7
PMCID: PMC5381966  NIHMSID: NIHMS802672  PMID: 26567018

Abstract

Objectives

Ambulatory assessment data collection methods are increasingly used to study behavior, experiences, and patient reported outcomes (PROs) such as emotions, cognitions, and symptoms in clinical samples. Data collected close in time at frequent and fixed intervals can assess PROs that are discrete or changing rapidly and provide information about temporal dynamics or mechanisms of change in clinical samples and individuals, but clinical researchers have not yet routinely and systematically investigated the reliability and validity of such measures or their potential added value over conventional measures. The present study provides a comprehensive, systematic evaluation of the psychometrics of several Proximal Intensive Assessment (PIA) measures in a clinical sample and investigates whether PIA appears to assess meaningful differences in phenomena over time.

Methods

Data was collected on a variety of psychopathology constructs on handheld devices every 4 hours for 7 days from 62 adults recently exposed to traumatic injury of themselves or a family member. Data was also collected on standard self-report measures of the same constructs at the time of enrollment, one week after enrollment, and two months after injury.

Results

For all measure scores, results showed good internal consistency across items and within persons over time, provided evidence of convergent, divergent, and construct validity, and showed significant between and within-subject variability.

Conclusions

Results indicate that PIA measures can provide valid measurement of psychopathology in a clinical sample. PIA may be useful to study mechanisms of change in clinical contexts, identify targets for change, and gauge treatment progress.

Patient-reported outcomes (PROs) are an essential source of information about psychological phenomena, experiences, and behavior in clinical research and treatment, but standard measures that collect PROs on one occasion and require retrospective summaries over long periods of time are subject to error from many sources [1] and do not allow study of the short-term temporal dynamics of phenomena or relationships among phenomena. Collecting reports closer in time to the phenomena of interest during daily life reduces error associated with retrospection and summary reports [2] and makes it possible to study clinically-relevant states, symptoms, emotions, cognitions, experiences, and behaviors [3]. Understanding of the temporal dynamics of such phenomena and the temporal relationships among them can provide insight into mechanisms that cause or maintain disordered behavior or are involved in clinical improvement [4]. As with any measures, empirical evidence of reliability and validity should be examined to determine whether such intensive summary reports accurately assess constructs [5]. In areas of social and personality psychology, some researchers have begun to include analyses of the psychometrics of intensive measures [6–9], but in clinical research, psychometric analyses of intensive measures has been infrequent and very limited in scope [10–12].

We discuss below how summary reports for relatively brief periods of time collected at regular intervals, which we refer to as proximal intensive assessment (PIA), fits into the rubric for ambulatory assessment or diary data collection methods. We then review the particular advantages of collecting PIA reports in clinical research and care, and present an example of a comprehensive, systematic evaluation of the psychometrics of data collected for several PIA measures in a clinical sample recently exposed to traumatic stress related to severe injury.

Types of Intensive Data Collection Methods and Their Uses

Wheeler and Reis [13] described three categories of intensive data collection methods that have been used in ambulatory assessment research to study daily life: signal-contingent sampling, event-contingent sampling, and interval-contingent sampling. Experience Sampling Method (ESM) [14] and Ecological Momentary Assessment (EMA) [15] are examples of signal-contingent sampling methods, which collect reports on current experience “in the moment” at randomly selected times within defined time intervals. Event-contingent sampling involves recording data at the time of a particular event or experience, such as recording one’s mood every time a cigarette is smoked. PIA refers to interval-contingent reports that summarize experiences during a relatively brief period of time at frequent intervals or set times of day (e.g., in the past four hours). Figure 1 shows examples of signal patterns and periods of time assessed for an ESM and a PIA measurement schedule with a frequency of four times per day. As the P and E arrows show, PIA measures assess blocks of time, while ESM and EMA sample experiences at specific moments in time.

Figure 1. Sample Signal Patterns and Reporting Period for Proximal, Fixed-Interval, Retrospective and Experience Sampling Method Data Collection Methods.

Figure 1

Open in a new tab

Schematic chart showing the time and reporting periods for the Proximal Intensive Assessment and Experience Sampling Method data collection methods. The P and E arrows delineate the periods of time that are reported on for each assessment period.

Distinguishing among these sampling methods is important when considering which type of sampling is optimal for studying various phenomena and what analyses are appropriate to establish the reliability, validity, and usefulness of measures. The developers of ESM were interested in capturing and understanding individuals’ momentary experiences, and the strategy to deliver signals at random times within intervals was adopted in order to reduce the influence of anticipation on reports of momentary experience [14]. In contrast, if a researcher or clinician wishes to obtain complete reports of experiences over a period of time in order to capture discrete or fleeting phenomena or to study fluctuations of phenomena over time, it may be optimal for signals to be delivered at the end of each time interval. Signaling at the end of time intervals also allows for reports on both continuous experiences (such as emotions) and discrete or fleeting experiences such as social interactions, symptoms that are brief, or behaviors. Figure 1 shows that there is a trade-off between minimizing sensitization to signals (with variable interval data collection) and data completeness (with fixed-interval data collection).

Potential Advantages of Proximal Intensive Assessments

The temporal dynamics of phenomena may be important indicators of disordered behavior, risk, or the “fit” of different behavior change methods to an individual, and they may influence later perceptions and related behavioral choices. In a study of pain in bone marrow transplant patients, the intensity of pain in the last hour of the day was more strongly associated with the summary report on the entire day than the intensity of pain reported in the other hours of the day [16]. A study of colonoscopy patients showed similar results, and those who had a period of mild pain added to the procedure reported less overall pain than those who had a routine colonoscopy [17]. Memories of pain at the end of a procedure also appear to affect important patient behaviors. In a randomized trial, patients undergoing a colonoscopy that had a less-painful ending added returned for follow-up colonoscopies at a significantly greater rate than those who had routine colonoscopies [18]. In this example, it seems that the temporal dynamics of pain affected perceptions which, in turn, affected behavior.

Temporal relations among different phenomnena may also be important in understanding how intrapsychic and situational factors influence emotions, cognitions, behavior, or physical responses. For example, in a study of the impact of daily experiences of task demand and control, the relationship between participants’ ratings every 45 minutes of how demanding activities seemed and how much control they felt while doing the activity was related to their subsequent blood pressure [19]. Fixed-interval intensive assessment methods such as PIA allow study of this type of complex interaction between perceptions and behaviors, emotions, and physical responses over relatively brief periods of time.

Investigating the Psychometric Properties of PIA and Meaningful Variation in PIA Data

The purpose of this study was to provide a comprehensive, systematic evaluation of the psychometrics of several Proximal Intensive Assessment (PIA) measures in a clinical sample and to investigate whether PIA appears to assess meaningful differences in phenomena over time. The primary differences between PIA reports and standard self-report measures are the intensity of data collection and the length of the recall period. For this reason, psychometric data analysis methods used for standard retrospective measures are also appropriate for evaluating PIA reliability and validity. In a sample of adults who were recently exposed to traumatic stress, we examined the reliability (internal consistency) of PIA measures of symptoms, moods, and cognitions and conducted analyses of their convergent and divergent validity and other aspects of construct validity. To investigate the capacity of PIA to assess meaningful within and between-subjects variability and to provide information beyond that available from standard self-report measures, we examined the interindividual and intraindividual variability of participants’ PIA reports and their convergence and divergence with standard measures given at different times.

Method

Participants

Participants were severely injured patients who were treated in a trauma center and admitted to a university hospital and spouses/partners or first degree relatives of severely-injured admitted patients. Only one patient or family member from a particular family was included to preserve independence of observations. No patients being treated in intensive care units were recruited, but family members of such patients were recruited. An initial sample of 20 participants was enrolled in an observational study of psychological responses to traumatic stress, and 65 additional participants were enrolled in a randomized controlled trial to study reactivity to PIA data collection (Carlson, unpublished data). The participants of both samples were recruited from the same patient and family population. The two samples were combined for this study to provide a larger sample for the psychometric analyses. Participant trauma type and demographics are provided in Table 1.

Table 1.

Trauma Type and Demographics

Trauma Type
 Motor vehicle accidents 44%
 Learning/witnessing injury of family member 36%
 Accidental injury at home or work 17%
 Interpersonal violence 3%
Gender
 Male 50%
 Female 50%
Age
 Mean = 42 (SD = 14.1); Range 16 to 73
Marital Status
 Married or living with partner 50%
 Separated, divorced, or widowed 26%
 Never married 24%
Self-Reported Race/Ethnicity
 Caucasian 60.5%
 Hispanic/Latino 14.5%
 Asian or Pacific Islander 10.5%
 African American 5.3%
 Mixed race or other 5.3%
Education
 Mean = 15 years; Range 10 to 21 years
Socioeconomic Status
 Upper or upper middle class 15%
 Middle class 37%
 Lower middle class 46%
 Lower class 1%
Open in a new tab

Measures

Standard Retrospective Summary Measures

The Screen for Posttraumatic Stress Symptoms (SPTSS) was used to assess PTSD symptoms. The SPTSS has demonstrated good internal validity (α=.91) and good concurrent validity when correlated with other measures of PTSD [20; 21]. The SPTSS has also demonstrated high sensitivity (.89 to .94) and adequate to high specificity (.60 to .89) to identify those with a PTSD diagnosis on a structured diagnostic interview [20; 21]. Cronbach’s alpha for baseline SPTSS scores was .86. The Dissociative Symptoms Scale (DSS) was used to assess disruptive dissociative symptoms including derealization, depersonalization, gaps in awareness and memory, and dissociative reexperiencing. The DSS has shown good internal validity (α’s from .89 to .94) and strong correlations with other measures of dissociation (r = .56-.66) and PTSD (r=.51-.55) in clinical and community samples [22]. Cronbach’s alpha for baseline DSS scores was .91. A modified version of the Post-Traumatic Cognitions Inventory (M-PTCI) was used to assess negative cognitions about the self, the world, and blame for the event. The PTCI has shown good test-retest reliability, strong internal consistency (α =.97), and correlations with symptoms of PTSD (r=.79), depression (r=.75), and anxiety (r=.75) [23]. In the original version of the PTCI, all items described negative cognitions, and response options ranged from totally agree to totally disagree. In order to produce items with response options that could be compared to PIA responses, PTCI items were modified to reverse the wording and scoring for half of the items and separate scores were calculated for positive M-PTCI and negative M-PTCI. We report here on negative M-PTCI only. Cronbach’s alpha was .83 for baseline negative PT cognitions scores. The Emotional Approach Coping (EAC) scales were used to assess emotional processing and emotional expression. Sample items include “I acknowledge my emotions” and “I let my feelings come out freely”. These scales have been found to be reliable and valid to assess situational coping without confounding with negative emotions or psychopathology [24]. Cronbach’s alpha was .93 for one-week EAC scores.

The time frame “since the event” was used for all standard retrospective summary measures when administered at the time of enrollment. While the time since the event differed across participants, the response options for these measures were grounded in time so that scores quantified the rates of symptoms. For example, response options of the SPTSS and DSS were not at all, once or twice, almost every day, about once a day, and more than once a day. A time frame of one week was used for the measures when administered at the end of the PIA assessments and at two months after the injury.

PIA measures

PIA items to assess symptoms of PTSD, dissociation, posttraumatic cognitions, and emotional approach coping items were drawn from the SPTSS, DSS, M-PTCI, and EAC scales described above. PTSD items included all 17 items from the SPTSS. Dissociation items were the ten DSS items most strongly related to total DSS score in several validation samples [22]. M-PTCI items included the ten items most strongly correlated with total PTCI scores in the initial validation study [25] with five items reversed to assess positive cognitions (as described above). Emotional approach coping items were those most strongly correlated with total scale scores in pilot data. The items used the stem “How much did you deal with your feelings by” and included two emotional processing items (“…realizing that your feelings are valid and important?” and “acknowledging your emotions”) and two emotional expression items (“…letting your feelings come out freely?” and “…allowing yourself to express your emotions.”). The mean for the four items was used in analyses. To investigate aspects of social support, we assessed variables hypothesized to be the mechanism of action of social support, including feeling free to talk about the trauma, emotional expression, emotional processing [26; 27]. One item also inquired about how supported the participant felt if he/she did talk about the event. The sum of feeling free to talk and feeling supported were used as an index of perceived social support. Mood items were drawn from Emmons, McCullough, and Tsang [28] to assess a wide range of positive (hopeful, attentive, grateful, strong, fortunate, forgiving, enthusiastic, alert, happy, energetic, good mood, and calm) and negative emotions (upset, frustrated, sad, stressed, angry, nervous, ashamed, guilty). We added three trauma-specific negative mood items (feeling numb, helpless, feeling overwhelmed), one relevant positive mood (relieved), and eliminated two items (proud and tired) that did not relate to either positive or negative mood factors in analyses of pilot data. All PIA items used a “past four hours” time frame for reporting. For each set of items assessing a construct, stems were displayed to remind participants of the reporting time frame: “In the past four hours, how much did you…” To minimize the time required to answer questions and avoid confusion due to changing response formats across items, response options were simple and distinctive, and the same options were used for all items (0 = not at all, 1 = a little bit, 2 = some, 3 = a lot).

Procedures

Admitted patients were approached in their hospital rooms between one and fourteen days after injury. Family members were approached in the patient’s room or in the intensive care waiting area. Family members were approached only when the patient was no longer in danger of death from injuries to avoid undue burden on persons under duress and because we were interested in studying recovery. After providing informed consent, participants completed the SPTSS, DSS, and M-PTCI at the time of enrollment (baseline). These were also completed one week later (at the end of PIA assessments), and two months post-trauma. The EAC was completed at the one-week and two-month time points. Participants completing the PIA assessments responded to PIA items administered automatically via Personal Digital Assistants (PDAs) four times per day. They were asked to collect data for at least seven days. The PDAs ran software that was a modified version of the iESP program. Intel Research Seattle developed iESP based on the Experience Sampling Program (ESP) [29]. Alerts were programmed to occur four times per day at 10AM, 2 PM, 6PM, and 10PM. If participants did not respond to an alert, it would be repeated every 5 minutes. If the participant did not respond to three alerts, the PDA would automatically turn off. To allow for interruptions, participants could pause in responding for up to 10 minutes during the session before the PDA would automatically turn off. PIA items were automatically presented one at a time and responses were date and time stamped and could not be changed once passed.

Results

Retention and Data Yield

Figure 2 shows the flow of participants from enrollment to data analysis. Analysis of baseline SPTSS scores (available for 74 of 85 enrolled) showed no difference in PTSD symptoms between those retained (M = 12.1, SD = 9.9) and those who withdrew (M = 15.3, SD = 14.2) with t (74) = 0.80, ns. Valid PIA data was available for 62 participants.

Figure 2.

Figure 2

Open in a new tab

Enrollment and retention flowchart.

Major analyses were conducted on data for subjects who completed at least 15% of assessments, for subjects who completed at least 30% of assessments, and for subjects who completed at least 50% assessments. Results of these analyses were essentially the same. Unless otherwise noted, analyses reported below are for the 54 participants (83%) completing at least 30% of assessments. This standard has been used by several investigators collecting intensive data in clinical samples [10; 30–34]. The mean number of sessions completed was 17.6 and the median number was 18.0. The median item response time was 2.5 seconds. Response times under one second were considered invalid. Comparison of response rates for different times of day showed no difference in frequencies of reporting across the four times of day (Chi-square = 2.85, n.s.).

Analyses to Investigate Psychometric Properties of PIA

Reliability: Internal consistency of PIA measures

To examine the internal consistency of the PIA measures, we used multilevel modeling to estimate the reliability adjusted for both person level and occasion level variance for several variables. Three level, unconditional models were specified using the methods described by Nezlek [35]. Reliability for the measures was .67 for PTSD, .74 for dissociation, .63 for negative affect, .70 for positive affect, .43 for negative cognitions, and .79 for emotional approach coping.

Convergent and divergent validity

To examine convergent and divergent validity of PIA reports, we conducted analyses of relationships between PIA reports and standard self-reports of the same variables administered at the end of the week of PIA data collection. Table 2 shows correlations between post-PIA standard measures of the same and different constructs and PIA ratings aggregated across all sessions. The Fisher z-corrected means of same-construct correlations and different-construct correlations were r = .77 and .26, respectively.

Table 2.

Relationships Among Scores on Aggregated PIA Ratings and Standard Measures

Standard Measures Given Post-PIA
PTSD Dissociation Neg Cogs EAC
PIA PTSD .87*** .55*** .28 .26
PIA Dissociation .64*** .76*** .09 .21
PIA Negative Cognitions .38** .17 .81*** .008
PIA Emotional Approach Coping .23 .15 -.01 .56***
Open in a new tab

Note. Same construct correlations are in boldface.

***

p < .001,

**

p < .01,

*

p < .05

Construct validity from analyses of theoretically expected relationships

Multilevel analyses were conducted to investigate whether time elapsed since the injury event predicted PIA reports of PTSD symptoms. Time was entered into the Level 1 regression model as a predictor of PTSD symptoms, and the resulting beta weight for Time was B = 0.031 (SE = .01, p < .001). The B value of 0.031 means that PTSD symptoms decreased on average by 0.031 units across each successive day of PIA data collection.

Construct validity was also investigated by conducting unconditional slope analyses to examine within-person relationships between various pairs of constructs assessed via PIA that were theoretically expected to be related to one another. Results are shown in Table 3. The B coefficients indicate the degree of relationship between the two variables within individuals. For example, the significant slope coefficient for PTSD in predicting dissociation (B = 0.78) indicates that a one unit within-person increase in PTSD is associated with a 0.78 unit increase in dissociation. Relationships within persons between PTSD symptoms and dissociation, PTSD symptoms and negative PT cognitions, and PTSD and negative affect were all significant. Feeling free to talk about the trauma was significantly associated within persons with emotional expression, emotional processing, and positive affect.

Table 3.

Within-Subject Relationships among Constructs Assessed with PIA

Predictor Variable Criterion Variable Slope B (SE)
PTSD Dissociation .78*** (.02)
PTSD Negative PT cognitions .52*** (.02)
PTSD Negative Affect 1.03*** (.03)
Feeling Free to Talk Emotion Expression .33*** (.03)
Feeling Free to Talk Emotion Processing .25*** (.03)
Feeling Free to Talk Positive Affect .25*** (.02)
Open in a new tab
***

p < .001

Interindividual and Intraindividual Variation

Visual inspection of PIA assessments of PTSD symptoms for each participant indicated considerable variability in levels of PTSD symptoms within and across participants, ranging from very low to very high. To investigate interindividual and intraindividual variability, we conducted multilevel analyses of variance in PIA scores between and within persons. The unconditional means model was used to examine variance across participants without regard to time [36]. The results of the unconditional means model analyses for five variables are shown in Table 4. The fixed factor mean score represents the average ratings across all participants for each variable (i.e., grand mean). Values for all Level 1 and Level 2 variance were statistically significant.

Table 4.

Within (Level 1) and Between (Level 2) Variance for Variables Assessed

PDA Variable Grand Mean Level 1 Variance Level 2 Variance
PTSD .47 .071*** .224***
Dissociation .32 .091*** .170***
Negative Affect .76 .126*** .381***
Positive Affect 1.36 .160*** .453***
Neg Posttraumatic Cognitions .36 .078*** .263***
Open in a new tab
***

p < .001

To examine the strength of relationships of PIA reports to more proximal and less proximal retrospective summary reports, PIA reports of PTSD, dissociation, and negative cognitions for the first two days and last two days of the assessment were correlated with standard measures of the same constructs given just before and just after the PIA assessment period. PIA data were aggregated for each participant from the first two days and last two days of PIA assessments by calculating a mean PIA score for each variable. On standard measures, data were available for 54 participants at baseline and for 44 participants just after PIA. Data were included in analyses for those completing two or more PIA reports during the two-day periods. Data for one participant on each standard measure were excluded because the scores were extreme outliers. Correlations are shown in Table 5 with the more proximal measures shown in bold.

Table 5.

Relationships Among Scores on PIA Ratings and Standard Measures

Aggregated PIA Measures
Standard Measures PTSD Dissociation Negative Cognitions
First 2 Days Last 2 Days First 2 Days Last 2 Days First 2 Days Last 2 Days
PTSD before PIA .73*** .66***
after PIA .48*** .87***
Dissociation before PIA .55*** .43***
after PIA .52*** .65***
Negative Cognitions before PIA .71*** .58***
after PIA .82*** .81***
Open in a new tab

Note: N = 44 for correlations with standard measures administered after PIA.

***

p < .001

Discussion

Overall, the results provide evidence that the PIA measures studied are reliable and valid measures and that they appear to assess meaningful between and within-subject variability. In regard to reliability, the PIA measures for PTSD, dissociation, negative affect, positive affect, and emotional approach coping showed internal consistency values that are comparable to previously reported multilevel reliabilities for daily measures of similar constructs in non-clinical populations in the U.S. and Japan [6–8]. It is important to note that while no standards have been established for multilevel reliability of intensive measures, they are likely to be lower than reliabilities for standard single occasion measures, because intensive measures almost invariably include fewer items [3]. The measure of negative cognitions appeared to be less reliable and should be investigated further.

High to very high correlations between PIA reports and standard measures of the same constructs over the same week shows very good convergent validity, while lower correlations between PIA reports and standard measures of different constructs shows good divergent validity. Correlations across PIA and standard measures for different, but related constructs were lower than those for the same construct. For example, theoretically, PTSD and dissociation are expected to be related [37]. The correlation between PTSD and dissociation assessed by different methods was moderately high (.55 and .64), but not as high as that between measures of the same constructs by different methods (.87 and .76). Very low, nonsignificant correlations were observed between constructs that are not theoretically related (e.g., negative posttraumatic cognitions and emotional approach coping). This evidence of divergent validity supports the specificity of the reports for the intended construct. In other words, it appears that the measures accurately assessed different, but closely related constructs.

In regard to other analyses related to construct validity, PIA reports within the week of assessment related to standard measure results and to other PIA reports as they should if they are indeed measuring constructs accurately over smaller units of time. The levels of association were comparable or larger than that reported in prior research on construct validity of intensive measures [7; 8]. Also, over the course of the week assessed, PIA assessments of PTSD symptoms declined as would be expected. This finding indicates that, as a group, participants showed a progressive decrease in PTSD symptoms over successive days of PIA sampling as expected based on prior research [38]. In addition, within-person relationships between PTSD symptoms and dissociation, PTSD symptoms and negative PT cognitions, and PTSD and negative affect were all significant. These findings are consistent with theory and research showing associations between PTSD symptoms and dissociation [22; 39] and PTSD symptoms and negative PT cognitions [23; 25] in groups of trauma-exposed persons.

The finding that feeling free to talk about the trauma was associated in real time with emotional expression, emotional processing, and positive affect is also consistent with prior research [26; 27] and with theory about the mechanism of action of social support [40; 41]. This latter finding suggests that PIA can provide a way to study relationships among constructs in ever smaller units of time so that the time course of related variables can be identified. For example, if feeling free to talk is related to positive affect within a four-hour time period, but the two were found to be unrelated within a one-hour time period, that would indicate that the effects of social support on mood are not immediate. In this way, the “close-up” data that PIA provides offers the opportunity to study mechanisms of change in human behavior that cannot be studied using conventional, retrospective summary measures that collapse reports over the periods of time when change is happening.

Findings of significant intra-individual variability for all of the variables studied were consistent with the possibility that this more proximal method of assessment provided meaningful information about within-person changes over time. The grand means in Table 4 show that average ratings for most variables were on the low end of the possible range of scores (0 to 3), but significant values for Level 1 and Level 2 variance indicate that participants reported significant fluctuation between samplings on the PIA measures and differed significantly from one another in their mean ratings for the variables. Additional evidence that the PIA measures reflected meaningful variation within and across persons are provided by the findings that PIA reports on days closest in time to standard measures were more strongly related than PIA reports on days that were more distant in time. The sole exception was that negative PT cognitions assessed via PIA during the first two days were correlated more strongly with a standard measure given after PIA than they were with a standard measure given just prior to PIA. This result could be because negative PT cognitions actually contribute to a worsening of the condition and a further increase in negative thinking. In other words, as hypothesized by Ehlers and Clark [23], it may be that negative thinking begets more negative thinking. On the whole, these findings provide some evidence that the PIA methodology has the sensitivity to detect meaningful variation within and across persons that is not assessed by standard measures.

Strengths of this study include a demographically diverse sample and use of intensive assessments in “real-life” at a time when participants were under considerable duress. This is a fairly rigorous test of the feasibility of PIA and suggests that PIA can provide valid measurement even in samples of individuals who are distressed, injured, and experiencing high levels of symptoms and disruptions in their normal routines. This is important because problems of incomplete recall, error in summarizing experiences, and the inability to reflect dynamics are compounded in research on individuals experiencing distress, extreme lability, or impaired self-perceptions, because these three elements impair cognitive functioning and consequently amplify inaccuracy of reports [42; 43].

There were several limitations to this study. Some participants withdrew before completing the PIA reports and some provided incomplete data. Although there was no difference between PTSD symptom levels at the time of enrollment of retained and withdrawn participants, participant withdrawal and incomplete data may indicate that this method of data collection is too demanding for some individuals and in some circumstances. Participants were recruited and completed measures upon enrollment at different times in relation to the event. While flexibility about the time of enrollment was necessary to complete the study within a reasonable period of time, the variability in time since the event may have increased variability in reports. In addition, while the findings provide evidence for the reliability and validity of the PIA measures studied, they do not provide any information about the psychometric properties of any other PIA measures. Similarly, the findings indicated that the PIA measures we studied provided information about meaningful variation within and between individuals, but such variation may not be present or measurable in all samples or situations. As for standard measures, further research is needed to determine the reliability, validity, and meaningfulness of PIA measures of other variables and in other populations.

It would also be valuable to study use of more frequent assessments and less frequent assessments and to investigate the impact of timing of the assessment on findings. Very frequent measurement may be necessary to accurately assess some variables in some contexts, while less intensive assessments may be adequate for other variables in other contexts. In addition, PIA and other types of intensive assessment such as ESM and EMA could be compared to determine if and when reporting on the entirety of experience versus momentary experiences produce results that differ. As Stone [44] notes, the meaning of momentary versus recalled or global assessments may be different, and research is needed to explicate how they differ and what degree of bias is associated with recall of different variables. Lastly, more research is needed to ascertain the degree of error associated with summary, retrospective measures of different variables compared to that associated with PIA data. In some cases, phenomena may be sufficiently stable and error sufficiently low that the costs in burden to participants and researchers would outweigh the benefits of PIA data collection. In other cases, PIA data may provide valuable new information about psychological dynamics and mechanisms and be useful in routine clinical monitoring. Taking this type of “closer look” with reliable and valid intensive measures may further our understanding of some clinical phenomena.

Acknowledgments

Funding: This study was funded by the National Institute of Mental Health (MH69876).

Contributor Information

Eve B. Carlson, National Center for PTSD and VA Palo Alto Health Care System, Dept. of Veterans Affairs

Nigel P. Field, Pacific Graduate School of Psychology, Palo Alto University

Josef I. Ruzek, National Center for PTSD and VA Palo Alto Health Care System, Dept. of Veterans Affairs

Richard A. Bryant, School of Psychology, University of New South Wales

Constance J. Dalenberg, Alliant International University

Terence M. Keane, National Center for PTSD and VA Boston Health Care System, Department of Veterans Affairs and Departments of Psychiatry and Psychology, Boston University School of Medicine

David A. Spain, Department of Surgery, Stanford University School of Medicine

References

  • 1.Schwarz N. Why researchers should think “real-time”: A cognitive rationale. In: Mehl MR, Conner TS, editors. Handbook of research methods for studying daily life. New York: Guilford Press; 2012. pp. 22–42. [Google Scholar]
  • 2.Reis HT, Gable SL. Event-sampling and other methods for studying everyday experience. In: Reis HT, Judd CM, editors. Handbook of research methods for social and personality psychology. New York: Wiley; 2000. pp. 190–222. [Google Scholar]
  • 3.Nezlek JB. Diary methods for personality and social psychology. Los Angeles: Sage; 2012. [Google Scholar]
  • 4.Myin-Germeys I, Oorschot M, Collip D, Lataster J, Delespaul P, van Os J. Experience sampling research in psychopathology: opening the black box of daily life. Psychological Medicine. 2009;39(9):1533–1547. doi: 10.1017/S0033291708004947. [DOI] [PubMed] [Google Scholar]
  • 5.Shrout PE, Lane SP. Psychometrics. In: Mehl MR, Conner TS, editors. Handbook of research methods for studying daily life. New York: Guilford; 2012. pp. 302–320. [Google Scholar]
  • 6.Kashdan TB, Nezlek JB. Whether, when, and how is spirituality related to well-being? Moving beyond single occasion questionnaires to understanding daily process. Personality and Social Psychology Bulletin. 2012;38:1523–1535. doi: 10.1177/0146167212454549. [DOI] [PubMed] [Google Scholar]
  • 7.Nezlek JB, Sorrentino RM, Yasunaga S, Otsubo Y, Allen M, Kouhara S, Shuper P. Cross-cultural differences in reactions to daily events as indicators of cross-cultural differences in self-construction and affect. Journal of Cross-Cultural Psychology. 2008;39:685–702. [Google Scholar]
  • 8.Nezlek JB, Gable SL. Depression as a moderator of relationships between positive daily events and day-to-day psychological adjustment. Personality and Social Psychology Bulletin. 2001;27:1692–1704. [Google Scholar]
  • 9.Wilhelm P, Schoebi D. Assessing mood in daily life: Structural validity, sensitivity to change, and reliability of a short-scale to measure three basic dimensions of mood. European Journal of Psychological Assessment. 2007;23:258–267. [Google Scholar]
  • 10.Kimhy D, Delespaul P, Corcoran C, Ahn H, Yale S, Malaspina D. Computerized experience sampling method (ESMc): Assessing feasibility and validity among individuals with schizophrenia. Journal of Psychiatric Research. 2006;40:221–230. doi: 10.1016/j.jpsychires.2005.09.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Granholm E, Loh C, Swendsen J. Feasibility and validity of Computerized Ecological Momentary Assessment in schizophrenia. Schizophrenia Bulletin. 2008;34:507–514. doi: 10.1093/schbul/sbm113. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Kimhy D, Delespaul P, Ahn H, Cai S, Shikhman M, Lieberman JA, Malaspina D, Sloan RP. Concurrent measurement of “real-world” stress and arousal in individuals with psychosis: Assessing the feasibility and validity of a novel methodology. Schizophrenia Bulletin. 2010;36:1131–1139. doi: 10.1093/schbul/sbp028. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Wheeler L, Reis HT. Self-recording of everyday life events: Origins, types, and uses. Journal of Personality. 1991;59:339–354. [Google Scholar]
  • 14.Csikszentmihalyi M, Larson R. Validity and reliability of the Experience-Sampling Method. Journal of Nervous and Mental Disease. 1987;175(9):526–536. doi: 10.1097/00005053-198709000-00004. [DOI] [PubMed] [Google Scholar]
  • 15.Stone AA, Shiffman S. Ecological Momentary Assessment in behavioral medicine. Annals of Behavioral Medicine. 1994;16:199–202. [Google Scholar]
  • 16.Ariely D, Carmon Z. Gestalt characteristics of experiences: The defining features of summarized events. Journal of Behavioral Decision Making. 2000;13:191–201. [Google Scholar]
  • 17.Redelmeier DA, Kahneman D. Patients’ memories of painful medical treatments: Real-time and retrospective evaluations of two minimally invasive procedures. Pain. 1996;66:3–8. doi: 10.1016/0304-3959(96)02994-6. [DOI] [PubMed] [Google Scholar]
  • 18.Redelmeier DA, Katz J, Kahneman D. Memories of colonoscopy: A randomized trial. Pain. 2003;104:187–194. doi: 10.1016/s0304-3959(03)00003-4. [DOI] [PubMed] [Google Scholar]
  • 19.Kamarck TW, Shiffman S, Muldoon MF, Sutton-Tyrrell K, Gwaltney CJ, Janicki DL, Schwartz JE. Ecological momentary assessment as a resource for social epidemiology. In: Stone AA, Shiffman S, Atienza A, Nebeling L, editors. The Science of Real-Time Data Capture: Self-Reports in Health Research. New York: Oxford University Press; 2007. pp. 268–285. [Google Scholar]
  • 20.Carlson EB. Psychometric study of a brief screen for PTSD: Assessing the impact of multiple traumatic events. Assessment. 2001;8:431–441. doi: 10.1177/107319110100800408. [DOI] [PubMed] [Google Scholar]
  • 21.Caspi Y, Carlson EB, Klein E. Validation of a screening instrument for posttraumatic stress disorder in a community sample of Bedouin men serving in the Israeli Defense Forces. Journal of Traumatic Stress. 2007;20:529–539. doi: 10.1002/jts.20237. [DOI] [PubMed] [Google Scholar]
  • 22.Carlson EB, Waelde L, Palmieri PA, Smith SR, McDade-Montez E, Macia K. Development and validation of the Dissociative Symptoms Scale. Assessment. doi: 10.1177/1073191116645904. (in press) [DOI] [PubMed] [Google Scholar]
  • 23.Ehlers A, Clark DM. A cognitive model of posttraumatic stress disorder. Behaviour Research and Therapy. 2000;38:319–345. doi: 10.1016/s0005-7967(99)00123-0. [DOI] [PubMed] [Google Scholar]
  • 24.Stanton AL, Kirk SB, Cameron CL, Danoff-Burg S. Coping through emotional approach: Scale construction and validation. Journal of Personality and Social Psychology. 2000;78:1150–1169. doi: 10.1037//0022-3514.78.6.1150. [DOI] [PubMed] [Google Scholar]
  • 25.Foa EB, Ehlers A, Clark DM, Tolin DF, Orsillo SM. The Posttraumatic Cognitions Inventory (PTCI): Development and validation. Psychological Assessment. 1999;11:303–314. [Google Scholar]
  • 26.Bolton EE, Glenn DM, Orsillo S, Roemer L, Litz BT. The relationship between self-disclosure and symptoms of posttraumatic stress disorder in peacekeepers deployed to Somalia. Journal of Traumatic Stress. 2003;16:203–210. doi: 10.1023/A:1023754820991. [DOI] [PubMed] [Google Scholar]
  • 27.Cordova MJ, Ruzek JI, Benoit M, Brunet A. Promotion of emotional disclosure following illness and injury: A brief intervention for medical patients and their families. Cognitive and Behavioral Practice. 2003;10:358–371. [Google Scholar]
  • 28.Emmons RA, McCullough ME, Tsang J-A. The assessment of gratitude. In: Lopez SJ, Snyder CR, editors. Positive psychological assessment: A handbook of models and measures. Washington, DC: American Psychological Association; 2003. pp. 327–341. [Google Scholar]
  • 29.Feldman Barrett E, Barrett DJ. Experience Sampling Program. Boston: Boston College; 1999. [Google Scholar]
  • 30.Jacobs N, Nicolson N, Derom C, Delespaul P, van Os J, Myin-Germeys I. Electronic monitoring of salivary cortisol sampling compliance in daily life. Life Sciences. 2005;76:2431–2443. doi: 10.1016/j.lfs.2004.10.045. [DOI] [PubMed] [Google Scholar]
  • 31.Wichers MC, Myin-Germeys I, Jacobs N, Peeters F, Kenis G, Derom C, Vlietinck R, Delespaul P, van Os J. Evidence that moment-to-moment variation in positive emotions buffer genetic risk for depression: A momentary assessment twin study. Acta Psychiatrica Scandinavica. 2007;115:451–457. doi: 10.1111/j.1600-0447.2006.00924.x. [DOI] [PubMed] [Google Scholar]
  • 32.Delespaul P, deVries M, Van Os J. Determinants of occurrence and recovery from hallucinations in daily life. Social Psychiatry and Psychiatric Epidemiology. 2002;37:97–104. doi: 10.1007/s001270200000. [DOI] [PubMed] [Google Scholar]
  • 33.Myin-Germeys I, van Os J, Schwartz JE, Stone AA, Delespaul PA. Emotional reactivity to daily life stress in psychosis. Archives of General Psychiatry. 2001;58(12):1137–1144. doi: 10.1001/archpsyc.58.12.1137. [DOI] [PubMed] [Google Scholar]
  • 34.Geschwind N, Peeters F, Jacobs N, Delespaul P, Derom C, Thiery E, van Os J, Wichers M. Meeting risk with resilience: High daily life reward experience preserves mental health. Acta Psychiatra Scandinavica. 2010;122:129–138. doi: 10.1111/j.1600-0447.2009.01525.x. [DOI] [PubMed] [Google Scholar]
  • 35.Nezlek JB. Mutilevel modeling for social and personality psychology. Los Angeles: Sage; 2011. [Google Scholar]
  • 36.Singer JD, Willett JB. Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence. New York: Oxford University Press; 2003. [Google Scholar]
  • 37.Dalenberg CJ, Carlson EB. Dissociation in Posttraumatic Stress Disorder Part II: How theoretical models fit the empirical evidence and recommendations for modifying the diagnostic criteria for PTSD. Psychological Trauma. 2012;4:551–559. [Google Scholar]
  • 38.Breslau N, Kessler RC, Chilcoat HD, Schultz LR, Davis GC, Andreski P. Trauma and posttraumatic stress disorder in the community: The 1996 Detroit Area Survey of Trauma. Archives of General Psychiatry. 1998;55:626–631. doi: 10.1001/archpsyc.55.7.626. [DOI] [PubMed] [Google Scholar]
  • 39.Carlson EB, Dalenberg CJ, McDade-Montez E. Dissociation in Posttraumatic Stress Disorder Part I: Definitions and review of research. Psychological Trauma. 2012;4:479–489. [Google Scholar]
  • 40.Kaniasty K. Social support and traumatic stress. PTSD Research Quarterly. 2005;16(2):1–7. [Google Scholar]
  • 41.Cordova MJ, Cunningham LLC, Carlson CR, Andrykowski MA. Social constraints, cognitive processing, and adjustment to breast cancer. Journal of Consulting & Clinical Psychology. 2001;69:706–711. [PubMed] [Google Scholar]
  • 42.Christianson S, editor. Handbook of emotion and memory: Research and theory. Hillsdale, NJ: Erlbaum; 1992. [Google Scholar]
  • 43.Myin-Germeys I. Psychiatry. In: Mehl MR, Conner TS, editors. Handbook of research methods for studying daily life. New York: Guilford Press; 2012. pp. 636–650. [Google Scholar]
  • 44.Stone AA. Thoughts on the present state of real-time data capture. In: Stone AA, Shiffman S, Atienza A, Nebeling L, editors. The science of real-time data capture: Self-reports in health research. New York: Oxford University Press; 2007. pp. 361–370. [Google Scholar]

RESOURCES