Comparing reported prescription drug misuse between ecological momentary assessment versus timeline follow-back among college students

Abstract

Background:

Accurate assessment of prescription drug misuse (PDM) is critical among young-adult college students, a particularly high-risk group for this substance behavior. No studies have compared assessments of college students’ reports of PDM obtained from their reporting in daily life (via ecological momentary assessment; EMA) to their retrospective accounts of PDM over the same period (via timeline follow-back interview; TLFB), an approach that is commonly used in substance use research.

Purpose/Objectives:

To determine day-level agreement and person-level agreement in college student reports of PDM in EMA versus TLFB methods.

Methods:

Participants were 297 college freshmen and sophomores (69% female) recruited based on misuse behavior in the past three months. PDM behaviors were captured in daily life using EMA for 28 days and TLFB administered during an in-person lab visit. Agreement was assessed at the person level (any PDM during the 28 days) and day level (PDM on a given day) using Cohen’s kappa and percent agreement.

Results:

PDM was reported more frequently using TLFB compared to EMA. Person-level agreement between the two methods was good (k=0.62, 95% CI: 0.53, 0.70), whereas day-level agreement was fair (k=0.23, 95% CI: 0.19, 0.28). Agreement in stimulant misuse reported across methods was more consistent compared to reports of other medication classes.

Conclusions:

Findings offer implications for the assessment of college student PDM data in substance use research.

Introduction

Young-adult college students are a particularly high-risk group for engaging in prescription drug misuse (PDM) and experiencing the consequences of this behavior (Schepis et al., 2021). It is important to have accurate assessment of prescription misuse in this population. Researchers have called for assessment of PDM in daily life contexts to improve the evidence to inform prevention and treatment approaches (Schepis et al., 2020). Emerging research suggests feasibility and utility of collecting prescription misuse reports close to the time of occurrence in naturalistic settings (Barringer & Papp, 2020; Papp et al., 2020; Schepis et al., 2020). Comparing momentary assessment procedures with global, retrospective methods commonly used in substance use research is necessary. Accordingly, the current study compares PDM behaviors reported at the time of their occurrence using ecological momentary assessment (EMA; Stone & Shiffman, 1994) with retrospective accounts of the behavior over the same period using the timeline follow-back interview (TLFB; Sobell & Sobell, 1992).

While comparison studies that administer both daily life reporting and retrospective TLFB methods to the same participants to collect information about their behaviors over the same period have yielded high levels of agreement for alcohol, cigarettes, and marijuana (e.g., Chow et al., 2017; Phillips et al., 2014), substantial variability also has been documented. For instance, college-based alcohol consumption has been found to be higher when derived from EMA as compared to TLFB (Patterson et al., 2019), whereas other research documents higher drinking estimates on the TLFB as compared to daily life reporting (Merrill et al., 2020). Of note, convergence of substance use indices across methods is weaker when comparing use on specific days than when comparing person-level aggregates (e.g., Shiffman, 2009). Extent of agreement may also depend on the frequency of the focal behavior, as some research shows that more commonly used substances correspond to higher levels of agreement across multiple methods (Liu et al., 2019; Simpson et al., 2011).

To improve understanding of PDM assessment, we drew from a sample of 297 college students with elevated risk for the behavior (based on endorsing misuse of one or more prescription medication in the past three months during screening procedures) who completed EMA and TLFB methods. We expected to document very good agreement when examining whether participants reported engaging in any misuse (across four focal medication classes of pain relievers, tranquilizers, stimulants, and sedatives/barbiturates) using EMA and TLFB approaches. We expected agreement to be relatively lower (fair to good) when comparing misuse reported on corresponding days (compared to person-level agreement) and when examining misuse by medication classes across methods (compared to any misuse).

Methods

The current study draws from the baseline phase of a recent longitudinal study on daily behaviors and health in college life. Please see the online supplemental material for additional methodological details.

Prescription misuse behaviors

Each EMA report assessed hypothesized triggers of misuse (not included in the current study) and misuse intention by asking participants if they were about to take one or more prescription drug classes in any way a doctor did not direct them to. When misuse intention was endorsed, a brief follow-up report was sent 15 minutes later (Thrul et al., 2014) to assess whether misuse occurred. The TLFB interviews conducted in the second lab session resulted in participants’ reports of PDM that occurred during the assigned daily life reporting period. Across EMA and TLFB, dichotomized variables (0=no, 1=yes) were created to represent prescription misuse occurrence across the four medication classes and for each class. These variables were created at both the person level (i.e., misuse over the entire reporting period) and the day level (i.e., misuse on a particular day).

Analysis plan

Person-level information was paired for 297 participants across EMA and TLFB methods reflecting any misuse overall and any misuse by drug class. Day-level data were selected from the 154 participants who had reported any misuse and misuse by class at least once via either method. All participants completed TLFB interviews. Paired-day data were retained for all days in which a participant completed at least one EMA report. If no EMA reports were completed on a particular day, their data were considered missing (vs. absence of misuse). Day-level information was then paired (by corresponding day) across methods. Agreement was determined using Cohen’s kappa and percentage of agreement. Strength of agreement for kappa values was evaluated as very good (0.81–1.00), good (0.61–0.80), moderate (0.41–0.60), fair (0.21–0.40), and poor ($\le .20$) (Cohen, 1960). The analysis plan was preregistered at Open Science Framework: https://osf.io/jdh56/

Results

The average length of time between participants ending their reporting in daily life and completing their TLFB interviews was approximately 1 week (M = 6.79 days, SD = 13.08, range: 0–186 days).

Table 1 includes frequencies of PDM endorsed at the person level and the day level, by method. Overall, 154 (52%) participants reported engaging in misuse via at least one method. Day-level analyses included 3,435 paired days. By medication class, 3,027 days from 135 participants were included in analyses for stimulant misuse, 818 days from 38 participants for tranquilizer misuse, 404 days from 19 participants for pain reliever misuse, and 137 days from 6 participants for sedative misuse. More students reported misuse on the TLFB than on EMA, and TLFB captured a greater number of misuse days compared to EMA. Participants reported an average of 1.62 PDM days (SD = 2.28) during daily life reporting and 2.69 days (SD = 2.86) on the TLFB; this reflected significantly more misuse days reported via TLFB than EMA reporting, t(153) = 6.78, p = < .001.

Table 1.

Frequency of Prescription Drug Misuse Endorsed on EMA and TLFB Methods.

	EMA		TLFB
Misuse Behavior	n	%	n	%
Person Level a
Any Misuse	103	34.7	148	49.8
Prescription Class
Stimulant Misuse	89	30.0	129	43.4
Tranquilizer Misuse	16	5.4	34	11.4
Pain Reliever Misuse	9	3.0	17	5.7
Sedative Misuse	4	1.3	4	1.3
Day Level b
Any Misuse	250	7.28	415	12.1
Prescription Class
Stimulant Misuse	199	5.79	325	9.46
Tranquilizer Misuse	39	1.14	70	2.04
Pain Reliever Misuse	14	0.41	25	0.73
Sedative Misuse	4	0.12	7	0.20

Note. EMA = ecological momentary assessment. TLFB = timeline follow-back.

^aPerson-level results derived from 297 participants who completed EMA and TLFB methods.

^bDay-level results derived from 154 participants who reported any prescription misuse at least once on either EMA or TLFB method.

Kappa agreement for any PDM at the person level was good (see Table 2). Of those reporting any PDM, 63% consistently reported misuse across both methods. By medication class, kappa agreement across methods for stimulant misuse was good, whereas agreement levels for misuse of tranquilizers, pain relievers, and sedatives were in the moderate range. Greater percent agreement was found for stimulant misuse (61%) compared to tranquilizers (32%), pain relievers (37%), and sedatives (33%).

Table 2.

Kappa Values for Prescription Drug Misuse Endorsed Across EMA and TLFB Methods.

Misuse Behavior	Cohen’s κ	95% CI
Person Level a
Any Misuse	0.62	[.53, .70]
Prescription Class
Stimulant Misuse	0.63	[.54, .72]
Tranquilizer Misuse	0.44	[.26, .62]
Pain Reliever Misuse	0.52	[.28, .75]
Sedative Misuse	0.50	[.07, .92]
Day Level b
Any Misuse	0.23	[.19, .28]
Prescription Class
Stimulant Misuse	0.23	[.18, .28]
Tranquilizer Misuse	0.13	[.03, .22]
Pain Reliever Misuse	0.17	[−.00, .34]
Sedative Misuse	−0.04	[−.06, .06]

Note. EMA = ecological momentary assessment. TLFB = timeline follow-back. CI = confidence interval.

^aPerson-level results derived from 297 participants who completed EMA and TLFB methods.

^bDay-level results derived from 154 participants who reported any prescription misuse at least once on either EMA or TLFB method.

Kappa agreement for any PDM at the day level was fair (Table 2). Across methods, 564 days included reports of any PDM; 18% of the days included corresponding PDM reports on both the EMA and TLFB. Within class, day-level kappa agreement across methods for stimulant misuse was fair, whereas day-level agreement was poor for tranquilizers, pain relievers, and sedatives. Day-level percent agreement for stimulant misuse (17%) was greater than agreement found for tranquilizer (10%) and pain reliever misuse (11%); no day-level agreement was found for sedative misuse.

Exploratory multilevel analyses were undertaken to account for the rich, nested structure of the data. The additional analyses tested the effect of reporting type (i.e., EMA vs. TLFB) on prescription misuse outcomes and examined potential moderators of this association, including timing factors (i.e., reporting day, number of days between EMA and TLFB approaches) and amount of misuse reported. Results are provided in the online supplemental material.

Discussion

Overall, participants were somewhat consistent across methods when endorsing whether they had engaged in any prescription misuse behavior across their entire reporting period, yet person-level agreement was not as high as we predicted. Agreement was even less robust when assessing misuse on particular days and when examining the behavior by drug type. TLFB yielded greater estimates of misuse compared to EMA, in contrast to studies based on other substances (e.g., alcohol; Patterson et al., 2019). Some of these differences may be due to methodological aspects of our study. We collected potential triggers and intention to misuse to strengthen the temporal precision of our ability to identify momentary predictors of misuse (i.e., the central aim of the broader study; Papp, Barringer, et al., 2020; Papp, Kouros, & Curtin, 2020). Researchers prioritizing the collection of all PDM instances in daily life would benefit from adding a look-back question that collects any misuse that occurred since the participant’s previous report (viz., Schepis et al., 2020). Additionally, our use of study-owned devices was critical to providing strong confidentiality assurances but likely resulted in missed instances of the behavior. In terms of medication-based results, it is possible that agreement in stimulant misuse reporting was higher due to the greater prevalence of that behavior relative to the reported misuse of pain reliever, tranquilizer, and sedative medication classes.

Together, we expect that our agreement estimates were not as robust as predicted due to lower occurrence of prescription misuse compared to other salient substance use in college as well as our methodological decisions that likely resulted in incomplete capture of the behavior in daily life. Moving forward, incorporating look-back questions and having participants complete surveys via their own smartphones may facilitate more comprehensive reporting and, accordingly, result in higher levels of agreement when comparing across EMA and TLFB.

An additional limitation is sample homogeneity. Study participants were mostly White and were recruited from a single university, and results may not be generalizable to more culturally diverse samples or to students attending other types of institutions.

EMA provides undeniable benefits when capturing contextual correlates of misuse. The TLFB may be particularly well suited to collecting person-level indicators of prescription misuse (i.e., any occurrence of misuse of a medication class) over specified timeframes. Research aims should guide decisions to use one method over the other (or to implement both types of methods), and future prevention and intervention PDM efforts are encouraged to consider both person-level and event-level risk factors.