The Validity of Self-Reported Medication Adherence as an Outcome in Clinical Trials of Adherence-Promotion Interventions: Findings from the MACH14 Study

Abstract

In medication adherence-promotion trials, participants in the intervention arm are often cognizant of the researcher’s aim to improve adherence; this may lead to their inflating reports of their own adherence compared to control arm participants. Using data from 1,247 HIV-positive participants across eight U.S. Studies in the Multisite Adherence Collaboration on HIV (MACH14) collaboration, we evaluated the validity of self-reported adherence by examining whether its association with two more objective outcomes [1], electronically monitored adherence and [2] viral load, varied by study arm. After adjusting for potential confounders, there was no evidence of greater overestimation of self-reported adherence among intervention arm participants, supporting its potential as a trial outcome indicator.

Introduction

Patient self-report as a method for assessing medication adherence has come under scrutiny. This is in part due to pharmacological and technological advances capable of measuring medication ingestion in a less subjective fashion. Indeed, in the case of antiretroviral therapy (ART), overestimation of self-reported adherence in comparison to electronic drug monitoring (EDM) has long been reported [1]. More recently, in pre-exposure prophylaxis trials for HIV prevention, low serum levels of detectable drug have been discordant with high self-reported adherence [2, 3], suggesting that self-reports are biased upward. This effect is usually attributed to participants’ wanting to provide “socially desirable” responses; however, three correlational studies examining the relation between social desirability (explicitly measured) and self-report adherence found no such association [1, 4, 5].

In adherence-promotion trials, participants in the intervention arm typically receive information or resources to promote greater adherence with the clear message that better adherence is desired by the researchers. This may create another threat to the validity of self-report occurs: demand characteristics [6]. Participants subject to demand characteristics know they are being evaluated with a specific goal in mind and then modify their behavior (or their reporting of the behavior) accordingly. To counteract demand characteristics, investigators often attempt to disguise the true purpose of the research. They may resort to deception in an attempt to “blind” participants to the study’s intent. However, in most behavioral research on adherence-promotion interventions, the intent of the study is difficult to mask. It is plausible that demand characteristics, compounded by the desire to be a “good” participant by providing socially desirable responses [7], may lead those in the intervention arm to overestimate their adherence to a greater degree than in the control arm (where adherence-promotion messages are generally less salient and systematic). If true, self-report of adherence might not be sufficiently rigorous for evaluating the efficacy of adherence interventions. We could locate no research that has systematically evaluated whether study arm assignment differentially affects the validity of self-reported adherence in studies to promote adherence to medications for HIV or other conditions, but it is a plausible hypothesis.

In the present study, we combined data from eight separate adherence-promotion interventions. We examined the association between self-reported adherence after the intervention phase and two other outcomes shown to be associated with self-reported adherence [1] but assumed to be less subjective or prone to demand characteristics: (a) EDM and (b) plasma HIV-1 viral load (VL). The null hypothesis was that there would be no difference in the overestimation of adherence between participants in either arm. We tested this by (a) comparing in each study arm the discrepancy between self-reported and EDM adherence and (b) assessing the moderation by arm of the association between self-reported adherence and both EDM and VL.

Methods

Data Source

Data came from the Multi-site Adherence Collaboration on HIV (MACH14), a National Institute of Mental Health-funded project combining data from 16 studies collecting EDM ART adherence data at 14 sites across the United States (U.S.) [8]. Demographic, psychosocial, and biological data from the individual studies were combined to create new variables in a pooled data set. The present study focused on eight behavioral studies conducted at seven research sites, representing 1,247 individuals, each including post-intervention measures of (a) 3-day self-reported adherence (b) EDM ART adherence, and VL. Seven studies were excluded because they did not include an intervention with one or more control or comparison arms, and one study was excluded because it did not include a 3-day self-report measure of adherence.

Participant Characteristics

The final analytic sample of 1,247 participants ranged in age from 19 to 66 years and was mostly (67 %) male. Participants were 48 % African American, 33 % White, 13 % Latino, and 6 % Asian American. With respect to education, 66 % of participants possessed a high school or general education degree, and 13 % had at least some postsecondary education. Sixty-one percent were heterosexual and 39 % were gay or bisexual. The majority of participants (86 %) were starting an ART regimen for the first time at baseline. With respect to ART regimens, 90 % of participants were prescribed a regimen that included a nucleoside or nucleotide reverse transcriptase inhibitor, 56 % included a protease inhibitor, 41 % included a non-nucleoside reverse transcriptase inhibitor, and <1 % included a fusion or integrase inhibitor.

Measures

Demographic characteristics

Demographic variables included as covariates were: age; sex; race/ethnicity; education; sexual orientation (gay or bisexual versus heterosexual); education (less than high school diploma, completed high school, some college or higher); and whether ART-experienced or ART-naïve.

Self-reported ART Adherence

The analyses used participant self-report of medication adherence from each available assessment following the intervention phase of the study (when, presumably, participants would be most primed to overestimate their adherence). To assess adherence, the different studies generally used modifications of the Adult AIDS Clinical Trial Group Instrument [9], which includes, for each antiretroviral medication the participant was prescribed, separate questions assessing the number of doses missed yesterday, the day before yesterday, and three days ago. From these data, we determined the 3-day dose adherence averaged across all ART medications, calculated as the fraction of doses taken during the 3-day period over the total number of doses prescribed.

EDM ART

Adherence Every study included an EDM device, a pill bottle and cap that electronically records the date and time of each opening as a presumptive dose. Post-intervention EDM adherence was calculated for the 3-day time frames of each available self-report assessment as the number of openings (i.e., presumptive doses taken) divided by the total number of doses prescribed. For individuals with multiple post-intervention assessments, a corresponding 3-day EDM score was calculated for each available self-report assessment. For individuals with more than one medication monitored, percent EDM adherence was averaged across all of the monitored antiretroviral (ARV) medications.

HIV-1 RNA VL

Laboratory-assessed plasma HIV-1 RNA VL was evaluated as a clinical outcome using the VL measurement closest in time to each SR adherence assessment (Mean = 45 days, 95 % CI = [41, 49]). Because of differences in test sensitivity across study sites, we defined a uniform minimum quantifiable VL of 400 copies/mL to allow for a consistent clinically relevant cutoff across individuals and studies. As VL data were positively skewed, we applied a log10 transformation and utilized the log-transformed values in all statistical analyses.

Statistical Analysis

Because the data originated from multiple studies, we first conducted preliminary analyses to assess for any site-specific associations between the outcome variables and the demographic and self-report adherence variables. For each of these predictor variables, the EDM adherence and VL outcomes were regressed on site, the predictor, and the predictor by site interaction in a single multivariate model. An omnibus Wald χ² test of the predictor by site interactions evaluated whether the relation between the predictor and each outcome varied by study site.

The race/ethnicity by site interaction was significant for VL, indicating a statistically significant difference in the association between race and VL across study sites. Consequently, we controlled for the main effect of site and its interaction with race in the final model of the VL outcome. There were no other statistically significant interactions by site for any for any of the other predictors. Site effects were modeled using fixed effects because of the small number of clusters. As site was not of substantive interest in the final results, we report the average predictor effect weighted by the relative sample size of the sites included in the analysis.

In the main analyses, a multivariate generalized estimating equation (GEE) model was used to evaluate if the association of self-reported adherence with (a) EDM adherence and (b) log10 HIV VL varied between individuals randomized to active treatment versus control. We controlled for potential demographic confounders. The statistical test of moderation was the self-reported adherence by treatment arm interaction, which was modeled separately for EDM adherence and VL. Omnibus Wald χ² tests evaluated whether treatment assignment moderated the association of self-reported adherence across both outcomes. GEE is a multilevel regression technique that adjusts standard errors to account for correlated outcomes and can accommodate both repeated measures and multiple dependent variables. This permitted the association of self-reported adherence with both EDM adherence and VL to be evaluated in a single multivariate longitudinal regression model. The model was estimated with an exchangeable correlation structure, which assumes equal correlations among outcome measurements. A robust standard error correction was used to accommodate the non-normal distribution of the EDM and VL outcomes and to accommodate departures from compound symmetry.

To retain participants with missing covariate or outcome data in the moderation analyses with GEE, multiple imputation was performed using the multivariate normal method. This insured that the analytic sample was consistent across both of the outcomes in the model. Since the adherence and VL outcomes were assessed at multiple time points, intra-individual correlations in these measures were adjusted for clustering among participants [10]. All analyses were replicated across ten imputed datasets, with the final results calculated as a pooled average of the individual analyses.

Results

Descriptive Analyses

Analyses of the unimputed post-baseline data indicated overall adherence as estimated by self-report was 91.28 % (95 % CI = [90.28, 92.28]), approximately 26 % points higher than EDM adherence-66.68 % (95 % CI = [64.23, 69.14]). More than half of the participants had achieved an undetectable viral load post-baseline-55.08 % (95 % CI = [51.90, 58.27]). Average post-baseline indicators by study arm are presented in Table 1. Note that the mean difference between self-reported and EDM adherence was nearly identical for control arm (25.25 % [21.59, 28.91]) and intervention arm (26.11 % [22.83, 29.39] participants, providing little evidence of overestimation of adherence by arm in the raw, unimputed data.

Table 1.

Post-Baseline Outcome Indicators Across 8 Adherence-Promotion Trials (Mean and 95 % Confident Intervals) by Study Arm (N = 1,247)

Indicator	Control Arm	Intervention Arm
Self-reported 3-day adherence	91.01 [89.45, 92.52]	91.49 [90.16, 92.81]
Electronically monitored adherence	67.16 [63.45, 70.87]	66.30 [63.03, 69.56]
VL (% undetectable)	51.92 [41.65, 50.56]	57.50 [53.26, 61.74]

Multivariate Longitudinal Regression Analysis

Table 2 shows the results of the GEE model evaluating for moderation due to treatment assignment in the association of self-reported adherence with the EDM adherence and VL outcomes. SR adherence had a statistically significant association with both EDM adherence and VL, while study arm assignment was not significantly associated with either. The omnibus test of the moderation effect was not statistically significant (F [2, 511.3] = 0.53, p = 0.59), indicating no evidence that the degree of association between self-reported adherence and either of the two outcomes varied by intervention arm assignment. The univariate self-reported adherence by intervention interaction was also not statistically significant for EDM adherence (Beta = 0.04, SE = 0.05, p = 0.42) or VL (Beta = −0.12, SE = 0.16, p = 0.46), indicating that the non-significant omnibus finding did not obscure a significant moderation effect in either of the outcomes. Thus, there was no evidence that participants, whether assigned to active treatment or control, estimated their adherence differently with respect to either EDM adherence or VL.

Table 2.

Summary of Multivariate Generalized Estimating Equation Model Evaluating the Association of Self-Report Adherence with Electronic Drug Monitoring (EDM) Adherence and HIV-1 Viral Load by Study Arm Assignment in 8 Adherence-Promotion Trials (N = 1,247)

	EDM Adherence				Log10 HIV-1 Viral load
	Beta	SE	95 % CI	P	Beta	SE	95 % CI	P
Main Effect:Self-reported adherence	0.21	0.04	[0.13, 0.29]	< 0.001	−0.28	0.12	[−0.51, −0.05	0.02
Main Effect:Active intervention versus control arm	−0.02	0.04	[−0.11, 0.07]	0.67	0.13	0.16	[−0.19, 0.44]	0.42
Interaction:Self-reported adherence × active intervention versus control arm	0.04	0.05	[−0.05, 0.13]	0.42	−0.12	0.16	[−0.44, 0.20]	0.46

Discussion

We investigated whether participants in trials evaluating ART adherence promotion strategies who were in the intervention arm were more likely than those in the control arm to succumb to demand characteristics and inflate self-reports of adherence. Findings from this analysis of 1,247 participants across eight U.S. Studies in the MACH14 collaboration provided no evidence that they were. Specifically, multivariate regression models indicated no interaction effect for self-reported adherence and study arm assignment for either EDM adherence or VL outcomes (despite a positive association between self-report and both EDM adherence and VL). Moreover, mean overestimates of self-reported adherence compared to EDM were nearly identical in each arm.

Three other studies that explicitly measured social desirability found no relation between it and self-reported adherence [1, 4, 5]. This suggests that even if social desirability differs between participants in intervention and control arms, this difference may not affect self-reported adherence estimates. However, Nieuwkerk et al. [11] found that the association between self-reported medication adherence and VL was statistically significant for Dutch HIV patients indicating low social desirability but not for patients indicating high social desirability. This would mean that if social desirability does differ between arms, it could have an effect on the accuracy of self-reported adherence estimates. Future work will need to explicitly measure social desirability in each study arm to confirm this. It could also be important for those evaluating adherence interventions to know if social desirability plays a greater role for specific populations or in certain contexts.

Limitations of our study included our exclusive use of a 3-day adherence measure (results may vary with other measures of self-reported adherence, particularly those with a longer time frame); high levels of self-reported adherence with little variance (that may produce ceiling effects); linear models (actual associations may be nonlinear); the variation in time between self-reported adherence assessment and VL results; and the potential for pre-existing resistance precluding virologic response that were not taken into account in this analysis. However, both self-report and EDM-measured adherence were associated with viral load, suggesting our models were able to detect important sources of variability when they were present. Additionally, the associations among key variables within each study arm were nearly identical, further suggesting low power was not a factor in the null findings. Finally, though we did control for potentially confounding socio-demographic factors, there may be additional variables not readily accessible in this pooled data set (e.g., dosing schedule) that obscure an actual moderation.

Limitations notwithstanding, our findings suggest self-report may constitute a valid outcome for the purposes of intervention efficacy evaluations, especially in real-world settings where more expensive assessments such as EDM are not feasible. This knowledge is relevant for work in diseases other than HIV such as diabetes, in which intervention efficacy can be evaluated with both self-report and biomedical indicators.

Note our conclusion is tempered by reports that self-reported adherence has been consistently shown to inflate adherence relative to more “objective” measures [1]. Indeed, in the present study, self-report overestimated adherence compared to EDM by an average of 26 percentage points. Thus, while we found no evidence that self-reported adherence estimates vary in accuracy according to arm, they are still likely to overestimate actual medication ingestion. If these overestimates of adherence by participants in both arms lead to ceiling effects, this could result in Type II error in the evaluation of intervention efficacy.

Ultimately, inexpensive, passive, non-intrusive adherence assessment technologies may become widely available for use in clinical trials of adherence-promotion interventions; meanwhile, self-report of adherence should not be categorically dismissed as a potential outcome indicator.