Abstract
This study examined the concordance between multiple measures of adherence, as well as sensitivity to detection of poor adherers, specificity, and predictive validity using a daily cholesterol-lowering regimen. Participants (N = 180) aged 24 to 60 years participated in an adherence ancillary study in a clinical trial. Males constituted 53.9% of this well-educated, community sample. Data on adherence were collected over a 6-month period, using electronic monitoring, self-report, specific recall, and pill counts. Electronically monitored (odds ratio [OR] = 5.348) and Shea self-report (OR = 2.678) predicted cholesterol lowering. Days (78.9%) and intervals (84.2%) adherent and the Shea (73.7%) were sensitive to the detection of poor adherers. Moderate associations were found between measures of the same type. Low correlations were found otherwise. The electronic monitor was the most accurate and informative measure. The Shea self-report was the most accurate brief, global estimate of adherence. Other measures were not associated with clinical outcome or sensitive to poor adherence.
Keywords: adherence, compliance, concordance of adherence measures, hyperlipidemia adherence, cholesterol-lowering regimen adherence
According to the Centers for Disease Control and Prevention (CDC), chronic disorders are responsible for 7 of 10 deaths (70%), are diagnosed in nearly one half of the population (CDC, 2010), and cost the United States nearly US$4.130 billion yearly (The Milliken Institute, 2011). Although effective treatments have been found for risk factors and disease management, poor outcomes remain for a significant number of persons. For example, only 50% of those with hypertension have their blood pressure within desirable limits (Egan, Zhao, & Axon, 2010); just 38.5% to 44% of those with diabetes have adequate blood glucose control (Fan, Koro, Bowlin, & Fedder, 2005); and just 17% of those with hyperlipidemia have their cholesterol within normal range (Ford, Li, Pearson, Zhao, & Mokdad, 2010). One of the contributing factors to inadequate outcomes is poor adherence to treatment. Estimates are that approximately one half of persons in treatment fail to fully adhere (Haynes, Ackloo, Sahota, McDonald, & Yao, 2008). Indeed, according to the World Health Organization (Sabate, 2003), health care dollars spent on improving adherence would have greater impact than on improvements of medical treatments. Poor adherence is also problematic in research studies. Low rates of adherence to treatment or control conditions lowers study statistical power, interferes with the ability to detect adverse responses to treatment, overestimates safety, and lowers the ability to detect treatment effects. Poor adherence is particularly problematic when it occurs differentially between research groups.
Efforts to detect poor adherence rely upon a variety of measures. These measures include biological indicators, pharmacy refills, pill counts, a variety of self-report measures, and electronic monitoring. Few studies have examined the association between measures beyond an examination of correlation coefficients to examine the sensitivity and specificity of measures. Although the correlation coefficient indicates the extent of a relationship, it does not indicate the degree of concordance or the level of specificity or sensitivity of the measure. Generally, correlations between measures of adherence are modest when looking at self-report contrasted with pill count or electronic monitoring (e.g., .14–.74; Hansen et al., 2009; Haynes et al., 1980; Pearson, Simoni, Hoff, Kurth, & Martin, 2007; Velligan et al., 2007). Cook, Wade, Martin, and Perri (2005) compared three standard adherence self-report inventories and pharmacy refill records in a predominantly female, White, Appalachian population with chronic disorders. Correlations between the measures ranged from .090 to .313. Garber, Nau, Erickson, Aikens, and Lawrence (2004) conducted an analysis of 86 comparisons of self-report and non-self-report, as published in the literature. They found varying levels of agreement between measures of adherence. However, the reports examined in this review utilized varying definitions of adherence, including continuous and categorical assessments, which may make comparisons difficult to interpret. Kappa coefficients, measures of agreement which are rarely undertaken, tend to be low (e.g., .2–.4; Walsh, Mandalia, & Gazzard, 2002). There are no data to support the utilization of one measure over another. Indeed, the World Health Organization (Sabate, 2003) reports that the current state of affairs in adherence measurement is the use of multiple concurrent measures. In clinical practice, as well as in research studies, the use of multiple measures is time-consuming and costly. With the variation between reports of adherence and measures, there is also confusion about which measures or combination of measures are most accurate and useful for the clinical or research purposes to which they are being put. There is a need for studies that examine the concordance between various measures, as well as the specificity and sensitivity of those measures, and their association with clinical outcomes.
Purpose
The purpose of this study was to examine the agreement between multiple measures of adherence to a standard dose pharmacological therapy as well as to identify their sensitivity and specificity using the measure that best predicts clinical outcome and the predictive validity of each measure in terms of clinical outcome. Adherence was defined as the consistency of doses taken with doses prescribed.
Method
This investigation was part of a larger randomized controlled trial, Evaluation of Adherence Interventions in Clinical Trials (NIH-HL48992), known hereafter as the ACT study, which examined two medication induction strategies. This adherence trial was an ancillary study embedded in a gender-balanced, randomized, controlled trial, Effect of Cholesterol Lowering on Behavior (NIH-HL46328), known as the CARE study (Muldoon et al., 2000). Participants within the adherence trial were randomized into adherence treatment conditions within the treatment groups of the cholesterol-lowering trial. The ACT study was approved by the University of Pittsburgh Institutional Review Committee as a subcomponent of the CARE study with a common consent form.
Participants
Participants from the cholesterol-lowering or CARE study were recruited from the community using newspaper advertisements, brochures, and posters. Criteria included newly diagnosed or recently diagnosed hyperlipidemia (low-density lipoprotein [LDL] ≥160 mg/dl), but otherwise healthy, willingness to be placed on an active lipid-lowering agent (20 mg lovastatin) or placebo, and between the ages of 24 and 60 years. Exclusion included marked hypertriglyceridemia, hypertension, cancer, diabetes, untreated hypothyroidism, morbid obesity, hepatic or renal insufficiency, alcoholism, major psychiatric disorder, and treatment with any lipid-lowering medication. The sample for the ACT study was entered into the trial at the second visit of the CARE trial.
Overall, 209 participants were entered into the CARE study, with 194 ultimately included in the final analysis (98 active drug and 96 placebo). A total of 180 participants, 90 active drug and 90 placebo, were entered into the ACT study. The difference of 29 participants was the result of the ACT study being funded later than the CARE study. All CARE participants from the time of the ACT award and Institutional Review Board (IRB) approval were enrolled.
Block randomization was utilized to ensure equivalent groups. Within the CARE study, randomization to drug or placebo was performed controlling for age (≤45 years vs. >45 years), race (White vs. non-White), and gender (male vs. female). Randomization to the two adherence intervention arms for ACT was accomplished within the two treatment arms for CARE considering the three blocks within each group. Thus, adherence treatment conditions were nested within drug-treatment conditions (see Figure 1).
Figure 1.
ACT design
UC = usual care
T1 = adherence intervention 1
T2 = adherence intervention 2
Participants for the measurement study included all those who completed the adherence measures at baseline and at 6 months, or at the end of the study. A total of 180 participants were available for analysis for the measurement study, the total number randomized. None were lost during the study. In all, 90 participants were in the active drug group and 90 in the placebo group.
Measures
Medication adherence to the study medication was measured via self-report, pill counts, and electronic monitoring. Self-report questionnaires were conducted at baseline and at 6 months. Pill counts were completed at 2, 4, and 6 months, consistent with the parent study visit schedule. Specific recall addressing 24-hr, 7-day, 30-day, and 6-month adherence was completed at the end of the study (6 months). Electronic monitoring was initiated at baseline with the start of study medication and continued throughout the 6 months of the study. Thus, 6 months, or approximately 180 days, of daily adherence data were collected.
Electronic event monitoring (EEM)
Multiple measures of medication taking were utilized to permit an examination of the agreement and utility of various types of measures. The specific measure used was the Aprex Medication Event Monitoring System (MEMS®, AARDEX Group Ltd., Sion, Switzerland) medication cap. This monitor consisted of a medication vial cap fitted with a microprocessor, which records the date and time the cap is opened and closed. Data are provided on medication-taking events and the interval between events. The monitor permits an examination of patterns of adherence, the timing of doses, and the detection of nonadherent episodes (e.g., Cramer, Vachon, Desforges, & Sussman, 1995; Kruse, Rampmaier, Ullrich, & Weber, 1994; Riekert & Rand, 2002). Adherence over a given period of time is typically summarized using any of the three definitions: (a) the proportion of total doses taken over the observation period of interest (doses), (b) the proportion of days in which the patient was adherent (days), and (c) the proportion of total doses ingested within an appropriate interdose interval (timing). The problem of extra doses can also be detected. Participants were given their study medication in a plastic vial fitted with a MEMs cap at the baseline visit. At the subsequent visits at 2, 4, and 6 months, the bottle and cap were retrieved, and in a private office, the cap was downloaded onto a computer, where the data were available for analysis. The vial was refilled with study medication for the next 2 months, and the MEMs cap was replaced on the vial. Adherence was treated as a continuous variable: proportion of prescribed doses taken or proportion of days with accurate dosing.
Self-report measures
The self-report measures consisted of a series of questions designed for the study, which assessed the participant’s recall of their adherence over various periods of time, and three specific published adherence questionnaires. These questionnaires were the Morisky Scale, the Shea Scale, and the Haynes Item. Questionnaires were administered by the two registered nurse research assistants during a clinic visit. No incentives were offered for participation or data completion for the adherence study.
Medication Taking Assessment II was a self-report inventory designed specifically for the study. It included a series of questions eliciting continuous information regarding medication adherence. Specifically, participants were asked to recall and report the number of doses taken, number of doses missed, and number of extra medications taken over the past 24 hr, 7 days, 1 month, and 6 months. Adherence was treated as a continuous variable.
Haynes Behavioral Compliance Assessment
The Haynes Behavioral Compliance Assessment (Haynes et al., 1976) was included in the medication adherence assessment. This single-item measure was designed to provide a simple and fast assessment of adherence problems. It asked whether the participant ever had problems taking their medication. Haynes et al. (1976) demonstrated the measure’s ability to discriminate compliance level between experimental and control group outcomes. The instrument has not had psychometric testing.
Morisky Scale
The Morisky Scale was included to assess medication adherence. This is a well-used, four-item self-report adherence measure of forgetting medicine, carelessness with medicine, stopping medicine when feeling better, and stopping medicine when feeling worse. The reliability of the scale, measured through Cronbach’s alpha, was .61. A positive response to an item received a score of 1 for a possible range of scores of 0 to 4. Total scores of 1 or above reflect poor adherence. Concurrent validity was demonstrated with 42% of patients with low adherence scores having good blood pressure control and 54% of those with high adherence scores having their blood pressure controlled (Morisky, Green, & Levine, 1986). Predictive validity was demonstrated with high scores predicting adequate control at Year 2 (75% had good control) and Year 5 (47% had good control) compared with those scoring low (p < .01).
Shea Scale
The Shea Scale is a five-item self-report scale derived from the four-item Morisky Adherence Scale (Morisky et al., 1986). Shea and colleagues made modifications to wording and added a fifth question inquiring about missing medication for any reason. Responses are scored as “yes” or “no.” Each question that endorsed missed medication was scored with 1 point, for a range of scores from 0 to 5. Scores of 1 or above reflect adherence problems. Internal consistency, measured with Cronbach’s alpha, was reported as .71. Predictive validity was demonstrated by discrimination of levels of hypertensive control (Shea, Misra, Ehrlich, Field, & Francis, 1992).
Participants, at the time of a visit, completed each of the self-report measures on a desktop computer in the study office. Data were available for analysis without further entry.
Analysis
Descriptive statistics (means and standard deviations for continuous type variables or medians and ranges for non-normally distributed continuous type and ordinal variables; frequency counts and percentages for categorical variables) were computed to summarize participant characteristics, self-report and EEM adherence measures, and change in total cholesterol from baseline to the final 6-month visit (a continuous variable computed as percentage change from baseline and a dichotomized variable defined as ≤−20% [adherent] versus >−20% [nonadherent]). The total cholesterol change cut point was based on the expected cholesterol reduction expected from the standard dose of lovastatin prescribed for active treatment group participants. Focusing on the participant subsample randomly assigned to the active lipid-lowering drug, correlational and linear/binary logistic regression analyses were performed to investigate the relationship between change in total cholesterol from baseline to the 6-month visit and the various self-report and EEM adherence measures, yielding Spearman rank order correlation coefficients (rs) and estimates of sensitivity and specificity when predicting percentage change in total cholesterol >−20% (nonadherent). Spearman rank order correlation coefficients were chosen due to the non-normal distribution of the data, resistant to transformation. Receiver operator characteristic (ROC) curve analysis was also applied to estimate the optimal cut point for EEM-based adherence measures when predicting percentage change in total cholesterol >−20% (nonadherent). Last, using the total sample, contingency table analyses were performed to yield estimates of sensitivity, specificity, and agreement/concordance (via κ coefficient) of self-report measures of adherence in predicting EEM adherence measures based on the doses taken over the same periods of time from the 6-month visit (past 24 hr, 7 days, 1 month, and overall 6-month study period).
Results
Participants
A total of 180 participants were enrolled, 90 placebo and 90 active drug, in the ACT study. Nearly 54% (53.9%) were male, or just slightly more than one half. The ethnic distribution was consistent with the local population: 87.8% White, 10.6% African American, 1.2% Hispanic, and 0.6% Asian. The sample was young to middle aged, with a mean (± SD) age of 46.11 ± 8.77 years, ranging from 24 to 60. The majority were married (67.2%), employed full-time (62.2%) or part-time (13.9%), earning an average income of approximately US$35,000 annually. The average household size was 1.9 (±1.4) persons in addition to the participant, with a range of 0 to 8. The sample was well educated, with 98.1% having at least a high school education and 63.9% having some higher education, ranging from technical or associate degree to PhD or MD. The majority were covered by some form of health insurance, with just 8.3% uninsured.
As noted, 90 participants completed the study in the active drug group. In this subsample, the gender distribution was nearly equal to the full-sized sample (56.7% male; 53.9% male, respectively). The ethnic distribution was consistent with the larger study: 86.7% were White, 11% were African American, and 2% Hispanic. The average age was 46.29 ± 9.35 years. The majority were married (68.9%) with an average of 1.99 ± 1.3 others living in the household. The majority were employed either full-time (58.9%) or part-time (10%) with an average family income of approximately US$35,000. Overall, the group was well educated, with 97.8% having at least a high school education and 61% having education beyond the high school level. Eighty-one percent had private health insurance, with just 6% on medical assistance. See Table 1 for a complete description of the sample.
Table 1.
Characteristics of the Sample
| Characteristic | Full sample (N = 180) |
Active drug subsample (Re = 90) |
|---|---|---|
| Gender (male) | 53.9% | 56.7% |
| Ethnicity | ||
| White | 87.8% | 86.7% |
| African American | 10.6% | 11.1% |
| Hispanic | 1.1% | 2.2% |
| Asian | 0.6% | N/A |
| Age (years) | 46.11 ± 8.77 (24–60) | 46.29 ± 9.35 |
| Marital status | ||
| Married | 67.2% | 68.9% |
| Widowed | 2.8% | 4.4% |
| Separated | 5.0% | 5.6% |
| Divorced | 15.0% | 12.2% |
| Never married | 10.0% | 8.9% |
| Employment | ||
| Working full-time | 62.2% | 58.9% |
| Working part-time | 13.9% | 10.0% |
| Not working | 15.0% | 19.0% |
| Homemaker | 5.6% | 8.9% |
| Income | ||
| <US$25,000 | 29.1% | 32.1% |
| ≥US$25–US$49,999 | 32.0% | 27.4% |
| ≥US$50,000 | 39.0% | 40.5% |
| Number in household (not including subject) |
1.92 ± 1.44 (0–8) | 1.99 ± 1.35 |
| Education | ||
| No degree | 1.1% | 2.2% |
| High school | 35% | 36.7% |
| Associates/technical | 22.2% | 24.4% |
| BS/BA | 22.8% | 18.9% |
| MS/MA | 12.2% | 13.3% |
| Professional (PhD, MD, etc.) | 6.7% | 4.4% |
| Health insurance | ||
| Private | 85.7% | 81.4% |
| Medical assistance | 3.5% | 7.1% |
| Medicare | 2.9% | 5.9% |
| VA benefits | 2.9% | 3.5% |
| No coverage | 8.3% | 10.6% |
Note: VA = Veterans Affairs.
Average adherence for the full sample and the sample in the active group was not different. Mean and median adherence for each measure is listed in Table 2.
Table 2.
Average Adherence by Measure
| Measure | n | M ± SD | Median (%) |
|---|---|---|---|
| MEMs dose (last 3 weeks) | 181 | 75.9% ± 37 | 90.5 |
| MEMs days (last 3 weeks) | 181 | 56.4% ± 29.4 | 61.9 |
| Pill count | 174 | 90.9% ± 18.5 | 96.2 |
| Shea | 166 | 1.7 ± 1.3 | 2 |
| Morisky | 166 | 1.2 ± .96 | 1 |
| Recall—1 day | 165 | 97.8 ± 23.0 | 100 |
| Recall—1 week | 166 | 95.1 ± 29.9 | 100 |
| Recall—1 month | 166 | 94.9 ± 6.6 | 96.7 |
| Recall—6 months | 164 | 92.6 ± 10.8 | 95.0 |
Note: MEMS = medication event monitoring system.
Best Adherence Score to Predict Less Than 20% Cholesterol Lowering
The ROC curve for the electronically monitored doses adherence for 3 weeks prior to the cholesterol assessment indicated that the best level of adherence to predict total cholesterol decrease of ≥20% was 83%. At this level of adherence, sensitivity was 84.8% and specificity was 55.3%. No other measure yielded a useful curve. It should be noted that the average total cholesterol lowering for the active drug group was 18% (95% CI of 16%–21%; Muldoon et al, 2000).
Detection of Cholesterol Lowering by Adherence Measure
The ability of each adherence measure to predict total cholesterol lowering of <20% or ≥20% was examined among the 90 participants assigned to the active cholesterol-lowering medication. The electronically monitored dose adherence measure and the Shea self-report measure significantly predicted cholesterol lowering (χ2 = 9.386, p = .002; χ2 = 4.431, p = .035, respectively). The odds ratio (OR) for the electronic monitor was 5.348, with a confidence interval of 1.72 to 16.61. The OR for the Shea self-report measure was 2.68 with a confidence interval of 1.06 to 6.78. No other measure was associated with cholesterol lowering. Each of the other measures had ORs with a confidence interval that included 1. Forward stepwise and backward elimination regression supported the finding that the Shea self-report and the electronically monitored dose adherence were the most accurate measures of adherence when predicting the degree of cholesterol lowering (B = .470, SE = .207, df = 1, p = .023, Exp(B) = 1.6; B = −.018, SE = .009, df = 1, p = .037, Exp(B) = .982, respectively). See Table 3 for the association of individual measures with cholesterol lowering.
Table 3.
Association of Adherence Measures to Cholesterol Lowering in Participants Randomized to an Active Lipid-Lowering Drug
| Adherence measure | n | Chi-square value |
p value | Odds ratio | 95% confidence interval |
|---|---|---|---|---|---|
| EEM (3-week doses) | 84 | 9.386 | .002 | 5.348 | [1.721–16.615] |
| EEM (3 week days) | 84 | 0.097 | .775 | 1.179 | [0.419–3.311] |
| EEM (3-week interval) | 84 | 0.851 | .356 | 1.676 | [0.556–5.057] |
| EEM (3-week coverage) | 84 | 0.106 | .744 | 1.155 | [0.485–2.748] |
| Pill count | 84 | 0.969 | .325 | 1.852 | [0.536–6.391] |
| Shea Scale | 83 | 4.431 | .035 | 2.678 | [1.058–6.781] |
| Morisky Scale | 83 | 2.352 | .125 | 2.214 | [0.792–6.190] |
| Haynes Item | 83 | 0.420 | .517 | 1.448 | [0.472–4.444] |
Note: EEM = electronic event monitoring.
Sensitivity and specificity for each measure was examined, setting the analysis to detect lack of cholesterol lowering with poor adherence on the measure (sensitivity) and cholesterol lowering with high adherence on the measure (specificity). High sensitivity was found for the electronic monitor for days adherent (78.9%), timing adherence (84.2%), the Morisky self-report (81.6%), and the Shea self-report (73.7%). High specificity was observed for the electronic monitor for doses adherent (89.1%), the Haynes self-report (84.4%), and the pill count (89.1%). For the two measures with significant relationships with cholesterol lowering, the electronic monitor for doses adherent and the Shea, the sensitivity and specificity were 39.5% and 89.1%, respectively, for the electronic monitor and 73.7% and 48.9%, respectively, for the Shea self-report measure. See Table 4 for sensitivity and specificity of each measure.
Table 4.
Sensitivity and Specificity of Adherence Measures in Detection of Cholesterol Change
| Adherence measure | n | Sensitivity (%) | Specificity (%) |
|---|---|---|---|
| EEM (3 weeks, doses) | 84 | 39.5 | 89.1 |
| EEM (3 weeks, days) | 84 | 78.9 | 23.9 |
| EEM (3 weeks, timing) | 84 | 84.2 | 23.9 |
| EEM (3 weeks, coverage) | 84 | 57.9 | 45.7 |
| Pill count | 84 | 18.4 | 89.1 |
| Shea Scale | 83 | 73.7 | 48.9 |
| Morisky Scale | 83 | 34.2 | 80.0 |
| Haynes Item | 83 | 21.1 | 89.1 |
Note: EEM = electronic event monitoring.
Association Between Self-Reported Recalled Adherence and Electronically Monitored Adherence
Adherence recall for 1 day, 1 week, 4 weeks, and 6 months were examined for their correspondence with the electronically monitored adherence recorded at the same time periods. The chi-square value at each time period was 4.419 (p = .05) at 1 day, 2.182 (p = .18) at 1 week, 5.288 (p = .04) at 4 weeks, and 7.300 (p = .01) at 6 months. The kappa coefficients, however, were low: 0.116 for 24 hr, 0.075 for 1 week, 0.121 for 4 weeks, and 0.159 for 6 months. The sensitivity for detecting electronically monitored poor adherence was low; significant numbers of missed doses were undetected by the self-report: 88% at 24 hr, 89% at 1 week, 87% at 4 weeks, and 83% at 6 months.
Association Between Measures of Adherence
Nonparametric testing is often necessary when analyzing adherence data due to absence of normality in the distribution. Using Spearman’s rank order correlation, the association between the electronically monitored measures, doses over 6 months and over 3 weeks, was high at rs = .84 (p < .001). Six-month and 3-week data were examined to determine the relationship with cholesterol lowering. The 3-week period was the minimum time on drug to the point of maximum cholesterol lowering. Electronically monitored adherence (doses), over 6 months and 3 weeks, had low correlations with the three questionnaires, ranging from .070 (ns) to −.294 (p < .001). Correlations were also low with the pill count, ranging from .201 (p = .008) to .282 (p < .001). Correlations with the recall were low for short-term assessment, ranging from .05 (NS) to .287 (p < .001) and moderate for 1 month to 6 months, ranging from .346 (p < .001) to .460 (p < .001).
Association between the Morisky Scale and the Shea Scale was also high with rs = .809 (p < .001). Associations between the Shea as well as the Morisky and the Haynes assessments were moderate to low (rs = −.403 and −.341, p = .000, respectively). Correlations with the pill count were also low, rs = .20 to −.29 (p = .01 to <.001). The Shea, Morisky, and Haynes Scales had low to moderate correlations with the 24-hr and 7-day recalls (.167 to −.391, p = .033 to <.001) and moderate correlations with the 30-day and 6-month recalls (.302 to −.592, p < .001). The Shea Scale was the most strongly related to the longer term recalls, with rs = −.557 with 6 months and −.592 with 30 days, both at p < .001.
Association between the pill count and recall was generally low, ranging from .049 (ns) with the 24 hr assessment to .389 (p < .001) with the 30-day assessment. As noted, associations with the questionnaires and the EEM were also low. See Table 5 for correlations between adherence measures.
Table 5.
Correlations (Spearman ρ) Between Adherence Measures
| Measure | EEM (6 months, doses) |
EEM (4-weeks, doses) |
Pill count | Shea Scale |
Morisky Scale |
Haynes Item |
24-hr recall |
7-day recall |
1-month recall |
6-month recall |
|---|---|---|---|---|---|---|---|---|---|---|
| EEM (6 months, doses) | 1.00 | |||||||||
| EEM (4 weeks, doses) | .726a | 1.00 | ||||||||
| <.001 | ||||||||||
| 168 | ||||||||||
| Pill count | .282 | .100 | 1.00 | |||||||
| <.001 | .198 | |||||||||
| 174 | 168 | |||||||||
| Shea Scale | −.294 | −.236 | −.290 | 1.00 | ||||||
| <.001 | .002 | <.001 | ||||||||
| 166 | 166 | 166 | ||||||||
| Morisky | −.282 | −.265 | −.246 | .809 | 1.00 | |||||
| <.001 | .001 | .001 | <.00 | |||||||
| 166 | 166 | 166 | 166 | |||||||
| Haynes | .168 | .048 | .200 | −.403 | −.341 | 1.00 | ||||
| .031 | .537 | .010 | <.00 | <.001 | ||||||
| 165 | 165 | 165 | 165 | 165 | ||||||
| 24-hr recall | .099 | .103 | .072 | −.116 | −.118 | .145 | 1.00 | |||
| .210 | .191 | .363 | .139 | .133 | .066 | |||||
| 163 | 163 | 163 | 163 | 163 | 162 | |||||
| 7-day recall | .206 | .165 | .115 | −.248 | −.203 | .108 | .162 | |||
| .008 | .034 | .141 | .001 | .009 | .170 | .038 | ||||
| 164 | 164 | 164 | 164 | 164 | 163 | 164 | 1.00 | |||
| 1-month recall | .456 | .372 | .389 | −.592 | −.499 | .305 | .143 | .234 | 1.00 | |
| <.001 | <.001 | <.001 | <.00 | <.001 | <.001 | .068 | .002 | |||
| 164 | 164 | 164 | 164 | 164 | 163 | 164 | 165 | |||
| 6-month recall | .430 | .340 | .299 | −.557 | −.453 | .305 | .132 | .278 | .605 | 1.00 |
| <.001 | <.001 | <.001 | <.00 | <.001 | <.001 | .095 | <.00 | <.001 | ||
| 164 | 164 | 164 | 164 | 164 | 163 | 161 | 162 | 162 |
Note: EEM = electronic event monitoring.
Statistics reported in cells are (from top to bottom) the estimated Spearman rank order (ρ) correlation coefficient, the two-tailed p value, and the available sample size (n).
Discussion
This study examined the concordance or agreement between multiple measures of adherence to a standard dose pharmacological therapy, as well as their sensitivity and specificity using the measure which best predicts clinical outcome, and the predictive validity of each measure in terms of clinical outcome. This study is unique in evaluating the quality of adherence measures within a sample all of whom were prescribed the same dosage of the same medication, thus, reducing measurement error that might be attributable to differences in medication schedules or regimen complexity. The sample included well-educated adults at or below age 60, avoiding cognitive difficulties that might be associated with aging or limited education.
Overall, we found low to moderate correlations, Spearman’s rho, between the various measures of adherence and low agreement using the kappa statistic. This is consistent with the existing literature on agreement between measures (e.g., Cook et al., 2005; Hansen et al., 2009; Haynes et al., 1980; Pearson et al., 2007; Velligan et al., 2007; Walsh et al., 2002). High correlations were found only within measurement types. The 6-month and 4-week electronic event monitor data were associated at rs = .726. The Morisky and Shea general self-reports were correlated at rs = .809. The 1-month and 6-month specific recalls were correlated at rs = .605. Other correlations ranged between .048 and .592. These data suggest that each type of adherence measure (general or specific self-report, short- or long-term electronic monitoring) captures similar information, but not necessarily overlapping information with other types of measures.
The selection of a measure that would yield the most useful information would be the measure associated with clinical outcome. In clinical practice, this permits the clinician to either regulate treatment or put an adherence improving treatment in place. In clinical research, this permits the investigator to introduce adherence improving interventions with some assurance that clinical outcome will be impacted. We examined the association of each of the adherence measures with cholesterol lowering. Two measures showed statistically significant associations, the electronically monitored “doses” adherence, or the number of doses of medication taken over the 3 weeks preceding the cholesterol determination (OR = 5.348, 95% CI = [1.721–16.615]) and the Shea (OR = 2.678, 95% CI = [1.058–6.781]). No other measure approached significance. Thus, we would suggest that when more precise data are of interest, the electronic monitor would be preferable and when a more global or brief estimate of adherence is acceptable, the Shea would be the preferred instrument.
As we were able to identify those measures associated with clinical outcomes, for example, cholesterol lowering, we were also interested in the sensitivity and specificity of the instruments in detecting poor adherence with poor cholesterol lowering. Our sensitivity and specificity analysis indicated that the electronically monitored doses adherence, the pill count, and the Haynes’s single-item assessment of adherence had the highest specificity (89.1% each) followed by the Morisky (80%). That is, these measures were most likely to detect cholesterol lowering with high adherence reported by the measure. Each of these measures had low sensitivity, that is, failed to detect lack of cholesterol lowering with poor adherence reported by the measure.
Three of the measures had high sensitivity, the electronic monitoring interval adherence (84.2%), electronic monitoring days adherence (78.9%), and the Shea Scale (73.7%). Each of these measures had low specificity. For clinical or research intervention purposes, high sensitivity permits the detection of poor adherers and, thus, reasonable and efficient targets for adherence improvement efforts.
Perhaps the optimal measure to ensure high sensitivity and high specificity would be the electronic event monitor, utilizing all three definitions: doses (specificity), days, and intervals (sensitivity). The use of multiple measures has been advocated in the HIV arena, where composite measures have been shown to be more meaningful in predicting viral load than single measures alone (Liu et al., 2001). Although we did not look at composite measures, it would appear that using the electronic monitored data in several ways independently would yield the maximum information.
Of the self-report measures, the Shea performed most successfully. It correlated with cholesterol lowering and had good sensitivity in detecting poor adherence. It also correlated well with the Morisky self-report and with the recall of medication adherence over 1 month and 6 months. Although the Shea Scale does not give detail on medication taking, it would appear to be a satisfactory measure when specific detail on adherence is unnecessary or when screening for poor adherence. It was surprising that the Shea and the Morisky did not perform equally as well as the Shea used the four questions, with some wording changes, and one additional question. This may reflect the sensitivity of measures to simple wording changes.
The other self-report measures did not fare well. They were not associated with cholesterol lowering. They had poor sensitivity and specificity. Furthermore, correlations with other measures were generally low.
Many factors can account for difficulties with self-report measures. Memory offers one of the most common barriers to accurate reporting. Reporting of adherence is an example of autobiographical memory, defined as those memories one holds of information about oneself (Wang & Brockmeier, 2002). Retrospective self-report of medication adherence requires individuals to recall individual repetitive events over varying time frames. Routine, recurring events may lead to the formation of a generic memory rather than being retained as separate specific occurrences, thus compromising accurate recall of individual events (Linton, 1982; Means & Loftus, 1991). It is also possible that individuals may confuse the memory of taking a medication for a thought about taking the medication, particularly when the action is repetitive (Einstein, McDaniel, Smith, & Shaw, 1998; Johnson & Raye, 1981). These memory distortions can preclude individuals from accurately recalling when and if medication was taken and may lead to estimation or guesswork.
Other factors may also influence the accuracy of self-report. It relies on the person’s ability to recall and summarize discrete events over time and then classify those into some evaluative category related to good or poor adherence. Social desirability may also play a role, or the desire to appear favorably, although this may not be a strong factor (Wagner & Miller, 2004). There may also be an influence of recall bias or different recall for events between good and poor adherers (Raphael, 1987; Wagner & Miller, 2004).
Other investigators have suggested that patients may be able to remember their adherence in the time period near the assessment, but would be more likely to forget their behavior in the more distant past (Rand, 2000). Our data indicate that this may not be the case. As noted, significant numbers of missed doses were undetected by the self-report when compared with the electronic monitor and the rate seemed to be stable over time; 88% at 24 hr, 89% at 1 week, 87% at 4 weeks, and 83% at 6 months. Thus, we would suggest that self-report of adherence poses significant problems in the detection of poor adherence and that report does not appear to be a function of the passage of time.
Pill counts also pose an interesting problem in the assessment of adherence. Pill counts are a common method of measuring adherence in clinical trials. We found that pill counts were not associated with clinical outcome and were not sensitive to the detection of poor adherence and lack of cholesterol lowering (sensitivity of 18.4%), although specificity was high (89.1%). The pill count also had low associations with other measures of adherence. Thus, if the intent is to identify poor adherers, the pill count is not an accurate method of detection.
The accurate assessment of adherence is important for many reasons. Of particular importance is the identification of poor adherers who would derive clinical benefit from efforts to improve their regimen management. Equally as important is the identification of true poor adherence in clinical trials testing the efficacy of new interventions. An inability to accurately assess poor adherence, or good adherence, can lead to misinterpretations of treatment efficacy, safety, and adverse outcomes. The literature on adherence is full of reports of measurement problems. Few studies have systematically and comparatively examined multiple measures of adherence in healthy individuals on a standard regimen, including their relationship to clinical outcomes, their sensitivity to the detection of poor adherence, and their interrelationships. Our results suggest that in general, self-report measures and pill counts are not related to clinical outcome and are not sensitive to the detection of poor adherence. Electronic monitors, using dosing, days, and intervals of adherence, appear to be associated with cholesterol lowering, are sensitive to the detection of low adherence, and have good specificity. When electronic monitoring is not feasible, the Shea self-report provides the most accurate assessment of adherence more globally. This study was limited to a once daily cholesterol-lowering regimen, and therefore we recommend further examination of the utility of adherence measures in multiple daily-dose regimens.
Acknowledgments
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the following grants—NIH NHLBI UO1 HL48992, NIH NHLBI UO1 46328, and NIH NINR P30 NR003924.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
The CARE study was a randomized, controlled study designed to examine the psychological and neurological consequences, if any, of cholesterol lowering using a Food and Drug Administration (FDA) approved statin. The study used a standard does of 20 mg of lovastatin designed to lower serum total cholesterol by 20%, compared with placebo. Enrolled participants had a low-density lipoprotein (LDL) cholesterol level of ≥160 mg/dl and were between 24 and 60 years old. Block randomization, race, sex, and age (> or ≤45 years old), assignment to treatment or control group was used. A total of 209 participants were enrolled. Of them, 15 participants withdrew. Thus, 194 participants completed the study. For more details, see Muldoon et al., 2000.
References
- Centers for Disease Control and Prevention. Chronic diseases and health promotion. 2010 Retrieved from http://www.cdc.gov/chronicdisease/overview/index.htm.
- Cook CL, Wade WE, Martin BC, Perri W., III Concordance among three self-reported measures of medication adherence and pharmacy refill records. Journal of the American Pharmacists Association. 2005;45:151–159. doi: 10.1331/1544345053623573. [DOI] [PubMed] [Google Scholar]
- Cramer J, Vachon L, Desforges C, Sussman NM. Dose frequency and dose interval compliance with multiple antiepileptic medications during a controlled clinical trial. Epilepsia. 1995;36:1111–1117. doi: 10.1111/j.1528-1157.1995.tb00469.x. [DOI] [PubMed] [Google Scholar]
- Egan BM, Zhao Y, Axon RN. US trends in prevalence, awareness, treatment, and control of hypertension, 1988–2008. Journal of the American Medical Association. 2010;303:2043–2050. doi: 10.1001/jama.2010.650. [DOI] [PubMed] [Google Scholar]
- Einstein GO, McDaniel MA, Smith R, Shaw P. Habitual prospective memory and aging: Remembering instructions and forgetting actions. Psychological Science. 1998;9:284–288. [Google Scholar]
- Fan T, Koro CE, Bowlin SJ, Fedder DO. Treatment patterns and glycemic control changes between NHANES III (1988–1994) and NHANES 1999–2002 among US adults diagnosed with type 2 diabetes. Paper presented at the American Diabetes Association Scientific Convention; San Diego, CA. 2005. Jun, Abstract Retrieved from http://professional.diabetes.org/Abstracts_Display.aspx?TYP=1&CID=50693. [Google Scholar]
- Ford ES, Li C, Pearson WS, Zhao G, Mokdad AH. Trends in hypercholesterolemia, treatment, and control among United States adults. International Journal of Cardiology. 2010;140:226–235. doi: 10.1016/j.ijcard.2008.11.033. [DOI] [PubMed] [Google Scholar]
- Garber MC, Nau DP, Erickson SR, Aikens JE, Lawrence JB. The concordance of self-report with other measures of medication adherence: A summary of the literature. Medical Care. 2004;42:649–652. doi: 10.1097/01.mlr.0000129496.05898.02. [DOI] [PubMed] [Google Scholar]
- Hansen RA, Kim MM, Song L, Tu W, Wu J, Murray MD. Comparison of methods to assess medication adherence and classify nonadherence. Annals of Pharmacotherapy. 2009;43:413–422. doi: 10.1345/aph.1L496. [DOI] [PubMed] [Google Scholar]
- Haynes RB, Ackloo E, Sahota N, McDonald HP, Yao X. Interventions for enhancing adherence to prescribed medications. Cochran Database of Systematic Reviews. 2008;16 doi: 10.1002/14651858.CD000011.pub3. CD000011. [DOI] [PubMed] [Google Scholar]
- Haynes RB, Sackett DL, Gibson ES, Taylor DW, Hackett BC, Roberts RS, Johnson AL. Improvement of medication compliance in uncontrolled hypertension. Lancet. 1976;1:1265–1268. doi: 10.1016/s0140-6736(76)91737-2. [DOI] [PubMed] [Google Scholar]
- Haynes RB, Taylor W, Sackett DL, Gibson ES, Bernholz CD, Mukherjee J. Can simple clinical measurements detect patient noncompliance? Hypertension. 1980;2:757–764. doi: 10.1161/01.hyp.2.6.757. [DOI] [PubMed] [Google Scholar]
- Johnson M, Raye CL. Reality monitoring. Psychological Review. 1981;88:67–85. [Google Scholar]
- Kruse W, Rampmaier J, Ullrich G, Weber E. Patterns of drug compliance with medications to be taken once and twice daily assessed by continuous electronic monitoring in primary care. International Journal of Clinical Pharmacology and Therapeutics. 1994;32:452–457. [PubMed] [Google Scholar]
- Linton ME. Transformations of memory in everyday life. In: Neisser U, editor. Memory observed: Remembering natural contexts. San Francisco, CA: W. H. Freeman; 1982. pp. 77–91. [Google Scholar]
- Liu H, Golin CE, Miller LG, Hays RD, Beck CK, Sanandaji S, Wenger NS. A comparison study of multiple measures of adherence to HIV protease inhibitors. Annals of Internal Medicine. 2001;134:968–977. doi: 10.7326/0003-4819-134-10-200105150-00011. [DOI] [PubMed] [Google Scholar]
- Means BNS, Loftus EF. When personal history repeats itself: Decomposing memories for recurring events. Applied Cognitive Psychology. 1991;5:297–318. [Google Scholar]
- Morisky DE, Green LW, Levine DM. Concurrent and predictive validity of a self-reported measure of medication adherence. Medical Care. 1986;24:67–74. doi: 10.1097/00005650-198601000-00007. [DOI] [PubMed] [Google Scholar]
- Muldoon MF, Barger SD, Ryan CM, Flory JD, Lehoczky JP, Matthews KA, Manuck SB. Effects of lovastatin on cognitive function and psychological well-being. American Journal of Medicine. 2000;108:538–546. doi: 10.1016/s0002-9343(00)00353-3. [DOI] [PubMed] [Google Scholar]
- Pearson CR, Simoni JM, Hoff P, Kurth AE, Martin DP. Assessing antiretroviral adherence via electronic drug monitoring and self-report: An examination of key methodological issues. AIDS Behavior. 2007;11:161–173. doi: 10.1007/s10461-006-9133-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Rand CS. I took the medicine like you told me doctor: Self-report of adherence to medical regimens. In: Stone AA, Turkkan JS, Bachrach CA, Jobe JB, Kurtzman HS, Cain VS, editors. The science of self-report: Implications for research and practice. Mahwah, NJ: Erlbaum; 2000. pp. 257–276. [Google Scholar]
- Raphael K. Recall bias: A proposal for assessment and control. International Journal of Epidemiology. 1987;16:167–170. doi: 10.1093/ije/16.2.167. [DOI] [PubMed] [Google Scholar]
- Riekert KA, Rand CS. Electronic monitoring of medication adherence: When is high-tech best? Journal of Clinical Psychology in Medical Settings. 2002;9:25–34. [Google Scholar]
- Sabate E, editor. Adherence to long-term therapies: Evidence for action. Geneva, Switzerland: World Health Organization; 2003. Retrieved from http://www.who.int/chp/knowledge/publications/adherence_full_report.pdf. [Google Scholar]
- Shea S, Misra D, Ehrlich MH, Field L, Francis CK. Correlates of nonadherence to hypertension treatment in an inner-city minority population. American Journal of Public Health. 1992;82:1607–1612. doi: 10.2105/ajph.82.12.1607. [DOI] [PMC free article] [PubMed] [Google Scholar]
- The Milliken Institute. An unhealthy America: The economic burden of chronic disease. 2011. Retrieved from http://www.chronicdiseaseimpact.com/ebcd.taf. [Google Scholar]
- Velligan DI, Wang M, Diamond P, Glahn DC, Castillo D, Bendle S, Miller AI. Relationships among subjective and objective measures of adherence to oral antipsychotic medications. Psychiatric Services. 2007;58:1187–1192. doi: 10.1176/ps.2007.58.9.1187. [DOI] [PubMed] [Google Scholar]
- Wagner G, Miller LG. Is the influence of social desirability on patients’ self-reported adherence overrated? Journal of Acquired Immune Deficiency Syndrome. 2004;35:203–204. doi: 10.1097/00126334-200402010-00016. [DOI] [PubMed] [Google Scholar]
- Walsh JC, Mandalia S, Gazzard BG. Responses to a 1 month self-report on adherence to antiretroviral therapy are consistent with electronic data and virological treatment outcome. Acquired Immunodeficiency Syndrome. 2002;16:269–277. doi: 10.1097/00002030-200201250-00017. [DOI] [PubMed] [Google Scholar]
- Wang Q, Brockmeier J. Autobiographical remembering as cultural practice: Understanding the interplay between memory, self and culture. Culture & Psychology. 2002;8:45–64. [Google Scholar]