Abstract
Purpose:
To assess the accuracy of five subjective self-assessment tools (three adherence measures and two psychometric scales) and pharmacy refill data in predicting objective electronically monitored non-adherence.
Design:
Prospective cohort study.
Subjects:
Glaucoma patients (≥40 years old, ≥1 medication with poor self-reported adherence) were recruited from University of Michigan Kellogg Eye Center for a study assessing the impact of a personalized glaucoma coaching program on medication adherence.
Methods:
Participants completed an initial assessment including five self-assessment tools and a three-month period of electronic monitoring of their glaucoma medication adherence (AdhereTech, New York, NY); pharmacy refill data were obtained. Electronically monitored adherence was calculated monthly as the percent of doses taken on time. The median of these adherence rates was designated as baseline adherence. Patients with adherence ≤ 80% by electronic monitoring were considered non-adherent. Self-assessment tools were scored, and pharmacy refill data were summarized as proportion of days covered. Correlation between measures of adherence was estimated with Pearson and Spearman correlation coefficients. Receiver operating characteristic curves including estimation of area under the curve (AUC), sensitivity, specificity, and accuracy, were used to compare measures of adherence with respect to predicting electronically monitored non-adherence.
Main Outcome Measures:
Accuracy of self-reported and pharmacy refill data adherence measures in predicting electronically monitored non-adherence.
Results:
95 patients completed three-months of electronic monitoring with a median monthly adherence of 74% (±21%); 53 patients (56%) were non-adherent. Pharmacy refill data were not significantly correlated with electronically monitored medication adherence (r=0.12, p=0.2). Of all the measures, a single-item adherence question (“Over the past month, what percentage of your drops do you think you took correctly?”) had the largest correlation with median electronically monitored adherence (r=0.47, p<0.0001), largest AUC for predicting non-adherence (AUC= 0.76, [95% Confidence Interval = 0.66, 0.87]), best accuracy (71%, [61, 82]), and good sensitivity (84%, [73, 96]).
Conclusions:
The single-item question was the most accurate in predicting electronically monitored non-adherence among participants with poor self-reported adherence. In clinical practice, where alternatives are often too resource-intensive, this free single-item screening question can help identify glaucoma patients at risk of poor medication adherence with reasonable accuracy.
Keywords: Electronically monitored medication adherence, self-reported medication adherence, glaucoma, screening medication non-adherence
Precis
Asking a single question, “Over the past month, what percentage of your drops do you think you took correctly?” is a simple and useful screening tool for medication nonadherence in glaucoma patients.
Glaucoma is a leading cause of blindness in the United States (US) despite the availability of effective treatments. This discrepancy is partly due to the fact that less than 50% of patients take medications as recommended.1,2 There are many reported reasons why patients are poorly adherent to their glaucoma medications, such as limited knowledge of glaucoma and its treatment, low self-efficacy, prohibitive cost, medication side effects, and even skepticism that glaucoma will lead to vision loss.3 Regardless of the reason, poor adherence to glaucoma medication is a pervasive problem that results in worse visual outcomes.4–6
Identifying and supporting patients who are non-adherent to their glaucoma medication is an urgent priority for ophthalmologists. In a cross-sectional survey, Sleath et al found that the odds of developing severe visual field loss was more than 7 times greater in study participants whose electronically monitored medication adherence rates were less than 80% versus those who were more than 80% adherent.4 Since then, many studies have targeted their intervention toward patients who are less than 80% adherent.7 Following this convention, we have defined non-adherence in our work as being ≤80% adherent according to electronic monitoring.
Although many physicians are aware that medication non-adherence is a significant problem, they may not know how best to identify patients at risk of poor adherence. The majority of eye care providers do not systematically ask about eye drop use, and even if they ask, many patients may not want to reveal that they are not taking the medication as prescribed.8,9 Additionally, studies have demonstrated that ophthalmologists are not accurate when they try to guess which patients are non-adherent.10,11
In response, many tools have been developed to assess medication adherence. These tools include pharmacy refill data, electronic medication monitors, and subjective self-assessment questionnaires. Out of the three, it is the self-assessment tools that are the most commonly used, as they are the least expensive and simplest to implement. Correspondingly, many self-assessment tools are available for free for non-commercial use and some have been validated for glaucoma patients such as the Boland measure,12 the Glaucoma Treatment Compliance Assessment Tool (GTCAT),13 the Glaucoma Medication Self-Efficacy Scale (GMSE),14 and Muir’s 2-item adaptation of the GMSE.15
The purpose of this study was to compare the accuracy of pharmacy refill data, three free self-reported adherence measures, and two psychometric scales to the gold standard of electronically monitored medication adherence. We calculated the sensitivity, specificity, and overall accuracy of these tools in identifying patients who are ≤80% adherent to their glaucoma medications by objective electronic monitoring.
Methods:
The study protocol of the overall intervention has previously been described.16, 17 Data for this study originated from subjects recruited for participation in the Support, Educate, Empower (SEE) Program pilot study (Clinical-Trials.gov Identifier #NCT03159247), a prospective non-controlled study that examined the effect of personalized glaucoma education and coaching on glaucoma patients who were ≤80% adherent to their medications. Written informed consent was obtained and this study was approved by the University of Michigan Institutional Review Board and adhered to all of the tenets of the Declaration of Helsinki.
Participants and Sample Selection
Patients who received ophthalmic care at a single academic medical center (Michigan Medicine), were ≥ 40 years old, took ≥ 1 glaucoma medication, and had a diagnosis of any type of glaucoma, glaucoma suspect or ocular hypertension were identified. Those with severe mental illness (defined as schizophrenia, history of a major depressive episode with psychosis, or bipolar disorder), cognitive impairment, did not speak English or did not instill their own eye drops were excluded. Potential participants were screened for non-adherence over the phone using the Boland adherence measure and the Morisky Medication Adherence Scale.12,18 To pass screening, patients had to self-report poor adherence, defined as <95% adherence on the Boland measure and a score ≤6 on the Morisky scale.
Patients who met eligibility and screening criteria were invited to an initial study visit with a coordinator to give written informed consent, collect data on participant demographics and clinical information, fill out surveys on self-reported adherence, provide consent to obtain pharmacy refill records, and receive electronic medication monitors for all of their glaucoma medications. This study compares the measures of self-reported adherence obtained at the initial study visit with objective electronically monitored glaucoma medication adherence over the following three months. (eFigure 1, supplemental material available at http://www.ophthalmologyglaucoma.org)
Self-assessment adherence tools:
Participants were asked to complete two self-assessment adherence measures at their initial study visit: Boland et al’s single question12 and Muir et al’s 2-item (confidence & forgetfulness) survey.15 Participants were asked to complete one additional self-assessment adherence measure, the Adherence to Refills and Medications Scale (ARMS), at their 3 month visit.19 Responses to Boland’s question were reported in 5% increments with higher percentages indicating better adherence. Responses to Muir’s confidence question were “very confident”, “confident” and “not very confident” converted to scores 1, 2, and 3 respectively. Responses to Muir’s forgetfulness question were “yes” scored as 1 and “no” scored as 0. Each response to the ARMS items was measured on a 4 point Likert scale with 8 items assessing non-adherence to medications (score range 8 – 32) and 4 items assessing non-adherence to refilling prescription (score range 4 – 16). Higher scores indicated more non-adherent behavior. ARMS was added to our study after recruitment began, thus not all patients completed the ARMS during their initial study visit but most completed it during their 3-month visit. As such, the association of the ARMS metric measured at the 3-month visit was compared to the electronic medication monitoring data.
In addition, participants completed two psychometric scales shown to be correlated with medication adherence in glaucoma patients: GTCAT13,20 and GMSE.14 Participants were asked 25 questions out of the 27-item GTCAT Version 3. Responses were graded on a 5-point Likert scale and grouped into 8 components where higher scores represented more positive outcomes. Two questions for the ninth category “Physical & Mental Health” were excluded as we already explored the participant’s physical and mental health through other validated questionnaires.21,22 Participants were also asked to complete the ten-item GMSE survey. Responses to GMSE were “not at all confident” (score = 1), “somewhat confident” (score = 2), and “very confident” (score 3). Mean score ranging from 1 to 3 was calculated with higher scores represent better self-reported medication adherence.
Pharmacy Data:
At the initial study visit, the study coordinator obtained consent to contact the patient’s preferred pharmacy or pharmacies and collect prescription refill records for their glaucoma medications. The coordinator called the pharmacies every three months to obtain data on any refills, including date dispensed, amount filled, and days covered. These data were obtained for each glaucoma medication while the patient was enrolled in the study. From these data, the proportion of days covered during the 90-day electronic adherence monitoring period was calculated as the estimated adherence. Estimated adherence was not calculated for patients who had no refills during the three-month electronic adherence monitoring period period, as they were assumed to have a prescription filled for at least 90 days. We also assumed that a participant was fully covered in the period after they joined the study and before they filled out their first refill, as we were unable to obtain records of dispensed refills before they were enrolled in the study.
Electronically Monitored Adherence:
Medication adherence was monitored electronically (AdhereTech, New York, USA) over the three-month study period.23 The patient’s glaucoma medications were placed within separate electronic monitoring bottles that recorded a date-time stamp every time the bottle was opened. Adherence to a dose of medication was confirmed if the monitor bottle was opened within a specified time window of the previous day’s dose: 24±4 hours for medications dosed once per day, 24±2 hours for medications dosed twice per day, and 24±1.3 hours for medications dosed 3 times per day. Adherence was calculated by dividing the total number of doses of all glaucoma medication(s) taken on time by the total number of doses of all medication(s) prescribed. This proportion was converted to a percentage. Percent adherence was calculated monthly during the three-month study period, cumulative over the entire baseline period, and as a median of the individual three months of medication monitoring to investigate and mitigate the Hawthorne effect.24 Medication non-adherence was defined as having a median adherence score of ≤80% over the three-month electronic monitoring period.
After finishing three months of electronic medication monitoring, patients returned for a second study visit at which point our intervention (the SEE personalized glaucoma coaching program) began for those whose median baseline adherence had been ≤80%. Just prior to the intervention, self-reported adherence was again obtained with the Boland, Muir, and ARMS measures for all participants regardless of whether they would continue with the SEE coaching program.
Statistical Methods:
All adherence measures were stratified by study visit then summarized using descriptive statistics. Correlation between electronically monitored medication adherence and self-reported adherence was estimated using Pearson correlation coefficients and with pharmacy refill adherence measures using Spearman correlation coefficients. Accuracy of self-reported adherence and pharmacy refill adherence measures compared to median medically monitored adherence ≤ 80% was assessed with receiver operating characteristic (ROC) curves including estimation of area under the curve (AUC), sensitivity, specificity, and accuracy. AUC was tested for differences between adherence measures using a logistic regression modeling the probability of median medically monitored adherence ≤ 80%. The optimal probability cut-points for predicting median medically monitored adherence ≤ 80% were chosen by maximizing the Youden index and diagnostics are reported at these values.4 Data are visually displayed using scatter plots with varying bubble size to represent subject density, ROC curves, stepped line plots to compare sensitivity and specificity across cut-points, and forest plots to compare ROC diagnostics between measures of adherence. SAS version 9.4 (SAS Institute, Cary, NC) was used for all statistical analysis.
Results:
Of the 100 people who met eligibility and screening criteria and were enrolled, 95 completed three months of electronic adherence monitoring. 91 completed all the self-reported measures of adherence at the initial study visit, had pharmacy refill data collected, and completed electronic medication monitoring; 69 participants completed all the self-reported measures of adherence at the 3-month visit after completing three months of electronic medication monitoring. The 95 participants in the sample were 63.2 years old (range = 40 – 88; SD=10.4), 50% male, 55% White (35% Black, 8% Asian, 2% Other race), and 99% non-Hispanic. The average number of total prescription medications our participants were taking was 6.6 (range = 1 – 23, SD = 4.4). Among the 95 participants, the average adherence rates for each of the three months were 77% (SD=19), 73% (SD=22), and 73% (SD=22), and the median monthly adherence was 74% (SD=21%). The average cumulative adherence over the entire three-month study period was 74% (SD=20%) and 53 participants (56%) had median monthly adherence ≤80%. Pharmacy refill data showed an average proportion of days covered during the three-month period of 0.92 (SD=0.17). Descriptive statistics of the self-reported adherence measures and their subscales are reported in Table 1.
Table 1.
Descriptive statistics of electronically monitored adherence, pharmacy refill adherence, and self-reported adherence measures.
| Adherence Measure | N | Mean | SD | Min | Max | Median |
|---|---|---|---|---|---|---|
| Electronic Adherence | ||||||
| Month 1 | 95 | 77.4 | 18.6 | 16.7 | 100.0 | 80.0 |
| Month 2 | 95 | 72.7 | 22.3 | 11.7 | 100.0 | 77.3 |
| Month 3 | 95 | 73.0 | 22.1 | 6.7 | 100.0 | 78.7 |
| Cumulative (Months 1–3) | 95 | 74.3 | 19.6 | 12.2 | 100.0 | 76.1 |
| Median (Months 1–3) | 95 | 73.8 | 21.0 | 13.3 | 100.0 | 76.7 |
| Pharmacy Adherence | ||||||
| Proportion Days Covered | 95 | 0.92 | 0.17 | 0.32 | 1.00 | 1.00 |
| Sensitivity Proportion Days Covered* | 60 | 0.86 | 0.20 | 0.32 | 1.00 | 1.00 |
| Self-Reported Adherence | ||||||
| Initial study visit Boland | 97 | 75.98 | 23.10 | 0.00 | 100.00 | 80.00 |
| 3 Month visit Boland | 84 | 82.26 | 19.63 | 5.00 | 100.00 | 90.00 |
| Initial study visit Muir Confidence | 100 | 1.98 | 0.70 | 1.00 | 3.00 | 2.00 |
| Initial study visit Muir Forget | 100 | 0.87 | 0.34 | 0.00 | 1.00 | 1.00 |
| 3 Month visit Muir Confidence | 95 | 1.74 | 0.72 | 1.00 | 3.00 | 2.00 |
| 3 Month visit Muir Forget | 95 | 0.72 | 0.45 | 0.00 | 1.00 | 1.00 |
| Initial study visit ARMS | 53 | 16.87 | 2.56 | 13.00 | 23.00 | 16.00 |
| Initial study visit ARMS Adherence | 55 | 11.35 | 1.84 | 8.00 | 17.00 | 11.00 |
| Initial study visit ARMS Refill | 58 | 5.57 | 1.60 | 4.00 | 10.00 | 5.00 |
| 3 Month visit ARMS | 79 | 15.73 | 2.29 | 12.00 | 22.00 | 15.00 |
| 3 Month visit ARMS Adherence | 79 | 10.59 | 1.56 | 8.00 | 15.00 | 11.00 |
| 3 Month visit ARMS Refill | 80 | 5.14 | 1.39 | 4.00 | 8.00 | 4.50 |
| Initial study visit GMSE Score | 100 | 2.49 | 0.42 | 1.40 | 3.00 | 2.60 |
| Initial study visit GTCAT Barriers | 99 | 3.41 | 0.60 | 2.00 | 4.67 | 3.33 |
| Initial study visit GTCAT Cues | 99 | 3.81 | 0.85 | 1.00 | 5.00 | 4.00 |
| Initial study visit GTCAT Knowledge | 99 | 3.87 | 0.58 | 2.25 | 5.00 | 4.00 |
| Initial study visit GTCAT Relationship | 99 | 3.82 | 0.68 | 1.33 | 5.00 | 3.67 |
| Initial study visit GTCAT Self-Efficacy | 99 | 3.66 | 0.68 | 1.75 | 5.00 | 3.67 |
| Initial study visit GTCAT Adherence | 99 | 2.32 | 1.30 | 1.00 | 5.00 | 2.00 |
| Initial study visit GTCAT Severity | 99 | 2.76 | 1.22 | 1.00 | 5.00 | 3.00 |
| Initial study visit GTCAT Susceptibility | 99 | 4.17 | 1.11 | 2.00 | 5.00 | 5.00 |
SD, Standard Deviation; Min, Minimum; Max, Maximum; ARMS, Adherence to Refills and Medications Scale; GMSE, Glaucoma Medication Self-Efficacy Scale; GTCAT, Glaucoma Treatment Compliance Assessment Tool;
Sensitivity measure of proportion of days covered restricted to only those subject who had at least 1 refill in the 90-day baseline period. Sample size reflects the number of patients that completed all items of the scale or subscale, which may not match those reported in Figure 1 (subjects who completed at least one item of the scale);
mean of a dichotomous variable represents the proportion of subjects who reported they sometimes forget to take their glaucoma drops.
Correlations of electronically monitored adherence with pharmacy refill adherence and self-reported adherence were calculated for participants whose assessment data was available and summarized in Table 2. Pharmacy refill adherence was not significantly correlated with electronically monitored medication adherence (range of correlations: −0.04 to 0.18; all p>0.05; Table 2). Most self-reported measures of adherence collected at the initial study visit were significantly correlated with the median electronically monitored medication adherence, such that better electronic medication adherence was correlated to better self-reported adherence. The Boland self-reported adherence measure showed the largest magnitude correlation with median electronically monitored adherence of 0.47 (p<0.0001) at the initial study visit and 0.65 (p<0.0001) at the 3-month study visit. (Figure 1A) Correlations for other self-reported measures of adherence with median electronically monitored adherence ranged from −0.55 to −0.20 for the Muir, −0.50 to −0.15 for the 3-month ARMS score, 0.03 to 0.36 for the GTCAT, and was 0.19 for the GMSE.
Table 2.
Correlation of electronically monitored adherence with pharmacy and self-reported adherence measures.
| Electronically Monitored Adherence | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Month 1 | Month 2 | Month 3 | Cumulative | Median | |||||||
| Adherence Measure | N | Corr | P-value | Corr | P-value | Corr | P-value | Corr | P-value | Corr | P-value |
| Pharmacy Adherence | |||||||||||
| Proportion Days Covered | 91 | 0.08 | 0.4764 | 0.05 | 0.6642 | 0.18 | 0.0836 | 0.11 | 0.2988 | 0.12 | 0.2444 |
| Sensitivity Proportion Days Covered* | 60 | 0.01 | 0.9430 | −0.04 | 0.7404 | 0.17 | 0.2004 | 0.06 | 0.6448 | 0.09 | 0.4850 |
| Self-Reported Adherence | |||||||||||
| Initial study visit Boland | 91 | 0.45 | <.0001 | 0.50 | <.0001 | 0.35 | 0.0006 | 0.47 | <.0001 | 0.47 | <.0001 |
| Initial study visit Muir Confidence | 91 | −0.27 | 0.0106 | −0.26 | 0.0127 | −0.26 | 0.0134 | −0.28 | 0.0069 | −0.26 | 0.0131 |
| Initial study visit Muir Forget | 91 | −0.19 | 0.0786 | −0.23 | 0.0257 | −0.18 | 0.0947 | −0.21 | 0.0413 | −0.20 | 0.0538 |
| Initial study visit ARMS Score | 49 | −0.64 | <.0001 | −0.57 | <.0001 | −0.59 | <.0001 | −0.63 | <.0001 | −0.62 | <.0001 |
| Initial study visit ARMS Adherence | 50 | −0.39 | 0.0053 | −0.33 | 0.0174 | −0.29 | 0.0397 | −0.35 | 0.0118 | −0.35 | 0.0124 |
| Initial study visit ARMS Refill | 54 | −0.55 | <.0001 | −0.49 | 0.0002 | −0.42 | 0.0015 | −0.51 | <.0001 | −0.53 | <.0001 |
| Initial study visit GMSE Score | 91 | 0.17 | 0.1046 | 0.19 | 0.0675 | 0.14 | 0.2010 | 0.18 | 0.0901 | 0.19 | 0.0671 |
| Initial study visit GTCAT Barriers | 91 | 0.28 | 0.0068 | 0.34 | 0.0009 | 0.29 | 0.0046 | 0.33 | 0.0014 | 0.36 | 0.0005 |
| Initial study visit GTCAT Cues | 91 | 0.12 | 0.2614 | 0.16 | 0.1257 | 0.11 | 0.2925 | 0.14 | 0.1813 | 0.16 | 0.1357 |
| Initial study visit GTCAT Knowledge | 91 | 0.09 | 0.3801 | 0.18 | 0.0897 | 0.23 | 0.0309 | 0.18 | 0.0824 | 0.21 | 0.0442 |
| Initial study visit GTCAT Relationship | 91 | 0.08 | 0.4699 | 0.18 | 0.0916 | 0.16 | 0.1270 | 0.15 | 0.1482 | 0.15 | 0.1476 |
| Initial study visit GTCAT Self-Efficacy | 91 | 0.07 | 0.4822 | 0.11 | 0.2829 | 0.14 | 0.1793 | 0.12 | 0.2549 | 0.14 | 0.1913 |
| Initial study visit GTCAT Adherence | 91 | 0.15 | 0.1489 | 0.17 | 0.1090 | 0.09 | 0.3964 | 0.15 | 0.1651 | 0.15 | 0.1658 |
| Initial study visit GTCAT Severity | 91 | −0.01 | 0.9043 | 0.04 | 0.7154 | 0.13 | 0.2101 | 0.06 | 0.5674 | 0.03 | 0.7662 |
| Initial study visit GTCAT Susceptibility | 91 | 0.09 | 0.3978 | 0.05 | 0.6235 | −0.01 | 0.9489 | 0.05 | 0.6664 | 0.03 | 0.7942 |
| 3 Month visit Boland | 69 | 0.59 | <.0001 | 0.67 | <.0001 | 0.53 | <.0001 | 0.63 | <.0001 | 0.65 | <.0001 |
| 3 Month visit Muir Confidence | 69 | −0.48 | <.0001 | −0.58 | <.0001 | −0.44 | 0.0002 | −0.53 | <.0001 | −0.55 | <.0001 |
| 3 Month visit Muir Forget | 69 | −0.26 | 0.0280 | −0.32 | 0.0068 | −0.26 | 0.0320 | −0.30 | 0.0122 | −0.31 | 0.0105 |
| 3 Month visit ARMS | 69 | −0.45 | <.0001 | −0.37 | 0.0016 | −0.38 | 0.0012 | −0.42 | 0.0003 | −0.43 | 0.0002 |
| 3 Month visit ARMS Adherence | 69 | −0.44 | 0.0001 | −0.45 | <.0001 | −0.41 | 0.0004 | −0.46 | <.0001 | −0.50 | <.0001 |
| 3 Month visit ARMS Refill | 69 | −0.25 | 0.0418 | −0.10 | 0.4003 | −0.16 | 0.1873 | −0.18 | 0.1485 | −0.15 | 0.2341 |
Corr, Correlation; ARMS, Adherence to Refills and Medications Scale; GMSE, Glaucoma Medication Self-Efficacy Scale; GTCAT, Glaucoma Treatment Compliance Assessment Tool;
Sensitivity measure of proportion of days covered restricted to only those subject who have at least 1 refill in the 90-day baseline period
Figure 1.
Plots displaying the relationship between electronically monitored adherence and Boland self-reported adherence, including A. Scatterplot for the linear relationship between median electronically monitored adherence and Boland self-reported adherence during the initial study visit (Bubbles are proportional to the number of cases at the observed point, such that larger bubbles represent more subjects), and B. Stepped line plot showing the sensitivity and specificity of having ≤80% median electronically monitored adherence at different cut-points of Boland self-reported adherence during the initial study visit (Youden’s statistic is maximized at 85% self-reported adherence from the single-item Boland)
Table 3 shows diagnostics of pharmacy and self-assessment tools for predicting ≤80% adherence in participants who completed all self-assessment measures during the initial study visit and the ARMS at the 3-month visit (n=77, of which 38 had electronically monitored adherence ≤80%). Pharmacy refill adherence had the lowest AUC at 0.50 (95% CI, 0.40–0.61) while the Boland question had the highest at 0.76 (95% CI, 0.66–0.87). (Figure 2A) The Boland, Muir (confidence item), ARMS (total score and adherence subscale), and the GTCAT (adherence subscales) all showed prediction better than chance alone (all 95% CI for AUC excluded 0.50). The AUC for Boland was significantly better than both Muir items (p=0.046 and p=0.001), ARMS refill score (p=0.014), and 5 of the 8 GTCAT subscales (all p<0.05). The AUC for the Boland was not significantly different from the ARMS total and adherence subscale (p=0.327 and p=0.513), and the GTCAT barriers, cues, and relationship subscales (all p>0.05). Combining the single item Boland question with any other measure of self-reported adherence did not significantly increase the AUC for predicting median electronically monitored adherence ≤80% (all p>0.05, eTable 1, supplemental material available at http://www.ophthalmologyglaucoma.org).
Table 3.
Diagnostic statistics for predicting median electronically monitored adherence ≤80% from pharmacy and self-reported adherence measures. Common sample size for subjects who completed the initial study visit self-reported adherence measures and the 3-month visit ARMS measure (n=77).
| ROC Curve Diagnostics at Cut-Point that Maximizes Youden Index | ||||||
|---|---|---|---|---|---|---|
| Variable | AUC (95% CI) | P-value* | Cut-Point | Accuracy (95% CI) | Sens (95% CI) | Spec (95% CI) |
| Pharmacy Adherence | ||||||
| Proportion Days Covered | 0.50 (0.40, 0.61) | 0.0008 | 0.76 | 55% (43, 66) | 16% (4,27) | 92% (84, 100) |
| Self-Reported Adherence | ||||||
| Initial study visit Boland | 0.76 (0.66, 0.87) | reference | 85.0 | 71% (61, 82) | 84% (73, 96) | 59% (44, 74) |
| Initial study visit Muir Confidence | 0.64 (0.52, 0.75) | 0.0457 | 2.0 | 61% (50, 72) | 87% (76, 98) | 36% (21, 51) |
| Initial study visit Muir Forget | 0.56 (0.49, 0.63) | 0.0011 | 1.0 | 56% (45, 67) | 95% (88, 100) | 18% (6, 30) |
| 3 Month visit ARMS | 0.68 (0.56, 0.80) | 0.3270 | 16.0 | 62% (52, 73) | 61% (45, 76) | 64% (49, 79) |
| 3 Month visit ARMS Adherence | 0.71 (0.60, 0.82) | 0.5132 | 10.0 | 66% (56, 77) | 95% (88, 100) | 38% (23, 54) |
| 3 Month visit ARMS Refill | 0.56 (0.44, 0.68) | 0.0142 | 5.0 | 57% (46, 68) | 58% (42, 74) | 56% (41, 72) |
| Initial study visit GMSE Score | 0.51 (0.38, 0.64) | 0.0002 | 2.2 | 53% (42, 64) | 32% (17, 46) | 74% (61, 88) |
| Initial study visit GTCAT Barriers | 0.62 (0.50, 0.75) | 0.0833 | 3.5 | 61% (50, 72) | 74% (60, 88) | 49% (33, 64) |
| Initial study visit GTCAT Cues | 0.62 (0.49, 0.74) | 0.0682 | 3.5 | 60% (49, 71) | 45% (29, 61) | 74% (61, 88) |
| Initial study visit GTCAT Knowledge | 0.60 (0.47, 0.73) | 0.0499 | 3.8 | 58% (47, 69) | 55% (39, 71) | 62% (46, 77) |
| Initial study visit GTCAT Relationship | 0.61 (0.49, 0.74) | 0.0507 | 4.0 | 64% (53, 74) | 87% (76, 98) | 41% (26, 56) |
| Initial study visit GTCAT Self-Efficacy | 0.56 (0.43, 0.69) | 0.0082 | 3.5 | 61% (50, 72) | 61% (45, 76) | 62% (46, 77) |
| Initial study visit GTCAT Adherence | 0.63 (0.51, 0.75) | 0.0434 | 1.0 | 62% (52, 73) | 42% (26, 58) | 82% (70, 94) |
| Initial study visit GTCAT Severity | 0.51 (0.37, 0.64) | 0.0019 | 2.0 | 55% (43, 66) | 79% (66, 92) | 31% (16, 45) |
| Initial study visit GTCAT Susceptibility | 0.50 (0.38, 0.62) | 0.0042 | 3.0 | 51% (39, 62) | 82% (69, 94) | 21% (8, 33) |
ROC, Receiver Operating Characteristic; AUC, Area Under the Curve; CI, Confidence Interval; Sens, Sensitivity; Spec, Specificity; ARMS, Adherence to Refills and Medications Scale; GMSE, Glaucoma Medication Self-Efficacy Scale; GTCAT, Glaucoma Treatment Compliance Assessment Tool;
test of AUC significantly different from Boland AUC.
Figure 2.
Diagnostics for predicting median electronically monitored adherence ≤80% from pharmacy adherence and self-reported adherence measures, including A. Receiver operator characteristic curves showing the area under the curve (AUC) for each adherence measures (each line represents a different measure), and B. Forest plot showing sensitivity and specificity with 95% confidence intervals. ARMS, Adherence to Refills and Medications Scale; GMSE, Glaucoma Medication Self-Efficacy Scale; GTCAT, Glaucoma Treatment Compliance Assessment Tool
A self-reported score of 85% or lower on the Boland adherence item during the initial study visit correctly classified 71% (95% CI, 61–82%) of participants as having median electronically monitored adherence ≤80% (Table 3). At this cut-point, which maximized Youden’s statistic, sensitivity of the Boland was 84% (95% CI, 73–96%) and specificity was 59% (95% CI, 44–74%; Figure 1B). Optimal cut-points for other measures of adherence showed lower overall accuracy for correctly categorizing electronically monitored adherence, where the GTCAT susceptibility adherence measure had the lowest overall accuracy of classification (51%; 95% CI, 39–62%, Table 3). Sensitivity of the adherence measures ranged from a low of 16% (for pharmacy refill adherence; 95% CI, 4–27%) to a high of 95% (for the Muir forget item and ARMs adherence subscale; both 95% CIs, 88–100%). Specificity ranged from a low of 18% (for the Muir forget item; 95% CI, 6–30%) to a high of 92% (for pharmacy refill adherence; 95% CI, 84–100%). (Table 3, Figure 2B)
Discussion:
We found that the Boland measure (“Over the past month, what percentage of your drops do you think you took correctly”) had the highest accuracy rate (71%) in predicting which participants were ≤ 80% adherent to their glaucoma medication according to objective electronic monitoring. This question maximized the balance between sensitivity (84%) and specificity (59%) with the highest AUC at 0.76. The Boland measure’s AUC was similar to reported values for digital mammograms (AUC = 0.76) and serum prostate specific antigen (AUC = 0.69) in breast and prostate cancer screening respectively.25,26
A common rule of thumb for screening tools is that sensitivity should be high, especially if the consequence of missing the disease is great.27 However, a screening tool that is highly sensitive with low specificity results in high false positive rates. Several tools we measured such as the Muir Forget Item and ARMS achieved a sensitivity rate of 95% but specificities of 18% and 38% respectively. Many participants who appear “non-adherent” according to these tools may actually be adherent to their medication. Using these tools to recruit patients into glaucoma self-management support programs would waste both patient and health system resources. Conversely, a tool that is only highly specific results in high false negative rates. For instance, pharmacy refill data had a sensitivity of 16% and specificity of 92%, meaning many patients who are non-adherent may appear adherent according to their pharmacy refill data. Using this tool would result in missed opportunities to provide additional glaucoma self-management support to those who may benefit. To ensure that a screening tool is as accurate as possible, it is important that it achieve the best balance between sensitivity and specificity. In our study, this balance was best achieved with the single Boland question.
Ideally, a screening test should be widely accessible, simple, inexpensive, safe, and reliable.27 Objective, electronically measured medication adherence is considered the gold standard in measuring adherence.1,28 However, electronic medication monitors are not yet reimbursable by medical insurance, and involve high out-of-pocket costs. For instance, the annual cost of using the electronic medication monitoring system that was used in this study was estimated to be around $1,210 per patient, making these monitors too expensive for widespread uptake.23 Pharmacy refill data has also been proposed as a way to measure adherence, but these data are not easily accessible and require clinicians to call the patient’s pharmacy or pharmacies. Given how time consuming this process is, it is logistically infeasible to incorporate pharmacy refill data into clinical practice. It also requires knowing whether a patient may have filled their prescription at a different pharmacy than their usual pharmacy – a relatively common occurrence. Given its high likelihood of error and time-intensive nature, pharmacy refill data may not be a realistic option to be used in clinical practice. Additionally, in our analysis, pharmacy data was one of the least accurate tools (55% accuracy). Though pharmacy data can be procured more definitively through pharmacy claims data, these must be purchased through Pharmacy Benefits Managers or companies at a high cost. Even then, the data may overestimate adherence if patients consistently fill their medications but do not take them regularly and on schedule.
Self-assessment medication adherence tools are free from many of these constraints. The tools we analyzed were available for free for academic use and did not require any proprietary scoring systems.12–15,19 However, even the self-assessment tools varied widely in terms of their simplicity and ease of use. For instance, participant response burden varied widely among the 5 self-assessment tools tested as the number of items asked ranged from 1 to 25. The Boland measure was the shortest as it asked only one question. It was also the only measure that did not require a scoring rubric. For a clinician who wants a simple tool that can be easily implemented as part of the check-in process, Boland’s single question on adherence was the best tool out of the five tested.
The Boland measure possesses many advantages as a screening tool, but it also has many limitations. The measure has an AUC of 0.76 as a predictor for electronic monitored non-adherence, which is considered “acceptable discrimination (0.7 – 0.8)” but below the range that is commonly considered “outstanding (>0.9)”.29 Furthermore, the Boland question had the best, but only a moderate correlation with electronic monitored adherence (r = 0.47). Thus, Boland should not replace electronic medication adherence tool as the definitive measurement tool for medication adherence. Instead, we propose to use the screening question in a similar way to how other screening tools with similar AUC values (i.e. mammograms and serum PSA levels) are used in clinical practice. Rarely are suspicious mammograms or elevated PSA levels alone definitive enough to diagnose cancer. Instead, an oncologist uses these tools to screen who needs further evaluation with more extensive testing such as a biopsy. Similarly, the Boland question, if incorporated into standard work ups, can help eyecare providers refer patients with a self-reported adherence rate of ≤ 85% on the Boland measure to more nuanced evaluation of medication adherence and self-management support. In our current health landscape, where electronic medication monitors and pharmacy claims data are too expensive or time-consuming to be feasibly incorporated into clinical practice, we believe that Boland is a useful and practical clinical tool.
There were several limitations to our study. First, the generalizability of our findings are limited by the study exclusion criteria and screening criteria as we only enrolled patients who spoke English, did not have severe mental illness or cognitive impairment, and self-reported poor adherence. Second, while Boland’s single question had the best AUC, there is no perfect self-assessment tool for medication adherence as all self-assessment tools are subject to social desirability bias, where the patient wants to please the doctor and is therefore less likely to report poor adherence. However, we believe that these shortcomings are more than compensated for by the ease of use of the Boland question, especially when we consider how resource and time consuming the alternatives such as pharmacy refill and electronic monitors are. Third, several questions asked in the original self-assessment tools were excluded in our study. We made this decision as the original study authors had already concluded that many were not associated with medication adherence. Fourth, we used electronically monitored medication adherence rates as the gold standard. There has been some debate as to whether or not a gold standard for measuring glaucoma medication adherence truly exists, as these monitors really only capture whether a patient intends to take the medication and not whether the medication is actually delivered to the eye. However, there are no current objective clinical measures that indicate long-term compliance such as assays for drug metabolites. Therefore, we chose to use the current gold standard in assessing medication adherence which are electronic monitors.30–32
In conclusion, asking all glaucoma patients whether “Over the past month, what percentage of your drops do you think you took correctly?” is an easy-to-implement screening tool in clinic to identify glaucoma patients at risk for poor medication adherence. Any patient who self-reports that they take 85% or less of their medication correctly on this question is at an increased risk of non-adherence and may benefit from additional self-management support.
Supplementary Material
Financial Support:
National Eye Institute (Bethesda, MD, K23EY025320, PANC), National Center for Advancing Translational Sciences (Bethesda, MD, TL1TR002242, JC), and Research to Prevent Blindness Career Development Award (New York, NY, PANC). The funding organizations had no role in the design or conduct of this research.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
No conflicting relationships exists for any author.
The material presented was presented in part at the American Glaucoma Society Annual Virtual Meeting on March 4–7, 2021.
This article contains additional online-only material. The following should appear online only: eFigure 1, eTable1
Supplemental material available at http://www.ophthalmologyglaucoma.org
References
- 1.Olthoff CMG, Schouten JSAG, van de Borne BW, Webers CAB. Noncompliance with Ocular Hypotensive Treatment in Patients with Glaucoma or Ocular Hypertension: An Evidence-Based Review. Ophthalmology. 2005;112(6):953–961.e957. [DOI] [PubMed] [Google Scholar]
- 2.Newman-Casey PA, Blachley T, Lee PP, Heisler M, Farris KB, Stein JD. Patterns of Glaucoma Medication Adherence over Four Years of Follow-Up. Ophthalmology. 2015;122(10):2010–2021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Newman-Casey PA, Robin AL, Blachley T, et al. The Most Common Barriers to Glaucoma Medication Adherence: A Cross-Sectional Survey. Ophthalmology. 2015;122(7):1308–1316. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Sleath B, Blalock S, Covert D, et al. The relationship between glaucoma medication adherence, eye drop technique, and visual field defect severity. Ophthalmology. 2011;118(12):2398–2402. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Rossi GCM, Pasinetti GM, Scudeller L, Radaelli R, Bianchi PE. Do Adherence Rates and Glaucomatous Visual Field Progression Correlate? Eur J Ophthalmol. 2010;21(4):410–414. [DOI] [PubMed] [Google Scholar]
- 6.Newman-Casey PA, Niziol LM, Gillespie BW, Janz NK, Lichter PR, Musch DC. The Association between Medication Adherence and Visual Field Progression in the Collaborative Initial Glaucoma Treatment Study. Ophthalmology. 2020;127(4):477–483. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Cate H, Bhattacharya D, Clark A, Fordham R, Holland R, Broadway DC. Improving adherence to glaucoma medication: a randomised controlled trial of a patient-centred intervention (The Norwich Adherence Glaucoma Study). BMC Ophthalmol. 2014;14:32. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Sleath B, Robin AL, Covert D, Byrd JE, Tudor G, Svarstad B. Patient-reported behavior and problems in using glaucoma medications. Ophthalmology. 2006;113(3):431–436. [DOI] [PubMed] [Google Scholar]
- 9.Winfield AJ, Jessiman D, Williams A, Esakowitz L. A study of the causes of non-compliance by patients prescribed eyedrops. Br J Ophthalmol. 1990;74(8):477–480. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Kass MA, Gordon M, Meltzer DW. Can ophthalmologists correctly identify patients defaulting from pilocarpine therapy? Am J Ophthalmol. 1986;101(5):524–530. [DOI] [PubMed] [Google Scholar]
- 11.Sleath B, Blalock SJ, Carpenter DM, et al. Provider Education about Glaucoma and Glaucoma Medications during Videotaped Medical Visits. J Ophthalmol. 2014;2014:238939. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Chang DS, Friedman DS, Frazier T, Plyler R, Boland MV. Development and Validation of a Predictive Model for Nonadherence with Once-Daily Glaucoma Medications. Ophthalmology. 2013;120(7):1396–1402. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Mansberger SL, Sheppler CR, McClure TM, et al. Psychometrics of a new questionnaire to assess glaucoma adherence: the Glaucoma Treatment Compliance Assessment Tool (an American Ophthalmological Society thesis). Trans Am Ophthalmol Soc. 2013;111:1–16. [PMC free article] [PubMed] [Google Scholar]
- 14.Sleath B, Blalock SJ, Stone JL, et al. Validation of a short version of the glaucoma medication self-efficacy questionnaire. Br J Ophthalmol. 2012;96(2):258. [DOI] [PubMed] [Google Scholar]
- 15.Kumar JB, Bosworth HB, Sleath B, et al. Quantifying Glaucoma Medication Adherence: The Relationship Between Self-Report, Electronic Monitoring, and Pharmacy Refill. J Ocul Pharmacol Ther. 2016;32(6):346–354. [DOI] [PubMed] [Google Scholar]
- 16.Newman-Casey PA, Niziol LM, Mackenzie CK, et al. Personalized behavior change program for glaucoma patients with poor adherence: a pilot interventional cohort study with a pre-post design. Pilot and feasibility studies. 2018;4:128–128. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Newman-Casey PA, Niziol LM, Lee PP, Musch DC, Resnicow K, Heisler M. The Impact of the Support, Educate, Empower Personalized Glaucoma Coaching Pilot Study on Glaucoma Medication Adherence. Ophthalmology Glaucoma. 2020;3(4):228–237. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Morisky DE, Green LW, Levine DM. Concurrent and predictive validity of a self-reported measure of medication adherence. Med Care. 1986;24(1):67–74. [DOI] [PubMed] [Google Scholar]
- 19.Kripalani S, Risser J, Gatti ME, Jacobson TA. Development and Evaluation of the Adherence to Refills and Medications Scale (ARMS) among Low-Literacy Patients with Chronic Disease. Value Health. 2009;12(1):118–123. [DOI] [PubMed] [Google Scholar]
- 20.Sanchez FG, Mansberger SL, Newman-Casey PA. Predicting Adherence With the Glaucoma Treatment Compliance Assessment Tool. J Glaucoma. 2020;29(11):1017–1024. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Kodjebacheva G, Coleman AL, Ensrud KE, et al. Reliability and validity of abbreviated surveys derived from the National Eye Institute Visual Function Questionnaire: the study of osteoporotic fractures. Am J Ophthalmol. 2010;149(2):330–340. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606–613. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Mauro J, Mathews KB, Sredzinski ES. Effect of a Smart Pill Bottle and Pharmacist Intervention on Medication Adherence in Patients with Multiple Myeloma New to Lenalidomide Therapy. J Manag Care Spec Pharm. 2019;25(11):1244–1254. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Ajit RR, Fenerty CH, Henson DB. Patterns and rate of adherence to glaucoma therapy using an electronic dosing aid. Eye. 2010;24(8):1338–1343. [DOI] [PubMed] [Google Scholar]
- 25.Pisano ED, Gatsonis C, Hendrick E, et al. Diagnostic Performance of Digital versus Film Mammography for Breast-Cancer Screening. N Engl J Med. 2005;353(17):1773–1783. [DOI] [PubMed] [Google Scholar]
- 26.Punglia RS, D’Amico AV, Catalona WJ, Roehl KA, Kuntz KM. Effect of Verification Bias on Screening for Prostate Cancer by Measurement of Prostate-Specific Antigen. N Engl J Med. 2003;349(4):335–342. [DOI] [PubMed] [Google Scholar]
- 27.Herman CR, Gill HK, Eng J, Fajardo LL. Screening for preclinical disease: test and disease characteristics. AJR Am J Roentgenol. 2002;179(4):825–831. [DOI] [PubMed] [Google Scholar]
- 28.Newman-Casey PA, Weizer JS, Heisler M, Lee PP, Stein JD. Systematic review of educational interventions to improve glaucoma medication adherence. Semin Ophthalmol. 2013;28(3):191–201. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Hosmer DW, Lemeshow S, Sturdivant RX, Hosmer DW Jr. Applied Logistic Regression. New York, UNITED STATES: John Wiley & Sons, Incorporated; 2013. [Google Scholar]
- 30.Riekert KA, Rand CS. Electronic Monitoring of Medication Adherence: When Is High-Tech Best? J Clin Psychol Med Settings. 2002;9(1):25–34. [Google Scholar]
- 31.Blowey DL, Hébert D, Arbus GS, Pool R, Korus M, Koren G. Compliance with cyclosporine in adolescent renal transplant recipients. Pediatr Nephrol. 1997;11(5):547–551. [DOI] [PubMed] [Google Scholar]
- 32.Fallab-Stubi CL, Zellweger JP, Sauty A, Uldry C, Iorillo D, Burnier M. Electronic monitoring of adherence to treatment in the preventive chemotherapy of tuberculosis. The International Journal of Tuberculosis and Lung Disease. 1998;2(7):525–530. [PubMed] [Google Scholar]
- 33.Tsai JC, McClure CA, Ramos SE, Schlundt DG, Pichert JW. Compliance barriers in glaucoma: a systematic classification. J Glaucoma. 2003;12(5):393–398. [DOI] [PubMed] [Google Scholar]
- 34.Nordstrom BL, Friedman DS, Mozaffari E, Quigley HA, Walker AM. Persistence and adherence with topical glaucoma therapy. Am J Ophthalmol. 2005;140(4):598–606. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.