A proposal for quality standards for measuring medication adherence in research

Abstract

A decade after widespread recognition that adherence to medication regimens is key to antiretroviral (ARV) effectiveness, considerable controversy remains regarding a “gold standard” for adherence measurement. Each adherence measurement approach has strengths and weaknesses and each rests on specific assumptions. The range of assumptions regarding adherence measurement and the diversity with which each approach is implemented strongly suggest that the evaluation of a particular measure outside of the context in which it was used (e.g. the study’s operational protocol) may result in undeserved confidence or lack of confidence in study results. The purpose of this paper is to propose a set of best practices across commonly used measurement methods. Recommendations regarding what information should be included in published reports regarding how adherence was measured are provided to promote improvement in the quality of measurement of medication adherence in research.

INTRODUCTION

A decade after widespread recognition that adherence to medication regimens is key to antiretroviral (ARV) effectiveness (1–6), considerable controversy remains regarding a “gold standard” for adherence measurement. The accuracy of methods used to measure medication adherence in research is of particular concern when study results are used to support recommendations for evidence-based practice. Recently, the Centers for Disease Control and Prevention (CDC) initiated an efficacy review of ARV adherence intervention trials with the goal of identifying evidence-based interventions to move into clinical practice (7, 8). A particular challenge for the CDC working group was assessment of the quality of the measurement of medication adherence in published studies. In addition to variation in the rigor of adherence measurement, details regarding operationalization of the assessment methodology used in individual studies were generally not included in the published reports, making it difficult to judge the relative strength of the findings.

Therefore, the purpose of this paper is to propose a set of best practices across commonly used measurement methods. Recommendations regarding what information should be included in published reports regarding how adherence was measured are provided to promote improvement in the quality of measurement of medication adherence in research.

While biological outcomes can be used to validate adherence measures and establish adherence intervention outcomes (e.g., quantitative HIV-RNA, ARV resistance, and immunological measures), neither biological measures nor biomarkers of drug availability are discussed as adherence outcome measures in this paper. Rather, we focus on measures that are commonly used to quantify adherence and adherence-execution behaviors including self-report, electronic monitoring devices (EMD), pill-count, and pharmacy refill based measures. These methods assess adherence by asking subjects directly about their adherence behavior or provide direct monitoring of behaviors that are part of adherence execution (e.g., opening a pill bottle, obtaining a refill before running out of medication).

VALIDITY

Across all methods of measuring medication adherence, two strategies have been used to establish validity. First, many studies address concurrent validity by examining the associations among measures such as self-report, EMD, pill counts, and pharmacy refill information (9–14). Second, examination of concordance between the estimates of adherence generated by a given measure and biological outcomes, such as CD4+ cell count (15–17), quantitative HIV-RNA (18–20), or clinical status (15, 21) is used to establish predictive validity. In addition, because non adherence and the resulting sub-optimal drug levels precede the development of antiretroviral resistance and subsequent treatment failure, the results of resistance tests are increasingly included as outcome variables in studies of medication adherence (15, 22, 23).

All methods of measuring medication adherence rest on assumptions regarding the characteristics of the method itself and its relationship to adherence behavior. Table I lists the assumptions associated with the adherence measurement methods discussed in this paper. The validity of these measures in any specific instance depends upon the extent to which these assumptions hold true. Quality control is enhanced when the underlying assumptions are articulated and addressed in the study design, implementation, and report.

Table I.

Assumptions Underlying Common Measures of Adherence

Measure	Assumptions
Self-Report	Participants can reasonably answer the questions (when doses were taken, how many, or provide general estimated).
	Cognitive deficits that impact memory/recall are not present.
	Immediate negative consequences (e.g., added procedures, reprimands, so on) of reporting non-adherence are absent.
	The scale used to measure adherence is reliable and valid.
	The scale used to measure adherence is culturally sensitive, worded clearly, and subjects know how to respond to the scaling response options with little difficulty.
	Social desirability bias is minimized or it is measured concurrently.
Electronic Monitoring	Each recorded opening equals one dose of medication consumed.
	The device is activated (e.g. cap is opened) once and only once when each dose of medication is taken.
	Multiple recordings at one time point are most likely artifact or improper use of the monitoring device.
	Periods when there is no recording of device activation indicate that the patient was not taking medication during that time; as opposed to other explanations (e.g., provider-directed hold; pocket dosing; borrowing medications).
	The novelty effect of using the devices wears off in about 35 to 40 days.
Pill Counts	The number of pills prescribed minus the number returned equals the number of pills actually consumed.
	No pills have been discarded, lost, given away, sold or disposed of in any other way.
	Pill returns were accurately counted.
	The patient returned pill containers (either empty or with left over pills) at each study visit.
Pharmacy Refill	Lack of a refill equals medication not consumed during that period.
	Pharmacy refill records are accurate.
	Medications are not purchased or borrowed from another person or venue.
	No health care provider-directed treatment interruptions occurred during the refill period.

MEASURES

Self-report

Description

Self-report is the longest standing and most widely used method to assess medication adherence in both the research and practice setting. The number of permutations of self-report measures specific to ARV adherence far exceeds that of any other method. Self-report adherence measures range from highly specific inquiries concerning the number of medication doses people have taken (or missed) to global estimates of how much or how often medication was taken as prescribed, anchored by a specified period of time, with or irrespective of additional guidelines (i.e. with or without food or water) and with or irrespective of dose-time window. Similarly diverse, the methods for administering these measures include individual structured interviews, computer delivered assessments, paper and pencil measurement, Short Message Service (SMS) text prompts, voice response systems, and web-based data collection.

The most frequently used self-report measures of ARV adherence are either count-based or estimation recall. Other strategies, including prospective monitoring (diary cards) and emerging methodologies using SMS, voice response, or web-based/smart phone strategies, also are available.

Count based recall measures, such as the frequently used recall portion of the AIDS Clinical Trials Group questionnaire (24), ask subjects to report on their ARV regimen in full (medications prescribed, when they are prescribed to be taken, and how many to be taken at each dose time) typically with the visual aid of a pill chart listing all available antiretroviral agents. Subjects then populate the last 3-days (or last week) one day at a time, typically beginning with yesterday and moving to more distant days, reporting on the time and number of doses taken, the number of pills taken, or alternatively, the number of doses missed or pills missed. These data then are used to calculate a proportion or rate of adherence that reflects the total number of doses (or pills) taken relative to the total number of doses (pills) prescribed over the assessed time period.

Estimation recall strategies include queries regarding adherence that ask subjects to reflect on a specific time period and provide a global estimate of approximately how much or how often they adhered to their ART regimen. Increasing specificity can be added to these queries by inquiring about adherence within dose-time windows or with respect to food and drink. Examples of estimation measures include visual analog scales (VAS) (25–27), the Swiss Cohort Study Adherence Questionnaire (28), Case Adherence Index Questionnaire (29), or general ability ratings (30).

Summary of Evidence

Many studies have examined concurrent validity of self-report measures with other measures of medication adherence including EMD, pharmacy refill records, and pill counts. Simoni and colleagues reviewed the literature evaluating the relation between self-report measures (predominantly count based) and other measures of adherence and viral load (31). They identified a range of association between self-report and EMD data of 0.30 to 0.55. Lu and colleagues evaluated self-report and EMD data matched for time period covered by each assessment and found a significant relationship between count-based self-report (0.49 to 0.50) and estimation self-report (0.45 to 0.55). In their work, estimation of one’s ability to adhere to the prescribed regimen outperformed count-based measures in relation to EMD data (30).

There are exceptions to these supportive findings, including Kerr and colleagues’ finding of a correlation as low as 0.06 between self-report and pharmacy refill data (32) and Wilson and colleagues’ review that placed the association between self-report and EMD below 0.25 (33). Additional validation work is needed that matches EMD assessment period to the self-report recall period and also targets the same behavior (EMD is frequently conducted on one medication while self-report is typically collected in relation to the entire regimen).

Although there is substantial support for the relation between self-report and EMD data, there is also clear evidence of unique variability in each measurement strategy. Sensitivity and specificity of self-report relative to EMD range from 24% to 57% and 66% to 97%, respectively (33–35). In general, self-report is thought to produce rates of adherence that are higher than those produced by EMD (discrepancies range from 5 to 15% in time matched contrasts) (33), although this too is not without exception (36).

Predictive validity with biological outcomes for self-report measures of adherence has been established across numerous studies. Studies and meta-analytic reviews have supported a consistent association between self-reported count-based adherence estimates and viral load, with this association ranging between 0.30 and 0.60 (31) [R² .17 (37); OR 2.31(20)]. Predictive validity with biological outcomes for estimation recall was supported in work by Mannheimer (29), Walsh (38), and Deschamps (34, 38).

Validity studies suggest that estimation items can perform as well as or better than count based measures, and also may provide added benefit by allowing assessment of longer time frames with global questions(30, 34). Counts become increasing more arduous for participants to complete and arguably more prone to memory error when the time interval assessed increases. In terms of time frames for estimation items, one month has been recommended as ideal (33).

It is important to note that the context in which self-report adherence data are collected (as an interview with a subject’s clinician, an adherence-counselor, a computer, or a questionnaire) and the subject’s beliefs about how the information provided will be used (what are the positive or negative consequences of reporting adherence or non-adherence) are likely influential determinants of the ultimate accuracy of the report. While considerable research has targeted self-report strategies in terms of what to ask, in what time interval, and how to interpret responses, limited research is available that examines the context in which adherence reports are collected or patient beliefs about the impact of reporting.

Strengths/Problems

The association between self-report and other measures of adherence proxies (EMD, unannounced pill counts) as well as with HIV-RNA, is most consistently observed in regards to specificity. That is, self-reports of non adherence are more likely to be associated with other outcome measures, such as virologic failure, than are self-reports of high adherence. Therefore, the relatively small subset of a sample that is characterized as non adherent via self-report is likely to be at increased risk for the negative health outcomes associated with ART non-adherence. In contrast, the far more common report of high adherence is less conclusively associated with positive outcomes. Amongst the potential causes of inaccuracy of self-report, the primary threat comprises recall errors and self-presentation bias.

As discussed by Wilson and colleagues (33), multiple cognitive processes influence an individual’s recall of adherence behavior. Limited salience in behaviors executed routinely may cause people to recall the routine or the intention to take doses, rather than the actual behavior. While less of a threat to prospective measurement, recall drift or inaccuracy poses considerable threat to recall measures. Also critical in recall, neurocognitive compromise associated with HIV progression or other comorbid conditions may limit the ability of some subjects to store and retrieve dose taking information accurately. Thus, even the most forthcoming and engaged individuals may provide inaccurate recall data. In the presence of known memory or cognitive functioning impairments, self-report is particularly suspect, more so as the number of days assessed increases. When asking participants to provide count recall data, shorter time periods appear reasonable (last 3 days) collected repeatedly over time (e.g., every 1–3 months). For estimate recall, last month appears reasonable for most patient groups (30).

The impact of social desirability on self-report of medication adherence has been evaluated with mixed results. Although social desirability as a personality trait was not associated with adherence reporting (37), others have found that self-report was more likely to be associated with viral outcomes when social desirability was low (39). Other motivations for consciously altering self-report of adherence include respondent beliefs about the potential consequences of reporting adherence or non-adherence and the desire to manage those consequences. For example, respondents may be motivated to modify their self-report of medication adherence if aware that their report will trigger unpleasant (negative reactions, lectures, real or perceived potential loss of benefits) or pleasant (inclusion in a study, access to a resource) consequences. In addition, when self-report adherence data is collected in the context of an ongoing relationship with a health care provider or health care system, that relationship may influence the report. Thus, how a respondent believes self-report of adherence will be used, what the consequences of different reports are, and the valuation of these most likely exert a slight to extreme influence on self-report.

An important strength of self-report is that it maps very well onto standard practice and costs relatively little to implement. Further, it is the only assessment available that asks subjects directly about adherence, while other measures directly assess proxies of adherence. However, a number of factors can compromise the accuracy of self-report and should be avoided. These include non-neutral assessment approaches (e.g., poor framing of the question that pulls social desirability one way or the other) or setting up a response task that is beyond the respondent’s memory capacity (e.g., asking to report number of tablets missed over the past three months).

Self-report adherence assessment approaches vary dramatically in what is asked, by whom, and in what context. Recent literature focuses on either count based recall using the last three days as the measurement window or gross-level estimates of adherence through one or two items with response scales assessing adherence over the last month. The longstanding belief that self-report overestimates true adherence likely still is valid with newer measurement strategies, but the discrepancy is outwardly small when comparing means between self-report and EMD based data (33). Discrepancies become more remarkable when evaluating proportions of people considered “non-adherent” between different measures. However, with self-report well supported by a number of studies in its relation to EMD and viral load, it is reasonable to suspect that self-report and EMD may measure different phenomenon, each with shared and unique variability related to clinical outcomes.

Proposed Best Practices

A research protocol that uses self-report as a measurement strategy for ARV adherence should take into account the assumptions associated with self-report measurement of ARV adherence (Table I) and seek to maximize the strengths of this strategy while addressing the problems described above. In striving for best practice, the following procedures should be considered and their use reported in studies using self-report as a primary or secondary outcome.

• Use an instrument and method of administration that has demonstrated both concurrent and predictive validity. If using a new instrument, establish validity in a pilot test.

• When memory is impaired or cognitive functioning limited, use an alternative to recall, use shorter time intervals, and consider estimation (versus count based) measures to decrease participant burden and increase accuracy.

• Minimize respondents’ perceptions of either positive or negative consequences related to self-reported adherence behavior in the following ways, (presented here as though interviewer-administered, but applicable to other assessment modalities as well).

  • Interviewers should limit praise in response to reports of high adherence.

  • Interviewers should communicate the expectation that reporting of full range of adherence is of value to the research project.

  • Minimize strategies that add procedures to interviews when non-adherence is reported.

  • Blind participants to details of the main outcome measures of a trial (e.g., that the study looks at adherence self-efficacy versus adherence rates).

  • Avoid phrasing questions in a manner that suggests negative evaluation (e.g., while the ability rating item has gathered some support, we are cautious about its use of “poor” or “very poor” as descriptors one would select to describe oneself).

  • Adopt items that offer no implied judgment (e.g., on how many days did you find yourself unable, for whatever reason, to take a dose or for whatever reason chose not to take it).

  • When using count based recall, review the prescribed regimen with the participant prior to collecting recall information about doses taken or missed. Photos or sample pills can be helpful.

  • Offer a permission statement such as “Taking pills is difficult for a lot of people. It is not uncommon for people to miss doses from time to time. These items/questions ask you about doses you took and doses you missed. Please try to remember as best you can what actually happened and not what you intended to have happen or what you think that other people want you to report. By answering these questions accurately you are making a big contribution to this research.”

  • Provide as much assurance as possible that adherence reports will not be shared with clinic staff or counselors and collect data via strategies that provide the greatest anonymity. Consider using one of the following:

    • Interviewers who are not part of the clinical or counseling program.

    • Computer assisted self-interview (CASI)

    • Audio computer assisted self-interview (ACASI)

    • SMS text messaging

    • Voice response with interviewers otherwise unassociated with the research

Electronic Monitoring

Description

Electronic monitoring of adherence behavior takes many forms. The underlying premise of EMD is that it is possible to monitor how often and when participants take their medications with a microprocessor embedded in a device such as a medication cap (e.g., Medication Events Monitoring System [MEMS]™), pill box (e.g., Med-eMonitor™) or by communication in real-time through transmission over a cellular network (e.g. Wisepill™). Each time the device is opened or activated a record of that event is recorded, stored and later uploaded to a computer for analysis. Additionally, some devices not only store adherence data, but are capable of sending messages to patients to remind them to take their medications (e.g. SimPill™). Some researchers consider EMD a possible gold-standard (40–43) or reference standard (44) to which other adherence measures should be compared.

Summary of Evidence

Multiple published reviews compare EMD data to other forms of adherence measurement (31, 34, 42, 45). Studies support the notion that EMD data correlate with pill counts, pharmacy refill and self report. However, rates of EMD data tend to be consistently lower than rates obtained using other measures. For example, EMD adherence rates are often 10–25% lower than self-reported adherence rates (10, 33, 46) and are not consistently associated with refill based measurement (9).

Electronic monitoring device data have been shown to correlate with important clinical outcomes such as HIV viral load (9, 10, 37, 40) and other adherence measures (10, 47), and to be highly sensitive (13, 14). Recently, Deschamps and colleagues (34) reported the sensitivity of electronic monitoring of antiretroviral adherence (based on the total period minus the first 40 days) to be 75% and the specificity as 85.6%, the positive predictive value as 28.6% and the negative predictive value as 97.8% (AUC = 0.80; CI = 0.62–0.99).

Strengths/Problems

Electronic monitoring devices monitor the opening of a device or bottle containing medication and thus directly monitor a behavior (opening a device or bottle) that is thought to be a necessary precursor to dose consumption most of the time. To the extent that dose consumption occurs in consistent and close proximity to device or bottle openings, this kind of assessment can provide the necessary data to closely mirror adherence. The association between EMD data and viral load provides considerable support for this assumption.

The advantages of EMDs include: measurement of adherence in real time (48, 49), tracking the timing of missed doses (50), avoidance of error due to recall or memory associated with self-report (51–53), evaluating medication dose response (42) and the ability to identify patterns of adherence behavior that would be difficult to detect with other types of measures (e.g. self-report, pharmacy refill) (54, 55). Among the key advantages of EMDs is the granularity of data collected, which is unmatched by other assessment strategies and can be used to estimate rates of adherence across long periods of time and can estimate adherence within specific dose times as well as persistence.

Problems associated with EMDs include: their high cost, the possibility of malfunction, possible interference with routine adherence activities, inability to confirm ingestion of the medication (42), inconsistent use of the electronic monitoring device (56), and the need to censor data (57). Two key assumptions under girding this method are that each recorded opening equals one dose of medication consumed and that the device is activated (e.g. the cap is opened) once and only once when each dose of medication is taken (Table I). These assumptions are violated when subjects remove more than one dose per cap opening, sometimes known as “pocket dosing”, and when subjects open the device multiple times without removing medication, referred to as “curiosity checks”. Bangsberg et al (11) described a procedure for adjusting electronically monitored data based on incorporating pocket dosing and extra device opening into self-reports over a 3 day period. Fennie and colleagues (57) reported on several ways to adjust EMD data based on pocket dosing and the appearance of excessive openings; including examining individual data based on diary or self-reports and adjusting the observation periods or changing the denominator.

There is also a potential intervention effect associated with these devices reflected in increased adherence immediately after participants begin using an EMD, which can take 35 (58) to 40 days (34, 59) of continued use to normalize. For this reason, a run-in or practice period with the EMD is recommended prior to starting data collection. This allows reactivity to the device, sensitization to adherence, and increased openings not related to drug consumption (such as curiosity checks) to dissipate. Where this is not possible, or in cases where participants use pocket dosing or situations that cannot accommodate an EMD, alternative measurement strategies may be more desirable.

Proposed Best Practices

Evidence suggests that EMDs are best used to measure adherence behavior when (1) highly granular data are needed to answer a research question, (2) therapeutic coverage is a major concern, (3) observing patterns of adherence is of interest, (4) a long-term trial is required, (5) the population has a high level of cognitive impairment or memory problems, and (6) the assumption that opening a device accurately reflects dose taking at the approximate time of the opening and that doses are not taken without opening the device is well supported. The use of regression analysis (60), modeling techniques (61), and the Bland and Altman method have been used to improve the validity of EMD-related adherence measures (62, 63). Finally, Knafl et al. demonstrated that using adaptive Poisson regression modeling with EMD data may help characterize individual adherence patterns and facilitate censoring approaches (64).

In striving for best practice, the following procedures should be considered and their use reported in studies using EMD data as a primary or secondary outcome.

• Train participants in accurate use of the device before beginning the study.

• Provide a run-in period, during which study participants are taught how to use the device and then take it home to use over a specific period of time (typically 30 - 40 days). Data collected during the run-in period are not used in the final analysis.

• Consider carefully and report the rationale for monitoring all medications or only one.

• Query study participants specifically about whether more than one dose was ever removed at one time from the device and clarify if the extra dose/s were subsequently ingested.

• Examine EMD data for multiple openings within a narrow time window and, if present, query the participant regarding what was happening at that time.

• Consult a statistician to determine the best statistical approach to accurately adjust EMD data based on self-report or diary data.

• Monitor devices for malfunction; establish a procedure for replacing devices if necessary, and be prepared to report these events.

• Review missing data for potential malfunctioning of device.

Pill Counts

Description

Pill-counting strategies for measuring medication adherence rest on the assumption that the number of pills in a patient’s possession reflects the number of pills dispensed minus those ingested (Table I). Techniques for counting pills include inspection of medication containers by clinicians or researchers during scheduled office visits to which subjects bring their medication container(s) (14, 65) or unannounced home visits (11, 25, 27, 66). Telephone-based pill counts ask the patient to count their pills during the course of a telephone conversation at baseline and during a second call at a later point in time, with the difference assumed to reflect pills taken (67). The pills estimated to have been consumed is the numerator and the pills that would have been required based on one’s regimen to cover the days between the first and subsequent count form the denominator. This produces an estimate of adherence (proportion of pills taken as prescribed) although more technically, it is the proportion of pills missing relative to the number that would be missing if the regimen were followed perfectly. For liquid medications, it is possible to measure the height of the liquid remaining in the medication bottle (14).

Summary of evidence

There is both concurrent and predictive evidence supporting pill counts as a measure of medication adherence. Pill counts have been positively associated with other measures of adherence such as EMD (11, 14, 27) and self report (11, 25, 27) and with concurrent viral load suppression (66, 68–70). Some studies, however, failed to demonstrate an association between pill count and viral load change (71). Kalichman et al. demonstrated concordance between unannounced telephone-based pill counts and unannounced home visit based pill counts (67).

Strengths/Problems

Pill counts have been described as an “objective” measure of adherence, because they don’t rely on subject’s self-report (72). Pill count based adherence measures provide an estimate of adherence on average over a given period of time. Consecutive days without medication and other variations in the pattern of medication use are not captured by this method.

The first challenge in operationalizing pill count based estimates of adherence is establishing an accurate baseline count (i.e., how many pills did the patient have in his or her possession at the beginning of the period to be assessed). The second challenge is ascertaining the complete number of pills remaining in the patient’s possession at the time of the count. When relying on the patient to bring medication to a scheduled visit or when making a scheduled home visit, there is the possibility that the patient will dispose of surplus pills to avoid the discovery of non adherence. Further, even in the most well-intentioned of circumstances, it is possible that pills stored elsewhere may inadvertently not be included in the count, leading to an over estimation of adherence.

Proposed best practices

There are considerable threats to validity of pill-count adherence measures; pill counts are used best in conjunction with other outcome measures. In striving for best practice, the following procedures should be considered and their use reported in studies using pill count data as a primary or secondary outcome measure of adherence.

• Control the amount of medication dispensed to improve accuracy of both baseline and follow up counts.

• Limit the time frame between counts to minimize error by keeping the total number of pills available relatively constant and uncomplicated by the addition of one or more refills between assessments.

• Consider unannounced pill-counts to limit the opportunity for subjects to dispose of uningested medication.

• Consider in-residence assessment, which provides better access to all available medications and also allows the home visitor to visually inspect the way medications are stored in the home. Although subjects must be informed about the potential for a home-visit, care can be taken to make these as unexpected as possible.

Pharmacy Refill

Description

The pharmacy refill approach to adherence assessment uses pharmacy records to assess medication acquisition, as distinct from medication consumption (73). Pharmacy refill assessment strategies fall into one of three categories: medication possession ratio (MPR), pill count (PC), or pill pick-up (PPU) (72). The medication possession ratio assesses the amount of time that an individual is in possession of a medication or prescription as a proportion of the time between two medication pick-ups. The time-to-refill approach (18) is a variation of the MPR which was developed to improve the precision of pharmacy refill data over short periods of time. In this approach, adherence is defined as (pills dispensed/pills prescribed per day)/days between refills) × 100%. Pill count reports the quantity of pills used between two medication pick-ups as a proportion of either the number of pills dispensed or the time between pick-ups. Pill pick-up measures whether all or a majority of prescribed medications are picked up as a proportion of the number of refills prescribed. Pill pick-up is typically expressed as a dichotomous variable where the ratio of the number of refills picked up/number of refills prescribed is assessed in comparison to a pre-defined standard, and then categorized as adherent or not adherent.

Pharmacy refill data offer information about medication availability or gaps, single or multiple refill intervals, and continuous or dichotomous adherence variables (73–75). The approach chosen by a researcher is determined by the overall goals of the study, but a careful description of the methods and definitions employed is crucial if comparisons between studies are to be made.

Pharmacy adherence assessment has been used successfully in a wide variety of studies, but has been particularly useful in large population studies, pharmaco-epidemiological studies, and health services research (73). However, the method can be used whenever there is access to pharmacy records that track use over time and whenever the following information is available: drug name, drug dosage, quantity of drug dispensed at each fill, and dates of prescription fills. Ideally, dosing instructions are also included. Some authors suggest that dosing may be imputed from pharmacy policies (73), however without specific dosing instructions, it is difficult to assess adherence using administrative data.

Summary of Evidence

Pharmacy refill has been validated as an approach to assess adherence to a wide variety of medications. In comparing it to other adherence assessment approaches, Kitahata and colleagues found that pharmacy refill was significantly correlated with concurrent patient self-assessment of adherence(76). It also demonstrated good predictive validity with viral load and CD4 response (18, 40, 72, 76). Acri and colleagues found little correlation between adherence by pharmacy refill and by EMD, or with viral load (9), but pharmacy data were collected retrospectively from a variety of different pharmacies, calling into question the accuracy of the assessment. MacMahon and colleagues have questioned the utility of pharmacy refill assessment in predicting future viral load or CD4 count (72).

Strengths/Problems

Interpretation of pharmacy refill data relies heavily on the assumption that the pharmacy record is complete, comprehensive, exclusive and accurate (Table 1). So long as these assumptions hold true, pharmacy refill data can be an effective method to measure medication adherence. However, there are numerous situations that threaten the stability of these underlying assumptions. For example, many pharmacy systems, Medicaid among them, do not include dosing instructions. Where this is the case, a daily dose estimate must be imputed. While this is likely satisfactory for most current antiretroviral agents, it may not be so for other medications and may not remain so over time as regimens change.

Another key assumption is that the pharmacies providing the refill information are the sole source of medications for the individuals participating in the study. In other words, medications can not come from multiple sources such as other pharmacies, friends, or clinic samples. However, a group in Amsterdam demonstrated that it is possible to collect data from multiple pharmacies, albeit with increased effort and time(77). Finally, unless a medication is tracked from when it is first prescribed, it is very difficult to determine if the patient has an oversupply.

Although the methodology of pharmacy refill assessment should be determined by the research question, there are some approaches that are less likely to provide accurate assessments. For example, averages or very gross measures such as number of refills in a certain number of months are typically too simplistic for a useful evaluation of refill adherence and may result in underestimating significant periods of nonadherence.

Proposed Best Practices

Evidence suggests that pharmacy-refill assessment is easiest and most likely to produce accurate data in prospective assessments, where medications are provided by a single payer such as Medicaid, the Veterans Health Administration, a universal health care system, or managed care organizations (40, 76). When these systems are available, pharmacy refill assessment has the benefit of reflecting use in real-world settings. This approach is useful in conjunction with other approaches to adherence assessment. While it cannot provide detail on what medications are taken when and how, it can provide an estimate of the highest possible level of medication consumption.

The ideal situation in which to use pharmacy refill information is as a component of an adherence assessment when the patient’s electronic medical record (EMR) is linked to the pharmacy refill record. The EMR provides additional information on apparent gaps in refills, such as clinician-directed medication holds or regimen changes.

In striving for best practice, the following procedures should be considered and their use reported in studies using pharmacy refill data as a primary or secondary outcome measure of adherence.

• Minimal information required from the pharmacy system includes the regimen, the medications that were dispensed, and the number of days of medications dispensed.

• Use a pharmacy where dosing instructions are included in the database.

• Link pharmacy data with an EMR when possible

• Track refills beginning when the regimen is first prescribed.

• Establish at baseline and at each subsequent data point whether the patient has additional sources through which to obtain medication.

• Follow the pharmacy refills over at least 3 potential dispensing events (75).

• Use an analysis approach to pharmacy refill data that provides a sufficiently nuanced assessment of medication use. For example, rather than exploring number of refills over the number of months of follow-up, assess refill adherence by calculating a percentage of the number of days of medication prescribed over the number of days between refills for a set number of refills.

DISCUSSION

Methods to measure ARV medication adherence have been dichotomized as “direct” versus “indirect” (40) and more recently as “objective” versus “subjective” (33). Electronic monitoring device data, pharmacy refill records, and pill counts are generally considered “objective”. Self-report measures, which rely on the individual to report his or her behavior, are categorized as “subjective”. Further, it sometimes is assumed that “direct” and “objective” measures are more rigorous and more tightly linked to behavioral, biological, and clinical outcomes than are “indirect” and “subjective” measures. However, the evidence supporting this assumption is inconsistent and likely affected by differences in implementation of measures whether classified as either objective or subjective.

In this paper we describe the ways in which adherence measurement is context-laden (e.g. influenced by social desirability, cognition, and data censoring approaches among other factors). Adherence “evidence” can be modified or influenced by personal views (i.e., the best way to censor EMD data), experience (i.e. personal preference for certain self-report measure) or background (i.e. the type of pharmacy system used to collect pill count data). Thus it is more accurate to suggest that all current measures of adherence are subjective to different degrees.

Measures in science are filled with subjective elements. Labeling measures as either objective or subjective is not useful for truly understanding human performance measures (78,79). Some concepts are not unitary constructs and require measures that capture multidimensionality (79). Adherence is one such concept; thus adherence measures should not be categorized as either objective or subjective.

Each adherence measurement approach has strengths and weaknesses and each rests on specific assumptions. The range of assumptions regarding adherence measurement and the diversity with which each approach is implemented strongly suggest that the evaluation of a particular measure outside of the context in which it was used (e.g. the study’s operational protocol) may result in undeserved confidence or lack of confidence in study results. Dichotomous or categorical classification of adherence assessment strategies inhibits the more nuanced evaluation which is essential to establish the evidence base of effective adherence interventions.

A more productive way of evaluating adherence outcomes is to apply Baye’s rule to the language of adherence measurement. Baye’s rule suggests that confidence in the accuracy of a specific measure can be informed by past evidence and current circumstances, along a continuum of probability (80). Confidence in the accuracy of adherence results will be based on how well the researcher described the study’s (a) assumptions (refer to Table 1), (b) operational parameters (refer to Table II), and (c) application of the published report check list items (refer to Table III). The consumer of such published reports can infer a high, moderate or low level of confidence in the results based on the application of these parameters. For example, a low level of confidence could be ascribed to a study that used EMD only to measure adherence without a concurrent method of tracking pocket-dosing or a run-in period, with infrequent uploading of data, no report of quality control, and no description of data censoring procedures. In contrast, a high level of confidence could be reported for a study that used self-report and pharmacy refill records to measure adherence when the researchers address the majority of assumptions, operational parameters and salient details outlined in Table III in their report. Thus, the results of adherence studies should be described in language that is transparent and context-specific.

Table II.

Operational parameters to consider when selecting an adherence measure

SELF REPORT

Will it be possible to separate the collection of self reported adherence information from interactions between the patients and counselors, clinicians, family members, or other interested parties? Will it be feasible to assure patients that self reported adherence information will not be shared with counselors, clinicians, family members, or other interested parties? What is the population’s literacy and numeracy level? Will it be feasible to devise a self report data collection technique that is consistent with this level of understanding? Will data collection be done by an interviewer or through self administered forms, including computer assisted tools? If interviewers will be collecting data, will they be known to the patient in any other role (e.g. as a counselor or member of the community)? If the forms will be self-administered, what level of comfort with the instrument itself will be required? Is this consistent with the population being studied? To what extent does the proposed self report measure require high level cognitive function, including accurate understanding of the regimen and a good memory? To what extent does the proposed self report measure intrude on the patient’s daily life? For example, will keeping a medication daily diary be burdensome? What would be the most effective normalizing or permission statement to provide?

ELECTRONIC MONITORING

Will it be possible to train the participants in use of the device before data collection begins? Will it be possible to provide a run-in period for practice with the device before data collection begins? Is it possible to measure electronically all elements of the medication regimen? If not, which medications are most important to measure? To what extent will the selected medication accurately reflect adherence to the full regimen? To what extent will use of the monitoring device intrude upon the patient’s daily life? Is this a feature than can be mitigated? Is there potential cash or other value associated with the monitoring device? Can the device be used for more purposes than monitoring adherence? How and how frequently will you upload data from the monitoring device? Will the information recorded by the monitoring device be available to the patient or to the patient’s treating clinicians? How will you address situations in which the patient removes more than one dose from the monitoring device (and takes the second dose at the appropriate time)? What is the frequency with which similar devices have malfunctioned? What is the most efficient way to monitor for malfunction? What is the strategy for responding to a malfunction?

PILL COUNTS

How will the baseline number of pills in the patient’s possession at the beginning of data collection be established? If patients are asked to bring pill bottles to an office or clinic for counting, will that be a burden? Will the counts be scheduled or unannounced? If pill counts are to be unannounced, What permissions will be needed ahead of time (for phone calls or home visits)? How will the study deal with patients who are not available for unannounced counts? Will all medications in a patient’s regimen be included in the pill count?

PHARMACY REFILL

Do the members of the population under study receive their medications consistently from the same pharmacy? How accurate are the records at the pharmacies used by the patients participating in the study? What parameters are included in the pharmacy data bases? If patients patronize a number of pharmacies, are the data bases consistent? What permissions are needed from the patients to access the pharmacy records? How engaged or interested in the project are the pharmacists and others at the pharmacies used by the participants?

Table III.

Checklist of adherence measurement items to include in a published report

METHOD	WHAT INFORMATION TO INCLUDE IN PUBLISHED REPORTS
Self-report	What was the question(s) asked? (Count of doses or pills missed or taken?
	Estimation of overall adherence over the time interval?)
	Reliability and validity information for the instrument.
	What was adherence in reference to in terms of agent(s)? (i.e., All ARVs or a selected ARV?)
	How many days or months were in the recall period?
	Was one full 7 day period (or other incorporation of a weekend) included?
	Who or what posed the question(s)? (Clinician, intervener, neutral interviewer, computer with voice, computer text only?)
	Was the question(s) asked one-on-one?
	Was the question(s) part of a larger assessment or research/clinic visit? If so, when did the adherence assessment occur in relation to other procedures and total duration of the visit? (Was it at the end of a long visit? End of a long computer delivered questionnaire?)
	Were there reasons to be concerned about patient-group/item-format mismatch? (Numeracy, wording that is uncommon in local culture, asked only in a second or third language of the patient-group?)
	What threats were there in terms of reporting bias? (Were real or perceived negative consequences associated with reports of non-adherence? Were positive or negative consequences from reports of high or low adherence equal between the study arms?)
	What assurances/strategies were in place to minimize potential self-reporting bias?)
	If data were treated categorically, how were the categorical parameters established and what was the rationale for the categories?
Electronic monitoring	What device (version, specification) was used?
	How and when were participants instructed on device use?
	Was there a lead-in time to control for the intervention effect associated with electronic monitoring devices?
	What quality control methods were used to evaluate device use during the trial?
	What was the malfunction rate during the course of the study?
	What strategies were used to account for non-device pill administration (ie “pocket-dosing)?
	What data censoring procedures were used during analysis?
	If data were treated categorically, how were the categorical parameters established and what was the rationale for the categories?
Pill counts	What was the setting in which the count was conducted (home, office, community)?
	Was the count conducted face-to-face or via communications technology (telephone, internet)?
	Who counted the pills (patient, family member, professional staff)?
	Was the pill-count unannounced or scheduled?
	Which medications were included in the count?
	How was the baseline number of pills from which the remainder was subtracted established?
	If data were treated categorically, how were the categorical parameters established and what was the rationale for the categories?
Pharmacy refill records	Did all patients obtain all medications from a single pharmacy?
	If multiple pharmacies provided data, how was a uniform data base established?
	What information was included in the pharmacy data base? Was dosing information part of the data base or was it imputed?
	How were left-over or carry-forward medications handled?
	Was information from an EMR available and, if so, how was it used?
	If data were treated categorically, how were the categorical parameters established and what was the rationale for the categories?

The ultimate goal of adherence measurement is to improve patient care through increasing our understanding of the relationship between behavior and biology and by establishing a basis on which clinicians and others can select appropriate and effective interventions to move into the clinic. It is a complicated and messy business, and no gold standard has emerged over the past decade of research. However, much has been accomplished and current approaches to measuring adherence are more nuanced and sophisticated than ever before. Thoughtful consideration of the assumptions under girding the selected measures, careful attention to the challenges of implementing each measure, and comprehensive reports will allow researchers, policy makers, clinicians, and activists to make informed choices for the benefit of patients and their communities.