Using Electronic Monitoring Devices to Assess Medication Adherence: a Research Methods Framework

Abstract

While electronic adherence monitoring devices (EAMDs) are the preferred and most objective medication adherence measurement strategy for many populations and research questions, there is no comprehensive methodological framework for EAMD use. We synthesize recommendations from experts in adherence science and the scientific literature to create a temporal framework of EAMD research methods. The goal of this framework is to provide a step-by-step guide that will enable researchers to design, prepare, implement, and clean data from rigorous, high-quality studies using EAMDs to assess adherence. Resources including a checklist of methodological considerations and example protocols have been created to assist readers in using this framework.

Medication non-adherence, or medication taking behavior that does not align with medical or health advice,¹ is a primary cause of treatment failure and a significant public health concern.² In the USA alone, medication non-adherence is estimated to result in over 100,000 preventable deaths and $100–$300 billion in potentially avoidable health care costs^{3, 4} each year. Increasing medication adherence, thus, has the potential to improve population health and reduce health care spending.⁵

Despite rapid growth in adherence science,⁶ effective methods of improving medication adherence remain largely unknown and there are few populations for which evidence-based adherence-promotion interventions are available.^{5, 7, 8} Efforts to improve adherence have been limited, in part, by the lack of accurate and reliable measures of medication-taking behavior.⁸ To date, medication adherence has primarily been assessed using self-report measures, a strategy known to overestimate adherence and increase the study’s risk of bias.⁸ A more objective measure of medication adherence can be obtained via electronic adherence monitoring devices (EAMDs), electronic pill bottles, pill boxes, or inhaler attachments which contain a computer chip that records dates and times of device manipulations (e.g., inhaler actuations, bottle/box openings; hereafter referred to as actuations). As technological advances improve their affordability and accuracy,^{9, 10} EAMDs are increasingly being recognized as the preferred and most objective adherence measurement strategy for many populations and research questions.¹¹

The features that make EAMDs attractive to researchers (e.g., collection of daily data) also pose numerous methodological questions (e.g., “How do I transform a series of actuations into a value representing adherence?”). Researchers have recognized the value of disseminating guidance for answering these questions and published manuscripts to inform EAMD selection⁹ and manuscript preparation.¹² Recommendations regarding other aspects of EAMD use, however, have largely been neglected or simply mentioned in discussion sections (e.g., “use EAMDs along with other adherence measures”¹³) without the contextual framework necessary to guide decision-making for specific studies and populations. The purpose of this manuscript is to create the first comprehensive framework of EAMD research methods that provides step-by-step guidance to inform decisions regarding the design, preparation, implementation, and data cleaning of studies using EAMDs. It is our hope that this framework will enable physicians and other health care providers to utilize EAMDs as they contribute to the adherence research critical for improving patient outcomes.⁸

FRAMEWORK DEVELOPMENT

Framework content was generated from published literature and experts in adherence science (co-authors and panel members) with more than 50 years of combined experience as principal investigators of grant-funded adherence research. First, recommendations for EAMD use were obtained from the co-authors’ standard operating procedures and manuscripts identified via a literature review (PubMed/MEDLINE search using the terms “electronic monitor” OR “electronic monitoring” AND adherence AND medication). Recommendations were then synthesized into a temporal framework detailing research methods for the design, preparation, implementation, and data cleaning of studies using EAMDs. Finally, the framework was refined with feedback from a panel of experts in adherence science (three adherence scientists and one postdoctoral fellow) and supplemented with exemplar materials.

FRAMEWORK OVERVIEW

The framework of EAMD research methods includes 10 steps (Table 1) and begins with the assumption that EAMDs have been deemed appropriate for the purpose, methods, population, and resources of the proposed research following a comparison of adherence assessment strategies.^{10, 11, 14–16}

Table 1.

Electronic Adherence Monitoring Device (EAMD) Research Methods Framework

1. Define medication adherence
	□ Define the summary statistic (N/P) that will be used to quantify the ratio of the patient’s medication-taking behavior (N) as compared to the prescribed medication regimen (P)
	□ Define the time period (t) over which data will be synthesized to compute adherence
	□ Determine the number and timing of adherence assessments
2. Select an EAMD
	□ Identify EAMDs that have demonstrated acceptable accuracy in laboratory-based evaluations
	□ Select an EAMD whose characteristics align with the study aims, population, and resources
	□ Conduct pilot field testing of the selected EAMD*
3. Design procedures for EAMD use
	□ Select EAMD use timeline (i.e., start and stop)
	□ Determine the timing of and procedures for EAMD downloads
	□ Determine which EAMD features (e.g., reminders) will be activated and when
4. Create and follow standardized EAMD preparation procedures
	□ Inspect each EAMD for physical damage
	□ Activate EAMDs in accordance with instructions and track activation date
	□ Synchronize EAMDs with other study devices (e.g., time on laboratory computer)
	□ Enable EAMD features relevant to the study aims
5. Design and conduct EAMD pre-testing
	□ Conduct a performance check of each EAMD to assess accuracy
	□ Conduct a laboratory-based test of a subset of EAMDs from each shipment
6. Educate study staff, patients, and health care providers
	□ Train study staff in EAMD use
	□ Discuss EAMD purpose and use with patients and providers
	□ Provide patients and providers with written educational materials regarding EAMD use
	□ Affix relevant instructions and study staff contact information to EAMDs
7. Tracking missing or extra actuations and periods of nonuse
	□ Track discrepancies between EAMD actuations and the patient’s medication-taking behavior
8. Assess and monitor changes in the prescribed medication regimen
	□ Asses the patient’s medication regimen upon enrollment
	□ Prospectively track medication changes, holds, and/or discontinuations throughout the study
9. Transform raw EAMD data to taken dose data
	□ Export data from EAMD
	□ For EAMDs recording dosing data, synthesize actuations to reflect taken doses*
	□ Remove extra actuations likely representative of user error, device malfunction, or dumping
	□ Combine EAMD data with patient-reported adherence data*
10. Compute adherence
	□ Define a “24-h day” for each patient
	□ Recode taken dose data into a daily variable (N) representing the patient’s behavior
	□ Create a daily variable (P) representing the prescribed medication regimen
	□ Compute the daily summary statistic (N/P × 100)
	□ Recode daily summary statistics to reflect restrictions in range*
	□ Adjust for periods of nonuse and regimen changes
	□ Compute adherence by averaging the daily summary statistics over the time period (t)

*Step may not be necessary or appropriate in all instances

STUDY DESIGN

Step 1: Define Medication Adherence

EAMDs are considered the “gold standard”¹¹ for assessing implementation adherence or the “extent to which a patient’s actual dosing corresponds to the prescribed dosing regimen.”⁶ Implementation adherence (hereafter referred to as adherence) compares the patient’s medication-taking behavior (N) over time (t) to the prescribed medication regimen (P) over time (t) using the formula $\frac{N_t}{P_t}\times \mathrm{100}.$⁶ EAMD use begins by defining the ratio of the patient’s medication-taking behavior as compared to the prescribed medication regimen, or the summary statistic $\left(\frac{N}{P}\right)$ and time (t).

There is no “correct” summary statistic. Often, the summary statistic quantifies the percentage of prescribed doses taken $\left(\frac{\mathit{\text{number of doses taken}}}{\mathit{\text{number of doses prescribed}}}\right)$, percentage of prescribed doses taken on time $\left(\frac{\mathit{\text{number of doses taken on time}}}{\mathit{\text{number of doses prescribed}}}\right)$, or percentage of days on which the prescribed number of doses was taken $\left(\frac{\mathit{\text{number of days with correct doses taken}}}{\mathit{\text{number of days}}}\right)$.⁶ Figure 1 illustrates how adherence for a once-daily regimen (panels a–c) and a twice-daily regimen taken on Monday, Tuesday, and Wednesday only (panels d–f) would be calculated using these three summary statistics over a 7-day period (t). Summary statistic selection should be informed by medication characteristics and the research question. For example, the percentage of prescribed doses taken on time (Fig. 1b) may be appropriate when medication dosing timing is related to outcomes (e.g., progestin-only oral contraceptives).¹⁷ Alternatively, if the prescribed regimen includes medication on specific days of the week, the percentage of days with prescribed doses taken would capture not only whether the patient was taking doses on the days prescribed but also if they appropriately did not take doses on the other days of the week (see Fig. 1f).

Fig. 1.

Computing adherence summary statistics using electronic adherence monitoring device data. *Shaded days represent days on which the daily summary statistic ≥ 100%.

As with the summary statistic, there is rarely a “correct” time period (t) over which data should be aggregated and a rationale for this decision should be articulated.¹⁸ In some instances, study aims may inform time period selection. When the study purpose is to examine correlates of medication adherence, a time period that aligns with that of the potential correlates should be considered. For example, in a study designed to examine the correlation between medication adherence and depressive symptoms (via PROMIS Depression),¹⁹ adherence data should be aggregated over the period assessed via the PROMIS Depression (past 7 days). In other instances, a time period may reflect a clinically relevant episode such as a medication regimen period (e.g., cancer treatment cycle).

Because EAMDs must be used for multiple days to obtain sufficient data to compute adherence, all studies including EAMDs are longitudinal. Often, EAMD data are used to compute adherence over multiple time periods. The assumptions and requirements of the planned analytic techniques (e.g., minimum number of data points¹⁸) should be considered when selecting the number of time points. In trials of adherence-promotion interventions, adherence should be computed at a minimum of two time points: pre-intervention and during or post-intervention. Pre-intervention data is necessary to control for baseline functioning in estimates of the treatment effect. The selection of subsequent time point(s) is dependent on the research question(s) of interest. Figure 2 details questions that may be answered by computing adherence pre-intervention and at various subsequent periods.

Fig. 2.

Periods of adherence measurement and potential associated study aims.

Step 2: Select an EAMD

Device selection should occur next as EAMD characteristics influence the study design. Because EAMDs may not perform as advertised, we recommend only considering EAMDs that have demonstrated high accuracy in independent evaluations.²⁰ Data on EAMD accuracy may be obtained from published laboratory-based evaluations^{9, 21–24} or collected by the research team (see sample protocol⁹).

Once EAMDs with acceptable accuracy have been identified, differences in EAMD design (e.g., pill boxes versus bottles), medication capacity, data storage capacity, battery life, data transmission capabilities, on-board reminder features, interface with other products or services (e.g., mobile applications), and cost can be reviewed.^{9, 25–27} We encourage researchers to obtain this information from multiple sources including: EAMD manufacturer-provided materials, discussions with manufacturer representatives, and/or consultations with researchers with experience using the EAMD. A framework for considering the impact of these features on EAMD performance is detailed by de Bleser et al.²⁰ Particularly when EAMD features (e.g., real-time remote data transmission) are essential for the study aims or feasibility, pilot field testing is recommended prior to finalizing EAMD selection (see examples^{23, 28–31}).

Step 3: Design Procedures for EAMD Use

Once an EAMD is selected, procedures for three aspects of EAMD use can be determined: EAMD use timeline, data downloads, and EAMD feature activation. Timeline selection begins with a decision regarding a run-in period. Empirical data and recommendations regarding run-in period inclusion are mixed¹⁰ with advocates arguing that passive EAMD use increases adherence¹⁸ and opponents citing studies in which EAMD use alone did not change adherence.^32–34 Given the lack of consensus, we recommend including a run-in period if feasible. If a run-in period is included, its duration should be sufficient to allow for reactivity to EAMD use to washout, or return to baseline, prior to the start of other study procedures.

EAMD data may be downloaded at intervals throughout the study or at study completion. Intermittent data downloads may be included to obtain data to inform study procedures (e.g., intervention delivery). Intermittent data downloads also allow researchers to assess for EAMD malfunctions and user errors. As addressing EAMD malfunctions and user errors during the study maximizes the likelihood of obtaining complete and accurate data, intermittent data downloads are recommended in all studies. Data download procedures differ by EAMD, and the download schedule should balance the potential benefits with feasibility and patient burden. For EAMDs in which data are automatically uploaded to a cloud or mobile application, downloads can be completed as frequently as useful (e.g., weekly) as they do not require patients to complete any additional study procedures. A less frequent schedule (e.g., monthly) may be more appropriate for EAMDs in which downloads require patients to complete steps at home (e.g., tap an EAMD to a near field-enabled device) or bring their EAMDs to an in-person study visit.

The final EAMD study design decision pertains to EAMD features such as alarms, text message reminders, and adherence calendars.⁹ Any EAMD feature beyond passive monitoring has the potential to change adherence behavior and should be disabled in observational research and intervention run-in, baseline, and post-treatment periods. In intervention studies, EAMD features should only be activated if their mechanisms of action and targets align with those of the proposed research. For example, it would be appropriate to activate EAMD reminder features in a behavioral intervention designed to improve adherence by targeting forgetting but not in an educational intervention designed to improve adherence by targeting regimen knowledge.^{35, 36}

STUDY PREPARATION

Step 4: Create and Follow Standardized EAMD Preparation Procedures

During study preparation, EAMDs should be inspected for physical damage¹⁰ and activated in accordance with manufacturer-provided instructions. For battery-powered EAMDs, the date of EAMD activation should be recorded to track the estimated date of battery expiration. Following activation, the clock on the EAMD should be synchronized to other relevant devices (e.g., time on laboratory computer)¹³ and relevant features should be enabled (e.g., setting auditory alerts to “on”).

Step 5: Design and Conduct EAMD Pre-testing

To pre-test EAMDs, we recommend conducting a performance check of each EAMD and a more thorough laboratory-based test of a subset of EAMDs.²⁵ A performance check evaluates whether each EAMD is accurately recording, storing, and transmitting actuations and may be conducted by (1) actuating the EAMD three times, (2) recording the date and time of each actuation on a tracking log, (3) downloading the EAMD data, and (4) comparing the EAMD-produced data with the lab-recorded data.^{10, 25} The performance check should also evaluate the accuracy of features that will be enabled during the study. When performance checks reveal discrepancies between EAMD-produced and lab-recorded data, the EAMD’s mechanism for detecting actuations (e.g., optical sensor), features, energy source, circuitry, internal memory, clock, and/or the associated software or hardware may be malfunctioning.^{25, 31} A threshold for passing the performance check should be set and failing EAMDs should be repaired with manufacturer guidance or replaced.²⁵

A laboratory-based test provides more detailed accuracy data and information on EAMD features which change with use (e.g., battery life). We recommend conducting a laboratory-based test on a subset of EAMDs in each shipment as minor changes in an EAMD model and/or associated hardware and software can impact accuracy.^{10, 25} The laboratory-based test should be conducted using a standardized protocol (see examples^{9, 21, 22, 25}) and include a threshold for passing.

Step 6: Educate Study Staff, Patients, and Health Care Providers

Health care provider, study staff, and patient behaviors can impact data quality. In previous studies, providers have advised patients to discontinue EAMD use,¹³ staff have failed to complete tasks required for continued EAMD use (e.g., SIM card activation),³¹ and patients have used the EAMD in a manner that did not record actuations (e.g., not closing the cap completely),^{18, 25} damaged the EAMD (e.g., submerged in water),¹⁸ discontinued EAMD use,¹³ and/or disposed of the EAMD.¹⁰ Education can minimize the likelihood of these events, and we recommend that research teams train all study staff in EAMD use, explain the purpose of EAMD use to providers and patients, provide written educational materials, and affix relevant instructions and study staff contact information to the EAMD.^{10, 13}

STUDY IMPLEMENTATION

Step 7: Track Missing or Extra Actuations and Periods of Nonuse

During EAMD use, there are likely to be instances in which EAMD-registered actuations do not align with the patient’s medication-taking behavior. These discrepancies have three common sources. First, actuations may be missing when the EAMD malfunctions or a patient “pocket doses” (takes out more than one dose at a time and takes one of those doses later).¹⁸ Second, extra actuations may be present when the EAMD is actuated for reasons other than medication taking (e.g., adding a refill).¹⁸ Third, there may be periods during which the EAMD is not used for medication administration. To facilitate reconciliation of EAMD actuations with patient behavior, missed and extra actuations and periods of nonuse should be tracked prospectively using medical record reviews, and/or patient self-report measures (e.g., diaries).^{10, 13}

Accurately tracking nonuse requires procedures for differentiating nonuse from non-adherence. A period during which no EAMD actuations were registered may be marked as nonuse when established data sources (e.g., medical record) indicate that the patient’s situation precluded EAMD use (e.g., inpatient hospitalization, incarceration).¹³ When there is not a clear explanation for why actuations are not being registered, standard procedures should be established and followed to determine whether this period should be coded as nonuse or non-adherence. For example, if the results of an intermittent data download (see step 3) suggest no EAMD actuations are being registered, the team may begin by contacting the patient to ensure the EAMD is charged, functioning, and currently in use. If the EAMD is not functioning properly or the patient has elected not to use the EAMD, it may be appropriate to code these periods as nonuse. If the phone-based check suggests the EAMD is working and/or the patient reports using the EAMD, however, the absence of actuations may more accurately reflect non-adherence.

Step 8: Assess and Monitor Changes in the Prescribed Medication Regimen

Accurate information on the prescribed medication regimen is essential to compute adherence. A standardized procedure (e.g., medical record review) should be used to assess the patient’s medication regimen upon enrollment. Medication prescription documentation procedures vary, and we recommend confirming this information with at least one additional source. To be equipped with the information necessary to adjust EAMD data to reflect medication changes, holds, or discontinuations, the regimen should be tracked prospectively throughout the study.

ANALYSIS PREPARATION

Step 9: Transform Raw EAMD Data to Taken Dose Data

Following study completion, actuation data are exported from the EAMD to a software program. Exported data typically appear as a list of events and must be transformed to a dataset of “taken” doses prior to analysis. Some EAMDs (e.g., inhaler attachments) register information on the number of doses or pills dispensed. For these EAMDs, data transformation begins by defining the number and timing of actuations required for a dose to be considered taken (e.g., 1 dose = X actuations within Y minutes). Each set of X actuations occurring within the specified time frame (Y minutes) should then be synthesized into a date and time stamp representing one taken dose.

Next, data should be examined for “extra” actuations. When multiple actuations occur in quick succession, it is likely that these actuations represent user error (e.g., not closing the cap completely), device malfunction (e.g., inhaler attachment EAMD registering multiple actuations due to movement while in a backpack), or intentional “dumping” rather than a series of taken doses. These extra actuations should be removed by following a procedure³⁷ that details a dose window during which only one actuation will be counted. For example, a researcher may set a dose window of 30 min for a once-daily pill, assuming that multiple actuations occurring within a 30-min period are reflective of user errors, device malfunctions, or “dumping.” Based on this rule, if the EAMD registers actuations at 8:03:10 a.m., 8:04:12 a.m., and 8:04:33 a.m., this series of events would count as one taken dose at 8:03:10 a.m.

After transforming raw EAMD data to a dataset of taken doses, some researchers choose to reconcile taken dose data with self-report data (e.g., diaries). For example, taken doses may be manually added to reflect instances in which patients reported taking their medication from a container other than the EAMD (e.g., “pocket dosing”) and manually removed when the EAMD-registered dose reflected a patient-reported non-dosing actuation (e.g., opening the EAMD for a refill transfer). It is important to note that combining EAMD data with patient-reported data introduces the limitations inherent in self-report adherence measures (i.e., over-reporting) and these methods and their resulting limitations should be detailed in the manuscript.

Step 10: Compute Adherence

While not required, it is common practice that adherence $\left(\frac{N_t}{P_t}\times \mathrm{100}\right)$ is computed by calculating the summary statistic at the daily level $\left(\frac{N}{P}\times \mathrm{100}\right)$ and then averaging these daily summary statistics over a time period (t). This process includes seven steps as described below and depicted in Table 2. First, a 24-h day is defined. Typically and in Table 2 (column 1), the 24-h day is defined as 12:00:00 a.m. to 11:59:59 p.m. For individuals taking doses around midnight (e.g., adolescents, shift workers), the time window may need to be shifted to reflect their 24-h day (e.g., to 3:00:00 a.m.–2:59:59 a.m.)Second, taken dose data are recoded into a new variable representing N (the patient’s behavior) over the 24-h period. When the summary statistic is the percentage of prescribed doses taken, N = the number of doses taken in a 24-h period (Table 2, column 3). To compute N for the summary statistic percentage of prescribed doses taken on time, the number of taken doses falling within a pre-determined window (e.g., 11:00:00 a.m. ± 60 min) is computed. For the summary statistic percentage of days on which the prescribed number of doses were takenis a dichotomous indicator of whether or not the correct number of doses was taken in the 24-h period.

Table 2.

Steps in Computing Adherence

Step 1		Step 2	Step 3	Step 4	Step 5	Step 6
Day in time period (t) (12:00:00 a.m.–11:59:59 p.m.)	“Taken” doses per EAMD	Number of doses taken per day (N)	Number of prescribed doses per day* (P)	Daily summary statistic $\left(\frac{N}{P}\times 100\right)$	Daily summary statistic with upper limit imposed	Daily summary statistic with periods of nonuse removed
Sun 10/27/2019	10:19:05 p.m.	1	1	100%	100%	100%
Mon 10/28/2019		0	1	0%	0%	‡
Tue 10/29/2019	9:24:18 p.m.	1	1	100%	100%	100%
Wed 10/30/2019	9:38:31 p.m.	1	1	100%	100%	100%
Thu 10/31/2019	8:55:28 p.m.	1	1	100%	100%	100%
Fri 11/01/2019		0	1	0%	0%	0%
Sat 11/02/2019	12:07:24 a.m. 9:02:15 p.m.	2	1	200%	100%†	100%
			Step 7: adherence $\left(\frac{100\%+100\%+100\%+100\%+0\%+100\%}{6}\right)$			83%

EAMD electronic adherence monitoring device

*Prescribed medication regimen includes a once daily medication

†Range of 0–100% imposed such that values > 100% were recoded to 100%

‡Value removed as patient took their medication during a day hospital visit and not from the EAMD

Third, the prescribed regimen (P) is determined for each day (Table 2, column 4). For regimens including a once-daily medication 7 days a week, P = 1 in each instance. Fourth, the daily summary statistic is computed using the formula $\left(\frac{N}{P}\times \mathrm{100}\right)$ (Table 2, column 5). Fifth, if instances in which patients take more doses than prescribed are not of interest and/or it is assumed that a missed dose on one day cannot be “made up” by taking an extra dose on another day, the researcher may wish to impose an upper limit on the summary statistic by adjusting values > 100% to 100% (Table 2, column 6).

Once a dataset of daily summary statistics is computed, the time period (t) of each patient should be adjusted for periods of EAMD nonuse and regimen changes. Often, periods of nonuse (defined in step 7) are excluded so these days do not “count against” the patient. For example, if adherence is defined as the percentage of prescribed doses taken over 90 days (t = 90 days) and the patient was hospitalized during the last 30 days, the time period may be truncated to days 1–60. The hypothetical example in Table 2 assumes that the patient took their medication during a day hospital admission (instead of from the EAMD) on Monday. This day of EAMD nonuse was accounted for by excluding the Monday daily summary statistic (Table 2, column 7). If a patient’s medication was held and/or discontinued during the study period, these periods should also be excluded. Finally, with the time period (t) defined for each patient, adherence is computed by averaging the daily summary statistics over this period (Table 2, column 7). As detailed in Figure 1, the value of adherence may differ across summary statistics (percentage of prescribed doses taken, percentage of prescribed doses taken on time, percentage of days on which the prescribed number of doses was taken), further highlighting the importance of articulating a rationale for the selected definition.

DISCUSSION

As a result of technological and functional advances in EAMDs, researchers increasingly have access to a valuable measurement strategy to aid in the field’s efforts to improve medication adherence and, ultimately, health and economic outcomes. Despite the complicated and nuanced nature of EAMD use, we believe that health care providers without extensive training in adherence science can capitalize on this technology with proper methodological guidance. To facilitate the development and implementation of rigorous and reproducible research including EAMDs, this manuscript provided the first comprehensive framework of EAMD research methods. It is hoped that this framework prompts discussions of methodological considerations that may emerge or evolve as adherence science and EAMD technologies advance.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they do not have a conflict of interest.