Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2019 Oct 29.
Published in final edited form as: Ann Intern Med. 2018 May 22;169(6):419–422. doi: 10.7326/M17-3306

An Independent Evaluation of the Accuracy and Usability of Adherence Electronic Monitoring Devices

Meghan E McGrady 1,2, Christina E Holbein 1, Aimee W Smith 1, Caroline F Morrison 3, Kevin A Hommel 1,2, Avani C Modi 1,2, Ahna L H Pai 1,2, Rachelle R Ramsey 1,2
PMCID: PMC6817974  NIHMSID: NIHMS1055048  PMID: 29800004

Background

Non-adherence to prescription medications is exhibited by approximately 50% of patients and is a primary cause of treatment failure, morbidity, and mortality, resulting in an estimated $300 billion in potentially avoidable U.S. health care costs each year (1, 2). Increasing adherence may improve health outcomes and reduce costs, even more so than the advent of new medical treatments (1, 3). Such efforts, however, have been limited by the lack of objective measures of adherence (3). Adherence electronic monitoring devices (AEMDs) are the most objective and preferred measurement strategy for clinical practice and research, but lack indpendently-obtained (versus manufacturer-provided) accuracy and usability data.

Objective

To assess the accuracy of commercially-available AEMDs and guide readers in selecting the optimal AEMD based on their needs/goals (4).

Methods and Findings

This study was exempt from IRB review. PubMed/MEDLINE, CINAHL, PsycINFO, and Scopus were searched to identify all AEMDs included in a published study of medication adherence. To maximize the likelihood that all relevant AEMDs were identified, additional AEMDs were solicited from experts in adherence (co-authors). The systematic review identified 12 AEMDs and experts identified an additional 10 AEMDs. Of the 22 AEMDs, 15 were available for purchase and 10 were obtained for accuracy testing and examination of usability features (n = 3 manufacturers did not reply, n = 1 manufacturer refused to sell their AEMD, n = 1 AEMD cost-prohibitive [>$500]).

Four of each AEMD were obtained, with three tested according to a 21 day protocol reflecting a twice daily medication prescribed seven days a week. Each protocol simulated four adherence patterns: 1) “perfect adherence” (AEMD opened at 100% of prescribed doses); 2) “non-adherence: additional doses” (AEMD opened at five randomly selected times in addition to prescribed doses [150%]); 3) “non-adherence: missed doses” (AEMD opened at 50% of prescribed doses); and 4) “device non-use” (AEMD opened at 0% of prescribed doses) (5). Perfect adherence, non-adherence: additional doses, and non-adherence: missed doses were tested over one of three work weeks (Monday-Friday). AEMDs were not opened on weekends to simulate device non-use. In all, each protocol included 25 scheduled openings. Three protocols were generated using a counterbalanced design where each of the three AEMDs of a given model were tested for a different adherence pattern simultaneously. Protocols are available from https://www.cincinnatichildrens.org/research/divisions/b/psychology/labs/mcgrady/projects.

The first author recorded the date and time of each opening on a paper form. A second individual independently recorded the date and time of a random 10% of openings (intra-class correlation = 1.00).

The primary measure of AEMD accuracy was defined as the number (out of 25) of AEMD-registered openings that fell within 120 seconds of the date/time recorded on the paper form (4). Additional measures of AEMD accuracy are defined in Table 1. AEMD characteristics (dimensions, data transmission, reminders, notifications, and data access) were obtained from publically-available information. Capacity, the number of 10 mm round tablets and 18 mm oval caplets held by each AEMD, was assessed independently by two co-authors.

Table 1.

Adherence Electronic Monitoring Device (AEMD) Accuracy

Device (Manufacturer) N Devices Working*
4		 N Registrations w/n 120 s†
25		 N Registrations w/n 600 s‡
25		 M Discrepancy in Seconds§ N Extra Openings||
eCAP™ 1-clic® (Information Mediary Corporation) 4/4 25/25; 25/25; 25/25 25/25; 25/25; 25/25 21; 24; 32 0; 0; 0
eCAP™ Argus-loc (Information Mediary Corporation) 4/4 25/25; 25/25; 25/25 25/25; 25/25; 25/25 16; 27; 35 0; 0; 0
iCAP (Medisafe iConnect) 2/4 13/25; 19/25; 13/25; 19/25; 18; 24; **
MEMS® SmartCap® (AARDEX Group) 4/4 24/25; 25/25; 25/25 24/25; 25/25; 25/25 29; 30; 39 0; 0; 0
MEMS® TrackCap® (AARDEX Group) 4/4 25/25; 25/25; 25/25 25/25; 25/25; 25/25 42; 49; 53 0; 0; 1
MedFolio Cellular Pillbox (MedFolio LLC) 4/4 †† 24/25; 24/25; 25/25‡‡ †† **
MedMinder™ Maya™ (MedMinder) 4/4 25/25; 25/25; 25/25 25/25; 25/25; 25/25 3; 5; 16 3; 34; 86
MedSignals® Pill Case/Monitor (MedSignals/VitalSignals LLC) 4/4 24/25; 24/25; 24/25 25/25; 25/25; 25/25 53; 61; 71 **
SimpleMed+ (Vaica) 3/4 25/25; 25/25; 25/25 25/25; 25/25; 25/25 3; 9; 16 **
Wisepill RT2000 Dispenser (Wisepill Technologies) 4/4 21/25; 23/25; 25/25 23/25; 25/25; 25/25 25; 29; 69 0; 0; 2
Open in a new tab

w/n = within

*

Four devices of each AEMD type were ordered, with three used for testing (one extra device ordered in case of malfunction). Value represents the number of each AEMD type (out of four) that were functioning after activation in accordance with manufacturer guidelines.

†

Primary measure of accuracy - the number of AEMD registrations that fell within 120 seconds of the date/time recorded on the paper test protocol out of the number of scheduled openings per the test protocol (25). By testing the “non-adherence: additional doses” pattern, it was realized that not all AEMDs are designed to record “additional” registrations as was being tested by this pattern (see **). As a result, the 5 planned additional openings included to reflect “non-adherence: additional doses” were excluded from the calculation of the primary measure of accuracy (resulting in 25 scheduled openings per test protocol as opposed to the originally planned 30 scheduled openings per test protocol).

‡

Secondary measure of accuracy – the number of AEMD registrations that fell within 600 seconds of the date/time recorded on the paper test protocol out of the number of scheduled openings per the test protocol (25).

§

Secondary measure of accuracy – the average difference per each AEMD (in seconds) in the time recorded by the AEMD and the time recorded on the paper test protocol.

||

Secondary measure of accuracy – the number of “extra” registrations recorded by the AEMD that were not recorded on the paper test protocol. For example, the MEMS® TrackCap® device with N Extra Openings = 1 represents an AEMD that registered 1 opening in addition to the 25 scheduled openings per the paper test protocol.

Because only 2 devices were operational, 2 instead of 3 devices were tested.

**

Prior to use, these AEMDs are programmed with the scheduled time(s) of medication administration (or times the AEMD should be opened). These AEMDs do not register “additional” openings. Specifically, after the AEMD is opened at the scheduled time, additional subsequent openings prior to the next scheduled medication administration are not recorded by the AEMD.

††

AEMD does not record the time stamp of openings (only the date), precluding the calculation of the primary measure of accuracy or mean discrepancy.

‡‡

As this AEMD records only date stamps and not times, this measure of accuracy was defined as the number of AEMD registrations that occurred on the same date as those recorded on the paper test protocol out of the number of scheduled openings per the test protocol (25).

All three tested devices of seven AEMD types registered ≥24/25 openings within 120 seconds of the date/time recorded by the research team. The average discrepancy between the AEMD-registered time and the date/time recorded by the researchers for each device ranged from 3–71 seconds. Four AEMDs do not register extra openings by design. Five AEMD types registered ≤2 additional openings per tested device. Table 1 details AEMD accuracy. Table 2 includes AEMD features relevant to usability.

Table 2.

Adherence Electronic Monitoring Device (AEMD) Characteristics*

On-Board Reminders
 Caregiver Notifications
Device Dimensions (mm) Data Transmission Audible Alarm Light Other Patient Reminder Missed Dose Incorrect Dose Refill Due Real-Time Data Access Data Editable Bottle or Bin Capacity
eCAP™ 1-clic® 48 × 25† Manual download with CertiScan RFID reader No No No No No No No Yes Multiple bottle sizes
eCAP™ Argus-loc 50 × 27† Near Field Communication (NFC) or Bluetooth No No No No No No Yes Yes Multiple bottle sizes
iCAP 40 × 19† Bluetooth Smart 4.1 Over the Air No No Push notifications in MediSafe application Push notifications in Medisafe application No No Yes No 72 10 mm tablets or
58 18 mm caplets
MEMS® SmartCap® 49 × 39† Manual download with MEMS® reader No No No No No No No Yes Multiple bottle sizes
MEMS® TrackCap® 50 × 39† Manual download with MEMS® reader No No§ No No No No No Yes Multiple bottle sizes
MedFolio Cellular Pillbox 366 × 178 × 58 pill box with 28 bins (4 doses a day × 7 days a week) Self-contained cellular modem Yes Yes SMS Text, Email SMS Text, Email No No Yes No 20 10 mm tablets or
16 18 mm caplets
MedMinder™ Maya™ 356 × 279 × 51 pill box with 28 bins (4 doses a day × 7 days a week) Self-contained cellular modem Yes Yes Phone SMS Text, Email SMS Text, Email SMS Text, Email Yes No 20 10 mm tablets or
16 18 mm caplets
MedSignals® Pill Case/Monitor 125 × 75 × 45 pill box with 4 bins Self-contained cellular modem Yes Yes No SMS Text, Email, Phone No SMS Text, Email, Phone Yes Yes 44 10 mm tablets or
36 18 mm caplets
SimpleMed+ 304 × 170 × 35 pill box with 28 bins (4 doses a day × 7 days a week) Self-contained cellular modem Yes Yes SMS Text, Email SMS Text, Email SMS Text, Email SMS Text, Email Yes Yes 15 10 mm tablets or
12 18 mm caplets
Wisepill RT2000 Dispenser 130 × 60 × 30 pill box with 2 bin or 7 bin insert Self-contained cellular modem No No SMS Text, Email SMS Text, Email SMS Text, Email No Yes Yes 7 bin option:
9 10 mm tablets or
10 18 mm caplets

2 bin option:
41 10 mm tablets or
36 18 mm caplets
Open in a new tab
*

Information obtained from testing, materials provided by the device manufacturer with purchase, or materials published by the device manufacturer via the following websites:

†

Diameter × Height

§

Features LCD display indicating number of doses taken in the past 24 hours or number of hours elapsed since last dose.

Discussion

In this independent evaluation, seven of 10 AEMDs accurately registered ≥24/25 openings. Among these, the AEMD that is “best” for a given clinical program or research study depends on the characteristics of the patient population (i.e., medication number/size, medication schedule, patient preferences) and clinical/research aim. Selecting an AEMD with features that align with patient characteristics may increase patient acceptability and uptake. Clinicians and researchers are also encouraged to consider the degree to which AEMD features are consistent with the patients’ geographical location (e.g., AEMDs requiring cellular connectivity may not function in areas with limited service). Further, users should consider the clinical/research goal of AEMD use (e.g., efforts to passively monitor adherence may preclude the use of AEMDs with reminders) and budget. AEMD costs vary substantially and unfortunately, pricing information is not publically available for all AEMDs. Interested users, thus, may need to independently solicit quotes and should be aware that some AEMD manufacturers will only provide confidential quotes and/or quotes after obtaining information about budget availability.

This study has multiple limitations. Not all available AEMDs were tested and AEMDs were tested in a controlled laboratory without activating all available features. The full functionality, usability, and connectivity of AEMDs requires future study.

In summary, there are multiple AEMDs that accurately assess medication adherence behavior. To identify the most appropriate AEMD among these, clinicians and researchers are encouraged to consider the characteristics of their clinical/research aim and patient population.

Acknowledgements

The authors gratefully acknowledge Gabriella Breen, B.S. for her assistance with data management and manuscript formatting.

Funding Source

M.E.M. is supported by the National Cancer Institute of the National Institutes of Health under Award Number K07CA200668. C.E.H. and C.F.M. are supported by the Eunice Kennedy Shriver National Institute of Child Health and Human Development of the National Institutes of Health under Award Number T32HD068223. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. M.E.M. and R.R.R. received funding from the Center for Adherence and Self-Management at Cincinnati Children’s Hospital Medical Center to support the collection and management of data.

Footnotes

Registration

Not Applicable

Conflicts of Interest

The authors have no conflicts of interest relevant to this manuscript to disclose.

References

  • 1.Sabate E Adherence to long-term therapies: Evidence for action. Geneva, Switzerland: World Health Organization; 2003. [Google Scholar]
  • 2.DiMatteo MR. Variations in patients’ adherence to medical recommendations: A quantitative review of 50 years of research. Med Care. 2004;42(3):200–209. [DOI] [PubMed] [Google Scholar]
  • 3.Haynes RB, McDonald H, Garg AX, Montague P. Interventions for helping patients to follow prescriptions for medications. Cochrane Database Syst Rev. 2002(2):CD000011. [DOI] [PubMed] [Google Scholar]
  • 4.De Bleser L, De Geest S, Vincke B, Ruppar T, Vanhaecke J, Dobbels F. How to test electronic adherence monitoring devices for use in daily life: A conceptual framework. Comput Inform Nurs. 2011;29(9):489–495. [DOI] [PubMed] [Google Scholar]
  • 5.Chan AH, Harrison J, Black PN, Mitchell EA, Foster JM. Using electronic monitoring devices to measure inhaler adherence: A practical guide for clinicians. J Allergy Clin Immunol Pract. 2015;3(3):335–349.e331–335. [DOI] [PubMed] [Google Scholar]

RESOURCES