A scoping review of studies comparing the medication event monitoring system (MEMS) with alternative methods for measuring medication adherence

Abstract

Different methods are available for measuring medication adherence. In this paper, we conducted a scoping review to identify and summarize evidence of all studies comparing the Medication Event Monitoring System (MEMS) with alternative methods for measuring medication adherence. A literature search was performed using the open database www.iAdherence.org that includes all original studies reporting findings from the MEMS. Papers comparing methods for measuring adherence to solid oral formulations were included. Data was extracted using a standardized extraction table. A total of 117 articles fulfilled the inclusion criteria, including 251 comparisons. Most frequent comparisons were against self‐report (n = 119) and pill count (n = 59). Similar outcome measures were used in 210 comparisons (84%), among which 78 used dichotomous variables (adherent or not) and 132 used continuous measures (adherence expressed as percentage). Furthermore, 32% of all comparisons did not estimate adherence over the same coverage period and 44% of all comparisons did not use a statistical method or used a suboptimal one. Only eighty‐seven (35%) comparisons had similar coverage periods, similar outcome measures and optimal statistical methods. Compared to MEMS, median adherence was grossly overestimated by 17% using self‐report, by 8% using pill count and by 6% using rating. In conclusion, among all comparisons of MEMS versus alternative methods for measuring adherence, only a few used adequate comparisons in terms of outcome measures, coverage periods and statistical method. Researchers should therefore use stronger methodological frameworks when comparing measurement methods and be aware that non‐electronic measures could lead to overestimation of medication adherence.

Introduction

Poor medication adherence has grown as a major health concern over the last decade. It is of rising concern to clinicians, healthcare systems and other stakeholders because of the increasing evidence that poor medication adherence is prevalent and associated with adverse outcomes and higher costs of care 1. The ABC taxonomy 2 defines medication adherence as the process by which patients take their medication as prescribed. This process of taking medication begins with the initiation of therapy, when patients take their first dose of a prescribed medicine. Initiation precedes the implementation phase, defined as the extent to which patients' actual dosing corresponds with their prescribed dosing regimen. Finally, discontinuation eventually occurs, which refers to the end of therapy.

Unbiased and precise measurement of the three elements of medication adherence (initiation, implementation and persistence) is the cornerstone for sound interpretation of clinical trials 3 as well as for the implementation of successful interventions to enhance medication adherence in medical practice 4. Measurement methods that rely on patients' recall and/or allow patients to censor information on their dosing histories (e.g. pill counts or self‐report) have repeatedly been discredited, as they are strongly biased toward overestimation of drug actual exposure 5.

A Medication Event Monitoring System (MEMS) consists in electronic detection of package entry by incorporating micro‐circuitry into pharmaceutical packages of various design, which detects, time‐stamps and stores the manoeuvres needed to remove a dose of the drug. This automatic compilation of times of medication intake (dosing history) provides a thorough characterization of medication adherence, with clear distinctions between initiation, implementation and discontinuation 3. It is, of course, an indirect method of estimating when and how much drug is administered, but it has been shown to predict well drug concentration in plasma 3. Electronic monitoring of package entry is the current gold standard for automatically compiling drug dosing history in trial settings 3, as illustrated by more than 700 peer‐reviewed publications (www.iAdherence.org).

Numerous studies have compared alternative methods against electronic monitoring. However, studies comparing methods of measuring adherence against each other often oversimplify the comparison and do not consider the richness of the dosing history data derived from electronic monitoring. Vrijens and Urquhart 6 introduced two attributes, reliability and richness of the measurement method. Reliability refers to the degree to which a method is biased, and richness refers to the precision of assessments over time. The less biased and the more precise the assessment, the better the method of measurement is able to distinguish and characterize the three elements of medication adherence 6. A comparison along these two attributes can offer a good understanding of the strengths and limitations of different methods for measuring adherence.

Given the numerous studies comparing multiple methods of measuring adherence, this review was designed to scope and synthesize the evidence of studies comparing MEMS with alternative non‐electronic methods, in order to see whether the comparisons made are optimal in terms of their outcome measure, precision and statistical method.

Method

Literature search

The iAdherence database (www.iadherence.org) is a repository, which contains all peer‐reviewed papers reporting studies where dosing histories are electronically compiled using MEMS. The database is constructed from a monthly search of company records and MEDLINE using a systematic approach described in the Appendix. Identified papers are then screened to confirm the use of MEMS and re‐confirmed using company records. The MEMS‐based papers are systematically filed in Reference Manager^® and potential duplicates are automatically removed.

As the objective of iAdherence is to provide a comprehensive list of all MEMS‐based peer‐reviewed papers, this database was used to identify all studies comparing MEMS with non‐electronic methods.

The literature search was conducted by one researcher (JD), according to a prespecified research protocol and using the iAdherence database. All abstracts available in the iAdherence database were systematically searched for the following terms: ‘self‐report OR self report OR questionnaire OR pill count OR refill OR diar* OR claims OR concentration OR compar* OR pharmacokinet* OR measur* OR valid*’. All articles published in the English language from January 1989 to 31 March 2015 were searched.

Selection of studies

All original studies that compared, within a same sample, MEMS with at least one other non‐electronic method for measuring adherence to solid oral formulations were included. The selection of studies took place in three stages. The first selection of studies based on title and abstract was performed by one researcher (ME). Some articles were discussed further during a meeting with three co‐authors (MH, BV, JD), which resulted in a second selection of studies based on title and abstract. In a third step, assessment of the full text of articles was performed by at least two independent reviewers (ME, MH or JD). In case of disagreement, consensus was reached after discussion during a face‐to‐face meeting (ME, MH, JD, BV).

Data extraction

Different characteristics were extracted from each publication, such as geographical area, year of publication and journal. Furthermore, study design, length of patient follow‐up period, sample size, therapeutic area and dosing regimen were recorded. For categorization of the therapeutic area, the classification of diseases according to the British National Formulary 7 was used. Dosing regimen was reported as once daily (OD), twice daily (BID), three times daily (TID), combined (multiple medications with different dosing regimens) and other (more than three times daily).

The measurement methods compared against MEMS were categorized as self‐report, pill count, rating by healthcare provider/caregiver/parent, chemical markers (pharmacokinetic or pharmacodynamics), pharmacy records, diaries and composite adherence score. All types of questionnaires or interviews were characterized as self‐report. The composite adherence score is a method that combines values of MEMS, pill count and interview 8.

For comparison of measurement methods and to check for reliability and precision of the measurement methods, five types of data were extracted:

1. The type of outcome derived from the measurement method of adherence could be expressed as a continuous outcome (i.e. a percentage), a dichotomous outcome (i.e. adherent or not adherent), or another type of outcome, which was reported as other.

2. The precision of each measure was extracted as the coverage period used to estimate the adherence outcome. This coverage period could be different for every method used. For instance, the coverage period for chemical markers is typically subject to the half‐life of the drug (e.g. 6 hours) while pill count could be performed over different periods of time (e.g. 30, 60 or 90 days).

3. The adherence estimate of each adherence assessment method was extracted. In studies where adherence was considered as a continuous outcome (i.e. the proportion of prescribed doses taken or the proportion of days with correct intake), the overall average percentage of adherence was extracted. In studies where adherence was considered as a dichotomous outcome (adherent vs. non‐adherent patients based on an arbitrary threshold), the proportion of adherent patients was retrieved. In the latter situation, the most frequent threshold used to dichotomize medication adherence was 80%, but other thresholds could be used.

4. The statistical methods used to compare the outcomes were extracted.

5. The results of each statistical method including their corresponding P‐values were extracted, if reported.

   Data were extracted using a standardized extraction table. The process of data extraction was carried out by one researcher (ME). A random selection of 10% of the articles was used to confirm the quality of data extraction by JD.

Analysis

In this phase of the review, compatible and optimal comparisons of measurement methods were identified. A comparison is defined as compatible if MEMS and the other non‐electronic methods used similar adherence outcomes and similar coverage periods, and if an optimal statistical method was used to assess the agreement between the two measurement methods. First, comparisons between MEMS and other non‐electronic methods were checked to determine whether a similar outcome was reached. Comparisons using continuous variables vs. continuous variables (%‐%) and comparisons using dichotomous variables versus dichotomous variables (y/n‐y/n) were considered similar outcomes. Other comparisons with different outcomes were reported as other. Second, each comparison was checked to see whether the adherence outcome was assessed over the same coverage period. Third, the comparisons that used similar outcomes and the same coverage periods were checked to see whether or not a statistical method was used, and, if one was used, to see whether the method was optimal or not to assess the agreement between measurement methods. Statistical methods used to assess the agreement between continuous outcomes were considered to be optimal when they evaluated the absolute difference between each pair of the two measurements 9 or when a formal measure of consistency was used 10. In those approaches, the null hypothesis that is tested is that there is an agreement between the two measures (difference = 0). Therefore, a P‐value below 0.05 indicates a significant inconsistency between the two methods being compared. Pearson and Spearman correlation coefficients were considered to be suboptimal statistical approaches, as they consider only relative position and the null hypothesis tested is the absence of linear association (rho = 0). A rejection of the null (P < 0.05) is an indication of some form of linear association between the two measures, but is less relevant to the question of agreement between the two methods 11.

Statistical methods that were considered optimal for evaluating the consistency between measurement methods when assessed as a dichotomous variable were Kappa/McNemar's test, as well as an approach based on sensitivity and specificity of the measurement methods. The null hypothesis tested with a Kappa or a McNemar's test is that there is an agreement between the two measures. A rejection of the null hypothesis (P < 0.05) will thus conclude that there is a significant disagreement between the methods. In calculating sensitivity and specificity, the optimal cut point for dichotomization is often identified using receiver operating characteristic (ROC) curves 12 and no formal hypothesis is tested (no P‐value is reported).

This screening resulted in a number of comparisons that were compatible in terms of similar outcomes and coverage periods and had optimal statistical methods. Compatible and optimal comparisons were further checked on significance and were classified as significant (P < 0.05), not significant or not reported (no P‐value). This classification was performed for both continuous outcomes and dichotomous outcomes.

Finally, the median adherence outcome was estimated for MEMS and each alternative non‐electronic method separately. To illustrate its reliability in comparison with MEMS, the difference between the most frequent methods was estimated. This estimation was performed based on compatible and optimal comparisons only.

Results

Study selection process

Figure 1 shows the flow chart of the studies' identification and selection, listing explicit reasons for exclusion. After the first screening of abstract and title, 136 potential eligible papers were identified.

Figure 1.

Flowchart describing the literature search

Uncertainties among the 136 papers were discussed during the second selection among three co‐authors, and nine additional studies were excluded. Of the 127 studies eligible for full text screening, ten additional articles were excluded, resulting in 117 articles included in the analysis.

Overview of included studies

The characteristics of the studies included are reported in Table 1. Sixty‐three (54%) studies were conducted in North America 8, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74 and 35 (30%) in Europe 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109. A few studies were conducted in Africa (n = 10) 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, Asia (n = 6) 5, 120, 121, 122, 123, 124, South America (n = 2) 125, 126 or Australia (n = 1) 127. Most of the studies compared one non‐electronic measurement method with the MEMS (n = 47), while 34 studies compared two measurement methods with the MEMS, 19 studies included three comparisons and 17 studies four or more comparisons.

Table 1.

Study characteristics

Variable	n (%)
Geographical area
North America	63 (54)
Europe	35 (30)
Africa	10 (9)
Asia	6 (5)
South America	2 (1)
Australia	1 (1)
Therapeutic area
Infections	44 (38)
Cardio‐vascular system	23 (20)
Central nervous system	18 (15)
Nutrition and blood	13 (11)
Malignant diseases and immunosuppression	7 (6)
Other	12 (10)
Publication year
1989–1994	8 (7)
1995–1999	17 (14)
2000–2004	25 (21)
2005–2009	37 (32)
>2010	30 (26)
Study design
Cohort	91 (78)
Randomized controlled trial from which:	26 (22)
Intervention	11 (9)
Treatment	15 (13)
Dosing regimen
Combined	80 (68)
OD	17 (15)
BID	12 (10)
TID	5 (4)
Other	3 (3)
Sample size
0–50	48 (41)
51–100	30 (26)
101–200	22 (19)
>201	17 (14)
Follow‐up period
<1 month	6 (5)
1–5 months	62 (53)
6–11 months	25 (21)
>12 months	21 (18)
Not reported	3 (3)
Comparisons
1	47 (40)
2	34 (29)
3	19 (16)
>4	17 (15)

OD, once daily; BID, twice daily; TID, three times daily.

Analysis of extracted data

The 117 articles included resulted in 251 comparisons between MEMS and alternative non‐electronic measurement methods. Table 2 presents the outcome measures and the corresponding statistical method used to assess the comparison. A total of 44% (n = 110) of all comparisons did not specify a statistical method at all or used a suboptimal method.

Table 2.

Statistical methods used to compare adherence methods

Outcome	Statistical method	n (%)
% ‐ % (132)	Absolute difference	46 (35)
	Consistency	2 (2)
	Correlation	53 (40)
	No	31 (23)
Y/n ‐ y/n (78)	Kappa/Mcnemar's test	38 (48)
	Sensitivity & specificity	27 (35)
	Correlation	4 (5)
	Absolute difference	3 (4)
	No	6 (8)
Other (41)	Yes	28 (68)
	No	13 (32)

Statistical method in bold are considered as optimal for continuous and dichotomous comparisons

Figure 2 presents a scatter plot of the coverage periods when adherence outcome was summarized from the MEMS or from an alternative non‐electronic method. The median of the coverage period for both MEMS and other non‐electronic methods was 30 days. Figure 2 highlights that 68% (n = 170) of the comparisons are on the diagonal line of the scatter plot, indicating that they used a similar coverage period.

Figure 2.

Scatter plot of coverage periods for all measurement methods; 68% (n = 170) of comparisons are on diagonal. Points below the diagonal, 21% (n = 53), are comparisons in which MEMS has wider coverage period than other measurement methods. Points above the diagonal, 7% (n = 18), are comparisons in which other measurement methods have a wider coverage period than MEMS

Among the 251 comparisons, 132 (52%) summarized medication adherence as a continuous outcome (%‐%) for both the MEMS and its comparator (see Figure 3). Of those 132 comparisons, only 41 (31%) used an optimal statistical method and a compatible coverage period, of which 37 reported a P‐value. The null hypothesis of agreement between the measurement methods was rejected (P < 0.05) in 28 cases (76%) 15, 16, 26, 44, 45, 46, 48, 52, 56, 57, 60, 66, 71, 89, 93, 101, 102, 103, 115, 119, leading to the conclusion that there is a significant disagreement between the adherence measurement methods.

Figure 3.

Flowchart of the evaluation of methodology of comparisons. Comparisons using compatible coverage periods and optimal statistical methods are shown in bold

Of note, when a continuous outcome was used, a correlation approach was considered in 36 comparisons to assess agreement between measurement methods leading to an average correlation of 0.35, ranging from −0.67 to 0.76 5, 8, 16, 19, 24, 29, 33, 34, 41, 43, 55, 62, 63, 65, 67, 69, 86, 88, 90, 91. For 31 of those correlations, the null hypothesis of no correlation was formally tested and rejected (P < 0.05) in 24 cases (77%) leading to the conclusion that there is a significant linear association between the measurement methods. This conclusion, based on correlation, is, however, less relevant to the question of agreement.

Among the 251 comparisons, 78 (31%) summarized medication adherence as a binary outcome (y/n‐y/n) for both the MEMS and its comparator (see Figure 3). Of those 78 comparisons, 46 (59%) used an optimal statistical method and a compatible coverage period. Fifteen of those comparisons used a statistical approach based on sensitivity and specificity 39, 92, 94, 107, 113, 117, 122. Taking MEMS as the reference method, the mean specificity was 65% and the mean sensitivity was 55%, leading to a mean positive predictive value of 35% and a mean negative predictive value of 78%.

Among the 46 comparisons that used an optimal statistical method over a compatible coverage period, 20 formally tested the null hypothesis of agreement between the measurement methods. The null was rejected (P < 0.05) in 18 cases (90%) 17, 25, 32, 64, 68, 76, 77, 89, 104, 118, leading to the conclusion that there is a significant disagreement between the measurement methods.

Taking together the optimal comparisons from both groups, continuous and dichotomous, the most frequently used methods were self‐report (n = 38), pill count (n = 27) and rating (n = 12). Sample sizes for these comparisons ranged from 7 to 669 with an average of 92 participants.

Besides assessing the appropriateness of the comparison and formally testing for agreement between the measurement methods, Figure 4 highlights the distribution of the difference in adherence estimates between MEMS and the comparison measurement method for the three most frequent methods that used compatible coverage periods and optimal statistical methods. When compared to MEMS, the median adherence per method was overestimated by 17% [range: −21%, 75%] for self‐report, 8% [−25%, 50%] for pill count and 6% [−15%, 50%] for rating.

Figure 4.

The extent to which other methods overestimate adherence compared with MEMS. The white line in the middle of the box is the median. The lower and upper bounds represent the 25th and 75th percentile of the distribution. The ends of the whiskers represent the minimum and maximum (n = number of comparisons)

Discussion

Of the 117 studies that reported 251 comparisons between MEMS and non‐electronic measurement methods for medication adherence, only a small percentage of comparisons (35%) used similar outcomes, similar coverage periods and optimal statistical methods. Of these compatible and optimal comparisons, 57 comparisons formally tested the null hypothesis of agreement between the measurement methods. The null hypothesis was rejected (P < 0.05) in 46 comparisons (81%), leading to the conclusion that there is a significant disagreement between adherence measures when assessed using MEMS vs. a non‐electronic method. Self‐report, rating by others and pill count tend to overestimate adherence compared to MEMS. For other methods, chemical markers, pharmacy refill, diaries or composite adherence score, there were too few comparisons to derive a formal conclusion.

When a specificity/sensitivity approach was used to evaluate how well an alternative measurement predicts the binary MEMS‐based classification as adherent versus non‐adherent, the specificity and sensitivity of such a test are respectively 65% and 55%, leading to poor positive predictive value (35%) and negative predictive value (78%).

Finally, it is interesting to see that a correlation, which is a measure of linear association rather than a measure of agreement, was used in 36 of the comparisons.

During the process of conducting this scoping review, some limitations could be identified. First, the analysis focused only on continuous and dichotomous outcomes (84% of all comparisons). Other outcomes (e.g. longitudinal) were not considered, as more advanced and unique statistical analysis was used. Those more advanced methods, while rarely used, are, however, required to better capture the dynamic and complex behaviour of medication adherence and should become the focus of further investigations. Second, the iAdherence database (www.iAdherence.org) includes only studies using MEMS and therefore other electronic methods for measuring medication adherence were not considered. In addition, studies comparing methods for measuring medication adherence using inhalation drugs, eye drops or topical medication were not considered in this review. Solid oral formulations (tablets and capsules) remain, however, the most widely used formulations of drugs used in ambulatory medical care. Third, as the main focus of this study was to evaluate the methodology used to compare MEMS against non‐electronic methods, the quality of the studies was not evaluated. Finally, the iAdherence database includes only peer‐reviewed journals, which could result in publication bias, as the grey literature was not searched.

While there are some limitations, this scoping review gives a comprehensive overview of the methods used in the peer‐reviewed literature to compare adherence measurement methods. The findings highlight major limitations and pitfalls in the methodological approaches used to compare and validate adherence measurement methods.

None of the comparisons addressed the three fundamental elements of medication adherence: initiation, implementation and persistence. Therefore, the average adherence values reported in those papers may be misleading as they may include treatment discontinuation, which is a time to event outcome.

Further research should use adequate approaches (including optimal statistical methods) in order to adequately compare methods for measuring medication adherence. Furthermore, researchers should be aware of the possible overestimation of medication adherence using certain non‐electronic methods (self‐report, rating and pill count), and should better define the conceptual framework to compare measurement methods. The different measurement methods need to be evaluated against their ability to measure the different elements of medication adherence 128.

Conclusion

Results from this review showed that among comparisons between MEMS and non‐electronic methods, only a few used similar outcomes, similar coverage periods and optimal statistical methods. From the comparisons of measurement methods that were compatible and optimal, about 80% showed a significant difference between MEMS and non‐electronic methods. Compared to MEMS, median adherence was overestimated by 17% using self‐report, by 8% using pill count and by 6% using rating. Those differences suggest that non‐electronic measures could lead to overestimation of medication adherence.

Competing Interests

All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organization for the submitted work; BV and JD had a specified relationship with WestRock Healthcare in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work.

Contributors

ME: study rationale and design, literature selection, data extraction, interpretation and reflection, writing the manuscript. BV: study rationale and design, literature selection, interpretation and reflection, reviewing the manuscript. JD: literature search, literature selection, interpretation and reflection, reviewing the manuscript. SE: interpretation and reflection, reviewing the manuscript. MH: study rationale and design, literature selection, interpretation and reflection, reviewing the manuscript.

Systematic approach of literature search

• ((electronic OR microelectronic or eDEM) AND monitoring) OR ((mems AND (electronic OR medication)) AND (“patient compliance”[MESH] OR treatment refusal “[MESH] OR “patient drop outs mesh))

• (compliance OR execution OR persistence) AND ((electronic OR microelectronic) monitoring)

• (“Patient compliance”[MESH] OR “treatment refusal”[MESH] OR “Patient Dropouts”[Mesh]) OR ((mems OR electronic OR microelectronic) AND monitoring)

• (“Patient Compliance”[Mesh]) AND “electronic”

• (“Patient Compliance”[Mesh]) OR (Treatment refusal[Mesh])

• “Patient Compliance”[Mesh] AND MEMS

El Alili, M. , Vrijens, B. , Demonceau, J. , Evers, S. M. , and Hiligsmann, M. (2016) A scoping review of studies comparing the medication event monitoring system (MEMS) with alternative methods for measuring medication adherence. Br J Clin Pharmacol, 82: 268–279. doi: 10.1111/bcp.12942.