Feasibility, performance, and acceptability of the Wisebag^™ for potential monitoring of daily gel applicator use in Durban, South Africa

Abstract

The Wisebag™, a lunchbag-style container with an electronic events-monitoring system, was designed as a real-time indirect objective measure of microbicide gel use. Due to cost, alternative functionalities (i.e. use of offline and dummy versions) were explored. We conducted a three-arm, double-blinded pilot study among fifty HIV-negative women in Durban, South Africa to assess participant adherence and Wisebag acceptability and performance. Participants were randomized 2:2:1 to Wisebag with online (events transmitted via cellular signal in real-time), offline (events stored in device memory) or inactive “dummy” devices. Participants were instructed to open the Wisebag daily for two weeks, retrieve a study sticker and affix it on a diary card. All participants completed the study. At exit, 94% did not know which device they had received, nor could they differentiate the Wisebag types when presented with the three options. Five offline devices failed (no data recorded). Per Wisebag events, 26% of women were perfectly adherent compared to 48% by self-report and 46% per diary card. Of reported non-adherence, 92% did not open the Wisebag (travelling or forgot) and 22% opened Wisebag >1x/day (curiosity). Participants liked and were comfortable carrying Wisebag. Successful blinding will allow inclusion of offline and/or dummy Wisebags in future study designs. Perfect adherence by opening events was significantly lower than by self-report, highlighting the importance of objective measures of adherence in clinical trials. Additional studies to validate Wisebag data with actual products, with and without SMS and online functionality, in different populations and settings, and in comparison to biomarkers are warranted.

INTRODUCTION

Accurate measurement of adherence to microbicide use is an ongoing challenge for the conduct and interpretation of clinical trials. Recently, CAPRISA 004, a trial of peri-coital use of tenofovir 1% gel was the first to report a significant protective effect of a topical microbicide [1]. Notably, the effectiveness of the gel was substantially greater among women with >80% adherence. Many other microbicide trials have reported no effect, futility or potential harm; and suboptimal adherence to product use may have contributed to these findings [2–5]. Unfortunately, self-reported adherence is notoriously inflated due to social desirability and recall bias and options to objectively measure microbicide gel use are currently limited [6]. These include applicator insertion tests [7–9], and drug levels in plasma or vaginal fluids, although these have not yet been quantitatively validated [3, 10].

Electronic monitoring, such as the Medication Event Monitoring System (MEMS), has been used successfully to objectively assess HIV treatment adherence to oral medications [6, 11]. One of the key advantages of an electronic Event Monitoring System (EMS) is that it provides detailed individual- level data on patterns of product use. Monitoring the quality of execution of a prescribed regimen (i.e. how well the product is taken in accordance with instructions) is a critical aspect of adherence which is not accurately measureable by self- report, biomarkers (e.g. drug levels), or other objective measures of gel use, such as applicator tests [6, 12]. Until recently, no EMS was available for microbicides. The Wisebag™ is an electronic EMS using wireless technology, and composed of a lunch-bag style container equipped with an internal device that records every lid opening event, thereby providing an indirect measure of gel applicator use. The Wisebag was piloted among 10 participants in the CAPRISA 004 trial, the first application of such a technology to the microbicide context [13]. Recently, the Wisepill, a similar wireless EMS, was piloted to monitor pill-taking for HIV treatment [14].

We assessed adherence to and acceptability of the Wisebag for daily use over a two-week period, in Durban South Africa. One of the challenges of Wisebag, however, is the cost – both the base cost of $200 USD (for the bag and EMS), and recurring monthly cellular costs. In this study, we also tested the feasibility of use, blinding success, and technical performance of three different functionalities of the EMS in the Wisebag, including two different lower cost versions, which could provide important cost saving options in the context of clinical trials.

METHODS

Study participants and location

The study was conducted at the Center for the AIDS Programme of Research in South Africa (CAPRISA). Eligibility criteria included being HIV-negative women, non-pregnant, aged 18–45, having screened-out of the Microbicide Trial Network VOICE trial (http://www.mtnstopshiv.org/studies/70) for administrative (e.g. enrollment targets met, out of screening period) or clinical (e.g. Hepatitis B positive, allergy to latex, anemia) reasons not requiring clinical monitoring or care beyond the scope of this study, and being willing to provide informed consent and participate in study procedures.

Study design and randomization

Wisebag™(Wisepill Technologies Adherence Management Solutions, South Africa) is an EMS composed of a portable, lunch-bag size container with a lid that opens/closes with a zipper (Figure 1). The Wisebag includes a self-contained battery-operated device that records an electronic time and date stamp each time an opening event occurs. The device is also designed to record a daily electronic control signal ( heart beat ) to signify proper battery functioning.

Figure 1.

Wisebag Container and Device

(a) Wisebag (closed) view from the outside; (b) device; (c) device positioned in the Wisebag. The Wisebag has a self-contained battery-operated device with a magnetic switch which is activated by a magnet in the Wisebag lid. When activated, the online device uses the cell phone network to send an electronic medication event record to the Internet Wisepill Server. This happens every time an opening event occurs (when the lid is opened and the magnetic contact is interrupted between the device and the magnet in the lid). For offline devices, each event is electronically stored in the device and can be downloaded onto a computer when it is returned to the clinic.

The device in the Wisebag can be set-up using one of three “modalities” – online, offline, and inactive or “dummy” – all of which were tested here using a randomized design. In the “online” modality, data is transmitted each time an opening event occurs to a central data server in real-time through a wireless cell phone system. Through this modality, it is possible to monitor adherence on an ongoing, real time basis and send reminder messages, although the latter feature was not used in this study. The online Wisebag modality costs $200 USD for the base unit, and $7 per unit per month, in South Africa, to support monthly charges for data monitoring, hosting and servicing using a commercially available network and the Wisepill server. The second modality, “offline,” stores data in the device until the information can be downloaded at the next clinic visit. It is less expensive ($120 USD per unit and no recurring costs) and simpler to set up, however this modality had not been previously tested in other studies.

To simulate the placebo effect of an EMS, inactive “dummy” Wisebags were also dispensed. This modality is the least expensive ($20 USD per unit, no recurring costs), and allows for the provision of similar looking bags to all participants in this or future trials, while collecting data only on a smaller subset. Participants in this pilot study were randomized in a 2:2:1 ratio to receive identical looking Wisebags that contained online, offline, or dummy devices, respectively. This scheme was selected to maximize the number of women with recorded events (equal numbers receiving online and offline), while still allowing for some comparison to a “control” population (dummy).

Study Procedures

Participants were seen for a total of two visits: at enrollment, and at study exit. At enrollment, participants were screened for eligibility, underwent written informed consent, and then completed a demographic questionnaire. Following randomization, participants received a blinded provision of a Wisebag by a pharmacist and then met with a study counselor, who provided them with a set of adhesive stickers and a paper diary card.

To simulate study procedures of daily gel use in a microbicide trial, participants were instructed to store the stickers (and only the stickers) inside the Wisebag, and to open the Wisebag only once per day, and if possible, at approximately the same time each day for the subsequent 13 days. Upon daily openings, participants were to remove one sticker, place it on the diary card (on the corresponding time period depicted for that day) and to close the Wisebag. If participants opened the bag more than once, they were asked not to add another sticker(s) on the diary card. If participants missed a day, they were instructed not to go back and add a sticker to the card for that day.

The Day 1 opening, sticker removal, and diary card placement was done at the clinic, under staff observation. Two weeks later, participants returned to the clinic for a study exit visit. Their Wisebag was returned to the pharmacy, and data from offline devices were downloaded. Participants responded to a quantitative behavioral and acceptability questionnaire, and were interviewed, using a qualitative semi-structured guide to gather more information about their experiences and attitudes towards using the Wisebag and study procedures. Topics discussed in the qualitative interviews included comfort with using and storing the bag, experiences with disclosure, challenges and concerns, unusual opening events, and potential future acceptability. Participants were administered these instruments in their local language (Zulu or English) by a trained staff member and responses were written verbatim or paraphrased (for longer responses) directly on the questionnaire and guide.

Analysis

Results from the structured questionnaires, diary cards and Wisebag data were tabulated, categorical measures were summarized as counts and percentages and continuous measures were summarized as means and median. To assess concordance amongst the various measures, the cumulative number of stickers placed on the diary card was compared with the cumulative number of Wisebag opening event data, and the proportion of participants who matched was calculated. Additionally, day-by-day comparisons were performed to assess the number of days on which participant data matched between diary card and event data. For participants with available self-reported adherence data, diary card data, and opening event data, the degree to which the various measures were associated was examined by calculating a Pearson correlation coefficient for binary variables. Adherence was assessed by calculating the number of days on which event data indicated one opening only, as well as the number of days on which no-openings (underuse) and multiple openings (overuse) were recorded. Variation with regard to time of day of openings throughout a participant’s 13-day course was also examined. All quantitative analyses were conducted in SAS v9.3 (Cary, NC). Qualitative textual data collected through semi-structured interviews were uploaded into NVivo v9.0 (Cambridge, MA) for structural coding by research question. Following coding, reports were run for each code and memos were developed summarizing responses by content and frequency.

RESULTS

Fifty women were enrolled in this study and all completed the two-week study. Participant background characteristics and user experiences did not differ across randomization groups, therefore results are presented for the combined sample (Tables 1–2).Study participants were on average 23.6 years old (range 19–42). Most were Zulu speakers (86%), 80% had completed secondary school or higher education, and 64% had a source of income. Most women (96%) were unmarried, although 92% had a primary partner, and two-thirds had one or more children.

Table 1.

Study Sample Characteristics

Characteristics	N = 50	%
Ethnic group/tribe
Zulu	43	86%
Xhosa	5	10%
Sotho	1	2%
Swati	1	2%
Currently Married
No	48	96%
Has primary sex partner
Yes	46	92%
Education level completed
Primary school, complete	1	2%
Secondary school, not complete	9	18%
Secondary school, complete	38	76%
Attended college or university	2	4%
Has personal income
Yes	32	64%
No	18	36%
Source of income a
Formal employment	4	8%
Self-employment	3	6%
Child social grant	25	50%
From parents	1	2%
Number of live births
None	17	34%
One	20	40%
Two or more	13	26%

^aMore than one may be selected so % does not sum to 64 (i.e. percent reporting a personal income)

Table 2.

User Experiences and Self-Reported Adherence (at Study Exit)

	N=50	Percent
Perfect adherence a	24	48%
Reasons for non-adherence b
I was travelling, out of the area	16	32%
I forgot	16	32%
I was sick or someone in my family was sick and I had to take care of him/her	2	4%
I could not find the bag, it was lost/stolen	1	2%
Reasons for non-protocol specified opening events
NA, the bag was only opened to retrieve stickers	39	78%
Wanted to show the bag to someone	8	16%
Curious/Just looking inside the bag	2	4%
Wanted to check the stickers were inside	1	2%
Experienced the following problems with Wisebag b
Problems opening the bag	9	18%
Problems closing the bag	6	12%
Problems with storage of the bag at your house	5	10%
Problems with others using your bag	5	10%
Problems with Wisebag getting lost, broken or stolen	1	2%
Problems with others gossiping about the bag	1	2%
Problems with stickers getting lost or stolen	-	0%
Ease with remembering to open WB daily
Very difficult or difficult	4	8%
Easy	20	40%
Very Easy	26	52%
Ease with finding time, privacy space to open WB daily
Very difficult or difficult	2	4%
Easy	11	22%
Very Easy	37	74%
Ease with opening the WB at approximately the same time every day
Very difficult or difficult	11	22%
Easy	22	44%
Very Easy	17	34%

^aPerfect adherence was defined as participants for whom the following option was checked to the question on reasons for non-adherence: “Not applicable, I opened the bag every day”

^bMore than one option could be selected. Total does not sum to 100%

Blinding

Blinding of the different Wisebag functionalities was successful: by the end of the study, 94% of participants said they did not know or “just guessed” which type of Wisebag they had received. When presented with three demonstration Wisebags at study exit and asked to designate which one was the dummy bag, 26% answered correctly, which could be expected by chance alone (data not shown).

Technical Performance, Concordance between Measures and Adherence to Study Regimen

Because the Wisebag might be used in hot climates, temperature inside and outside the Wisebag was monitored for 15 days, at temperatures ranging from 17 to 35°C. Temperature inside the Wisebag was not different from that outside the bag; thus there should not be a concern that the temperature of an investigational product would be altered by storage in the Wisebag (data not shown).

Forty women received a Wisebag with an active device: 20 received an online and 20 an offline device. All online devices correctly recorded opening events. Five offline devices failed to record events due to a technical failure (reason for malfunction is unknown), leaving 35 participants with analyzable Wisebag event data. Among functional devices, no difference were found between online and offline device performances, so we combined adherence data for these two groups. Additionally, two women had unusable (incorrectly completed) diary data, leaving 33 participants with analyzable data for concordance and adherence across measures.

On day 1 (enrolment visit), the Wisebag was opened under direct observation at the clinic and at least one Wisebag opening event was recorded for all women. Thereafter, during the 13-days of out-of-clinic unobserved opening period, across all participants, there was a median of 11 days (range 1–13 days) with exactly one opening event per day (range 0–9 openings per day). For each day of use, a mean of 76% of participants opened their bag a single time as instructed; of the remaining women each day, more had under use (15% with no opening) than had over use (9% with >1 opening), but no trend over time was noted (Figure 2). Forty-three percent of participants had perfect concordance on all 13 days, between the number of daily stickers on the diary card and the number of daily Wisebag events.

Figure 2.

Percent of participants with correct (1x/day), under, and over use of Wisebag (N=35).

Adherence

Perfect adherence was defined as opening the Wisebag once a day, removing a sticker and placing it on a diary card, and closing the Wisebag. Several measures of adherence were used and compared: participants self-reported adherence at study exit, diary card completion, and Wisebag opening events. Per electronically-recorded events, 51% were adherent on most days (11+days, or ≥ 85% adherent), while 26% were perfectly adherent to the 1x/day opening instruction, compared to 46% per diary card and 48% per self-report FTFI at study exit. The correlation between these measures was high for the two self-reported methods (rho=0.96, p<.0001), but, although statistically significant, it was only modest (rho~0.55) between Wisebag events and either of the self- reported measures (Table 3). Figure 3 provides an illustrative example of two participants with different patterns of Wisebag openings, highlighting the individual-level variability in the data. Note how opening events immediately after midnight can generate ambiguous results depending on the cut off time chosen for the new day. Only three participants had ambiguous time openings between midnight and 2:59am, so overall, the change in cut off time for the day (from midnight to 3am) did not change the adherence results.

Table 3.

Percent adherence by three different measures and correlation between measures.^a

	Quantitative interview (N=50)	Diary Card (N=48)	Wisebag Events (N=35)
100% Adherence	48%	46%	26%
Correlation b
Diary Card	0.96 p<.0001 (n=48)
Wisebag Events	0.54 p<.0008 (n=35)	0.56 p<.0007 (n=33)

^aNote: Analysis in sub-set of 33 participants who had data for all three measures gave essentially the same results

^bPearson correlation coefficients and associated p-values

Figure 3.

Example of two participants with different patterns of Wisebag openings.

Participants in panels (A) and (B) had opening gaps and >1 opening on one day, but showcase different patterns of opening. Participant (A) opened the WB approximately at the same time each day (10pm +/− 2 hours) while participant (B) opened the WB more or less across the entire range of a 24 hour-day. Note how right-after midnight opening events can generate ambiguous results depending on the cut off time chosen for the new day. While participant (A) did not have an opening on day 6, her first opening was just after midnight on day 7, within 2 hours of her typical opening time. When the cut off time for day change was shifted from midnight to 3am, this participant no longer had a missing opening event on day 6. With a later cut off, the first opening for participant (B) on day 1, would be shifted to a last opening on day 0 (enrolment day) and she would only have 1 opening on day 1. Only 3 participants had ambiguous time openings between midnight and 2:59am, so overall, the change in cut off time for the day (from midnight to 3am) does not change the adherence results.

We also asked participants about their perceptions of the ease and difficulty with adherence and reasons for non-adherence. The majority of participants (52%) reported it was very easy to remember to open the Wisebag every day, and only four (8%) said it was difficult or very difficult. Similarly, 74% said it was very easy to find time and space to open the Wisebag every day; however, only a third (34%) thought it was very easy to open the bag at approximately the same time every day, and 22% said it was difficult or very difficult (Table 2).

Qualitative responses from participants provided additional insight into the strategies participants employed to make Wisebag use easier. The most common strategy was the use of a reminder system: over half of participants used an alarm – primarily a cell phone alarm. Five participants described using a television show, and six individuals said that a friend or family member reminded them, one of whom specified she used this method due to her lack of a cell phone. A handful of participants each said that the visibility of where they kept the Wisebag or that liking the bag helped them remember.

Despite this, 52% reported instances of not opening the Wisebag, most often because they forgot and/or were travelling (Table 2). In the qualitative interviews, these reasons were corroborated, along with other disruptive situations (e.g. when family members were sick or taken to the hospital, when taking school exams). Yet, over half of the women felt there was nothing specific that would have made it easier for them to remember to open the Wisebag every day. Indeed, while women provided several suggestions for reminder mechanisms, these generally reflected the strategies already being employed by participants during the study. One participant offered the novel suggestion of having the bag make a noise (“beep”) as a reminder.

Approximately one-fifth reported opening the bag more than once on a given day, most often to show it to a friend or relative (Table 2). Qualitative responses also reflected this desire to show a family member, friend, or other person. One woman said that a security guard at a shop asked her to open the bag and another participant asked her sister to open her bag in her absence, suggesting other potential confounders to accurate opening data.

User Experiences

Storing and accessing the Wisebag was generally not a problem for participants: 88% never had a problem with storage and 98% had no problems with accessing it from its storage location. Qualitative interviews provided additional insight into storage locations and their visibility: twenty-two participants described storing the Wisebag out-of-sight, either underneath or inside a piece of furniture (e.g. cupboard, wardrobe, or suitcase), while twenty-six participants described storing the device in-sight (e.g. on top of their wardrobe, a table or cupboard). A minority of women reported problems with others using their Wisebag (10%), or having trouble opening (18%) or closing (12%) the bag s zipper (see Table 2). One bag was stolen while a participant was travelling on public transport. Twelve percent of women reported storing items other than the stickers (e.g. study provided diary card) in their Wisebag. In the qualitative interviews, one participant described her sister s use of the bag to store her cell phone charger.

Acceptability and Willingness to Use the Wisebag in the Future

Most (94%) participants liked using the Wisebag. Only one participant disliked it, which was due to the discomfort of carrying the Wisebag along with her handbag. Most were very comfortable (88%) or comfortable (10%) with others seeing them carrying the Wisebag and all were very/comfortable with the idea of using Wisebag in the future for gel applicators. This high level of comfort was echoed in the qualitative interviews, where 41 participants explained that others most likely thought that the Wisebag was a lunchbox. Only one participant reported the perception that others “thought I was carrying [HIV] treatment because I was coming from [a] clinic.”

Almost all (94%) liked the way the Wisebag looked overall, and the majority of participants found its color, shape, size, zipper, and strap acceptable (range 80% – 98%). Qualitative discussions confirmed that few had concerns related to physical look of the Wisebag. Only three participants described being physically uncomfortable carrying the bag. One of them explained that “it was uncomfortable because of the positioning of the belt and the weight of the device makes the bag unsteady.” A few others discussed some discomfort related to carrying two bags or having the Wisebag inside of a personal purse.

Reasons as to why women felt the Wisebag would be acceptable to others were provided during the qualitative interviews. Five women said that the bag would protect the study products and eleven said that what would be enticing is that the bag did not allow others to see what was inside, with some specifying that the bag therefore would hide the study products from view. Other attributes of the bag that women thought would be appealing included: its attractiveness, its small or “right” size, its portability, and its ease of use. A few women felt that others would be motivated to use the bag (and/or product) because it would record their openings. Five women said that other women had expressed some interest in the bag already, which indicated to them a willingness to use it. One participant said “VOICE participants in the waiting room commented that they preferred these bags to the one that they are getting (in the main trial).”

DISCUSSION

In this two-week pilot study conducted in Durban, South Africa, among healthy, HIV-negative women who had been screened-out from the VOICE trial, the Wisebag was well received. The blinding of the online, offline and dummy Wisebags was successful, showing that it would be feasible to deploy electronic monitoring at lower cost in large trials. The offline device, is technologically similar to MEMS, and is appropriate for events monitoring on a monthly basis, if reminder SMS or phone calls are not needed. The dummy Wisebag would preserve the placebo effect of receiving a bag, while lowering cost, if adherence in a random subset of participants at a site is sufficient to monitor overall adherence. While initial cost is high for the online Wisebag, it allows for real-time follow up with non-adherent participants [13],[14]. The cost of the offline Wisebag is similar to the MEMS monitor (AARDEX Group Ltd). Notably, Wisebags can be recycled and re-used, allowing researchers to amortize the cost over several sets of participants.

During the short study period, the Wisebag was deemed easy to use and acceptable. While provision of study-products in a novel container could hypothetically introduce stigma, instigate rumors, or generate other user-related problems that may affect its acceptability or feasibility of use in research or real-world settings, this was generally not the case here. No Wisebag-associated social harms, defined as non-clinical adverse consequences such as psychological or emotional problems associated with study participation, were reported, and only one participant who reported neighbors gossip that associated the bag with receipt of antiretrovirals from an HIV clinic, thereby implying an HIV-positive status. However, for most, its resemblance to a lunch bag probably provided a neutral appearance, which may actually reduce perceived research study associated-stigma as it was found to be potentially useful in disguising study product.

While adherence was not high across any of the study measures, it was much lower by Wisebag opening events compared to self- report at exit or diary card completion. One reason for the low adherence may be that the Wisebag made the simple task of peeling off stickers to affix on a paper diary more difficult to accomplish. This may not be the case for gel applicators, which are stored in and must be retrieved from a box and/or a bag. Social desirability bias, which typically inflates self-reported adherence [5, 11], may have influenced participants responses, even though no study products was used in this study. Hence, the correlation and concordance between opening events and other adherence measures was not high. This is not unexpected, as correlation between electronic monitoring and other adherence measures tend to be in the moderate range [15, 16]. Additional studies to validate Wisebag data compared to other objective measures of product use are warranted, to determine how useful Wisebag can be for monitoring gel applicator adherence in clinical trials.

The most common reason for not opening the Wisebag daily was forgetfulness. Despite its higher cost, this supports the use of the online version of Wisebag, which includes SMS reminders as an adherence support intervention. Travelling was another common reason for not opening the Wisebag. While meant to be portable, the Wisebag and its complete contents may not be perceived as practical or appropriate to carry around for trips of short duration. Women may still have strategies to take products while travelling (e.g. “decanting” the required doses into another container), but removing more than one product at a time from Wisebag would affect the accurate interpretation of opening events as a measure of gel insertion. “Curiosity events,” unintended use of the Wisebag and other causes of extra openings, although less frequent were described as well. These too may yield biased estimates of product use with EMS. For MEMS, it’s been well documented that MEMS tend to underestimate pill use, due to “pocket dosing” (taking several pills out of the container at a time)[11]. Some investigators have suggested creating composite adherence scores, which perform better at predicting virological outcomes in HIV treatment contexts, than any single measure [15]. The composite score requires collecting questionnaire data on behaviors such as pocket dosing, curiosity events or unintended uses of the container, which can be incorporated through an algorithm to adjust the electronic data for these situations. A drawback of a composite score, however, is that it is a combination of subjective and objective measures, thus diluting a major benefit of objective adherence measurement, Other researchers have developed algorithms to specifically correct for curiosity events when using electronic data [14]. While either of these approaches may need further refinement and validation, specifically in the prevention context, they too could be applied to data collected with the Wisebag.

From a technical performance standpoint, there were several problems with the offline devices used in this study. These were initial prototypes, and since then, the technical problems we identified were addressed by the manufacturer. There were no problems with temperature control, indicating that the Wisebag would not affect the stability of investigational products in a variety of settings, including those with tropical conditions.

This study has several important limitations: it was very short in duration, it did not use a study product, and there were no other objective measures of adherence to validate the Wisebag event data. Despite these limitations, the study also has several strengths. For one, it included participants from VOICE screen-outs, a population that is very similar to a clinical trial population where a Wisebag would be used. Additionally, the study tested daily use, which had not previously been done, as well as different functionalities of the Wisebag, which could increase the options available to investigators while designing a trial.

In conclusion, the Wisebag seems a promising, feasible, and acceptable technology to implement in the context of microbicide trial to electronically monitor product use. More studies are needed to validate this approach against objective markers of product use, and to evaluate the added benefit of the SMS component in the online system, to serve as a reminder to support high product use.