Challenges and Solutions Implementing an SMS Text Message-based Survey CASI and Adherence Reminders in an International Biomedical HIV PrEP Study (MTN 017)

Abstract

Background

We implemented a text message-based Short Message Service computer-assisted self-interviewing (SMS-CASI) system to aid adherence and monitor behavior in MTN-017, a phase 2 safety and acceptability study of rectally-applied reduced-glycerin 1% tenofovir gel compared to oral emtricitabine/tenofovir disoproxil fumarate tablets. We sought to implement SMS-based daily reminders and product use reporting, in four countries and five languages, and centralize data management/automated-backup.

Methods

We assessed features of five SMS programs against study criteria. After identifying the optimal program, we systematically implemented it in South Africa, Thailand, Peru, and the United States. The system consisted of four windows-based computers, a GSM dongle and sim card to send SMS. The SMS-CASI was, designed for 160 character SMS. Reminders and reporting sessions were initiated by date/time triggered messages. System, questions, responses, and instructions were triggered by predetermined key words.

Results

There were 142,177 total messages: sent 86,349 (60.73%), received 55,573 (39.09%), failed 255 (0.18%). 6,153 (4.33%) of the message were errors generated from either our SMS-CASI system or by participants. Implementation challenges included: high message costs; poor data access; slow data cleaning and analysis; difficulty reporting information to sites; a need for better participant privacy and data security; and mitigating variability in system performance across sites. We mitigated message costs and poor data access by federating the SMS-CASI system, and used secure email protocols to centralize data backup. We developed programming syntaxes to facilitate daily data cleaning and analysis, and a calendar template for reporting SMS behavior. Lastly, we ambiguated text message language to increase privacy, and standardized hardware and software across sites, minimizing operational variability.

Conclusion

We identified factors that aid international implementation and operation of SMS-CASI for real-time adherence monitoring. The challenges and solutions we present can aid other researchers to develop and manage an international multilingual SMS-based adherence reminder and CASI system.

Graphical abstract

1. INTRODUCTION

Adherence to product use in microbicide and other biomedical intervention HIV prevention trials is critical to measuring the product’s safety and efficacy.[1–3] The outcome of clinical trials is closely related to participant adherence to the study’s biomedical and behavioral protocols;[4] low patient adherence with prescribed treatments is a very common problem that can seriously distort the validity and generalizability of outcomes. Variations in adherence to biomedical protocols can lead to diverse statistical outcomes and conclusions.[4,5] In contrast, when adherence is consistently high, studies can recruit, enroll, and follow fewer people, which decreases costs.[2,5,6] Better adherence also increases statistical certainty of results.[7] However, since product adherence is often participant-dependent, it is difficult to ensure that proper dosing, administration, and product use are taking place. Therefore, real-time monitoring and adherence improvement mechanisms help to guarantee that trials proceed successfully.[7]

Short message service (SMS) or text messaging has been used to collect real-time data and to monitor adherence.[3,8,9] Collecting data in real-time mitigates recall-bias and allows researchers to address many data collection, analytic, or behavioral (non-response or non-adherence) issues early.[3,8] Researchers also have a greater number of opportunities to identify and mitigate many adverse experiences during the course of product use.[3,10] For instance, Coomes et al. reviewed the literature on people living with HIV, use of SMS, and medication adherence and found that SMS-based interventions that incorporate the elements of interactivity, timing, and tailoring of messages could be implemented to encourage greater medication adherence.[11] Moreover, SMS has been shown to impact other mutually reinforcing behaviors and factors (e.g., increasing patient involvement and social support, reducing risk behaviors, and promoting general health and well-being) that support better healthcare quality and clinical outcomes.[11–13] Additional studies have evaluated the use of SMS in medical settings and have found a positive effect on adherence.[10,11,14]

As mobile phone technology increases in popularity and surpasses 6.8 billion users worldwide, its varied implementation in research has grown.[3,15–17] Many studies on the ubiquity of SMS have shown that SMS allows researchers to provide behavioral reminders and collect adherence data remotely and in real-time, with low levels of intrusiveness, and at a low-cost, respective to other modalities.[11,15,16,18]

This paper describes the challenges and solutions to implementing an international SMS-based adherence reminder and computer-assisted self-interview (CASI) system in an HIV prevention trial, MTN-017. We describe a “system specification assessment” of several SMS text messaging applications, detail critical challenges to developing and managing an international SMS-based CASI system and explain how challenges were resolved. In MTN-017, SMS was used for real-time data collection and to send reminder messages to support behavioral adherence to the study’s three biomedical regimens (daily oral tablet, daily rectal gel, receptive anal intercourse [RAI] associated rectal gel). Lastly, we will determine the feasibility of developing and managing an international multilingual SMS-based CASI and adherence reminder system.

2. MATERIALS AND METHODS

2.1 Study design

The Microbicide Trials Network (MTN), funded by the United States National Institutes of Health, is a worldwide collaborative clinical trials network focused on developing methods for prevention of the sexual transmission of HIV. MTN-017 was a phase 2 trial designed to evaluate the safety and acceptability of a reduced glycerin formulation of 1% tenofovir gel used daily (i.e., daily rectal regimen) or used before and after sex (i.e., RAI rectal regimen), as well as daily use of the oral tablet of emtricitabine/tenofovir disoproxil fumarate (FTC/TDF, i.e., daily oral regimen) in HIV-negative transgender women and men who have sex with men (MSM). Participants (N=187) followed each of the three study regimens for eight weeks in a crossover design. Each washout period was one week long (see Figure 1).[19] Study protocols and participant samples are detailed in two other publications.[20,21]

Figure 1. MTN-017 study design.

PERIOD 1 - PERIOD 3 [Pd1-Pd3]: Are the different treatment crossover periods for each of the three regimens. V1 – V10: Are the visits that participants were required to make during each period of their treatment regimens. [19]

To support and monitor adherence, we implemented an SMS system that allowed for daily reminders and data collection on product use. SMS-CASI was conducted daily for each of the eight-week regimen periods. SMS-CASI also continued during both one-week washout periods and up to one week during scheduling and completion of the participant’s final interview, for a total of 27 weeks of SMS-CASI reporting. This system was implemented in four countries (i.e., South Africa, Thailand, Peru, United States) and five languages (i.e., Xhosa, Afrikaans, Thai, Spanish, and English).

2.2 SMS protocol and system design

2.2.1 Adherence measurement

SMS was employed in MTN-017 as part of a mixed methods “convergence” approach to promote and monitor adherence. Figure 2 shows how SMS was one part of our convergence of multiple data sources to get at the truth of the data. Additional methods used to monitor product use included counts of returned study product at each visit and pharmacokinetic drug levels taken from plasma samples at study visits. At mid- and end-period visits, participants met with an adherence counselor to review their adherence data together, and a participant-centered approach was used to discuss any discrepancies in order to converge the data and determine the most likely number of doses taken. A detailed description of the convergence process and adherence measurements are published elsewhere.[19]

Figure 2. The process of data convergence from multiple sources.

SMS was used in concert with other data forms to produce the clearest picture possible of participant adherence. SMS: Short message service, PK: pharmacokinetic results, CASI: Computer Assisted Self-Interview. SMS Reports consisted of SMS-CASI data. *Traditional computer-based CASI data was reviewed at the end of study.[19]

2.2.2 SMS system description

Programming for the SMS-CASI schema was based in the principals and on the structure of eHealth Behavior Management model, which describes the use of technological tools (e.g., SMS, computers, mobile phones) as incorporated into the trans-theoretical (stages of change) model, to increase personal involvement and self-efficacy (see Figure 3). The eHealth Behavior Management model was important for this study because one of our major goals was increasing adherence to the biomedical and behavioral protocols of the study.

Figure 3.

The eHealth Behavior Management model

Starting at the enrollment visit, participants indicated the time of the day they wanted to receive the SMS reminder. Participants were encouraged to match the time of the reminder with the time of the day they expected to self-administer a product dose. A password, corresponding to an abbreviated version of their study site and participant ID (e.g., PT15 for participant number 15 in Pittsburgh, PA), was used to initiate the SMS-CASI when prompted. Participants had to answer only one survey question regarding their use of study product by sending a number (0–90). Upon receipt of the number of times of product use, the system generated a final message indicating that that day’s session had been completed (see Figure 4).

Figure 4.

SMS-CASI schema adapted to the eHealth Behavior Management model

2.2.3 SMS-CASI system architecture and programming

SMS-CASI systems for each country were programmed at a central location in the United States. The software program used was then federated by downloading it to each country’s computer system and text messages were sent and received via a Global System for Mobile Communications (GSM) modem dongle connected to that country’s computer via Universal Serial Bus (USB). Also, each dongle contained a Subscriber Identification Module (SIM) card activated with SMS messaging service to facilitate data transmission on that country’s specific networks (see Figure 5).

Figure 5.

SMS-CASI system architecture

From this structure a master text script (refer back to figure 4) was first developed in English and then translated and back-translated into each participating country local language(s). English was used in the United States and South Africa; Spanish was used in Peru and for Spanish speakers in the United States and Puerto Rico; Thai was used in Thailand; and Xhosa and Afrikaans were used in South Africa.

Programming involved developing four major functions: data capture, adherence, privacy, security. For data capture, keywords were associated with both questions. When the keyword was received, it triggered a pre-designated subsequent question. Adherence was aided by a daily reminder prompting the participant to report use of the study product and completion of the biomedical protocol. The time of the reminder was designated by the participant. Timing options were any hour on the hour of the 24 hour time cycle. Privacy was addressed by programming code terms into the questions and teaching participants the meaning of the code terms (i.e. the word “product” was used in place of “rectal microbicide gel” or “Truvada® (Gilead Sciences, Foster City, CA) (FTC/TDF) tablet”). An additional privacy procedure involved dissociating the participants’ personal information (i.e. phone number) from the collected data and the re-association of those data with an SMS ID number unique to each server and each site. Security was achieved through the use of text-able passwords that were the only way to initiate the SMS-CASI (refer back to Figure 4). These processes are further detailed in a separate publication.[22]

The SMS data federation and centralization schema used automated SMS forwarding via email to send data to a secure and centralized SMS repository (Data Push). It also included manual remote access to all SMS systems and databases so that the Columbia University behavioral team could download the data (Data Pull). Both the automated and manual data aggregation methods were secure and acted as reciprocal redundancies (see Figure 6).

Figure 6.

Centralized data management with a federated SMS-CASI system schema

2.2.4 SMS system preparation and procedures

At screening, site staff conducted an SMS Readiness Assessment to determine whether the participant had a cell phone he/she could use to respond to the SMS system or if the site needed to provide one. The one-page SMS Readiness Assessment consisted of four questions that helped staff to determine each participant’s familiarity with sending and receiving text messages so that the appropriate level of training could be provided. Prior to enrollment, study staff readied site-specific access number for the SMS system, the participant’s password, a cell phone for the participant if needed, and the Texting 101 Guide or user’s guide for the cell phone being provided, if the participant was unfamiliar with text messaging.

Participants were provided general instructions and trained on using the SMS system to report product use. Training involved using each participant’s phone to join the SMS system and conducting a mock SMS survey session while the staff monitored. Site staff followed the general use instructions and training script available on the MTN-017 web page under Study Implementation Materials. All SMS related protocol documents including the SMS Training Script, data cleaning syntax, SMS-CASI instructions, and the SMS readiness assessment are publically available at: http://www.mtnstopshiv.org/node/4524.

2.2.5 SMS system feature specifications

Feature availability of several SMS systems was assessed based on ideal criteria, including 1) open source, 2) low cost, 3) keyword response, 4) email capability, 5) participant identification and grouping, 6) text message scheduling, 7) multiple language operating system, 8) international SMS capability, and 9) no requirement for internet connectivity (see Table 1). After identifying the optimal SMS system, it was systematically implemented it in each country, working with information technology (IT) staff at clinical research sites.

Table 1.

Summary of SMS system feature specification assessment.

	Open Source	Cost	Keyword Response	Email Capable	Participant Grouping	Text Message Scheduling	Multiple Language Interface	International SMS Capability	Need Internet Connection
RapidSMS	Yes	Free/Pay	Yes	No	No	No	No	Yes	Yes
FrontlineSMS v1	Yes	Free	Yes	Yes	Yes	Yes	Yes	Yes	No
FrontlineSMS v2	Yes	Free	Yes	No	Yes	No	No	Yes	No
FrontlineSMS Cloud	No	Pay	Yes	No	Yes	No	No	Yes	Yes
Telerivet	No	Pay	Yes	Yes	Yes	Yes	No	Limited	Yes

Pay = The service costs money, Free/Pay = hybrid plan of free service with pay service for additional features.

Several SMS systems were evaluated based on the above nine highly desired criteria specifically needed for MTN-017. Open source allowed the system to be modified to fit new needs of the study as the study protocol evolved. A low cost (or free) service with no fixed monthly cost would allow the study team to pay for service as needed due to a limited budget for developing and operating the system. The system had to allow for automated keyword responses as specified in the SMS script and email capability so that anonymized messages could be backed-up securely using an Internet Message Access Protocol (IMAP) with a Transport Layer Security (TLS) made specifically for protected health information (PHI). It also had to contain a participant grouping feature so that one message could be sent to multiple participants simultaneously (e.g. for announcements and updates to participants or to send automated reminders at commonly selected times). A multi-language interface was vital so that non-English speaking site staff in other countries could help operate and troubleshoot the system. Finally, the system had to allow for international SMS capability and work without an internet connection due to the potential of internet brownouts.

3. RESULTS

3.1 Quantities of sent and received messages, and errors by SMS-CASI system

There were 142,177 total messages. Of those, 86,349 (61%) messages were “sent” messages. Sent messages consisted of the initial reminder message asking if the participant was ready to report for the day and, if so, requesting that the participant send their password. Sent messages also consisted of questions regarding product use, confirmation of completion of the participant’s SMS-CASI session, and a security reminder for the participant to delete the session from their phone and/or make sure their phone is password protected.

There were 55,573 (39%) “received” messages from the participants to the SMS system. Messages of this type consisted of passwords, answers to questions, number of product uses, comments, questions, concerns, and response errors. Less than one percent of the messages, 255 (0.18%), were failed messages. These messages were message attempts by the system that were either not sent or sent in error. For example, some messages were not sent because the network was down for a short period of time.

Message volume was directly related to number of participants recruited and participant activity in the study. Of the 142,177 total messages, The United States and Thailand systems represented the greatest volume of messages sending and receiving over two-fifths 59,084 (41%) and slightly under one-third 43,695 (31%), respectively. Peru had the next largest volume of message making up one-fifth 279,42 (20%) of the total volume of messages sent and received. Lastly, South African and Puerto Rico represented the smallest volume of sent and received messages, both sending and receiving less than five percent of all messages, 5,860 (4%) and 5,626 (4%) respectively.

Of the 142,177 total messages, 6,153 (4.33%) of the message were errors generated from either our SMS-CASI system or by participants. Volume of message errors for each country was fairly consistent with total message volume for that country. The United States, Thailand, and Peru had the highest number of errors at 1,788 (29%), 1,740 (28%), and 1,606 (26%) respectively, followed by South Africa, 580 (10%) and Puerto Rico 439 (7%), respectively.

Common errors included unsent messages or reminders, and participants sending non-keywords, wrong passwords, or duplicate texts. SMS-CASI system errors, such as unsent messages or reminders, made up 4.14% of all errors. The remaining 95.86% was primarily comprised of non-keywords, wrong passwords, and duplicate texts. Since there was significant overlap in these three major categories (wrong passwords and duplicate texts were sometimes non-keyword, and some wrong passwords were sent in duplicate) we could only approximate the percentage of each category, which we estimated as each being slightly less than a third of the total number of errors. There were a few spam messages from SMS bots, but they were deleted and their source numbers were blocked during early stages of implementation.

3.2 Results from SMS system feature specification assessment

In the development phase and over the course of the study we vetted and assessed five different SMS systems: RapidSMS, FrontlineSMS v1 (FLSMSv1), FrontlineSMS v2 (FLSMSv2), FrontlineSMS Cloud, and Telerivet. The vetting was based on the nine highly desired criteria specifically needed for MTN-017, as mentioned above.

We preferred that the platform used for each SMS system be open source so that over the course of the study it could be modified from its original programming to fit evolving needs of the study. RapidSMS, FLSMSv1, and FLSMSv2 were the only open source platforms at the time. Though Telerivet is not technically open source, it did provide the functionality to create and integrate with any API (application program interface) that we might develop for a specific task. Thus, although we could not change the source code, we could write our own code for specific functions and configure Telerivet to interact with it and trigger desired functions.

Due to the size of the study, and because we hoped that resource-limited sites might be able to utilize the technology after the study, sustainability related to cost was a large consideration. Thus, we sought a system that was either free or low cost with the flexibility of no fixed monthly cost, but with the option of “pay for services” as needed. FLSMSv1 and FLSMSv2, were the only systems that had completely free cost options at any scale. RapidSMS, FLSMS Cloud, and Telerivet all had free testing options so that one could try out the product, but with limited functionality, which prevented it from being used to the scale necessary for MTN-017. Our survey required a system that allowed for automated keyword response so that we could program the question and answer response protocol as specified in our SMS script. Fortunately, this type of functionality seems to be standard among most SMS systems because we found that all of the services we vetted provided this functionality.

To improve versatility and functionality in each country, it was necessary to develop a federated SMS survey system, but data also had to be stored in a centralized repository system. Specifically, a federated architecture would allow interoperability and information sharing between our semi-autonomous de-centralized SMS systems. In order to achieve this, one of the mission critical features for MTN-017 was to have the SMS system allow for email capability of the SMS messages. We were able to transmit and store anonymized messages through a secure IMAP with a TLS made specifically for PHI. At the time, email was the lowest common denominator technology at all sites. FLSMSv1 and Telerivet were the only systems that provided this functionality. We wanted a system that included the feature of participant grouping so that one message could be sent to multiple participants simultaneously. FLSMSv1, FLSMSv2, FLSMS Cloud, and Telerivet all provided this functionality as a basic feature. RapidSMS did not list this function as a basic feature, however their website noted that such functionality could be incorporated in development but might impact the cost of the project. Another critical feature was the ability to schedule messages to go out at a specific time of day or at a specific interval. This was the crux of our SMS text message reminder feature to support biomedical and behavioral protocol adherence. Surprisingly, only two systems, FLSMSv1 and Telerivet, listed this as a basic feature. Again, RapidSMS could provide this feature in scale up, which could impact cost.

Again, it was necessary that our SMS system be federated, with a specific installation in each country, and that the country staff be able to use it during the study and after for their own purposes. Having a system with an interface that supports multiple languages was important. Thus, non-English speaking site staff in other countries could assist in operating and troubleshooting the system if or when necessary. Only one system provided this functionality, FLSMSv1. Given the international nature of the study, in order for the systems to communicate with each other during testing, and in case one system needed to be used as a temporary surrogate for another malfunctioning system in a different country, our SMS system needed to allow for international SMS capability. Fortunately, all of the vetted systems provided this functionality; though, in the case of Telerivet this functionality was limited to a select number of countries (which did not include South Africa). Lastly, some countries had unique challenges. Internet connectivity was not always guaranteed (i.e., internet brownouts). Thus, we also desired a system that could operate without internet connection. FLSMSv1, and FLSMSv2 were the only systems with this functionality. All other systems needed the internet for access.

Table 1 shows the five systems that were vetted and how they rated based on each criterion. FLSMSv1 had all of the desired base features and was thus chosen as the system to use for MTN 017. Ultimately, FrontlineSMS (FLSMS) was chosen for its versatility in programming and because it could be used as a local cloud-based system.[23]

3.3 Challenges and solutions identified

After we set up FrontlineSMS v1, several challenges became apparent during the development process. These challenges were categorized into six primary areas: privacy and security, speed and functionality, standardization across sites, cost, malfunctions, and data management.

3.3.1 Privacy/Security

There were four challenges directly related to general privacy and security. Participant security needed to be safeguarded by preventing access to the SMS system from unregistered users (e.g., family members, friends) through the participant’s cell phone. Two other security challenges involved preventing unauthorized access to our databases through the internet and during data transfer from remote sites to our central repository over the internet. To prevent unauthorized access to the SMS system by unregistered users, a unique text message password with three to four alphanumeric digits was provided to each participant for use in initiating the session. Message content was worded using ambiguous terms like “product” (“Since your last response, how many times have you used the product?”). Also, at the completion of the report, participants were instructed to delete all messages and lock their phones. Internet security was enhanced using firewall technology and administrative passwords for each machine that ran our SMS system. Finally, secure messaging protocols were used to transfer programming specifications and anonymized participant data to and from the federated SMS systems and the central repository (see Table 2). An assessment of participants’ privacy and security concerns is detailed in a related article. [22]

Table 2.

Privacy and security, challenges by country.

Challenge	South Africa	Thailand	Peru	United States
Security	X	X	X	X
Privacy	X	X	X	X
Internet Security Measures	X	X	X	X
Programming Transfer	X	X	X

3.3.2 Speed/Functionality

There were three main challenges related to speed and functionality. Solutions to these challenges were implemented at the site level. First, internet speeds at three sites were slow, which hindered daily, real-time data transmission. Message speeds ranged from 5 seconds to less than or equal to 60 seconds. Most messages averaged under 10 seconds. To increase internet speed, computers used to generate the text messages at the study sites were relocated and routed to more reliable wired high-speed internet networks. Second, the SMS system itself lagged at three sites due to the computer servers on which it was hosted. To mitigate system lag, computers were periodically restarted and defragmented. The SMS system was cleaned of old data periodically to increase message download speed and to reduce duplicate data. Lastly, a GSM modem with a SIM card was used to send and receive messages. Though this helped to protect against power and internet outages (see “Malfunctions” section), it left the system vulnerable to variations in signal strength. In order to identify antenna locations where mobile device signal strength was the strongest, a cartographic analysis of antennas and towers was performed at United States sites.[24] For other sites, signal strength was estimated by site staff using mobile phones. If necessary, the computer and/or mobile device was moved to the location with greatest identified signal strength (see Table 3).

Table 3.

Speed and functionality, challenges by country.

Challenge	South Africa	Thailand	Peru	United States
Slow Internet Speed	X	X	X
System Lag		X	X	X
Variations in Signal Strength	X			X

3.3.3 Standardization across sites

Standardization increases efficiency of management across sites. To standardize the system across sites we had to overcome three major challenges. First, SMS messaging cost structures varied across participating countries, which required each site to identify appropriate participant SMS compensation based on local cost calculations. Each site provided its own computer on which to run the SMS system, but some operating systems were more recent than others, which affected SMS system functionality. To standardize operating systems (OS) across sites, an optimal OS for both Mac and Windows computers (i.e. OSX 6.4.4 and Windows XP) was identified and installed at all international sites. In addition, each system’s survey instrument had to be programmed in a different language or in multiple languages. To standardize the translation of materials involved in the programming, translations were verified by local site staff in each country of origin and messages for each language were programmed into each site’s system via remote access by a primary central translation team (see Table 4).

Table 4.

Standardization across sites, challenges by country.

Challenge	South Africa	Thailand	Peru	United States
Differing Local SMS Costs	X	X	X	X
Standardize Operating Systems (OS)	X	X	X	X
Program Multiple Languages	X	X	X	X

3.3.4 Cost

The cost of sending international text messages was the major financial challenge of implementing the SMS system. The system was programmed to send a daily reminder to each participant. If participants responded as instructed, the system would send a total of three messages daily to each participant and receive two replies from the participant per day. Local unlimited data plans were secured when available over pay-per-text. This allowed sites to budget for a fixed monthly cost that allowed them to send as many messages as necessary (see Table 5.).

Table 5.

Cost, challenges by country.

Challenge	South Africa	Thailand	Peru	United States
International SMS Cost	X	X	X

Cost was variable by country, dictated by site, and set in the country’s currency, according to each sites’ economics and participants’ costs of living. Participants were compensated for one completed text message session per day (defined as responding to their reminder with their password and sending the number of times the product was used). Each study site determined a locally appropriate compensation structure. This structure ranged from US$0.15 to US$2.00 per session, and participants were informed they would receive a bonus at the end of the month, which ranged from US$1.00 to US$10.00, for completing at least six sessions per week.[22] An exact estimate of cost to MTN 017 isn’t possible due to variability in participant texting behavior, staff time, fluctuating international currency conversions, and periodic changes in service costs. However, the United States SMS team made suggestions for how much to spend on SMS system setup, management, and participant SMS compensation, but these decisions were at the discretion of each site (SMS compensation table available at http://www.mtnstopshiv.org/node/4524). However, we developed a basic cost estimation algorithm in the “RESULTS: Cost” section. Cost considerations critical for implementation include a computer to run the system and manage the data, SMS text messaging software, staff time, cost of an unlimited SMS subscription to send messages to participants (A-sms), compensation for SMS costs incurred by the participant (P) to send and receive SMS messages (B-sms), Sim card (sim) and Sim card computer dongle (sim-d). The algorithm should be applied per SMS system needed at each site. It is only necessary to implement one system per country.

$$[(\text{computer}+\text{software})]+(\text{sim}+\text{sim-d})+\text{staff}\text{time}(\mathrm{\%}\text{effort})+\text{A-sms}+\text{B-sms}(\mathrm{P})(\#\text{messages})]\times \#\text{of}\text{countries}=\text{total}\text{cost}$$

The software, FLSMS v1, was open access (free), staff time was estimated at 25% effort, cost of an unlimited SMS subscription to send messages to participants was $40/month (changed over the duration of the study), compensation for SMS costs incurred by the 72 United States participants to send and receive SMS messages was reimbursed at $1 per reporting session per day, the two Sim cards were $30 and Sim card computer dongles were $60.

3.3.5 Malfunctions

Malfunctions, which represented the widest range and largest number of challenges, occurred throughout study implementation. These included: power outages, internet outage, automatic wrong keyword response triggered by spam, random spam texts, SIM card failure or USB modem issues, broken text in SMS messages, daylight saving time-induced system clock malfunction, physical malfunctions at remote sites, and participants without a functioning cell phones. Solutions rendered had to be relative to the resources available at each site (see Table 6).

Table 6.

Malfunctions, challenges and solutions, by country.

Challenge	South Africa	Thailand	Peru	United States	Solution
Malfunctions
Power Outage	6	0	N/A	0	Multiple federated SMS systems with international SMS capability were used so that one system could temporarily take the place of another system in a different country.
Internet Outage	18	0	N/A	0	SMS system that did not require internet was used.
Auto Wrong Keyword Response Triggered by Spam	8	1	0	0	Auto “wrong keyword” responses were deactivated to prevent accidental activation of error response when the system received random text message spam.
Random Spam Texts	8	1	0	0	SMS spam was blocked by blocking the number or message content.
SIM Card Failure	1	1	N/A	2	SIMs were tested using mobile phone. SIMs were replaced and their number transferred.
SIM USB Modem Issues	1	0	N/A	1	List of optimal modems was provided by central site based on software-specific recommendations.
Broken Text in SMS Messages	X	X	X	X	Formatting imbedded in the text caused broken text. So, we changed the text to Unicode unformatted text.
Daylight Savings Induced System Clock Malfunction	N/A	N/A	N/A	1	All systems in the United States were monitored and reset immediately after Daylight Saving time change
Physical Malfunctions at Remote Sites	X	X	X	X	IT specialists at each site were identified to assist local setup and maintenance of hardware
Participants Without Cell Phone	2	0	0	0	A cell phone was provided for participants that did not already have a cell phone

Numbers indicate the number of occurrences. “X” indicates untracked or unquantifiable instance.

3.3.6 Data management

Data management challenges included: centralizing data, managing site resources, data analysis, and data dissemination. As a solution to data management issues, a federated SMS-system architecture to mitigate SMS message costs and manage data access was employed. The SMS system was a hybrid data management system using both a federated schema where by each site had its own SMS and data storage system and data centralized schema where data from each site was backed up to a central location. SMS data was pushed from three international SMS sites (South Africa SMS-CASI, Peru SMS-CASI, and Thailand SMS-CASI) to the main site (United States SMS-CASI) located in the United States using an automated SMS forwarding function that sent each text message into a centralized protected repository. SMS data were also pulled from each one of the federated systems (SMS 2, SMS 3, and SMS 4) via remote access and downloaded using secure email protocols to centralize data backup (see Figure 6). In addition, de-identified data was downloaded daily by the behavioral team at the United States site via remote access to site computers. This process was done every day and immediately helped us have quick assess for daily data cleaning, analysis, and dissemination, which was necessary due to the high volume of data coming in on a daily basis.

Data were reviewed, cleaned, analyzed, and summarized daily (Monday-Friday) by staff in the U.S. using programming syntaxes to facilitate daily data analysis. Data summary sheets outlined the total number of times participants received reminders completed product use reports, and reported using the product. Summary sheets were updated and uploaded daily to a secure data management website for site staff to access. In addition, we developed a calendar template to share SMS behavior reports with sites to review with participants. All sites were challenged by the same data management issues.

4. DISCUSSION

Real-time data collection using SMS text messaging can help monitor adherence in biomedical HIV prevention trials, but SMS system development for, and management in, large multisite trials can be challenging.[8,10] Using SMS text messaging helped us to monitor adherence, despite initial implementation challenges.[19] Current literature on addressing challenges with developing and managing an SMS-based CASI system in an international study is non-existent. Literature on SMS use in smaller studies has not focused on mitigating message costs.[15,16] Efforts to address data access, data cleaning, and data analysis have not been reported. A common suggestion of paying for assistance from a third party service may be feasible for a smaller study, but could add to cost or contribute to security and privacy concerns.[15,16] The use of a federated schema to keep the systems local and costs lower is proposed. Providing SMS data, metadata, and management information to multiple sites has not previously been reported as an issue, but this is because most studies reported to date have not used SMS systems across multiple international sites. Since most studies do not have a multi-site international component, they can rely on a single system and do not need to mitigate variability in system performance across sites.[3,15–17] Nevertheless, for those studies that do have multiple international sites our methods and findings can assist in the development and management of a functional and effective SMS-based CASI.

Our system successfully sent and received 141,922 messages and very few errors (<1%), demonstrating both efficiency and reliability. Though there were over 6,153 errors, over 94% of those errors were human errors made by participants. Additional training could potentially reduce these errors. In this trial, we were able to: 1) develop an SMS system that allowed for daily reminders and collection of data on product use; 2) develop a centralized data management system with automated backup; and 4) address critical challenges to SMS-CASI implementation. MTN-017 provided a rare opportunity to utilize the SMS-CASI system over a complex, longitudinal multisite crossover study. With the growing ubiquity of SMS worldwide and the increased use of SMS in biomedical and behavioral research, studies that resemble MTN-017 will become more common. The challenges highlighted in the article were the most critical and universal ones we faced, but are certainly not the only challenges in implementing an international SMS system. It is highly recommended that teams implementing such a system work closely with SMS system developers and site staff to tailor solutions to their specific SMS challenges.

During the execution of this SMS system, challenges were identified and solutions implemented to ensure smooth implementation and operation of SMS as an adherence reminder system and real-time data collection modality. Solutions included: developing a federated SMS-system architecture to mitigate SMS message costs and manage data access, using secure email protocols to centralize data backup, developing several programming syntaxes to facilitate daily data cleaning and analysis, developing a calendar template for reporting SMS behavior to sites, ambiguating of text message language to increase privacy, and standardizing operating systems and hardware across sites to minimize variability in system performance. It is important to note that in the era of cloud computing, there are a growing number of SMS programs becoming available online, eliminating the necessity of a native program to run on specific operating systems. However, even to access these cloud services there are minimum system requirements (e.g., having a compatible OS or CPU), minimum resource requirements (e.g., reliable power and internet services), and minimum personnel requirements (e.g., competent IT staff). Thus, even in the era of software virtualization, meeting and standardizing minimum requirements across the systems of multiple international research sites is still a good practice.

4.1 Future work

For future studies that implement an SMS system as a method of real-time data collection, it may be beneficial to incorporate other platforms such as web-based messaging apps. If participants can select their preferred method of reporting product use, this may lead to higher usage of the reporting system. Future work should also include more robust methods for assessing signal viability in countries outside the United States.

4.2 Limitations

There are several limitations to this study. One limitation is the disproportionate sample sizes of the non-United States sites compared to the United States (Thailand n=54, Peru n=36, South Africa n=18, United States n=79). Therefore, the participant-specific challenges identified in sites outside of the United States may not be representative in the United States. In addition, SMS was not the most common form of text messaging communication in all countries (i.e., in South Africa, many users prefer instant messaging through phone apps). Thus, other SMS incompatible messaging platforms (i.e., WhatsApp) may be preferable to some populations, which may hinder the use of an SMS system. Lastly, though we believe medical, clinical, hospital facilities may be able to draw value from our systems general architecture and programming, we are hesitant to make this direct link in this paper. Our system was designed in the context of field research in multiple countries with very different organizations and diverse facilities. We feel trying to draw a direct link between our systems and the challenges that a hospital would face, would over simplify the critical administrative, legal, and structural challenges that are unique to the clinical and medical institution space.

5. CONCLUSION

We successfully deployed our SMS system in four countries and developed a centralized data management system with federated SMS systems and automatic backup. After assessing feature availability of several SMS systems based on ideal criteria, we found that FrontlineSMS v1 had the most ideal features for our research needs. Working with IT and site staff at each research site, we successfully addressed challenges to implementing our SMS system. By tracking our process and progress we identified factors that aid international implementation and operation of SMS-CASI for real-time adherence monitoring. The wide international use of the system allowed us to identify challenges to implementation and through collaboration with IT and site staff we adopted related solutions. Lastly, methods discussed in this article can aid other researchers to develop and manage an international multilingual SMS-based adherence reminder and CASI system.

Highlights.

• Ambiguate short message service (SMS) text message language to increase privacy.

• Standardize hardware/software across sites to minimizing system error variability.

• Federate the SMS system to reduce international SMS message costs.

• Use secure data transfer protocols to centralize, improve, and secure data access.

• Develop programming syntaxes to facilitate daily data cleaning and analysis.