Text Messaging and Disaster Preparedness Aids Engagement, Re-Engagement, Retention, and Communication Among Puerto Rican Participants in a Human Immunodeficiency Virus (HIV) Self-Testing Study After Hurricanes Irma and Maria

Abstract

Objective:

Hurricanes can interrupt communication, exacerbate attrition, and disrupt participant engagement in research. We used text messaging and disaster preparedness protocols to re-establish communication, re-engage participants, and ensure retention in a human immunodeficiency virus (HIV) self-test study.

Methods:

Participants were given HIV home test kits to test themselves and/or their non-monogamous sexual partners before intercourse. A daily text message-based short message service computer-assisted self-interview (SMS-CASI) tool reminded them to report 3 variables: (1) anal sex without a condom, (2) knowledge of partners’ testing history, and (3) proof of partners’ testing history. A disaster preparedness protocol was put in place for hurricanes in Puerto Rico. We analyzed 6315 messages from participants (N = 12) active at the time of Hurricanes Irma and Maria. Disaster preparedness narratives were assessed.

Results:

All participants were able to communicate sexual behavior and HIV testing via SMS-CASI within 30 days following María. Some participants (n = 5, 42%) also communicated questions. Re-engagement within 30 days after the hurricane was 100% (second week/89%, third week/100%). Participant re-engagement ranged from 0–16 days (average = 6.4 days). Retention was 100%.

Conclusions:

Daily SMS-CASI and disaster preparedness protocols helped participant engagement and communication after 2 hurricanes. SMS-CASI responses indicated high participant re-engagement, retention, and well-being.

Retention of study participants is of serious methodological concern, given the importance of follow-up to the validity of research findings. If attrition is too high, additional resources must be committed to compensate for participants who are lost to follow-up and to avoid loss of statistical power.¹ While some attrition is anticipated in all studies, benchmarks of 70 or 80 percent retention per study arm are common criteria for affirming the validity of a behavioral intervention.¹

Attrition is particularly threatening to the validity and reliability of study outcomes in longitudinal research. Variations in retention rates within and between research sites may lead to distorted results. Thus, factors that may lead to greater site-specific attrition are of particular interest for researchers. Sudden events, such as natural disasters, that affect an entire city or country add particular challenges to overall retention. For example, the disruption may make clinic visits or other procedures more difficult and favor retention at 1 site over another.

When major natural disaster-level disruptions (eg, hurricanes) occur, researchers typically face extraordinary difficulties just to maintain contact with study volunteers and to sustain research activities.² In these critical situations, effective communication is key to continued engagement, but how to retain participants in the immediate aftermath of a natural disaster is not well known. Little attention has been paid to the relative merits of different communication and disaster preparedness strategies to address these unique obstacles.^2–4

For a study on the use of human immunodeficiency virus (HIV) self-testing to screen sexual partners, researchers designed a daily sexual diary for research participants in Puerto Rico (PR) to record information on their sexual encounters via text messaging using short message service computer-assisted self-interview (SMS-CASI). When Hurricanes Irma and Maria, 2 of the most intense tropical cyclones worldwide of 2017, struck the island, both communication lines and travel infrastructure suffered widespread damage.^4,5 The unexpected events provided valuable lessons for engagement, re-engagement, retention, and communication, with participants in the event of catastrophic natural disasters.

METHODS

This study was a 5-year, randomized controlled trial exploring the effectiveness of HIV self-testing (HIVST) as a risk reduction tool for high-risk populations. Participants were recruited in New York City and PR between March 2014 and June 2018, and participant follow-up was completed in December 2018. Participants were recruited through mixed-methods using social media and traditional strategies, detailed elsewhere.⁶

Participants’ eligibility criteria were HIV-negative, 18 years of age or older, identifying as a man or transgender woman who has sex with men, reporting 3 or more occasions of condomless anal intercourse (CAI) with serodiscordant or unknown status partners in prior 3 months, 2 or more sexual partners in previous 3 months, and not on oral pre-exposure prophylaxis (PrEP) at the time of recruitment. Participants responded to a brief pre-screening survey. Those who qualified were invited to an in-person screening visit (Visit 1).

In Visit 1, participants completed a computer-assisted self-interview (CASI) to determine their eligibility for enrollment. The CASI assessed demographic information, sexual risk behaviors in the past 3 months, HIV knowledge, alcohol and substance use history, and willingness to use a self-test with partners. Furthermore, participants self-administered an oral HIV antibody test using the OraQuick® In-Home HIV Test while monitored by a researcher. Those who fulfilled the eligibility criteria were invited to enroll in the study and asked to return for an enrollment visit (Visit 2).

In Visit 2, participants were randomized into either control or intervention and enrolled in our SMS-CASI system for daily reporting of sexual behavior and HIVST kit use. Participants were asked to return for a follow-up visit in 3 months (Visit 3), while those in the intervention were also asked to return for an additional visit in 6 months (Visit 4). In subsequent follow-up visits (Visits 3 and 4), participants were tested for HIV and completed a CASI on-site.

Puerto Rico Research Site Participant Tracking Protocol

To maintain contact during the study participation, participants were asked to provide their personal cell phone number, the cell phone number of a friend or family member, an e-mail address, a mailing address, and their social media (eg, Facebook Messenger) contact information if they had one and were willing to provide it. Participants were asked to choose contact preferences (methods and time of day). Participants were routinely called 1 week before their site visit appointments as a reminder and again the day before the visit to confirm. If necessary, the appointment date and time could be changed as needed by the participant.

Puerto Rico Research Site Disaster/Hurricane Preparedness Protocol and Aftermath of Hurricanes

During the hurricane season, the PR research site had a protocol in place to protect the safety and property of patients, research participants, and personnel. Once alerted of a hurricane by the local weather service, all clinic appointments were canceled until staff could return safely to the site. Researchers called all participants who had appointments that week to cancel appointments and secured participants’ contact information for the next week. Participants were instructed that research staff would contact them as soon as operations resumed.

During September 2017, 2 hurricanes landed in PR. On September 6, Hurricane Irma passed over the northeastern shore of the island, causing flooding, wind damage, and loss of power on a significant part of the island.⁵ The PR research team was able to return to normal activities during the week of September 11, 2017. Researchers called participants to confirm their visits for that week and to reschedule visits missed due to Hurricane Irma. On September 18, the team received a warning alerting them of a major hurricane, so that day the clinic staff prepared for closure of the research offices, and appointments were again canceled.

The morning of September 20, 2017, Hurricane Maria made landfall as a Category 4 hurricane causing catastrophic damage to the whole island. The entire power grid was destroyed, and cell phone communication was almost completely cut off. In addition, because most antennas were down, there also was no Internet service, no functioning TV stations, and no radio broadcasting, except for 1 station that remained on the air for 72 hours. Running water was cut off because the water pumps had no power. All roads were either flooded or closed due to debris. A night curfew was put in place by emergency authorities to avoid robbery and looting (from discussions with the PR team).^4,7

SMS-CASI System

Per protocol, participants received daily text messages at a time of their choosing until the end of their participation in the study. Participants were first sent a reminder that asked about their readiness to report sexual behavior and HIVST kit use to the SMS-CASI system. They were then asked to report daily on the number of CAI occasions, the number of CAI occasions with a negative partner, the number of occasions in which they saw a partner’s negative test results, and the number of unused test kits remaining in their possession. Participants received monetary compensation of US $1 for each completed SMS-CASI session, plus a 50% bonus if they reported at least an average of 6 out of 7 sessions per week.

All procedures were approved by the Institutional Review Boards at the New York State Psychiatric Institute and the University of Puerto Rico Medical Sciences Campus.

MEASURES AND DATA ANALYSIS

We used Hurricane Maria as the time index cut-point for pre-post hurricane analysis because of the significantly greater devastation that it caused to the island of PR as compared with Hurricane Irma. Descriptive statistics for key variables were used to examine and test for frequencies related to participant communication, engagement, retention, and re-engagement. In addition to the daily SMS-CASI, participants completed a baseline assessment on-site via CASIs. The survey included a demographics section with questions regarding age, race, ethnicity, employment/student status, and income. Descriptive data were generated using SPSS Statistics, Version 24.0, 2016 (IBM Corp, Armonk, NY).

Operationalization of Categories and Measurements

Using sent, received, error, and meta-data, we measured 4 categories: engagement, re-engagement, retention, and communication. Engagement, operationalized as maintaining active participation, was assessed by looking at all messages received from the participant during the period of analysis. Re-engagement was operationalized as the participants’ ability to re-connect with the SMS-CASI system after the hurricane and was assessed by measuring the number of days elapsed until participants started to report again. In this way, we were able to measure participants’ capacity to continue participation in the study, despite the difficulties created by the hurricane. Retention was measured according to how many participants remained in the study and included data on participants’ return for their next study visit. It also included instances where participants were actively engaged with the SMS-CASI system within the 30 days following Hurricane Maria. Communication was defined as the receipt of any type of information and was measured using descriptive statistics of text messages that were received from participants, analyzed by context and content. We excluded messages sent to participants from the SMS-CASI system and focused on all instances where the participants tried to communicate with or through our SMS-CASI. This included responses to reminder prompts, general questions sent, and communicating sexual behavior during and after the hurricane.

RESULTS

Characteristics of the Participant Sample

The mean age of the sample was 31.67 (range 18–51, SD = 10.27) years. Most participants identified as male (N = 11; 91.7%) and 1 participant identified as transgender female. Similarly, most participants identified as gay or homosexual (N = 10; 83.3%), with the remainder of participants identifying as bisexual (N = 1; 8.3%) or other (eg, heterosexual transexual female; N = 1; 8.3%). Employment among participants was split, with most participants either working full-time (N = 5; 41.7%) or part-time (N = 5; 41.7%), and a minority unemployed (N = 2; 16.7%). Most participants were not currently in school (N = 10; 83.3%), though a few were (N = 2; 16.7%). Finally, participants’ annual income had a wide range (US $0–$98 000). Four participants reported income of < US $20 000. Of them, 2 reported earning US $0. Two participants reported annual earnings of > US $30 000 (1 reported US $31 000 and the other US $98 000). Two participants didn’t provide data for annual income (Table 1).

TABLE 1.

Sample Characteristics

Age	N = 12	%	Employment	N = 12	%
18–24	4	33	Working full-time	5	42
25–29	2	17	Working part-time	5	42
30–34	2	17	Not working	2	17
35+	4	33
Gender	N = 12	%	Currently a Student	N = 12	%
Male	11	92	Yes	2	17
Transgender male-to-female	1	8	No	10	83
Sexual Orientation	N = 12	%	Income	N = 10	%
Gay/homosexual	10	83	Less than 10 000	2	17
Bisexual	1	8	10 001–19 999	2	17
Other: heterosexual transsexual Female	1	8	20 000–29 999	4	33
			30 000–39 999	1	8
			98 000	1	8

SMS-CASI Functionality

SMS-CASI Performance Data

The SMS-CASI system continued to send out reminder messages from a server in New York during both hurricanes and thereafter. For the 30 days prior to Hurricane Maria, the SMS-CASI system processed 6315 messages from our site in PR (∼211 messages/day), of which 3972 were sent by the system, and 1 of which failed. On the day of the hurricane, September 20, 2017, the SMS-CASI system processed 161 messages, of which 110 were sent by the system, with zero failed messages. Finally, for the 29 days following the hurricane, the SMS-CASI system sent 3419 messages, 1 of which failed. It processed a total of 5380 sent and received messages (∼185 messages/day). A total of 46 errors (explained below) were recorded before the hurricane, only 1 during the hurricane, and 78 after the hurricane, for a total of 125 (1.97% of total messages) errors detected within the period of analysis. Messages received from the PR participants were classified as engagement and are further discussed later in the “Engagement” section.

SMS-CASI and Participant-Generated Errors

Internal SMS-CASI system assessment tools indicated that, in the period before the hurricane, there were 40 errors generated by participants and 6 errors generated by the CASI system. In contrast, there were 77 errors made by participants and 1 error generated by the SMS-CASI system in the period after the hurricane. On the day of the hurricane, the system generated only 1 error, and there were no errors generated by participants. Participant errors included sending the wrong response type, providing cumulative responses rather than discrete responses, sending the wrong password, and sending invalid responses. The 1 system error was due to a failed data transmission to the SMS-CASI service provider. According to the assessment metrics of our SMS-CASI service provider (Nexmo), all messages received by our service provider were successfully delivered to our SMS-CASI system. However, local reports of damaged infrastructure suggested that there could have been challenges, delays, or failures from the participants’ phones due to issues with their local cellular service providers. An exact number is unknown due to lack of access to the SMS data curated by the participants’ individual local cellular service providers.

Engagement

Total Messages Received From Puerto Rico's Participants Before and/or After Hurricane Maria

Engagement was measured by the cumulative number of messages received from PR’s participants before, during, and after the hurricane. All participants who were participating in the study during the 30-day periods before and after the hurricane (N = 9) continued to respond to the SMS-CASI system during the time of the hurricane and/or thereafter. Most participants consistently responded to the SMS-CASI system at the time of the hurricane and thereafter, with some decreases and fluctuations (Table 2). For the 30 days prior to Hurricane Maria, participants in PR sent 2343 messages (∼78/day). On the day of the hurricane, participants sent 51 messages, and for the 29 days after the hurricane participants sent 1960 messages (∼68/day).

TABLE 2.

SMS-CASI Use Before and After Hurricane Maria

Participant IDs	Study Enrollment Date	Number of SMS-CASI Completions in Days 08/20/17–09/19/17	Number of SMS-CASI Completions in Days 09/20/17–10/20/17	Study Completion Date	Date of First Response on or After Hurricane Maria	Number of Days After Hurricane Maria Until First Response	Randomized Condition
w1012	05/16/17	28	15	11/15/18	09/26/17	7	Intervention
w1098	07/14/17	19	10	N/A	09/28/17	7	Intervention
w1100	07/13/17	27	20	N/A	09/27/17	8	Intervention
w1151	08/21/17	31	30	11/06/18	09/21/17	1	Intervention
w1173	10/04/17	N/A	17	01/09/18	10/04/17	0	Intervention
w954	03/01/17	12	N/A	09/01/18	N/A	N/A	Intervention
w968	03/07/17	19	N/A	09/13/18	N/A	N/A	Intervention
w988	05/05/17	21	12	N/A	10/06/17	16	Intervention
w993	03/31/17	30	19	11/01/17	09/28/17	9	Intervention
x1058	05/26/18	6	N/A	08/31/17	N/A	N/A	Control
x1122	07/05/18	32	7	10/05/17	09/30/17	10	Control
x1171	09/18/17	3	27	12/19/17	09/20/17	0	Control
Total		228	157			58

Re-Engagement

Participant Re-Engagement With the SMS-CASI System Measured as Time to First Response Post-Hurricane

Re-engagement was characterized and measured by the participants’ ability to resume communication with the system after disruption. Three participants had no comparison data because they completed the study before the hurricane. The remaining 9 participants re-engaged with the system at various times during and after the hurricane. The range of time that it took participants to re-engage with the study via the SMS-CASI system was 0 to 16 days; 89% of participants re-engaged with the system within 2 weeks, and 100% of participants re-engaged within 3 weeks. The average number of days needed for re-engagement was 6.4.

Challenges to Re-Engagement After Catastrophic Disaster: A Narrative

When contact occurred with participants, many shared with site staff their traumatic experiences of the consequences of this event in their lives (see section, Communication). Some expressed being anxious during and after the event. There were many simultaneous stressors: lack of power, water, housing, communication, transportation, and health services. However, the greatest burden was the delay in returning to a normal way of life.

Research team members in PR described experiencing back-to-back hurricanes as a catastrophic event and fearing for the safety of their family members, friends, co-workers, and clinic participants. For the first few days following Hurricane Maria, people sought shelter, clean water and food, along with fuel for their cars and, for those who had them, their backup power generators. Despite the catastrophe, staff members described how people helped each other, and workers, doctors, nurses, police, and emergency personnel worked hard to restore some sense of normality. Within each community, in both urban and rural areas, people helped each other and worked day-by-day to stay well. Although the response to the emergency by both local government leaders and the federal government was suboptimal, the Puerto Rican people were resilient and willing to resume a normal way-of-life in the midst of chaos. Clinic staff strove to help the participants re-engage in the study and resume the SMS, which helped assure 100% retention.

Retention

Participants Who Started Before and Returned After the Hurricane

We defined retention as the total number of participants who started before Hurricane Maria hit and continued to report, or re-engaged, with the system after Hurricane Maria (see Table 2). We also included data from study visits (Table 3). Overall, 100% of participants continued participating in the study within the 30 days after Hurricane Maria. Eight (66.6%) of the participants were actively reporting to the SMS-CASI system before, during, and after Hurricane Maria. Three participants (25%) were active before Hurricane Maria but had completed their final study visit before the hurricane. These participants were not included in the retention analysis.

TABLE 3.

Participant Follow-Up Visits to Study Site

Participant ID	Study Enrollment Date	Visit 3	Visit 4
w1012	5/16/17	8/15/17	11/15/17*
w1098	7/14/17	10/19/17*	1/26/18
w1100	7/13/17	11/20/17*	3/5/18
w1151	8/21/17	11/6/17*	2/1/18
w1173	10/4/2017*	1/9/18	4/9/18
w988	5/5/17	8/18/17	–
w993	3/31/17	8/10/17	11/1/17*
x1122	7/5/17	10/5/17*	–
x1171	9/18/17	12/19/17*	–

^*First visit to the study site after Hurricane Maria.

Communication

Communication of Study Protocol Behavior

All 6 data elements continued to be communicated from the participant (readiness to report, daily number of unprotected sexual encounters, knowledge of partners’ HIV test results, number of confirmation of partners’ HIV test results, number of test kits left [intervention only], and other general participant-to-researcher communications) post Hurricane Maria.

Reports of Sexual Activity, by Participant

Participants consistently reported whether they were having sexual intercourse without a condom (Figure 1 and Table 4), predominantly reporting that no sexual intercourse without a condom occurred. If their answer to P1 was equal to 0, then P2 was skipped, so P1 – P2 = Instances of no sexual behavior (see Figure 1). Though the data suggest decreased sexual activity in the month after the hurricane, there isn’t enough parallel data for each participant before the hurricane to reliably compare (see Table 4). For instance, participant w1173 (see Tables 2 and 3) began the study after the hurricane and thus has no comparison data and does not show up in Table 4.

FIGURE 1. SMS-CASI Questions in Spanish (With English Translations).

P = pregunta, which means “question” in Spanish.

AS = anal sex; C = condom.

TABLE 4.

Counts of Reports by Question Before and After Hurricane Maria

Before Hurricane				After Hurricane Maria
PTID	Q1	Q2	Q3	PTID	Q1	Q2	Q3
x1058	12	6	6	-	-	-	-
x1122	4	0	0	x1122	0	0	0
x1171	2	1	0	x1171	5	5	1
Total (x)	18	7	6	Total (x)	5	5	1
w1012	40	0	0	w1012	24	0	0
w1098	7	3	2	w1098	3	2	2
w1100	71	0	0	w1100	63	2	12
w1151	20	13	12	w1151	5	5	5
w954	12	11	0	w1173	11	9	2
w968	4	3	0	-	-	-	-
w988	0	0	0	w988	0	0	0
w993	3	3	3	w993	0	0	0
Total (w)	157	33	17	Total (w)	106	18	21

Q1-Q3 = Questions 1–3; w = intervention group; x = control group; (-) = no data available.

Disaster Preparedness Narratives

During the week of October 2, 2017, the research team re-started clinic activities and began to contact participants through a mix of contact modalities (phone calls, text messages, e-mail, and Facebook messages). Direct phone calling was not successful; however, text messages did get through. Researchers contacted all participants in PR who were enrolled at that time to be sure they were safe and to let them know that the clinic was open. Researchers began to reschedule missed appointments and to keep scheduled ones. The participants slowly renewed contact, and, ultimately, no participants were lost to follow-up.

DISCUSSION

Hurricanes Irma and Maria, 2 of the most intense tropical cyclones worldwide of 2017,⁵ were regarded as the worst natural disasters on record in PR.^4,7 Retaining participants is 1 of the most critical challenges of any study. Retaining participants in the wake of a catastrophic natural disaster greatly increases that challenge. The literature on participant retention and re-engagement in the aftermath of a hurricane is sparse. However, Packenham et al. suggest that having a call-in system in place for researchers and participants to communicate is key.² Our research on using text messaging confirms that direct lines of communication are important to reach and retain participants and to share critical information. Despite that loss of electricity and damaged telecommunication poles, wires, and antennas hindered communication for both the participants and the local research site, backup generators allowed people to charge their phones, and Google provided cellular service using Loon balloons, which is why cellular communication was the most stable.

In addition, we found that having a mobile, real-time, automated, low-burden communication system, such as SMS-CASI text messaging, greatly aided our participants’ ability to stay connected and engaged in the study. It also aided our ability to observe participant resilience in resuming SMS-CASI reporting, communicate important information, and continue to monitor their sexual and HIV testing behavior. Daily SMS-CASI text reminders and SMS-CASI reports acted as a “heartbeat” system. Some participants resumed responding to the SMS-CASI within 1 to 2 days, which helped staff to know that they were still willing and, more importantly, able and safe enough to participate.

The SMS-CASI system also acted as a direct line of communication from researchers to participants. Though the SMS-CASI reminders and SMS-CASI were automated, SMS-CASI text messages could also be sent manually. We used the system to communicate with participants in cases where they had not responded in 3 or more days. This method was effective in making sure that participants were not lost to follow-up, communicating information to help them troubleshoot issues with the system, and making sure that the participants had a direct line of communication to site staff in real-time. Participants would often confirm that they received the message, using complex or simple responses or an emoji (ie, thumbs-up or smiley face).

Physical and psychological stresses and the decrease in basic resources led participants to shift priorities away from study participation.^2,8 Thus, the high level of participant engagement with the SMS-CASI system after Hurricane Maria was unexpected and had 3 very important effects. First, it reduced anxiety and tension among the research staff because it indicated that no participants had been killed or critically injured. Second, it showed that participants cared about the study and wished to stay engaged with the research. Third, it exemplified that Puerto Ricans immediately strove to reclaim their normal lives, even as early as the next day, returning to their routines however they could. Hence, participants’ continued engagement with the study and the SMS-CASI system following Hurricane Maria demonstrated that their level of re-engagement was very high.

Last, the breakdown of civic infrastructure made it difficult for participants to travel to study sites to complete testing, interviews, questionnaires, and other protocols. Both participants and research staff in PR were affected by damaged infrastructure. During the period immediately after Hurricane Maria, there were no participants who needed to travel to the study site. Nevertheless, our experience demonstrated the importance of having protocols in place in future studies for communicating, engaging, and retaining participants in the event of a natural disaster.

FUTURE WORK

Future research should look at the mechanisms that aid participant re-engagement after a natural disaster. Research should also measure the impact of incentives and compensation on participant engagement and retention after a natural disaster. Much of the literature suggests the need to develop emergency protocols for both participant and site staff.^2,4,5,8 Surveying site staff and participants, including using posttraumatic stress disorder and harm reduction interviewing techniques, could provide insight on those procedures that are critical to participant and researcher safety and re-engagement in the wake of a catastrophic natural disaster.

Limitations

This study was not designed or intended in any way to collect data relevant to Hurricane Maria or any natural disaster. However, use of an SMS-CASI for daily automated reporting provided a rare opportunity to analyze participant behavior in real-time before, during, and after Hurricane Maria. Our sample size of 12 participants limits the statistical strength and generalizability of our results. Unfortunately, only 9 participants were still in the phase of reporting sexual behavior via SMS-CASI immediately before, at the time the hurricane hit, and/or immediately after. Nevertheless, given that we were collecting daily reports via SMS-CASI text messaging, those 9 participants generated 5490 data points during the time period of interest and analysis, which gave us sufficient data to conduct an analysis of the effects of the hurricane on texting behavioral data in a research study.

CONCLUSIONS

The devastation of Hurricane Maria continues to negatively impact the livelihood of Puerto Rican citizens and critical research studies being conducted there. Lack of participant retention is detrimental to the thousands of important and groundbreaking studies in PR that have contributed to its economy and population welfare for generations. Daily SMS-CASI helps participants stay engaged in research studies after hurricanes or similar natural disasters. SMS-CASI text message responses from participants indicated to researchers that participants were not critically injured during Hurricane Maria. Participants were able to stay engaged with research staff in real-time, which had a positive impact on participant participation. Similarly, when participants reported to the SMS-CASI system, it helped curb the psychological stress of research staff. Thus, disaster preparedness protocols are critical for hurricane prone research environments, and daily SMS-CASI text messaging as a reporting protocol may have positive effects in the wake of a natural disaster.