A randomized controlled trial of daily text messages versus monthly paper diaries to collect bleeding data after intrauterine device insertion

1. Introduction

Patient-reported outcomes are an important measure in clinical research, and diaries are a key example of information that depends on the quality of data gathered. Bleeding patterns can inform a woman's choice of contraception and researchers frequently collect such data using paper diaries [1–3]. The validity of data collected with paper diaries could be compromised if study participants fill out multiple entries at once [4–7].

The use of text messages for data collection in clinical research merits investigation because more than 90% of American adults own a cell phone [8]. Nearly all 18–29 years old report using their phones for text messaging [8]. Text messages are increasingly used in healthcare for result notification and behavior modification and to improve adherence to medication [9–19]. Several studies have used text messaging to collect data in clinical trials [12,13,20–22], with high response rates and good reliability and validity [20,22]. Text messages increase oral contraceptive pill continuation rates and reduce loss to follow-up in clinical trials [13,23].

Within a cohort study that evaluated bleeding and cramping patterns for 90 days following insertion of contraceptive intrauterine devices (IUDs) [24], we carried out a randomized controlled trial in which we compared the quantity of data provided with two-way text messages versus paper diaries. We hypothesized that bleeding data collected daily using text messages would have fewer missing values than paper diaries returned monthly. To evaluate whether text messages provided superior data, we explored variations in the content of the responses that could indicate differential validity. Finally, we assessed satisfaction with the two data collection methods.

2. Materials and methods

We enrolled women from two clinical sites: New York- Presbyterian Hospital's Family Planning Center (a Title X-funded clinic primarily serving low-income women) and Columbia University's Division of Family Planning faculty practice (predominantly serving college students and women with commercial insurance). Women were eligible to participate if they spoke English or Spanish, were willing to keep a daily bleeding diary, had cellular phones with text messaging functionality and were receiving an IUD for contraceptive indications. They chose either the copper T380A (ParaGard®, Teva Pharmaceuticals) or the levonorgestrel IUD (Mirena®, Bayer HealthCare Pharmaceuticals). Women having an IUD removal and reinsertion at a single visit were ineligible. The institutional review board at Columbia University Medical Center approved the study, which is registered in clinicaltrials.gov (NCT01730911).

Women enrolled in the study following completion of routine clinical services, including the IUD insertion. After informed consent, they completed a brief questionnaire about demographic characteristics, medical history and past contraceptive use. The research coordinator called all participants’ cell phones to ensure that they were in service.

We then randomized participants to report the presence or absence of bleeding and cramping each day for 90 days, by replying to daily text messages or by completing paper diaries and mailing these to the study center each month. Here we will discuss bleeding data collected; cramping data will be reported separately. Randomization stratified by clinical site and IUD type used a 1:1 fixed allocation ratio with a block size of 4. One investigator, not involved in participant-related activities, prepared sequentially numbered opaque envelopes designating data collection assignment. The investigators analyzing the diary data (SN and PC) were blinded to randomization assignment and IUD type.

Participants randomized to text messages received a text at 9 p.m. nightly inquiring about bleeding experienced that day with the same answer categories as the paper diaries (“Did you have any bleeding Sunday? REPLY (1) no bleeding, (2) light bleeding, (3) moderate bleeding or (4) heavy bleeding”). A second text message was sent 12 h later if a participant had not replied (“Did you have any bleeding Sunday?” with the same answer choices). For a given day, answers were accepted for 24 h following the first text message; there was no opportunity to input values after this period elapsed. MIR3 (San Diego, CA), a company that provides emergency mass notifications, sent the fully automated text messages.

Paper diaries contained study-specific monthly calendars with the numbers 1 through 4 listed in each day and instructions to circle the amount of bleeding corresponding to each number: (1) no bleeding; (2) light bleeding necessitating use of a panty liner, toilet paper or no protection; (3) moderate bleeding requiring a sanitary pad or tampon; and (4) heavy bleeding that leaked beyond a pad or through a tampon.

At enrollment, the research coordinator gave all participants, regardless of group assignment, study-specific calendars and addressed, stamped envelopes. Although abbreviated definitions were used in daily text messages and on each paper calendar, all participants received identical definitions quantifying bleeding. Participants randomized to paper diaries received monthly text messages reminding them to mail the diaries. Paper diaries could serve as a backup for those randomized to text message in case of a cell phone service interruption. All participants received a text message reminder at 4 weeks to schedule an IUD follow-up visit and a request for a telephone exit interview at 3 months. The paper diary group received no other contact from study staff. All study instruments were available in English and Spanish.

During the telephone exit interview, we asked participants whether they would have chosen the same data reporting method to which they were assigned. They rated satisfaction and ease of use of their reporting method using a 4-point Likert scale. In addition to US$10 received at the enrollment visit, they received up to US$35 for completion of 3 months of bleeding data; those who completed at least 60 days of diary data and the exit interview were eligible for a lottery to receive one of four US$500 gift cards.

The primary outcome was number of days for which participants reported bleeding data (maximum: 90 days). We stratified this analysis by age, race, primary language spoken, level of education, parity, whether the participant usually tracks her menses, clinical site and IUD type. Secondary outcomes were descriptive information from the exit interviews. We also describe technical challenges encountered by study participants using text messages.

If participants using paper diaries made up for missed entries by filling out several days at once, then they might have more sequential days with identical answers; in contrast, women responding to text messages in a 24-h window might have more day-to-day variability in their responses. To explore any such differences, we calculated how often participants gave the same response over consecutive days. We defined a ‘run’ as any period of at least two consecutive days during which identical answers were given. For participants providing 60 or more responses, we summed the days that were part of runs. We treated data as missing if no ‘1,’ ‘2,’ ‘3’ or ‘4’ response was received for that day for text messages or if the participant did not circle any number on the paper calendar for a given day. In the event of a discrepancy between paper and text for the few (17) participants providing data via both, we chose the text value. We did not impute missing data. We counted the frequency of these runs and calculated an average run length. Finally, we compared the average run length for paper diary versus text data collection. We also compared the distribution of responses (1, 2, 3 and 4) between groups.

We performed intention to treat analyses using the Wilcoxon rank-sum test to compare medians for data that were not normally distributed. Categorical variables were compared with the Fisher's exact or χ2 test using SAS (Cary, NC). We calculated the sample size for the main cohort study (N= 230) to compare differences in number of bleeding days between IUD types; the sample size for the bleeding diary analysis thus had 80% power to detect a 15-day difference in number of responses, assuming a standard deviation of 40 and a two-sided α of 0.05.

3. Results

From September 2012 to February 2013, 451 women sought IUD insertion (Fig. 1). Two hundred thirty were eligible, interested and enrolled, with 115 randomized to paper bleeding diaries and 115 randomized to text messages. Randomization yielded similar groups (Table 1). We also evaluated body mass index, smoking status, marital status, prior diagnosis of a sexually transmitted infection, cycle day of insertion and previous IUD use; these were similarly distributed by randomization group and were not associated with the study outcome.

Figure 1. Recruitment, randomization, and participant flow.

Table 1.

Participant characteristics by randomization group^a

	Text (n=115)	Paper (n=115)
Age (median, range)	26 (16-45)	26 (16-44)
Race/Ethnicity
Hispanic (any race)	82 (71.3)	83 (72.2)
Non-Hispanic White	25 (21.7)	26 (22.6)
Non-Hispanic Black	3 (2.6)	4 (3.5)
Non-Hispanic other	5 (4.4)	2 (1.7)
Primary language
English	53 (46.1)	59 (51.3)
Spanish	62 (53.9)	56 (48.7)
Level of education completed
High school or less	68 (59.1)	68 (59.1)
Associate's/technical certificate	13 (11.3)	10 (8.7)
Bachelor's/postgraduate degree	34 ( 29.6)	37 (32.2)
Parity
Nulliparous	47 (40.9)	49 (42.6)
Parous	68 (59.1)	66 (57.4)
Usually tracks menses
Yes	76 (66.1)	80 (69.6)
No	39 (33.9)	35 (30.4)
Clinical site
Family Planning Center	84 (73.0)	85 (73.9)
Faculty Practice	31 (27.0)	30 (26.1)
IUD type
CopperT380A	58 (50.4)	61 (53.0)
LNG-IUS	57 (49.6)	54 (47.0)

Abbreviation: IUD, intrauterine device; LNG-IUS, levonorgestrel intrauterine system

^aAll data presented are n(%) unless otherwise stated

Overall, the text message group had more days where bleeding data were reported. Days of data were not necessarily consecutive. The text group reported a median of 82 days [interquartile range (IQR) 40–89] and the paper group reported a median of 36 days (IQR 0–88) (p≤.001) (Table 2). This was true for all participant characteristics. Forty-seven women (20%) provided no bleeding data, 11 in the text group and 36 in the paper diary group. Thirty-one participants (13%) provided 1–30 days of bleeding data, 13 in the text group and 18 in the paper diary group. One hundred six (47%) provided 30–89 responses, 70 in the text group and 36 in the paper group. Forty six (20%) provided complete 90-day bleeding data, 21 in the text group and 25 in the paper group. The number of responses received each day decreased gradually over the 90-day study period but was always higher in the text group (Fig. 2). These differences are statistically significant from day 10 onward (p=.01).

Table 2.

Days that bleeding data were reported (90 possible) by randomization group

	Text		Paper
	n	Median [IQR])	n	Median [IQR]	P
Total	115	82 [40-89]	115	36 [0-88]	<.001
Age
16-24 (<=33rd percentile)	38	79 [31-89]	43	36 [0-89]	0.12
25-29	41	86 [64-89]	34	67.5 [0-90]	0.07
>29 (>66th percentile)	36	81 [23.5-89]	38	26.5 [0-83]	0.03
Race/Ethnicity
Hispanic (any race)	82	80 [21-88]	83	31 [0-86]	0.008
Non-Hispanic White	25	88 [79-90]	26	76 [22-90]	0.10
Non-Hispanic Black	3	84 [40-89]	4	0 [0-45]	0.39
Non-Hispanic other	5	89 [77-90]	2	45 [0-90]	0.85
Primary language
English	53	86 [73-90]	59	31 [0-88]	<.001
Spanish	62	79.5 [18-87]	56	42.5 [0-87.5]	0.22
Level of education completed
High school or less	68	79 [27-88]	68	15 [0-65]	<.001
Associate's/technical certificate	13	80 [40-89]	10	20 [0-90]	0.27
Bachelor's/postgraduate degree	34	88 [81-90]	37	84 [55-90]	0.25
Parity
Nulliparous	47	88 [79-90]	49	59 [0-90]	0.01
Parous	68	65.5 [17.5-86.5]	66	26.5 [0-84]	0.02
Usually tracks menses
Yes	76	85 [61-89]	80	34 [0-86.5]	<.001
No	39	64 [14-88]	35	37 [0-90]	0.61
Clinical site
Family Planning Center	84	79.5 [23-88]	85	21 [0-80]	0.001
Faculty Practice	31	88 [79-90]	30	84 [28-90]	0.24
IUD type
Copper IUD	58	81 [21-88]	61	36 [0-88]	0.15
LNG-IUS	57	84 [62-89]	54	34.5 [0-84]	<.001

IUD, intrauterine device; LNG-IUS, levonorgestrel intrauterine system.

Figure 2.

Daily responses over the 90 day study period

In the text message group, 38 (33%) women had some difficulty receiving messages or sending responses. Most of these (35/38) were recruited from the Family Planning Center. For 17 (15%), we were unable to resolve the technical problems and we asked them to submit paper diaries. The 11 who did not provide any data were all from this group. Of the remaining six, three provided data with paper diaries and text, and three provided data entirely with paper diaries.

Stratification demonstrated that those with more than a high school education and those enrolled from the Faculty Practice provided more complete diaries in both randomization groups (Table 3). Conversely, those who were less educated and randomized to paper diaries and who enrolled from the Family Planning Center had the lowest response rates. We repeated these analyses excluding the 38 participants who had technical problems with text messages and the results were similar. A multiple linear regression analysis using a backward selection model with α=0.05 also identified lower education level and randomization group as associated with median number of days reported.

Table 3.

Responses reported (90 possible) by site of care and education level

	Text					Paper
	<= HS		>HS		P	<=HS		>HS		P
	n	Median [IQR]	n	Median [IQR]		n	Median [IQR]	n	Median [IQR]
Family Planning Center	64	79 [21.5-88]	20	80.5 [30.5-88.5]	0.55	62	11.5 [0-61]	23	79 [8-90]	0.01
Faculty Practice	4	67 [45.5-84]	27	89 [83-90]	0.14	6	60.5 [0-90]	24	84 [41.5-90]	0.69

HS, high school; IQR, interquartile range.

Most participants completed an exit interview (203, 88%), regardless of randomization assignment or IUD type (Fig. 1). Seven of the 102 participants who used text messages and completed an exit interview would have preferred to use paper. In contrast, 47 of the 100 assigned to paper diaries and who completed an exit interview would have chosen text messages (pb.0001). Fifty five (54%) in the text messages group reported being “very satisfied” with their method of data collection, compared to 27 (27%) of those using paper diaries. Six (6%) women using text messages and 19 (19%) using paper diaries found it “difficult” or “very difficult” to use their assigned reporting method (p=.005). Whether a participant would choose the same reporting method was not associated with the number of responses.

We explored the validity of the data received among participants who provided at least 60 responses (n= 131, 57%); women in this subgroup gave nearly complete answers in both the text and the paper groups (median: 88; IQR 81–90). We counted the consecutive days that the woman gave the same response (run length) and the number of runs. Although the number of runs was similar between the two groups (12.4 and 11.1 for text and paper, p=.07), the median run length was shorter for the text message group (5.3 days, IQR 3.6–6.7 for text and 6.4 days, IQR 5.0–8.5 for paper, p=.004). However, the number of days that participants reported no bleeding, light bleeding, moderate bleeding or heavy bleeding did not differ by data reporting modality for participants giving at least one response (Table 4). We repeated the analysis shown in Table 4 for those giving more than 60 responses, and the results were no different.

Table 4.

Median number of 1, 2, 3, and 4 responses

Participants who gave at least 1 day of data
	Text (n=104)	Paper (n=79)	P
No bleeding (1)	49 [23.5-61]	33 [14-59]	0.05
Light bleeding (2)	16 [8-30]	13 [9-23]	0.41
Moderate bleeding (3)	5 [2-9]	5 [2-10]	0.73
Heavy bleeding (4)	1 [0-3]	1 [0-3]	0.64

Participants who gave >60 responses
	Text (n=82)	Paper (n=49)	P
No bleeding (1)	55 [38-65]	56 [36-63]	0.77
Light bleeding (2)	18 [12-34]	18 [11-35]	0.96
Moderate bleeding (3)	6 [3-9]	8 [5-12]	0.04
Heavy bleeding (4)	1 [0-5]	1 [0-4]	0.90

4. Discussion

Collecting diary data is a challenging, but important, way to gather patient-reported outcomes. In this study, bleeding data were far more complete among the women assigned to text messages compared to women assigned to paper diaries. Our data indicated that women who attained higher levels of education did well regardless of data collection modality, but text messages improved response rates for those with a high school education or less. Most striking was that low level of education and recruitment from the Family Planning Center characterized the women who did poorly with the paper diaries. In contrast, when assigned to text messages, such women provided a quantity of answers similar to the generally more educated participants recruited from the Faculty Practice.

Diary responses in the paper group were lower than reported in other studies [25–29]. However, we made little effort to collect paper diary data with only one reminder per month. Our participants could have filled out multiple entries at once. While we know that text responses were within 1 day of the events and therefore less susceptible to recall bias, we do not know when paper diaries were filled out.

We calculated the number of runs and average run length to assess variability in the data, and we found that participants who made mandatory daily entries with text messages had more day-to-day variability in their responses. Their answers are a contemporaneous reflection of the bleeding they experienced. Nonetheless, the overall number of bleeding and nonbleeding days reported was similar for the two groups.

Our study required participants to log data daily for 90 days, which can be burdensome. Daily text message data submission may be more taxing for some women than daily logging with monthly submission of paper diaries. We observed lower response rates in those using paper diaries, however, over the course of the study period. Although there is ongoing benefit from a daily text message intervention through 6 months [30], the ideal time period for collecting patient-recorded via text message needs further study. Researchers should be cognizant of the possibility of diminishing study reactivity as study duration increases.

Operationalizing text message services can be difficult. A small portion of the population does not own cell phones, and more experience service interruptions. Such individuals could not use this methodology. Another option that these investigators have used is a separate study-specific data entry diary device. These proved to be expensive and a burden to participants and, in our experience, data entry was poor. In this study, we required participants to use their own cell phones, taking advantage of devices with which they were already familiar. However, one third of the women in the text message group experienced problems sending and receiving text messages. Some were straightforward to resolve, such as a participant not knowing how to answer a message or sending multiple responses when only one could be accepted. Other problems were more difficult to address, such as participants not receiving text messages. For some, their cell phone carriers did not successfully relay the messages from MIR3 to their phones. Others had exhausted their usage of text messages for their billing cycle. Our success in resolving these problems (~ 50%) often depended on our participants’ willingness to assist in that process.

The start-up cost of establishing data collection with text messages should be weighed against the salary (time and effort) devoted to increased follow-up with paper diaries. It is more cost efficient to program for an entire group over a defined period of time where the number of participants and approximate number of text messages to be sent is specified in advance. Although there is a large expense incurred from initial programming, per-message costs are low. As such, using text messages for data collection may best be suited for research purposes rather than in clinical practice. Costs may be prohibitive for small studies but acceptable for larger studies. Another advantage of this modality is direct data entry, which reduces the burden of transcription and associated errors.

Text messaging for daily data collection in a research setting is a promising modality. It is superior to paper diaries in quantity of data collected, particularly for those with less educational attainment. Text messaging effectively elimi- nates recall bias, but in this study, we could not identify any such bias among the women who provided nearly complete data. High start-up costs and potential technical difficulties require investment both in terms of funding and time; however, using text messages to collect data offers advantages that may be well worth the costs, particularly in research settings or when optimizing patient-reported outcomes.