Abstract
Objective:
Millions of infants worldwide remain under-immunized and at risk for unnecessary morbidity and mortality. Text messaging may offer a low-cost solution. We aimed to evaluate text message reminders to improve infant immunization in Guatemala.
Methods:
A randomized clinical trial was conducted at four public health clinics in rural and urban Guatemala. Infants ages six weeks to six months presenting for the first visit of the primary immunization series were randomly and equally allocated to an intervention or usual care group. Intervention participants were sent three text reminders before the second and third vaccine visits. The main outcome was timeliness of the second and third visits of the primary immunization series.
Results:
Of 1088 families approached for enrollment between March to November 2016, 871 were eligible and 720 (82.7%) participated; only 54 families did not own a cell phone. Due to country-wide vaccine shortages, visit completion was used as a proxy for overall immunization coverage. In intention to treat analysis, both intervention and usual care groups had high rates of visit completion, but intervention participants presented on the scheduled date more often (151 [42.2%] of 358 intervention vs. 111 [30.7%] of 362 usual care participants for visit 2, p=0.001, and 112 [34.0%] of 329 intervention vs. 90 [27.0%] of 333 usual care participants for visit 3, p=0.05). Intervention caregivers were significantly more likely to want to receive future text message reminders for vaccines and other appointments and were more willing to pay for these reminders.
Conclusion:
Caregivers who were sent text message reminders in urban and rural Guatemala were less delayed for their child’s immunization visits and reported high user satisfaction. Text message reminders may be an effective tool to increase infant vaccination coverage in low-income settings by reminding parents to vaccinate.
Trial Registration:
Keywords: childhood immunization, vaccine coverage, SMS, text messaging, reminder-recall, Guatemala
Introduction
Vaccination protects children from serious illness and long-term complications of vaccine-preventable diseases. Despite considerable effort to promote immunization, global coverage of routine infant vaccination has stagnated at around 86%, leaving an estimated 19.5 million infants under-immunized and at risk for unnecessary morbidity and mortality [1]. The Global Vaccine Action Plan (GVAP) goal to reach more than 90% of children worldwide continues to be a significant challenge, especially in many low- and middle-income countries (LMICs) where the majority of vaccine-preventable diseases occur [2]. Furthermore, the importance of the timeliness of immunizations to maximize protection and prevent disease outbreaks is now being recognized as a public health issue that should be given greater global emphasis [3–5].
While there are many potential reasons for undervaccination, a major barrier to delivering immunizations in LMICs is often the lack of a functioning preventive health service, which exacerbates demand-side barriers such as a lack of knowledge about the importance and timeliness of vaccines and either not knowing or forgetting when vaccines are due. Patient reminder systems can be an important mechanism for improving childhood preventive health coverage by informing parents when children are due for specific services. Reminder systems for immunizations have been shown to effectively improve immunization coverage in children, with most studies being done in high-income countries [6]. Despite these evidence-based interventions, such methods are often being under-utilized in lower-income settings [7,8].
Mobile technologies, such as Short Message Service (SMS) texts, have significant potential for improving health [9,10], including in LMICs [11,12]. Studies have shown that reminder systems for childhood immunizations can be adapted to SMS platforms. SMS reminders have been used in the United States for pediatric patients receiving vaccinations for influenza [13–15], MCV4/Tdap (meningococcal and tetanus, diphtheria, and pertussis) [16,17], HPV (human papillomavirus) [17,18], MMR (measles, mumps, and rubella) [19], and the infant primary immunization series [20]. Several recent studies have also been conducted in LMICs, including in Guatemala [21], Bangladesh [22], India [23], Pakistan [24], Kenya [25–27], Zimbabwe [28], and Vietnam [29]. Text messaging is emerging as a promising public health tool to improve childhood immunization programs with the potential for low-cost scalability, especially as global mobile phone access continues to increase.
Completion rates for the primary immunization series in Guatemala remain below the GVAP goal of 90%, with some rural areas having less than 70% coverage [30]. In collaboration with the Ministry of Public Health and Social Assistance (MPHSA) of Guatemala and the Pan American Health Organization (PAHO), our study team previously developed an SMS platform for immunization reminders hosted at the MPHSA and demonstrated the feasibility and acceptability of the system in Guatemala City [21]. In this study, we aimed to evaluate the preliminary efficacy of an optimized commercial SMS system to improve the completion and timeliness of the infant primary immunization series in geographically diverse urban and rural settings in Guatemala.
Material and Methods
Study design and participants
This randomized clinical trial was conducted at four public health clinics of the MPHSA in Guatemala. These government clinics serve a low-income population with two clinics located in the urban metropolitan setting of Guatemala City (Zona 11 and Villa Nueva) and two clinics located in the rural southwest region of the country (Colomba and Coatepeque, Quetzaltenango). Eligible participants included infants between the ages of six weeks to six months who presented for their first visit of the infant primary immunization series. At least one caregiver needed to self-report owning an active mobile phone capable of receiving SMS, be able to use SMS, and be literate and able to decipher the messages for themselves or by a surrogate in the household. Participants were excluded if they were not medically cleared to receive vaccines, the study clinic was not the patient’s primary clinic, the consenting caregiver was under 18 years of age, or the caregiver did not speak Spanish (Mayan languages are not commonly spoken in our study populations). Equal numbers of rural and urban participants were enrolled. Guatemala experienced significant political instability and corruption from 2015 to 2016 leading to unusually high levels of vaccine shortages during our study period. Two months after enrollment commenced, one of the initial rural study clinics (La Blanca) had to be administratively withdrawn from the study due to a complete lack of vaccine supplies. A demographically similar nearby rural health clinic (Colomba) was selected as a replacement study clinic and began enrollment in May 2016. Although the other study sites experienced intermittent vaccine shortages throughout the study period, these clinics maintained some vaccine supply for the primary immunization series at all times. The Colorado Multiple Institutional Review Board, Universidad del Valle Ethics Committee, and Guatemala National Ethics Committee of the MPHSA approved the research. Participation was voluntary and patients were not given any incentives. Written informed consent was obtained from all caregivers.
SMS intervention
The primary immunization series in Guatemala consists of three visits that are scheduled to occur when children are 2-months-old (visit 1: pentavalent [DTwP-Hib-HepB], pneumococcal [PCV13], poliomyelitis [IPV or bOPV], and rotavirus), 4-months-old (visit 2: pentavalent, pneumococcal, poliomyelitis, and rotavirus), and 6-months-old (visit 3: pentavalent and poliomyelitis). Pentavalent vaccine in Guatemala is a combination vaccine for diphtheria, tetanus, pertussis, Haemophilus influenza type B, and hepatitis B antigens. Participants were enrolled during visit 1 of the primary immunization series and then followed-up at visits 2 and 3. Scheduled visit dates were determined by the clinic nurses as per the usual standard of care and were written in the patient’s vaccination card. This meant that visits were scheduled during normal clinic operating hours generally two months after the current visit, but some visits were scheduled using minimum acceptable intervals between doses within a series for patients who were behind in receiving immunizations (similar to recommendations by the Advisory Committee on Immunization Practices in the United States). Vaccine shortages, which could not be predicted at the time of the visit, did not affect these scheduled return visit dates.
Following the enrollment visit, participants were allocated to either an intervention or usual care group using a computer-generated block randomization scheme (block size of 10) based on urban and rural clinic sites. Caregivers and study nurses were blinded to study group allocation at the initial enrollment visit. The statistician was blinded to the randomization. Based on input collected prior to the study commencement from caregivers of 2-month-old children at our study clinics, we planned that those randomized into the intervention group would receive the following text messages translated into Spanish at three, two, and one days before the next scheduled date for visits 2 and 3: “Your child [autopopulate child’s name] is due on [autopopulate date] at [autopopulate clinic name] for vaccines.” The local data manager entered each child’s name, scheduled visit date, clinic, and caregiver’s mobile number into a computer system connected to the SMS server both at enrollment and after visit 2. Automated SMS texts were generated using a customized computer-based software program hosted by the mobile telecommunications company Claro™. The commercial system cost $60 US dollars (USD) per month to send up to 1500 text messages. Participants did not need to have minutes on their phone or pay to receive messages. As part of the standard of care, clinic staff also printed the next vaccination visit date in the child’s government-issued immunization card for both intervention and usual care participants. All children were followed for at least eight months after enrollment. Study nurses were present at each clinic until eight months after the last patient was enrolled at that site. If a patient presented during this period, the study nurses included them in the follow-up.
Data collection
At enrollment (visit 1), all caregivers were consented and completed a baseline questionnaire designed by the research team that included information about maternal and paternal demographics, household characteristics, food insecurity using the Household Food Insecurity Access Scale [31], vaccine hesitancy using the WHO SAGE Working Group Vaccine Hesitancy Scale [32], parental depression scores using the Patient Health Questionnaire or PHQ-9 [33], and mobile phone access and usage characteristics. At visits 2 and 3, data were collected about mobile phone turnover, receipt of SMS reminders, adverse events following immunization, vaccines given, and the scheduled date of the next visit. A completion survey was administered to caregivers during the final study visit that included several questions related to caregiver attitudes regarding the SMS reminder intervention. A study nurse assigned to each clinic verbally administered the data collection surveys to caregivers in a quiet and confidential location at the clinic. Surveys were pretested by our study group in Guatemala. Study data were collected using a hand-held tablet device and managed using REDCap (Research Electronic Data Capture), a secure and web-based electronic data capture tool hosted at the University of Colorado [34]. Electronic immunization records reporting dates of vaccinations given were kept at each clinic and confirmed for children who returned to any of the four study sites at the end of the eight-month follow-up period for each patient. This electronic immunization record also allowed us to capture patients who may have been missed by the study nurse during their return visit. The mobile carrier report included the number and dates of SMS reminders sent to each participant but could not verify if messages were received or read by the caregiver. Messages listed in the mobile carrier report as “sent with error” or “error” were considered as not sent.
Statistical analysis
A sample size of 720 total participants (360 at each of the two geographical study sites) was calculated to provide us with 80% power to detect a difference of 10% (p1=.80 and p2=.90). Due to the unusually high levels of vaccine shortages experienced during our study, we used visit completion as a proxy for overall immunization rates given the difficulties in tracking the completion of specific vaccines that may not have been available when study participants presented for their initial, intended immunization visit (which was counted as their visit completion). We feel that this is an accurate approximation since we found low levels of previous vaccine refusal and delayed vaccination in our population using a vaccine hesitancy survey included with our baseline questionnaire [35]. In fact, according to this survey, no parent in our study had ever refused a vaccination, and only eight parents (1.1%) had been reluctant or hesitated to get a vaccination for their children. The primary outcomes for this study were completion and timeliness of visits 2 and 3 of the primary immunization series. Secondary outcomes included feasibility and acceptability results related to the proportion of caregivers who were sent SMS messages and caregiver satisfaction with the intervention. Descriptive statistics were used for baseline demographic characteristics. Chi-square test or Fisher’s exact were used to test for differences between categorical data and t-tests were used for continuous data. Both per protocol and intention-to-treat analyses were done. Per protocol analyses included both intervention patients who were sent at least one SMS as well as all three SMS reminders within the three days prior to the scheduled date for visits 2 and 3. All analyses were conducted using SAS (SAS 9.4, SAS Institute, Cary, NC).
Results
Of 1088 children screened for eligibility from March 1 to November 30, 2016, 871 were eligible and 720 (82.7%) participated (Figure 1). Nearly all caregivers screened for eligibility owned a mobile phone capable of SMS, were able to use SMS on their phone, and were literate and able to decipher SMS messages (n=1011, 92.9%). Table 1 summarizes the baseline demographic characteristics of the study participants at enrollment. Characteristics of vaccine hesitancy [35] and mobile phone access and usage [36] are described elsewhere. Three hundred and fifty-eight participants were allocated to the intervention group and 362 to the usual care group. There were no significant differences between the baseline demographics of intervention and usual care children and their caregivers.
Figure 1.
Trial profile
Table 1.
Baseline demographic characteristics at enrollment by treatment and study cohort
| Characteristics | Usual care n=362 n(%) | Intervention n=358 n(%) | p-valuea | Study cohort n=720 n (%) |
|---|---|---|---|---|
| CHILD | ||||
| Child’s age at enrollment (days) | ||||
| Mean (SD) | 71.8 (±17.8) | 71.4 (±17.6) | .78 | 71.6 (±17.7) |
| Child’s gender | ||||
| Male | 172 (47.5) | 186 (52.0) | .23 | 358 (49.7) |
| Female | 190 (52.5: | 172 (48.0) | 362 (50.3) | |
| MOTHER | ||||
| Mother’s age (years) | ||||
| Mean (SD) | 24.8 (±5.5) | 24.7 (±5.5) | .84 | 24.8 (±5.5) |
| Mother is able to read and write | ||||
| No | 9 (2.5) | 7 (2.0) | .63 | 16 (2.2) |
| Yes | 353 (97.5) | 351 (98.0) | 704 (97.8) | |
| Mother’s education | ||||
| No education | 4 (1.1) | 6 (1.7) | .93 | 10 (1.4) |
| Completed or some primary education | 92 (25.4) | 92 (25.7) | 184 (25.6) | |
| Completed or some secondary education | 88 (24.3) | 85 (23.7) | 173 (24.0) | |
| Completed or some higher education | 178 (49.2) | 175 (48.9) | 353 (49.0) | |
| Mother’s employment status | ||||
| Unemployed | 261 (72.1) | 253 (70.7) | .67 | 514(70.7) |
| Employed | 101 (27.9) | 105 (29.3) | 206 (28.6) | |
| Mother owns a mobile phoneb | ||||
| No | 28 (7.8) | 16 (4.5) | .07 | 44 (6.1) |
| Yes | 333 (92.2) | 342 (95.5) | 675 (93.9) | |
| FATHERc | ||||
| Father is involved in child care | ||||
| No | 54 (14.9) | 65 (18.2) | .24 | 119 (16.5) |
| Yes | 308 (85.1) | 293 (81.8) | 601 (83.5) | |
| Father’s age (years) | ||||
| Mean (SD) | 28.5 (±7.3) | 28.2 (±6.2) | .59 | 28.4 (±6.8) |
| Father is able to read and write | ||||
| No | 4 (1.3) | 6 (2.0) | .54 | 10 (1.7) |
| Yes | 304 (98.7) | 287 (98.0) | 591 (98.3) | |
| Father’s education | ||||
| No education | 4 (1.3) | 5 (1.7) | .77 | 9 (1.5) |
| Completed or some primary education | 50 (16.2) | 53 (18.1) | 103 (17.1) | |
| Completed or some secondary education | 71 (23.1) | 73 (24.9) | 144 (24.0) | |
| Completed or some higher education | 183 (59.4) | 162 (55.3) | 345 (57.4) | |
| Father’s employment | ||||
| Unemployed | 3 (1.0) | 5 (1.7) | .50 | 8 (1.3) |
| Employed | 305 (99.0) | 288 (98.3) | 593 (98.7) | |
| Father owns a mobile phoneb | ||||
| No | 105 (35.8) | 116 (42.2) | .12 | 221 (38.9) |
| Yes | 188 (54.2) | 159 (57.8) | 347 (61.1) | |
| HOSEHOLD | ||||
| Number of children | ||||
| Mean (SD) | 1.9 (±1.0) | 1.9 (±1.0) | .63 | 1.9 (±1.0) |
| Family monthly income leveld | ||||
| ≤Q1000 | 102 (28.2) | 105 (29.3) | .78 | 207 (28.8) |
| Q1001 – 2000 | 75 (20.7) | 80 (22.4) | 155 (21.5) | |
| Q2001 – 3000 | 81 (22.4) | 85 (23.7) | 166 (23.1) | |
| Q3001 – 4000 | 57 (15.8) | 46 (12.9) | 103 (14.3) | |
| ≥Q4001 | 47 (13.0) | 42 (11.7) | 89 (12.4) | |
| Landline present in home | ||||
| No | 265 (73.2) | 274 (76.5) | .30 | 539 (74.9) |
| Yes | 97 (26.8) | 84 (23.5) | 181 (25.1) | |
| Family owns a radio | ||||
| No | 103 (28.5) | 114 (31.8) | .32 | 217 (30.1) |
| Yes | 259 (71.5) | 244 (68.2) | 503 (69.9) | |
| Family owns a television | ||||
| No | 28 (7.7) | 25 (7.0) | .70 | 53 (7.4) |
| Yes | 334 (92.3) | 333 (93.0) | 667 (92.6) | |
| Family owns a refrigerator | ||||
| No | 95 (26.2) | 108 (30.2) | .24 | 203 (28.2) |
| Yes | 267 (73.8) | 250 (69.8) | 517(71.8) | |
| Family owns an electric/gas stove | ||||
| No | 42 (11.6) | 37 (10.3) | .59 | 79 (11.0) |
| Yes | 320 (88.4) | 321 (89.7) | 641 (89.0) | |
| Family owns a car | ||||
| No | 242 (66.9) | 255 (71.2) | .20 | 497 (69.0) |
| Yes | 120 (33.1) | 103 (28.8) | 223 (31.0) | |
| Number of amenitiese | ||||
| ≤2 | 63 (17.4) | 68 (19.0) | .36 | 131 (18.2) |
| 3 | 81 (22.4) | 96 (26.8) | 177 (24.6) | |
| 4 | 123 (34.0) | 115 (32.1) | 238 (33.1) | |
| 5 | 95 (26.2) | 79 (22.1) | 174 (24.2) | |
| Food insecurity scoref | ||||
| 0 | 290 (80.1) | 282 (78.8) | .57 | 572 (79.4) |
| 1 | 18 (5.0) | 28 (7.8) | 46 (6.4) | |
| 2 | 7 (1.9) | 6 (1.7) | 13 (1.8) | |
| 3 | 10 (2.8) | 11 (3.1) | 21 (2.9) | |
| ≥4 | 37 (10.2) | 31 (8.7) | 68 (9.4) | |
| Depression scoreg | ||||
| Minimal/no depression (0–4) | 345 (95.3) | 338 (94.4) | .59 | 683 (94.9) |
| Mild depression (5–9)h | 17 (4.7) | 20 (5.6) | 37 (5.1) |
Chi-square test or Fisher’s exact for category and t-test for continuous variables.
One participant (usual care =1) left this answer blank for mothers, and thirty-three participants (usual care = 15, intervention = 18) responded “I don’t know” for fathers.
Father characteristics include a total of n = 601.
Q = Guatemalan Quetzal (Q1 = $0.14 USD).
Amenities scale includes ownership of radio, television, refrigerator, stove, and car.
Food insecurity is measured with the Household Food Insecurity Access Scale (score 0 – 27). Higher scores equal more food insecurity.
Depression is measured with the PHQ-9 (score 0 – 27).
One individual had a PHQ-9 score of 10 but was added into the mild depression group for analysis.
According to the mobile carrier report (Table 2), a total of 842 SMS reminder messages were sent prior to visit 2 and 652 prior to visit 3. Most of the intervention caregivers were sent at least one SMS reminder prior to visit 2 (n=301, 84.1%) and visit 3 (n=239, 72.6%), with most caregivers sent the intended three messages (n=259, 72.4% for visit 2 and n=194, 59.0% for visit 3). Several messages (n=20 for visit 2 and n=25 for visit 3) were listed in the mobile carrier report as “sent with error” or “error” and were, therefore, considered as not sent.
Table 2.
Number (%) of intervention parents sent text message reminders
| Number of text message reminders sent | Visit 2, (n=358) n(%) | Visit 3, (n=329) n(%) |
|---|---|---|
| 0 | 57 (15.9) | 90 (27.4) |
| 1 | 19 (5.3) | 20 (6.1) |
| 2 | 23 (6.4) | 25 (7.6) |
| 3 | 259 (72.4) | 194 (59.0) |
In intention-to-treat analysis (Table 3), both intervention and usual care groups had similarly high rates of visit completion for visits 2 and 3 (i.e. 88–92%). Intervention infants presented earlier for their scheduled visits compared to usual care participants, with more intervention children presenting on the scheduled visit date for both visit 2 (42.2% vs. 30.7%, p=.001) and visit 3 (34.0% vs. 27.0%, p=.05) and within the first week following the scheduled visit date for visit 2 (71.0% vs. 63.5%, p=.03). Models based on a cluster analysis that controlled for clinic level variation, including urban versus rural clinic sites, had no effect on the outcomes and, therefore, are not included in this manuscript. There was no dose-response seen between the number of messages sent and visit completion rates or visit timeliness among the intervention participants. Per protocol analyses (Appendix) were consistent with intention-to-treat.
Table 3.
Completion rates and timeliness for visits 2 and 3 of the primary immunization series
| VISIT 2 | VISIT 3a | |||||||
|---|---|---|---|---|---|---|---|---|
| Total | Usual care | Intervention | Total | Usual care | Intervention | |||
| n(%) | n(%) | n(%) | P-value | n(%) | n(%) | n(%) | P-value | |
| Intention-to-treat | (n=720) | (n=362) | (n=358) | (n=66) | (n=33) | (n=329) | ||
| Visit was on scheduled date | 262 (36.4) | 111 (30.7) | 151 (42.2) | .001 | 202 (30.5) | 90 (27.0) | 112 (34.0) | .05 |
| Visit was up to 7 days late | 484 (67.2) | 230 (63.5) | 254 (71.0) | .03 | 392 (59.2) | 191 (57.4) | 201 (61.1) | .33 |
| Visit was up to 14 days late | 550 (76.4) | 269 (74.3) | 281 (78.5) | .19 | 451 (68.1) | 225 (67.6) | 226 (68.7) | .76 |
| Visit was up to 30 days late | 599 (83 2) | 296 (81.8) | 303 (84.6) | .30 | 509 (46.9) | 256 (76.9) | 253 (76.9) | .99 |
| Visit was completed | 663 (92.1) | 333 (92.0) | 330 (92.2) | .92 | 586 (88.5) | 297 (89.2) | 289 (87.8) | .59 |
| Sent at least 1 text reminderb | (n=663) | (n=362) | (n=301) | (n=572) | (n=333) | (n=239) | ||
| Visit was on scheduled date | 243 (36.7) | 111 (30.7) | 132 (43.9) | <.001 | 179 (31.3) | 90 (27.0) | 89 (37.2) | .01 |
| Visit was up to 7 days late | 446 (67.3) | 230 (63.5) | 216 (71.8) | .02 | 347 (60.7) | 191 (57.4) | 156 (65.3) | .06 |
| Visit was up to 14 days late | 507 (76.5) | 269 (74.3) | 238 (79.1) | .15 | 400 (69.9) | 225 (67.6) | 175 (73.2) | .15 |
| Visit was up to 30 days late | 552 (83.3) | 296 (81.8) | 256 (85.1) | .26 | 449 (78.5) | 256 (76.9) | 193 (80.8) | .27 |
| Visit was completed | 613 (92.5) | 333 (92.0) | 280 (93.0) | .62 | 510 (89.2) | 297 (89.2) | 213 (89.1) | .98 |
Individuals who did not have a visit 2 and therefore did not receive a visit 3 scheduled date (n=29 usual care and n=28 intervention) or the visit 3 scheduled date was missing despite having a visit 2 (n=1 for intervention) were excluded from the visit 3 analysis since there was no scheduled vaccine visit date for these participants.
Visits for the intervention group that were not sent at least one text reminder prior to the scheduled visit date were excluded from the given visit analysis (n=57 for visit 2 and n=90 for visit 3).
Most caregivers completed follow-up surveys at both visits 2 (n=606, 91.4%) and 3 (n=524, 89.4%). Children were almost always brought to clinic by their mother (n=586, 96.7% for visit 2 and n=514, 98.1% for visit 3). Most participants reported that their mobile phone number did not change between visits (n=554, 91.7% for visit 2 and n=481, 92.1% for visit 3). Most intervention caregivers remembered receiving SMS messages (n=216, 71.1% for visit 2 and n=169, 64.3% for visit 3) with nearly all saying that the messages helped them remember the appointment (n=213, 98.6% for visit 2 and n=168, 99.4% for visit 3). No serious adverse events following immunization were reported.
Completion survey response rates were 73.5% (n=263) for intervention caregivers and 71.3% (n=258) for usual care. Intervention caregivers had more favorable views towards SMS reminders than those in the usual care group (Table 4). On a 4-point ordinal scale, all intervention caregivers reported that the SMS reminders in the study were either “very helpful” (n= 121, 58.7%) or “helpful” (n=85, 41.3%) in reminding them of their child’s vaccination visit and they would be either “very likely” (n=103, 45.2%) or “likely” (n=124, 54.4%) to recommend SMS vaccination reminders to friends and family (one person replied “not likely” to this question).
Table 4.
Caregiver attitudes regarding the reminder intervention
| Completion Survey Questions | Usual care (n=258) % (n) | Intervention (n=263) % (n) | p-valuea |
|---|---|---|---|
| How helpful would it be for you to receive text message reminders for vaccination appointments in the future? | |||
| Very helpful | 28.7 (74) | 49.1 (129) | <.001 |
| Helpful | 67.8 (175) | 49.8 (131) | |
| Not helpful | 3.5 (9) | 1.1 (3) | |
| Not at all helpful | 0 | 0 | |
| How willing would you be to pay for a text message to be reminded about your child’s vaccinations? | |||
| Very willing | 8.5 (22) | 20.9 (55) | <.001 |
| Willing | 45.0 (116) | 47.9 (126) | |
| Not willing | 28.7 (74) | 21.3 (56) | |
| Not at all willing | 17.8 (46) | 9.9 (26) | |
| How much would you be willing to pay per month for a phone application to remind you about vaccine appointments for your child?b | |||
| <Q10 | |||
| Q10 – Q30 | 86.1 (222) | 78.3 (206) | .13 |
| Q31 - Q50 | 2.3 (6) | 3.8 (10) | |
| >Q50 | 3.1 (8) | 3.8 (10) | |
| 8.5 (22) | 14.1 (37) | ||
| How helpful would it be for you to receive text message reminders for other things in the future, such as reminders for other clinic appointments or to take medications? | |||
| Very helpful | 31.0 (80) | 46.0 (121) | <.001 |
| Helpful | 62.8 (162) | 52.1 (137) | |
| Not helpful | 6.2 (16) | 1.9 (5) | |
| Not at all helpful | 0 | 0 | |
| What is the best time of the day to receive text message reminders? | |||
| Night (12 am - 6 am) | 2 7 (7) | 1.9 (5) | .56 |
| Morning (6 am - 12 pm) | 77.9 (201) | 74.9 (197) | |
| Afternoon (12 pm - 6 pm) | 14.0 (36) | 18.3 (48) | |
| Evening (6 pm - 12 am) | 5.4 (14) | 4.9 (13) |
Fisher’s Exact test
Q=Guatemalan Quetzal (Q1 = $0.14 USD)
Discussion
We evaluated the effectiveness of SMS reminders to caregivers to improve the completion and timeliness of infant immunization visits in urban and rural settings in Guatemala. Our study results show similarly high rates of visit completion for both the intervention and usual care groups, but intervention participants presented earlier for their scheduled visits and expressed high parental satisfaction with the SMS reminders. Almost all caregivers assessed in this low-resource setting had access to a mobile phone with SMS capability and were able to use text messaging. Additionally, the intervention was low-cost, creating the potential for future scalability, especially if clinic staff could be trained to use the SMS platform. The program could be further utilized to share important information about child wellness and to alert caregivers to vaccine shortages as well as episodic opportunities for vaccine completion such as national immunization days or health fairs. This is one of the largest randomized controlled trials to evaluate the efficacy of text message reminders on improving immunization coverage in both urban and rural populations in an LMIC and, to our knowledge, we are the only study group to test this in a Latin American country, a region that enjoys high overall coverage for immunizations.
Immunization schedules are designed to provide optimal safety and protection against infection at the most vulnerable ages. While achieving high immunization coverage is needed to protect children from vaccine-preventable infections, age-appropriate vaccination (i.e. vaccination as early as possible according to the immunization schedule) is also important for several reasons. Late administration of vaccines results in a longer period of susceptibility to infections, which places children at risk for specific vaccine-preventable diseases and outbreaks and is particularly important with the cumulative effect of being late for each subsequent visit of the primary immunization series [37–42]. Age-appropriate vaccination also has important administrative implications for clinics and government programs as they plan for vaccine supplies and services, especially for vaccines such as rotavirus that are only offered during specific age-ranges. Standard measures estimating population vaccination coverage are commonly based on up-to-date completion rates at a specific age, without regard to the timing of the vaccinations and the degree to which children experience delays prior to reaching the predetermined age for which they are labeled “up-to-date”. It has been argued that relying on up-to-date vaccination status masks these important delays relative to age-appropriate standards and understates the degree of underimmunization rates within a population [43–45]. For example, recent studies have found high up-to-date childhood vaccination coverage in many LMICs but considerably lower rates of age-appropriate vaccination [4,5,46–49]. Age-appropriate vaccination, therefore, is an important indicator to both assess and promote, especially in LMICs where the majority of vaccine-preventable diseases occur.
SMS reminder platforms that improve the timeliness of childhood vaccinations, such as ours, could have far-reaching implications in providing children with optimal protection against infections at the youngest and most vulnerable age possible. While there is growing evidence to support SMS applications for childhood vaccine promotion, most research has been done in high-income countries, sample sizes have been small, and study designs have varied. The evaluation of these interventions in LMICs has generally not been done consistently or rigorously and, to date, only limited evidence of effectiveness exists. A few small randomized controlled trials (RCTs) have recently shown promising results for SMS reminders for childhood vaccines in Zimbabwe [28], Pakistan [24], and our own initial study in Guatemala [21], while an evaluation in Kenya showed that SMS reminders decreased dropout rates for the third dose of pentavalent vaccine [26]. Additionally, a quasi-experimental study in Bangladesh [22] and an evaluation in Vietnam [29] both demonstrated that a digital immunization registry capable of sending text reminders could improve vaccine coverage. Two recent studies also found improved immunization coverage using SMS reminders coupled with compliance-linked incentives, including a pragmatic RCT in India [23] using a phone credit incentive and a cluster RCT in Kenya [27] using monetary incentives. Our results are similar to the Mobile Solutions for Immunization (M-SIMU) trial in Kenya [27] where high baseline immunization coverage existed and the SMS intervention resulted in modest improvements in immunization timeliness. However, the most significant findings from the Kenyan study coupled SMS reminders with monetary incentives that could present cost implications prohibiting future scalability and replication in other LMIC settings.
Our study has important limitations. The significant political instability that led to unusually high levels of vaccine shortages affected our study and our ability to track specific vaccine completions as described in the Material and Methods section. We do not know whether these shortages had an impact on the likelihood of caregivers bringing infants to clinic for vaccinations or their decision to adhere to the primary immunization schedule. Even if reminded, some caregivers may not have brought their infants to clinic on the scheduled date if they thought there were insufficient vaccine supplies. We anticipate that the effects of reminding people to adhere to a vaccine schedule will be diminished in an environment where shortages are common. Additionally, we did not see an improvement in overall visit completion rates with our intervention. This may have been partly because our study population had higher baseline immunization coverage than we were adequately powered to assess. These higher than expected completion rates were likely due in part to a selection bias in enrolling children as they presented for their first immunization visit, which would have unintentionally excluded children who either presented significantly delayed or not at all for vaccines. Enrolling infants in the hospital shortly after delivery would likely be more representative of the population but would have required additional resources that were outside the scope of this study. Another limitation was that while most of our intervention participants were sent all three reminder texts, we do not know why several patients were not sent any messages according to the mobile carrier report and why fewer messages were sent prior to visit 3 than visit 2. The automated SMS system was subject to user error where incorrectly entered visit dates or telephone numbers by either the study nurses or data manager may have caused some intervention participants to be sent messages after the visit or to the wrong phone. Unsent or lost messages may have been an additional problem with the server. Future optimization of the SMS system will be critically important and will likely result in greater efficacy of the intervention. Furthermore, the MPHSA of Guatemala has developed an electronic immunization registry that is currently functional at the clinic level. Immunization records were confirmed for patients presenting to our study clinics through this registry. However, if caregivers brought their child to another non-study clinic, the visit was unable to be recorded in our study. Population-based immunization registries have been suggested as a means to improve age-appropriate vaccination and could be linked to SMS platforms for even better efficacy [22,29,43]. This will be important to study further once a national immunization registry is successfully implemented.
There are several important concepts that will be important to explore in future studies. For example, we did not ask about shared mobile telephones among family members. In our completion survey, 30–40% of intervention caregivers who brought their child to clinic did not recall receiving a text message reminder, which may have been partly due to the sharing of phones. This will be an important area for future exploration since phone sharing could have significant implications for whether SMS reminder interventions are reaching the intended recipients. While bidirectional messaging has been shown to improve some mHealth intervention results [10], we did not use this in our study because it would have required participants to have minutes on their phone and pay to send a message. Finally, we did not explore the ownership and use of smartphones, which could be used with Multimedia Message Service (MMS) technology or free mobile-messaging applications (or apps) such as WhatsApp, Kik, Snapchat, and Facebook Messenger. Smartphone apps will be important to examine in future studies [50].
Conclusions
We found that SMS reminders led to significant, albeit modest, improvements in age-appropriate childhood immunization visit completion in both urban and rural Guatemala, even in the setting of high baseline immunization rates and unusually high levels of vaccine shortages. Timely adherence to the recommended schedule for the primary immunization series has important implications for the effectiveness, safety, and ultimate success of childhood immunization programs. SMS technology may offer a high-reach and low-cost way to improve adherence to the childhood vaccination schedule by reminding caregivers when their children are due for immunizations. Future studies will be needed to evaluate the impact of text messaging reminders on improving childhood vaccination in other LMIC settings and with regard to large-scale implementation.
Highlights.
A novel SMS vaccine reminder platform was optimized and implemented in a LMIC
The SMS vaccine reminder platform was evaluated in both urban and rural settings
SMS reminders led to significant improvements in visit timeliness
SMS vaccine reminders were acceptable to use in a LMIC with high user satisfaction
SMS vaccine reminders have the potential for widespread scalability at low cost
Acknowledgments:
We thank the caregivers and children who participated in our study. We also thank our data managers, project administrators, and study nurses. We greatly appreciate the support and collaboration of the Ministry of Public Health and Social Assistance in Guatemala.
Funding: This work was supported by the Eunice Kennedy Shriver National Institute of Child Health & Human Development at the National Institutes of Health [grant number 1R21HD084115–01] and the Thrasher Research Fund Early Career Award Program [award number 12797]. REDCap was supported by the National Institutes of Health and National Center for Research Resources Colorado CTSI [grant number UL1 RR025780]. The funders had no involvement in the study design and conduct, data collection and analysis, interpretation of the data, decision to publish, or preparation of the manuscript.
Appendix.
Completion rates and timeliness for visits 2 and 3 of the primary immunization series by per protocol study groups
| VISIT 2 | VISIT 3 | |||||||
|---|---|---|---|---|---|---|---|---|
| Total | Usual care | Intervention | Total | Usual care | Intervention | |||
| n(%) | n(%) | n(%) | P-value | n | n(%) | n (%) | P-value | |
| At least 1 text reminder per protocola | (n=655) | (n=362) | (n=293) | (n=557) | (n=333) | (n=224) | ||
| Visit was on schedule date | 237 (36.2) | 111 (30.7) | 126 (43.0) | .001 | 177 (31.8) | 90 (27.0) | 87 (38.8) | .003 |
| Visit was up to 7 days late | 439 (67.0) | 230 (63.5) | 209 (71.3) | .03 | 340 (61.0) | 191 (57.4) | 149 (66.5) | .03 |
| Visit was up to 14 days late | 500 (76.3) | 269 (74.3) | 231 (78.8) | .18 | 393 (70.6) | 225 (67.6) | 168 (75.0) | .06 |
| Visit was up to 30 days late | 545 (83.2) | 296 (81.8) | 249 (85.0) | .27 | 437 (78.5) | 256 (76.9) | 181 (80.8) | .27 |
| Visit was complete | 606 (92.5) | 333 (92.0) | 273 (93.1) | .57 | 497 (89.2) | 297 (89.2) | 200 (89.3) | .97 |
| All three text reminders per protocolb | (n=611) | (n=362) | (n=249) | (n=514) | (n=333) | (n=181) | ||
| Visit as on schedule date | 223 (36.5) | 111 (30.7) | 112 (45.0) | <.001 | 159 (30.9) | 90 (27.0) | 69 (38.1) | .009 |
| Visit was up to 7 days late | 415 (67.9) | 230 (63.5) | 185 (74.3) | .005 | 310 (60.3) | 191 (57.4) | 119 (65.8) | .06 |
| Visit was up to 14 days late | 470 (76.9) | 269 (74.3) | 201 (80.7) | .06 | 359 (69.8) | 225 (67.6) | 134 (74.0) | .13 |
| Visit was up to 30 days late | 510 (83.5) | 296 (81.8) | 214 (85.9) | .17 | 402 (78.2) | 256 (76.9) | 146 (80.7) | .32 |
| Visit was complete | 565 (92.5) | 333 (92.0) | 232 (93.2) | .59 | 459 (89.3) | 297 (89.2) | 162 (89.5) | .91 |
Visits for the intervention group that were not sent at least one text per protocol were excluded from the given visit analysis (n=65 for visit 2 and n=105 for visit 3).
Visits for the intervention group that were not sent all three texts per protocol were excluded from the given visit analysis (n=109 for visit 2 and n=148 for visit 3).
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Declarations of interest: None.
Declaration of interests
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
References
- [1].WHO. 2017 Assessment Report of the Global Vaccine Action Plan: Strategic Advisory Group of Experts on Immunization. Geneva: World Health Organization; 2017. Available from: https://www.who.int/immunization/global_vaccine_action_plan/sage_assessment_reports/en/ [accessed 15 September 2018]. [Google Scholar]
- [2].WHO. Global Vaccine Action Plan: 2011–2020. Geneva: World Health Organization; 2013. Available from: https://www.path.org/resources/global-vaccine-action-plan-2011-2020/ [accessed 5 October 2018]. [Google Scholar]
- [3].Luman ET, Barker LE, Shaw KM, McCauley MM, Buehler JW, Pickering LK. Timeliness of childhood vaccinations in the United States: days undervaccinated and number of vaccines delayed. JAMA 2005;293(10):1204–11. 10.1001/jama.293.10.1204. [DOI] [PubMed] [Google Scholar]
- [4].Clark A, Sanderson C. Timing of children’s vaccinations in 45 low-income and middle-income countries: an analysis of survey data. Lancet 2009;373(9674):1543–9. 10.1016/S0140-6736(09)60317-2. [DOI] [PubMed] [Google Scholar]
- [5].Akmatov MK, Mikolajczyk RT. Timeliness of childhood vaccinations in 31 low and middle-income countries. J Epidemiol Community Health 2012;66(7):e14 10.1136/jech.2010.124651. [DOI] [PubMed] [Google Scholar]
- [6].Jacobson Vann JC, Jacobson RM, Coyne-Beasley T, Asafu-Adjei JK, Szilagyi PG. Patient reminder and recall interventions to improve immunization rates. Cochrane Database Syst Rev 2018;1:CD003941 10.1002/14651858.CD003941.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [7].Tierney CD, Yusuf H, McMahon SR, Rusinak D, O’Brien MA, Massoudi MS, et al. Adoption of reminder and recall messages for immunizations by pediatricians and public health clinics. Pediatrics 2003;112(5):1076–82. 10.1542/peds.112.5.1076. [DOI] [PubMed] [Google Scholar]
- [8].Oyo-Ita A, Wiysonge CS, Oringanje C, Nwachukwu CE, Oduwole O, Meremikwu MM. Interventions for improving coverage of childhood immunisation in low- and middle-income countries. Cochrane Database Syst Rev 2016;7:CD008145 10.1002/14651858.CD008145.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [9].Head KJ, Noar SM, Iannarino NT, Harrington NG. Efficacy of text messaging-based interventions for health promotion: a meta-analysis. Soc Sci Med 2013;97:41–8. 10.1016/j.socscimed.2013.08.003. [DOI] [PubMed] [Google Scholar]
- [10].Hall AK, Cole-Lewis H, Bernhardt JM. Mobile text messaging for health: a systematic review of reviews. Annu Rev Public Health 2015;36:393–415. 10.1146/annurev-publhealth-031914-122855. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [11].Déglise C, Suggs LS, Odermatt P. Short message service (SMS) applications for disease prevention in developing countries. J Med Internet Res 2012;14(1):e3 10.2196/jmir.1823. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [12].Gurman TA, Rubin SE, Roess AA. Effectiveness of mHealth behavior change communication interventions in developing countries: a systematic review of the literature. J Health Commun 2012;17(Suppl 1):82–104. 10.1080/10810730.2011.649160. [DOI] [PubMed] [Google Scholar]
- [13].Stockwell MS, Kharbanda EO, Martinez RA, Vargas CY, Vawdry DK, Carmargo S. Effect of a text messaging intervention on influenza vaccination in an urban, low-income pediatric and adolescent population. JAMA 2012;307(16):1702–8. 10.1001/jama.2012.502. [DOI] [PubMed] [Google Scholar]
- [14].Hofstetter AM, Vargas CY, Camargo S, Holleran S, Vawdrey DK, Kharbanda EO, et al. Impacting delayed pediatric influenza vaccination: a randomized controlled trialof text message reminders. Am J Prev Med 2015;48(4):392–401. 10.1016/j.amepre.2014.10.023. [DOI] [PubMed] [Google Scholar]
- [15].Stockwell MS, Hofstetter AM, DuRivage N, Barrett A, Fernandez N, Vargas CY, et al. Text message reminders for second dose of influenza vaccine: a randomized controlled trial. Pediatrics 2015;135(1):e83–91. 10.1542/peds.2014-2475. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [16].Stockwell MS, Kharbanda EO, Martinez RA, Lara M, Vawdrey D, Natarajan K, et al. Text4Health: impact of text message reminder-recalls for pediatric and adolescent immunizations. Am J Public Health 2012;102(2):e15–21. 10.2105/AJPH.2011.300331. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [17].O’Leary ST, Lee M, Lockhart S, Eisert S, Furniss A, Barnard J, et al. Effectiveness and cost of bidirectional text messaging for adolescent vaccines and well care. Pediatrics 2015;136(5):e1220–7. 10.1542/peds.2015-1089. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [18].Kharbanda EO, Stockwell MS, Fox HW, Andres R, Lara M, Rickert VI. Text message reminders to promote human papillomavirus vaccination. Vaccine 2011;29(14):2537–41. 10.1016/j.vaccine.2011.01.065. [DOI] [PubMed] [Google Scholar]
- [19].Hofstetter AM, DuRivage N, Vargas CY, Camargo S, Vawdrey DK, Fisher A, et al. Text message reminders for timely routine MMR vaccination: a randomized controlled trial. Vaccine 2015;33(43):5741–6. 10.1016/j.vaccine.2015.09.042. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [20].Ahlers-Schmidt CR, Chesser AK, Nguyen T, Brannon J, Hart TA, Williams KS, et al. Feasibility of a randomized controlled trial to evaluate Text Reminders for Immunization Compliance in Kids (TRICKs). Vaccine 2012;30(36):5305–9. 10.1016/j.vaccine.2012.06.058. [DOI] [PubMed] [Google Scholar]
- [21].Domek GJ, Contreras IL, O’Leary ST, Bull S, Furniss A, Kempe A, et al. SMS text message reminders to improve infant vaccination coverage in Guatemala: a randomized controlled trial. Vaccine 2016;34(21):2437–43. 10.1016/j.vaccine.2016.03.065. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [22].Uddin MJ, Shamsuzzaman M, Horng L, Labrique A, Vasudevan L, Zeller K, et al. Use of mobile phones for improving vaccination coverage among children living in rural hard-to-reach areas and urban streets of Bangladesh. Vaccine 2016;34(2):276–83. 10.1016/j.vaccine.2015.11.024. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [23].Seth R, Akinboyo I, Chhabra A, Qaiyum Y, Shet A, Gupte N, et al. Mobile phone incentives for childhood immunizations in rural India. Pediatrics 2018;141(4):e20173455 10.1542/peds.2017-3455. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [24].Kazi AM, Ali M, Zubair K, Kalimuddin H, Kazi AN, Iqbal SP, et al. Effect of mobile phone text message reminders on routine immunization uptake in pakistan: randomized controlled trial. JMIR Public Health Surveill 2018;4(1):e20 10.2196/publichealth.7026. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [25].Wakadha H, Chandir S, Were EV, Rubin A, Obor D, Levine OS, et al. The feasibility of using mobile phone based SMS reminders and conditional cash transfers to improve timely immunization in rural Kenya. Vaccine 2013;31(6):987–93. 10.1016/j.vaccine.2012.11.093. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [26].Haji A, Lowther S, Ngan’ga Z, Gura Z, Tabu C, Sandhu H, et al. Reducing routine vaccination dropout rates: evaluating two interventions in three Kenyan districts, 2014. BMC Public Health 2016;16:152 10.1186/s12889-016-2823-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [27].Gibson DG, Ochieng B, Kagucia EW, Were J, Hayford K, Moulton LH, et al. Mobile phone-delivered reminders and incentives to improve childhood immunisation coverage and timeliness in Kenya (M-SIMU): a cluster randomised controlled trial. Lancet Glob Health 2017;5(2):e428–38. 10.1016/S2214-109X(17)30072-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [28].Bangure D, Chirundu D, Gombe N, Marufu T, Mandozana G, Tshimanga M, et al. Effectiveness of short message services reminder on childhood immunization programme in Kadoma, Zimbabwe – a randomized controlled trial, 2013. BMC PublicHealth 2015;15:137 10.1186/s12889-015-1470-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [29].Nguyen NT, Vu HM, Dao SD, Tran HT, Nguyen TXC. Digital immunization registry: evidence for the impact of mHealth on enhancing the immunization system and improving immunization coverage for children under one year old in Vietnam. mHealth 2017;3:26 10.21037/mhealth.2017.06.03. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [30].Minesterio de Salud Pública y Asistencia Social, Instituto Nacional de Estadística, Secretaría de Planificación y Programación de la Presidencia. VI Encuesta Nacional de Salud Materno Infantil: ENSMI 2014–2015 Informe de Indicadores Básicos. Guatemala, 2015. Available from: http://www.osarguatemala.org/osartemporal/Archivos/PDF/201603/259_4.pdf [accessed 1 August 2018]. [Google Scholar]
- [31].Coates J, Swindale A, Bilinsky P. Household Food Insecurity Access Scale (HFIAS) for Measurement of Food Access: Indicator Guide (v. 3). Washington, D.C.: Food and Nutrition Technical Assistance Project, Academy for Educational Development; August 2007. Available from: http://www.fao.org/fileadmin/user_upload/eufao-fsi4dm/doc-training/hfias.pdf [accessed 1 August 2018]. [Google Scholar]
- [32].Larson HJ, Jarrett C, Schulz WS, Chaudhuri M, Zhou Y, Dube E, et al. Measuring vaccine hesitancy: the development of a survey tool. Vaccine 2015;33(34):4165–75. 10.1016/j.vaccine.2015.04.037. [DOI] [PubMed] [Google Scholar]
- [33].Kroenke K, Spitzer R, Williams W. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001;16(9):606–16. 10.1046/j.1525-1497.2001.016009606.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [34].Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap) – a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009;42(2):377–81. 10.1016/j.jbi.2008.08.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [35].Domek GJ, O’Leary ST, Bull S, Bronsert M, Contreras-Roldan IL, Bolaños Ventura GA, et al. Measuring vaccine hesitancy: field testing the WHO SAGE Working Group on Vaccine Hesitancy survey tool in Guatemala. Vaccine 2018;36(35):5273–81. 10.1016/j.vaccine.2018.07.046. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [36].Domek GJ, Contreras-Roldan IL, Asturias EJ, Bronsert M, Bolaños Ventura GA, O’Leary ST, et al. Characteristics of mobile phone access and usage in rural and urban Guatemala: assessing feasibility of text message reminders to increase childhood immunizations. mHealth 2018;4:9 10.21037/mhealth.2018.03.05. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [37].The National Vaccine Advisory Committee. The measles epidemic. The problems, barriers, and recommendations. JAMA 1991;266(11):1547–52. 10.1001/jama.1991.03470110093039. [DOI] [PubMed] [Google Scholar]
- [38].von Kries R, Böhm O, Windfuhr A. Haemophilus influenzae b-vaccination: the urgency for timely vaccination. Eur J Pediatr 1997;156(4):282–7. 10.1007/s004310050601. [DOI] [PubMed] [Google Scholar]
- [39].Siedler A, Hermann M, Schmitt HJ, Von Kries R. Consequences of delayed measles vaccination in Germany. Pediatr Infect Dis J 2002;21(9):826–30. 10.1097/01.inf.0000027665.74040.bf. [DOI] [PubMed] [Google Scholar]
- [40].Grant CC, Roberts M, Scragg R, Steward J, Lennon D, Kivell D, et al. Delayed immunisation and risk of pertussis in infants: umatched case-control study. BMJ 2003;326(7394):852–3. 10.1136/bmj.326.7394.852. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [41].Kolos V, Menzies R, McIntyre P. Higher pertussis hospitalization rates in indigenous Australian infants, and delayed vaccination. Vaccine 2007;25(4):588–90. 10.1016/j.vaccine.2006.08.022. [DOI] [PubMed] [Google Scholar]
- [42].Glanz JM, Narwaney KJ, Newcomer SR, Daley MF, Hambidge SJ, Rowhani-Rahbar A, et al. Association between undervacination with diphtheria, tetanus toxoids, and acellular pertussis (DTaP) vaccine and risk of pertussis infection in children 3 to 36 months of age. JAMA Pediatr 2013;167(11):1060–4. 10.1001/jamapediatrics.2013.2353. [DOI] [PubMed] [Google Scholar]
- [43].Bolton P, Hussain A, Hadpawat A, Holt E, Hughart N, Guyer B. Deficiencies in current childhood immunization indicators. Public Health Rep 1998;113(6):527–32. [PMC free article] [PubMed] [Google Scholar]
- [44].Dombkowski KJ, Lantz PM, Freed GL. The need for surveillance of delay in age-appropriate immunization. Am J Prev Med 2002;23(1):36–42. 10.1016/S0749-3797(02)00442-7. [DOI] [PubMed] [Google Scholar]
- [45].Dayan GH, Shaw KM, Baughman AL, Orellana LC, Forlenza R, Ellis A, et al. Assessment of delay in age-appropriate vaccination using survival analysis. Am J Epidemiol 2006;163(6):561–70. 10.1093/aje/kwj074. [DOI] [PubMed] [Google Scholar]
- [46].Akmatov MK, Kretzschmar M, Krämer A, Mikolajczyk RT. Timeliness of vaccination and its effects on fraction of vaccinated population. Vaccine 2008;26(31):3805–11. 10.1016/j.vaccine.2008.05.031. [DOI] [PubMed] [Google Scholar]
- [47].Fadnes LT, Nankabirwa V, Sommerfelt H, Tylleskär T, Tumwine JK, Engebretsen IM; PROMISE-EBF Study Group. Is vaccination coverage a good indicator of age-appropriate vaccination? A prospective study from Uganda. Vaccine 2011;29(19):3564–70. 10.1016/j.vaccine.2011.02.093. [DOI] [PubMed] [Google Scholar]
- [48].Hu Y, Chen Y, Guo J, Tang X, Shen L. Completeness and timeliness of vaccination and determinants for low and late uptake among young children in eastern China. Hum Vaccin Immunother 2014;10(5):1408–15. 10.4161/hv.28054. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [49].Gibson DG, Ochieng B, Kagucia EW, Obor D, Odhiambo F, O’Brien KL, et al. Individual level determinants for not receiving immunization, receiving immunization with delay, and being severely underimmunized among rural western Kenyan children. Vaccine 2015;33(48):6778–85. 10.1016/j.vaccine.2015.10.021. [DOI] [PubMed] [Google Scholar]
- [50].Chen L, Wang W, Du X, Rao X, van Velthoven MH, Yang R, et al. Effectiveness of a smart phone app on improving immunization of children in rural Sichuan Province, China: study protocol for a paired cluster randomized controlled trial. BMC Public Health 2014;14:262 10.1186/1471-2458-14-262. [DOI] [PMC free article] [PubMed] [Google Scholar]