Monitoring polio supplementary immunization activities using an automated short text messaging system in Karachi, Pakistan

Abstract

Problem

Polio remains endemic in many areas of Pakistan, including large urban centres such as Karachi.

Approach

During each of seven supplementary immunization activities against polio in Karachi, mobile phone numbers of the caregivers of a random sample of eligible children were obtained. A computer-based system was developed to send two questions – as short message service (SMS) texts – automatically to each number after the immunization activity: “Did the vaccinator visit your house?” and “Did the enrolled child in your household receive oral polio vaccine?” Persistent non-responders were phoned directly by an investigator.

Local setting

A cluster sampling technique was used to select representative samples of the caregivers of young children in Karachi in general and of such caregivers in three of the six “high-risk” districts of the city where polio cases were detected in 2011.

Relevant changes

In most of the supplementary immunization activities investigated, vaccine coverages estimated using the SMS system were very similar to those estimated by interviewing by phone those caregivers who never responded to the SMS messages. In the high-risk districts investigated, coverages estimated using the SMS system were also similar to those recorded – using lot quality assurance sampling – by the World Health Organization.

Lessons learnt

For the monitoring of coverage in supplementary immunization activities, automated SMS-based systems appear to be an attractive and relatively inexpensive option. Further research is needed to determine if coverage data collected by SMS-based systems provide estimates that are sufficiently accurate. Such systems may be useful in other large-scale immunization campaigns.

Background

Pakistan is one of only three countries where poliovirus remains endemic despite extensive elimination efforts.¹ Karachi, Pakistan’s largest city, is now the world’s only metropolitan area that has not been able to interrupt poliovirus transmission. Nine cases of poliomyelitis were confirmed in Karachi during 2011 and these cases were reported from six of the 18 “towns” or districts that form the metropolitan area.² Although in 2012 no cases of polio were identified in Karachi, poliovirus was found in samples of the city’s sewer water.¹ The first case of polio identified in Pakistan in 2013 was in a child in Karachi.³

Problem

The use of supplementary immunization activities (SIAs) has helped several countries increase their immunization coverage against polio and has played a major role in decreasing the global incidence of polio by more than 99% since 1988.⁴ In Karachi, however, more than 100 such activities based on oral polio vaccine have failed to interrupt local poliovirus transmission.⁵ A major reason for this failure is poor vaccine coverage among vaccination-eligible children.⁶ The current system of monitoring the coverage of each SIA in Pakistan has several limitations. In this system, the children receiving vaccine during an SIA are marked with black “permanent” ink on their right little fingers.⁷ A third-party evaluation team then visits a randomly selected sample of households in the target area and counts the number of eligible children – that is, children aged less than 5 years – and the number of children who have black ink on their right little fingers.⁸ This monitoring system is resource-intensive, utilizes convenience sampling and is prone to misreporting bias since the same individuals often supervise both the vaccinations and the evaluation of vaccine coverage.⁹ It also exposes the members of the evaluation teams to the risk of violence, especially in certain locations with poor security. More than 16 vaccinators associated with polio vaccination campaigns have been assassinated in various parts of Pakistan since December 2012.¹⁰

In Pakistan, mobile phones have become very common over the past 10 years and there are currently more than 119 million mobile phone users.¹¹ Many of these phones are used to send short message service (SMS) texts. In 2011, for example, 237.58 billion person-to-person texts – the equivalent of about 175 text messages per mobile phone per month – were generated in Pakistan.¹² Mobile health (“m-health”) technology is increasingly being considered for various health interventions, such as sending reminders for clinic visits, immunizations and medication adherence.¹³ We recently wondered if m-health could be used to measure vaccine coverage in polio SIAs, particularly in resource-limited areas and those with poor security. We therefore conducted a feasibility study to assess if a low-cost automated system that could send SMS messages to the mobile phones of caregivers could be used to monitor the coverage of SIAs in Karachi.

Approach

The study protocol was approved by the ethical review committees of the Aga Khan University in Pakistan and the World Health Organization (WHO). Thirty clusters of households were randomly selected in each of three districts of Karachi – Gulshan-e-Iqbal, Baldia and SITE (Sindh Industrial and Trading Estate) – and across the entire city. A case of polio had been detected in each of the three study districts in 2011.² Each cluster consisted of 200 to 250 households.⁸ Within each selected cluster, at least 28 households with a member aged less than 5 years were selected by systematic random sampling. If a selected house contained more than one child younger than 5 years, then one of the children was randomly chosen for enrolment. After informed consent was obtained from a caregiver, baseline demographic information on each enrolled child, the mobile phone number of one of the child’s caregivers and the phone-owning caregiver’s preferred language were recorded. The phone owners were told that they might be contacted – either via SMS texts or a direct phone call – after one or more forthcoming SIAs and that, if contacted, they would be asked two questions. Although the questions were basically “In the last SIA, did the vaccinator visit your house?” and “In the last SIA, did [named of enrolled child] receive oral polio vaccine?”, they – or their translation into the preferred languages of the caregivers – were slightly revised during the study to improve the probability of an adequate response. The revisions of the questions were based on feedback from the caregivers who had been asked the questions.

After each of seven SIAs run in Karachi, seven households were randomly selected from the 28 originally identified in each study cluster. The caregivers in these households were sent the two questions as SMS texts. Those who did not answer the text messages were contacted via direct phone calls and asked the same questions by an investigator. Coverage data obtained via text messages and phone calls were compared with the coverage data obtained – in the same SIAs – by WHO, using conventional lot quality assurance sampling.¹⁴

The SMS texts were sent automatically – and relatively cheaply – by a server-based software package developed specifically for our study. The two questions were sent as two separate texts, with a request for the caregiver to reply to each text with the letter “P” followed by a single digit: 1 if the answer was “yes”, 2 if the answer was “no” and 3 if the answer was “do not know”. As an incentive, 20 Pakistani rupees – equivalent to about 0.20 United States dollars – were credited to a caregiver’s mobile phone each time the caregiver responded to an automated message. All questions – whether sent as texts or spoken over the phone – were posed in the caregiver’s preferred language.

If no response to a text had been received within 24 hours, repeat messages were sent daily for up to 1 week. If there had been no reply to a question sent as an SMS text after 1 week, an investigator attempted to speak by phone to the caregiver involved, to obtain an answer to the question that had been posed and to determine why the caregiver had not sent a reply to the SMS message.

Results

Seven SIAs conducted in Karachi – six in 2012 and one in 2013– were monitored using the SMS-based system. Of the 840 caregivers who were sent the messages in each SIA, the median proportions who responded to the first SMS text – “In the last SIA, did the vaccinator visit your house?” – and to the second text – “In the last SIA, did [named of enrolled child] receive oral polio vaccine?” – were 23% (interquartile range, IQR: 19–30) and 14% (IQR: 10−21), respectively. Of the 4404 caregivers who were sent SMS messages but responded to none of them, 56% (IQR: 54–58) were successfully spoken to by an investigator, by phone. We failed to get any response – by text or phone call – from 1831 (32%) of the caregivers sent SMS messages: four refused to talk to an investigator, 208 never answered the phone when called at least three times by an investigator and 1619 could not be reached because they had changed their numbers, their phones were faulty, uncharged or otherwise unusable or, perhaps, because their phone numbers had been recorded incorrectly.

In the sample representing the entire city of Karachi, the lowest vaccine coverage was detected during the SIA in June 2012, when 59% of the caregivers who were sent SMS messages and successfully contacted said that a vaccinator had visited their household and the same proportion said that the enrolled child had received a vaccine dose. The highest coverage recorded in the sample representing the entire city was observed slightly earlier – in the SIA in April 2012 – when the corresponding proportions were 83% and 82%, respectively. The median coverage revealed for the entire city by the responses of the caregivers sent SMS messages was 74.5% (95% confidence interval, CI: 71.6–77.4).

Using the responses of the caregivers in the district-level samples who were sent SMS messages and successfully contacted, the median coverages by vaccinators were estimated to be 71% (95% CI: 68.2–73.8) in Gulshan-e-Iqbal, 79% (95% CI: 76.4–81.6) in Baldia and 81% (95% CI: 78.6–83.5) in SITE. The corresponding vaccine coverages among the enrolled children were 67% (95% CI: 63.9–70.1), 75% (95% CI: 72.1–78.0) and 78% (95% CI: 75.2–80.8), respectively.

In general, the data collected during phone calls to caregivers who had not responded to SMS messages indicated levels of vaccine coverage similar to those estimated from the responses to SMS messages (Table 1). However, the overall response rate to phone calls made by investigators was almost twice as high as the corresponding response rate to SMS messages (44% versus 23%). In the high-risk districts that were investigated, the vaccine coverages estimated by WHO – using clustered lot quality assurance sampling – were slightly higher than the estimates made using the SMS system (Table 1).

Table 1. Vaccine coverages estimated by three methods, Karachi, Pakistan, 2012–2013.

Study areas and date of SIA	na	Estimated vaccine coverage,b % (95% CI)
		SMS	LQASc	PCd
SITEe
March 2012	9/–/135	78 (51–100)	ND	93 (89–97)
April 2012	21/–/81	95 (86–100)	ND	93 (87–99)
June 2012	22/150/96	91 (79–100)	92 (88–96)	83 (75–91)
July 2012	24/–/87	83 (68–98)	ND	90 (84–96)
September 2012	77/–/61	29 (15–43)	ND	56 (44–68)
October 2012	37/50/68	86 (75–97)	82 (71–93)	82 (73–91)
April 2013	20/–/75	65 (44–85)	ND	56 (45–67)
Baldia
March 2012	8/150/138	63 (30–96)	93 (89–97)	93 (89–97)
April 2012	14/200/77	93 (80–100)	92 (88–96)	96 (92–100)
June 2012	14/150/85	86 (68–100)	94 (92–98)	87 (80–94)
July 2012	18/–/69	94 (83–100)	ND	91 (84–98)
September 2012	33/–/66	73 (58–88)	ND	82 (73–91)
October 2012	31/50/75	97 (91–100)	88 (79–97)	88 (81–95)
April 2013	29/–/73	52 (34–70)	ND	56 (45–68)
Gulshan-e-Iqbal
March 2012	15/100/108	60 (51–69)	91 (85–97)	80 (72–88)
April 2012	22/200/92	68 (49–87)	89 (84–93)	83 (75–91)
June 2012	38/150/77	82 (70–94)	89 (84–93)	81 (72–90)
July 2012	32/–/82	78 (64–92)	ND	82 (74–90)
September 2012	46/–/51	61 (48–74)	ND	76 (64–88)
October 2012	57/50/68	60 (47–73)	92 (84–100)	84 (75–93)
April 2013	32/–/73	72 (56–88)	ND	66 (55–77)
Entire city of Karachi
March 2012	–/–/142	8 (0–21)	ND	77 (70–84)
April 2012	24/–/84	92 (81–100)	ND	89 (83–96)
June 2012	26/–/83	46 (27–65)	ND	72 (62–82)
July 2012	35/–/82	86 (75–98)	ND	89 (82–96)
September 2012	53/–/71	26 (14–38)	ND	49 (37–60)
October 2012	43/–/72	84 (73–95)	ND	85 (76–93)
April 2013	–/–/–	ND	ND	ND

CI, confidence interval; LQAS, lot quality assurance sampling; ND, not determined; PC, phone calls; SIA, supplementary immunization activity; SMS, short message service.

^a Numbers of households investigated using the SMS system/LQAS/PC that provided adequate data. We excluded the households of caregivers who claimed not to know if the eligible children in their care had been vaccinated.

^b Estimated percentage of children younger than 5 years who received oral polio vaccine in each immunization activity.

^c Data collected by the World Health Organization.¹⁴

^d Data collected in phone calls to the caregivers who did not respond to SMS messages.

^e Sindh Industrial and Trading Estate.

The major reasons given by caregivers for not responding to an SMS message were that they were too busy (36%), not interested in replying (32%) or unable to read the message (20%). The most common time of day for a caregiver to reply to an SMS message was the evening – between 18:00 and 22:00. Overall, 80% of the replies to SMS messages were received within three days of the messages being sent. Of the caregivers who responded to an SMS message asking if the enrolled child had received oral polio vaccine in the last SIA, 6% answered that they did not know.

Discussion

We were able to monitor the vaccine coverages achieved in SIAs in Karachi by using a low-cost, automated SMS-based system – augmented with follow-up phone calls to persistent non-responders. To the best of our knowledge, this study is the first to evaluate the effectiveness of mobile phones and automated SMS messages in monitoring the quality of polio SIAs.

The rate of response to the first SMS message that we sent out to caregivers was disappointing but it soon became clear that many of the targeted caregivers had simply not received the message because of problems with the modems then being used. The subsequent addition of a commercial SMS gateway (Sidat Hyder, Karachi, Pakistan) to the server-based SMS system substantially improved the percentage of targeted caregivers who received the SMS message that we had intended to send to them (data not shown). The rate of response to our automated SMS messages gradually increased over time, as we improved the format and language of the messages. It would be relatively easy to scale up our SMS-based system to cover SIAs or other vaccine campaigns at the provincial or country level. However, there may not be sufficient human resources to permit follow-up phone calls to be made to all of the non-responders in such large-scale campaigns.

Automated SMS-based monitoring allows for the separation of the vaccine delivery and monitoring teams. It eliminates errors made by human data collectors, reporting bias by field teams – who may exaggerate coverage to improve their apparent performance – and the risk of data collectors being attacked. The latter advantage makes SMS-based monitoring particularly valuable in security-compromised settings.^10,15 If an SMS-based system is implemented, the human resources previously occupied with monitoring can focus exclusively on the delivery of vaccines.

In general, the methods that we and WHO used to estimate vaccine coverage in SIAs in Karachi gave similar results. However, there were some discrepancies. For example, for the SIA in March 2012, the coverage estimated using data collected with the SMS-based system was substantially lower than that estimated using lot quality assurance sampling (Table 1). However, the SMS gateway had not been deployed at this time and many of the caregivers who were sent SMS messages never received them. The overall response rate to our SMS messages was disappointingly low – despite the resending of messages to non-responders and the offer of a financial incentive for replying. It remains possible that the vaccine coverage in the households of caregivers who were never successfully contacted after being sent SMS messages differed from that in the households of responders. When investigators phoned most of the caregivers who were never successfully contacted after being sent SMS messages, they simply received a “number unobtainable” message or tone. It seems likely that many caregivers had changed their mobile phone numbers after the baseline survey. In Pakistan – as in Cambodia¹⁶ and other developing countries – it is easy to switch service providers and such providers offer attractive packages to gain new customers.

Another limitation of our study was that many of the caregivers contacted by SMS messages said that they simply did not know if the enrolled child in their household had been vaccinated. In many of these cases, the caregiver was responding while out of contact with the enrolled child and any other of the child’s caregivers. This problem might be reduced by educating target populations – so that the caregivers who are to be contacted anticipate the contact and check on vaccinations within their households before that contact is made.

We could compensate for low response rates by increasing the number of caregivers sent SMS messages but if there is a systematic difference in the baseline characteristics of responders and non-responders, or of those who do and do not own a mobile phone, a larger sample will not reduce the risk of non-response bias. The use of mobile phones is now very common in Pakistan but it is not universal. Caregivers who are resistant to vaccinating children – who are important in the interruption of polio transmission – may well be unlikely to share their mobile phone numbers with those tasked with assessing vaccine coverage.

Although we collected mobile phone numbers and located children who were eligible for polio vaccination through household visits, the same information may become available online in the future. For example, if local laws, rules and regulations permit it, the phone numbers of potential caregivers could be collected from service providers and the identities and addresses of eligible children could be obtained from health records. Caregivers could be encouraged to provide the same data for inclusion in national registries. Vaccine coverage could then be assessed with very little human intervention.

In conclusion (Box 1), in the monitoring of SIA, SMS-based systems have the advantages of low cost, fairly representative sampling of the population, low burdens on monitoring staff and low risk of the coverage being exaggerated. These advantages make such systems an attractive alternative to more conventional methods of SIA monitoring, in Pakistan and elsewhere.

Box 1. Summary of main lessons learnt.

• For the monitoring of coverage in supplementary immunization activities, automated systems based on short message service (SMS) texts appear to be an attractive and relatively inexpensive option.

• Further research is needed to determine if coverage data collected by SMS-based system provide estimates that are sufficiently accurate.

• Such systems may be useful in other large-scale immunization campaigns.