Patient experience with, and use of, an electronic monitoring system to assess vaccination responses

Abstract

Objective  To evaluate the user experience and acceptability of an electronic patient monitoring system.

Setting and participants  822 Military and civilian personnel at a health clinic at a major US military headquarters used an Internet and telephone‐based electronic monitoring system to report vaccination‐site responses and symptoms after receiving the smallpox vaccination. Focus groups of vaccinees were conducted to help develop a survey about the experience that was distributed to 379 vaccinees (96% completion rate).

Results  Users of the electronic monitoring system reported that it was fast and easy to use and reported they would use a system like this again and recommend an electronic monitoring system to a friend or relative. Most users (84%) were comfortable with a physician tracking their vaccine reaction using their electronic reports, but only half (51%) were comfortable with eliminating the post‐vaccination follow‐up visit with their health‐care provider based on their electronic reports.

Conclusions  This electronic monitoring system was well received by vaccinees and allowed health‐care providers to track the status of vaccinees. However, vaccinees were not comfortable replacing a physician visit with electronic monitoring, at least for the smallpox vaccination. A monitoring system like this may be useful in public health settings, such as mass vaccination or prophylaxis during a bioterrorism event, a pandemic influenza outbreak, or another public health emergency.

Introduction

The National Research Council (NRC) report, Networking Health: Prescriptions for the Internet, highlights the potential for information technology (IT) and the Internet to revolutionize health‐care delivery in the near future. ¹ For individual patients, the Internet and IT can facilitate information seeking and promote disease self‐management and patient autonomy, as well as improve patient–physician communication. ² , ³ , ⁴ When applied to populations, electronic monitoring of many individual patients with chronic or infectious diseases can improve surveillance and management of chronic diseases, as well as reducing health‐resource utilization. The NRC cites numerous applications of such technology for chronic disease management and even self‐management of weight loss. In public health practice, a mass vaccination program, whether for pandemic influenza or smallpox, may be strengthened and easier to administer if an electronic system were used to monitor vaccinee reactions and side effects.

The US Department of Defense (DoD) pilot tested a telephone and Internet‐based electronic patient monitoring system during its smallpox vaccination campaign in 2003. ⁵ Centers for Disease Control and Prevention (CDC) guidelines encourage smallpox vaccinees to maintain a written diary of local and systemic symptoms and vaccination‐site appearances for 28 days after vaccination. ⁶ Anecdotal information suggests that adherence to this recommendation was limited, and that many vaccinees did not complete the diaries. DoD's pilot program was designed to permit people who received smallpox vaccination to report symptom and vaccination‐site descriptions electronically in lieu of maintaining a daily written diary.

Such a system can reduce the need for clinical assessments of vaccine responses, cutting travel time, work absenteeism and clinician time to assess vaccine sites. In addition, a system such as this can act as an early warning device for adverse events. We previously reported on the clinical data obtained through this pilot program and the finding that the reliability of vaccinee self‐assessments of their vaccine sites was high. ⁵ However, such a system would not be useful in the future if people found it difficult to use or could not trust it. In this paper, we report on the user experience and acceptability of the system.

Methods

System enrolment and use

This research was approved by RAND, Abt Associates and DoD Institutional Review Boards. A description of how vaccinees were recruited and enrolled in the electronic monitoring system is described elsewhere. ⁵ Briefly, military personnel and civilian DoD employees from four sites were offered the opportunity to use an automated electronic system developed by Voxiva Corporation (Washington, DC, USA) to report vaccination signs and symptoms for 28 days after vaccination. Participation in the monitoring system was completely voluntary. Participants registered for the system and received instructions on its use, including a pocket‐sized colour brochure depicting expected vaccination‐site responses and symptoms. Vaccinees could log onto a secure website or call an automated, password‐protected telephone system to record their data. Confidentiality was maintained by the fact that only health professionals directly involved in the care of the patients and the personnel who built and maintained the system had access to the electronic data. User reports were de‐identified before these analyses were performed.

Each vaccinee had the option to request contact with a nurse, who answered the vaccinee's questions and recommended referral to additional medical care if needed. In addition to participant‐initiated reports, a call centre was set up to contact vaccinees to receive and stimulate reporting. The monitoring system was evaluated from March to September 2003. This paper reports on the use of the system by the 822 vaccinees, who signed up for the system at the largest of the four sites, the DiLorenzo Tricare Health Clinic in the Pentagon, Arlington, VA, USA. In addition to the reports from the system users, we surveyed a subset of vaccinees at the DiLorenzo clinic to assess their experience using the electronic system.

Focus group and survey

To develop a survey with which to assess user acceptability, we first discussed the system in four focus groups of smallpox vaccinees (20 people in total). Focus group questions explored the mechanics of using the system, ease of use and concerns that vaccinees had with using the system. The resulting survey included questions about the use of the system (e.g. frequency, ease of use, reasons for non‐use), whether vaccinees could accurately report vaccine site appearance and symptoms, the likelihood that the vaccinee would use a similar system again, the level of comfort with having a physician track their electronic reports and determine vaccine response based on the electronic reports, and general demographic characteristics. Respondents used a five‐point Likert scale with response choices appropriate to the question (e.g. very easy to very difficult, very comfortable to not at all comfortable). Surveys were distributed to all 379 vaccinees, who returned to the clinic to check on the status of their vaccination site (‘take checks’) from June 13 to July 15, on August 13, and from August 20 to September 11, 2003. These surveys were completed by 362 of the 379 vaccinees who received one (96% completion rate).

Analysis

For ease of analysis, we dichotomized responses to several survey questions that used a five‐point Likert scale. We used Fisher's Exact test and Mantel–Haenszel chi‐square tests to compare demographic characteristics across different population strata: vaccinees who did not sign up for the system, vaccinees who signed up for the system but did not use the system, and vaccinees who signed up for the system and used the system.

In analysing the electronic reports by vaccinees, logistic regression was used to determine which vaccinee characteristics related to use (≥1 report) and non‐use (0 reports) of the electronic monitoring system. Independent variables included age, gender and race. Among monitoring system users, we fit a negative binomial model to determine which demographic characteristics were associated with higher use of the system. The dependent variable is the number of reports made by a vaccinee. A Poisson regression model was originally fit to the data but model‐fit diagnostics indicated overdispersion, hence a negative binomial model was used. We used SAS (SAS System version 8.2, Cary, NC, USA) for all analyses.

Results

Of the 822 vaccinees who signed up for the system at the clinic, 708 (86%) made at least one report with the system (for more detail on the number of reports made and information in the reports, see Ref. [5]). The 362 survey responders included 220 vaccinees (61%) who signed up for the system and 142 vaccinees (39%) who did not sign up for the monitoring system. Of the 220 survey responders who signed up for the system, 177 individuals (80%) used it at least once, which is similar to the overall percentage of total enrollees who used the system at least once. Demographics of the survey responders and system enrollees are shown in Table 1.

Table 1.

 Demographics of study population

	Survey responders
	Total survey sample (N = 362), n (%)	Did not enrol in electronic system (N = 142), n (%)	Enrolled in but did not use the system (N = 43), n (%)	Enrolled in & used system at least once (N = 177), n (%)	Fisher's Exact test, P‐value	Total system enrollees (Pentagon) (N = 822), n (%)
Age (years)
<26	11 (3)	5 (4)	2 (5)	4 (2)	0.15	36 (4)
26–35	56 (16)	19 (14)	11 (26)	26 (15)		118 (14)
36–45	184 (51)	68 (48)	20 (47)	96 (55)		363 (44)
>45	108 (30)	49 (35)	10 (23)	49 (28)		305 (37)
Gender
Female	41 (11)	21 (15)	2 (5)	18 (10)	0.17	104 (13)
Male	318 (89)	121 (85)	40 (95)	157 (90)		718 (87)
Race
White	287 (81)	121 (86)	35 (83)	131 (77)	0.40	669 (81)
Black	45 (13)	13 (9)	5 (12)	27 (16)		89 (11)
Other	21 (6)	7 (5)	2 (5)	12 (7)		64 (8)
Rank
Enlisted	68 (20)	24 (18)	9 (22)	35 (21)	0.74	159 (19)
Officer	259 (74)	102 (74)	30 (73)	127 (75)		536 (65)
Civilian	21 (6)	11 (8)	2 (5)	8 (5)		127 (15)
Education
HS or Associates degree	50 (14)	13 (9)	6 (14)	31 (18)	0.20	Not available
Bachelors degree	72 (20)	36 (26)	3 (7)	33 (19)		Not available
Masters degree or higher	236 (66)	92 (65)	34 (79)	110 (63)		Not available

Percentages may not add to 100% because of rounding error and not all questions were answered by all survey responders. For age and education, a Mantel–Haenszel chi‐square test is performed.

Use of the system

The electronic system was well received by the vaccinees. Survey responders reported that the system was fast and easy to use (Table 2). 34% (58/171) of respondents who used the system reported using the telephone to make at least one report and 84% (147/175) reported using the Internet at least once. Both the Internet and telephone methods of making reports were reported as being easy. No users reported that the system took too long to make reports. Users also reported that the picture on the web or pocket card closely matched the appearance of their vaccine site (143/169; 85%) and that they were confident that what they reported matched their true reaction to the vaccine (147/171; 86%).

Table 2.

 System usability based on survey responses (N = number responding to each survey item)

	n (%)
Written and verbal information received described the system well (N = 217)	205 (94)
If vaccinees had questions about the system they (N = 209):
Did not have any questions about the system	162 (78)
Did not ask for help	30 (14)
Read the information and brochures handed out to me	20 (10)
Called into the system and spoke with an operator	12 (6)
The first time making a report was easy:
Via the Internet/Web (N = 126)	118 (94)
Calling in by phone (N = 40)	37 (93)
Overall, using the system was easy:
Via the Internet/Web (N = 147)	142 (97)
Calling in by phone (N = 58)	50 (86)
Time to complete the electronic diary report was (N = 177):
≤2 min	132 (75)
3–5 min	42 (24)
6–10 min	2 (1)
>10 min	1 (1)
Amount of time it took to complete the electronic diary reports was (N = 177):
Very short	103 (58)
About right	74 (42)
Very long	0 (0)

Among the 822 vaccinees who signed up to use the system at the DiLorenzo clinic, the logistic regression model suggested that older respondents, civilians and whites (compared with other races) were more likely than others to use the electronic monitoring system at least once; however, none of these demographic characteristics achieved statistical significance (Table 3). To investigate whether there were characteristics that led to more frequent use among users, we fitted a multivariate negative binomial model. Among users of the system (≥3 reports), this model indicates that older vaccinees were more likely than younger (P < 0.01), and whites were more likely than African Americans (P < 0.03), to use the system frequently (Table 3). We restricted this model to users who reported at least three times because analyses of call centre data (data not shown) indicated that approximately 75% of the reports made by people who used the system only once or twice were initiated by the call centre. As we were interested in which characteristics (if any) were related to frequent system usage, we subsetted our negative binomial model to the population who reported ≥3 times, i.e. those who used the system as the designers intended. The resulting N for this model is 560. For the negative binomial model, the deviance and Pearson chi‐square divided by the degrees of freedom are close to 1 (1.03 each) indicating good model fit.

Table 3.

 Demographic characteristics relating to any use (logistic regression) and frequent use (negative binomial regression) of the system

	Logistic regression model (N = 822) (>0 reports vs. 0 reports)		Negative binomial model (N = 560) (among frequent users: ≥3 reports)
	Odds ratio (95% CI)	P‐value	Incidence rate ratio (95% CI)	P‐value
	Gender
Male (reference group)
Female	1.05 (0.58, 1.91)	0.87	0.95 (0.83, 1.08)	0.41
Race/ethnicity
White (reference group)
Black	1.06 (0.55, 2.02)	0.87	0.83 (0.71, −0.98)	0.03
Other	0.70 (0.35, 1.37)	0.29	1.05 (0.88, 1.25)	0.59
Age group (years)
<26 (reference group)
26–35	1.04 (0.33, 3.28)	0.95	2.08 (1.31, 3.30)	<0.01
36–45	1.19 (0.37, 3.76)	0.77	1.92 (1.20, 3.07)	<0.01
>46	1.23 (0.37, 4.07)	0.73	1.95 (1.21, 3.13)	<0.01
Rank
Junior Enlisted Personnel (E1–E4) (reference group)
Senior Enlisted Personnel (E5–E9, W1–W5)	0.85 (0.24, 3.01)	0.80	1.06 (0.66, 1.68)	0.82
Officer	1.78 (0.48, 6.66)	0.39	1.03 (0.64, 1.64)	0.92
Civilian	1.79 (0.45, 7.11)	0.41	1.04 (0.64, 1.67)	0.89
Goodness‐of‐fit	Hosmer and Lemeshow, P = 0.83		Deviance/d.f. = 1.03; Pearson chi‐square/d.f. = 1.03

E1–E4, E5–E9 are enlisted soldiers (grades 1–4 are junior; 5–9 are more senior enlisted soldiers). W1–W5 (Warrant officers) are more senior than enlisted men but are not commissioned officers.

The mean number of reports made by the 822 vaccinees over the 28‐day period was 7.4 (±6.6). On the survey, vaccinees reported a variety of reasons for not making daily reports: they forgot (52/137; 38%), they did not have access to a computer or phone (23/137; 17%), they did not have the phone number or web site with them (22/137; 16%), they did not have any abnormal symptoms to report (18/137; 13%) or their symptoms did not change from their previous report (14/137; 10%). Daily reminders (40/124; 32%) and being able to enter ‘no changes since the last report’ (35/124; 28%) were suggested as ways to increase reporting frequency.

The survey also probed whether vaccinees would be likely to use a system like this again. Almost 90% (154/172) reported being likely to use an electronic monitoring system instead of a written diary and to recommend an electronic system to a friend or relative receiving the smallpox vaccination (149/171). Most respondents (143/171; 84%) reported that they would recommend an electronic monitoring system to a friend or relative for other health‐care issues, as well.

Most of the vaccinees (144/172; 84%) reported being comfortable with the idea of having a physician track their reaction to the vaccine using their electronic reports, and 67% (116/172) reported being comfortable having their take determined using the electronic reports. However, only 51% (87/172) reported that they were comfortable eliminating the actual take check follow‐up visit with their health‐care provider if their electronic reports indicated they had a positive vaccine reaction.

The vaccinees who chose not to enrol in the system differed little from enrollees who did not use the system or system users. Notably, the percentage of responders who reported feeling anxious about the smallpox vaccination in the groups were 14% (20/142), 12% (5/43) and 9% (16/177).

Discussion

Increasing patient–physician communication, facilitating disease management and medical surveillance are some of the ways that IT can have an impact on health‐care delivery. Recent advances in Internet and telephone capabilities have improved the potential timeliness of data gathering from patients. This paper reports on one such system used to collect post‐smallpox vaccination data directly from vaccinees.

The system was easy to use, entering reports took little time, and users were generally satisfied with the system. Almost all of the survey responders who used the system reported that they would use a system like this again, in place of a paper diary, and would recommend it to a friend or relative. While consistent use over 28 days was not achieved, ⁵ use of the system was not mandated. We were able to identify subgroups of people who were less likely to use the system, and targeted interventions could be developed to increase use with these groups. This system was also implemented without any electronic reminders or incentives. We expect that such features would enhance reporting.

The ability of physicians to track progress of patient care without seeing the patients, thereby reducing patient utilization of the health‐care system, is one advantage of an electronic monitoring system. While survey responders in this study were comfortable with a physician tracking their vaccination status via their electronic reports, and many were comfortable with having their take check determined electronically, half of the respondents were not comfortable eliminating the follow‐up visit with a health‐care provider. In other words, respondents were not yet comfortable replacing a provider visit with an electronic report, at least for this situation. However, this should be viewed in light of the fact that, for some, smallpox vaccination was a highly charged issue. In contrast, we also note that substantial proportions (∼40%) never returned to have their vaccine reaction assessed. ⁵ We report elsewhere that self‐reported take information has high sensitivity (98.8%) and high specificity (99.6%). ⁵ Not only could these reports be used as surrogates in situations where vaccine reactions could not be assessed in person, a similar reporting system may prove useful in public health settings in which large numbers of people will need treatment and follow‐up in a short period of time, such as a mass vaccination or prophylaxis during a bioterrorism event, an influenza pandemic or another public health emergency. While we cannot determine the portion of the population who might use such a system in a public health emergency, this pilot test suggests that it may be useful for a substantial portion of the population, enabling physicians and public health officials to identify those individuals who might require follow‐up.

There are three significant limitations to this study. System users and survey participants were US military and DoD civilian personnel who work at the Pentagon, and are highly educated, predominantly white, and male. Thus the generalizability to the general population is limited. Additionally, the electronic reporting system was piloted during the build up and start of Operation Iraqi Freedom, which posed logistical problems with some personnel being deployed soon after getting vaccinated. Finally, we were unable to systematically determine how many people did not sign up for the system, why they did not sign up, and if there were any systematic differences between those who did and did not sign up. The demographics and the level of concern reported by the survey responders, however, suggests that there were no significant differences between those who signed up for the system and those who did not sign up.

Electronic diaries such as this are becoming popular, especially in the clinical trials, ⁷ but few published studies use Internet technologies ² or telephones ⁸ , ⁹ to capture diary information that is then immediately available for health‐care providers or public health officials to review. This, we believe, is the first study to report on a system that combines both the Internet and telephone capabilities for diary entries. Based on this experience, DoD used the system for a different, but related, purpose‐monitoring individuals who received injectable or intranasal influenza vaccines during the 2004–2005 influenza season. The system was enhanced to provide frequent voice‐ and e‐mail reminders to report using the system, and vaccine recipients were asked to report symptoms over a 2‐week, rather than 4‐week period. The ease of use of the Internet, coupled with the widespread availability of telephones, makes a dual system such as this attractive. Additionally, the ability for the researcher or clinician to evaluate diary data and make decisions about patient care in real time is an advantage of this type of monitoring system.

Funding/support

This research was funded by the US Army Military Vaccine Agency in the Office of the Surgeon General of the US Army under separate contracts to RAND and Eagle Group International, in collaboration with Abt Associates Inc. and Voxiva Inc. The work was conducted jointly by Abt Associates, Voxiva, and the RAND Center for Military Health Policy Research and RAND Arroyo Center. The latter is a federally funded research and development centre for the US Army.