A comparison of 4 epinephrine autoinjector delivery systems: usability and patient preference

Abstract

Background

Prefilled epinephrine autoinjectors are sometimes improperly used by patients, caregivers, and physicians. A user-centered design process led to the development of 2 prototype epinephrine autoinjectors (INT01 and INT02) that have a unidirectional perceived injection end, a self-retracting needle, and, with INT02, voice instructions to assist in guiding users through administration.

Objective

To compare the usability and patient preference among 4 epinephrine autoinjectors: EpiPen, TwinJect, INT01, and INT02.

Methods

A total of 48 participants were divided equally among 3 age groups: 7 to 10, 11 to 15, and 16 to 55 years. In each group, half had prior TwinJect or EpiPen training. In 1-hour sessions, without training, participants performed simulated-use testing under observation for all 4 epinephrine delivery systems. Usability (ie, the ability to perform the manufacturer’s labeled instructions), task completion time, and preferences were assessed and analyzed based on device, age, previous experience, sex, device malfunction, and testing order.

Results

More participants correctly followed all device instructions with INT02 (22 [46%]) than with INT01 (13 [27%]), EpiPen (6 [12%]), or TwinJect (0 [0%]). The difference among devices was significant (P < .01) after adjusting for device malfunctions and age group (the youngest age group [those aged 7–10 years] performed significantly worse than the other 2 groups). Prior experience, sex, and testing order did not significantly affect this measure. The first choice of overall preference was greater (P < .001) for INT02 (35 participants [73%]) vs INT01 (7 participants [15%]), EpiPen (5 participants [10%]), and TwinJect (1 participant [2%]).

Conclusion

The user-centered device design may have a significant impact on correct epinephrine autoinjector use and patient preference.

INTRODUCTION

Anaphylaxis is a serious allergic reaction that is rapid in onset and can be fatal. The at-risk, severely allergic population has increased substantially during the last decade, with the current incidence now estimated to be 49.8 per 100,000 person-years.¹ In the occurrence of an anaphylactic reaction, an injection of epinephrine (adrenaline) within minutes of the onset of symptoms can be lifesaving. Prefilled epinephrine autoinjectors, such as the EpiPen and TwinJect, are commonly prescribed for out-of-hospital treatment of at-risk patients. Such autoinjectors are designed to be used by individuals in a prehospital environment, yet there have been several recent reports focusing on errors made by patients who experience anaphylaxis and their caregivers when administering epinephrine autoinjections.^2–7 In a study of EpiPen administration by junior and senior medical staff at a major tertiary pediatric hospital, Mehr et al³ found that only 2% of practitioners performed all 6 administration steps correctly; in 37% of cases, adrenaline would not have been delivered to the patient. Another study assessing use of self-administered epinephrine autoinjectors among food-allergic adolescents, caregivers, and pediatricians found that only 93 (38%) of patients/parents, 25 (21%) of attending pediatricians, and 11 (36%) of pediatric residents could demonstrate the correct use of the EpiPen device. More than 50% of the demonstrations would not have resulted in any administration of epinephrine during an allergic emergency.⁴

Having a more intuitive and usable device could abrogate many of the common use errors. Thus, 2 alternative autoinjector prototypes (INT01 and INT02) were designed following a human factors engineering method, an approach that strives to build systems that conform to human capabilities and limitations.^8,9 The design process was iterative, with early prototypes evaluated by expert usability practitioners and by potential users, including patients, caregivers (ie, parents of food-allergic individuals), and health care providers (ie, physicians and nurses).

To compare the INT01 and INT02 prototypes with the EpiPen and TwinJect systems, we conducted a simulated-use experiment and evaluated participants’ ability to complete a successful injection; this experiment was followed by a preference study that had participants rank the devices on several predefined criteria.

METHODS

Study Design

This study was approved by the University of Virginia, Charlottesville, institutional review board.

Figure 1 depicts the 4 devices used in the study, shown to scale in terms of relative shape and size. Both TwinJect and EpiPen are cylindrical and are 16 cm tall. The INT01 and INT02 devices are half that height and are rectangular, similar to a cell phone. The INT02 device uses visual signals from light-emitting diodes and incorporates a voice-prompt system to assist in guiding the user through the injection process. Both the INT01 and INT02 devices have a needle that retracts into the casing after injection to eliminate any postinjection sharps hazards and to prevent the user from seeing the needle before, during, or after the injection. A single safety guard covers the injection end to avoid confusion about where the needle comes out.

Figure 1.

Study devices: 4 tested devices, with protective cases removed, shown to scale.

To compare devices for multiple user populations with sufficient statistical power, a within-participants protocol with 48 participants was designed. Specifically, there were 16 participants in each of 3 age groups: 7 to 10, 11 to 15, and 16 to 55 years; half of the participants in each age group were defined as experienced with epinephrine autoinjectors (ie, they had seen, used, or been trained on how to use an EpiPen or TwinJect within the last 8 years), and the other half were defined as inexperienced (ie, they had no prior experience with TwinJect or EpiPen) (Table 1).

Table 1.

Demographics of the 48 Participants by Age and Experience Level

	Experience with autoinjectors according to age group						Total (N = 48)
	Younger pediatric (7–10 y) (n = 16)		Older pediatric (11–15 y) (n = 16)		Adults (≥15 y) (n = 16)
	Yes (n = 8)	No (n = 8)	Yes (n = 8)	No (n = 8)	Yes (n = 8)	No (n = 8)
Age, mean y (SD)	8.25 (0.89)	8.25 (0.71)	12.88 (1.36)	12.75 (1.67)	36.88 (12.89)	27.63 (6.14)	NA
Sex, No. (%)
Male	5 (62)	5 (62)	6 (75)	1 (12)	2 (25)	5 (62)	24 (50)
Female	3 (38)	3 (38)	2 (25)	7 (88)	6 (75)	3 (38)	24 (50)

Abbreviation: NA, not applicable.

Trainer models of the 2 current commercially available epinephrine autoinjectors in the United States (ie, EpiPen and TwinJect), and trainer models of 2 epinephrine autoinjectors in development (ie, INT01 and INT02), were used in the study. The device testing order was counterbalanced across participants of each age and experience combination (ie, each device appeared in the first, second, third, and fourth testing position the same number of times). In addition, INT01 and INT02 were never tested in direct succession because they have common design elements.

Native English speakers with at least a second-grade reading proficiency were recruited in April and May 2008 via flyers posted at community centers, hospitals, clinics, and grocery stores throughout the Charlottesville, Virginia, community until recruitment goals were met. The study was held between May 10, 2008, and June 12, 2008, in a designated room at the University of Virginia School of Engineering and Applied Science. Participants were informed of all study risks and benefits via a written informed consent form, were provided with a parking pass if needed, and were paid $50 for participation. The study took approximately 1 hour per participant, and each trial was videotaped. Individuals were not allowed to observe other participants before participation. Parents were able to watch the trial from a separate room on videotape, if desired. The participants were not given information about the sponsor company until the end of the study.

Procedures

The first phase was a simulated-use test (without blinding for manufacturer), and the second phase involved a preference evaluation. Participants were not trained or retrained on the use of the devices and were not allowed to view the devices before beginning the study. For the simulated-use phase, participants were first read an introductory scenario that described anaphylaxis. Then, for each product in turn, a labeled paper bag with the product inside was placed in front of the participant and an identical scenario description was read, except for the product name, which was changed to reflect the name of the product about to be used. The scenario described a life-threatening allergic reaction, including a description of typical symptoms, and asked the participant to self-inject epinephrine via an autoinjector. At this point, the participant was asked to begin and a series of beeps was played through speakers until the participant finished. The beeps started at a frequency of 1 per second; at 30 seconds, the beeps increased in pace and intensity to 2 per second; and at 1 minute, they increased again to 3 per second to induce stress to the testing environment. Throughout the study, the investigator did not provide any training or aid; participants were expected to rely on the instructions provided on the device and/or previous experience or intuition to perform the simulated injection correctly. The product was put away, and the next product, in its bag, was placed in front of the participant and the procedure repeated until all 4 devices had been tested. Even though TwinJect has a second dose available, it is manually injected after disassembly of the autoinjector; this was not included in the procedure to have an accurate comparison across autoinjector products.

After completing all 4 simulated-use tests, participants were shown an actual, nontrainer, version of each of the autoinjector devices (including the devices post injection with needles exposed, as applicable) and were asked to complete a questionnaire that included items on the clarity of written and graphic instructions, the durability of the product, ease of use, and portability. After the questionnaires were completed, the investigator informed the participant of any errors that he or she made during the simulated-use phase. Finally, participants were asked to rank the 4 products by moving them on the table into the preferred ranking order from left to right and to explain their reasoning as they did so for each of the following criteria: size preference, shape preference, preference to carry, ease of use, clarity of instructions, perceived safety, and overall preference. Ties were allowed for the preference rankings. For example, a participant could rank both INT01 and INT02 first, EpiPen third, and TwinJect fourth. In this case, the ranks would be coded as 1 for INT01 and INT02, 3 for EpiPen, and 4 for TwinJect. The procedure for the usability test was specifically developed to compare the 4 epinephrine autoinjector delivery systems by the authors (S.G., A.H., and L.W.), who are experienced in human factors design and evaluation.

Statistical Analyses

All participants were included in the analysis in the groups to which they were originally assigned. Following device protocol was strictly defined as following the device instructions (Table 2) (not including the step to “call 911”) and not making a single error (Table 3). Two investigators (A.H. and L.W.) reviewed the simulated-use test videotapes to make these judgments, and the results were compared. Any scoring discrepancies were reviewed and resolved by consensus. To model the outcomes, a generalized linear model with an autoregression correlation structure for the correlated measures within each participant was used. Binary outcomes were converted to continuous measures using the logit transformation. Outcomes were modeled with a single covariate: device, age group, testing order, sex, and prior experience. Then, the effect of device on outcomes was adjusted using additional covariates that were identified from the single covariate models with P < .05.

Table 2.

Manufacturer Instructions for Correct Injection

TwinJect	EpiPen	INT01 and INT02
Pull off the GREEN end cap labeled “1.”	Pull off the gray safety release.	Pull off RED safety guard.
Pull off the GREEN end cap labeled “2.”	Swing and jab firmly into outer thigh until it clicks.	Place BLACK end against outer thigh.
Put the rounded RED tip against the middle of the outer side of your thigh (upper leg).	Hold firmly against thigh for approximately 10 seconds.	Push down firmly and hold for 5 seconds.
Press down hard until the needle enters your thigh (upper leg) through your skin.	Remove unit from thigh.	Call 911.
Hold it in place while slowly counting to 10.	Call 911 and seek immediate medical attention.
Remove the TwinJect from your thigh.
Get emergency medical help right away.
Call 911.

Table 3.

Data for Each Type of Error for Each Device for the 48 Participants

Type of error	INT01	INT02	EpiPen	TwinJect	P value
1. Does not remove safety cap(s)	1	3	0	7	.07
2. Touches injection tip before injection	6	5	11	11	.30
3. Unintentional injection into hand or digit	0	0	6	2	.10
4. Injected in location other than muscle (ie, hip, arm, stomach)	6	5	8	7	.57
5. Injected in muscle, but not the outer thigh (ie, top of thigh, deltoid, or calf)	9	6	4	12	.05
6. Did not hold for correct amount of time	27	11	40	42	<.001
7. Moved device significantly during injection	4	1	4	3	.25
8. Injected more than once	6	7	4	5	.76
9. Attempted to disassemble device	0	0	2	5	.18
10. Dropped device before injection, and misfire could have occurred	0	0	0	1	.39
11. Did not jab device or press down hard enough	2	2	5	2	.40
12. Removed caps in wrong ordera	0	0	0	18	<.001
13. Did not inject at all	2	3	1	1	.76
Total No. of errors	63	43	85	116	<.001

^aThis is not possible for INT01, INT02, or EpiPen because they only have 1 safety cap.

RESULTS

Completing Labeled Device Instructions (Following Protocol)

Table 4 gives the number and proportion of participants who correctly followed the device protocol by device type, experience, and age group. Age group significantly affected this measure (P = .01). The younger pediatric group had a lower protocol completion rate than both the older pediatric group (odds ratio [OR], 9.17; P = .005) and the adult group (OR, 6.14; P = .01). Device malfunction also affected the likelihood of following the device protocol (P = .01). When the device malfunctioned during the test, the likelihood of failing to follow protocol was higher (OR, 15.67; P = .045). The INT02 prototype malfunctioned 7 times (a voice instruction was skipped, repeated, or out of place), the INT01 prototype malfunctioned 3 times (the safety cap came off at the same time as the outer case), and the EpiPen malfunctioned 1 time (the safety cap came off at the same time as the outer case). This caused errors in 4, 0, and 1 case, respectively.

Table 4.

Frequency of Correctly Following Protocol

Age group, ya	Experience with autoinjectors	Correctly following protocol, %b
		INT01	INT02	EpiPen	TwinJect
7–10	No	0 (0)	0 (0)	0 (0)	0 (0)
	Yes	1 (12)	3 (38)	1 (12)	0 (0)
11–15	No	4 (50)	5 (62)	1 (12)	0 (0)
	Yes	4 (50)	4 (50)	2 (25)	0 (0)
16–55	No	1 (12)	7 (88)	2 (25)	0 (0)
	Yes	3 (38)	3 (38)	0 (0)	0 (0)

^aThe age group mean values were as follows: 7 to 10 years, 7.8%; 11 to 15 years, 31.2%; and 16 to 55 years, 25.0%.

^bData are given as number (percentage) of each group, using 8 as the denominator. The device mean values were as follows: INT01, 27.1%; INT02, 45.8%; EpiPen, 12.5%; and TwinJect, 0.0%.

After adjusting for age group and device malfunction, following protocol was significantly different among the devices tested (P < .001). Participants followed the labeled device instructions more often with INT01 and INT02 compared with EpiPen (OR, 8.36 [P < .001] and 3.95 [P = .002], respectively) and TwinJect (no participant followed the TwinJect protocol; therefore, ORs are not reported).

Preference

Figure 2 shows the first-choice rankings by product and preference category. An overwhelming majority ranked INT02 first compared with the existing products (EpiPen and TwinJect) for safety, clarity of instructions, ease of use, ease of carrying, shape, and size. We report on final overall preference, which was significantly different between devices (P < .001). None of the covariates (sex, order, age, experience, or device malfunction) was significant. Contrasts show that INT02 was significantly preferred to INT01 (OR, 14.68; P < .001), EpiPen (OR, 20.80; P < .001), and TwinJect (OR, 96.63; P < .001) and that INT01 was significantly preferred to TwinJect (OR, 6.58; P = .03).

Figure 2.

User device preference. Percentage of respondents (n = 48) who ranked each device first (tie allowed, sum could be more than 100%).

DISCUSSION

Studies have shown that patients and caregivers do not always correctly administer epinephrine autoinjector devices. Such concerns may be surmountable with patient education on correctly injecting the epinephrine, but a lack of time and resources may minimize the support health practitioners are able to provide.^10–12 In addition, some physicians are not familiar with these devices and may fail to review their use with patients.⁶ There may also be a large time lapse (several years) between when a person is trained on an autoinjector and when it must be used during an allergic reaction. Finally, a patient or care provider may be under significant stress while attempting to provide the potentially life-saving dose of epinephrine when it is used. Thus, devices that can be used with little to no prior training are preferable.

Recently, legislative bills have been passed that allow children 8 years and older with a history of food allergy reactions to carry their epinephrine autoinjector in school.¹³ The results from this study indicate that the untrained younger pediatric group (aged 7–10 years) was much less capable of using these devices than the adolescent and adult groups. Training, however, helped patients use the devices correctly. In our study, 7 of the 8 children aged 7 to 10 years who had prior EpiPen experience used the EpiPen correctly, whereas only 1 of 8 of the inexperienced 7- to 10-year-old EpiPen users used the EpiPen correctly. For the EpiPentrained 7- to 10-year-old children, their training may have carried over to some degree, improving their likelihood of following labeled instructions for the other devices, but not for the other age groups (Table 4). This may indicate that the young children who had experience with EpiPen were trained more recently or more carefully (as a result of their young age) than those in the other 2 age groups. Future studies should measure time since last training and use as a continuous rather than a dichotomous variable.

An additional benefit of usability testing during the development stage is that it enables the manufacturers to identify and correct any mechanical malfunctions or design concerns that may arise. Malfunctions occurred in the study for several of the trainer devices. The INT02 device had 7 voice-prompting malfunctions that, in 4 cases, resulted in users making errors. For example, in 5 cases, the device continued to repeat the following instruction: “To inject, press black end against outer thigh,” even though the participant had already done so. This caused 2 participants to try to inject the device twice; however, for the other 3 participants, no error occurred. The interactive voice-prompting system is completely independent of the device injection mechanism. Thus, even though a voice-prompting failure might occur, in all cases the system will inject and retract the needle if the safety cap is removed and the device is pressed down against an injection site. Because of these voice-prompting malfunctions, the device manufacturer has redesigned the voice-prompting system. Future usability testing is planned to validate these design changes and to determine whether they result in an additional increase of “followed protocol” percentage for the INT user population.

The other type of malfunction that occurred once with both INT01 and EpiPen was that when the outer case was removed, the safety cap was removed inadvertently as well. For the participant using INT01, this did not cause any errors. For the participant using EpiPen, this resulted in 2 errors (errors 2 and 9): touching and attempting to unscrew the black needle end.

Studies have shown that testing a product on as few as 4 or 5 potential users allows investigators to discover 80% of a product’s usability problems.¹⁴ This study identified a variety of usability errors that were committed by participants during the study (Table 3). For example, known usability issues with existing autoinjectors include the possibility of attempting an injection using the wrong end of the device (causing an unintentional injection into a hand or digit).¹⁵ Two types of errors occurred in this study, highlighting this usability concern. Touching the injection tip before injection was more prevalent using the TwinJect and EpiPen devices than using the INT01 and INT02 prototypes; digital injection occurred for participants using the EpiPen and the TwinJect devices but never the INT prototypes. For the TwinJect and EpiPen, this latter error occurred presumably because of the symmetrical design, with both ends equally “affording” a push action and an injection action. The INT prototypes were designed in such a way so as to eliminate this hazard. The TwinJect device caused 2 errors relating to the device’s 2 safety caps. The first error (n = 7) was because participants failed to recognize the need to remove the second cap on the device. The 2 caps also contributed to participants removing the caps in the wrong order 18 times.

Not holding for the correct amount of time was the most common error for each device. The INT02 device resulted in participants committing this error 11 times compared with 27 (INT01), 40 (EpiPen), and 42 (TwinJect) times. The count-down feature on the INT02 device likely aided in reducing the occurrence of this error. Training may prevent several other errors from occurring, including becoming familiar with the correct injection site, knowing how hard to press or jab the device for activation, and not moving the device during injection.

The results from this study are preliminary given the sample size, the discovery of an electronic malfunction with the INT02 interactive voice-prompting system, and the relatively unstressful test setting. Although the study was designed to induce stress to participants via the script that was followed and the simultaneous, timed beeps, participants were clearly not as stressed as one might imagine during a real allergic emergency.

Participants did not have a long time to evaluate the devices; they were asked to perform a single-use scenario without any prior training during the study. This may be indicative of how anaphylactic patients may use the device in a real-world scenario because patients would likely not be up-to-date on their autoinjector training before an event occurred. The fact that less than 50% of participants across all devices could follow the labeled instructions without committing a single error provides confirmation that the need for training on the use of epinephrine autoinjectors is still important.

The INT02 prototype performance suggests that significant progress can be made in reducing the overall likelihood of participants making errors by changes in device design. Despite the interactive voice prompting malfunctioning 7 times, the results showed significantly fewer overall errors with this device than those currently on the market. Future usability studies can incorporate training users on the proper injection technique based on the labeled instructions and then having users conduct the usability testing.

In addition, a longer-term “field test” is recommended to evaluate other features, such as the likelihood of keeping the device with the patient at all times, because failure to carry the device at all times is a known issue with existing epinephrine autoinjectors. A future field study could include collecting objective measures, by attaching a Global Positioning System to the patient’s device(s) and to the patient, to determine if the device is always in close proximity. In addition, a temperature sensor could be included on the devices to determine whether users are storing them in the correct environment.

Finally, health care practitioners were not included in the study; their use and preferences should be evaluated as well, particularly in terms of ease of training patients and patient compliance.