Abstract
Objective
To evaluate the ability of different types of vaccine storage units to maintain appropriate temperatures for the storage of vaccines and to characterize deviations from recommended temperatures.
Data sources
Continuous temperature monitoring devices, or digital data loggers, from vaccine providers who participated in a continuous temperature monitoring pilot project.
Study design
We computed descriptive statistics on the percentage of runtime with an out-of-range temperature, or excursion, for different storage unit types (freezers and refrigerators) and for different storage unit grades (household-grade combination, household-grade stand alone, and purpose-built or pharmaceutical grade). We developed frequency histograms for the percentage of storage unit runtime outside of the normal range. We plotted the duration and temperature extrema for identified excursions. Analyses were stratified by storage unit type and grade.
Results
Household-grade combination units underperformed relative to household-grade stand-alone and purpose-built units. Among refrigerators, household-grade combination units operated in the normal temperature range an average of 98.9% of their observed runtime, which was lower than 99.4% (p value = 0.038) for household-grade stand-alone and 99.9% (p value < 0.001) for purpose-built units. Among freezers, household-grade combination units operated in the normal temperature range an average of 95.0% of their observed runtime, which was lower than 99.3% (p value < 0.001) for household-grade stand-alone units and 99.7% (p value < 0.001) for purpose-built units.
Conclusion
These findings, in particular the underperformance of household-grade combination units relative to household-grade stand-alone and purpose-built units, support current CDC recommendations to avoid the use of household-grade combination storage units when possible.
Keywords: vaccines, storage units, cold-chain, continuous temperature monitoring, digital data logger
The management of temperature levels in vaccine storage units and vaccine supply systems is an important issue for maintaining viable supplies of vaccine for immunization programs [1–3]. This is an issue that affects immunization programs in the United States [4] and those around the world [5, 6]. Recent studies have suggested that cold-chain lapses were potential causes of vaccine wastage [7–9]. Given the substantial public health and economic value attributed to vaccines [10–13], research and intervention efforts have emerged to promote and maintain the cold-chain in vaccine distribution. Recent research efforts have investigated vaccine transportation [14], packaging and logistics [15, 16], thermostability [3, 17–19], environmental factors [15], and improved temperature monitoring of vaccine storage units [2, 20–24]. The thermostability of vaccines has been a long-standing subject of research, recognized as a source of vaccine supply reliability and a way to ensure potency under diverse storage and handling conditions [18, 19].
The most recent Vaccine Storage & Handling Toolkit (Toolkit) provides several recommendations for vaccine storage practices [25]. The Toolkit recommends vaccines be stored between 2°C and 8°C (36°F and 46°F) for refrigerated vaccines and between −50°C and −15°C (−58°F and 5°F) for frozen vaccines. The Toolkit recommends the use of purpose-built storage units, also called pharmaceutical-grade storage units, over household-grade stand-alone storage units or household-grade combination storage units. If a purpose-built storage unit is not available, the Toolkit recommends household-grade stand-alone storage units over the use of combination storage units, which may also be referred to as dual-zone units [26]. In addition, the Toolkit explicitly recommends against the use of either dormitory-style units or the freezer compartments of combination units. The Toolkit also recommends using digital data loggers (DDLs) to monitor temperatures in storage units [25]. DDLs are electronic devices connected to a temperature probe that is usually suspended in buffered material such as a glycol solution. Temperature data is collected by DDL at small time intervals, such as 5, 10, or 15 minute intervals and recorded electronically. The value of DDLs in monitoring storage unit temperatures has been supported by findings from a series of reports by the National Institute of Standards and Technology [15, 22, 26].
In terms of both the numbers of storage units assessed and the length of the observation periods, recent research on vaccine storage units has focused on a smaller scope, with sample sizes equal to seven vaccine providers [15, 26], seven digital data loggers [22], or 45 provider offices over 2 weeks [4]. Our study adds to the literature by investigating the stability of vaccine storage unit temperatures collected in a real-world setting from a large number of provider offices, capturing 320 provider offices with 783 storage units implementing DDLs. Our study evaluates the performance of different types of storage units and provides evidence that supports the Toolkit recommendations on the prioritization of purpose-built storage units.
Methods
In 2012 a cooperative agreement funded by the Prevention and Public Health Fund (PPHF) was awarded to twenty-two state immunization programs to improve vaccine storage and handling at the awardee and provider level. One of the priorities for this cooperative agreement was to promote and expand the use of DDLs [27]. Raw DDL data was transmitted from provider offices to the immunization program project staff. Data included in this study came from three immunization programs and included 320 providers. The immunization project staff utilized a software application to compress and aggregate the raw data into a processed format that was easier to store and more amenable to later analysis and interpretation. Each observation in the processed data is associated with a time period of normal temperature or a time period of out-of-range temperature. For each time period, the minimum and maximum observed temperature was recorded. An illustrative example of the raw DDL data and the corresponding processed data can be found in Table 1. This study used processed data from 783 storage units. From the processed data, two analytical datasets were constructed. One analytical dataset aggregates the observations from the processed data into a dataset where each individual storage unit is the observational unit. The other analytical dataset contains only the observations representing temperature excursions. In the cases where household-grade combination storage units stored vaccines in both the refrigerator and freezer compartments, these two compartments are evaluated as separate units in the analytical dataset. For this study, storage units were characterized by their type, which indicates if the storage unit is a refrigerator or a freezer, and by their grade, which indicates if the storage unit is a household-grade combination unit, household-grade stand-alone unit, or a purpose-built unit. The characterizations of the storage units were all self-reported. As an analysis of data without reference to any human subjects or any other identifiers, the study did not require institutional review board review.
Table 1.
Illustrative example of (a) raw data from a digital data logger and the corresponding (b) processed data from one refrigeration storage unit.
| (a) Raw data | |||||
|---|---|---|---|---|---|
| Observation | Provider ID | Storage unit ID | Date | Time | Temperaturea (°C) |
| 1 | 1234 | R1 | 7/1/14 | 8:00 AM | 5.0 |
| 2 | 1234 | R1 | 7/1/14 | 8:20 AM | 5.0 |
| 3 | 1234 | R1 | 7/1/14 | 8:40 AM | 5.5 |
| 4 | 1234 | R1 | 7/1/14 | 9:00 AM | 6.0 |
| 5 | 1234 | R1 | 7/1/14 | 9:20 AM | 6.5 |
| 6 | 1234 | R1 | 7/1/14 | 9:40 AM | 7.0 |
| 7 | 1234 | R1 | 7/1/14 | 10:00 AM | 7.5 |
| 8 | 1234 | R1 | 7/1/14 | 10:20 AM | 7.7 |
| 9 | 1234 | R1 | 7/1/14 | 10:40 AM | 8.5 |
| 10 | 1234 | R1 | 7/1/14 | 11:00 AM | 9.0 |
| 11 | 1234 | R1 | 7/1/14 | 11:20 AM | 9.0 |
| 12 | 1234 | R1 | 7/1/14 | 11:40 AM | 9.0 |
| 13 | 1234 | R1 | 7/1/14 | 12:00 PM | 8.7 |
| 14 | 1234 | R1 | 7/1/14 | 12:20 PM | 8.5 |
| 15 | 1234 | R1 | 7/1/14 | 12:40 PM | 8.2 |
| 16 | 1234 | R1 | 7/1/14 | 1:00 PM | 7.4 |
| 17 | 1234 | R1 | 7/1/14 | 1:20 PM | 7.1 |
| 18 | 1234 | R1 | 7/1/14 | 1:40 PM | 7.0 |
| 19 | 1234 | R1 | 7/1/14 | 2:00 PM | 6.5 |
| 20 | 1234 | R1 | 7/1/14 | 2:20 PM | 5.0 |
| 21 | 1234 | R1 | 7/1/14 | 2:40 PM | 5.0 |
| 22 | 1234 | R1 | 7/1/14 | 3:00 PM | 4.5 |
| 23 | 1234 | R1 | 7/1/14 | 3:20 PM | 4.0 |
| 24 | 1234 | R1 | 7/1/14 | 3:40 PM | 10.1 |
| 25 | 1234 | R1 | 7/1/14 | 4:00 PM | 7.0 |
| 26 | 1234 | R1 | 7/1/14 | 4:20 PM | 5.0 |
| 27 | 1234 | R1 | 7/1/14 | 4:40 PM | 4.5 |
| 28 | 1234 | R1 | 7/1/14 | 5:00 PM | 5.0 |
| (b) Processed data | ||||||||
|---|---|---|---|---|---|---|---|---|
| Observation | Provider ID | Storage unit ID | Start date-time | End date-time | Excursion | Min. Temp. (°C) | Max. Temp. (°C) | Durationb (Hours) |
| 1 | 1234 | R1 | 7/1/14 8:00 AM | 7/1/14 10:20 AM | No | 5.0 | 7.7 | 2.50 |
| 2 | 1234 | R1 | 7/1/14 10:40 AM | 7/1/14 12:40 PM | Yes | 8.2 | 9.0 | 2.33 |
| 3 | 1234 | R1 | 7/1/14 1:00 PM | 7/1/14 3:20 PM | No | 4.0 | 7.4 | 3.67 |
| 4 | 1234 | R1 | 7/1/14 3:40 PM | 7/1/14 3:40 PM | Yes | 10.1 | 10.1 | 0.33 |
| 5 | 1234 | R1 | 7/1/14 4:00 PM | 7/1/14 5:00 PM | No | 4.5 | 7.0 | 1.17 |
Note(s): C refers to Celsius.
The acceptable temperature range for this example is 2–8°C, so the out-of-range temperatures found in the raw data correspond to observations numbered: 9 to 15, and 24.
When calculating durations, the amount of time in between two observations (20 minutes in this example) is divided equally between two sequential observations.
Data cleaning
As part of the data verification process, we checked that the reported temperature ranges were appropriate for the type of storage unit that was identified. Some data appeared inconsistent, for example, data was labeled as being from a ‘freezer’ but temperatures were consistent with refrigerator units. These observations were excluded from the analysis. In other cases, several observations appeared to represent a storage unit that had only one recorded time interval, which was outside of the normal temperature range. These observations were also excluded due to the likelihood that the DDLs could have been installed improperly or could represent a single erroneous reading. We excluded data from storage units for which the observation period was less than one week. Having an observation period of less than a week might indicate a test or trial period for the DDL or for the storage unit. The one week threshold was partially based on the Toolkit recommendation that new storage units and recently repaired storage units undergo a trial period of 2 to 7 days before being used to store vaccine [25]. As a final precaution, in the main analysis excursion observations that were in the most extreme 2.5th percentile of temperatures observed were excluded. This removed additional excursions that appeared to be inconsistent with unit type, such as a temperature of −15°C in a refrigerator unit. We conducted two sensitivity analyses where we repeated our main analysis where (1) we remove this restriction so these extreme observations are included and (2) we increased this restriction so that excursion observations in the 5th percentile of extreme temperature were excluded.
Analysis
For the first analytical dataset, which contains data aggregated to the level of individual storage units, the means and standard errors were computed for the percentage of runtime that a storage unit experienced high excursions, low excursions, and normal temperatures, and for the number of high or low excursions per 30 days. The number of excursions per 30 days was computed from an excursion rate, which was equal to the number of observed excursions divided by the cumulative observed runtime for each storage unit. This excursion rate was then scaled up to a 30-day interval to be more concrete and to simplify presentation. To provide additional context and to make our results more comparable to a previous study [4], we extrapolated the average percentage of runtime to represent a two week period and reported cumulative hours of runtime with high, low, and normal temperatures for each type and grade of storage unit. Welch two-sample statistical tests were used to compare the mean percentage of runtime without excursions between the groups of different storage unit grades. Results from the Welch two-sample statistical tests are reported in terms of statistical significance and estimated p-values. A difference in means that is statistically significant at the 5% level reflects that the probability of the null hypothesis being true is equal to or less than 0.05. In our case the null hypothesis for each of the tests is an equal mean runtime without excursion between two groups of storage units. For both refrigerators and freezers, these tests compared household-grade combination vs. household-grade stand-alone, household-grade combination vs. purpose-built, household-grade stand-alone vs. purpose-built, and household-grade combination vs. non-combination, where the non-combination group was the pooled sample of household-grade stand-alone and purpose-built. In sensitivity analyses, the Welch two-sample statistical tests on the difference mean percentage of runtime without excursions across vaccine storage unit type were conducted on the two subsamples with different exclusion criteria for extreme value temperatures.
To visualize the distribution of storage unit performance and to characterize temperature excursion data, we generated two figures. The percentage of runtime with any excursion for each storage unit was presented as a frequency histogram, with a histogram bin width of 0.5% in Figure 1. The second analytical dataset is used to characterize temperature excursions. For the second analytical dataset, which tabulated individual excursions, the temperature level and length of time of each duration are plotted in Figure 2. All analyses were stratified by storage unit type and storage unit grade. This analysis was conducted using the R statistical package, version 3.53.
Figure 1. Frequency histograms of the percentage runtime with out-of-range temperatures (either high or low) from several types of vaccine storage refrigerators: (a) household-grade combination, (b) household grade, stand-alone, (c) purpose-built; and freezers: (d) household-grade, combination, (e) household-grade, stand-alone, and (f) purpose-built.
Note(s): Histogram x-axis bin width is 0.5%. The x-axis limit of 0.2 was chosen to simplify presentation and ease comparison across panels. This truncation resulted in dropping from presentation the following number of observations: 2 from panel a, 16 from panel d, and 1 from panel e.
Figure 2. Duration and temperatures associated with identified excursions from several types of vaccine storage refrigerators: (a) household-grade combination, (b) household-grade, stand-alone, (c) purpose-built; and freezers: (d) household-grade, combination, (e) household-grade, stand-alone, and (f) purpose-built, purpose-built.
Note(s): Refrigerator out-of-range temperature thresholds are high > 8 °C, low < 2 °C, and freezer out-of-range threshold is high > −15 °C. Each point in the figures above represents the most extreme temperature and the duration of a single excursion. Points are partially transparent so the regions with greater numbers of observations appear darker.
Results
From a total of 43,951 hours of observation time, household-grade combination refrigeration units exhibited normal temperatures 98.9% of observed runtime and exhibited low (and high) excursions 0.5% (and 0.6%) of the time (Table 2). Of a total of 18,428 hours of observation time, stand-alone refrigeration units exhibited normal temperatures 99.4% of observed runtime and exhibited low (and high) excursions 0.2% (and 0.3%) of the time. Of a total of 15,012 hours of observation time, purpose-built refrigeration units exhibited normal temperatures 99.9% of observed runtime and exhibited low (and high) excursions less than 0.1% (and less than 0.1%) of the time. The difference in observed time across storage-unit grades is related to sample size. This sample contained more household-grade combination units (243) than household-grade stand-alone units (105), and more household-grade stand-alone units than purpose-built units (71) (Table 2). In Welch two sample t-tests, the mean percentage of runtime in the normal-temperature range was statistically different at the 5% level (p value = 0.038) between the household-grade combination and household-grade stand-alone units, and was statistically different at the 1% level (p value < 0.001) between the household-grade combination refrigerator and the purpose-built units. Household-grade stand-alone units were statistically different at the 1% level (p value = 0.005) from the purpose-built units. Household-grade combination units were found to be statistically different from non-combination units at the 1% level (p value < 0.001). When these statistical tests were repeated in sensitivity analyses, all results were qualitatively similar except the household-grade combination unit was not statistically different from the household-grade stand-alone unit (p value = 0.209) in the sample with no exclusions (Appendix).
Table 2.
Characteristics of a sample of vaccine storage units related to temperature stability, stratified by storage unit type (refrigerator or freezer) and storage unit grade (household-grade combination, household-grade stand-alone, purpose-built).
| Refrigerators | Freezers | |||||||
|---|---|---|---|---|---|---|---|---|
| Household-grade | Purpose-built | Household-grade | Purpose-built | |||||
| Combination | Stand-alone | Combination | Stand-alone | |||||
| Percent of runtime in a, given temperature state | Low temperature | Mean (%) SE | 0.50 | 0.24 | 0.02 | a | a | a |
| 0.15 | 0.10 | 0.01 | ||||||
| High temperature | Mean (%) SE | 0.63 | 0.34 | 0.04 | 4.99 | 0.65 | 0.30 | |
| 0.12 | 0.12 | 0.01 | 0.63 | 0.22 | 0.16 | |||
| No excursion (normal range) | Mean (%) SE | 98.87 | 99.42 | 99.94 | 95.01 | 99.35 | 99.70 | |
| 0.20 | 0.18 | 0.02 | 0.63 | 0.22 | 0.16 | |||
| Number of excursions per 30 days | Low temperature | Mean SE | 3.52 | 2.83 | 0.23 | |||
| 0.99 | 1.22 | 0.10 | ||||||
| High temperature | Mean SE | 1.76 | 1.33 | 0.21 | 13.63 | 1.87 | 1.47 | |
| 0.32 | 0.29 | 0.06 | 1.39 | 0.41 | 0.77 | |||
| Hours of observation | Mean SE | 180.87 | 175.50 | 211.43 | 195.52 | 179.39 | 171.77 | |
| 7.62 | 11.38 | 11.46 | 9.67 | 10.06 | 16.01 | |||
| Number of storage units | Total Count | 43,950.66 | 18,427.76 | 15,011.74 | 37,734.57 | 24,576.31 | 5,840.23 | |
| 243 | 105 | 71 | 193 | 137 | 34 | |||
|
Cumulative runtime extrapolated to two weeks (or 336 hours) |
||||||||
| Low Temperature excursions | Hours | 1.68 | 0.80 | 0.06 | ||||
| High temperature excursions | Hours | 2.12 | 1.13 | 0.15 | 16.77 | 2.20 | 1.02 | |
| No excursion | Hours | 332.20 | 334.07 | 335.79 | 319.23 | 333.80 | 334.98 | |
| Total | Hours | 336 | 336 | 336 | 336 | 336 | 336 | |
Note(s): SE refers to standard error of the mean; Combination refers to units with a freezer and a refrigerator compartment.
Low temperature excursions among freezer units were not documented in the study.
Of a total of 37,735 hours of observation time, household-grade combination freezer units exhibited normal temperatures 95.0% of observed runtime and exhibited high excursions 5.0% of the time (Table 2). Of a total of 24,576 hours of observation time, household-grade stand-alone freezers units exhibited normal temperatures 99.3% of observed runtime and high excursions 0.7% of the time. Of a total of 5,840 hours of observation time, purpose-built freezer units exhibited normal temperatures 99.7% of observed runtime and exhibited high excursions 0.3% of the time. As with the refrigerators, the difference in observed time across storage unit grades is related to sample composition, which contained more household-grade combination units (193) than household-grade stand-alone units (137), and more stand-alone units than purpose-built units (34) (Table 2). In Welch two sample t-tests, the mean percentage of runtime in the normal-temperature range was statistically different at the 1% level (p value < 0.001) between household-grade combination and household-grade stand-alone units, and was statistically different at the 1% level (p value < 0.001) between household-grade combination refrigerator and purpose-built units, but was not statistically different (p value = 0.193) between household-grade stand-alone and purpose-built units. Household-grade combination units were found to be statistically different from non-combination units at the 1% level (p value < 0.001). All results were qualitatively similar in sensitivity analyses with different exclusion criteria for the most extreme temperatures. The number of excursions per 30 days follows a similar pattern as the percentage of runtime in a given temperature state. In particular, household-grade combination storage units exhibit larger average number of excursions per 30 days (13.6), followed by household-grade stand-alone (1.9) and purpose-built units (1.5).
In the frequency histograms of Figure 1, for all but household-grade combination freezers, the majority of storage units appear to exhibit an average excursion percentage of runtime between 0 to 0.5%. We observed the greatest dispersion among combination freezer units. The figure also makes apparent the volume of data available for each unit type and unit grade, which corresponds to the number of storage units identified in Table 2. There are fewer purpose-built storage units in this sample relative to the household-grade units. Characterizations of observed temperature excursions, the duration and extreme temperatures, are presented in Figure 2, stratified by storage unit type and grade. A larger number of temperature excursions appear in the combination units (Figure 2a and 2d), due to both the performance of combination units and due to the characteristics of this sample which contains a disproportionately larger number of combination units. In terms of excursion duration, a majority of temperature excursions appear to be relatively short, and therefore excursions tend to be clustered around the lower levels of time in each panel of Figure 2. This seems particularly the case for most storage unit types (Figure 2a, 2b, 2c, 2e, 2f), perhaps except for the freezer compartments of household-grade combination units (Figure 2d). The freezers belonging to household-grade combination storage units appear to experience excursions with longer durations than the other storage unit types. The fewest number of excursions occurred among purpose-built units, likely due to both the higher performance of these units and the nature of this sample, which contains relatively fewer purpose-built storage units.
Discussion
This study presented an analysis of data collected from DDLs that were placed in different types and grades of vaccine storage units. The primary findings of this study reinforce several recommendations that have been adopted by the CDC regarding the storage and handling of vaccine supplies. We found household-grade combination units underperform relative to both household-grade stand-alone units and purpose-built units. In particular, among refrigerator units, statistically significant differences in cumulative excursion rates were found between household-grade combination and purpose-built units. Among freezer units statistically significant differences in cumulative excursion rates were found between household combination units and both purpose built units and household stand-alone units. These results were consistent across the main analysis and in sensitivity analyses. The underperformance of household-grade combination freezer units comports with findings from a study on storage units in Michigan [23]. The poor performance of household-grade combination freezer units also supports the guidance laid out by the Storage & Handling Toolkit [25] and the findings of a study by the National Institute of Standards and Technology [26], both of which advise against the use of household-grade combination freezer compartments for the storage of vaccines.
Among refrigerators, we found evidence for differences in performance between household-grade stand-alone units and purpose-built units. However, among freezers, we did not find a statistically significant difference in the percentage of runtime without excursions. This result could be due to a few potential factors. First, among freezers, temperature stability performance may in fact be very similar across household-grade stand-alone and purpose-built storage units. Secondly, while the Storage and Handling Toolkit [25] describes purpose-built storage units as having a microprocessor-based temperature control and fan-forced air circulation, in practice and by extension in our dataset, the distinctions between stand-alone storage units and purpose-built units may not be sufficiently clear or well-known to providers, who self-reported the storage-unit grade. By contrast, the distinction between combination and non-combination units would likely be less ambiguous for providers to discern and report. In the main analysis and in the sensitivity analyses, combination units underperformed relative to non-combination units.
In the most recent version of the Toolkit [25], an update was made to the Fahrenheit-based range of normal temperatures for refrigeration units. Prior to June 2016, the range in the Toolkit had been 35–46°F, whereas the new, current range is 36–46°F1. The data used in this project came from data logger systems that were using the older Fahrenheit range of 35–46°F. The revision to the normal range on the Fahrenheit scale would shift the low temperature excursion threshold in refrigeration units from 1.67°C to 2.22°C. This change would potentially reduce the number of excursions observed across all refrigerator units in the study but would be unlikely to affect any of the conclusions drawn by looking across storage unit grades, such as the underperformance of household-grade combination units.
The Advisory Committee on Immunization Practices provides recommendations for immunizations against 17 vaccine-preventable diseases with storage temperature recommendations for 22 different vaccines [28]. Several vaccines, including the diphtheria-tetanus-containing vaccines, the pneumococcal conjugate vaccine, and the human papillomavirus vaccine, are accompanied by the explicit recommendation of “do not freeze” and a justification that “irreversible loss of potency occurs with exposure to freezing temperatures” [28]. Since a onetime exposure to freezing temperatures may render all or part of a vaccine supply ineffective, evaluating storage unit performance on the basis of cumulative excursion rate alone would be an incomplete assessment of storage unit performance. In some cases, excursions might only account for 1–2% of the runtime for a vaccine storage unit, but if any single excursion exposes vaccine supplies to freezing temperatures then some vaccines may lose potency. With this in mind, particular attention may be directed towards the number of low excursion instances as opposed to just accumulated excursion time. Even if freezing excursions are short in duration, these excursions may have substantial consequences for the effectiveness of certain vaccines.
This dataset and analyses are subject to some limitations and therefore a cautionary discussion around the generalizability of results is warranted. Refrigerators and freezers from this dataset may not perform the same as would be found in the broadest possible sample of storage units, because the data used in this study comes from a convenience sample. Furthermore, this sample consisted of DDL measurements occurring in practice, so provider staff may have made adjustments to improve storage unit performance and if so, we would not be able to observe any of these adjustments. Information on the types and grades of storage units were all self-reported. Therefore, there is a risk of incorrect categorization by type or grade if, for example, in practice the distinctions between household-grade stand-alone and purpose-built storage units are at all ambiguous.
Temperature variability may occur due to other factors, such as the placement of the DDL probe within a storage unit or improper implementation of the buffered temperature probe, which could not be captured by our data sample and analysis. To account for the influence of temperature outliers, we conducted two sets of sensitivity analyses, which on balance yielded similar qualitative findings as the main analysis.
Overall, this study underscores the benefits of DDLs in the context of the vaccine supply system and, in particular, monitoring temperature stability in vaccine storage units. A large portion of storage units in this study, with the notable exception of household-grade combination freezer units, appear to perform with greater than 98.9% of observed runtime in the normal temperature range. Immunization program managers and vaccine providers who observe substantially lower levels of performance from storage units may want to investigate further to identify any issues with their storage units and any risks to their vaccine supplies. Given differences in performance across storage unit grades and the prevalence of household-grade combination storage units, future research could investigate the preferences among providers for the selection of different vaccine storage units. Another potential avenue for future research would be to better understand the costs of improving cold-chain supply systems, including the costs of different types of vaccine storage units and different temperature monitoring and data management systems. These improved systems can yield substantial benefits in terms of reducing the risk of loss of potency and minimizing potential replacement costs associated with any vaccines in storage that may experience temperature excursions. DDL-based temperature measurements, or any assessment of temperature excursions, in vaccine provider offices is just one of many factors that contribute to a resilient and effective immunization system, which includes among others: vaccine temperature stability along vaccine distribution networks, vaccine expiration dates, and vaccine coverage at the local, state and national levels.
Key points:
Both refrigerator and freezer compartments of household-grade combination storage units do not perform as well as household-grade stand-alone or purpose-built storage units, in terms of maintaining stable temperatures in the normal range.
Performance estimates support current vaccine storage and handling guidelines to minimize the use of household-grade combination storage units, particularly the freezer compartments.
Appendix A.
Table A1.
Welch two-sample statistical tests on the difference in percentage of runtime without excursions between types of vaccine storage units, stratified by sample exclusion criteria
| Unit types | Comparison | Sample description, the percentile of extreme temperature that was excluded | |||||
|---|---|---|---|---|---|---|---|
| Exclude 2.5% most extreme temperatures (main analysis) | No exclusions | Exclude 5% most extreme temperatures | |||||
| p-value | p-value | p-value | |||||
| Refrigerators | Household-combination vs. household stand-alone | 0.0378 | ** | 0.2089 | 0.0232 | ** | |
| Household-combination vs. purpose-built | 0.0000 | *** | 0.0000 | *** | 0.0000 | *** | |
| Household stand-alone vs. purpose-built | 0.0050 | *** | 0.0106 | ** | 0.0048 | *** | |
| Combination vs. non-combination | 0.0008 | *** | 0.0073 | *** | 0.0005 | *** | |
| Freezers | Household-combination vs. household stand-alone | 0.0000 | *** | 0.0000 | *** | 0.0000 | *** |
| Household-combination vs. purpose-built | 0.0000 | *** | 0.0000 | *** | 0.0000 | *** | |
| Household stand-alone vs. purpose-built | 0.1927 | 0.1502 | 0.5469 | ||||
| Combination vs. non-combination | 0.0000 | *** | 0.0000 | *** | 0.0000 | *** | |
Note(s):*, **, and *** refer to statistical significance at the 10%, 5%, and 1% levels.
Publisher's Disclaimer: Acknowledgements/Disclaimer:
The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention.
This change was made to bring the normal range presented in the Toolkit into better alignment with the normal range presented in the majority of vaccine package inserts.
References
- [1].Boros CA, Hanlon M, Gold M, Roberton D. Storage at− 3 C for 24 h alters the immunogenicity of pertussis vaccines. Vaccine. 2001;19:3537–42. [DOI] [PubMed] [Google Scholar]
- [2].Gazmararian JA, Oster NV, Green DC, Schuessler L, Howell K, Davis J, et al. Vaccine storage practices in primary care physician offices: assessment and intervention. American Journal of Preventive Medicine. 2002;23:246–53. [DOI] [PubMed] [Google Scholar]
- [3].Milstien JB, Galazka AM, Kartoglu Um, Zaffran M. Temperature sensitivity of vaccines. World Health Organization, Department of Immunization, Vaccines and Biologicals; 2006. p. 1–62. [Google Scholar]
- [4].Levinson DR. Vaccines for children program: vulnerabilities in vaccine management. Department of Health and Human Services: Office of Inspector General; 2012. [Google Scholar]
- [5].Lloyd J, Lydon P, Ouhichi R, Zaffran M. Reducing the loss of vaccines from accidental freezing in the cold chain: The experience of continuous temperature monitoring in Tunisia. Vaccine. 2015;33:902–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [6].Karp CL, Lans D, Esparza J, Edson EB, Owen KE, Wilson CB, et al. Evaluating the value proposition for improving vaccine thermostability to increase vaccine impact in low and middle-income countries. Vaccine. 2015;33:3471–9. [DOI] [PubMed] [Google Scholar]
- [7].Setia S, Mainzer H, Washington ML, Coil G, Snyder R, Weniger BG. Frequency and causes of vaccine wastage. Vaccine. 2002;20:1148–56. [DOI] [PubMed] [Google Scholar]
- [8].Mukherjee A, Ahluwalia TP, Gaur LN, Mittal R, Kambo I, Saxena NC, et al. Assessment of vaccine wastage during a pulse polio immunization programme in India. Journal of Health, Population and Nutrition. 2004:13–8. [PubMed] [Google Scholar]
- [9].Glista K Hartford HealthCare Warns of Possibly Ineffective Vaccines. Hartford Courant. Hartford, Connecticut; 2014. [Google Scholar]
- [10].Zhou F, Santoli J, Messonnier ML, Yusuf HR, Shefer A, Chu SY, et al. Economic evaluation of the 7-vaccine routine childhood immunization schedule in the United States, 2001. Archives of Pediatrics & Adolescent Medicine. 2005;159:1136–44. [DOI] [PubMed] [Google Scholar]
- [11].Whitney CG, Zhou F, Singleton JA, Schuchat A. Benefits from immunization during the Vaccines for Children program era - United States, 1994–2013. Morbidity and Mortality Weekly Report. 2014;63:352–5. [PMC free article] [PubMed] [Google Scholar]
- [12].Leidner AJ, Murthy N, Chesson HW, Biggerstaff M, Stoecker C, Harris AM, et al. Cost-effectiveness of adult vaccinations: A systematic review. Vaccine. 2019;37:226–34. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [13].Dabestani NM, Leidner AJ, Seiber EE, Kim H, Graitcer SB, Foppa IM, et al. A review of the cost-effectiveness of adult influenza vaccination and other preventive services. Preventive Medicine. 2019;126:1–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [14].East A, Smale N. Combining a hybrid genetic algorithm and a heat transfer model to optimise an insulated box for use in the transport of perishables. Vaccine. 2008;26:1322–34. [DOI] [PubMed] [Google Scholar]
- [15].Chojnacky M, Miller W, Ripple D, Strouse G. Thermal analysis of refrigeration systems used for vaccine storage. NIST Interagency / Internal Report (NISTIR). United States Department of Commerce: National Institute of Standards and Technology; 2009. [Google Scholar]
- [16].Muegge S Protecting Refrigerated Vaccines with Water Bottles: An Evidence-Based Strategy. AJN The American Journal of Nursing. 2012;112:61–9. [DOI] [PubMed] [Google Scholar]
- [17].Schlehuber LD, McFadyen IJ, Shu Y, Carignan J, Duprex WP, Forsyth WR, et al. Towards ambient temperature-stable vaccines: The identification of thermally stabilizing liquid formulations for measles virus using an innovative high-throughput infectivity assay. Vaccine. 2011;29:5031–9. [DOI] [PubMed] [Google Scholar]
- [18].Galazka A, Milstien J, Zaffran M. Thermostability of vaccines. Geneva: World Health Organization; 1998. [Google Scholar]
- [19].McAleer WJ, Markus HZ, McLean AA, Buynak EB, Hilleman MR. Stability on storage at various temperatures of live measles, mumps and rubella virus vaccines in new stabilizer. Journal of Biological Standardization. 1980;8:281–7. [DOI] [PubMed] [Google Scholar]
- [20].McColloster P, Vallbona C. Graphic-output temperature data loggers for monitoring vaccine refrigeration: implications for pertussis. American journal of public health. 2011;101:46. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [21].Kartoğlu Ü, Nelaj E, Maire D. Improving temperature monitoring in the vaccine cold chain at the periphery: An intervention study using a 30-day electronic refrigerator temperature logger (Fridge-tag®). Vaccine. 2010;28:4065–72. [DOI] [PubMed] [Google Scholar]
- [22].Chojnacky M, Miller W, Strouse G. Data logger thermometers for vaccine temperature monitoring. NIST Interagency / Internal Report (NISTIR). United States Department of Commerce: National Institute of Standards and Technology; 2012. [Google Scholar]
- [23].Fisun H, Leidner AJ, Wildt JK, Stevenson J. Assessing digital data logger temperature monitoring ability and storage unit performance in Michigan. National Immunization Conference. Atlanta, Georgia; 2016. [Google Scholar]
- [24].Leidner AJ, Lee C, Tippins A, Stevenson JM, Messonnier ML. Evaluation of non-continuous temperature monitoring practices for vaccine storage units: a Monte Carlo simulation study. Journal of Public Health. 2019. Forthcoming. DOI: 10.1007/s10389-020-01207-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- [25].Centers for Disease Control and Prevention. Vaccine Storage & Handling Toolkit. US Department of Health and Human Services; 2018. [Google Scholar]
- [26].Chojnacky MJ, Miller WW, Strouse G. Thermal analysis of refrigeration systems used for vaccine storage: Report on pharmaceutical grade refrigerator and household refrigerator/freezer. NIST Interagency / Internal Report (NISTIR). United States Department of Commerce: National Institute of Standards and Technology; 2010. [Google Scholar]
- [27].Stevenson J, Hicks M, Fisun H, Leidner AJ. Summary of progress by recipients of a cooperative agreement to improve vaccine management, storage and handling at the provider and awardee levels. National Immunization Conference. Atlanta, Georgia; 2016. [Google Scholar]
- [28].Kroger AT, Sumaya CV, Pickering LK, Atkinson WL. General recommendations on immunization: recommendations of the Advisory Committee on Immunization Practices (ACIP). Morbidity and Mortality Weekly Report: Recommendations and Reports. 2011:1–60. [PubMed] [Google Scholar]