Skip to main content
NCBI home page
HHS Author Manuscripts logoLink to HHS Author Manuscripts
. Author manuscript; available in PMC: 2022 Aug 2.
Published in final edited form as: J Public Health (Bangkok). 2020 Mar 22;29(6):1253–1260. doi: 10.1007/s10389-020-01207-0

Evaluation of non-continuous temperature monitoring practices for vaccine storage units: A Monte Carlo simulation study

Andrew J Leidner a,*, Carla E Lee b, Ashley Tippins a, Mark L Messonnier c, John M Stevenson a
PMCID: PMC9345596  NIHMSID: NIHMS1789478  PMID: 35923301

Abstract

Objectives:

Evaluate different non-continuous temperature monitoring practices for detection of out-of-range temperatures (above or below the recommended temperature range of 2–8°C for refrigeration units), which are called excursions, within vaccine storage units.

Methods:

Simulations based on temperature data collected by 243 digital data loggers operated in vaccine storage units at health care providers who participated in a CDC-sponsored continuous temperature monitoring pilot project, from 2012 to 2015. In the primary analysis, we evaluate: (1) twice-daily current temperature readings without minimum and maximum readings (min/max), (2) twice-daily current temperature readings with once-daily min/max, and (3) twice-daily current temperature readings with twice-daily min/max.

Results:

Recording current temperature twice-daily without min/max resulted in the detection of 4.8—6.4% of the total number of temperature excursions. When min/max readings were introduced, the percentage of detected temperature excursions increased to 27.8—96.6% with once-daily min/max and to 34.8—96.7% with twice-daily min/max.

Conclusions:

Including min/max readings improves the ability of a temperature monitoring practice to detect temperature excursions. No combination of the non-continuous temperature monitoring practices were able to consistently detect all simulated temperature excursions.

Keywords: vaccines, storage units, cold-chain, simulation

Introduction

Immunization of populations against harmful diseases by the use of vaccines has been one of the most beneficial endeavors of modern medicine and public health. The economic returns and health gains associated with vaccination are substantial (Whitney et al. 2014). In the early 2000s, the worldwide vaccine market was estimated to account for approximately $6 billion (Gréco 2002; Whitehead and Pasternak 2002). To maintain the effectiveness and value of these vaccines, public health researchers have focused on ways to reduce risk to vaccines due to temperature fluctuations in the vaccine distribution system (Chojnacky et al. 2009; Kristensen et al. 2011) and in storage at vaccine provider locations (Bell et al. 2001; McColloster and Vallbona 2011). The most important risk to vaccines from being subjected to inappropriate temperatures is the loss of effectiveness, which can be due to exposure to either heat or freezing temperatures, depending on the type of vaccine (Kumru et al. 2014; Matthias et al. 2007).

Events where vaccines are exposed to temperatures outside the recommended range are often referred to as temperature excursions. Temperature excursions are identified as a potential cause of vaccine wastage in a variety of settings (Mukherjee et al. 2004; Setia et al. 2002). In response to the risk posed to vaccine supplies from temperature excursions, a number of interventions have emerged to promote and maintain temperatures in vaccine distribution, which may also be referred to as maintaining the cold-chain. Research efforts have focused on manufacturing more thermally stable vaccines (Schlehuber et al. 2011), improving transportation and distribution of vaccines (East and Smale 2008), and applying different methods to pack and arrange vaccines within storage units (Chojnacky et al. 2009). There have also been investigations into the effects of ambient environmental factors such as ambient air temperature or power outages (Chojnacky et al. 2009) and improving temperature monitoring in vaccine storage units (Fisun et al. 2016; Kartoğlu et al. 2010; McColloster and Vallbona 2011).

Continuous temperature monitoring (CTM) consists of using technology to record temperatures in storage units at regular intervals. These temperatures are recorded and saved electronically in a digital data logger (DDL). The CDC currently recommends the use of a continuous temperature monitoring device (Centers for Disease Control and Prevention 2018), and began requiring these devices for storage units containing federally funded Vaccines For Children vaccines in January 2018. The World Health Organization (WHO) also recommends the use of CTMs (World Health Organization 2015). However, in specific situations CTM devices may not be feasible. As an example in Haiti, several challenges were encountered after DDLs were implemented, including spans of time when the DDL did not collect data or communicate data to the central data collection hub (Cavallaro et al. 2017). In addition to the use of CTM, both CDC and WHO also recommend daily manual temperature checks, which we refer to as non-continuous temperature monitoring practices. Non-continuous temperature monitoring helps to actively monitor temperatures and this serves as a backup or supplement to any data collected with DDLs.

The focus of this study is to assess the potential benefits of different types of non-continuous temperature monitoring practices. Findings from this study would be relevant to vaccine provider locations where non-continuous temperature monitoring practices are being used as a complement to CTM, but would be especially important for vaccine provider locations where CTM is not yet feasible. Using a simulation methodology, we evaluated the ability of different non-continuous temperature monitoring practices to detect the occurrence of temperature excursions. To our knowledge, no thorough investigation has been conducted that quantifies the relative strengths of these temperature monitoring practices for their ability to detect and record temperature excursions. Our study aims to fill this knowledge gap by providing an assessment of several temperature monitoring practices with a simulation model.

Background on vaccine storage and handling practices

In spite of substantial efforts to improve the vaccine supply chain across the world, ubiquitous and effective vaccine storage and handling practices remain elusive in many parts of the world (Ashok et al. 2017; Hanson et al. 2017). For example, improper storage and handling practices were identified, among other violations of proper vaccination procedures, at a workplace influenza vaccination clinic in New Jersey (Taylor et al. 2015). Freezing temperatures in vaccine storage units have been linked to pertussis incidence in Texas (McColloster and Vallbona 2011). Specific deliveries of live attenuated influenza vaccine for the 2013–2014 influenza season appear to have been subjected to excessive temperatures, and these excessive temperatures were found to be associated with low vaccine effectiveness for particular influenza strains (Caspard et al. 2016). More globally, low vaccine effectiveness, which may have been related to cold chain inadequacies or waning immunity, was acknowledged as a contributing factor in a 2014 measles outbreak in the Federated States of Micronesia (Hales et al. 2016). A literature review looking at vaccine cold chain studies found that in lower income countries, as many as 19% of vaccine shipments and 37% of vaccine storage units experienced temperatures below recommended ranges (Hanson et al. 2017). Another review of cold chain effectiveness in developing countries identified the “lack of the latest technology or optimal equipment” as a key issue that has been limiting cold chain performance (Ashok et al. 2017). In more specific geographic regions, vaccine cold chain management has been studied by public health researchers in the context of Cameroon (Yakum et al. 2015), Pakistan (Buledi et al. 2017), Tunisia (Lloyd et al. 2015) and Haiti (Cavallaro et al. 2017).

Non-continuous practices consist of vaccine provider staff checking storage unit temperatures from a temperature monitoring device at one or more times during the workday to ensure the storage unit temperature is within the recommended range. By contrast, CTM utilizes DDLs, which are small electronic devices, typically weighing less than a pound and about the size of a scientific calculator, connected to a temperature probe that is usually suspended in buffered material such as a glycol solution. Temperature data is collected by DDL at small time intervals, such as 5, 10, or 15 minute intervals. The CDC Vaccine Storage and Handling Toolkit recommends these intervals be no longer than 30 minutes (Centers for Disease Control and Prevention 2018). DDL data is either transmitted or uploaded to a computer where past temperatures of a storage unit may be inspected, analyzed, or stored for later review. Many DDLs also feature an alarm system that can indicate in real time if a temperature excursion is occurring. As an illustrative example, data from a DDL is presented in Figure 1. This data exhibits several temperature excursions, where recorded temperatures fall below the lower bound of the normal range, along with a corrective action, after which the recorded temperatures remain within the normal range.

Figure 1. Illustrative example of temperature excursions with evidence of a corrective actiona in household-grade combination refrigeration vaccine storage unit.

Figure 1.

Open in a new tab

a.In this illustrative example, the corrective action takes place around 22 hours, several hours after the first temperature excursion that would be detected by continuous temperature monitoring. In an ideal scenario, if staff are present and available to address the excursion, the corrective action would occur as soon as the first temperature excursion is detected.

Vaccine providers can have storage units that are either refrigerators or freezers, depending on the types of vaccines distributed by the provider. Storage units may be household-grade combination units (i.e., refrigerator and freezer combined into a single appliance), household-grade stand-alone units, purpose-built stand-alone units, and purpose-built combination units. These storage units may vary in their ability to sustain a constant temperature. Household-grade combination units can be problematic for vaccine storage due to relatively dramatic fluctuations in temperature, frequently caused by de-frost cycling within the freezer section. For this reason, CDC does not recommend using the freezer section (Centers for Disease Control and Prevention 2018).

Methods

We developed a Monte Carlo simulation model that represents a household-grade refrigeration unit. As a matter of scope, this study focuses on refrigerated storage units only, not freezer units. For vaccine storage in refrigerators, we define the normal temperature range as between 2°C and 8°C, which is the range stated in all manufacturer package inserts for all routinely recommended vaccines (Centers for Disease Control and Prevention 2018). Monte Carlo is a type of simulation methodology that uses repeated, random sampling from a distribution. Assumptions in the simulations can be adjusted so that the simulations represent a storage unit that is relatively reliable and has a fewer number of temperature excursions. Alternatively, assumptions in the simulations can be adjusted so that the simulations represent a storage unit that is functioning less reliably. To investigate the simulation model across a range of reliability assumptions, we put simulations into four groups (numbered I, II, III, and IV in the results) that represent different levels of storage unit reliability. Within each group, we ran the simulation 500 times, or for 500 iterations. Each iteration constructs a hypothetical new, randomly generated storage unit with a unique distribution of temperature excursions. For each iteration, we recorded the total number of temperature excursions, the total temperature excursion runtime, the temperature excursions as a portion of total runtime, and the number of temperature excursions per day. Results from the simulation models are summarized using averages and standard errors of the 500 iterations for each of the four simulation groups.

Pilot project data

The simulations in this study are based on data collected during a larger project to pilot (DDL Pilot Project) and distribute DDLs to US vaccine providers, which ran from 2012 to 2016 (Stevenson et al. 2016). Our study utilizes the temperature and excursion data collected by DDLs from a convenience sample of 243 household-grade, combination refrigeration units that had received DDLs in the DDL Pilot Project (Stevenson et al. 2016). Across the 243 units, temperature data was collected for an average of 4 305 hours, or 179 days. Summary statistics are reported for these data in Table 1. Other aspects of the DDL Pilot Project have been described and summarized elsewhere (Stevenson et al. 2016). The DDL Pilot Project was a Prevention and Public Health Fund awardee grant, funded by CDC.

Table 1.

Summary data for recorded temperature excursions in household-grade storage units used in the simulation model.

Time in different excursion status from 243 storage units
Time (hours) per storage unit Proportion of observed time in status
Mean SE Mean SE Median 70th Percentile 90th Percentile
No excursion 4 305.1 181.8 0.989 0.002 0.999 1.000 1.000
Any excursion 35.8 5.3 0.011 0.002 0.001 0.005 0.022
 High excursion 25.4 4.8 0.006 0.001 0.000 0.002 0.015
 Low excursions 10.4 1.7 0.005 0.002 0.000 0.001 0.008
Time and temperatures of excursion events
Number of excursion events Time (hours) per excursion Recorded temperatures (°C)
Mean SE Mean SE Min Max
Any excursion 4 972 1.749 0.090
 High excursion 2 140 2.888 0.202 8.448 0.021 15.500
 Low excursions 2 832 0.889 0.037 1.151 0.011 −1.100
Open in a new tab

Note(s): SE=standard error of the mean. This excursion data was collected from storage units that were using digital data loggers to monitor temperature stability. From this sample of 243 storage units, there were 54 units with no detected excursions, 72 units with high excursions only, 28 units with low excursions only, 89 units with both high and low excursions detected.

Brief description of the simulation model

The four simulation groups are characterized by the amount of possible excursion time that can occur in a simulation iteration. Simulation group I, II, III, and IV allows excursions to occur between 0 to 0.001, 0 to 0.005, 0 to 0.022, and 0 to 0.900 of total runtime, respectively. The ranges for simulation groups I, II, and III are based on the DDL Pilot Project data, where 0.001, 0.005, and 0.022 were the median, 70th, and 90th percentiles of the total excursion time in the DDL Pilot Project data, among our sample of household-grade combination units. Simulation group IV allows for excursion time to occur up to 90% of the total runtime, which provides a worst-case scenario that is the least reliable in terms of maintaining a stable temperature within the normal range.

In each iteration, the model randomly generates simulated data that represents 90 days of runtime in a household-grade refrigeration storage unit. In particular, the simulated data represents temperature values recorded at 5-minute intervals for 90 days. We simulated data at 5-minute intervals to capture temperature excursions at a high resolution and we limited the time horizon to 90 days to keep the size of simulated data files manageable. These simulated temperatures are either within the normal range (between 2°C and 8°C) or outside the normal range. The simulation uses two key inputs based on data from the DDL Pilot Project: (1) the portion of time outside the normal range for a given storage unit, and (2) the extreme temperatures and the length of time associated with excursions. Additional technical details on the simulations are presented in the Appendix.

Non-continuous temperature monitoring practices

In this model, temperature readings can occur once per day at 9am or twice per day at 9am and 4pm. At each of these times, the current temperature and the min/max temperatures can be recorded. The current temperature readings record the current temperature in the storage unit. The min/max temperature reading records both the minimum and maximum temperature that was detected since the last time the min/max temperature was checked1.

The times of temperature readings occurring in the model are based on temperature monitoring activities in provider offices, during the business days of a 5-day workweek. We assumed no readings occur on the weekends so that for every 7 days in the simulation, 2 of the days have no readings no matter which monitoring practice is being evaluated. As an example, consider twice daily min/max readings. The last min/max reading of any given week in the simulation occurs at 4pm on Friday. The next min/max reading occurs the following Monday at 9am. The Monday 9am reading documents the minimum and maximum temperatures experienced by the storage unit since Friday at 4pm. For simplicity, we assumed no holidays occur during the 90-day simulation. We also assumed that the temperature readings are never skipped and temperatures are recorded with perfect accuracy. In practice, holidays, sick days, and occasional human error would likely result in skipped or inaccurate readings.

Analysis

In the primary analysis, three temperature monitoring practices were evaluated: (1) twice daily current temperature readings without min/max readings, (2) twice daily current temperature readings with daily min/max readings, and (3) twice daily current temperature readings with twice daily min/max readings. The outcomes of interest were the number of excursions and portion of total excursions that were detected by a given temperature monitoring practice. Outcomes for all three practices were reported as averages across the 500 simulation iterations for each of the four simulation groups. The list of temperature monitoring practices evaluated in the primary analysis centered on three common practices and was not exhaustive. For this reason, we implemented sensitivity analyses, where a more complete list of potential temperature monitoring practices could be evaluated. The monitoring practices evaluated in the sensitivity analysis include all possible combinations of once daily, twice daily, current temperature readings, and min/max temperature readings. For these sensitivity analyses, we also assumed our widest range (0 to 0.9) for the possible portion of cumulative excursion time.

Results

Table 1 presents summary statistics for storage units and excursion events from the DDL pilot data. The portion of time that the household-grade combination refrigerator units from the DDL pilot operated in the normal range was 0.989. Of the 4 972 excursions, the portion that were lower excursions (i.e. temperature below the bottom of the acceptable range, putting the vaccine at risk of freezing) was 0.57 (2 832) and the portion that were high excursions was 0.43 (2 140). The average time of a temperature excursion from the DDL pilot was 1.749 hours, with high excursions lasting longer (2.888 hours per excursion) on average than low excursions (0.889 hours per excursion). Figure 2 presents the distribution of each excursion with respect to the length of time each excursion lasted and the most extreme recorded temperature in each excursion. The majority of excursions appear to be clustered in areas relatively near the normal temperature range and in areas that indicate relatively short-duration excursions.

Figure 2. Duration and temperatures associated with excursions from household-grade combination refrigeration units with a grey box indicating the normal range for temperatures.

Figure 2.

Open in a new tab

Note(s): Each point in the figure represents the duration (x-axis) and the most extreme temperature (y-axis) of a temperature excursion from the DDL Pilot Project data. As an example, points just above the area labeled the normal temperature range constitute relatively long excursions that had a maximum temperature that was only slightly warmer than the recommended temperature range. By comparison, points in the bottom left region constitute relatively brief temperature excursions with a low minimum temperature recorded.

The results of the primary analysis from the simulation model can be found in Table 2. The average portion of temperature excursions detected by twice daily current temperature readings with no min/max was between 0.047 and 0.068 of all temperature excursions. Including a once daily min/max temperature reading increased the portion of detected temperature excursions to a range of 0.275 to 0.963, depending on simulation group. Twice daily current temperature readings and twice daily min/max temperature readings yielded a range of the portion of detected temperature excursions from 0.347 to 0.965. The number of excursions detected by any of the three monitoring practices increased in the simulation groups with greater numbers of temperature excursions.

Table 2.

Results from four groups of 90-day simulations of storage unit temperature excursions, evaluating three temperature monitoring practices.

Simulation Results
Simulation group identifiera I II III IV
Range of cumulative excursion time as portions of total time 0 to 0.001 0 to 0.005 0 to 0.022 0 to 0.9
Mean number of excursions 3.3 9.5 31.9 713.2
Mean elapsed excursion time (hours) 4.2 7.4 25.0 945.9
Mean portion of runtime in excursion 0.002 0.003 0.012 0.438
Mean excursions per day 0.036 0.106 0.355 7.925
Evaluations of temperature monitoring practices
Current temperatures Min/Max temperatures Simulation group identifiera I II III IV
Twice daily None Mean number of excursions detected 0.2 0.4 1.4 42.4
Mean portion of excursions detected 0.068 0.059 0.047 0.061
Twice daily Once daily Mean number of excursions detected 3.2 8.9 25.7 134.1
Mean portion of excursions detected 0.963 0.940 0.846 0.275
Twice daily Twice daily Mean number of excursions detected 3.2 9.1 27.2 176.4
Mean portion of excursions detected 0.965 0.952 0.882 0.347
Number of simulations 500 500 500 500
Open in a new tab

Note(s): Measures of statistical significance, such as standard errors or confidence intervals or p-values are not computed or presented here because, in this case, those measures are primarily a function of the number of simulations.

a.

The ranges of cumulative excursion time for simulation groups I, II, and III correspond the median, 70th percentile, and 90th percentile from the sample of DDL Pilot Project data. Simulation group IV used a substantially wider range than the other groups to provide a context for a storage units with a relatively large number of temperature excursions.

Results from the sensitivity analyses are summarized in Figure 3, which plots the portion of temperature excursions detected as a function of the portion of cumulative excursion time in a simulation iteration, stratified by different non-continuous temperature monitoring practices in each panel. The negative relationship between cumulative excursion time of a simulation and the portion of temperature excursions detected by monitoring practices with at least one daily min/max reading is apparent in the sensitivity analysis (Figure 3c–h). This analysis also supports the main finding from Table 1, which is that monitoring practices that have no min/max reading (Figure 3a and 3b) do not detect as many excursions as monitoring practices with at least one min/max reading (Figure 3c–h).

Figure 3. Portion of detected excursions verses total excursion time in a sensitivity analyses, with each panel representing different temperature monitoring practices and 500 simulation iterations.

Figure 3.

Open in a new tab

Note(s): Each point in the figure represents the outcome of a single iteration of the simulation model. For example, a point in the bottom left of panel a, represents a simulation iteration that experienced relatively few temperature excursions, and a relatively small portion of these excursions that was detected by once daily current temperature readings. For a comparison, consider a point in the top left of panel (d). A point in this region represents a simulation iteration that experienced a relatively small number of temperature excursions but the majority of which were detected by the twice daily min/max temperature readings.

Discussion

Proper storage and handling along the vaccine supply chain is a necessary component of a well-functioning immunization system. Temperature excursions can compromise vaccine effectiveness and contribute to vaccine wastage. Failure to maintain the cold chain may have played a role in past outbreaks of vaccine-preventable diseases (Paunio et al. 1998; Yeung et al. 2005). The objective of this study was to use a simulation model to assess the potential benefits of different types of non-continuous temperature monitoring practices. In terms of detecting temperature excursions, we found min/max temperature readings to add substantial value, relative to relying on current temperature readings alone, particularly when a storage unit experiences relatively few temperature excursions. However, none of the non-continuous temperature monitoring practices that were assessed in this study were able to reliably detect 100% of temperature excursions, which underscores the additional value of using DDLs for CTM.

Even though CTM devices are broadly recommended, non-continuous temperature monitoring practices may continue to be an important part of vaccine storage and handling procedures for at least two reasons. First, for some geographic regions across the world, particularly those where information technology infrastructure remains underdeveloped, CTM may not be technically feasible. Secondly, even when CTM is in practice, non-continuous temperature monitoring provides an additional, documented record of temperature stability in the event that certain functional issues with the DDL or data management system occur. Our study demonstrates, in both of these cases, the use of min/max temperature readings may likely provide greater awareness of any temperature excursions when compared to non-continuous practices that only utilize current temperature readings.

Findings from this study support and underscore the benefits of CTM. While non-continuous temperature monitoring practices that utilized min/max readings outperformed those that only utilized current temperature readings, in the majority of simulation iterations there was no non-continuous temperature monitoring practice that detected all of the temperature excursions that occurred. For this reason, CTM and the use of DDLs offer the current, best available means of capturing the most complete assessment of temperature stability in vaccine storage units. Temperature excursions can have a varied effect on the effectiveness of stored vaccines, which may depend on several factors including: the temperatures experienced during an excursion, the duration of any out of range temperatures, and the types of vaccines being stored. Currently, the CDC recommends the use of a continuous temperature monitoring device (Centers for Disease Control and Prevention 2018), and began requiring these devices for storage units containing federally funded Vaccines For Children vaccines in January 2018. At least one state vaccine program adopted the requirement of CTMs before January 2018 (Oregon Health Authority 2015).

Our model and results should be interpreted in consideration of some important limitations and context. First, human behavior is not modeled in these simulations. In practice, recording of temperatures by staff will likely be subject to skipped observations or imprecisely documented temperature readings. Second, storage units modeled were household-grade, combination-style units. So some caution may be warranted when considering the generalization of this study’s results to other types of storage units, such as stand-alone refrigerator units, purpose-built or pharmaceutical-grade storage units, or even freezers. Third, this study models temperature excursions in vaccine units as intermittent, non-permanent events. One likely consequence of a permanent temperature excursion would be the replacement of a vaccine storage unit. We did not model the replacement of storage units. Finally, the data used as inputs into the simulation model in this study did not come from a sample that was known to be representative of storage units or vaccine providers in the United States or any other jurisdiction. As such, generalizations of these results to a nation-wide sample may not be appropriate.

In light of these limitations, we can conclude that the simulations presented in this study provide a useful, evidence-based comparison of several non-continuous temperature monitoring practices using data generated in a simulation. The results of this study highlight the benefits of min/max as a non-continuous temperature monitoring practice. These results also provide support for the transition to CTM as a way to ensure vaccination effectiveness, which can improve immunity, lead to healthier communities, and confer greater protection from vaccine preventable diseases.

Supplementary Material

1

Acknowledgements/Disclaimer:

The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention.

Footnotes

Conflict of interest: The authors declare they have no conflict of interest.

Ethical statement: This article does not contain any studies with human participants or animals performed by any of the authors.

1

In practice, a temperature monitoring device has a ‘reset’ button that can reset min/max temperature reading. Each time the min/max temperatures are recorded, the ‘reset’ button is pushed, and the previous reading of the min/max temperature is deleted so that new values are recorded for the next min/max temperature reading.

References

  1. Ashok A, Brison M, LeTallec Y (2017) Improving cold chain systems: Challenges and solutions. Vaccine 35:2217–2223. DOI 10.1016/j.vaccine.2016.08.045 [DOI] [PubMed] [Google Scholar]
  2. Bell KN, Hogue CJ, Manning C, Kendal AP (2001) Risk factors for improper vaccine storage and handling in private provider offices. Pediatrics 107:e100. DOI 10.1542/peds.107.6.e100 [DOI] [PubMed] [Google Scholar]
  3. Buledi R, Butt ZA, Ahmed J, Alizai AA (2017) Status of cold chain in routine immunisation centres of the Expanded Programme on Immunisation in Quetta, Pakistan. JPMA J Pak Med Assoc 67:739–744 [PubMed] [Google Scholar]
  4. Caspard H, Coelingh KL, Mallory RM, Ambrose CS (2016) Association of vaccine handling conditions with effectiveness of live attenuated influenza vaccine against H1N1pdm09 viruses in the United States. Vaccine 34:5066–5072. DOI 10.1016/j.vaccine.2016.08.079 [DOI] [PubMed] [Google Scholar]
  5. Cavallaro KF et al. (2017) Demonstration of the Use of Remote Temperature Monitoring Devices in Vaccine Refrigerators in Haiti. Public Health Rep 133:39–44. DOI 10.1177/0033354917742119 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Centers for Disease Control and Prevention (2018) Vaccine Storage & Handling Toolkit. United States Department of Health and Human Services [Google Scholar]
  7. Chojnacky M, Miller W, Ripple D, Strouse G (2009) Thermal analysis of refrigeration systems used for vaccine storage. National Institute of Standards and Technology, United States Department of Commerce [Google Scholar]
  8. East A, Smale N (2008) Combining a hybrid genetic algorithm and a heat transfer model to optimise an insulated box for use in the transport of perishables. Vaccine 26:1322–1334. DOI 10.1016/j.vaccine.2007.12.055 [DOI] [PubMed] [Google Scholar]
  9. Fisun H, Leidner AJ, Wildt JK, Stevenson J (2016) Evaluating DDL ability and storage unit performance in Michigan. 47th National Immunization Conference, Atlanta, Georgia; September 13–15 [Google Scholar]
  10. Gréco M (2002) The future of vaccines: an industrial perspective. Vaccine 20:S101–S103. DOI 10.1016/s0264-410x(01)00293-6 [DOI] [PubMed] [Google Scholar]
  11. Hales CM et al. (2016) Measles outbreak associated with low vaccine effectiveness among adults in Pohnpei state, federated states of Micronesia, 2014. Open forum Infect Dis 2: ofw064. DOI 10.1093/ofid/ofw064 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Hanson CM, George AM, Sawadogo A, Schreiber B (2017) Is freezing in the vaccine cold chain an ongoing issue? A literature review. Vaccine 35:2127–2133. DOI 10.1016/j.vaccine.2016.09.070 [DOI] [PubMed] [Google Scholar]
  13. Kartoğlu Ü, Nelaj E, Maire D (2010) Improving temperature monitoring in the vaccine cold chain at the periphery: An intervention study using a 30-day electronic refrigerator temperature logger (Fridge-tag®). Vaccine 28:4065–4072. DOI 10.1016/j.vaccine.2010.03.076 [DOI] [PubMed] [Google Scholar]
  14. Kristensen D, Chen D, Cummings R (2011) Vaccine stabilization: research, commercialization, and potential impact. Vaccine 29:7122–7124. DOI 10.1016/j.vaccine.2011.05.070 [DOI] [PubMed] [Google Scholar]
  15. Kumru OS, Joshi SB, Smith DE, Middaugh CR, Prusik T, Volkin DB (2014) Vaccine instability in the cold chain: mechanisms, analysis and formulation strategies. Biologicals 42:237–259. DOI 10.1016/j.biologicals.2014.05.007 [DOI] [PubMed] [Google Scholar]
  16. Lloyd J, Lydon P, Ouhichi R, Zaffran M (2015) Reducing the loss of vaccines from accidental freezing in the cold chain: The experience of continuous temperature monitoring in Tunisia. Vaccine 33:902–907. DOI 10.1016/j.vaccine.2014.10.080 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Matthias DM, Robertson J, Garrison MM, Newland S, Nelson C (2007) Freezing temperatures in the vaccine cold chain: a systematic literature review. Vaccine 25:3980–3986. DOI 10.1016/j.vaccine.2007.02.052 [DOI] [PubMed] [Google Scholar]
  18. McColloster P, Vallbona C (2011) Graphic-output temperature data loggers for monitoring vaccine refrigeration: implications for pertussis. Am J Public Health 101:46. DOI 10.2105/AJPH.2009.179853 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Mukherjee A, Ahluwalia TP, Gaur LN, Mittal R, Kambo I, Saxena NC, Singh P (2004) Assessment of vaccine wastage during a pulse polio immunization programme in India. J Health Popul Nutr 22:13–18. DOI 10.3329/jhpn.v22i1.237 [DOI] [PubMed] [Google Scholar]
  20. Oregon Health Authority (2015) Oregon VFC Thermometer Guide. Available from: https://www.oregon.gov/pharmacy/Imports/Rules/November15/VFCthermguide.pdf. Accessed on 4/7 2016
  21. Paunio M, Peltola H, Valle M, Davidkin I, Virtanen M, Heinonen OP (1998) Explosive school-based measles outbreak intense exposure may have resulted in high risk, even among revaccinees. Am J Epidemol 148:1103–1110. DOI 10.1093/oxfordjournals.aje.a009588 [DOI] [PubMed] [Google Scholar]
  22. Schlehuber LD et al. (2011) Towards ambient temperature-stable vaccines: The identification of thermally stabilizing liquid formulations for measles virus using an innovative high-throughput infectivity assay. Vaccine 29:5031–5039. DOI 10.1016/j.vaccine.2011.04.079 [DOI] [PubMed] [Google Scholar]
  23. Setia S, Mainzer H, Washington ML, Coil G, Snyder R, Weniger BG (2002) Frequency and causes of vaccine wastage. Vaccine 20:1148–1156. DOI 10.1016/S0264-410X(01)00433-9 [DOI] [PubMed] [Google Scholar]
  24. Stevenson J, Hicks M, Fisun H, Leidner AJ (2014) Summary of progress by recipients of a cooperative agreement to improve vaccine management, storage and handling at the provider and awardee levels. 46th National Immunization Conference, Atlanta, Georgia; September 29–30 [Google Scholar]
  25. Taylor L et al. (2015) Notes from the field: injection safety and vaccine administration errors at an employee influenza vaccination clinic–New Jersey, 2015. MMWR Morb Mortal Wkly Rep 64:1363–1364 [DOI] [PubMed] [Google Scholar]
  26. Whitehead P, Pasternak A (2002) Lessons learned: new procurement strategies for vaccines. Final report to the GAVI Board. Chicago, IL: Mercer Management Consulting, Inc. [Google Scholar]
  27. Whitney CG, Zhou F, Singleton JA, Schuchat A (2014) Benefits from immunization during the Vaccines for Children program era - United States, 1994–2013. MMWR Morb Mortal Wkly Rep 63:352–355 [PMC free article] [PubMed] [Google Scholar]
  28. World Health Organization (2015) How to monitor temperatures in the vaccine supply chain. WHO Vaccine Management Handbook, Module VMH-E-01.1. World Health Organization, Geneva [Google Scholar]
  29. Yakum MN, Ateudjieu J, Walter EA, Watcho P (2015) Vaccine storage and cold chain monitoring in the North West region of Cameroon: A cross sectional study. BMC Res Notes 8:145. DOI 10.1186/s13104-015-1109-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Yeung LF et al. (2005) A limited measles outbreak in a highly vaccinated US boarding school. Pediatrics 116:1287–1291. DOI 10.1542/peds.2004-2718 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1
NIHMS1789478-supplement-1.pdf (317.6KB, pdf)

RESOURCES