Validity and reliability of a protocol to establish human critical environmental limits (PSU HEAT Project)

Abstract

The PSU HEAT protocol has been used to determine critical environmental limits, i.e., those combinations of ambient temperature and humidity above which heat stress becomes uncompensable and core temperature rises continuously. However, no studies have rigorously investigated the reliability and validity of this experimental protocol. Here, we assessed the 1) between-visit reliability and 2) validity of the paradigm. Twelve subjects (5 M/7W; 25 ± 4 yr) completed a progressive heat stress protocol during which they walked on a treadmill (2.2 mph, 3% gradient) in a controllable environmental chamber. After an equilibration period, either dry-bulb temperature (T_db) was increased every 5 min while ambient water vapor pressure (P_a) was held constant (T_crit experiments) or P_a was increased every 5 min while T_db was held constant (P_crit experiments) until an upward inflection in gastrointestinal temperature (T_gi) was observed. For reliability experiments, 11 subjects repeated the same protocol on a different day. For validity experiments, 10 subjects performed a T_crit experiment at their previously determined P_crit or vice versa. The between-visit reliability (intraclass correlation coefficient, ICC) for critical environmental limits was 0.98. Similarly, there was excellent agreement between original and validity trials for T_crit (ICC = 0.95) and P_crit (ICC = 0.96). Furthermore, the wet-bulb temperature at the T_gi inflection point was not different during reliability (P = 0.78) or validity (P = 0.32) trials compared with original trials. These findings support the reliability and validity of this experimental paradigm for the determination of critical environmental limits for maintenance of human heat balance.

NEW & NOTEWORTHY The PSU HEAT progressive heat stress protocol has been used to identify critical environmental limits for various populations, clothing ensembles, and metabolic intensities. However, no studies have rigorously investigated the reliability and validity of this experimental model. Here, we demonstrate excellent reliability and validity of the PSU HEAT protocol.

INTRODUCTION

Body core temperature (T_c) equilibrates proportionally to metabolic heat production across a wide range of environmental conditions (1, 2). Thermal environments above which heat balance cannot be maintained, resulting in a continuous rise in T_c, are termed critical environmental limits. Lind (1, 2) first defined this critical limit as the “upper limit of the prescriptive zone” using an approach in which subjects completed a series of ∼1 h exposures to a wide range of thermal environments during treadmill walking. Belding and Kamon (3) developed a less time-intensive model to determine critical ambient water vapor pressures for a variety of exercise intensities and air movements at an ambient temperature of 36°C. Subsequently, our laboratory refined the approach by separating experiments in which heat balance is limited by sweat evaporation (critical water vapor pressure, P_crit) from those in which maximal sweating capacity and skin blood flow limitations are the primary limiting factors (critical dry-bulb temperature, T_crit) (4–7). The experimental approach has a long history of use in identifying critical environmental limits in a variety of populations, metabolic intensities, and clothing ensembles (4, 8–10). The PSU HEAT (Human Environmental Age Thresholds) project aims to establish critical environments for a variety of vulnerable populations that may be used for evidence-based alert communication, triage for impending heat events, and global climate change initiatives to mitigate the deleterious impact of severe heat events on human health. As such, it is imperative to ensure the experimental method used to determine these thresholds is rigorous, repeatable, and valid.

Our laboratory has previously reported reliability (r = 0.97) of this experimental paradigm; however, only a subgroup of four young, healthy adults were tested at a metabolic intensity of 30% V̇o_2max, an intensity designed to approximate that across an 8-h workday in industrial settings (4). There has yet to be an investigation to specifically determine the reliability and validity of determining critical environmental limits in conditions that approximate the metabolic intensity of daily living or minimal activity. This is a critical component of the PSU HEAT project, given the National Institutes of Health initiative for advancement in scientific research that is repeatable, reproducible, and rigorous (11, 12).

The purpose of the present investigation was to examine the between-visit reliability of the experimental paradigm used to identify those critical environmental conditions in which heat stress becomes uncompensable. In addition, to assess the validity of the findings from these experiments, we conducted a series of trials in which the T_crit or P_crit trials were reversed [i.e., progressive heat stress was conducted with dry-bulb temperature (T_db) or ambient water vapor pressure (P_a) clamped at the previously observed T_crit or P_crit value for a given subject] to determine whether critical environmental loci would occur at the same combinations of ambient temperature and humidity.

METHODS

Subjects

All experimental procedures were approved in advance by the Institutional Review Board at the Pennsylvania State University. Oral and written consents were obtained voluntarily from all subjects before participation and in accordance with the guidelines set forth by the Declaration of Helsinki. All testing was conducted in Noll Laboratory at the Pennsylvania State University.

Twelve young healthy adults (5 men/7 women) aged 18–35 yr were recruited for the study. All of the original trials (15 experiments) overlapped with our companion paper (13). Nine subjects returned for both a reliability and validity trial, two subjects returned for a reliability trial only, and one subject returned for a validity trial only. All visits were separated by at least 5 days and were conducted during the same season of the year for each subject. The average duration between visits was 4 wk (range = 1–16 wk) for reliability tests and 3 wk (range = 1–7 wk) for validity tests. All participants were healthy, normotensive, nonsmokers, and were not taking any prescription medications that might affect the physiological variables of interest in this study. No attempt was made to control for menstrual status or contraceptive use (4). Maximal aerobic capacity (V̇o_2max) was determined using open-circuit spirometry during a graded exercise test performed on a motor-driven treadmill. During the experiments, participants wore thin, short-sleeved cotton tee-shirts, shorts, socks, and walking/running shoes.

Testing Procedures

Before each experimental session, participants were instructed to abstain from alcoholic beverages and vigorous exercise for 24 h and from caffeine for 12 h. Upon arrival, participants provided a urine sample to ensure euhydration, defined as urine specific gravity ≤ 1.020 (USG; PAL-S, Atago, Bellevue, WA) (14). Subjects walked on a motor-driven treadmill at a speed of 2.2 mph with a grade of 3% until a clear rise in gastrointestinal temperature (T_gi) was observed.

Two protocols were used to determine P_crit and T_crit as previously described (6, 15). The P_crit and T_crit for the upward inflection of T_c was determined in an environmental chamber at three distinct T_db conditions of 36°C, 38°C, and 40°C and three P_a conditions of 12, 16, and 20 mmHg, respectively. During P_crit experiments, T_db was held constant while, after a 30-min equilibration period, P_a was increased by 1 mmHg every 5 min. Conversely, during T_crit experiments, P_a was held constant and T_db was increased by 1°C every 5 min after a 30-min equilibration period. Progressive heat stress continued until a clear and continuous rise in T_gi was observed (Fig. 1).

Figure 1.

Representative tracings of the time course of gastrointestinal temperature (T_gi), dry-bulb temperature (T_db), and ambient water vapor pressure (P_a) for an original trial (A) with increasing T_db and a validity trial (B) with increasing P_a. During the original trial, P_a is held constant at 11.7 mmHg and the T_gi inflection point occurs at T_db = 43.1°C. During the validity trial, T_db is held constant at 43.1°C and the T_gi inflection point occurs at P_a = 12.1 mmHg.

To assess the reliability and validity of critical environmental limits identified using this protocol, each subject first completed either a T_crit or P_crit trial and then returned for 1) a reliability trial in which subjects repeated the same protocol as their first trial, and/or 2) a validity trial in which the critical P_a or T_db observed during the first trial was held constant while either T_db or P_a, respectively, was systematically increased to determine if T_gi inflected at the same combination of ambient temperature and humidity. A representative tracing of the time course of T_gi and the environmental conditions for an original T_crit and its P_crit validity trial is presented in Fig. 1.

Measurements

T_gi was measured using telemetry capsules that were ingested 1–2 h before each experiment, in accordance with previous data demonstrating that ingestion times from 1–12 h before use do not influence the precision of T_gi data (16) (VitalSense, Philips Respironics, Bend, OR). T_gi data were transmitted continuously to a PowerLab data acquisition system and LabChart signal processing software (AD Instruments, Colorado Springs, CO) using an Equivital wireless physiological monitoring system (Equivital Inc., New York, NY).

Oxygen consumption V̇o₂; L/min) and respiratory exchange ratio (RER; unitless) measurements were collected for 5 min at the 30-min and 60-min time points of each trial. The average V̇o₂ and RER of the final 3 min of each collection period were used to calculate metabolic heat production (M; W/m²) (17) as follows:

$$M={\dot{\mathrm{V}}\mathrm{o}}_2·\mathrm{ }\frac{\left[\left(\left(\frac{\mathrm{R}\mathrm{E}\mathrm{R}\mathrm{ }-\mathrm{ }0.7}{0.3}\right)·21.13\right)\mathrm{ }+\mathrm{ }\left(\left(\frac{1.0\mathrm{ }-\mathrm{ }\mathrm{R}\mathrm{E}\mathrm{R}}{0.3}\right)·19.62\right)\right]}{60}\mathrm{ }·1000·{{\mathrm{A}}_{\mathrm{D}}}^{-1}$$ (1)

where A_D is Dubois surface area (m²). External work (W; W/m²) was calculated as follows:

$$W= 9.81·{\mathrm{m}}_{\mathrm{b}}·{\mathrm{v}}_{\mathrm{w}}· {\mathrm{F}}_{\mathrm{g}} · {{\mathrm{A}}_{\mathrm{D}}}^{-1}$$ (2)

where m_b is body mass (kg), v_w is walking velocity (m/min), and F_g is fractional grade of the treadmill (17). M_net (W·m⁻²) was then calculated as M – W.

Sweat rate was calculated as the difference between pre and postexercise nude body mass on a scale accurate to ±20 g. Subjects were not allowed to intake fluid during the experimental trial, and no effort was made to account for respiratory losses as they are negligible in the environments of interest.

Statistical Analysis

No a priori power analysis was conducted; however, a post hoc power analysis using a slope of 0.9 from the relation between original and reliability trials suggested that we were adequately powered (Power = 0.99) to detect significant correlations in the present study. In an effort to ensure the most comprehensive and transparent results, reliability and reproducibility were assessed using multiple indices. For reliability trials, intraclass correlation coefficients (ICCs; two-way mixed effects, single measures, and absolute agreement) were used to compare the original and repeat T_db or P_a at which each critical point was observed (IBM SPSS Statistics v26, Armonk, NY). For validity trials, because the dependent variable of interest was flipped from the original trial to the validity trial (i.e., if the original trial was to determine the T_crit at a given P_a, the validity trial was to determine the P_crit at the critical T_db from the original trial), ICCs were performed for both T_db and P_a between original and validity trials. In addition, data were converted to wet-bulb temperature (T_wb) values and Bland–Altman plots (along with the calculated upper and lower limits, bias, and correlation coefficients) are presented for reliability and validity trials. One-way ANOVA was used to evaluate potential differences in M_net between trials with Tukey’s multiple-comparison procedure for post hoc pairwise comparisons of interest. Sweat rates for original and repeat trials were compared using paired-samples t tests. No comparison was made for sweat rate for validity trials because the environmental conditions (and, thus, sweat rates across the entirety of the trial) were not directly comparable.

Group data are reported as means ± standard deviation. Significance was set at P < 0.05.

RESULTS

Subject Characteristics

Subject characteristics are presented in Table 1. All subjects were young, healthy adults with body mass indices in the normal range.

Table 1.

Subject characteristics

Reliability Trials	Means ± SD
Age, yr	25 ± 4
Height, m	1.72 ± 0.12
BMI, kg/m2	23.5 ± 3.5
V̇o2max, mL·kg−1·min−1	50.1 ± 12.2

Validity Trials	Means ± SD
Age, yr	25 ± 4
Height, m	1.71 ± 0.13
BMI, kg/m2	23.7 ± 3.4
V̇o2max, mL·kg−1·min−1	47.7 ± 9.7

BMI, body mass index; V̇o_2max, maximal oxygen consumption.

Metabolic Heat Production and Sweat Rate

There were no differences in M_net between original and reliability trials (133 ± 12 vs. 127 ± 16; P = 0.11), nor between original and validity trials (131 ± 13 vs. 135 ± 14; P = 0.41). In addition, there were no differences in sweat rate between original and reliability trials (219 ± 82 vs. 208 ± 67; P = 0.49). Similarly, there were no differences in percent body mass loss between original and reliability trials (1.06 ± 0.46 vs. 0.98 ± 0.44; P = 0.43), nor between original and validity trials (0.89 ± 0.31 vs. 1.01 ± 0.43; P = 0.41).

Reliability

Table 2 shows the combination of T_db and P_a, as well as the corresponding T_wb, at the T_gi inflection point for original and reliability trials. There were no differences in critical T_wb between original and reliability trials (P = 0.78). The reliability for the determination of T_crit and P_crit was excellent (ICC = 0.98). Similarly, the reliability of the critical T_wb between original and reliability trials was excellent (ICC = 0.91; Fig. 2).

Table 2.

Psychrometric loci for ambient dry-bulb temperature (T_db) and water vapor pressure (P_a), with the corresponding wet-bulb temperature (T_wb), at the core temperature inflection point for all subjects during original and reliability trials

	Trial 1		Trial 2
Subject	Tdb, °C; Pa, mmHg	Twb	Tdb, °C; Pa, mmHg	Twb
Subj 1	(38.1, 25.1)	28.9	(37.9, 26.1)	29.3
Subj 2	(36.0, 24.2)	28.0	(35.8, 23.4)	27.6
Subj 3	(37.8, 19.3)	26.1	(41.1, 20.0)	27.3
Subj 4	(38.1, 24.0)	28.4	(38.0, 22.0)	27.5
Subj 5	(40.5, 19.2)	26.7	(40.0, 22.5)	28.2
Subj 6	(42.2, 16.2)	25.7	(43.8, 16.3)	26.1
Subj 7	(35.8, 24.6)	28.1	(36.0, 23.0)	27.4
Subj 8	(44.7, 11.3)	23.8	(45.5, 12.4)	24.6
Subj 9	(43.1, 16.0)	35.8	(40.6, 16.0)	25.2
Subj 10	(47.8, 12.0)	25.0	(47.7, 11.3)	24.6
Subj 11	(42.6, 12.3)	23.7	(41.2, 12.3)	23.3
Means ± SD		26.4 ± 1.8		26.5 ± 1.8

Figure 2.

Bland–Altman plot of critical wet-bulb temperature (T_wb) for original and reliability trials (n = 11; 5 men, 6 women). The solid line at y = 0 represents perfect reliability. The dashed lines represent the 95% confidence interval of the bias. ICC, intraclass correlation coefficient.

Validity

Table 3 shows the combinations of T_db and P_a, and the corresponding T_wb, at the T_gi inflection point for original and validity trials. There were no differences in critical T_wb between original and validity trials (P = 0.32). There was excellent agreement in the T_db (ICC = 0.95) and P_a (ICC = 0.96) at the T_gi inflection point between original and validity trails. Likewise, there was excellent agreement in the critical T_wb between original and validity trials (ICC = 0.91; Fig. 3).

Table 3.

Psychrometric loci for ambient dry-bulb temperature (T_db) and water vapor pressure (P_a), with the corresponding wet-bulb temperature (T_wb), at the core temperature inflection point for all subjects during original and validity trials

	Trial 1		Trial 2
Subject	Tdb, °C; Pa, mmHg	Twb	Tdb, °C; Pa, mmHg	Twb
Subj 1	(43.1, 11.7)	23.5	(43.1, 13.0)	24.3
Subj 2	(38.1, 21.0)	27.0	(38.2, 20.3)	26.7
Subj 3	(37.8, 19.3)	26.1	(38.1, 18.8)	26.0
Subj 4	(35.9, 24.3)	28.0	(34.3, 24.2)	27.6
Subj 5	(39.2, 19.8)	26.7	(38.3, 22.4)	27.9
Subj 6	(42.2, 16.2)	25.7	(42.2, 17.7)	26.5
Subj 7	(35.8, 24.6)	28.1	(36.8, 25.4)	28.7
Subj 8	(40.5, 23.5)	28.8	(40.5, 23.0)	28.5
Subj 9	(43.1, 16.0)	25.8	(42.9, 14.9)	25.2
Subj 10	(39.9, 20.7)	27.3	(42.0, 21.0)	28.0
Means ± SD		26.7 ± 1.5		26.9 ± 1.5

Figure 3.

Bland–Altman plot of critical wet-bulb temperature (T_wb) for original and validity trials (n = 10; 4 men, 6 women). The solid line at y = 0 represents perfect reliability. The dashed lines represent the 95% confidence interval of the bias. ICC, intraclass correlation coefficient.

DISCUSSION

PSU HEAT is an ongoing project aimed at determining the critical environmental conditions above which human heat balance cannot be maintained. The objective of the present study was to determine reliability (between-visit reliability) and validity of the experimental paradigm used. Our data demonstrated a high degree of reliability for M_net and SR, as well as the T_wb, T_db, and P_a at which the T_gi inflection point occurred. In addition, the protocol yielded critical environmental loci that were not significantly different regardless of whether T_db or P_a is being systematically increased. As such, these findings provide evidence of validity, rigor, and reproducibility of this experimental approach.

Reliability of this progressive heat stress protocol has been previously reported (r = 0.97), although that study included only a small subgroup of individuals who completed reliability trials (4). Given the importance of robust experimental design that provides reliable data, providing evidence for experimental rigor and reproducibility is critical. Indeed, the National Institutes of Health have placed a strong emphasis on rigor and reproducibility in recent years (18). The reliability data in the current study confirm the previous findings from our laboratory (4) that this experimental paradigm is highly reliable. Similarly, the validity trials provide further evidence supporting the use of this experimental approach to establishing thermal environments that place various populations at risk of heat-related morbidity and mortality.

LIMITATIONS

Only young, healthy subjects were tested in the present study. However, comparisons were conducted within subjects, so it is unlikely that age would affect reliability and validity. In addition, this progressive heat stress protocol has been used to identify critical environmental limits for a range of exercise intensities. The present study only reported reliability and validity for physical activity approximating light activities of daily living. However, our findings were similar to those previously reported for an exercise intensity of 30% V̇o_2max (4). Thus, we do not expect that exercise intensity is likely to influence the reliability and validity of this experimental approach.

Conclusions

The present study demonstrates excellent reliability of the progressive heat stress protocol used to identify critical environmental limits for the maintenance of heat balance in humans. In addition, these data suggest that uncompensable heat stress occurs at the same combination of ambient temperature and humidity (i.e., psychrometric loci), regardless of which environmental variable is being experimentally manipulated. Together, these data provide strong evidence that this experimental paradigm is rigorous, reliable, and valid, supporting its use to investigate critical environmental limits to the maintenance of heat balance.

GRANTS

This work was supported by the National Institutes of Health (NIH) Grant RO1 AG067471 (to W.L.K.) and National Institute on Aging (NIA) Grant T32 AG049676 (to D.J.V.).

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

R.M.C., S.T.W., and W.L.K. conceived and designed research; R.M.C., S.T.W., and Z.S.L. performed experiments; R.M.C. analyzed data; R.M.C., S.T.W., Z.S.L., and W.L.K. interpreted results of experiments; R.M.C. prepared figures; R.M.C. drafted manuscript; R.M.C., S.T.W., Z.S.L., and W.L.K. edited and revised manuscript; R.M.C., S.T.W., Z.S.L., and W.L.K. approved final version of manuscript.