Indicators to assess physiological heat strain – Part 2: Delphi exercise

ABSTRACT

In a series of three companion papers published in this Journal, we identify and validate the available thermal stress indicators (TSIs). In this second paper of the series, we identified the criteria to consider when adopting a TSI to protect individuals who work in the heat, and we weighed their relative importance using a Delphi exercise with 20 experts. Two Delphi iterations were adequate to reach consensus within the expert panel (Cronbach’s α = 0.86) for a set of 17 criteria with varying weights that should be considered when adopting a TSI to protect individuals who work in the heat. These criteria considered physiological parameters such as core/skin/mean body temperature, heart rate, and hydration status, as well as practicality, cost effectiveness, and health guidance issues. The 17 criteria were distributed across three occupational health-and-safety pillars: (i) contribution to improving occupational health (55% of total importance), (ii) mitigation of worker physiological strain (35.5% of total importance), and (iii) cost-effectiveness (9.5% of total importance). Three criteria [(i) relationship of a TSI with core temperature, (ii) having categories indicating the level of heat stress experienced by workers, and (iii) using its heat stress categories to provide recommendations for occupational safety and health] were considered significantly more important when selecting a TSI for protecting individuals who work in the heat, accumulating 37.2 percentage points. These 17 criteria allow the validation and comparison of TSIs that presently exist as well as those that may be developed in the coming years.

Introduction

Occupational heat stress is detrimental for worker health and productivity [1–8]. This is a day-to-day issue for billions of people who perform their duties in hot environments [9], often exceeding their bodies’ capacity for thermoregulation [1,8,10–12]. When elevated ambient temperatures are accompanied by work tasks requiring high metabolic demands, such as carrying or lifting weights, physiological heat strain increases drastically [7,8,13,14]. This is an unavoidable combination in many occupations [15] and sometimes leads to fatal accidents at work [1,10–12], as well as constant danger for life-threatening complications in cases where underlying cardiovascular diseases are present [16]. This is exacerbated by global warming which is worsening the conditions in which people perform their daily activities. Though this is expected to be more hazardous for people working and living close to the Earth’s equator [9], individuals employed in heavily mechanized and industrialized workplaces are also subjected to increased physiological strain from heat stress [1]. The importance of this topic has driven hundreds of scientists to develop sophisticated equations, known as thermal stress indicators (TSIs), which use information from the prevailing environment to assess the physiological heat strain experienced by workers. These TSIs can play an important role in mitigating heat-related illness and deaths in occupational settings through alerts and as a basis for heat-health advisories [17–19].

In a series of three companion papers published in this Journal, we identified the TSIs developed since the dawn of scientific research (part 1) [20], we conducted a Delphi exercise to understand what is important to consider when adopting a TSI to protect individuals who work in the heat (part 2), and we conducted field experiments across nine countries to evaluate the efficacy of each TSI for quantifying the physiological strain experienced by individuals who work in the heat (part 3) [21]. In the 1^st paper of this series, we conducted a systematic review which identified 340 unique TSIs that have been developed between 200 BC and 2019 AD to assess the heat stress and strain experienced by people performing various activities over a wide operating range and conditions. Of these TSIs, 153 represent nomograms, specific instruments, and complex models, while the remaining 187 TSIs are formulas that can be calculated utilizing only meteorological data (air temperature, relative humidity, wind speed, and solar radiation).

Many TSIs have been adopted in different health advisories as well as by scientists and industrial organizations [17–19], but there are no criteria weighing the relative importance of different aspects influencing health outcomes. Indeed, we do not currently know if physiological strain indices of heat stress such as the body core temperature, skin temperature, heart rate, or hydration state are parameters that a TSI should be able to accurately align with [22]. Moreover, even if we assume that these parameters are important for maintaining a healthy and productive life, and therefore should always be considered, we still do not know their relative importance when assessing the physiological strain experienced by someone. For instance, is it more important for a TSI to detect a high body temperature or an increased level of dehydration?

The present article is the second in the above-mentioned series of three companion papers published in this Journal. The purpose of the work described here was to conduct a Delphi exercise aiming to identify the criteria to consider when adopting a TSI to protect individuals who work in the heat, as well as to weigh the relative importance of these criteria. The Delphi methodology is a well-established [23] structured communication technique based on collecting opinions from a panel of experts while eliminating the influence of participants upon one another (i.e., using the rules of brainstorming where participants withhold judgment). This analysis informed the subsequent parts of this series of companion papers, where the efficacy of different TSIs was evaluated based on physiological data [21].

Methodology

Selection of experts and communication

Following standard Delphi methodology [23–26] adopted recently to address issues related to occupational heat stress [27,28], we asked an interdisciplinary group of 21 experts from different countries to participate in a Delphi exercise. The invited experts were either prolific scientists recognized internationally for their work in one of the different disciplines that came together in the present research (environmental physiology, biometeorology, physics, chemical engineering, and occupational health/safety), or policy makers with significant experience and contribution in occupational health and safety (see Results section). Some of the authors of the present paper were included in the experts invited but were not involved in data curation and analysis. The inclusion criteria for participating in the Delphi exercise were as follows: (i) holding a Doctoral Degree in one of the above-mentioned disciplines, (ii) proven expertise in occupational heat stress, as defined by >5 years of relevant professional experience and/or >3 relevant scientific publications, as well as (iii) willingness to participate in the study. Participation was voluntary and no financial compensation was provided. The moderator (ADF) communicated with each expert on an individual basis and the survey scores provided were recorded anonymously on a master datasheet. The survey was distributed as a spreadsheet document via e-mail accompanied by a description of the Delphi exercise (Table 1). Anonymity amongst experts was carefully maintained throughout, as it is a key part of the Delphi methodology [23–25], contributing to (i) reduced risk of participant unwillingness to abandon publicly expressed opinions, (ii) protection from group pressure in cases of changing a previously expressed opinion, and (iii) elimination of the “halo effect” (i.e., participants favoring recommendations by dominant/prominent individuals) [26,29,30].

Table 1.

Individual messages to each respondent during the two iterations of the Delphi exercise.

1st iteration

“The attached Delphi survey aims to score different criteria for their importance for adopting a thermal indicator to protect individuals who work in the heat. It also aims to uncover other important criteria who were not initially included in our list. The Delphi method is a way of obtaining a collective view from experts about issues where there is no or little definite evidence and where opinion is important. This survey incorporates 12 criteria where you have to score from zero (0) to one hundred (100) based on their importance for adopting a thermal indicator to protect people who work in the heat. It is important to note that although you can score a single item with one hundred (100) points, the total score of all criteria should not exceed one hundred (100), which means that, in this case, the remaining 11 criteria should be scored with zero (0). The idea behind this, is that you should distribute one hundred score points among all 12 criteria. If you feel that an item is not important at all, please insert zero (0).” Please note that the Delphi survey incorporates the following item, allowing you to add other important criteria that had not been considered: “If you feel that there are other important criteria that have not been considered, please list them here and insert a relevant score.”

2nd iteration

“Following the completion of the 1st iteration of the Delphi survey that you completed a few weeks ago, please find attached here the 2nd iteration. Once again, I would be grateful if you could complete the survey keeping in mind that this 2nd iteration includes the following two important differences: 1. Next to each item, you will find the mean and the standard deviation of the score from the 1st iteration, which was completed by 20 evaluators who are all experts in environmental physiology, and/or biometeorology, and/or thermal modelling, and/or occupational health policy. 2. Below the initial 12 criteria, you will find five more criteria that were suggested by experts during the 1st iteration of the Delphi survey. If you feel that these criteria are important for adopting a thermal indicator to protect individuals who work in the heat, please score them accordingly. As in the 1st iteration, please keep in mind that the total score of all criteria should not exceed one hundred (100).”

Delphi criteria

The criteria used in the Delphi exercise are listed in Figure 1 and are further described in an Online Supplement. A small monitor team (LGI and ADF) designed the 1^st iteration of the survey, which then was circulated to the panel of experts. The criteria included in the 1^st iteration were developed based on existing literature in occupational health-and-safety [2,10,18,31–35], heat strain during physical work [1,2,4,9,11,22,31,36], and biometeorological factors related to work in hot environments [13,17,18,37,38]. During the 1^st iteration we asked the experts to score 12 criteria in the survey for their importance when considering to adopt a TSI to protect individuals who work in the heat (Figure 1, criteria #1-12). Experts were informed that they could distribute a total of 100 points across all 12 criteria. In the same iteration, we also asked them to list any additional criteria that had not been considered in the survey. The experts added the need for a TSI to reflect the level of dehydration (Figure 1, criterion #13) as well as higher thresholds (>39°C) for mean body temperature (Figure 1, criteria #14-17). During the 2^nd iteration, a revised version of the survey with all 17 criteria was sent to the same experts accompanied with the score (mean ± sd) of criteria #1-12 from the 1^st iteration. Experts were informed that they could distribute a total of 100 points across all 17 criteria and were encouraged to consider their answers in the 1^st iteration in light of the general group responses, a process that facilitates converging towards consensus [23–26]. For this purpose, we used the arithmetic mean because (i) it is intuitive, (ii) it is the mostly used measure of central tendency in Delphi [39], and (iii) led to a cumulative of 100 percentage points (each point representing 1% of importance when adopting a TSI) in our analysis which was convenient and intuitive for evaluating the TSIs. Therefore, the relative importance of each criterion was indicated by its score.

Figure 1.

Scores (means ± SD) of different criteria for their importance for adopting a thermal stress indicator to protect individuals who work in the heat in the 1^st (faded bars) and the 2^nd (solid bars with gray border) iteration of the Delphi exercise. The criteria used in the Delphi exercise are listed at the bottom and are further described in the Online Supplement. They are colored to indicate the three occupational health-and-safety pillars: contribution to improving occupational health (green), mitigation of worker physiological strain (blue), and cost-effectiveness (red).

Analysis of Delphi data

In the present Delphi exercise, the proposed criteria were deemed important only when being endorsed (i.e., receiving a score higher than “0”) by more than 7 evaluators (one-third of the panel of experts). The concept of consensus within the respondent group was defined as a condition of consistency of opinion among the experts who work in the different disciplines that came together in the present study. To examine the extent of consensus within the group of experts we utilized three statistics. First, Cronbach’s coefficient alpha (α) [40] was used to examine the reliability and thus the internal consistency of the panel of experts [41], as follows:

$$Cronbach{ }^{\mathrm{\prime }}scoefficienta=\frac{n}{n-1}\times \left(1-\frac{{\mathrm{\Sigma }}_i{\sigma }_i^2}{{\sigma }_t^2}\right)$$

where “n” is the number of experts, ${\sigma }_i^2$ is the variance of each individual expert responses, and ${\sigma }_t^2$ is the variance of the sum responses for each individual expert [41]. The smaller the variance among the experts’ scores, the closer Cronbach’s coefficient α will be to 1.0, indicating perfect internal consistency. Following previous literature on Delphi technique [42], an overall Cronbach’s coefficient α value higher than 0.8 was considered as a threshold demonstrating a good internal consistency (≥0.7: acceptable, ≥0.8: good; and ≥0.9: excellent) [43]. Error variance in the scores was calculated by squaring the Cronbach’s coefficient α, multiplying by 100, and subtracting from 100 [44]. That is to say, if an iteration reached the set reliability threshold of 0.8, there is a 36% error variance (e.g., random fluctuation) in the scores of experts [44].

Pearson’s correlation coefficient was used to examine potential associations in the scores of experts between the Delphi iterations. Moreover, an additional correlation analysis was conducted to investigate potential associations in the mean scores of each diagnostic indicator between the Delphi iterations. This is because a very strong association between the Delphi iterations is considered a good indication for the extend of consensus reached in the panel of experts [41,45,46].

In addition to the Cronbach’s α and Pearson’s correlation, paired t-tests were used to examine potential differences in the mean scores of the Delphi criteria between the two iterations. Paired t-tests were also used to compare mean values amongst the 17 criteria of the 2^nd iteration. Statistical significance was set at p < 0.05. Following previous methodology on the Delphi technique [42], the Bonferroni correction was used to adjust p values for multiple comparisons, resulting in a threshold of p < 0.004 in cases of comparisons of mean values between the two Delphi iterations and a threshold of p < 0.0004 in cases of comparisons of mean values amongst the 17 criteria of the 2^nd iteration.

Statistical analyses were conducted using both the SPSS v27.0 (IBM, Armonk, NY, USA) and Excel spreadsheets (Microsoft Office, Microsoft, Washington, USA). All results are presented as mean ± SD, unless otherwise indicated.

Results

Of the 21 invitations sent, 20 (95%) experts agreed to participate in the present Delphi exercise. This group size is appropriate to reach a meaningful outcome, as groups of ≥12 experts are considered adequate, while groups of ≥20 are considered robust [24,25,47]. The experts in our study were professors, researchers, policy makers, or health-and-safety professionals working in institutions located across 12 countries: Australia, Canada, Denmark, Greece, Israel, Netherlands, Portugal, Slovenia, Spain, Sweden, United Kingdom, and the United States of America. The expert panel included 18 men and 2 women aged 49.5 ± 12.6 years who, as a group, had published more than 2,600 relevant peer-reviewed articles and their research was cited more than 130,000 times. As the panel included professionals from national meteorology services and policy making organizations who had contributed to occupational health and safety legislation but tended to publish less often than researchers and academics, the group’s citations (6402.9 ± 8278.0) and h-index (32.5 ± 25.8) based on Google Scholar varied considerably.

The study took place from December 2019 to March 2020. Two Delphi iterations were enough to reach consensus within the expert panel (see statistics below). The overall response rate was 100% both for the first and second Delphi iterations. During the first iteration, the experts submitted their surveys within a period of two weeks (2.3 ± 3.4 days), ranging from 0 to 14 days. The different criteria incorporated in the Delphi survey were given an average score of 8.3 ± 3.4 percentage points (pp) ranging from 4.4 to 15.4 pp (Figure 1). Additionally, five new criteria were proposed by the experts and were added in the next iteration of the survey (Figure 1). During the second iteration, the experts submitted their surveys within two months (7.3 ± 15.4 days), ranging from 0 to 59 days. The relatively longer response time in the 2^nd iteration was due to issues related to the COVID-19 outbreak that had just become a global pandemic. The criteria incorporated in the Delphi survey were given an average score of 5.9 ± 3.5 pp ranging from 1.9 to 14.4 pp. It is important to note that the five new criteria were given an average score of 3.6 ± 1.7 pp which was almost half of the score (6.8 ± 3.7 pp) given to the 12 original criteria (Figure 1). In total, all 17 criteria were endorsed by more than one-third of the experts (71.5 ± 18.4%), and thus, they were all deemed important to consider when adopting a thermal stress indicator to protect individuals who work in the heat. The identified criteria addressed three occupational health-and-safety pillars: (i) contribution to improving occupational health, (ii) mitigation of worker physiological strain, and (iii) cost-effectiveness. Across all tables and figures, distinct colors are used to clearly indicate the criteria in each pillar.

Cronbach’s coefficient α for the first iteration of the present Delphi exercise was 0.77 and was increased to 0.86 in the second iteration. This indicates that the error variance among the experts was reduced by ~15%, from ~40% to ~25%, after the second Delphi iteration. Moreover, the correlation in the individual scores given by each expert between the two iterations was strong (r = 0.69, p < 0.001), ranging between 0.32 and 0.98 for the different experts who participated in the present Delphi exercise. This suggested that some experts were outliers in the first Delphi iteration and modified their scores to reach consensus. The overall correlation in the average scores of each Delphi item between the two iterations was very strong (r = 0.98, p < 0.001), revealing the same pattern of importance between the two iterations (Figure 1). This very strong correlation confirmed further the high level of consensus reached within the panel of experts after the second iteration [45,46]. Moreover, there were no statistically significant differences in the scores of the different Delphi criteria between the two iterations (Bonferroni adjusted p = 0.004; Table 2), which is considered an extra indication for establishing consensus in the Delphi technique [48].

Table 2.

Results (mean ± sd) and comparisons in the two iterations of the Delphi exercise across the 17 criteria.

No statistically significant differences were identified between the two iterations at a Bonferroni adjusted alpha of 0.004. Criteria are colored to identify the three occupational health-and-safety pillars: contribution to improving occupational health (green), mitigation of worker physiological strain (blue), and cost-effectiveness (red). Key: d = Cohen’s d effect size; p = p value from two-tailed paired samples t tests comparing the two iterations.

Paired t-tests comparing mean values amongst the 17 criteria of the 2^nd iteration of the Delphi exercise demonstrated that three criteria were considered significantly more important (Bonferroni adjusted p = 0.0004; Table 3), accumulating to a score of 37.2 pp (Figure 2). These were as follows: (i) relationship (Pearson’s correlation coefficient) of a TSI with core temperature (criterion #1); (ii) having categories indicating the level of heat stress experienced by workers (criterion #9); (iii) using the heat stress categories of the TSI to provide recommendations for occupational safety and health (criterion #10). The same analysis demonstrated that criteria #5, #7, #8, and #14-17 were perceived as significantly less important (Bonferroni adjusted p = 0.0004; Table 3), accumulating to a score of 22.4 pp.

Table 3.

Results (p values) from two-tailed paired samples t tests comparing the scores across the different criteria (indicated by numbers in colored boxes) in the 2^nd Delphi iteration.

Grey boxes indicate statistical significance at a Bonferroni adjusted alpha of 0.0004. Criteria are colored to indicate the three occupational health-and-safety pillars: contribution to improving occupational health (green), mitigation of worker physiological strain (blue), and cost-effectiveness (red).

Figure 2.

Stacked bar indicating the importance scores of the different criteria for adopting a thermal stress indicator to protect individuals who work in the heat in the 2^nd iteration of the Delphi exercise. The numbers in the stacked bars or the callouts indicate the number of each criterion. Criteria are colored to indicate the three occupational health-and-safety pillars: contribution to improving occupational health (green), mitigation of worker physiological strain (blue), and cost-effectiveness (red).

Discussion

In this Delphi exercise, there was a high consensus in the panel of experts regarding the criteria that should be considered important when adopting a TSI to protect individuals who work in the heat. The initially suggested 12 criteria were all considered important by the panel of experts, who also proposed five more criteria, reaching a total of 17 unique criteria with varying weights that a TSI should be able to align with. In particular, during the 1^st iteration the panel of experts added the need for a TSI to associate with the level of workers’ dehydration, which is known to induce a wide spectrum of deleterious effects on worker health and well-being [1,3,5]. Also, the panel added a higher level of mean body temperature threshold (>39°C) for the criteria #14-17, reflecting a state of extreme hyperthermia and/or exertional heat stroke [49].

The present Delphi exercise was performed based on standard methodology [23–26] adopted recently to address issues related to occupational heat stress [27,28]. The identified criteria were distributed across three occupational health-and-safety pillars: (i) contribution to improving occupational health, (ii) mitigation of worker physiological strain, and (iii) cost-effectiveness. The contribution of a TSI to improving occupational health (i.e., pillar 1) accumulated to a score of 55.0 pp across ten Delphi criteria (#5-10 and #14-17 in Figure 1). Although some of the criteria in this pillar were considered comparatively less important, the accumulated importance score of 55.0 pp is higher than the total score of the remaining two pillars. The importance given to these criteria reflects the value of proactivity in preventing heat-induced occupational injuries and illnesses. Of course, to ensure any benefit from this Delphi exercise, a collective worldwide effort is required to bring the voice of experts to the surface [50]. That is to say, mass media [17,18] together with implementation of heat-health legislation [2,51,52] could be utilized as a means to deliver a strong message toward adopting a TSI on worker day-to-day activities, and thus preventing the adverse effects of workplace heat stress on worker health and wellbeing.

Mitigation of worker physiological strain accumulated to a score of 35.5 pp across five Delphi criteria (#1-4 and #13 in Figure 1), making it the second most important pillar. This pillar reflects the importance given by the experts to the real-time evaluation of the physiological heat strain experienced by people who work in the heat. The physiological criteria of “core temperature”, “mean skin temperature”, “mean body temperature”, “heart rate”, as well as the “hydration state” of a worker represent the main variables that the current technological advances allow us to continuously measure in ecological studies [6,22,53], and thus all of them were considered in the present Delphi exercise. It is logical to assume that the rapid technological progress that is currently taking place will soon enable new capabilities for ecological studies [6], and therefore the current Delphi exercise should be repeated in due course of time, probably within the next decade. By then, new criteria such as “skin blood flow”, “respiratory rate”, “sweating rate”, and “vasomotion” may be able to be examined in ecological studies and thus eligible for being considered by experts in a future Delphi exercise.

The cost-effectiveness characterizing a TSI reached a total score of only 9.5 pp across two Delphi criteria (#11-12 in Figure 1). This lower importance attributed by our experts to the costs associated with heat mitigation is in contrast to opinions by policy makers stating that cost is one of the main barriers for adopting heat mitigation strategies [54]. Although one could argue that the latter is not entirely accurate [55], science and policy should come together and form much-needed legislation to address occupational heat stress, as recently seen in some countries [2,51,52]. This can be achieved by (i) identifying the priorities for science and for policy, (ii) holding discussions with participants from both science and policy with a mediator, and (iii) involving actors who have decision making capacity in either science or policy, but understand both [56].

This Delphi exercise showed that three criteria were considered significantly more important when selecting a TSI for protecting individuals who work in the heat. Specifically, the relationship of a TSI with core temperature (criterion #1), having categories indicating the level of heat stress experienced by workers (criterion #9), as well as using its heat stress categories to provide recommendations for occupational safety and health (criterion #10) were considered the most important criteria by all experts, accumulating to a score of 37.2 pp. The requirement for a TSI to be associated with core temperature (importance score: 12.32) is overwhelmingly supported by the literature, as core temperature is the single most important criterion for heat-related illnesses [1,9,11,22,27,32,35]. Also, the high importance placed in having categories indicating the level of heat stress experienced by workers (importance score: 12.61) as well as in using them to provide recommendations (importance score: 15.36) is also reflected in some of the most widely accepted existing guidelines, including those proposed by the American Conference of Governmental Industrial Hygienists [32], the International Standardization Organization [33], and the North Atlantic Treaty Organization [34].

The present article is the second in a series of three companion papers published in this Journal. In these articles, we identified the TSIs developed since the dawn of scientific research (part 1) [20], we conducted a Delphi exercise to understand what is important to consider when adopting a TSI to protect individuals who work in the heat (part 2; present article), and we conducted field experiments across nine countries to evaluate the efficacy of each TSI for quantifying the physiological strain experienced by individuals who work in the heat (part 3) [21]. We used the criteria developed in the present Delphi exercise along with their associated weights to inform the third part of this series of companion papers, where the efficacy of different TSIs was evaluated based on physiological data [21].

In conclusion, this Delphi exercise demonstrated a high consensus toward the identification of 17 criteria with varying weights that should be considered when adopting a TSI to protect individuals who work in the heat. These criteria allow the validation and comparison of TSIs that presently exist as well as those that may be developed in the coming years. This is important since occupational heat stress affects workers’ health and productivity [1,3,4,6], and it is projected to worsen with the ongoing global warming [9]. This Delphi exercise showed that three criteria were considered significantly more important when considering to adopt a TSI for protecting individuals who work in the heat. Specifically, the relationship of a TSI with core temperature (criterion #1), having categories indicating the level of heat stress experienced by workers (criterion #9), as well as using its heat stress categories to provide recommendations for occupational safety and health (criterion #10) were considered the most important criteria by all experts, accumulating to a score of 37.2 pp. This Delphi exercise was specific to occupational settings, and it is important for future studies to identify the criteria that should be considered important for adopting a TSI to protect non-occupational populations, as well as the general population including older individuals, people with underlying diseases, warfighters, and athletes. This will enable the development of a wide new range of heat mitigation measures, including legislation and guidance, to protect people who work under heat stress.

Funding Statement

The study has received funding from the European Union’s Horizon 2020 research and innovation programme [project HEAT-SHIELD; Grant agreement No 668786].

Author contributions

Conceptualization: LGI, ADF, LN, GH, GPK; Data curation: LGI, ADF; Formal Analysis: LGI, ADF; Funding acquisition: ADF; Investigation: LGI, ADF; Methodology: LGI, GH, GPK, LN, ADF; Project administration: LGI, ADF; Supervision: ADF; Validation: ADF; Visualization: LGI, ADF; Writing – original draft: LGI, ADF; Writing – review & editing: LGI, SRN, FG, GAG, PCD, MB, YE, GH, MS, PB, IM, GPK, TEB, LN, ADF.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplementary material

Supplemental data for this article can be accessed here.