Comprehensive evaluation framework for intervention on health effects of ambient temperature

Abstract

Despite the existence of many interventions to mitigate or adapt to the health effects of climate change, their effectiveness remains unclear. Here, we introduce the Comprehensive Evaluation Framework for Intervention on Health Effects of Ambient Temperature to evaluate study designs and effects of intervention studies. The framework comprises three types of interventions: proactive, indirect, and direct, and four categories of indicators: classification, methods, scope, and effects. We trialed the framework by an evaluation of existing intervention studies. The evaluation revealed that each intervention has its own applicable characteristics in terms of effectiveness, feasibility, and generalizability scores. We expanded the framework's potential by offering a list of intervention recommendations in different scenarios. Future applications are then explored to establish models of the relationship between study designs and intervention effects, facilitating effective interventions to address the health effects of ambient temperature under climate change.

1. Introduction

Climate change impacts the weather, ecosystems, and human systems, leading to increased exposure to extreme events such as extreme temperatures, heavy rainfall, and droughts, all posing significant threats to human health [[1], [2], [3], [4]].

To address the direct health impacts of climate change due to extreme weather, the World Health Organization (WHO) and other United Nations organizations provide guidance on effective actions for climate change and health [5]. Among those actions, there are a variety of interventions to reduce heat exposure, or to cope with the negative health effects of heat exposure, which can be grouped into three main categories [6]. First, urban planning measures (landscape or urban level), aimed at the role of the physical environment affecting humans, which mainly include green space [7], blue space [8], albedo [9], and water mist [10]. Second, interventions for large populations and groups, from the perspective of public health or policy measures, which provide protective effects at the population level, such as cold and heat early warning systems [11,12], publicity and education [13,14], and health action plans [[15], [16], [17]]. Third, individual-level interventions, in which individuals directly address the impact of environmental temperatures, such as through clothing [18,19], personal comfort systems [20] hot- and cold-stimulated behavior [[21], [22], [23]], and melatonin and sapropterin drug intervention [24,25].

However, the unclear effectiveness of these interventions under extreme temperatures, coupled with the lack of assessment of their feasibility and generalizability, make it challenging to guide their implementation. Despite the existence of a Comprehensive Framework for Implementation Research (CFIR) to guide implementation research [26] and qualitative recommendations by scholars to advance health adaptation to climate change through this approach [27], there is still a need for a quantitative assessment framework specifically designed for interventions for ambient temperature health effects. Most studies currently evaluating the methodological quality of interventions are qualitative descriptions or systematic reviews using methodological quality assessment tools, such as Grading of Recommendations Assessment, Development, and Evaluation (GRADE) [28], or a critical appraisal tool for systematic reviews that includes randomized or non-randomized studies of healthcare interventions, or both (AMSTAR2) [29]. However, these assessments do not sufficiently reflect the methodologies of interventions for ambient temperature, nor do they comprehensively characterize the effects of intervention studies in terms of effectiveness, feasibility, and generalizability. Despite the WHO prioritizing a need for a comprehensive evaluation of the effectiveness more than a decade ago [30], only a few studies have responded to the need from the perspective of ambient temperature [6,31,32].

Here, based on a range of intervention reviews [[32], [33], [34], [35]], reports or guidelines [36,37], frameworks [[38], [39], [40], [41]], and standards of evidence [42], we developed an evaluation framework for the quantitative assessment of the effectiveness, feasibility, and generalizability of interventions, according to three types of health interventions for ambient temperature. Then, we conducted an evaluation of the research of intervention on the health effects of ambient temperature to trial the framework. We offered the recommendation level of each specific intervention in different scenarios and explored the potential of the framework to develop models of the relationship between research designs and intervention effects, facilitating the implementation of effective interventions to address the health effects of climate change.

2. Components of CEFI-HEAT framework

2.1. Definitions of interventions

We developed a comprehensive evaluation framework for intervention on health effects of ambient temperature (CEFI-HEAT) (Fig. 1), which includes definitions of interventions and evaluation indicators.

Fig. 1.

A comprehensive evaluation framework for intervention on the health effects of ambient temperature (CEFI-HEAT). (A) CEFI-HEAT framework. Only the evaluative indicators are shown in this figure. The detailed scoring items for each indicator are listed in Supplemental Information (Table S1). (B) The evaluation process based on CEFI-HEAT. It should be noted that only indicators spanning all the evaluation processes within each intervention type possess comparability. For instance, the research object in indirect interventions and all indicators of effects across the three intervention types. (C) The scoring steps. For detailed procedures, refer to Section3.3 on scoring procedures.

We defined an intervention as a certain measure, action, or program that can reduce people's disease burden, have a positive impact on people's physical and mental health, or ultimately promote health by improving the physical environment under the influence of ambient temperature. This broad definition includes both actual and simulated interventions extending beyond those already implemented, which enabled us to identify all measures that may be effective for health, thus ensuring a comprehensive basis for evaluating interventions. It should be noted that indicators used to represent intervention effectiveness or detailed information used to describe intervention are not the intervention we have defined.

Based on the three types of public health evidence and assessment templates in the Climate and Health Intervention Assessment [36] and the actions in the WHO guidance on climate change and health [5], we comprehensively considered the scale, subject of implementation, research object, and characteristics of the research field, and summarized three types of interventions: (1) proactive intervention, an urban-scale intervention usually led by the public sector, which mainly affects the physical environment and proactively affects health; (2) indirect intervention, a population-scale intervention that cannot be directly implemented by individuals, with the public sector as the implementer, and requires infrastructure or promotion by the public sector to benefit the health of the population; and (3) direct intervention, an individual-scale intervention that can be directly implemented by individuals through behavioral changes, commodity purchases, and adaptations that directly benefit health.

We have summarized specific interventions of these three types (Fig. 1A) with their characteristic keywords shown in Fig. 3. Our framework primarily assesses the study design and effects of each intervention based on methodological features. Therefore, distinct from the differentiation of intervention types, our classification of intervention subtypes is primarily driven by considerations related to study design. Thus, further considering the methodologies of different intervention studies, we divided the three types of interventions into five subtypes. The proactive intervention was further classified into “model simulation” and “empirical study”. The indirect intervention was further classified into “statistical analysis” and “empirical study”.

Fig. 3.

Descriptive statistics for intervention effects. The number represents the total number of studies corresponding to a specific intervention. Mean is the average score of all studies for a specific intervention, and SD is the standard deviation of the scores. The score derived from CEFI-HEAT reflects the quality of the research, and indicates how well our evaluation criteria have been met. Higher scores indicate that the intervention was more effective, feasible, and generalizable.

2.2. Evaluation indicators for study design

For the three types of interventions, we established four categories of indicators, namely “classification”, “methods”, “scope”, and “effects” (Fig. 1A). We first subjectively select some indicators, which we then optimized based on the methodological features ultimately included in the subsequent evaluation practice (Section 3), culminating in distinctive scoring items. The scoring items of indicators are listed below with the rationale for selection.

“Classification” reflects the characteristics of the intervention. The corresponding indicators used in studies were implemented as categorical variables in the analysis: temperature exposure (heat wave or cold wave, high or low temperature, or normal temperature); level of health indicators (environmental thermal comfort index, morbidity or mortality, psychological or physiological indicators); research levels (urban environment, building environment, indoor or outdoor environment); and research methods (model simulation, statistical analysis, or empirical study). Since heat loads are transmitted top-down from the urban environment to the built environment and then to individuals in the heat cascade [6], research level and research method were the exclusive indicators of proactive interventions for a finer classification.

In Fig. 1A, “Methods” indicated the quality of the intervention study design, including intervention design (intervention scenarios, multiple interventions, control groups, crossover trial), temporal resolution (yearly, monthly, daily), spatial resolution (country, city, individual), recruitment design (e.g., probability sampling, cohort recruitment, basic health requirements), and intervention duration (e.g., no more than 2 h, more than a decade).

“Scope” indicated the conditional control in the implementation of the intervention study, including data scale (e.g., cities included greater than 100, data time range greater than five years, population more than one million), sample size (e.g., number of tests more than 400), sample distribution (e.g., no significant differences between control groups and treatment groups), research object (environment, population, characteristic occupation or disease group, vulnerable group), and types of health indicators (environmental thermal comfort index, mortality or morbidity, behavior, psychological, physiological indicators). The types of health indicators here represent a more detailed differentiation for subsequent scoring, whereas the health indicator level serves as a rough classification for drawing the overall research roadmap (Fig. 2C).

Fig. 2.

Spatiotemporal distribution and roadmap of intervention studies. (A) Temporal distribution of publications. (B) Nationwide interventions counted by the capital city; those involving specific districts and counties were counted by the jurisdictional city or provincial capital; model simulation studies with unclear locations were counted by the address of the first author's unit. Studies involving five cities or fewer were counted for each city; studies covering the entire country or more than five cities were counted once by capital city. For research involving multiple countries, if there was a specific location, it was counted once for each specific city; otherwise, it was counted according to the capital of each country; Global-scale studies such as the Global Burden of Disease were not counted. (C) Overall research roadmap for all three interventions.

For indicators in these three categories, a higher score, indicating satisfaction of more scoring items, shows greater alignment of the study with our expectations. By extracting the details of these indicators, applying our CEFI-HEAT framework to assign scores, and combining them with the scores of intervention effects, we might be able to identify the precise conditions under which a specific intervention is effective.

2.3. Evaluation indicators for effects of intervention

“Effects” indicated the evaluation of intervention effects across three dimensions: effectiveness, feasibility, and generalizability. A higher score in the “effects” indicators indicated that the intervention study satisfied more items and was, therefore, more effective, feasible, and generalizable.

“Effectiveness” reflected whether the interventions were effective based on the primary outcomes reported by publications, with five items included: (1) “Positive effect” indicated that the intervention showed positive effects on human health under the influence of temperature. (2) “Significance” indicated that the statistical analysis was based on the design, and the main results passed the statistical test, referring to the standards for efficacy endorsed by the Society of Prevention Research [42]. (3) “Vulnerable group” indicated that the research objects included (or mainly were) vulnerable groups, such as the elderly and children, and the intervention was effective for them. (4) “Related research” considered that the three elements for grading the strength of the evidence were quality, quantity, and consistency [43]. We define that the intervention results consistent with at least two related experimental studies can be identified as scientifically supported by evidence. Our scoring criteria for this indicator follow the items in “types of health indicators” (Table S1), requiring consistency in impact direction across related studies, without delving into more detailed comparisons of specific health outcomes. (5) “Mechanism” indicated that a clear theory of causal mechanisms was stated.

“Feasibility” was used to assess the feasibility of transforming results into field applications from a practical point of view and included five items. (1) “Real condition” indicated that the intervention was implemented as intended in the real world [39], and was not only a simulated analysis. (2) “Detailed information” indicated that the intervention was described with detailed steps and parameters that would allow others to implement or replicate the intervention [38,42]. (3) “Quality control” indicated that the research described the factors that needed to be controlled during the implementation process to ensure the quality of the intervention. (4) “Completion rate” indicated the proportion of subjects who completed the entire intervention process. If a study primarily analyzed effects on the physical environment, this item was not scored. (5) “Application suggestions” indicated that the researchers provided suggestions to facilitate the implementation of the intervention in addition to the above information, such as for long-term maintenance [39], scale-up [35], and applying sustainable cooling strategies in heat-vulnerable settings [6]. “Generalizability” is a subjective indicator that was comprehensively reflected by technology maturity, cost-effectiveness, and willingness to participate.

2.4. Advance over existing frameworks

The CEFI-HEAT framework is a tailored framework for heat interventions, whereas the RE-AIM framework is too generic to assess specific heat interventions. Our framework includes three types of interventions and incorporates additional indicators, such as temperature exposure, which can capture the unique characteristics of studies focusing on heat interventions. By narrowing the scope to heat interventions, our framework ensures that all the components are directly relevant to evaluating the effects of interventions in heat-related contexts. Furthermore, as the health risks related to urban heat are expected to increase in the coming decade due to climate change and urbanization, there is an urgent need to cope with extreme heat through modifications in building and city design [44]. The CEFI-HEAT framework, through the three types of interventions, especially urban-scale interventions, emerges as a promising solution to these challenges.

As for the current evaluation frameworks, the five dimensions (reach, efficacy, adoption, implementation, maintenance) in the RE-AIM framework focus more on evaluating the impact of an intervention program or policy [39], while CFIR evaluates the factors that interfere with the implementation in the real world [26]. Some studies proposed conceptual frameworks to explore factors for scaling up public health interventions [34,45]. These frameworks mainly reflect the “effects” of intervention, while our framework adds “methods” and “scope” to reflect study design, which can more comprehensively characterize the process from theoretical research to implementation.

RE-AIM framework evaluates five dimensions and then combines them multiplicatively to derive the final public health impact score [39]. But how to evaluate the scores (ranging from 0 to 1) for each of the dimensions for specific heat intervention is unclear. In contrast, the new proposed CEFI-HEAT framework provides detailed, actionable scoring guidance (Section 3.3), compared with the more generic approach in previous frameworks. This makes it easier for researchers and evaluators to apply the CEFI-HEAT framework.

3. Application of the framework

3.1. Evaluation process

To demonstrate the potential of the CEFI-HEAT framework, we trialed it by conducting an evaluation of the research on interventions for health effects of ambient temperature. The evaluation process is shown in Fig. 1B.

First, we confirmed the literature retrieval strategy, including retrieval keywords, time range, and literature database. Second, we formulated corresponding inclusion and exclusion criteria according to the purpose of the evaluation and screened out qualified studies accordingly. Third, after determining the final included studies, the corresponding information was extracted according to the indicators of the CEFI-HEAT framework. Then, each study was evaluated according to the scoring procedures. Each indicator was used as a variable, and the corresponding score was assigned to form an intervention scoring data set for subsequent analysis.

3.2. Retrieval of intervention studies

We first searched two electronic literature databases (Web of Science and Scopus) from January 1, 2000 to June 1, 2022, with a broad search strategy to identify all interventions on health effects of ambient temperature (see Table S2 for the full search strategy). Only intervention studies focusing on temperature related to human health, characterizing the intervention outcomes, and published in English were included. We included only research articles and excluded reviews, non-human studies, studies that did not focus on health effects related to ambient temperature, and studies that did not quantify the effects of interventions (detailed selection criteria are shown in Table S3).

A total of 27,057 studies were initially retrieved. After removing duplicates, the remaining 22,591 studies were screened by title and abstract. A total of 540 studies were eligible for full-text review, and 278 intervention studies were included, of which 141 were proactive interventions, 51 were indirect interventions, and 86 were direct interventions (Fig. S1). The list of included literature after the full-text screening is shown in Table S4.

Titles, abstracts, and full texts were assessed independently for inclusion by two reviewers (J.W. and P.W.), and discrepancies were resolved by discussion with a third reviewer (L.H.). Data and information were extracted using standardized tables for subsequent scoring and statistical analysis.

It should be noted that while heat action plan involves a lot of interventions, it is still classified and evaluated as an indirect intervention, considering that it is implemented by the public sector and its purpose is to affect health. In addition, studies involving multiple interventions are distinguished according to their main intervention. For those interventions that are difficult to distinguish by definition (for example, air conditioning may be a housing facility, part of a heat action plan, or individual behavioral adaptation), the type is distinguished according to its research scale and method.

3.3. Scoring procedures

The example of scoring is shown in Fig. 1C. This process was used to score each included study. First, based on the research information extracted after literature retrieval, we confirmed the type and subcategory of the intervention, and found the corresponding evaluation process from Fig. 1A. Then, we found the evaluation indicators and scoring items along the process (Fig. 1A, Table S1), and derived the score of each indicator according to the two scoring methods.

(1) Direct scoring, in which each scoring item was matched directly with relevant research information. The score assigned to a particular indicator is denoted by the number marked before the respective scoring item (indicated by “1-, 2-, 3-, …”). For example, there are four items for “research object”, which are “1-environment”, “2-population”, “3-characteristic group (disease, occupation)”, and “4-sensitive group (the elderly, children)”. If the research object was the population, the indicator score was 2. Most indicators in CEFI-HEAT were scored in this manner.

(2) Cumulative scoring, in which the initial score for each indicator was set at 0, and the indicator items were compared with the extracted information one-by-one (indicated by “①②③ … ”). Whenever an item was met, the score was incremented by one. The total score obtained through this process represents the overall score of the indicator. For example, there are five items in “effectiveness”. If the research confirmed that the intervention had significant positive health effect with supporting results from related research, but was ineffective for sensitive groups and lacked exploration of the underlying mechanism, then only three of the five items were met, and its effectiveness score was 3.

Among the indicators of the four categories, those of “classification” were only used as categorical variables, and the other three were scored according to the indicator items for subsequent analysis.

During the scoring process, there were many cases where an item was not perfectly met or not met, especially for the items of indicators in the “Effects” category. For example, there were some unexpected situations: not all the indicators of health outcomes in the studies reflected that the intervention was effective (we think the study satisfies this item as long as one indicator shows a positive effect); the intervention was effective at the population level covering all age groups, but there was no stratified analysis for sensitive groups; multiple mechanisms were mentioned without definitive mechanism.

Therefore, in the actual scoring process, we adjusted the full score of the indicators in the “Effects” category from 5 to 7. The evaluation still refers to these items but allows certain subjective evaluations to deal with unexpected situations so as to better distinguish the difference in the “Effect” of the intervention. That is, if one item is met, the score will be 1 or 2; two items met, score 2 or 3; three items met, scored 3 or 4 or 5; four items met, scored 5 or 6; score 7 when all five items are met.

3.4. Statistical analysis

After several steps of literature retrieval, screening, inclusion, information extraction, and scoring according to CEFI-HEAT, a data set containing each indicator variable of all the included studies was formed. We performed a descriptive statistic based on this data set. By comparing the mean and standard deviation of the effectiveness, feasibility, and generalizability of each specific intervention, we identified the most effective, feasible, and generalizable interventions from the ranking of scores, and preliminarily analyzed the reasons for the differences in “effect” scores based on the contents of the included studies. All analyses were performed with Stata SE Version 15.1.

3.5. Evaluation result: spatiotemporal distribution and roadmap of intervention studies

Following the evaluation process, we conducted an evaluation by searching the Web of Science and SCOPUS databases for intervention studies since 2000 (see experimental procedures). We performed statistical analyses based on the score datasets of the 278 included studies (Table S4).

There was a marked increase in the number of original articles per year on the topic (Fig. 2A). Most of the included intervention studies were located in the United States, China, and European countries, with a few in Africa, South America, and other parts of Asia (Fig. 2B). A study has indicated that heat events are increasing in frequency and intensity, especially in the tropics and developing countries [46]. However, similar to the spatial distribution of intervention studies, research on heat and health effects was concentrated in developed countries, while insufficient attention was given to low-latitude, low- and middle-income countries [47]. These areas with lower socioeconomic development levels usually face a higher risk of temperature-related health burdens [48,49]. Consequently, countries that bear a disproportionate heat burden do not receive adequate academic support to address inequalities in heat impact.

From the research roadmap (Fig. 2C), empirical research on healthy populations (people not suffering from heat-related diseases, not taking prescription drugs, and not lacking other dietary or lifestyle requirements) was the mainstream, followed by model simulation research on the physical environment. However, few intervention studies targeted characteristics (occupation, location, ethnicity) or vulnerable groups. It is crucial to prioritize health interventions by combining spatial analysis information on climate hazards and risks for vulnerable populations [50]. Interventions targeting specific vulnerable groups can help protect human rights, promote social justice, and avoid creating new problems or exacerbating existing problems for vulnerable populations [51].

3.6. Evaluation result: descriptive statistics

The results of descriptive statistics are shown in Fig. 3. We distinguished the difference in the “effect” of specific interventions by comparing their scores to reveal the most effective, feasible, and generalizable interventions. According to the overall mean, there was no significant difference in the effectiveness and generalizability of the three types of interventions. Feasibility score ranked as follows: proactive (3.851) < indirect (4.431) < direct (4.663), indicating that, based on the definition and evaluation results of the CEFI-HEAT framework, individual interventions had the highest feasibility, followed by public health interventions, and built environment interventions were the least feasible.

Among proactive interventions, “green space”, “blue space”, and “water mist” interventions were found to be more effective. Compared with other interventions, “green space” has a clear mechanism, and thus a higher effectiveness score. Specifically, it can improve health outcomes through three potential mechanisms: people living in green areas engage in more frequent physical activity to promote health; green spaces reduce environmental hazards such as air pollution; and green spaces help restore attention, alleviate stress, and improve relaxation [52,53]. However, the feasibility and generalizability may be less than expected due to challenges such as insufficient green space provisions in densely populated areas, low priority of green space planning, and the potential risk of cementing social inequality [54]. “Blue space” usually had a better mitigation effect on the impact of high temperatures, but is highly dependent on water bodies (oceans, rivers, lakes, reservoirs) around the city [55], so this intervention cannot be extended to drylands that are likely to experience increasing heat waves and warm spells [56]. “Water mist” was feasible and scalable, and was expected to relieve thermal stress through two main routes: reducing the air temperature and lightly wetting the skin with evaporation [10]. But more evidence is needed to ensure its effectiveness.

Among indirect interventions, “publicity and education” and “behavior intervention/heat adaptation” were effective, feasible, and generalizable. The former increases awareness and understanding of how to act in response to heat wave events through TV media, radio, brochures, and other channels [57,58], while the latter focuses on improving people's health adaptability [59]. The effects of a “heat action plan” were expected, but because it is usually the overall effect of a series of intervention measures at the city scale, it is difficult to distinguish the role of a specific intervention [60]. In addition, the “early warning system” was not as effective as expected, possibly due to external factors such as failure to receive early warning information, inadequate awareness or responses to health risks, and economic constraints that could not be managed [12,16]. Despite the limitations, it is still a useful intervention, given its potential to guide the health sector in implementing targeted actions to respond to meteorological emergencies and formulate a joint multisectoral plan to effectively manage heat-related health risks [61].

In direct interventions, heating for low-temperature environments, such as floor/desk-mounted thermode apparatus, local heating, and bed heating [21,23,62], was more effective than cooling for high temperatures, probably due to more significant temperature differences. A “personal comfort system (PCS)” typically uses commercially available equipment, such as a cooling vest, cooling collar, or Embr Wave, to cope with the impact of ambient temperature [[63], [64], [65]]. Although the PCS has high feasibility and generalizability, its effectiveness was no better than that of drug intervention, heating, or behavior adaptation. “Drug intervention” generally followed a randomized controlled trial design with strict experimental conditions [66], and was scored with higher effectiveness and feasibility than other direct interventions. However, the generalizability was poor due to the limitations of RCTs to evaluate population-based interventions, including but not limited to population availability, contamination, time for follow-up, external validity, and cost [67].

To further enhance the interpretability of our evaluation, we propose a simple scoring system for reference: When a specific intervention's mean score exceeds the mean score of its corresponding type and the SD is smaller than the type's overall SD, we consider it to have reliable effects and label it as “perfect”. For example, the effectiveness mean score for green space in proactive interventions (4.476) exceeds the overall mean for upstream interventions (3.759), and the SD (1.064) is smaller than the overall SD (1.183), making it a “perfect” case. Similarly, if the mean score exceeds the total mean and the corresponding SD is smaller than the total SD, we classify it as “good”. Otherwise, it falls into the “ordinary” status.

In conclusion, among all specific interventions, “publicity/education” and “behavior intervention” were found to be the most effective, feasible, and generalizable. In contrast, “heating” and “drug intervention” as direct interventions had high effectiveness and feasibility but often lacked generalizability due to cost constraints. “Early warning system” and “personal comfort system” were promising interventions, but further research is needed to improve their effectiveness. Furthermore, in terms of effectiveness, interventions like “green spaces” and “albedo” in proactive interventions, “publicity/education”, and “behavioral interventions” in indirect interventions, as well as “heating” and “drug interventions” in direct interventions, all achieved a “perfect” status. This application demonstrates the ability of the CEFI-HEAT framework to characterize the methodologies and score the effects of interventions, which helps to understand the effectiveness, feasibility, and generalizability of various interventions.

4. Critiques of the framework

4.1. Effectiveness of the framework

The CEFI-HEAT framework provides a comprehensive evaluation framework applicable for interventions to deal with the impact of temperature under climate change, and can evaluate methodologies and effects of interventions by scoring. We defined three types of interventions: proactive, indirect, and direct, and provided four categories of indicators: classification, methods, scope, and effects. Then, we trialed the CEFI-HEAT framework by conducting an evaluation practice on intervention studies on the health effects of ambient temperature. Through this practice, our framework has responded to WHO's call for a comprehensive evaluation of interventions from the perspective of temperature effects under climate change.

Despite a comprehensive framework that has been developed and tested, there remains a scarcity of mechanistic studies investigating the negative health effects and potential side effects of interventions. This makes it difficult to obtain a more comprehensive understanding of the effects of interventions with the CEFI-HEAT framework. Moreover, it hinders the identification of actions that may increase the risk of adverse climate-related outcomes, called maladaptation [1]. In proactive interventions, “albedo” involves covering building surfaces with highly reflective materials and reflects more solar radiation, which reduces local air temperature and energy requirements for cooling [9,68,69]. However, this may lead to an increased heating energy demand in winter and does not necessarily reduce global warming [9]. The indirect intervention “heat action plan” includes a series of measures, such as early warning systems, medical care, community services, telephone calls, and door-to-door visits [15,70], resulting in decreased heat-related mortality. This can be attributed to individual adaptation, improvements in the healthcare system, and technological advances [16]. But heat plans typically comprise a range of interventions, making it difficult to ascertain the effect of each intervention separately on any reduced health impacts [71]. Among the direct interventions, “behavioral adaptation” produces physiological adaptations, including a lower resting and exercise heart rate, lower core and skin temperature, increased skin blood flow, hypervolemia, and enhanced sweat sensitivity, output, and efficiency [72]. However, it should be noted that not all behavioral changes turn out well, and these adaptations may lead to maladaptation because they are often unable to address the root causes of vulnerability [73]. Prioritizing research toward fundamental mechanisms and diagnostic biomarker discovery is crucial for the design of specific management approaches [31]. Despite several studies that have mentioned or discussed potential mechanisms of interventions, high-quality mechanistic studies are still lacking. The publication bias [74] may also limit the identification of ineffective interventions to gain a comprehensive understanding of their effects with CEFI-HEAT framework.

4.2. From theoretical research to implementation

As mentioned in the introduction, the existing comprehensive evaluations still have challenges in guiding the design and implementation of interventions, while the CEFI-HEAT framework offers a solution. The previous content has shown that the framework can provide useful information for research design, but is still insufficient to guide the implementation of interventions. We would like to expand the potential of the framework and, therefore, propose a list of recommended interventions to guide the implementation (Fig. 4).

Fig. 4.

Recommended interventions in typical scenarios. Blank, not applicable; empty star, not recommended; half star, recommended; full star, highly recommended. “Environment” in “object” refers to the environmental effect, which means that the intervention was effective on environmental temperature, humidity, or wind speed without considering the impact on human health. In “duration”, “short-term” is several days; “long-term” is several months to years.

Referring to the indicators in “methods” and “scope”, such as temperature, duration of intervention, and research object (Fig. 1A and Table S1), we configured several typical scenarios, which included different temperatures, areas, populations, and intervention durations. According to the spatiotemporal distribution, roadmap, and descriptive statistics of intervention studies in results of the application of the framework, we offered the recommendation level of each specific intervention in different scenarios, providing a reference for protective measures for policymakers, vulnerable groups, and individuals.

We acknowledge that the current version of the recommended list of interventions is not yet optimal for guiding implementation. Even though the CEFI-HEAT framework has included indicators of “feasibility” and “generalizability”, there are still many challenges in the pathways of implementation and scale-up of interventions, such as characteristics of the intervention, practical strategies, advocacy, political will, priorities, multisectoral collaboration, cost-effectiveness, human resources, social support, reflection, and evaluation [26,27,34,35,45]. However, these are beyond the scope of our framework. Given the above factors, we call upon researchers to conduct more high-quality intervention studies that can clarify that “Intervention X is efficacious for producing Y outcomes for Z population at time T in setting S” [42]. This effort aims to build a comprehensive and practical list of intervention characteristics that can assist governments and individuals in selecting strategies that are effective, feasible, and tailored to local conditions.

4.3. Inspirations and limitations of the framework

Similar to the four scales of measurement [75], the indicators of CEFI-HEAT include more than just the nominal scale involved in those existing frameworks. Additionally, it involves ordinal scales like the four levels of evidence quality in GRADE (high, moderate, low, and very low) [76]. For example, the specific scoring items for each indicator are set in order. In addition, our framework covers interval scales by assigning score values to each indicator using a scoring process, which enables quantitative evaluation based on uniform scoring of intervention studies. The idea of the scoring process can also provide a solution for evaluating interventions to deal with climate-sensitive diseases, air pollution, extreme weather events, and other hazards caused by climate change [77]. However, considering that study design characteristics were associated with overestimation of intervention effects in RCTs reporting subjectively assessed outcomes [78], we do not want to add subjectivity to the evaluation on this basis. Thus, our framework does not involve the indicators of the ratio scale and is unable to make a more precise quantitative evaluation of interventions.

Although our framework defines three types of interventions and summarizes corresponding specific interventions, health interventions addressing health effect of ambient temperature should not be limited to just these three types. WHO has summarized a range of actions to mitigate or adapt to the health impacts of climate change [5], but some were not included as they did not meet our inclusion criteria. In fact, according to the International Classification of Health Interventions (ICHI) [79], there are multifarious health interventions, many of which could potentially be used to address the health effects of ambient temperature.

The indicators of the CEFI-HEAT framework are designed to reflect the methodology of each intervention study using general indicators as much as possible. In this manner, we could extract sufficient information and reflect the features of each intervention study during the follow-up evaluation practice. However, this only captures the common features of interventions rather than their specific methods, such as types of land-use/land-cover indicators [80], the threshold of early warning systems [81], and the specific devices for heating or cooling [82,83]. Furthermore, our framework can only provide an assessment of the broader health indicators within “types of health indicators”, overlooking more specific health outcomes, such as specific causes of morbidity or mortality. Encompassing a wide array of disciplines within the included literature with general indicators might not adequately capture each discipline's unique features of study design. Moreover, due to the absence of a specific bias indicator within the effects, it was challenging to assess the reliability of the outcomes of the included intervention studies.

In the evaluation practice, our framework can only evaluate research articles, and does not include the gray literature of interventions [84]. Additionally, it can only evaluate the content of the publication report, not the systematic and robust evaluation of the intervention measures, let alone the evidence quality of the research [76]. There are also practical factors that are not taken into account, such as multilevel contextual factors relevant to program implementation throughout all stages from planning to sustainment [85]. In addition, the framework cannot encompass the entire six steps in quality intervention development (6SQuID) and only made an attempt at the sixth step, collecting sufficient evidence of effectiveness to proceed to a rigorous evaluation [86].

5. Future applications

The CEFI-HEAT, offering a comprehensive evaluation framework for interventions on the health effect of ambient temperature, has been tested to evaluate the effectiveness, feasibility, and generalizability of interventions. While we have offered a list of recommended interventions to enhance its capacity to inform the design and implementation, we also believe that its potential should not be limited to these suggestions alone.

Quantitative structure–activity relationships (QSARs) models are commonly used in medicinal chemistry [87], while technology and phenotyping tools are used to explore genotype–phenotype relationships [88]. Additionally, many epidemiological studies exist for investigating temperature–morbidity relationships [89]. Although the specific applications and methodologies may differ, the fundamental concept of establishing quantitative relationships between relevant indicators remains applicable. In this context, it may be possible to develop empirical models to understand design–effect relationships of intervention studies. Moreover, considering the presence of multiple effect modifiers, including climate change-related factors, within the included interventions, we acknowledge the need to address these confounders more explicitly. These empirical models could help quantify the impact of confounders on intervention outcomes, providing a clearer understanding of the relationship between study design and intervention effects.

Specifically, the design component covers the indicators of “methods” and “scope” in our CEFI-HEAT framework, as well as the scenarios in the recommendation list. By considering combinations of multiple parameters, this model can even analyze complex interventions involving interactions among various factors [40], including confounders that can affect the outcome of interventions. The effect component refers to the desired result, which can be measured at different levels, such as physiological health parameters, odds ratios, and effectiveness scores of the framework. Through the empirical model, it is possible to understand exactly how each factor of the study design quantitatively affects the intervention effect. Additionally, if further group modeling is carried out according to the intervention definitions and “classification” indicators in the CEFI-HEAT framework, it is even expected to obtain a dedicated empirical model for each specific intervention. In the emerging area of RE-AIM framework, it is recognized that mixed-methods, combining quantitative measures and qualitative integrated with newer predictive modeling approaches, should be used across framework components to identify explanatory processes across RE-AIM dimensions [90]. By analogy, we imagine that the CEFI-HEAT framework can upgrade the evaluation index of the ordinal scale to the ratio scale, which enables precise quantification intervention characteristics. Regression analysis or machine learning can be employed to develop the design-effect empirical model based on a large number of intervention studies.

The design-effect models are particularly useful for untested interventions, as they can reduce the need for extensive and expensive intervention experiments. Moreover, by estimating the effects of different intervention strategies, these models can guide policymakers and stakeholders in choosing the most effective and efficient interventions to address a particular problem. This helps achieve the desired effect of our CEFI-HEAT framework—optimizing resource allocation and prioritizing interventions with the highest potential for positive impact on the health effects of climate change.

Data and code availability

Intervention study score data sets generated in this perspective have been deposited at https://github.com/Carl723000/Extreme-temperature-and-health-intervention and are publicly available as of the date of publication.

CRediT authorship contribution statement

J.W., P.W., and L.H. performed the literature search, study selection, and data extraction. They had full access to all the study data, and were responsible for the integrity of the data and the accuracy of the data analysis. J.W. performed the analysis and had the primary responsibility of writing the manuscript. J.W., P.W., and K.C. interpreted the results. P.W., L.H., B.L., P.K., and K.C. conceived, designed, and supervised the study and revised the manuscript. All authors contributed to writing the manuscript and approved the final manuscript.

Declaration of competing interests

The authors declare no competing interests.

Appendix A. Supplementary data

The following is the Supplementary data to this article.