Contextual influences on the role of evidence in e-cigarette recommendations: a multi-method analysis of international and national jurisdictions

Abstract

Background

E-cigarette policy has varied across jurisdictions, contrasting with the previous coordinated approach of international tobacco control communities.

Aims and objectives

A multi-method case study approach was used to understand the role of evidence and external and internal contextual factors in the development of public health recommendations across four purposively selected jurisdictions (WHO, UK, Australia and USA).

Methods

Informed by Dobrow et al.’s (2004) conceptual framework for context-based evidencebased decision-making, four data sources were drawn upon: 1) 15 public health bodies’ e-cigarette recommendation documents, 2) seven development documents produced by the public health bodies, 3) sources of evidence cited in the public health bodies’ recommendation documents and 4) 15 qualitative interviews with experts. Thematic analysis and citation analysis were conducted to aid triangulation of evidence.

Findings

We found a complex interplay between internal and external factors which influence the role and use of evidence in the development of e-cigarette recommendations. For example, recommendation documents’ remit (internal factor) was influenced by various external factors such as epidemiology and policy history, with decisions made over time having reshaped the external context. Considering the findings with respect to evidence utilisation, we propose a modified version of Dobrow et al.’s (2004) framework, highlighting the important interplay between internal and external contextual factors.

Discussion and conclusion

This research suggest internal and external contextual factors mutually interact and influence how evidence is incorporated into recommendations. This dynamic interplay of contextual factors may help explain the why different policy approaches are pursued concerning public health topics, particularly e-cigarettes.

Background

Evidence and context are two fundamental components of evidence-based decision-making (Dobrow et al, 2004). Evidence can be defined as facts (actual or asserted) in support of a conclusion, statement or belief (Oxman et al, 2009). Context can broadly be defined as the factors that influence decision-making (Dobrow et al, 2004 ; Mirzoev et al, 2017). Dobrow et al. (2004) argue the most crucial aspect of the development of evidence-based decision-making is the interaction between evidence and context. Even when there is agreement on what constitutes evidence, research has shown that the same evidence, utilised in different contexts can lead to different decision outcomes (Walls et al, 2017).

Evidence can take a variety of forms from peer reviewed research to personal experiences and opinions (Oxman et al, 2009). However, evidence does not tell decision-makers what to do with the results (Black, 2001). Evidence may also be conflicting, limited or rapidly changing, making the development of recommendations challenging, as has been the case with e-cigarettes. Researchers and policymakers can have different perspectives about what constitutes evidence and how it should be used. Dobrow et al. (2004) distinguishes between the ‘philosophical’ and ‘practical’ aspects of evidence. The ‘philosophical-normative orientation’ focuses on the properties of evidence (e.g., validity and rigour) and introduces the claim that some forms of evidence are to be preferred over others (Djulbegovic et al, 2009; Dobrow, 2003). Therefore, what constitutes evidence is based on quality (such as consideration of the its validity and reliability) with the supposition being that higher-quality (i.e., less susceptible to bias) evidence should lead, in turn, to higher quality decisions (Dobrow et al, 2004). Initially, the evidence based medicine (EBM) movement supported the philosophical-normative orientation and emphasised the use of evidence produced through systematic and rigorous research, while de-emphasising expert judgement, unsystematic clinical experience, and patient and professional values (Evidence-Based Medicine Working Group, 1992). However, there is now wide recognition that scientific evidence alone is insufficient to make decisions (Goldenberg, 2006 ; Greenhalgh et al, 2014). In contrast, the ‘practical-operational orientation’ is context-based and “defines evidence less by its quality, and more by its relevance, applicability or generalisability to a specific context” (Dobrow et al, 2004, p.209). This perspective proposes that evidence is subjective, with different perspectives producing different explanations for the same decision outcome. It is argued that this perspective is more aligned with decision-making as it focuses on the variety of factors that contribute to an outcome and has somewhat been incorporated into EBM (Dobrow et al, 2004 ; Oxman et al, 2009). There are frameworks distinguishing between macro, meso, and micro levels of context (e.g., Evans, 2001 ; Hudson and Lowe, 2009 ; Ricketts, 2010) and others distinguishing between inner and outer context (Bate et al, 2008 ; Pye and Pettigrew, 2005). However, settling on a definitive categorisation is problematic given that, as Squires et al. (2015) note “no one framework is sufficiently inclusive or comprehensive about what comprises context.” (p.137)

Dobrow et al.’s (2004) conceptual framework for evidence-based decision-making can provide insights into the numerous contextual factors influencing the role and use of evidence in the decision-making process. Similar to Pye and Pettigrew (2005) and Bate et al. (2008), Dobrow et al. (2004) distinguish between external and internal contextual influences. The external context accounts for the environment in which a decision is applied and includes epidemiological features of the health issue being addressed, extrajudicial factors (e.g., experiences in other jurisdictions that may help inform decision-making), and political factors (e.g., ideological, social, and economic issues) (Dobrow et al, 2004). External factors play a role in decision-making with some factors being uncontrollable and according to Dobrow et al. (2004) “cannot be manipulated by decision-makers (at least in the short term” (p. 210). As such the mutability of external factors depends to some extent on where the decision-makers themselves are positioned and whether or not they have any intermediary partners who may help to align external factors in ways that support the decision/issue being addressed. The internal context refers to the environment in which a decision is made and includes factors related to the purpose of the decision-making activity, the role of participants and the processes used to arrive at decisions (Dobrow et al, 2004). Both internal and external contextual factors impact how evidence is weighed and how that evidence is utilised (Dobrow et al, 2004).

Electronic cigarettes (e-cigarettes, also known as Electronic Nicotine Delivery Systems or ENDS) offer a highly relevant case study for investigating the role and use of evidence in public health recommendations, due to the rapidly developing evidence base. Across the world, a range of different public health policy approaches towards e-cigarettes products has been pursued; from being completely prohibited to being regulated as consumer products, tobacco products, or medicinal devices (Hawkins and Ettelt, 2019).

This paper aims to identify and explore how different contextual factors influence the role of evidence in the development of e-cigarette recommendations across four jurisdictions (WHO (World Health Organization), United Kingdom (UK), Australia (AUS) and United States of America (US)) and how it may contribute to different policy approaches.

Methods

A multi-method case study approach was used, with four data sources drawn upon: 1) 15 public health bodies’ e-cigarette recommendation documents, 2) seven development documents produced by the public health bodies, 3) sources of evidence cited in the recommendation documents and 4) 15 qualitative interviews with experts working in the selected study jurisdictions

Selection of study jurisdiction

We purposively selected four different influential jurisdictions: WHO, UK, AUS, and US. These were chosen due to the variation in regulatory frameworks and their relative importance for setting the agenda on public health recommendations for e-cigarettes (Erku et al, 2020). The UK has adopted a ‘harm reduction’ approach towards e-cigarettes, arguing that e-cigarettes are likely to be less harmful than combustible cigarettes and proposing that smokers who are unable to quit should be encouraged to switch to e-cigarettes (Erku et al, 2020). The contrasting ‘precautionary’ approach adopted by WHO, AUS and US, is based on the argument that smokers should be encouraged to quit smoking and not switch to e-cigarettes (Pisinger et al, 2019). Sub-national level bodies within the UK were included in the sample to investigate diversity within a jurisdiction. The UK has four public health systems, corresponding to its four different political systems. Scotland, Wales, and Northern Ireland each have an autonomous legislature that makes health policy while the UK Government directly runs England’s National Health Service (NHS) (Greer, 2016). This, therefore, makes the UK an interesting and complex jurisdiction to examine. However, it was not feasible to include sub-national level bodies within AUS and US.

Identification of public health bodies’ recommendation documents

Within each of the chosen jurisdictions, we identified public health bodies that had produced recommendation documents, position papers, or policy statements on e-cigarettes that included recommendations for health policy and practice, A ‘public health body” was defined as an organisation whose aims stated, or whose role within local/national/international policy is to protect and improve the health of a population. Several public health bodies had been identified during an initial literature review stage and through correspondence with experts. Additional public health bodies were identified using online searching (conducted between July and August 2019). As the literature surrounding e-cigarettes is continuously evolving, another online search was conducted in December 2019 to ensure no documents had been missed from the sample. Websites of public health bodies were searched for any publicly available documents using the key terms “e-cigarettes”, “electronic cigarettes”, “e-liquids”, and “tobacco”. Citation lists within the identified documents were examined for additional relevant recommendation documents, position, or policy statements. Criteria for sample inclusion are shown in Appendix A. Through snowballing from websites, policy documents, and personal networks, a list of relevant experts within each jurisdiction was compiled. These experts were emailed with a list of the documents making up the sample and asked to provide details of any recommendation documents, positions, or policy statements they believed to be influential that were not included in the original sample (Appendix B for detailed search strategy of the sampling of public health recommendation documents).

A total of 15 recommendation documents across 10 public health bodies (two from WHO, eight from the UK, two from AUS and three from the US) met the inclusion criteria for further analysis.

Identification of public health bodies’ development documents

In addition to examining the recommendation documents, we examined their development documents or manuals (i.e., documents that detailed the processes the corresponding public health body follow to develop recommendations). Websites of the included public health bodies were searched for any publicly available documents or manuals detailing the process used to develop their recommendations. If a document or manual was not publicly available, the body was contacted via email and asked to provide any details on the process used to develop recommendations.

Of the 10 public health bodies, seven development documents were identified, and three public health bodies had no available developments documents. One of these three did provide information about the development process in personal communication.

Citation network analysis

Citation analysis measures the importance or impact of an author, an article, or a publication by counting times cited in other works, and network analysis can be used to study patterns of connections between documents, where a citation is considered a link between documents in the network (Aksnes et al, 2019 ; Lefebvre et al, 2020). The aim of the citation network analysis was to investigate the sources of evidence cited by the 15 recommendation documents when making recommendations about e-cigarettes (more methodological details in Smith et al. (2021)).

In addition to examining the sources of evidence drawn upon we conducted qualitative analysis to determine if the interpretation of the citations varied across recommendation documents. The surrounding text of when the reference was cited within the recommendation document was examined and coded. Coding was initially developed inductively using descriptive codes. Using NVivo 12 the text was firstly coded based on the topic area discussed (e.g., e-cigarettes as a smoking cessation tool) and secondly by the interpretation (e.g., may have benefits). The coding framework is in Appendix C.

Coding was an iterative process and was discussed at team meetings to help refine and adapt the framework. MJS conducted all coding and a random sample of 20% was independently double coded by KS and any disagreements were discussed and clarified.

Expert interviews

Sampling of experts

Expert interviews allow in-depth insights into the decision-making process and the role and use of evidence within it. Ethical approval was received from the University of Glasgow’s College of Medicine and Veterinary Science research committee (reference 200180098).

We used a purposive sampling approach and identified potential experts from the author and/or contributor lists of included recommendation documents. Alongside purposive sampling, snowball sampling was also used as authorship and contributorship details were not always publicly available. Snowball sampling therefore allowed both authors and contributors (who may not have been authors, but still closely involved) to be interviewed. In addition, participants were asked to recommend additional respondents from their knowledge of the field and involvement in the decision-making process. To provide an insight into the e-cigarette debate and the decision-making process, we included academics, policymakers, and methodologists (i.e., people with expertise in applying evidence to produce recommendations).

To ensure anonymity, the names and identifying details of the study participants are not reported.

Data collection

Potential participants were approached by email and provided with an information sheet. Interested participants were contacted by MJS to arrange a suitable time and mode (Skype, Zoom, telephone) for interview. Participants completed a consent form beforehand (13 provided written consent and two provided verbal consent) and interviews conducted by MJS.

Interview schedules were informed by the document analysis and Grading of Recommendations Assessment, Development and Evaluation (GRADE) Evidence to Decision (EtD) Framework (Alonso-Coello et al, 2016), to probe on public health bodies’ processes for developing recommendations, including how evidence was used in that process. Two different topic guides were tailored to interviewees: e-cigarette academics (Appendix D) and policymakers and methodologists (Appendix E). These were framed around similar key topics but probed in different areas depending on the category of experts, e.g., academics were probed more about tobacco control, and e-cigarette regulation, whereas policymakers and methodologists were questioned more on the development process.

Interviews lasted between 34 minutes and 84 minutes (median 53 minutes). All interviews were audio recorded and transcribed verbatim by a third party, subject to a confidentiality agreement. Reflective field notes were made immediately after the interview, to assist with analysis.

Triangulating across data sources

Triangulation of data was guided by Dobrow et al.’s (2004) conceptual framework which discussed three key components of decision-making: evidence, context and utilisation. The framework for context-based evidence-based decision-making discussed the three key components of decision-making and acknowledges the influence of contextual factors on the process. We selected this framework because of its explicit focus on context in potentially shaping how evidence is incorporated into policymaking which therefore aligns closely with our research focus on trying to understand variation in policymaking across contexts. Dobrow et al.’s (2004) framework was refined in 2006 to acknowledge the three layers of policy objectives: effectiveness, appropriateness, and implementation (Dobrow et al., 2006). In the refined framework, Dobrow et al. (2006) attempt to unpick the types of evidence used in decision-making. In contrast, Dobrow et al.’s (2004) framework focuses in more detail on context and the influence of contextual factors on the decision-making process which more closely aligned with our research aim and was therefore used as the starting point for our analysis (Figure 1).

Figure 1.

Conceptual framework for context-based evidence-based decision-making. Adapted from Evidence-based health policy: context and utilisation M.J Dobrow and R.E Upshur, 2004, Social Science and Medicine, 58, p.216. Copyright [2004] by the Elsevier.

Utilisation was understood to mark the critical interaction between evidence and context. Utilisation was based on a three-stage process model developed by Rich (1997), which addressed 1) introduction of evidence, how evidence is identified and brought to the decision-making table; 2) interpretation of evidence, how the internal and external validity of evidence is evaluated; and 3) application of evidence, the influence each source of evidence has on the decision outcome.

Framework analysis is a systematic approach that identifies commonalities and differences across qualitative data, to identify patterns and relationships (Spencer et al, 2003) and allows for the combined inductive and deductive coding. Data were imported into NVivo 12 and deductively coded based on Dobrow et al.’s (2004) conceptual framework with inductive codes iteratively added (Figure 2) (full coding framework in Appendix F).

Figure 2.

Coding process for integrating recommendation documents, development documents and expert interviews.

Three large data synthesis frameworks were produced in NVivo, one from each stage of evidence utilisation. MJS led the analysis, with the framework and a sample (20%) of the coding double-checked by SVK and KS. MJS, SVK and KS met to discuss and compare codes, any disagreements were discussed and clarified.

Descriptive summaries of the data were generated and allowed for cross-comparisons to be made between the data sources. It highlighted where there were similarities or disagreement around evidence use, as well as showing how contextual factors were related and addressed during decision-making.

Data availability: All reviewed documents are available on the public health bodies’ websites.

Interview transcripts cannot be shared due to confidentiality. All materials, including details about the search strategy and coding framework, are available at https://osf.io/8azex/.

Results

Drawn from the 10 public health bodies, 15 recommendation documents and seven development documents were identified through online searching of the public health bodies’ website (Table 1).

Table 1. Titles of the 15 recommendation documents, seven development documents drawn from 10 public health bodies across four policy jurisdictions.

Jurisdiction	Public health body	Recommendation document	Development document
International	World Health Organisation	Electronic nicotine delivery systems (2014a)	WHO Handbook for Guideline Development (2014b)
		Electronic Nicotine Delivery Systems and Electronic Non-Nicotine Delivery Systems (ENDS/ENNDS) (2016)
UK	National Institute for Health and Care Excellence	Stop smoking intervention and services [NG92] (2018)	Developing NIcE guidelines: the manual (2022)
	NHS Health Scotland	Smoke-free prisons and e-cigarettes (2016)	No development document available. Personal communication indicated it was an ‘in person’ development process, with no relevant formal development documentation.
		Consensus statement on e-cigarettes (2017)
	Public Health England	E-cigarettes: an evidence update (2015)	Knowledge strategy: Harnessing the power of information to improve the public’s health (2013)
		Use of e-cigarettes in public places and workplaces (2016)
		Evidence review of e-cigarettes and heated tobacco products (2018)
		Vaping in England: an evidence update (2019)
	Public Health Wales	E-cigarettes (Electronic Nicotine Delivery Systems (ENDS)) (2017)	Process for developing Position Statements for Public Health Wales (2016)
Australia	National Health and Medical Research council	National Health and Medical Research council CEO Statement: Electronic cigarettes (E-cigarettes) (2017)	Guidelines for Guidelines Handbook (2016)
	Public Health Association Australia	E-cigarettes policy position statement (2018)	No development document available. Personal communication stated that a Special Interest Group (SIG) propose a new policy position statement and draft it. National Office and the Vice President, Policy, review for content and consistency with the existing policy position statements. Draft is then made available to all Public Health Association Australia members to review and comment. Final draft approved by the Board, and formally voted on and adopted by the annual general meeting.
US	American Public Health Association	Supporting Regulation of Electronic Nicotine Delivery Systems (2018)	American Public Health Association Policy statement development process (2019)
	U.S Department of Health and Human Services	E-Cigarette Use Among Youth and Young Adults: A Report of the Surgeon General (2016)	Development process detailed in recommendation document.
	U.S. Food and Drug Administration	Deeming Tobacco Products To Be Subject to the Federal Food, Drug, and Cosmetic Act, as Amended by the Family Smoking Prevention and Tobacco Control Act; Restrictions on the Sale and Distribution of Tobacco Products and Required Warning Statements for Tobacco Products (2016)	No development document available. Personal communication indicated it was an ‘in person’ development process.

Twelve participants were recruited via purposive sampling and three participants were recruited via snowball sampling. In total, 15 interviews (eight academics, five policymakers and two methodologists) were conducted between January and June 2020. All participants authored/contributed to at least one included recommendation document and several participants authored/contributed to more than one document. Due to confidentiality, it is not possible to further breakdown participants beyond the broad sector. Eleven interviews were conducted by video call (using Skype/Zoom platforms) and four by telephone.

We begin with an overview of the results from the citation network analysis, followed by results from the document analysis of e-cigarette recommendation documents. Analysis of contextual influences on the development of recommendations is structured by internal and external context (broad themes) and is followed by identification of potential interactions across these two contexts.

A total of 1700 unique citations were included across the 15 public health recommendation documents. Many citations appeared in only one or two of the recommendation documents; 1508 (89% of 1700 citations) were cited by only one recommendation document, 139 (8.2%) by two and 53 (3.1%) across three or more. Our analysis of the sources of evidence cited found public health bodies to be introducing similar sources of evidence (Smith et al, 2021). However, this evidence was used to articulate different policy approaches, with the UK pursuing a ‘harm reduction’ approach and WHO, AUS, and US pursuing a ‘precautionary’ approach.

Although this examination highlighted an overlap in the evidence sources used, it did not indicate why recommendations and regulatory approaches diverged (Smith et al, 2021). To try and understand and explain the divergence, we examined what other factors in addition to evidence plays a role.Evidence is a key factor in the decision-making process and “should be absolutely central in policymaking” (Policymaker, UK). However, other factors need to be considered during the process as this may impact the policy or recommendations pursued within a jurisdiction.

“[Evidence] is not the whole story, it has to be contextualised to what’s going to be realistic.” (Academic, AUS)

“It is important to consider how contextual factors can modify the benefits and harms of an intervention, and how various barriers and facilitators can affect implementation and impact.” (World Health Organisation, 2014b, p.153)

External contextual factors

We identified the epidemiological features of smoking and vaping and political and institutional factors as external contextual factors influencing the development process.

Epidemiological features of smoking and vaping

Epidemiological data have been used to draw attention to the growing rate of e-cigarette use. US participants discussed that the prevalence of vaping has increased among young people, highlighting a problem, and this caused concern among policymakers and public health researchers.

“In the US context, it is really increasing the prevalence among young people. This is something that we see here, and the data (on youth prevalence) is really strong, it is increasing. It is increasing, the vaping among young people is increasing, we see not only increase in experimentation, but we see that the daily use pattern among the young people, many of those never smoked tobacco cigarettes.” (Academic, US)

In contrast, this was not being seen in the UK. While e-cigarette use among youths is being monitored, focus was on using e-cigarettes as a smoking cessation tool due to the adult smoking prevalence.

“The priority for us arguably is adult smoking cessation. Whereas in other countries including many low and middle-income countries where smoking rates are still quite low and youth prevention is a big priority for them there are going to be worried about youth and if that’s your focus then the natural response is to ban or heavily restrict the products.” (Academic, UK)

As a result, e-cigarettes are being framed in two different ways. Data and evidence relating to the epidemiology of smoking and vaping (external context factor) provide a basis for goals to be prioritised and recommendations to be developed accordingly. In turn, this will influence the remit of the recommendation document (internal context factor), subsequent interpretation of evidence and framing of policy goals. Therefore, this highlights the connection between external and internal contextual factors.

“The problem’s priority is determined by its importance and frequency (i.e., burden of disease, disease prevalence or baseline risk).” (World Health Organisation, 2014b, p.124)

Maintaining and interacting policy positions

Previous tobacco control policies are important to consider when developing recommendations, as they provide insights into what policies have been successfully implemented and what future goals are (e.g., reducing smoking prevalence within a timeframe).

“The position statement must pay due regard to current legislation and policy, outlining how the organisation’s proposed position aligns with the existing policy context.” (Public Health Wales, 2016, p.4)

In our examination of the interpretation of the 53 influential evidentiary sources, we found broadly similar interpretations of the evidence. Interpretation of evidence involves considering the validity and reliability of evidence (e.g., risk of bias) and its applicability and relevance to a particular decision (e.g., generalisability) (Dobrow et al, 2004). We found that the framing of the interpretation of the evidence was in line with tobacco control policies, particularly a jurisdiction’s ‘stance’ on e-cigarettes. For example, recommendation documents that cited McRobbie et al. (2014) broadly interpreted the evidence the same but framed it differently (WHO (2016), PHE (2015), PHE (2016) and SGR (2016)). For example, WHO (2016), which has adopted a ‘precautionary approach’, stated the study shows e-cigarettes “had a similar, although low, efficacy for quitting smoking, the overall quality of the evidence was low” (World Health Organisation, 2016, p.4). In contrast, PHE (2015; 2016) emphasised the potential of e-cigarettes.

“Recent studies support the [McRobbie et al. (2014)] Cochrane Review findings that EC [e-cigarettes] can help people to quit smoking and reduce their cigarette consumption.” (Public Health England, 2016, p.13)

The UK, AUS and US have implemented various successful tobacco control policies over the last few decades, and participants noted that new policies need to pay attention to these. Australian participants in particular spoke of the importance of these previous policies when developing new policies.

“I think in Australia, our response has been a little bit ego-driven, wanting to recognise the success of previous tobacco control measures. There’s a real sense of responsibility to not undermine those successes, like plain packaging, like the excise increases.” (Policymaker, AUS)

Although it is important to acknowledge previous and future tobacco control policies and goals, new evidence was also acknowledged as important, especially in an area such as e-cigarettes where evidence is rapidly developing.

“We’re open to the fact that new evidence might come to hand which means we have to change our minds. I think that’s the essence of good science, is the willingness to change your mind when new evidence comes to hand that suggests that what you thought yesterday is not going to apply today or tomorrow.” (Academic, AUS)

Policy transfer could impact evidence utilisation and recommendations pursued in a jurisdiction. Participants discussed how the public health body they worked with considered other jurisdictions’ e-cigarette regulations to learn about regulations they could either pursue or use as an indication of why not to pursue. For example, Australian participants discussed New Zealand’s regulatory approach and why they would not be transferring that approach.

“E-cigarettes aren’t going to solve all tobacco control problems by flooding the market with these devices, completely unregulated, which is what New Zealand has done, which is very frightening, and they’re now retroactively putting in legislation in place.” (Academic, AUS)

“In New Zealand, I think we’ve been a little more like the UK in terms of a wider, open free-ranging debate. We’ve been looking at their approach and looking at what works for us here in our country.” (Academic, AUS)¹

Participants also discussed what would happen if the regulatory position was to change. For example, if e-cigarettes were to become more readily available though still regulated, there would be potential repercussions that could not be controlled.

“That’s one of the big concerns in Australia is if you take a more liberal approach and that liberal approach includes advertising and marketing of those products to adults, then you will see an upswing in the use by youth because you cannot isolate advertising just to adults.” (Policymaker, AUS)

Peters (2016) states that the assumption is that once an institution selects a policy approach, it is likely to persist unless there are strong pressures to divert from that approach. This was discussed by one Australian participant:

“We had the pre-existing statement that was about the precautionary approach. Typically, when we go to do these things unless there’s really strong evidence of a need to change that position. We didn’t necessarily go into it with a clean slate, I’ll be honest about that. There wasn’t necessarily a discussion about, let’s only support precautionary approaches, but there was a general sense that that was the position that had been supported by the overarching and that they would need to come up with a very strong and robust argument for changing that position.” (Policymaker, AUS)

Our analysis also highlighted the asymmetric power of some public health bodies across jurisdiction, particularly PHE. Policies, reports and recommendations produced by PHE are cited by and drawn upon by other public health bodies thus they may be consider influential to other institutions e.g. McNeill et al. (2015) (which is the PHE recommendation document (Public Health England, 2015)) is cited by national and international public health bodies (WHO (2016), PHW (2017) and SGR (2016)). This was also reflected in discussions:

“A lot of people are concerned that the UK relies way too much on that original Public Health England report, about 95% [the PHE 2015 report stated that e-cigarettes were 95% less harmful than cigarettes]. I don’t think that that conclusion was worth all of the reliance it received at the time and has continued to receive.” (Policymaker, US)

Similarly, political feasibility was influential, as consideration is needed as to whether the recommendations are feasible and supported by decision-makers and officials responsible for implementation.

“The committee should also judge to what extent it will be feasible to put the recommendations into practice.” (National Institute for Health and Care Excellence, 2022, p.178)

Political feasibility and policy transfer (external factors) will shape the decision-making process by indicating what the tobacco control goals are of each jurisdiction and if the policy will be feasible and supported. This, in turn, influences the remit of the document (internal factor), evidence utilisation and policy framing, highlighting the interaction of external and internal contextual factors in decision-making.

Internal contextual factors

We refer to internal contextual factors as the remit of decision-makers, documents and participants involved in decision-making.

The remit of the document

The included recommendation documents have slightly differing remits. Some focused on the broader topic of e-cigarettes (e.g., WHO (2014a)), while others focused on narrower topics (such as e-cigarettes in the workplace). Dobrow et al. (2004) argue that the conception of evidence could be broadened or narrowed in response to the remit of the document, resulting in the introduction of different evidentiary sources. However, results from the citation network analysis highlighted that the remit of the document did not markedly alter the introduction of different evidentiary sources, as we found considerable overlap in the sources cited. For example, there is considerable overlap in the SGR (2016) document focusing on e-cigarette use among youths and young adults, and the PHE (2019) document focusing on its use in the workplace. An indication that despite different remits being pursued similar evidence sources were drawn upon.

An examination of how the remit of the document is defined, found an external organisation most frequently determines the remit. The remit of NICE documents is determined by NHS England or the Department of Health and Social (National Institute for Health and Care Excellence, 2022). Another participant discussed that the remit of documents produced by the public health body they worked with was determined by their funder.

“This [the remit] is determined by the [UK organisation] and they fund us. In the tender, they outlined the areas they wanted us to cover, for example, vulnerable populations, pregnancy, mental health, etc.” (Academic, UK)

In contrast, one participant stated that the remit of the document they worked on was stipulated by the US Government.

“The original authority from the government was only for cigarettes, smokeless tobacco, roll your own tobacco, and cigarette tobacco. […] A few years later we were asked to expand this to everything else, e-cigarettes, cigars, pipes, hookah, dissolvable products, anything that met the statutory definition of a tobacco product.” (Policymaker, US)

The remit of the document (internal factor) can therefore be influenced by political factors, highlighting the interplay between internal and external contextual factors in decision-making.

What constitutes evidence?

Analysis of the development documents showed variation in what public health bodies consider evidence Appendix G).The WHO (2014b) discussed only drawing explicitly on formal types of evidence (e.g., peer-reviewed studies) while, other public health bodies (such as NHMRC (2016) and PHE (2013)) consider both formal and informal (e.g., grey literature) types of evidence.

“Recommendations in WHO guidelines should be based on a systematic review of the scientific literature guided by specific key questions about the intervention, exposure or approach under consideration. Non-systematic reviews and low-quality systematic reviews should not inform WHO guidelines […].” (World Health Organisation, 2014b, p.93)

Participants discussed their own ideas of what constitutes evidence in the decision-making process and indicated that drawing upon published SRs or conducting a SR was considered typical. This highlighted that use of evidence is influenced by values and beliefs about different evidentiary sources.

“The standard for developing guidelines these days is to use systematic reviews.” (Methodologist, International).

Although participants discussed the importance of SRs, they also discussed drawing upon other types of evidence.

“Narrative review of the papers of the evidence I thought were more comprehensive [than a systematic review] in my opinion.” (Academic, US)

The NICE (2022) development document states that in some instances there may be a lack of evidence in relation to the topic being addressed, therefore, other types of information can be drawn upon. This point was raised by participants:

“I didn’t conduct a systematic review partly because there was so little evidence to review at all.” (Policymaker, UK)

Participants involved in the decision-making process

Our analysis shows the influence of interpersonal relationships, potential conflicts of interest (COI) (such as financial relationships or non-financial (e.g., personal, political, or religious beliefs or social relationships) and individual responsibilities can influence the use of evidence. The process for managing COI of participants involved in the development process varied across public health bodies. It was mentioned by two public health bodies (WHO (2014b) and NHMRC (2016)) and participants that to allow transparency in the decision-making process, individuals who disclose COI were not allowed to participate.

“The people who are putting the policy together aren’t in any way, shape, or form conflicted.” (Policymaker, AUS)

In contrast, one participant stated that the public health body they worked with required individuals to declare any COI; however, no further action was taken.

“[Conflicted individuals] weren’t prevented from taking part in the [development] process, they just had to declare their conflicts of interest.” (Academic, AUS)

This statement was not consistent with the COI policy of the public health body which this participant worked with.

Most commonly, individuals who declare COI can be involved in the development process but were excused from certain other stages of the process.

“It is kind of limited involvement and limited to the discussion and giving an opinion on the evidence, providing insight about the evidence but not making judgement about how to interpret this evidence and how to develop a recommendation accordingly.” (Methodologist, International)

“A conflicted individual being present but not taking part in any discussions or decision-making related to the specific area or issue.” (National Health and Medical Research Council, 2016)

Vested interests

Vested interests might be influential in the decision-making process, not only concerning the utilisation of evidence but also in relation to those permitted to participate in decision-making. Drawing on the public health bodies’ development documents, citation network analysis and interview data, showed that there were differences in the involvement of the tobacco industry in decision-making across public health bodies (with some drawing upon tobacco industry data and involving tobacco industry representatives in decision-making).

Concerning drawing upon industry-associated data, most participants stated that they “excluded tobacco industry data” (Academic, US) and that research funded by or associated with industry “should not be part of the formal literature” (Academic, AUS). Conversely, two participants stated industry associated data could be drawn upon.

“That evidence [tobacco industry evidence] isn’t just excluded automatically, we include it, but we make it very, very clear, when we’re presenting to the committee, and when we’re writing things up, which bits of evidence are related to tobacco organisation.” (Methodologist, UK)

Results from the citation network analysis suggested that some evidence influencing public health recommendation documents stems from research where important conflicts exist, such as industry (including pharmaceutical, tobacco and e-cigarette) associated COI.

Regarding the involvement of the tobacco industry in the development process there were differences across jurisdictions.

“Nobody can claim they weren’t allowed to participate in the debate, apart from the tobacco industry because they were systematically excluded from the whole process from start to finish.” (Policymaker, UK)

Another UK participant stated that the tobacco industry could not be stopped from responding to consultations and voicing their opinions; however, their responses were dealt with separately. When it comes to classifying tobacco organisations, they “aren’t officially stakeholders, they’re responders, so they are given a different status” (Methodologist, UK).

“When the comments from stakeholders are all gathered in, all those comments go straight to the commissioning team, internally. They would go through and very carefully take out all the tobacco organisation and would look at them to see whether there was anything that was important that we needed to address.” (Methodologist, UK)

Several Australian participants stated that the tobacco industry should not be involved in developing public health recommendations. In doing so, this would suppress the voice of the tobacco industry.

One policymaker explained their public health body has internal processes on “how to safeguard the development of policies from those groups [tobacco industry]” (Policymaker, AUS). These processes included the organisation “being very acutely aware of the influence that the tobacco industry can have [on the development of policies] and “not investing or having any interactions with the tobacco industry” (Policymaker, AUS).

Similarly, the WHO (2014b) stated that those involved in the decision-making process should not be conflicted.

“The majority of members of the GDG [guideline development group] should have no conflicts of interest, either financial or nonfinancial. Individuals with financial conflicts of interest should generally not be members of GDGs. This applies especially to individuals with substantial financial interests in an intervention under consideration in the guideline.” (World Health Organisation, 2014b, p.71)

However, this was not a total restriction, as the same document also stated:

“If the GDG [guideline development group] must include some members with financial and/or intellectual conflicts of interest, every effort should be made to balance the perspectives of these individuals in the group. This can be achieved by selecting people whose opinions are known to differ, including a variety of stakeholders.” (World Health Organisation, 2014b, p.71)

Although participants included e-cigarette companies when discussing the tobacco industry, they did discuss that this was not a general classification. The “[e-cigarette companies] are tobacco companies” (Academic, US); however, “the lines have gotten blurred” (Academic, US) meaning that e-cigarette industry data are drawn upon when developing policies and recommendations. Participants argued that e-cigarette and tobacco companies should be treated the same.

“You can’t really disentangle it [the vaping industry] anymore from the tobacco industry. I think we had romantic notions a few years ago that somehow the vaping industry and the tobacco industry were separate entities, and I don’t think that holds up anymore. […] At the beginning, they [tobacco companies] weren’t involved and now they are, I mean they bought them [e-cigarette companies] all up. So, they are all the same and be treated that way too.” (Academic, AUS)

While it is important to engage a variety of individuals in the decision-making process, participants highlighted that they may have their own agendas on how to frame policy goals and recommendations.

“Although we have to be aware that stakeholders might have their own kind of angles that they are trying to push.” (Methodologist, UK)

Discussion

Cross-jurisdiction comparison revealed that there is considerable divergence relating to e-cigarettes recommendations despite similar evidence drawn upon. Public health decision-making does not take place in a vacuum and our study highlights the various contextual factors influencing the decision-making process. Internal contextual factors (e.g., the remit of the participants involved) were found to influence decision-making but they did not account for the differing e-cigarette policy approaches being pursued. Analysis of the external factors suggest their importance in the framing of policy goals, including differences in epidemiology and the need to be consistent with broader political contexts. An important difference between our findings and previous frameworks includes the two-way interaction between internal and external contextual factors and interaction between different jurisdictions. Considering the findings with respect to evidence utilisation, we have amended Dobrow et al.’s (2004) conceptual framework (Figure 3).

Figure 3.

A modified conceptual framework for the influence of external and internal contextual factors in the development of e-cigarette policies and recommendations.

While modified, the basic elements of the framework are the same as in the original model (Figure 1). The main modification is acknowledging the interplay between external and internal contextual factors. For example, the remit of the participants (internal factor) is influenced by various external factors (e.g., epidemiology and policy history) and over time, decisions are made that will reshape the external context. This reflects the inherent complexity of decision-making and highlights the methodological challenge to understanding how the different elements (decision-making participants, processes and context) are intertwined. Considering the three-stage process model of evidence utilisation (i.e., introduction, interpretation and application of evidence) together with the external and internal contextual factors, our modified conceptual framework essentially creates a framework where the three elements (evidence utilisation, internal contextual factors and external contextual factors) overlap resulting in evidence-informed decision-making. Within the evidence utilisation element of the framework, we have shown the process not as sequential since evidence may be drawn upon to justify a policy position (e.g., moving from introduction to application).

Our findings suggest that although evidence is a key factor, context forms an important set of influences on evidence-based public health decision-making, which is consistent with other similar studies (such as Dobrow et al, 2004 ; Hutchinson, 2011 ; Mirzoev et al, 2017). Previous research (de Savigny et al, 2012 ; Mirzoev et al, 2017 ; Ricketts, 2010) has illustrated the interplay between factors across macro, meso and micro levels. Although we describe contextual factors as external and internal, our study highlights the complexity of decision-making by showing the variety of contextual factors influencing decision-making and the subtle ways in which context interacts to frame the focus of recommendations and the evidence chosen to underpin them. Our findings echo similar studies in not finding any dominance of contextual influences at any of the three stages of evidence utilisation (de Savigny et al, 2012 ; Mirzoev et al, 2017 ; Ricketts, 2010).

Our study has several strengths. We purposively identified different jurisdictions for investigation, systematically identified recommendation and development documents for consideration and carried out detailed qualitative coding (with double-coding of a sample). In addition, we conducted 15 in-depth expert interviews with individuals involved in developing e-cigarette recommendations (with a sample double-coding again). Another strength was the data from different perspectives generated by employing multi-methods. By combining multiple data sources, the data are woven together to promote a greater understanding of the case. Triangulation of four data sources has enriched our analysis by moving beyond demonstrating the existence of different framings of the e-cigarette policy debate, to highlighting the contextual factors influencing the decision-making process. Further, we were able to capture the interplay between internal and external contextual factors, a unique feature that was able to provide a vital understanding of the role and use of evidence in developing e-cigarette recommendations. Though, some limitations should be noted when interpreting the findings. Firstly, the small sample size of expert interviews and the limited number of each expert type due to the COVID-19 pandemic makes it difficult to understand the diversity of arguments made within these different expert groups. However, as discussed above, the use of multiple data sources has allowed for a deeper understating of the role and use of evidence in decision-making. Secondly, it is argued that when employing multi-methods it is not enough to simply compare different data from different methodological sources, without understanding data collection process of each (Flick, 2004). We analysed each data source individually, paying attention to the data collection processes, and then synthesised the data sets to analyse issues from these varying perspectives with reference to the different data collection processes. Finally, we did not include sub-national bodies within AUS and US which would have been of interest given the decentralisation of public health decision-making. While national public health guidance predominately comes from a central body (e.g., NHRMC or U.S. Health and Human Services); state public health agencies may make recommendations which may or may not be enforced by county and city public health divisions in addition to executing federal and state decisions. Our analysis did to some extent consider this multi-level policymaking landscape within the UK, but limited resources precluded us from being able to explore the role of sub-national bodies elsewhere. Our analysis might therefore provide an incomplete exploration of the complex interplay across different levels of policy in some contexts, but our UK data do highlight their likely importance.

Our study highlights several areas of research that contribute to understanding the role of evidence in public health recommendations. Our research focussed on the case study of e-cigarettes and future research could be conducted to determine if these findings are generalisable to other public health topics, such as the COVID-19 pandemic. In the absence of long-term evidence on COVID-19, global policies and recommendations focused on behavioural changes (e.g., social distancing) and vaccine uptake to reduce the number of cases and spread of the virus. Similarly, the role of context could be explored in relation to drug policy, where different jurisdictions have pursued different approaches (e.g., use of drug consumption rooms). These further case studies could help understand the contextualisation of evidence from the pandemic and drug-related policy when developing policies and recommendations.

Conclusion

A greater understanding of contextual influence is helpful in appreciating the complexity of decision-making and the fact that policymakers wrestle with myriad different factors (including evidence, politics and policy ambitions) when developing public health recommendations, especially on novel public health issues (such as e-cigarettes and COVID-19), where the evidence base is still emerging.

Key messages.

• E-cigarette public health recommendations have varied widely across countries.

• Little is known about how evidence has shaped divergent e-cigarette policies.

• Numerous contextual factors influence the role and use of evidence in e-cigarette recommendations.

• Contextual variation across countries may help explain divergent e-cigarette policies.

• Contextual factors interact in subtle ways to frame the focus of recommendations.

Funding

MS is a self-funded PhD Student and acknowledges funding from Cancer Research UK grant PPRCTAGPJT\100003. SVK acknowledges funding from an NRS Senior Clinical Fellowship (SCAF/15/02), the Medical Research Council grant MC_UU_00022/2 and the Chief Scientist Office of the Scottish Office of the Scottish Government Health Directorates grant SPHSU17. KS is funded by the Medical Research Council grant MC_UU_00022/3, the Chief Scientist Office of the Scottish Government Health Directorates grant SPHSU18, and is also supported by a Medical Research Council Strategic Award (MC_PC_13027). SH is funded by the Medical Research Council grant MC_UU_00022/1, the Chief Scientist Office of the Scottish Government Health Directorates grant SPHSU17, and Cancer Research UK grant PPRCTAGPJT\100003.