Critical appraisal of interventional clinical trials assessing heated tobacco products: a systematic review

Abstract

Objective

To critically assess the methodological characteristics and quality of interventional clinical trials investigating the effects of heated tobacco products (HTPs).

Data sources

Web of Science (Core collection and MEDLINE), Scopus, MedRxiv, ClinicalTrials.gov and ICTRP trial databases and transnational HTP manufacturer online publication libraries were searched for clinical trials on HTPs published between January 2010 and April 2022.

Study selection

Interventional clinical trials of any design, in which at least one group of adult participants used a currently marketed HTP, were selected by two reviewers with good or very good agreement.

Data extraction

Data relating to trial characteristics and effects of intervention on primary outcomes were extracted using a predesigned form. Risk of bias was assessed using Cochrane’s Risk of Bias tool v1.

Data synthesis

40 trials were included, 29 of which were tobacco industry affiliated. Methodological characteristics, such as registration, design, setting, comparator interventions, participants, outcomes and analyses, varied between trials, though there were few significant differences between industry-affiliated and independent trials. Of the 40 trials, 33 were judged to be at high risk of bias and 6 at unclear risk of bias. Trial findings were not significantly associated with either affiliation or risk of bias.

Conclusions

The conduct and reporting of HTP interventional clinical trials were poor in many respects and limited to investigating effects of short-term exposure. These trials fall short of what is needed to determine whether HTPs are beneficial to public health, meaning they may not be a sound basis for tobacco control policy decisions.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Previous research has shown industry-sponsored studies are more likely to have pro-industry results, potentially due to reduced quality and increased bias, yet the quality of interventional clinical trials on heated tobacco products (HTPs) and associations between findings on HTPs and affiliation or risk of bias have not been investigated.

WHAT THIS STUDY ADDS

• Of the 40 identified interventional clinical trials assessing HTPs, 29 were industry affiliated and 11 were independent.

• Many characteristics of these trials, such as short durations, confined settings and choice of comparators and participants, are not representative of real-world use and fail to adequately investigate whether HTPs reduce harm and are beneficial to public health.

• Trial findings on the effect of HTPs relative to cigarettes were not significantly associated with trial affiliation or overall risk of bias.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• Existing intervention clinical trials on HTPs are largely inadequate in assessing the impact of HTPs on public health and may not, therefore, be reliable in tobacco control policy decision making.

Introduction

The harms of inhaling toxicants from combusted tobacco (ie, cigarettes) are well known.¹ Heated tobacco products (HTPs) are designed to heat tobacco to relatively low temperatures. The purpose of this is to produce an inhalable nicotine aerosol which purportedly reduces the amounts of toxicants released and thus reduces health risks compared with cigarettes.² The potential to reduce health risks is fundamental to HTP marketing³ and a contributing factor in their uptake and use by consumers.^4–6 As HTP sales grow globally⁷ accurate assessment of their relative risks is essential. However, this assessment currently relies mostly on short-term laboratory research due to a lack of epidemiological studies.^{8 9}

Previous reviews have highlighted the difficulties in interpreting the existing clinical evidence. The majority of clinical research into HTPs is conducted by the tobacco industry,^{8 9} which has a history of research manipulation.¹⁰ Tobacco industry studies largely show the potential health benefits of HTPs in smokers, while some independent studies have identified potentially harmful effects⁸ and found key industry studies do not comprehensively investigate all toxicants present.¹¹ The association between a conflict of interest and industry-favourable findings has previously been observed in other tobacco and nicotine research.^{10 12} Poor or biased study design and reporting have been proposed as possible contributors to this phenomenon.¹⁰ Some methodological shortcomings have already been noted in HTP clinical research, such as short intervention durations, inconsistent reporting of data and potentially unethical practices, particularly in industry-affiliated studies.8 9 13–15 However, the quality of all HTP clinical trials has not yet been thoroughly examined.

Before consumers and policy makers make important decisions based on the results of these studies, it is crucial the quality of the evidence is assessed. Therefore, this review sought to critically appraise HTP interventional clinical trials by answering the following questions:

1. What are the methodological characteristics (ie, study details, design, interventions, participants, outcomes and analyses) and affiliations (ie, industry or independent) of interventional clinical trials on HTPs?

2. What is the risk of bias in these trials?

3. Are there differences in the methodological characteristics and risks of bias in industry-affiliated trials compared with trials with no industry affiliation?

4. What is the association between trial findings and: (a) trial risk of bias and (b) trial affiliation?

Methods

This systematic review followed recommendations set out by PRISMA.¹⁶ The protocol was registered on PROSPERO (CRD42021240676, https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021240676).

Search strategy and study selection

Web of Science (core collection and MEDLINE), Scopus, MedRxiv, ClinicalTrials.gov and the International Clinical Trials Registry Platform databases were searched on 28 April 2021. Searches were restricted to studies published from 2010 to exclude those on HTPs no longer marketed. Search terms included HTP terminology, brand names (‘IQOS’, ‘Ploom’, ‘Glo’) and clinical study terms (‘trial’, ‘participant’, ‘clinical’, ‘random*’). The online publication libraries of transnational HTP manufacturers (Philip Morris International, PMI; British American Tobacco, BAT; Japan Tobacco International, JTI; Imperial Brands, IB) and the reference lists of included literature were also searched. The full search strategy was reported in the protocol (https://www.crd.york.ac.uk/PROSPEROFILES/240676_STRATEGY_20210429.pdf). The searches were repeated to identify any relevant literature published between 28 April 2021 and 12 April 2022.

Trial publications were managed in Covidence. After duplicates were removed, title and abstract screening was piloted on 10% of the literature. Two reviewers (SB and AvdA) then independently screened all titles and abstracts, followed by full-text assessment against the eligibility criteria. Inter-rater agreement was measured using Cohen’s Kappa (k).

Inclusion criteria

Study design: Interventional clinical trials (studies in which human participants are prospectively assigned an intervention to evaluate its effects on health-related outcomes)¹⁷ of any design were included. Eligible studies did not need to be peer-reviewed or formally published.

Population: Adults (≥18 years).

Intervention: Studies were included if at least one arm was assigned a currently marketed HTP.

Comparison: Any comparator interventions.

Outcomes: Any outcomes.

Exclusion criteria

• Studies published before 2010.

• Studies that were not clinical trials.

• Observational clinical studies.

• Studies in which participants were not adults.

• Studies in which an intervention was not a currently marketed brand of HTP.

• Studies for which methodology and results data were not available, for example, ongoing studies.

Data extraction

Trial characteristic data were extracted into a predesigned form in Covidence by one reviewer (SB) and verified by a second reviewer (AvdA). The following data were extracted: study details (citation, country, trial registration date and ID, start and end dates, sponsor and affiliation); trial design (design, duration, comparators, setting); participant characteristics (eligibility criteria, age, sex, ethnicity, smoking history, comorbidities); intervention (type, cointerventions, mode of exposure); analysis (analysis population, unit of analysis, sample size calculation); outcomes (types, outcomes measured and reported, outcome matrices, time points measured) and results (participant flow, direction of effect in primary outcomes between HTP and cigarette groups at last follow-up). Two reviewers (SB and AvdA) independently coded trial affiliation. The full coding scheme is provided in online supplemental appendix 1.

Last follow-up exhaled carbon monoxide means and SD were independently extracted by two reviewers (SB and AvdA). Where SD was not reported, it was calculated as per the Cochrane Handbook for Systematic Reviews of Interventions.¹⁸ Study authors were contacted to request missing data relevant to the meta-regression analysis.

Risk of bias assessment

Risk of bias was assessed using Cochrane’s Risk of Bias tool V.1.¹⁹ The assessment consists of six domains: random sequence generation and allocation concealment (selection bias); blinding of participants and personnel (performance bias); blinding of outcome assessment (detection bias); incomplete outcome data (attrition bias) and selective outcome reporting (reporting bias).

The evident differences between HTPs and comparator interventions means special considerations had to be made when assessing risk of bias. Unblinded trials were rated at low risk of performance bias if they were randomised and used an active comparator of similar intensity (ie, also contains tobacco/nicotine and all arms receive same cointerventions, if any). Unblinded trials were rated at low risk of detection bias if the primary outcome was objectively measured. Selection bias was rated high for all non-randomised trials.

The assessment was piloted on 20% of included trials (SB) and checked by an experienced assessor (JHB). Then, two reviewers (SB and AvdA) independently assessed risk of bias in all trials, resolving disagreements through discussion. The overall risk of bias for each trial was rated as ‘low’ when there was low risk of bias in all domains, ‘unclear’ when there was unclear risk of bias in ≥1 domains or ‘high’ when there was high risk of bias in≥1 domains. Risk of bias plots and graphs were generated using RobVis.²⁰

Data synthesis and analysis

Trial characteristics data were summarised using descriptive statistics, distinguished by affiliation and tabulated where possible. Where comparisons involved two categorical variables, Fischer’s exact test was used to investigate associations between trial characteristics and affiliation. Due to inconsistent reporting and heterogeneity of available data, we could not conduct the meta-regression analyses per our protocol. Instead, we created an effect direction plot (as described in the Cochrane Handbook¹⁸ and by Boon and Thomson²¹) and used Fisher’s exact test to investigate associations between primary outcomes in each study and affiliation or risk of bias. We excluded studies with mixed effects for primary outcomes and/or were rated unclear risk of bias from these analyses.

Statistical analyses were conducted in Stata V.17. Significance level was 0.05.

Results

Included trials

A total of 987 and 214 records were identified through the first and second searches, respectively, of which 79 were included. There was good or very good agreement²² between reviewers for screening (first search: k=0.74, second search: k=0.81) and eligibility assessment (first search: k=0.64, second search: k=0.76). The 79 records related to 40 trials. Additional records pertinent to these trials (ie, registrations, protocols, reports and so on) were then collected, meaning 120 total records were included (figure 1). Key trial characteristics are provided in table 1 and full characteristics in online supplemental table 1. Two ‘actual use’ studies were identified. Typically used in pharmaceutical research, actual use studies investigate how a product is used under simulated real-world conditions.²³ Although usually observational, these two studies met our definition of an interventional clinical trial and were, therefore, included.

Figure 1.

PRISMA flow diagram for study selection. HTP, heated tobacco product.

Table 1.

Overview of included trials

Trial*	Country	Sponsor (affiliation)	Design	Interventions (brand/model)
ISRCTN1343952946 47	Italy	BAT (Industry-affiliated)	Crossover RCT	HTPs (Glo1.0, Glo1.1) cigarettes (OB), NRT (Nicorette inhaler)
ISRCTN14301360/UMIN00002498848–51	Japan	BAT (Industry-affiliated)	Parallel RCT	HTPs (Glo1.0, Glo1.0M, IQOS) cigarettes (Lucky Strike Regular, Lucky Strike Menthol), tobacco and nicotine cessation
ISRCTN8065190952 53	UK	BAT (Industry-affiliated)	Parallel RCT	HTPs (Glo1.0, unknown brand HTP) cigarettes (Lucky Strike Regular), e-cigarette (IS1.0(TT)), tobacco and nicotine cessation
ISRCTN8107576054–60	UK	BAT (Industry-affiliated)	Parallel RCT	HTPs (Glo1.1, THD2.4T20), cigarettes (OB), smoking cessation
Dalrymple et al (2022)26	Germany	BAT (Industry-affiliated)	Repeated measures	HTP (Glo), cigarettes (N491), e-cigarette (ePen 3)
Gee et al (2018)61	Japan	BAT (Industry-affiliated)	Actual use study	HTPs (Glo1.0, Glo1.0M, IQOS) cigarettes (Lucky Strike Regular, Lucky Strike Menthol)
Jones et al (2020)62	Italy	BAT (Industry-affiliated)	Actual use study	HTPs (Glo1.0, IQOS) cigarettes (Lucky Strike Regular), e-cigarettes (IS1.0(TT))
UMIN00001729763 64	Japan	JTI (Industry-affiliated)	Crossover RCT	HTP (Prototype NTVP), cigarettes (unknown brand)
UMIN00002577765–67	Japan	JTI (Industry-affiliated)	Parallel RCT	HTP (NTVP), cigarettes (OB), smoking cessation
UMIN00004153968 69	Japan	JTI (Industry-affiliated)	Parallel RCT	HTPs (Ploom TECH+, Ploom S2.0, 2 HTPs of unknown brands), cigarettes (OB), smoking cessation
ISRCTN8868243570 71	UK	JTI (Industry-affiliated)	Crossover RCT	HTP (HNB2.1), cigarettes (unknown brand)
NCT0370011272 73	New Zealand	JUUL Labs (Industry-affiliated)	Crossover RCT	HTP (IQOS), e-cigarettes (JUUL, Myblu, MarkTen Bold Classic, VUSE Solo, PHIX, NJOY Daily), cigarettes (Marlboro Red)
NCT0178068874 75	UK	PMI (Industry-affiliated)	Crossover RCT	HTP (IQOS2.1), cigarettes (OB)
NCT0178071476–78	Poland	PMI (Industry-affiliated)	Parallel RCT	HTP (IQOS2.1), cigarettes (OB)
NCT0195960779–82	Japan	PMI (Industry-affiliated)	Crossover RCT	HTP (IQOS2.2), cigarettes (OB), NRT (Nicorette gum)
NCT0195993283–89	Poland	PMI (Industry-affiliated)	Parallel RCT	HTP (IQOS2.2), cigarettes (OB), tobacco and nicotine cessation
NCT0196770679 90–94	Japan	PMI (Industry-affiliated)	Crossover RCT	HTP (IQOS2.2M), cigarettes (OB, M), NRT (Nicorette gum)
NCT0196771995–99	USA	PMI (Industry-affiliated)	Crossover RCT	HTP (IQOS2.2M), cigarettes (OB, M), NRT (Nicotrol nasal spray)
NCT01967732100–103	UK	PMI (Industry-affiliated)	Crossover RCT	HTP (IQOS2.2), cigarettes (OB), NRT (Nicotrol nasal spray)
NCT01970982104–109	Japan	PMI (Industry-affiliated)	Parallel RCT	HTP (IQOS2.2), cigarettes (OB), tobacco and nicotine cessation
NCT01970995110–115	Japan	PMI (Industry-affiliated)	Parallel RCT	HTP (IQOS2.2M), cigarettes (OB, M), smoking cessation
NCT01989156116–121	USA	PMI (Industry-affiliated)	Parallel RCT	HTP (IQOS2.2M), cigarettes (OB, M), smoking cessation
NCT02396381122–125	USA	PMI (Industry-affiliated)	Parallel RCT	HTP (IQOS2.2), cigarettes (OB)
NCT02466412126–128	Japan	PMI (Industry-affiliated)	Crossover RCT	HTP (CHTP1.1M), cigarettes (OB, M)
NCT02503254129–134	Poland	PMI (Industry-affiliated)	Parallel RCT	HTP (CHTP1.0), cigarettes (OB)
NCT02641587135–138	Poland	PMI (Industry-affiliated)	Parallel RCT	HTP (CHTP1.2), cigarettes (OB)
NCT02649556139–141	USA	PMI (Industry-affiliated)	Parallel RCT	HTP (IQOS2.2), cigarettes (OB)
NCT03364751142–145	Japan	PMI (Industry-affiliated)	Parallel RCT	HTP (IQOS), cigarettes (OB)
Caponnetto et al (2018)24	Unknown	University of Catania (Industry-affiliated)	Crossover RCT	HTPs (IQOS, Glo), cigarettes (OB)
DRKS00012919146 147	Germany	University Medical Centre Schleswig-Holstein (Independent)	Crossover RCT	HTP (IQOS2.2), cigarettes (Marlboro Gold), e-cigarettes (eGo-T with and without nicotine)
NCT03301129148 149	Italy	University of Roma La Sapienza (Independent)	Crossover RCT	HTP (IQOS2.2), cigarettes (Marlboro Gold), e-cigarette (Blu Pro)
NCT03435562150 151	USA	Virginia Commonwealth University and NIDA (Independent)	Crossover RCT	HTP (IQOS), cigarettes (OB), e-cigarette (JUUL)
NCT03452124152 153	Greece	National and Kapodistrian University of Athens (Independent)	Crossover RCT+Case Control Study	RCT: HTP (IQOS), cigarettes (Marlboro Red), sham cigarette Case Control: HTPs (IQOS), cigarettes (unknown brand)
NCT03889990/NCT03995329154–156	Greece	Aristotle University Of Thessaloniki (Independent)	Single-group assignment	HTP (IQOS)
aspredicted.org #6896157 158	Belgium	KU Leuven and Thomas More University of Applied Sciences (Independent)	Crossover RCT	HTP (IQOS), cigarettes (OB), e-cigarette (Eleaf iStick)
Iokeimidis et al (2021)159	Greece	Athens Medical School, Hippokration Hospital	Crossover RCT	HTP (IQOS), cigarettes (unknown brand), sham cigarette
Lopez et al (2016)160	USA	NIDA and CTP (Independent)	Crossover RCT	HTP (PAX), CC (OB), e-cigarette (eGo)
Nga et al (2020)161	Malaysia	International Medical University (Independent)	Quasi-experimental	HTP (IQOS), cigarettes (OB), e-cigarette (Aspire AVP)
Phillips-Waller et al (2021)162	UK	Tobacco Advisory Group project grant, Cancer Research UK	Non-randomised crossover	HTPS (IQOS), cigarettes (OB), e-cigarettes (JUUL, KangerTech EVOD, Innokin iTaste MVP 2)
Yaman et al (2021)163	Cyprus	Near East University and Mersin City Training and Research Hospital	Crossover RCT	HTP (IQOS), cigarettes (OB)

*Registration ID for registered trials. Author and date for unregistered trials.

BAT, British American Tobacco; [C]HTP, [carbon] heated tobacco product; CTP, Center for Tobacco Products, U.S. Food and Drug Administration; JTI, Japan Tobacco International; M, menthol; NIDA, National Institute on Drug Abuse; NRT, nicotine replacement therapy; NTVP, novel tobacco vapour product; OB, participant’s preferred own brand of cigarettes; PMI, Philip Morris International; RCT, randomised controlled trial.

Of the 40 trials, 11 (27.5%) had no known industry affiliation and 29 (72.5%) were industry affiliated. PMI conducted 16 trials, BAT conducted 7, JTI conducted 4 and JUUL conducted 1. The first and last authors of one study (Caponnetto, 2018)²⁴ were funded by the Foundation for a Smoke Free World between 2018 and 2019, which was established with funding from PMI.²⁵

Trial registration and reporting

Thirty-one trials (77.5%) were registered (figure 2). Only 12 (30%) were registered prior to enrolment of the first participant (ie, registered a priori). Most trials did not submit results for publication in a peer-reviewed journal (n=23, 57.5%) or post key outcome data on trial registries (n=26, 65%) within 12 months of trial completion (figure 3). Trial completion date was not reported in 12 (30%) trials; thus, timeframe for publishing results was unclear.

Figure 2.

Number of trials that were registered on a clinical trial registry (‘Registered?’) and whether they were registered prior to enrolment of the first participant (‘Registered a priori?’). Size of bar indicates percentage of trials. Number within bar indicates number of trials.

Figure 3.

Number of trials that reported results via peer-reviewed publications and posting on trial registrations within 12 months of trial completion. Size of bar indicates percentage of trials. Number within bar indicates number of trials.

There were no significant associations between affiliation and whether the trial was registered (p=0.08), whether it was registered a priori (p=0.70) or published results within 12 months of completion (p=0.07).

Trial design and setting

Thirty trials (75%; 20 industry-affiliated and 10 independent) were conducted in confined settings (ie, controlled environments, like clinics), 4 (11%; all industry-affiliated) in ambulatory settings (ie, uncontrolled environments, like participants’ homes) and 6 (15%; 5 industry-affiliated and 1 independent) in confined followed by ambulatory settings. Intervention duration ranged from single use up to 6 months. One BAT trial (ISRCTN81075760) was 12 months long, but at time of literature, collection results had only been reported for the first 6 months.

Thirty-four trials (85%; 26 industry and 8 independent) were randomised: 15 of parallel design, 18 crossover and 1 crossover followed by a case control study (table 1). The repeated measures study randomised the placement of interventions on participants’ skin, but all participants received all interventions and in the same order. Non-randomised designs included: a quasiexperimental trial, a non-randomised crossover and a study comprising two single-group assignment trials, one in which smokers used HTPs and one in which non-smokers used HTPs. In the two BAT actual use studies, products were allocated in random order within each group, but subject assignment to groups was not randomised.

There were no significant associations between affiliation and setting (confinement or ambulatory; p=0.25) or randomisation (p=0.32).

Interventions

The minimum number of intervention arms in any one trial was one and the maximum was eight. IQOS was the most common HTP intervention across both industry-affiliated (n=18) and independent trials (n=10). Excluding Caponnetto (2018), who used PMI’s IQOS, all industry-affiliated trials used the company’s own brand of HTP in at least one arm. Comparators included cigarettes, e-cigarettes, cessation, nicotine replacement therapy and non-smokers (table 1). Independent trials included an e-cigarette group significantly more often than industry-affiliated trials (p=0.0003). Only industry-affiliated trials included nicotine replacement therapies and cessation arms.

In most trials, participants used interventions ad libitum, regardless of confined or ambulatory setting. In seven confined trials, use was restricted (ie, puffing topography restricted). Three trials (9%) implemented both restricted and ad libitum use in confined settings and the mode of exposure was unclear in two (6%) trials. There was no significant association between mode of exposure and affiliation (p=0.27).

Participants

Four trials (10%; 3/29 industry-affiliated and 1/11 independent) failed to report the number of participants enrolled, randomised and/or completed. A total of 4098 participants were randomised (or enrolled in non-randomised trials) across the remaining 36 trials. A total of 3675 participants completed these trials, yielding an attrition rate of 10.3%: 10.5% across 26 industry-affiliated trials and 8.2% across 10 independent trials. Attrition was higher in ambulatory-only trials (average attrition=20%, n=3) than confinement-only trials (2.9%, n=28). Eighteen trials had withdrawals, 15 of which reported reasons for withdrawals and 3 did not.

Twenty-six (65%) trials reported baseline characteristics for the randomised/enrolled population, 8 (20%) reported them for the completed population, 5 (12.5%) reported them for analysis populations and 1 (2.5%) did not report any baseline characteristics. Based on available data, the mean age of participants was 40.1 years old and the ratio of male to female was 1.41:1 (n=4310 across 37 trials). In 35 trials, all participants were described as being in good health or without relevant morbidities. In one PMI trial, some participants had mild or moderate chronic obstructive pulmonary disease (COPD). In another PMI trial, all participants had chronic generalised periodontitis. Three trials did not report whether participants had any relevant morbidities.

Participants were smokers in all but two trials (NCT03889990/NCT03995329 and Dalrymple, 2022). Minimum eligible cigarette consumption across the trials ranged from ≥5 to ≥11 cigarettes per day and having smoked for ≥6 months to ≥10 years. One industry-affiliated and five independent trials did not define eligible smoking history.

Outcomes

A total of 214 different outcomes were measured across the 40 trials (online supplemental table 2). There was a wider variety of biomarkers of potential harm, but biomarkers of exposure were most measured (table 2). Number of outcomes measured in any one trial ranged from 1 to 71. The mean number of outcomes measured in industry-affiliated trials was 27 (mode=19, range=1–71), whereas for independent trials, it was 11 (mode=7, range=1–28). Seventeen trials (42.5%; 14/29 industry-affiliated and 3/11 independent; p=0.29) did not report results data for all outcomes measured.

Table 2.

Outcomes measured in heated tobacco product clinical trials

Outcome type	Number of outcomes		Number of trials
	Measured	Reported in ≥1 trial	Measured outcome	Reported data on outcome
Biomarker of exposure	25	25	32	28
Biomarker of exposure*	2	2	28	24
Biomarker of potential harm	125	104	21	19
Nicotine pharmacokinetics	18	16	17	16
Subjective effects (questionnaire)	20	17	28	22
Other measures	15	12	22	18
Safety profile	9	8	24	23

*Two biomarkers of exposure were also measured as biomarkers of potential harm in one trial.

Analysis characteristics

A total of 275 trials (67.5%; 22/29 industry-affiliated and 5/11 independent; p=0.12) reported sample size calculations. The unit of analysis in 39 trials was individuals and areas of skin in 1 trial.²⁶ The analysis populations used were: full analysis set (n=5, all industry-affiliated); full analysis set as exposed (n=3, all industry-affiliated); per-protocol population (n=5, all industry-affiliated); pharmacokinetic (PK) population (n=5, all industry-affiliated); per-protocol and PK populations (n=1, industry-affiliated); per-protocol and CEVal-compliant populations (n=1, industry-affiliated); not specified or unclear (n=20, 9 industry-affiliated and 11 independent). Population definitions are provided in online supplemental table 1.

Risk of bias

Thirty-four trials were judged to be at high risk of bias and for six trials risk of bias was judged to be unclear (online supplemental figure 1). Twenty-seven (93%) industry-affiliated trials were judged to be at high risk of bias and 2 (7%) unclear (figure 4A). Seven (64%) independent trials were judged to be at high risk of bias and 4 (36%) unclear (figure 4B) (significance not estimable as no low ratings). Judgement justifications are provided in online supplemental table 1.

Figure 4.

(A) Risk of bias across industry-affiliated trials. (B) Risk of bias across independent trials. Size of bar indicates percentage of trials. Number within bar indicates number of trials. BPP, blinding of participants and personnel.

The 5 trials (3/11 independent and 2/29 industry-affiliated) judged to be at high risk of selection bias were due to these being non-randomised trials, meaning there was no random sequence generation or allocation concealment. There was no significant association between affiliation and rating for random sequence generation (p=0.07), but industry-affiliated trials had a significantly higher proportion of low ratings for allocation concealment than independent trials (p=0.0065). Selection bias could not be assessed for Dalrymple 2022 as the unit of randomisation was not individuals.

Risk of performance bias (blinding of participants and personnel) was judged to be high in 25 (86%) industry-affiliated and 2 (18%) independent trials (p=0.11). The numerous high ratings were commonly due to inability to conceal visually distinctive products and the control being non-active (cigarettes). As these factors are expected in HTP clinical research, we also determined overall risk of bias excluding this domain (figure 2; ‘Overall (exc. BPP)’). While this had no effect on overall risk of bias judgements across independent trials, 3 industry-affiliated trials went from high to unclear ratings, 10 went from high to low and 1 went from unclear to low. When excluding performance bias, there was evidence that industry-affiliated trials were judged to have low risk of overall bias significantly more often than independent trials (p=0.03).

Risk of detection bias (blinding of outcome assessment) was judged to be high in 3 (10%) industry-affiliated trials and 2 (18%) independent trials (p=0.5875). In all instances, this was due to some primary outcomes being subjectively measured in combination with either the trial being open-label (ie, no blinding) or a lack of information on blinding.

Risk of reporting bias (selective reporting) was high in 12 (41%) industry-affiliated and 4 (36%) independent trials (p=1). In all trials, this was because at least one outcome measured during the trial was not reported on at all in any trial literature.

Other biases were identified in two PMI trials due to all results data being grouped by participant product use (ie, ‘full analysis set as exposed’ analysis population), not randomisation.

Association between trial findings and affiliation or risk of bias

Table 3 (and online supplemental table 3) shows whether HTPs had a positive, mixed or negative effect on each trials’ primary outcomes compared with cigarettes at last follow-up. One independent study (NCT03889990/NCT03995329) had no cigarette arm and therefore direction of effect compared with the HTP was not estimable. Most industry-affiliated trials (59%) found HTPs had positive effects on primary outcomes compared with cigarettes, while most independent trials (60%) found they had mixed or negative effects. However, there was no convincing evidence that the proportion of effect directions was different between industry-affiliated and independent trials (p>0.05). We could not investigate associations between overall risk of bias and trial findings because no studies were rated low. Overall risk of bias judgements excluding performance bias were not significantly associated with trial findings (p=0.18).

Table 3.

Summary of direction of effect in primary outcomes at last follow-up between heated tobacco product and combustible cigarette arms

Trial	Affiliation	Design	Primary outcomes	RoB (all domains)	RoB (exc. BPP)
ISRCTN13439529	Industry-affiliated	RCT	▼7	High	High
ISRCTN14301360/UMIN000024988	Industry-affiliated	RCT	▲16	High	High
ISRCTN80651909	Industry-affiliated	RCT	▲19	High	High
ISRCTN81075760	Industry-affiliated	RCT	▲1	High	High
Gee et al (2018)61	Industry-affiliated	NRT	◄►6	High	High
Jones et al (2020)62	Industry-affiliated	NRT	◄►5	High	High
ISRCTN88682435	Industry-affiliated	RCT	▼3	High	High
NCT03700112	Industry-affiliated	RCT	◄►3	High	High
NCT01780714	Industry-affiliated	RCT	▲4	High	High
NCT02466412	Industry-affiliated	RCT	▼2	High	High
NCT02503254	Industry-affiliated	RCT	▲4	High	High
NCT02641587	Industry-affiliated	RCT	▲5	High	High
NCT02649556	Industry-affiliated	RCT	◄►8	High	High
NCT03364751	Industry-affiliated	RCT	▼1	High	High
UMIN000017297	Industry-affiliated	RCT	◄►3	High	Unclear
UMIN000025777	Industry-affiliated	RCT	▲16	High	Unclear
UMIN000041539	Industry-affiliated	RCT	▲15	High	Unclear
NCT01780688	Industry-affiliated	RCT	▼2	High	Low
NCT01959607	Industry-affiliated	RCT	▲2	High	Low
NCT01959932	Industry-affiliated	RCT	▲4	High	Low
NCT01967706	Industry-affiliated	RCT	▲2	High	Low
NCT01967719	Industry-affiliated	RCT	◄►2	High	Low
NCT01967732	Industry-affiliated	RCT	▲2	High	Low
NCT01970982	Industry-affiliated	RCT	▲4	High	Low
NCT01970995	Industry-affiliated	RCT	▲5	High	Low
NCT01989156	Industry-affiliated	RCT	▲5	High	Low
NCT02396381	Industry-affiliated	RCT	◄►8	High	Low
Dalrymple et al (2022)26	Industry-affiliated	RMS	▲9	Unclear*	Low*
Caponnetto et al (2018)24	Industry-affiliated	RCT	▲1	Unclear	Unclear
NCT03889990/NCT03995329	Independent	NRT	NE†	High	High
Nga et al (2020)161	Independent	NRT	▲1	High	High
Lopez et al (2016)160	Independent	RCT	◄►6	High	High
DRKS00012919	Independent	RCT	▼1	High	High
NCT03435562	Independent	RCT	▼1	High	High
NCT03452124	Independent	RCT+CCS	▲4	High	High
Phillips-Waller et al (2021)162	Independent	NRT	▼7	High	High
aspredicted.org #6896	Independent	RCT	◄►5	Unclear	Unclear
NCT03301129	Independent	RCT	▲2	Unclear	Unclear
Ioakeimidis (2021)	Independent	RCT	▲6	Unclear	Unclear
Yaman et al (2021)163	Independent	RCT	◄►24	Unclear	Unclear

Effect direction: ▲=HTP had a positive effect compared with cigarettes; ▼=HTP had a negative effect compared with cigarettes; ◄►=mixed or conflicting effects.

Numbers next to arrows describe number of primary outcomes within each synthesis.

Trial quality: RoB (all domains)=overall risk of bias based on all domains; RoB (exc. BPP)=overall risk of bias based on all domains except blinding of participants and personnel.

*This is excluding selection bias, which could not be assessed in this study.

†Not estimable due to lack of cigarette arm.

BPP, blinding of participants and personnel; CCS, case-control study; HTP, heated tobacco product; NE, not estimable; NRT, non-randomised trial; RCT, randomised controlled trial; RMS, repeated measures study.

Despite attempting to adapt our methods, we were unable to perform the planned analysis. Nonetheless, the issues we encountered provide further insight into the quality of available data. First, there were few objectively measured outcomes which were measured in 10 or more trials (recommended minimum for meta-regression¹⁸) and measured in both industry-affiliated and independent trials. Data were also highly variable: last follow-up exhaled carbon monoxide (eCO) means ranged from 0.5 to 17.2 ppm across HTP arms and 0.8 to 25.6 ppm across cigarette arms. A possible solution to the issues of variability could have been to compare change in eCO from baseline to last follow-up, but few trials reported this. Moreover, the SD were relatively large compared with the means. This suggests the eCO data were positively skewed, as has been noted in other large population trials,^{27 28} yet most trials did not provide log-transformed eCO data.

Discussion

To our knowledge, this is the first study to critically assess the design and reporting of HTP interventional clinical trials and investigate associations between characteristics, affiliations and results. Despite worldwide use increasing,² the number of clinical trials assessing HTPs remains low, especially those conducted independently of the tobacco industry, and most HTP trials were judged to be at high risk of bias.

In contrast with existing literature demonstrating industry sponsorship is associated with proindustry findings,^{12 29} we found no significant differences between findings from industry-affiliated and independent trials. Further, a 2017 Cochrane review found risk of bias did not differ between industry and independent studies, except for domains regarding blinding, which were more often rated low in industry studies. However, we found most industry-affiliated trials were at high risk of performance bias. When this was omitted, a significantly higher proportion of low overall risk of bias ratings were observed among industry-affiliated compared with independent trials. The differences between our findings and previous reviews’ findings may be due to the smaller sample size, most trials being limited laboratory-based studies of short-term exposure and using primary outcome data rather than overall conclusions of each trial to investigate associations. Additionally, selection bias could not be assessed in 1 of the 11 studies rated at low overall risk excluding performance bias. Full study reports were available for the other 10, which provided more information than can be presented in typical trial publications, like journal articles, thus reducing the chances of unclear judgements.

We noted numerous shortcomings in the design and quality of HTP trials. First, most trials were not registered a priori and did not publish results within 12 months, as recommended by the WHO and World Medical Association’s Declaration of Helsinki.^{30 31} Second, around half the trials did not report data for all prespecified outcomes. Selective reporting compromises the validity of trials, especially if significant outcome results are reported while non-significant results are omitted.³² It is disconcerting to find safety measures and biomarkers of potential harm particularly neglected given the health impact of HTPs remains uncertain.

Third, three independent and three BAT studies did not use a randomised controlled design and three PMI trials analysed data by exposure rather than random allocation, effectively derandomising the data. Lack of or compromised randomisation may reduce validity of results by creating an imbalance in subject characteristics (ie, possible confounding factors) between groups.³³

Fourth, there were many characteristics which diminish the representativeness of the findings in real-world populations, including very short follow-up, which may not be long enough for adverse effects to manifest, and use of controlled confined settings. Many trials also used per-protocol or similar analysis populations, which exclude participants who deviated from the protocol or product assigned. In doing this, the trials can only estimate the effects of HTPs in ideal circumstances, that is, when smokers make a complete, or near-complete, switch from cigarettes. This may overestimate their true effects across real-world populations,³⁴ in which consumers may use HTPs in conjunction with cigarettes or other products.

The choice of participants and products may also not be representative of real-world settings. Most trials included healthy participants, yet 12% of UK smokers report being in ‘bad’ or ‘very bad’ health³⁵ and 15% of US smokers have COPD.³⁶ Likewise, most trials did not include a ‘next best’ comparator based on options already available to smokers looking to reduce health risks, such as e-cigarettes, smokeless tobacco and nicotine replacement therapy. Notably, only five industry-affiliated trials included an e-cigarette arm, despite all the companies except PAX, manufacturing both HTPs and e-cigarettes.³⁷ This could be to avoid directly comparing HTPs to a more established and popular competitor.

Although these short-term, confined trials can provide evidence on exposure to toxicants compared with cigarettes, they fall short of what is needed to determine whether HTPs reduce the risks of tobacco-related diseases and whether they are beneficial to public health in real-world settings. Furthermore, high risks of bias and notable weaknesses in trial conduct and reporting are concerning in regard to existing reviews by governments and health authorities, including in the USA,³⁸ UK,³⁹ Netherlands⁴⁰ and Belgium,⁴¹ on which regulatory decision have been made. While methodological limitations were noted, most did not include systematic assessments of trial quality.

Although, to our knowledge, BAT’s actual use studies have not been reviewed by regulators, similar studies by PMI have.^{38 42} However, there is limited guidance on these studies and ethical approval can be complex to obtain.^{23 43} Indeed, ethical approval was obtained from BAT’s internal Human Research Committee in Jones 2020 and there was no mention of ethical approval in Gee (2018). In the absence of clear guidance, the design and reporting of actual use studies noticeably varies and raises concerns over their consistency and ethicality in tobacco research and regulation.

Strengths and limitations

This review included more trials than previous reviews,8 9 13 44 45 likely in part due to our less restrictive eligibility criteria. Following the guidance of the Cochrane Tobacco Addiction Group, we used Risk of Bias V.1 over the newer Risk of Bias V.2 tool because the latter requires an assessment for each outcome. This may have yielded different results, but it would have been impractical to do for all the outcomes we were interested in. Heterogenic data and inconsistent reporting meant the planned meta-regression analyses could not be conducted. Instead, we used direction of effect plots, but these do not consider statistical significance, the magnitude of effects or sample size differences between studies.

Conclusion

We found HTP interventional trials to be substandard in many aspects of their design and reporting, with most being at high risk of bias. Though our analyses detected few statistically significant differences between trials of different affiliation and risk of bias, this should only be interpreted as absence of evidence, not evidence of absence. Research in this area remains relatively sparse and results may change as further studies become available. The findings of this review highlight the inadequacy of existing clinical trial data in determining the health impacts of HTPs as used in real-world markets and thus calls into question their utility in regulatory decisions.

Data availability statement

All data relevant to the study are included in the article or uploaded as supplementary information.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

Not applicable.