Modeling current practices in critical care comparative effectiveness research

Abstract

Objective: To determine whether contemporaneous practices are adequately represented in recent critical care comparative effectiveness research studies.

Design: All critical care comparative effectiveness research trials published in the New England Journal of Medicine from April 2019 to March 2020 were identified. To examine studies published in other high impact medical journals during the same period, such trials were subsequently also identified in the Journal of the American Medical Association and The Lancet. All cited sources were reviewed, and the medical literature was searched to find studies describing contemporary practices. Then, the designated control group or the comparable therapies studied were examined to determine if they represented contemporaneous critical care practices as described in the medical literature.

Results: Twenty-five of 332 randomised clinical trials published in these three journals during this 1-year period described critical care comparative effectiveness research that met our inclusion criteria. Seventeen characterised current practices before enrolment (using surveys, observational studies and guidelines) and then incorporated current practices into one or more study arm. In the other eight, usual care arms appeared insufficient. Four of these trials randomly assigned patients to one of two fixed approaches at either end of a range of usually titrated care. However, due to randomisation, different subgroups within each arm received care that was inappropriate for their specific clinical conditions. In the other four of these trials, common practices influencing treatment choice were not reflected in the trial design, despite a prior effort to characterise usual care.

Conclusion: One-third of critical care comparative effectiveness research trials published in widely read medical journals during a recent year did not include a designated control arm or comparable therapies representative of contemporary practices. Failure to incorporate contemporary practices into critical care comparative effectiveness trials appears to be a widespread design weakness.

The goal of comparative effectiveness research is to determine which commonly employed interventions are most beneficial, least harmful, and most costeffective in a real-world setting.¹ Critical care is a medical discipline that encompasses a range of technologically advanced, life-sustaining therapies. Because comparative effectiveness research commonly examines routinely used interventions, such research is widely perceived as safe. However, when it comes to studying life-sustaining therapies in critically ill patients, unappreciated deviations from usual care can undermine this assumption.2, 3, 4, 5, 6, 7, 8, 9 Comparative effectiveness research trialists may not realise that randomisation can sometimes create practice misalignments that result in unconventional and unsafe care for patient subgroups.2, 3, 4, 5, 6, 7, 8, 9 The absence of study arms representing contemporaneous practices not only compromises safety, but also the consent process and the validity of randomised comparative effectiveness research trials.2, 3, 4, 5, 6, 7, 8, 9 It is unclear how often critical care comparative effectiveness research trials fail to incorporate generally accepted usual care.

We investigated the scope of the problem by reviewing and analysing reports of critical care comparative effectiveness research published in a recent year in one widely read medical journal. Subsequently, we reviewed research published in two additional high impact journals over the same period using the same methods. We intentionally selected high impact medical journals with rigorous review processes to obtain a potentially conservative estimate, and because comparative effectiveness research findings published in such journals are influential and can produce rapid practice changes.

Methods

To obtain an up-to-date assessment, we first reviewed all comparative effectiveness research studies published during the period April 2019 to March 2020 in the New England Journal of Medicine (NEJM). Owing to our initial findings, we then reviewed similar studies that were published in two other journals known for having similarly rigorous review processes — the Journal of the American Medical Association (JAMA) and The Lancet — during the same 1-year period. We considered only randomised clinical comparative effectiveness research trials of adult or paediatric patients in critical care settings. We used the definition of comparative effectiveness research published by the Institute of Medicine: "the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care.”¹ We excluded trials that studied novel therapies or novel indications for commonly used therapies. We also excluded reports of follow-up data from trials published earlier.

We determined whether each trial had characterised usual care before trial initiation, by conducting observational studies or surveys and/or by referencing contemporaneously available data. We defined “usual care” and “unusual care” as care consistent or inconsistent with contemporary practices, respectively. Unusual care included interventions and therapies that deviated markedly from what study patients would have received routinely outside of the trial based on the following:

• •a comparison to cited sources relevant to contemporary practice provided by the investigators;
• •a comparison to relevant studies performed by the investigators; and
• •our own searches of contemporary literature documenting how patients would have been treated outside of the trial.

Initially, one of us (WNA, JW or IC-P) reviewed each published study and protocol along with references cited to support the design of the designated control or usual care arm. By comparing the care given in the trial to that described as usual care in recent surveys, guidelines, observational studies or other published data provided by the investigators, we determined whether contemporary practice was adequately represented in each study. First, data provided and cited in the manuscript and trial protocol supporting either arm or the designated control arm as representative of usual care were examined. Next, if usual routine care was not well documented and verified in the original publication or protocol, we conducted literature searches to identify observational studies, surveys, or clinical practice guidelines not cited by the investigators that described contemporaneous usual care practices at the time of enrolment. We also reviewed all references in such publications to find additional relevant sources.

Following this initial review process, three of us conducted a second review of each study to audit the accuracy of the initial reviewer’s assessment. Thus, four of us (WNA, JW, IC-P and CN) — three of whom were not involved in the preliminary decision — reviewed all the data for each trial to determine whether the initial assessment was valid. Unanimous agreement was necessary to classify a study. In the event of a disagreement, we decided that all of us should convene to review the data again and make a final determination by majority vote. This was never necessary, however, because we reached consensus in all cases. Full search queries and more details of the overall methods are available in the Online Appendix (methods).

Results

Twelve of 146 randomised clinical trials (RCTs) (8%) published in NEJM from April 2019 to March 2020 met the criteria for critical care comparative effectiveness research (Figure 1). Six used contemporaneous data to define and incorporate current practices into their trial design, while the other six did not. A summary of these studies, with information on whether contemporary practices supported the design of each trial, is provided in Table 1.10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21 For the six trials that effectively incorporated usual care, there was a total of 19 articles reporting studies characterising usual care that had been performed by the trial investigators. In addition, 66 other articles describing usual care practices, either cited by trial investigators or found in our independent search, were consistent with the usual care arm of the respective trial. Only 5 articles cited by investigators or found in our independent search did not fully support the usual care arm in the respective published trial.

Figure 1.

Flow diagram of studies from the New England Journal of Medicine that were excluded and included

CER = comparative effectiveness research. RCT = randomised controlled trial.

Table 1.

Articles documenting usual care for critical care comparative effectiveness research trials published in the New England Journal of Medicine during the period April 2019 to March 2020

		Number of articles with information on usual care
		Investigator performed		Referenced or found
Study name	Month of publication	Supports design of usual care arm in published trial	Does not support design of usual care arm in published trial	Supports design of usual care arm in published trial	Does not support design of usual care arm in published trial	Number/total (%) of articles that support design of usual care arm in published trial	Number/total (%) of articles that do not support design of usual care arm in published trial	Number of additional articles, referenced or found, that neither support nor refute design of usual care arm in published trial
Trials that inadequately incorporated usual care arm(s)
ESETT10	Nov 2019	3*	16	1†	37	4/57 (7%)	53/57 (93%)	5
ROSE11	May 2019	0‡	1§	0	3	0/4	4/4 (100%)	4
SIFT12	Oct 2019	0	1	0	2	0/3	3/3 (100%)	11
loco213	Mar 2020	0	10	0	31	0/41	41/41 (100%)	23
HYPERION14	Dec 2019	0	13	0	54	0/67	67/67 (100%)	12
NONSEDA15	Mar 2020	0	1	1¶	4	1/6 (17%)	5/6 (83%)	1
Trials that effectively incorporated usual care arm(s)
ICU-ROX16	Mar 2020	8	0	4	0	12/12 (100%)	0/12	1
SPICE III17	June 2019	3	0	1	0	4/4 (100%)	0/4	6
HUNTER18	May 2019	3	0	3	0	6/6 (100%)	0/6	0
COACT19	Apr 2019	1	0	26	1	27/28 (96%)	1/28 (4%)	0
PREVENT20	Apr 2019	1	0	28	4	29/33 (88%)	4/33 (12%)	3
ISAR-REACT 521	Oct 2019	3	0	4	0	7/7 (100%)	0/7	3

COACT = Coronary Angiography after Cardiac Arrest. ESETT = the Established Status Epilepticus Treatment Trial. HUNTER = High-Flow Use in Non-Tertiary Centres for Early Respiratory Distress. HYPERION = HYPothermie thERIque après arrêt cardiaque avec un rythme nON choquable à l’arrivée des secours [Therapeutic Hypothermia after Cardiac Arrest in Nonshockable Rhythm]. ICU-ROX = the Intensive Care Unit Randomized Trial Comparing Two Approaches to Oxygen Therapy. ISAR-REACT = Intracoronary Stenting and Antithrombotic Regimen: Rapid Early Action for Coronary Treatment. LOCO₂ = Liberal Oxygenation versus Conservative Oxygenation in Acute Respiratory Distress Syndrome. NONSEDA = non-sedation or light sedation in critically ill, mechanically ventilated patients. PETAL = Prevention and Early Treatment of Acute Lung Injury. PREVENT = Pneumatic Compression for Preventing Venous Thromboembolism. ROSE = Reevaluation of Systemic Early Neuromuscular Blockade. SIFT = the Speed of Increasing Milk Feeds Trial. SPICE = Sedation Practice in Intensive Care Evaluation.

^⁎Three reviews written by ESETT investigators described usual care practices as consistent with what was done in this trial.22, 23, 24

^†Article was a basic science study suggesting that individuals with Alpers–Hutten–Locher syndrome were at increased risk of fatal hepatotoxicity if administered valproate (ESETT excluded individuals with known or suspected metabolic diseases to avoid enrolment of these patients).²⁵

^‡The ROSE investigators describe in their protocol an internal unpublished canvasing of 35 PETAL investigators on sedation and neuromuscular blockade practices,

^§in addition to the guideline by Rhodes and colleagues referenced in the trial.²⁶

^¶Article was a survey of sedation practices used in Danish intensive care units performed in 1996 — 18 years before NONSEDA enrolment began, an era when heavy sedation was common. .²⁷

Owing to our finding that half of the comparative effectiveness research studies published in NEJM that met our inclusion criteria did not have representative usual care arms for their designated control arms or comparable therapies, we conducted the same type of analysis on studies published in two other widely read medical journals: JAMA and The Lancet. A summary of studies published in these journals is provided in Table 2.28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40 We found 13 other RCTs in these journals that met our criteria and were published during the same 1-year period — ten of 80 RCTs published in JAMA (Online Appendix, figure S1) and three of 106 RCTs published in The Lancet (Online Appendix, figure S2).28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40 Of the ten studies published in JAMA, two failed to include either a satisfactory and fully defensible usual care control arm or comparable therapies consistent with usual care for most subjects.²⁸,³² One employed a commonly used therapy in an unusual manner in the control arm.³² The other randomly assigned patients to two experimental groups at opposite ends of the usual care treatment spectrum (Table 2; Online Appendix, supplementary results, S13 and S14).²⁸ In The Lancet, all three studies incorporated usual care into the trial design.38, 39, 40

Table 2.

Summary of critical care comparative effectiveness research trials published in JAMA and The Lancet, April 2019 to March 2020^⁎

Journal	Month of publication	First author	Title	Comparison that was studied	Was care in at least one arm for most patients enrolled consistent with contemporaneous practice?
JAMA	June 2019	C Subirà	Effect of pressure support vs T-piece ventilation strategies during spontaneous breathing trials on successful extubation among patients receiving mechanical ventilation: a randomized clinical trial28	Effect of an SBT consisting of 30 minutes of pressure support ventilation (considered by authors as a “less demanding” approach) versus an SBT consisting of 2 hours of T-piece ventilation (a “more demanding” approach) on rates of successful extubation	No†
JAMA	July 2019	RG Rosa	Effect of flexible family visitation on delirium among patients in the intensive care unit: the ICU visits randomized clinical trial29	Flexible family visitation policy in the ICU versus standard restricted visitation defined by each ICU, to reduce the incidence of delirium (cluster crossover trial)	Yes
JAMA	July 2019	M Garrouste-Orgeas	Effect of an ICU diary on posttraumatic stress disorder symptoms among patients receiving mechanical ventilation: a randomized clinical trial30	Effect of an ICU diary on the occurrence of mental health consequences in patients and their families in the ICU setting versus no ICU diary	Yes
JAMA	July 2019	K Johnston	Intensive vs standard treatment of hyperglycemia and functional outcome in patients with acute ischemic stroke: the SHINE randomized clinical trial31	Efficacy of intensive treatment of hyperglycaemia versus standard care during treatment for acute ischaemic stroke	Yes
JAMA	Oct 2019	AW Thille	Effect of postextubation high-flow nasal oxygen with noninvasive ventilation vs high-flow nasal oxygen alone on reintubation among patients at high risk of extubation failure: a randomized clinical trial32	High flow nasal oxygen therapy with prophylactic non-invasive ventilation applied immediately after extubation versus with high flow nasal oxygen therapy alone, to reduce the rate of reintubation in patients at high risk of extubation failure in the ICU	No‡
JAMA	Oct 2019	J Callum	Effect of fibrinogen concentrate vs cryoprecipitate on blood component transfusion after cardiac surgery: the FIBRES randomized clinical trial33	Fibrinogen concentrate versus cryoprecipitate for treatment of bleeding related to hypofibrinogenaemia after cardiac surgery (non-inferiority trial)	Yes
JAMA	Dec 2019	PC Spinella	Effect of fresh vs standard-issue red blood cell transfusions on multiple organ dysfunction syndrome in critically ill pediatric patients: a randomized clinical trial34	Effect of transfusion of fresh red blood cells (7 or fewer days old) versus standard-issue red blood cells on clinical outcomes in critically ill children	Yes
JAMA	Dec 2019	B Guihard	Effect of rocuronium vs succinylcholine on endotracheal intubation success rate among patients undergoing out-of-hospital rapid sequence intubation: a randomized clinical trial35	Rocuronium versus succinylcholine for tracheal intubation in out-ofhospital emergency situations	Yes
JAMA	Jan 2020	PEPTIC Investigators	Effect of stress ulcer prophylaxis with proton pump inhibitors vs histamine-2 receptor blockers on in-hospital mortality among ICU patients receiving invasive mechanical ventilation: the PEPTIC randomized clinical trial36	Effect of proton pump inhibitors versus histamine-2 receptor blockers, used for stress ulcer prophylaxis, on in-hospital mortality rates	Yes
JAMA	Jan 2020	F Lamontagne	Effect of reduced exposure to vasopressors on 90-day mortality in older critically ill patients with vasodilatory hypotension: a randomized clinical trial37	Effect of reducing exposure to vasopressors through permissive hypotension (mean arterial pressure target of 60-65 mmHg), to reduce mortality at 90 days in ICU patients aged 65 years or older with vasodilatory hypotension	Yes
The Lancet	Apr 2019	MD Lyttle	Levetiracetam versus phenytoin for second-line treatment of paediatric convulsive status epilepticus (EcLiPSE): a multicentre, open-label, randomised trial38	Levetiracetam versus phenytoin as second line therapy for children aged 6 months to 18 years with status epilepticus	Yes
The Lancet	Apr 2019	SR Dalziel	Levetiracetam versus phenytoin for second-line treatment of convulsive status epilepticus in children (ConSEPT): an open-label, multicentre, randomised controlled trial39	Levetiracetam versus phenytoin as second line therapy for children aged 3 months to 16 years with status epilepticus	Yes
The Lancet	May 2019	RESTART Collaboration	Effects of antiplatelet therapy after stroke due to intracerebral haemorrhage (RESTART): a randomised, open-label trial40	Effects of starting versus avoiding antiplatelet therapy in patients with a history of intracerebral haemorrhage while on antithrombotic therapy	Yes

ICU = intensive care unit. JAMA = Journal of the American Medical Association. SBT = spontaneous breathing trial.

^*Methods used to find critical care comparative effectiveness research trials in JAMA and The Lancet and the numbers of studies found and excluded are provided in the Online Appendix (methods).

^†This trial randomly assigned intubated patients eligible for an SBT to one of two strategies at opposite ends of the usual treatment spectrum (one that was considered less demanding and one that was considered more demanding); the investigators did not document current practices at enrolling institutions. A detailed discussion on why neither arm studied in this trial fully represented usual care at the enrolling institutions is provided in the Online Appendix (results, SBT section).

^‡In this trial, the purported control arm was a usual therapy given in an unusual manner, by extubating patients at high risk of needing reintubation to high flow nasal cannula, but concurrent guidelines recommended extubating such patients to non-invasive positive pressure ventilation. A detailed discussion on why the control arm studied in this trial did not represent usual care is provided in the Online Appendix (results, HIGHWEAN section).

A more comprehensive presentation than found below in the Results of each of the 12 comparative effectiveness research trials published in NEJM and the two trials from JAMA that appeared deficient is included in the Online Appendix (results). In tables S1–S12 of the Online Appendix, we have detailed and classified the summary findings in Table 1, the individual studies that were potentially important for documenting usual care in the design of each of the trials that met our inclusion criteria as supporting or not supporting design of the usual care arm in the published trial.

Trials published in NEJM without a representative usual care arm

Trials in which patients were randomly assigned to two fixed levels of treatment

The HYPERION (Therapeutic Hypothermia after Cardiac Arrest in Nonshockable Rhythm) trial randomly assigned cardiac arrest patients resuscitated from a non-shockable rhythm to therapeutic hypothermia or normothermia control to determine whether neurological outcomes differed.¹⁴ In the therapeutic hypothermia arm, patients were cooled to a core temperature of 32.5°C–33.5°C for 24 hours, and this was followed by a rewarming period. The protocol specifies that patients with temperatures < 36.5°C at randomisation were to be warmed until their core temperature reached 36.5°C–37.5°C. However, guidelines published during trial enrolment warned that “actively or rapidly warming patients [acutely after arrest] is not suggested”.⁴¹ HYPERION also targeted a higher temperature in the normothermia control arm than contemporary guidelines recommended for clinical trials of hypothermia.41, 42 Information relevant to contemporary practice — including 13 articles authored by HYPERION investigators, and an additional 54 articles published by others, either cited in the HYPERION manuscript or protocol, or found during literature searches — did not support the notion that actively warming patients to the temperature range prescribed for the normothermia control group constituted usual care (Table 1). While actively warming patients increased temperature differences between study arms, this design potentially disadvantaged the designated control arm. Details of our analysis of this trial are provided in the Online Appendix (results and table S1).⁴³,⁴⁴ Possible harm caused by warming patients in the normothermia designated control arm could have made hypothermia appear beneficial even if it was not.⁴⁵

The LOCO₂ (Liberal Oxygenation versus Conservative Oxygenation in Acute Respiratory Distress Syndrome) trial of acute respiratory distress syndrome compared the mortality effect of targeting two ranges of arterial blood oxygen (Pao₂): high (liberal, 95–105 mmHg) versus low (conservative, 55–70 mmHg).¹³ No trial arm allowed clinicians to uniformly practice usual care — that is, titrate supplemental oxygen by reducing fractional inspired oxygen (Fio₂) concentrations to minimise exposure to toxic oxygen levels while avoiding hypoxia (arterial oxygen saturation [Spo₂] ≥ 91%).46, 47, 48, 49, 50, 51 Consequently, some patients in the conservative arm received non-toxic Fio₂ levels (0.35–0.43) but were unnecessarily maintained in a relatively hypoxic range to achieve the low Pao₂ target. For some in the liberal arm, Fio₂ was increased to potentially toxic levels (> 0.60) despite unnecessarily high blood oxygen levels (Pao₂ > 90 mmHg). We found no data supporting the idea that either arm of LOCO₂ represented contemporary practices (Table 1) in the ten articles published by LOCO₂ investigators, or in the 31 articles either published by others and referenced in the LOCO₂ manuscript or protocol or found through literature searches. This trial design compared two arms in which different subgroups received unusual care with potentially increased risks (Online Appendix, results and table S2).⁵²

NONSEDA (the Non-sedation Versus Sedation with a Daily Wake-up Trial in Critically Ill Patients Receiving Mechanical Ventilation) compared no sedation versus light sedation in intubated, mechanically ventilated patients.¹⁵ The Richmond Agitation–Sedation Scale (RASS) target for light sedation in the control arm was –2 to –3 with periodic release of sedation. However, according to the RASS scoring system, –2 to –3 represents moderate sedation.⁵³ Contemporaneous international guidelines recommended light sedation, a RASS score from –2 to –1, in mechanically ventilated patients.⁵⁴ Among six articles found or referenced in the NONSEDA manuscript or protocol that offered information relevant to usual care, only one supported the moderate sedation level targeted in the designated control arm of NONSEDA as being usual care, and this was a survey of Danish intensive care units performed 18 years before NONSEDA, when heavy sedation was common (Table 1).²⁷ More recently, sedation that is even lighter than recommended in international guidelines was commonly employed at local and regional centres that were enrolling patients into the NONSEDA trial.⁵⁵ Deeper sedation targets have been associated with adverse outcomes in mechanically ventilated patients (Online Appendix, results and table S3).56, 57, 58, 59, 60 Safety monitoring, conclusions and informed consent were potentially weakened by the designated control patients receiving deeper sedation goals (RASS scores of –2 to –3) than recommended by current guidelines or practised regionally.

SIFT (the Speed of Increasing Milk Feeds Trial) investigated whether advancing enteral feeds at faster (30 mL/kg/day) or slower (18 mL/kg/day) rates in preterm (< 32 weeks’ gestation) or very low birthweight (< 1500 g) infants would affect survival without neurodevelopmental disability in the United Kingdom.¹² In the 2017 protocol,⁶¹ SIFT referenced a recent survey of 302 neonatologists and chart review of 670 neonates performed by a co-author to determine feeding advancement practices in the UK and Canada.⁶² The survey found that usual care entailed tailoring the rate of feeding based on weeks of gestational age at birth. Furthermore, an observational study of preterm infants (23–28 weeks’ gestation) found that slower advancement to full feeds was prescribed for lower birthweight, younger gestational age, and small-forgestational age diagnosis.⁶³ Omitting this common, patient-specific practice from the trial design introduced a potential bias: in one arm, very low birthweight infants had feeds advanced faster than was commonly done; in the other arm, higher birthweight infants were advanced more slowly than was typical in common practice (Online Appendix, results and table S4). Both arms represented unusual care practices which compromised conclusions, safety monitoring and informed consent.

Trials in which usual therapies were administered in an unusual manner

In ESETT (the Established Status Epilepticus Treatment Trial), patients with benzodiazepine-refractory status epilepticus were randomly assigned to receive one of three antiepileptic drugs (AEDs) to determine which would control seizures best at 60 minutes without additional anticonvulsant medications.¹⁰ Investigators reported that, in routine practice, physicians selected AEDs based on what patients were already taking, as well as compliance history, age, comorbidities and underlying conditions.64, 65, 66 Support for and characterisation of these usual practice patterns was found in at least 16 articles authored by ESETT investigators as well as 37 other studies which were either referenced in the published ESETT manuscript or protocol or were found in our literature searches (Table 1). However, these usual practices were not incorporated into the ESETT trial. Patients were randomly assigned to receive any one of the three trial AEDs irrespective of these usual care practices. Furthermore, since the trial capped bodyweight-based dosing at 75 kg, patients weighing > 75 kg were at risk of underdosing during status epilepticus (Online Appendix, results S5).⁶⁶ Finally, unblinding the trial drug assignment before 60 minutes comprised a protocol deviation, discouraging individualised patient-based AED therapy for 60 minutes. Thus, AEDs were delivered in an unusual manner in all three arms, with potentially increased risks (Online Appendix, results and table S5).

The ROSE (Reevaluation of Systemic Early Neuromuscular Blockade) trial evaluated whether early neuromuscular blockade in moderate-to-severe acute respiratory distress syndrome would improve outcomes when compared with usual care.¹¹ Based on canvassing primary site investigators, neuromuscular blockade was restricted in the usual care arm to refractory hypoxia, and to plateau pressures above 32 cmH₂O for at least 10 minutes that persisted despite increasing sedation and decreasing positive end expiratory pressure and tidal volume. However, data from a broad survey and most observational data available at the time indicated that clinicians did not restrict use of neuromuscular blockade to these situations and instead administered them most commonly for other indications, including ventilator asynchrony (Online Appendix, results and table S6).^46,67, 68, 69

Trials published in NEJM that adequately represented usual care in one or more arms

Trials that replicated usual care as a control

ICU-ROX (the Intensive Care Unit Randomized Trial Comparing Two Approaches to Oxygen Therapy) investigated routine oxygen therapy, reducing Fio₂ to as low as 0.21 while maintaining Spo₂ at ≥ 91%, in patients with acute respiratory failure.¹⁶ Before enrolment began, eight studies were conducted by ICU-ROX investigators, documenting usual oxygen therapy in mechanically ventilated patients. The findings were consistent with conclusions from four observational studies done by others and referenced in the published trial, the protocol, and a pilot study done by the investigators (Table 1; Online Appendix, results and table S7). In routine practice, Fio₂ was titrated downward to levels that were non-toxic (0.30–0.59) if Spo₂ remained ≥ 91%.47, 48, 49, 50, 51 One arm of the trial was unrestricted usual care, and the other was a more conservative approach which decreased Fio₂ to 0.21, if Spo₂ was maintained, as usually done, at ≥ 91%. The trial maintained usual titration practices in both arms and tested whether a more conservative strategy could further limit unnecessary hyperoxia and injury.

The SPICE (Sedation Practice in Intensive Care Evaluation) III trial compared early dexmedetomidine administration to sedate all intubated, mechanically ventilated patients with usual care.¹⁷ Three observational studies published by the investigators, and another referenced in the SPICE III trial and protocol, support the design of the usual care arm employed (Table 1). A prospective observational study found that dexmedetomidine was used in only 7.6% of patients in the first 48 hours following intubation.⁷⁰ The control arm was designed to replicate that usual care by discouraging dexmedetomidine use but fully permitting dexmedetomidine if additional sedation proved necessary (Online Appendix, results and table S8).

Trials that restricted enrolment to subpopulations routinely receiving study interventions

The HUNTER (High-Flow Use in Non-Tertiary Centres for Early Respiratory Distress) trial studied whether high flow nasal cannula (HFNC) therapy in premature infants with respiratory distress was non-inferior to nasal continuous positive airway pressure (nCPAP).¹⁸ A survey (Online Appendix, results and table S9) indicated that, in infants, most health care providers used nCPAP and not HFNC therapy.⁷¹ Enrolment was limited to centres using nCPAP alone. Given this restriction, investigators did not have to determine whether other factors influenced the choice of treatment and did not need to incorporate such factors into the trial design. Focusing enrolment on the chosen centres made it possible to draw conclusions for institutions that primarily used nCPAP. Further studies could extend the results to institutions using both modalities.

The COACT (Coronary Angiography after Cardiac Arrest) trial investigated the timing of follow-up percutaneous coronary angiography after cardiac arrest with return of spontaneous circulation and no evidence of ST-segment elevation myocardial infarction (STEMI).¹⁹ Patients were randomly assigned to receive angiography immediately or have it delayed until after neurologic recovery (generally after discharge from the intensive care unit). An observational study by the investigators found that angiography timing varied based on individualised risk assessments.⁷² Twentysix articles cited in the published manuscript or protocol also support the notion that usual care involved making an individual risk assessment to guide angiography timing (Table 1). The trial excluded enrolment of patients who would have usually undergone immediate angiography, such as those with STEMIs, as well as those who would not usually undergo immediate angiography, such as patients with a non-cardiac cause of arrest. This trial preserved usual practice patterns by focusing on post-cardiac arrest patients who would require angiography but for whom timing of the procedure was uncertain (Online Appendix, results and table S10).

The ISAR-REACT (Intracoronary Stenting and Antithrombotic Regimen: Rapid Early Action for Coronary Treatment) trial compared the efficacy of two antiplatelet agents, ticagrelor and prasugrel, in patients with acute coronary syndromes for whom angiography was planned.²¹ Current guidelines recommended that all patients with non- STEMIs and unstable angina at presentation could receive ticagrelor but were only eligible to receive prasugrel if angiography was performed and angioplasty (stenting) was planned.⁷³ Three articles authored by the investigators and four studies, which were either referenced in the published manuscript or protocol or found in our independent search, support these recommendations (Table 1; Online Appendix, results and table S11). Enrolment was restricted to acute coronary syndrome patients undergoing angiography, a subpopulation who might be administered prasugrel or ticagrelor during usual care. This design allowed for head-to-head comparison while incorporating usual care into both study arms.

The PREVENT (Pneumatic Compression for Preventing Venous Thromboembolism) trial investigated whether pharmacologic venous thromboembolic prophylaxis alone versus combined with intermittent pneumatic compression would decrease the incidence of deep vein thrombosis in critically ill patients.²⁰ By performing one observational study and evaluating more than two dozen additional studies, the investigators characterised usual care and found critically ill patients received only pharmacologic prophylaxis in the overwhelming majority of these studies (Table 1; Online Appendix, results and table S12).74, 75, 76 However, some guidelines recommended dual therapy for patients at very high risk of deep vein thrombosis.77, 78, 79 Therefore, because high risk patients might also be treated with compression devices in addition to pharmacologic therapy, the study excluded any patient treated with such devices for over 24 hours, whether or not they had received pharmacologic prophylaxis. This restriction excluded most high risk patients. By defining and then enrolling a subpopulation that would have only received pharmacologic prophylaxis during usual care, the investigators maintained usual care in the control arm. An alternative, superior design might have excluded all patients who were at high risk or treated with compression.

Discussion

We found twelve comparative effectiveness research trials enrolling critically ill patients published in NEJM from April 2019 to March 2020.10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21 Half of these trials relied on contemporary published data, prevailing local or national guidelines, and/or investigator-initiated surveys and observational studies to determine contemporary practices, and designed studies that simulated usual care practices in one or both trial arms after randomisation (Table 1). These studies employed control arms that closely followed usual care16, 17 or limited enrolment to patient subpopulations so that, after randomisation, management still conformed to contemporaneous medical practices.18, 19, 20, 21 These designs created a well informed consent process, guaranteed patient safety, and ensured that the trial results could reliably advise current routine practices.

Conversely, the other six comparative effectiveness research trials did not include a representative usual care arm. Three of these six trials did not adequately characterise current practices before enrolment using published guidelines, surveys or observational studies, and did not ensure that the designated control arm or comparable therapies studied were consistent with contemporaneous practices for most patients after randomisation.13, 14, 15 These three trials compared two relatively fixed levels of treatment from the opposite ends of the usual treatment spectrum but failed to exclude patients for whom these extremes represented unusual practice.13, 14, 15 While this methodologic approach simplified the study designs and maximised the potential for outcome differences, it compromised safety monitoring and the ability of these trials to inform current practices. Protocols for the remaining three trials failed to adhere to documented practices at enrolling institutions, resulting in unusual care for many patients after randomisation.10, 11, 12 While these three trials purportedly studied existing therapies, the interventions given during the trials did not account for patient characteristics that would routinely influence treatment choices.10, 11, 12 This omission again simplified the study designs but produced results which could not reliably inform current practice and may have increased risks for trial patients. All six of these trials ultimately investigated experimental care without a well defined, designated, representative usual care control arm or comparable therapies. In addition, investigators and institutional review boards for at least two of the six trials did not appear to appreciate the experimental nature of the study arms, as the need for informed consent was waived.¹⁰,¹⁴

After finding that half of the critical care comparative effectiveness research studies published in one journal incorporated non-representative usual care in their designated control arm or the comparable therapies studied, we performed the same analysis in two additional high impact, widely read medical journals. This revealed that comparative effectiveness research lacking study arms representative of contemporaneous practices was not confined to one journal (Table 2). Despite previous descriptions and growing awareness, failure to incorporate contemporary practices remains a persistent design weakness in critical care comparative effectiveness research. But any contention that usual care cannot be characterised or implemented in critical care medicine is not supported by our findings. In more than half of the critical care comparative effectiveness research studies that we analysed, contemporary practices were meticulously characterised a priori and rigorously simulated in the design and conduct of the trial.

Our study has potential limitations. Although four authors carefully reviewed and reached unanimous consensus on the adequacy of representative arms for each trial, this approach was conducted using an open, unblinded adjudication process. A 1-year time frame, studying only three journals and focusing on critical illness, may not have been sufficient to fully characterise the scope of the problem. Nonetheless, we were able to quantify the prevalence of this design weakness in trials that included vulnerable patients and had recently been published in high impact, widely read medical journals. Despite a careful review of cited publications and the protocols of these trials, defining current usual practices can be challenging and may be subject to differences in opinion. However, to minimise bias, we evaluated studies based on direct comparisons to the published literature describing current practices. Literature searches performed with the aid of a librarian (DC) were done to further support notions of what constituted usual care. Notwithstanding, these reviews and searches could have missed relevant articles that would mitigate some of the shortcomings we identified. Furthermore, the published literature may not have accurately reflected current practices and might have wrongly suggested or overemphasised practice inconsistencies. Practices might have been rapidly evolving, making it difficult throughout enrolment to characterise practice with any certitude. We found no such examples, or cases where practice appeared random and not possible to characterise. Care may differ across institutions and regions, and vary according to specific patient characteristics that require flexible treatment options to avoid giving unusual care. Examples of how these design challenges can be successfully mitigated to adequately capture usual care are described among the trials examined here.

At least one usual care control arm that reflects up-to-date practices is widely accepted as the gold standard comparator for driving practice change. Studies lacking a usual care arm are difficult to justify because they often pose hard-to-recognise risks for critically ill research patients, for three main reasons:

• •monitoring boards cannot determine whether an intervention is harmful or beneficial as compared with usual care;
• •if usual care is altered, no firm conclusions to improve current patient management can be drawn; and
• •informed consent — which should explain usual practices and how care will differ in the context of the trial — is compromised.

Our findings suggest that better safeguards are needed for future critical care comparative effectiveness research trials. Investigators should provide data characterising usual care and specify whether one or both arms represent contemporary practices. This information could be registered with the protocol before enrolling patients. The published trial should subsequently state which arm(s) represented usual care and detail data sources used by the authors to make that determination. Manuscript reviewers might be asked to confirm whether at least one or both arm(s) of every randomised comparative effectiveness research study constitutes usual care practices. This would further incentivise investigators involved in comparative effectiveness research to characterise current usual care at enrolling institutions before designing a trial. The scientific community, including journals, should take responsibility for the safety and quality of comparative effectiveness research trials, rather than invite more external regulatory oversight. Defining comparative effectiveness research standards as a prerequisite for publication would improve trial design and increase patient safety in critical care research. It would also enhance the ability of such research to advance future patient care.