Abstract
BACKGROUND
Whole blood trauma resuscitation is conceptually appealing and increasingly used but lacks evidence. A randomized controlled trial is needed but challenging to design. A Bayesian approach might be more efficient and more interpretable than a conventional frequentist design. We report the results on an elicitation meeting to create prior probability distributions to help develop such a trial.
METHODS
In‐person expert elicitation meeting, based on Sheffield Elicitation Framework methodology. We used an interactive graphical tool to elicit the quantities of interest (24‐hour mortality and certainty required). Two rounds were conducted, with an intervening discussion of deidentified responses. Individual responses were aggregated into probability distributions.
RESULTS
Fifteen experts participated. The pooled belief was that the median 24‐hour mortality of trauma patients with hemorrhagic shock treated with component therapy (the current standard of care) was 19% (95% credible interval [CrI], 6%‐45%), and the median 24‐hour mortality of those treated with whole blood, 16% (95% CrI, 5%‐39%). The pooled prior distribution for the relative risk had a median of 0.84 (95% CrI, 0.26‐3.1), indicating that the expert group had a 64% prior belief that whole blood decreases 24‐hour mortality compared to component therapy.
CONCLUSIONS
Experts had moderately strong beliefs that whole blood reduces the 24‐hour mortality of trauma patients with hemorrhagic shock. These data will assist with the design and planning of a Bayesian trial of whole blood resuscitation, which will help to answer a key question in contemporary transfusion practice.
ABBREVIATIONS
- CrI
credible interval
- PROPPR
Pragmatic, Randomized Optimal Platelet and Plasma Ratios (trial)
- TROOP
Trauma Resuscitation With Group O Whole Blood Or Products (trial)
Bleeding is the most common cause of preventable death after injury, and blood transfusion is an essential part of trauma resuscitation. The current standard of care is the balanced administration of red blood cells, plasma, and platelets,1, 2 which essentially attempts to reconstitute whole blood. Use of whole blood from the outset, rather than reconstituting it from its components, is therefore conceptually and logistically attractive.
Whole blood was the resuscitation product of choice until the development of modern blood component separation techniques.3, 4 Compared with component therapy, whole blood offers several potential advantages, including its balanced composition, ease of administration, and longer shelf life compared with thawed plasma and platelets. Recent military experience with fresh whole blood has stimulated renewed interest in its use. Several observational studies5, 6, 7, 8, 9 suggest improved outcomes, but these types of studies are susceptible to confounding and other issues, such as inability to match.10, 11 A single, small, randomized controlled trial has confirmed the feasibility of such a study, but was not powered to detect mortality differences.12
Despite this paucity of high‐quality evidence, more than 30 US trauma centers and a smaller number of emergency medical services9 have begun to use whole blood. There have been no large‐scale multicenter clinical trials testing the effectiveness of whole blood in trauma resuscitation. We have been working on designing such a trial, the Trauma Resuscitation With Group O Whole Blood Or Products (TROOP) trial. In brief, the aims of the TROOP trial will be to evaluate the clinical and cost effectiveness of whole blood for in‐hospital trauma resuscitation of patients in hemorrhagic shock, compared to standard component therapy. The primary outcome will be 24‐hour mortality, as it is increasingly recognized that the evaluation of hemostatic interventions may benefit from the use of shorter‐term mortality outcomes.
Designing such a trial is challenging. Traditional trial designs would demand large numbers of participants. Assuming a 15% 24‐hour mortality rate with standard of care (based on the results of the Pragmatic, Randomized Optimal Platelet and Plasma Ratios [PROPPR] trial), 80% power, and an alpha of 5%, demonstrating a 2% reduction in mortality would require approximately 10,000 participants (5000 in each arm). Based on experiences with the PROPPR trial, such a study would be prohibitively expensive, and take between 10 and 15 years to complete. Even a less conservative (or more ambitious) estimate of the benefits of whole blood, say a 4% absolute reduction in 24‐hour mortality, would require around 2300 patients – nearly four times the number enrolled in the PROPPR trial.
In addition to sample size considerations, the interpretation of frequentist designs is also difficult.13, 14, 15 In contrast, Bayesian trial designs provide direct estimates of the probability of treatment benefit or harm.16 Bayesian analyses can also incorporate information from prior studies and experts' judgments, providing results that reflect the influence of existing knowledge or assumptions of effect. While a detailed description of the Bayesian statistical framework is beyond the scope of this article, the approach—as applied to a clinical trial—involves the combining of existing (or prior [the term prior probability distribution, often shortened to prior or prior distribution, refers to the mathematical representation of previously available information]) knowledge regarding the effect of the intervention (e.g., in terms of 24‐hour mortality) with new data, gathered from the trial itself (likelihood), yielding the posterior probability distribution.17, 18, 19, 20 Such beliefs may be based on an examination of the published evidence, personal experience, other sources of information, or a combination of the above. The process of quantifying such beliefs into a prior distribution is known as an elicitation.
In this study, we describe an elicitation workshop to elicit expert opinion of the efficacy of whole blood. The elicited prior distributions will be used to plan and model the proposed TROOP trial, comparing whole blood to component therapy in trauma resuscitation.
METHODS
Design
We used the Sheffield Elicitation Framework methodology, as described by O'Hagan, with some modifications.21 Whenever possible, we adhered to good practice recommendations for eliciting expert (the term expert is commonly used to denote the participants of the elicitation exercise; in the context of this report, the term refers to both participation in the elicitation, and individuals' subject matter expertise regarding whole blood) opinion20, 22, 23 including preparation of the participants for the elicitation workshop, use of an IRB‐approved elicitation protocol, provision of feedback to experts, and an opportunity to revise elicited responses.20 There are a large number of techniques available for eliciting information.24, 25, 26 None have been shown to be superior, but the “roulette method” (also called “bins and chips”) may be the most intuitive for clinicians.27
Setting
The elicitation meeting was conducted at the Houston Marriott Airport Hotel, Texas, on May 7, 2019.
Participants
We invited civilian and military clinicians and researchers with experience in whole blood resuscitation from US trauma centers that currently use whole blood. We reasoned that these individuals would have both knowledge of the published evidence for using whole blood and personal experience. Participants were paid travel and accommodation expenses, and a small honorarium. Eleven trauma surgeons, two transfusion medicine specialists, one pediatric intensivist, and one nonclinical scientist, representing 15 trauma centers currently using whole blood, participated. As transfusion practices and the availability of whole blood differ between countries, we restricted the participants to those working in the United States.
Quantities of interest
The quantities of interest chosen to inform the design of the proposed TROOP trial were 1) the 24‐hour mortality rate for trauma patients in hemorrhagic shock who receive component therapy (the current standard of care); 2) the 24‐hour mortality rate for trauma patients in hemorrhagic shock who receive whole blood; and 3) the probability of benefit experts would need to consider using whole blood for trauma patients in hemorrhagic shock, assuming no observed differences in rates of adverse events or cost (i.e., all other things being equal).
Online elicitation tool
We developed an interactive online graphical tool based on previous work by Mason et al.,25 using R software and the Shiny package (https://mms-ccrebm.shinyapps.io/TROOPOnline/).28, 29 Users first entered a subjective probability interval to describe a plausible range for the 24‐hour mortality rate. Participants were then asked to provide their “best guess” or “most likely” estimate. These quantities were encoded using a beta probability distribution, which is bounded by [0,1], as probabilities are, where the plausible range represents a 95% probability interval and the best estimate of the mode. The interactive tool then displayed a graphical representation of the corresponding beta distribution (highlighting the 95% probability interval) to give real‐time feedback to the participants and allow them to change their estimates until the distribution matched their opinion. When satisfied with their estimates, participants proceeded to the next component of the elicitation.
Information provided prior to workshop
Before the workshop, we provided the participants with an overview of the elicitation process and the concept of subjective probabilities (Fig. S1, available as supporting information in the online version of this paper). We also provided a simple example (based on Grigore et al.30) to familiarize them with the online tool. Finally, we included examples of Bayesian trials and analyses using informative priors.31, 32 Two weeks before the elicitation workshop, we sent an evidence dossier to participants. The dossier consisted of a list of relevant published studies and conference abstracts on whole blood and component therapy (Fig. S2, available as supporting information in the online version of this paper.). We also asked participants to provide us with any other published or unpublished studies or abstracts of which they were aware.
Workshop program
The workshop was split into seven parts:
Presentation of background information on the rationale for the planned TROOP trial of whole blood versus component therapy. We did not present evidence relating to component therapy or whole blood resuscitation at this point to avoid bias by “anchoring” the participants.
Introduction to Bayesian principles focusing on the distinction between probability under frequentist and Bayesian paradigms. We also emphasized that Bayesian probability represents the subjective level of uncertainty of an event happening and can vary among individuals.
Introduction to elicitation. We then detailed the parameters that we wanted the participants to provide: the lower and upper bound and most likely value of 24‐hour mortality in patients in hemorrhagic shock treated with either component therapy (standard of care) or whole blood.
Elicitation training exercise. As an example, we used the number of squirrels in Central Park, New York. We worked through this example with the participants, who used an online tool similar to the one used for the actual elicitation, to increase familiarity with the process.
Elicitation—Round 1. Participants' beliefs for the parameters of interest were elicited using the online elicitation tool. Figure S1, available as supporting information in the online version of this paper, shows a screenshot of the questions posed to participants. We clarified any and all questions about the statistical terms that participants had during the elicitation exercise but refrained from providing any numbers or information pertaining to the parameters of interest. We calculated prior distributions for each participant's elicited beliefs, and then graphed and presented deidentified individual responses.
Group discussion. Participants were then encouraged to discuss their choices. We emphasized that the purpose of the discussion was not to come to a consensus but rather to calibrate individual opinions, and to resolve any questions relating to process.
Elicitation—Round 2. The second round was designed to allow participants to revise and calibrate their beliefs, and therefore used the same questions as the first. Participants were provided with an individual code, to allow first‐ and second‐round responses to be compared. The results were, once again, presented as deidentified individual responses.
The elicitation was supported by three biostatisticians, who explained the concepts and were available throughout the day to answer questions.
Analysis
We used mathematical pooling to aggregate the individual elicited distributions. First, we evaluated participants' elicited priors for the 24‐hour mortality rate with use of component therapy. We used equal‐weighting linear pooling of the elicited priors to calculate the clinical prior. This method can help mitigate overconfidence and overoptimism,33 and when there is extensive overlap of all the individual priors, average pooling is an efficient method to derive a group prior. We also calculated implied prior distributions for relative risks from each individual's elicited priors as well as the pooled clinical priors for the two groups.
Permissions and approvals
This study was approved by the University of Alabama at Birmingham (UAB) Institutional Review Board. Participants were advised of this status at the beginning of the elicitation meeting.
RESULTS
Round 1
Of the 15 participants, 13 submitted responses to all questions, one participant submitted responses only for the questions about component therapy, and one participant did not submit any responses. (Nonresponses are thought to have been due to lack of understanding of how to use the app.) The individual priors for the 24‐hour mortality of patients with hemorrhagic shock treated with component therapy (the current standard of care) are shown in Fig. 1A. There were a wide range of responses, in terms of both the “best guess” measure of central tendency and the credible interval. Some of the distributions were skewed. The pooled distributions are shown in Fig. 1B. The pooled belief was that the median 24‐hour mortality of trauma patients with hemorrhagic shock treated with component therapy was 22%, with a 95% credible interval (CrI) of 7% to 74%. (Table 1). The individual priors for the 24‐hour mortality of trauma patients with hemorrhagic shock treated with whole blood are shown in Fig. 1C. Again, there was a wide range of responses, in terms of both central tendency and degree of confidence. The pooled belief was that the median 24‐hour mortality of trauma patients with hemorrhagic shock treated with whole blood was 19%, with a 95% CrI of 5% to 65% (Table 1).
Figure 1.
Individual and pooled beliefs regarding 24‐hour mortality of trauma patients with hemorrhagic shock, after first round of elicitation.
Table 1.
Characteristics of pooled prior probability distributions of 24‐hour mortality of trauma patients with hemorrhagic shock treated with (a) component therapy (standard of care) and (b) whole blood
| Round 1 | Round 2 | |||
|---|---|---|---|---|
| Median | 95% CrI | Median | 95% CrI | |
| Component therapy | 22% | 7%‐74% | 19% | 6%‐45% |
| Whole blood | 19% | 5%‐65% | 16% | 5%‐39% |
CrI, credible interval.
Most participants indicated that they would want to be 50% to 90% certain that whole blood treatment was beneficial, assuming no observed differences in rates of adverse events or cost (i.e., all other things being equal) between whole blood and component therapy, to continue to use whole blood for trauma resuscitation (Fig. 2). Following presentation of the results of the first round to the group, participants were then encouraged to discuss their choices before progressing to round 2.
Figure 2.
Level of certainty required that whole blood treatment is beneficial to continue to use whole blood for trauma resuscitation.
Round 2
All 15 participants submitted responses in this round. Overall, there was considerable convergence of the distributions. The individual priors for the 24‐hour mortality of patients with hemorrhagic shock treated with component therapy (the current standard of care) are shown in Fig. 3A, and the pooled responses in Fig. 3B. The pooled belief, after this second round, was that the median 24‐hour mortality of trauma patients with hemorrhagic shock treated with component therapy was lower than in Round 1 at 19%, with a 95% CrI of 6% to 45% (Table 1). The corresponding responses for trauma patients with hemorrhagic shock treated with whole blood are shown in Fig. 3C and the pooled responses in Fig. 3D. The pooled belief after this second round was that the median 24‐hour mortality of trauma patients with hemorrhagic shock treated with whole blood was also lower at 16%, with a 95% CrI of 5% to 39% (Table 1).
Figure 3.
Individual and pooled beliefs regarding 24‐hour mortality of trauma patients with hemorrhagic shock after second round of elicitation.
The pooled prior distribution for the relative risk had a median of 0.84 and 95% CrI of 0.26 to 3.1 (Fig. 4) with the participants as a group having a 64% prior belief that whole blood will decrease 24‐hour mortality compared to component therapy.
Figure 4.
Pooled prior distribution for the relative risk, linear (A) and log scale (B).
After this second round, participants indicated that they would want to be 50% to 60% certain that whole blood treatment was beneficial, assuming no observed differences in rates of adverse events or cost (i.e., all other things being equal) between whole blood and component therapy, to continue to use whole blood for trauma resuscitation.
Changes between elicitation rounds
Figure S2, available as supporting information in the online version of this paper, shows how participants' responses changed from Round 1 to Round 2. Figure S2A, available as supporting information in the online version of this paper, shows participants' estimates of the measure of central tendency for trauma patients undergoing resuscitation with component therapy. Figure S2B, available as supporting information in the online version of this paper, shows the width of the corresponding credible interval. Figure S2C and D, available as supporting information in the online version of this paper, show the evolution of responses for whole blood. The graphs again demonstrate a convergence of beliefs and increased confidence. They also demonstrate how outliers (very high or low mortality) tend to revise their estimates as a result of the discussion.
DISCUSSION
The ability to incorporate prior knowledge and beliefs into the design of trials is one of the greatest strengths of Bayesian methodology. Prior information is often available, and prior beliefs almost always exist. The inclusion of prior information can help increase trial efficiency and enable trials that would be unfeasible under a frequentist framework.
The results of our elicitation exercise thus provide useful data on experts' current beliefs regarding the use of whole blood for trauma resuscitation. The participants' pooled estimate of 24‐hour mortality of patients treated with component therapy (19%) was somewhat higher than the 24‐hour mortality observed in patients enrolled in the PROPPR trial (15%), who were treated with a 1:1:1 transfusion strategy (now the standard of care). We did not explore why the participants' estimate was higher than that reported in the literature.
However, the inclusion of informative priors can also affect the internal validity of studies. If high‐quality meta‐analyses or randomized controlled trials are available, such priors are rarely contentious. Unfortunately, high‐quality data often do not exist. Previous studies may have been mostly observational, not generalizable, or have investigated different outcomes to the ones that the trial planners would like to use. All of these caveats apply to the available literature on whole blood. Specifically, a significant proportion of the data on whole blood stems from its use as a freshly collected product to treat military trauma victims. However, clinicians and researchers will often still have beliefs, extrapolated from their reading of the evidence and personal experience, similar to how they practice medicine. Rather than discount this knowledge, elicitation attempts to structure and quantify it, thereby converting that knowledge into a mathematical form that can be used for trial planning and analysis.
As “early adopters,” the participants, overall, expectedly had a positive view of the effects of whole blood. The pooled belief was that the use of whole blood resuscitation reduces 24‐hour mortality of trauma patients with hemorrhagic shock to 16%. This elicited absolute risk reduction of 3% is relatively small in comparison to the assumed effect sizes that have been used to design similar trials, despite the participant group being self‐proclaimed “enthusiasts.” It therefore has high face validity and represents a realistic target for a trial. It also highlights the need for an innovative approach, as a frequentist trial design based on a mortality reduction from 19% to 16% would require approximately 5000 patients. We anticipate that a Bayesian design with informative priors, as we have developed here, will require considerably fewer patients, but determination of the precise number will require additional work (which is in progress).
Our expert panel furthermore indicated that a probability of 50% to 60% of whole blood being superior to component therapy would be sufficient for them to continue (or start) to use whole blood (with 50% indicating an equal chance of whole blood or component therapy being better). This probability was, again, relatively low. This makes sense given that the outcome in question was mortality (“even a small probability that whole blood is better than component therapy is good enough for me to use whole blood, especially given the logistical superiority”). It also provides additional justification for the use of a Bayesian trial design, which permits the reporting of actual probabilities of an intervention being superior, rather than dichotomizing results by means of an arbitrarily chosen p value.
Having two elicitation rounds, with an intervening face‐to‐face discussion, proved useful. The discussion helped to bring out issues, both relating to the methodology of the elicitation, the baseline 24‐hour mortality of trauma patients with hemorrhagic shock resuscitated with component therapy, and the perceived benefits of whole blood. This finding correlates with previous studies.27 The discussions also brought to light a number of other important issues, such as leukoreduction, and the number of units of whole blood available in each site, which we will consider in the design of the trial. These were not formally recorded or analyzed but included questions about whether children should be included, whether whole blood should be leukoreduced, what antibody titer is acceptable, and whether O‐positive whole blood can or should be given to women of childbearing age. Similarly, the importance of other outcomes—including safety, cost effectiveness, and that the administration of whole blood, especially in pressured clinical settings, is much easier than trying to maintain fixed transfusion ratios—was emphasized.
We also discussed a number of methodological issues, including that of participant selection. We had invited clinicians and researchers from trauma centers currently using whole blood, both for their knowledge of the literature, current outcomes in transfused trauma patients, and their experience of using whole blood. As expected, these participants had generally favorable views of whole blood, and the elicited priors should be regarded as being “enthusiastic.” When planning and analyzing Bayesian trials, it is often helpful to consider a range of priors, and the TROOP trial will incorporate not only the prior elicited here but also a neutral prior that assumes a 50‐50 chance of whole blood decreasing 24‐hour mortality and a more skeptical prior, as well as different weightings of these priors, to ensure that the entire spectrum of possibilities has been accounted for.
CONCLUSION
This was, to our knowledge, the first elicitation related to the design of a Bayesian trauma trial. None of the participants had previously participated in an elicitation meeting, and most had had very little exposure to Bayesian trial design. Overall, the feedback that we received was positive, and the exercise has generated useful data and interest in innovative trial designs. Experts believe that whole blood resuscitation of trauma patients with hemorrhagic shock reduces 24‐hour mortality by 3%. These data will inform the design and planning of the TROOP trial.
CONFLICTS OF INTEREST
JAH, JR, EA, SWS, VTTT, MBM, SMD, JMY, and CP declare no conflicts of interest. JBH is a cofounder and on the board of directors of Decisio Health, on the board of directors of Zibrio and QinFlo, a coinventor of the Junctional Emergency Tourniquet Tool, and an adviser to PotentiaMetrics, Cellphire, and Arsenal Medical.
Supporting information
Fig. S1: Screenshot of elicitation questions posed to participants using an online elicitation tool.
Fig. S2: Evolution of responses from Round 1 to Round 2
ACKNOWLEDGMENTS
The authors thank the participants of the elicitation meeting: Jeremy W. Cannon, MD, Department of Surgery, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA; Juan C. Duchesne, MD, Department of Surgery, Tulane University, New Orleans, LA; Jennifer Gurney, MD, USAISR, Joint Trauma System, Defense Center of Excellence, San Antonio, TX; Donald H. Jenkins, MD, Department of Surgery, University of Texas Health Sciences Center, San Antonio, TX; Shaun D. Lawicki, MBBS, Department of Pathology, LSU Health Sciences Center, New Orleans, LA; Alan D. Murdock, MD, Department of Surgery, Allegheny General Hospital, Pittsburgh, PA; Gregory J. Pomper, MD, Department of Pathology, Wake Forest Baptist Health, Winston‐Salem, NC; John M. Porter, MD, Department of Surgery, Cooper University Hospital, Camden, NJ; Timothy Pritts, MD, PhD, Department of Surgery, The University of Cincinnati College of Medicine, Cincinnati, OH; Amy Rushing, MD, Department of Surgery, The Ohio State University Wexner Medical Center, Columbus, OH; Martin A. Schreiber, MD, Department of Surgery, Oregon Health & Science University, Portland, OR; Michael Smith, DO, Department of Surgery, UPMC Susquehanna, Williamsport, PA; Philip C. Spinella, MD, Department of Pediatrics, Washington University in Saint Louis, St. Louis, MO; Charles E. Wade, PhD, Department of Surgery and Center for Translational Injury Research The University of Texas McGovern Medical School at Houston, Houston, TX; and Martin D. Zielinski, MD, Department of Surgery, Mayo Clinic, Rochester, MN.
[Correction added on 8 February 2020, after first online publication: “Division of Acute Care Surgery” has been removed from the first affiliation.]
Funding: The study was supported by grant U34HL148472 from the U.S. National Heart, Lung, and Blood Institute.
REFERENCES
- 1. Cannon JW, Khan MA, Raja AS, et al. Damage control resuscitation in patients with severe traumatic hemorrhage: a practice management guideline from the Eastern Association for the Surgery of Trauma. J Trauma Acute Care Surg 2017;82:605‐17. [DOI] [PubMed] [Google Scholar]
- 2. Holcomb JB, Tilley BC, Baraniuk S, et al. Transfusion of plasma, platelets, and red blood cells in a 1:1:1 vs a 1:1:2 ratio and mortality in patients with severe trauma: the PROPPR randomized clinical trial. JAMA 2015;313:471‐82. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Bahr MP, Yazer MH, Triulzi DJ, et al. Whole blood for the acutely haemorrhaging civilian trauma patient: a novel idea or rediscovery? Transfus Med 2016;26:406‐14. [DOI] [PubMed] [Google Scholar]
- 4. Holcomb JB, Jenkins D. Get ready: whole blood is back and it's good for patients. Transfusion 2018;58:1821‐3. [DOI] [PubMed] [Google Scholar]
- 5. Nessen SC, Eastridge BJ, Cronk D, et al. Fresh whole blood use by forward surgical teams in Afghanistan is associated with improved survival compared to component therapy without platelets. Transfusion 2013;53(suppl 1):107S‐13S. [DOI] [PubMed] [Google Scholar]
- 6. Perkins JG, Cap AP, Spinella PC, et al. Comparison of platelet transfusion as fresh whole blood versus apheresis platelets for massively transfused combat trauma patients (CME). Transfusion 2011;51:242‐52. [DOI] [PubMed] [Google Scholar]
- 7. Spinella PC, Perkins JG, Grathwohl KW, et al. Warm fresh whole blood is independently associated with improved survival for patients with combat‐related traumatic injuries. J Trauma 2009;66(4 suppl):S69‐76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Cotton BA, Williams J, Merutka N, et al. Safety profile and impact of low‐titer group O whole blood for emergency use in trauma. J Trauma and Acute Care Surg 2020;88(1):87‐93 [DOI] [PubMed] [Google Scholar]
- 9. Zhu CS, Pokorny DM, Eastridge BJ, et al. Give the trauma patient what they bleed, when and where they need it: establishing a comprehensive regional system of resuscitation based on patient need utilizing cold‐stored, low‐titer O+ whole blood. Transfusion 2019;59(S2):1429‐38. [DOI] [PubMed] [Google Scholar]
- 10. Holcomb JB, Swartz MD, DeSantis SM, et al. Multicenter observational prehospital resuscitation on helicopter study. J Trauma Acute Care Surg 2017;83(1 suppl 1):S83‐91. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Greene TJ, DeSantis SM, Fox EE, et al. Utilizing propensity score analyses in prehospital blood product transfusion studies: lessons learned and moving toward best practice. Mil Med 2018;183(suppl_1):124‐33. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Cotton BA, Podbielski J, Camp E, et al. A randomized controlled pilot trial of modified whole blood versus component therapy in severely injured patients requiring large volume transfusions. Ann Surg 2013;258:527‐32. [DOI] [PubMed] [Google Scholar]
- 13. Amrhein V, Greenland S, McShane B. Scientists rise up against statistical significance. Nature 2019;567:305‐7. [DOI] [PubMed] [Google Scholar]
- 14. Wasserstein RL, Schirm AL, Lazar NA. Moving to a world beyond “p<0.05.”. Am Stat 2019;73(suppl 1):1‐19. [Google Scholar]
- 15. McShane BB, Gal D, Gelman A, et al. Abandon statistical significance. Am Stat 2019;73(suppl 1):235‐45. [Google Scholar]
- 16. Jansen JO, Pallmann P, MacLennan G, et al. Bayesian clinical trial designs: another option for trauma trials? J Trauma Acute Care Surg 2017;83:736‐41. [DOI] [PubMed] [Google Scholar]
- 17. Berry DA. Bayesian clinical trials. Nat Rev Drug Discov 2006;5:27‐36. [DOI] [PubMed] [Google Scholar]
- 18. Spiegelhalter DJ, Abrams KR, Myles JP. Bayesian approaches to clinical trials and health care evaluation. Chichester; Hoboken, NJ: Wiley; 2004. [Google Scholar]
- 19. Brownstein NC, Louis TA, O'Hagan A, et al. The role of expert judgment in statistical inference and evidence‐based decision‐making. Am Stat. 2019;73:56‐68. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. O'Hagan A. Uncertain judgements: Eliciting experts' probabilities. Hoboken, NJ: John Wiley & Sons; 2006. [Google Scholar]
- 21. Oakley JE, O'Hagan A. SHELF: the Sheffield Elicitation Framework (version 3.0). Available from: http://tonyohagan.co.uk/shelf [cited 2019 Oct 5].
- 22. O'Hagan A. Eliciting expert beliefs in substantial practical applications. J Royal Stat Soc D 1998;47:21‐35. [Google Scholar]
- 23. O'Hagan A. Expert knowledge elicitation: subjective but scientific. Am Stat 2019;29(73 (suppl 1)):69‐81. [Google Scholar]
- 24. Chaloner K, Church T, Louis TA, et al. Graphical elicitation of a prior distribution for a clinical trial. J Royal Stat Soc D 1993;42:341‐53. [Google Scholar]
- 25. Mason AJ, Gomes M, Grieve R, et al. Development of a practical approach to expert elicitation for randomised controlled trials with missing health outcomes: application to the IMPROVE trial. Clin Trials 2017;14:357‐67. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Morris DE, Oakley JE, Crowe JA. A web‐based tool for eliciting probability distributions from experts. Environ Model Softw 2014;52:1‐4. [Google Scholar]
- 27. Dallow N, Best N, Montague TH. Better decision making in drug development through adoption of formal prior elicitation. Pharm Stat 2018;17:301‐16. [DOI] [PubMed] [Google Scholar]
- 28. Chang W, Cheng J, Allaire J, Xie Y, McPherson J. Shiny: Web Application. Framework for R. R package version 1.2.0. [cited 2019 Oct 5]. Available from: https://cran.r-project.org/package=shiny.
- 29. Team R. A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. 2016. Available from: https://www.r-project.org/. [cited 2018 Mar 2].
- 30. Grigore B, Peters J, Hyde C, et al. A comparison of two methods for expert elicitation in health technology assessments. BMC Med Res Methodol 2016;16:85. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Laptook AR, Shankaran S, Tyson JE, et al. Effect of therapeutic hypothermia initiated after 6 hours of age on death or disability among newborns with hypoxic‐ischemic encephalopathy: a randomized clinical trial. JAMA 2017;318:1550‐60. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Goligher EC, Tomlinson G, Hajage D, et al. Extracorporeal membrane oxygenation for severe acute respiratory distress syndrome and posterior probability of mortality benefit in a post hoc bayesian analysis of a randomized clinical trial. JAMA 2018;320:2251‐9. [DOI] [PubMed] [Google Scholar]
- 33. Johnson SR, Tomlinson GA, Hawker GA, et al. Methods to elicit beliefs for Bayesian priors: a systematic review. J Clin Epidemiol 2010;63:355‐69. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Fig. S1: Screenshot of elicitation questions posed to participants using an online elicitation tool.
Fig. S2: Evolution of responses from Round 1 to Round 2