Why are not There More Bayesian Clinical Trials? Perceived Barriers and Educational Preferences Among Medical Researchers Involved in Drug Development

Abstract

Objective and Background

The clinical trials community has been hesitant to adopt Bayesian statistical methods, which are often more flexible and efficient with more naturally interpretable results than frequentist methods. We aimed to identify self-reported barriers to implementing Bayesian methods and preferences for becoming comfortable with them.

Methods

We developed a 22-question survey submitted to medical researchers (non-statisticians) from industry, academia, and regulatory agencies. Question areas included demographics, experience, comfort levels with Bayesian analyses, perceived barriers to these analyses, and preferences for increasing familiarity with Bayesian methods.

Results

Of the 323 respondents, most were affiliated with pharmaceutical companies (33.4%), clinical research organizations (29.7%), and regulatory agencies (18.6%). The rest represented academia, medical practice, or other. Over 56% of respondents expressed little to no comfort in interpreting Bayesian analyses. “Insufficient knowledge of Bayesian approaches” was ranked the most important perceived barrier to implementing Bayesian methods by a plurality (48%). Of the approaches listed, in-person training was the most preferred for gaining comfort with Bayesian methods.

Conclusions

Based on these survey results, we recommend that introductory level training on Bayesian statistics be presented in an in-person workshop that could also be broadcast online with live Q&A. Other approaches such as online training or collaborative projects may be better suited for higher-level trainings where instructors may assume a baseline understanding of Bayesian statistics. Increased coverage of Bayesian methods at medical conferences and medical school trainings would help improve comfort and overcome the substantial knowledge barriers medical researchers face when implementing these methods.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43441-021-00357-x.

Introduction

Traditionally, clinical trials data are analysed using “classical” approaches based on frequentist statistical methods where analyses are usually centred on a null hypothesis of no treatment effect. The main result from this null hypothesis testing is a single p value, the probability of data as extreme as those observed, assuming that the null is true. The probability threshold for deciding that the null hypothesis is false is typically set at p < 0.05. The p value does not give us any inference on whether we have made the correct decision, rather, inference for this study is within the context of inferences among all studies one might encounter under identical conditions [1]. This interpretation limits the usefulness of a p value in a medical setting [2, 3]. A decision-making structure based on methods utilizing Bayes Theorem can provide meaningful results associated with a hypothesis beyond what is seen with a p value [4].

In order to gain insight into barriers for implementation of Bayesian methods among medical researchers, the Drug Information Association Bayesian Scientific Working Group (DIA BSWG) Medical Outreach subteam sent a questionnaire to non-statisticians in drug development asking them to complete the survey if interested. Part of the objectives of the DIA BSWG is to enhance understanding of Bayesian methods with the vision to ensure that Bayesian methods are well understood and utilized where appropriate for design and analysis throughout the medical product development process. Previous work by this group for this includes a similar survey for statisticians [5] along with a number of articles in a special issue on Bayesian Methods in Medical Product Development and Regulatory Reviews [6–9]. Here, the subteam, which includes individuals from academia, industry and regulatory authorities, reports some results from the survey, and proposes approaches to overcome barriers based on insights from the survey.

The DIA BSWG delegated its Medical Outreach subteam to create and administer the survey, as well as report and interpret the results to propose actions based on insights learned. The subteam takes responsibility for all opinions and any errors expressed in the paper.

Materials and Methods

We sent the survey to over 1500 medical researchers involved in clinical trials through academia, pharmaceutical companies, clinical research organizations (CROs), or regulatory institutions. Volunteers from organizations were recruited to identify appropriate groups and secure necessary permissions to send the survey to potential respondents. While we did not know the total numbers contacted through some volunteers due to listservs and forwarding, we estimated the total number of researchers contacted as approximately 1600.

Surveys were sent to organizations primarily based in the U.S.A., although many also have a global presence in the pharmaceutical industry. The survey was run by the DIA BSWG medical outreach subteam using the REDCap software system at Vanderbilt University.

There were 323 respondents who answered at least one question. Most respondents answered all survey questions. The survey link opened on 11/11/2019. The last response was submitted on 12/20/2019.

The survey tool included 22 questions, which can be grouped into five sections. The first section consisted of seven questions to capture demographic data, including education background and work environment. The second section had six questions assessing previous Bayesian exposure/education and comfort level with these methods. The third section consisted of two rank questions assessing respondents’ perceived barriers to using Bayesian methods in clinical trials and preferences for learning approaches that would make them more comfortable with these methods. The options for what would increase respondents’ comfort with Bayesian methods not only had an active educational or experience components but also included a write-in choice for those who would prefer other options. For conciseness, we call these options for increasing respondents’ comfort levels “Educational Preferences”. The fourth section included a short scenario describing the trial design and results for a pilot study. Respondents were given four questions asking them to interpret the p value, posterior probability, confidence interval, and credible interval from this study. The last section, designed to be a usefulness check, included two questions assessing which interpretations respondents thought were most useful for decision making, and one question assessing the survey itself as a learning tool.

Most survey questions were multiple choice, with some questions allowing respondents to choose more than one category. Some questions had an “Other” option allowing write-in responses. Write-ins were examined by multiple members of the DIA BSWG and grouped with multiple choice categories if deemed appropriate, or in new categories. Answering all questions was not mandatory for completing the survey.

Descriptive statistics were run on demographics, including background education and work experience, to better characterize respondents. Proportions of respondents giving rankings of one, two, or three for rank-based questions of perceived barriers to using Bayesian methods and preferred learning approaches were calculated. We plotted stacked bar charts to compare results for rankings.

We examined whether certain background characteristics could affect how respondents ranked barriers and educational preferences; particularly, rankings subset by whether respondents’ work involved Phase 3 trials, work organization, having previous Bayesian training, and type of training. Respondents could have had multiple trainings. Percentages for rankings in each group are shown in dot plots. Percentages are those of respondents classifying the category as their top one, two, or three barriers/educational preferences. Some respondents had fewer than three rankings.

Results

Of the approximately 1600 medical researchers contacted, there were 323 respondents. Table 1 summarizes the education status, organization, work role, previous training in Bayesian statistics, and comfort level of the 323 respondents. Most received post-baccalaureate degrees within the past 20 years (61%), 67% graduated from medical school (MD; DO; MD/PhD), and 19.2% obtained a PhD without a medical degree. The respondents worked for pharmaceutical or health technology product development (33.4%), CROs (29.7%), regulatory agencies (18.6%), academia (11.8%), or medical practice (5.9%). Most (80.4%) described their role as a “Clinical Research Physician/Scientist” (33.4%), a “Medical Monitor/study lead/medical safety” (30.3%), or a “Reviewer for a Regulatory Agency” (16.7%).

Table 1.

Respondent demographics

	Overall (N = 323)
Highest degree
Bachelors	2 (0.6%)
Masters	35 (10.8%)
MD, DO, MD/PhD	216 (66.9%)
PharmD	7 (2.2%)
PhD	62 (19.2%)
Missing	1 (0.3%)
Number of years since degree completion
Under 5 years	33 (10.2%)
5 to 10 years	58 (18%)
10 to 20 years	106 (32.8%)
20 + years	124 (38.4%)
Missing	2(0.6%)
Organization
Academic	38 (11.8%)
Clinical Research Org	96 (29.7%)
Medical practice	19 (5.9%)
Pharma/HTA Development	108 (33.4%)
Regulatory	60 (18.6%)
Missing	2 (0.6%)
Work Role
Academic	16 (5.0%)
Clinical research physician/scientist	108 (33.4%)
Management	22 (6.8%)
Medical monitor/study lead	98 (30.3%)
Medical practice	15 (4.6%)
Regulatory reviewer	54 (16.7%)
Other: regulatory manager	3 (0.9%)
Other	4 (1.2%)
Missing	3 (0.9%)
Has previous Bayesian training
No training	137 (42.4%)
Has some training	186 (57.6%)
Comfortable Interpreting Bayes
Comfortable interpreting	27 (8.4%)
Little/no comfort	183 (56.7%)
Some, but not interpreting	110 (34.1%)
Missing	3 (0.9%)

Responses for previous training indicated that 31.3% and 19.5% respondents had a short course in statistics and/or on-the-job training, respectively, while 42.4% stated that they had neither training nor experience in Bayesian statistics. Only 8.4% reported being comfortable interpreting Bayesian analyses. Most (56.7%) had little to no comfort with interpreting Bayesian statistics, and 34.1% were not yet comfortable interpreting Bayesian analysis despite having some knowledge of the area.

Respondents were asked to rank the top three barriers to using a Bayesian method as “the primary statistical approach to a clinical trial” in their organization(s). Perceived barrier options included

1. Knowledge: Insufficient knowledge of Bayesian approaches

2. Regulators: Lack of clarity/guidance from regulators

3. Clinical Team: Reluctance from my internal clinical team

4. NA: The Bayesian approach is not applicable, and my organization sees no benefit

5. Reg Team: Reluctance from my internal regulatory team

6. Stat Team: Reluctance from my internal statistical team

7. Mngmnt: Reluctance from upper management

8. Other

Figure 1 is a stacked bar chart of these rankings of perceived barriers. “Insufficient knowledge of Bayesian approaches” was ranked highest by 155 (48%) respondents. The next most commonly selected barrier was “Lack of clarity/guidance from regulators”, which was ranked first by 61 (18.9%) respondents. These first two categories far outranked all other categories, including Bayesian approaches were not applicable (2.2%), reluctance from the internal regulatory team (1.9%), reluctance from the internal statistical team (2.2%), reluctance from the internal clinical team (3.7%), reluctance from upper management (1.9%), and “Other” (5%). Write-in responses for “Other” included a general reluctance in the community, unknown, and no barriers.

Fig. 1.

Perceived barriers to using Bayesian methods

For categories ranked as top two, there is very little change in order. Knowledge is still the highest with 212 responses ranking as 1 or 2 (65.6%). Lack of regulatory guidance was again second with 160 (49.6%) response rankings of one or two; although it was not the highest ranked barrier, it did receive the most votes as the second ranked barrier (30.7%). The order of highly ranked perceived barriers does not change much when examining categories ranked in the top three.

When examining these findings by whether respondents reported spending some time working in Phase 3 of product development (Fig. 2), there was little difference seen for top-ranked categories. However, those in Phase 3 have a slightly stronger perception of a regulatory barrier (20.9% vs. 14.8%).

Fig. 2.

Perceived barriers by Phase 3 work

Barriers by organization type showed more variation (Fig. 3). Lack of knowledge remained the top-ranked perceived barrier for all organizations. Those in academia were most likely to rank this as the top barrier (76.3%). Those in pharmaceutical companies were least likely (35.2%) with a much smaller gap between knowledge and the second ranked regulatory barrier (24.1%). When assessing the top two rankings, the knowledge barrier is tied with their regulatory barrier (51.9%).

Fig. 3.

Perceived barriers by work organization

When comparing those with some Bayesian training to those with none, there was little difference in top-ranked perceived barriers (Fig. 4). Those with no training were more likely to not rank the barriers which could be influencing some of the small differences seen in these subsets.

Fig. 4.

Perceived barriers by previous Bayesian training

We further analysed the rankings by the types of training respondents indicated that they have had. While the top-ranked categories remained the same, perceived lack of knowledge seemed to be a less important barrier among those who had taken a graduate course in Bayesian methods.

Respondents were asked to rank the top 5 factors that would make them “more comfortable using a Bayesian design and analysis on a primary objective in a clinical trial”. Options that were given included:

1. Workshop: In-person training at a workshop, conference, or internal to my organization.

2. Online: Online training with Q&A (e.g. live webinar, online course), with slides and recording available

3. Collaboration: Close collaboration between the clinical statisticians and medical teams for a project

4. Hypothetical: Participating in the creating of a hypothetical study in which the primary analysis is Bayesian with guidance from an instructor

5. Paper: A white paper written for clinicians to better understand Bayesian methods

6. Consult: 1–1 consultation with Bayesian expert(s)

7. Study: Written case studies

8. Self-Train: Self-training via books/journals, etc.

9. Other; write-in responses for this were generally along the lines of “regulatory acceptance”

Figure 5 presents stacked bar charts for the top three rankings for Bayesian education options. In-person training like a workshop, conference, or internal organization training was top ranked with 142 (44%) respondents ranking this as their top choice. Online training was a distant second with 49 (15.2%) respondents ranking it first. Other top choices include close collaboration for a project, 42 (13%), and a hypothetical study with guidance from an instructor, 30 (9.3%). The remaining educational options were preferred by some respondents. These rankings should be interpreted recognizing the survey was conducted before the COVID-19 pandemic.

Fig. 5.

Educational preferences for Bayesian training

The order of rankings for educational options did not differ much for those working primarily in Phase 3 vs no Phase 3 (Fig. 6). Phase 3 respondents had a stronger preference for in-person workshop training.

Fig. 6.

Educational preferences by Phase 3 work

Educational preferences by organization (Fig. 7) did not show differences in top preferences. However, those in CROs, pharmaceutical companies, or regulatory organizations had a stronger preference for in-person workshops than did those in academia or medical practice. When examining the top 2 or 3 preferences, academics tended to favour other forms of learning over in-person workshops.

Fig. 7.

Educational preferences by work organization

The order of the top education preferences did not change much depending on whether the respondent had previous Bayesian training (Fig. 8). There was a stronger preference for in-person workshops among those with no previous Bayesian training.

Fig. 8.

Educational preferences by previous Bayesian training

Discussion

To gain insight into the barriers for using Bayesian methods in clinical research and potential pathways that could increase uptake where appropriate, we collected and evaluated responses from interested medical researchers. The survey provided actionable insight on possible reasons for the slow uptake by the clinical development community. Key benefits of Bayesian methods include mechanisms for formally incorporating prior knowledge into the current trial analysis (thus, not ignoring what is already known about a disease state and an intervention) and for estimating the probability of a pre-specified treatment effect size [10] which are incredibly useful in clinical research. Incorporating prior knowledge can augment the information from the current trial and increase the precision [11].

Recently published perspectives [15–17] pointed out a trend in the scientific community to promote evaluating effect sizes and differences instead of only using p value thresholds. The FDA statement on p values, while stating that they are “fit for use” and necessary benchmarks for regulatory review purposes, also concludes that “scientific inference involving p values requires additional input and judgement regarding study design and conduct, data integrity, clinical relevance and method of analysis, among other factors” [14]. Bayesian methods are one way to incorporate additional information relevant to some of these factors.

However, implementing Bayesian methods in the regulated pharmaceutical industry does entail challenges, including extensive preplanning and collaboration of industry and regulators on formulating prior information and selecting Bayesian analysis methods [11]. There is also the need for standards for what constitutes substantial evidence of effectiveness in Bayesian analyses [12–14]. Implementation of Bayesian methods require specific statistical and computational expertise to ensure results that are sound [11].

According to the 323 survey participants, among all perceived barriers to implementing Bayesian approaches more widely, insufficient knowledge of these approaches far overshadowed all others. A lack of regulatory guidance came in as a clear second. We were expecting to see more differences in perceived barriers between those with no previous training and those with some. The fact that we did not see more separation could be indicative of insufficient Bayesian training currently available for medical researchers.

When examining what would make respondents more comfortable with Bayesian methods, in-person training was the top choice with online training the clear second preference. This survey was given before COVID-19 in-person restrictions, so now, some might switch their responses if given the option between in-person and non in-person categories. However, in-person training’s top ranking had almost three times as many respondents as the next highest preference, and when looking at the top two and three ranked preferences, this category is still clearly highly ranked.

Based on these survey results, we believe that introductory level training would be best presented through an in-person workshop that could also be broadcast online with live Q&A for those who prefer not to meet in person. Stronger preferences for online training or a collaborative project among those who have previous Bayesian training could be a useful pathway for higher-level training that may assume some baseline understanding of Bayesian methods.

Our findings and recommendations should be evaluated keeping some limitations in mind. The response rate of less than 20% suggests that the available responses might not represent the population who were given the chance to complete survey, so interpretation is limited to medical researchers who were motivated to respond. The results do provide insight into the comfort level and perceived barriers to Bayesian implementation among 323 medical researchers and can, thus, help inform methods that can be undertaken to increase the appropriate use of Bayesian methods in medical research. Although self-selection among respondents by definition introduces biased results, the results indicated that most responders were not Bayesian enthusiasts as many had little to no comfort with this methodology, and there was a healthy amount of curiosity and scepticism about Bayes in written comments about the survey. It is also important to note that interpretations of the results along with recommendations based on them are our own.

Conclusion

We found that a lack of knowledge of Bayesian methods was the top barrier to implementing those methods more broadly, and that in-person training was the top-ranked option for helping respondents become comfortable with using these methods. Based on insights gained from the survey, the DIA BSWG believes that there is a need for education on Bayesian methods and guidance from competent authorities. We suggest increasing coverage of Bayesian methods at medical conferences and medical school trainings, especially because most currently available educational courses and workshops are designed for a statistical audience. Guidance documents and case studies from regulatory authorities such as the US FDA’s 2010 “Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials” [11], the 2019 “Adaptive Designs for Clinical Trials of Drugs and Biologics” [13], and trial designs developed through the FDA’s Complex Innovative Trial Designs Pilot Program provide useful training materials outside a formal workshop.

Supplementary Information

Below is the link to the electronic supplementary material.

Author Contributions

All authors contributed to the survey conception and design. Organizing and contacting survey distribution volunteers were run by Ross Bray. Access and organization of the survey on the REDCap database were provided by Jennifer Clark. Material preparation and data analysis were performed by Jennifer Clark, Ross Bray, Andrew Hartley, and Fanni Natanegara. All authors commented and provided input and edits for the manuscript.

Funding

This research was supported by resources available to members of the DIA BSWG.

Declarations

Conflict of interest

The authors have no conflicts of interest relevant to this article to disclose.