Designing an evidence-based Bayesian network for estimating the risk versus benefits of AstraZeneca COVID-19 vaccine

Abstract

Uncertainty surrounding the risk of developing and dying from Thrombosis and Thrombocytopenia Syndrome (TTS) associated with the AstraZeneca (AZ) COVID-19 vaccine may contribute to vaccine hesitancy. A model is urgently needed to combine and effectively communicate evidence on the risks versus benefits of the AZ vaccine. We developed a Bayesian network to consolidate evidence on risks and benefits of the AZ vaccine, and parameterised the model using data from a range of empirical studies, government reports, and expert advisory groups. Expert judgement was used to interpret the available evidence and determine the model structure, relevant variables, data for inclusion, and how these data were used to inform the model.

The model can be used as a decision-support tool to generate scenarios based on age, sex, virus variant and community transmission rates, making it useful for individuals, clinicians, and researchers to assess the chances of different health outcomes. Model outputs include the risk of dying from TTS following the AZ COVID-19 vaccine, the risk of dying from COVID-19 or COVID-19-associated atypical severe blood clots under different scenarios. Although the model is focused on Australia, it can be adapted to international settings by re-parameterising it with local data. This paper provides detailed description of the model-building methodology, which can be used to expand the scope of the model to include other COVID-19 vaccines, booster doses, comorbidities and other health outcomes (e.g., long COVID) to ensure the model remains relevant in the face of constantly changing discussion on risks versus benefits of COVID-19 vaccination.

1. Introduction

Vaccine safety is a key consideration for public health managers, policy makers and the public. The controversy surrounding the safety of the AstraZeneca (AZ) COVID-19 vaccine (Vaxzevria) in relation to fatalities from rare, atypical severe blood clots (Thrombosis and Thrombocytopenia Syndrome [TTS]) [1] has contributed to vaccine hesitancy in many countries including Australia [2]. This hesitancy is in-part due to the lack of access to comprehensive, up-to-date scientific information, displayed in an easily understandable and objective manner. The problem has been exacerbated by continually-evolving information, with the publication of new scientific studies and agency reports. Moreover, many scientific studies often address only part of the overall puzzle. The lack of aggregated information presents challenges not only to members of the public, but also to the clinicians who are tasked with helping patients make an informed decision about the AZ vaccine. Delving into the extensive and constantly-changing scientific literature is also beyond the resources of most public health practitioners and policy makers who need to make decisions in a highly dynamic environment. Similar challenges face epidemiologists and other scientists who need to assess and account for vaccine safety in related studies.

While there have been attempts to compile results into a meaningful comparison with respect to the risks versus benefits of the AZ vaccine, these have typically been confined to specific scenarios, for example when community transmission of SARS-CoV-2 in Australia was low [3]. A more flexible framework is therefore urgently needed to effectively combine and communicate the existing evidence surrounding the risks versus benefits of vaccines such as the AZ vaccine. It is crucial that this framework is both transparent in its assumptions and data sources, as well as easily updatable to account for new evidence and changes in the pandemic landscape, such as new variants, changes in vaccine effectiveness, or fluctuations in the rates of community transmission. The framework must also be able to incorporate data from a wide range of sources, and in different formats.

Bayesian network (BN) modelling [4] is well suited for implementing such a framework. The transparency and flexibility of BNs for integrating different data sources [5], [6], [7] has seen them used for a variety of different analyses relating to COVID-19, such as examining the limitations of contact tracing [8] using expert-elicited data for interpretation of SARS-CoV-2 testing [9] and estimating SARS-CoV-2 infection and fatality rates [10]. BNs can be designed as causal models and are easily interpretable, which makes them suitable for use in decision-support contexts, for example those developed by Fenton et. al. [10] to illustrate the need for more random testing of community members when estimating prevalence of COVID-19.

The COVID-19 Risk Calculator (CoRiCal) was developed to address the need for a user-friendly risk–benefit analysis tool to assist clinicians and the public to make informed decisions about COVID-19 vaccinations [11], [12]. The specific objectives of the model were to estimate and compare: (i) the risk of developing and dying from TTS following the AZ vaccine; (ii) the background risk of developing and dying from atypical severe blood clots (cerebral venous sinus thrombosis [CVST] and portal vein thrombosis [PVT]) in the general population; (iii) the risk of developing and dying from COVID-19-associated atypical severe blood clots (CVST or PVT); and iv) the risk of SARS-CoV-2 infection and related deaths under different transmission intensities [12]. In this paper, we describe in detail the methods used to design and validate the BN model that integrates the best available evidence to compare the risks versus benefits of the AZ vaccine for the Australian population.

2. Methods

2.1. Modelling approach

The modelling process was a hybrid evidence-driven and expert-led approach (Fig. 1 ). This process was modified from the approach used by Ticehurst et. al. [13], and was chosen as an efficient way of creating a useful and evidence-based model in as short a timeframe as possible. Seven subject matter experts with experience in virology (KRS, JES), clinical practice (AB, CLL, JL), biostatistics (MW, KM), and infectious disease epidemiology (CLL, JL) were involved in gathering and interpreting available information. The modelling team (HJM, KM, CLL, JES) facilitated the design process and implemented the model using the GeNIe BN modelling software version 3.0.5703 [14].

Fig. 1.

Model design process used for implementing the CoRiCal Bayesian network.

Evidence was initially collected for the incidence of the adverse events following immunisation (AEFI) for both the AZ and Pfizer vaccines, as well as for a range of adverse outcomes and complications from COVID-19 (including death), and comorbidities (e.g., obesity, diabetes) that could influence COVID-19-related outcomes. We selected CVST and PVT to provide a comparator to TTS which occurs after administration of a drug or vaccine. CVST and PVT are atypical severe blood clots that can occur at a background rate in the general population [15], [16], [17] and have also been reported in COVID-19 patients [18].

The scope of the model was narrowed in response to the changing pandemic landscape, where hesitancy for the AZ vaccine was growing, despite it being the only vaccine available to older age groups at the time of model development. To facilitate the urgent need for decision-support tools for clinicians and the general public, analysis of the Pfizer vaccine, comorbidities, and other COVID-19-related outcomes (e.g., ICU admission, long COVID) were excluded from this initial model. The results presented here therefore focus on (i) the risk of developing and dying from TTS following the AZ vaccine; (ii) the background risk of developing and dying from atypical severe blood clots (iii) the risk of developing and dying from COVID-19-associated atypical severe blood clots (CVST or PVT) and (iv) the risk of SARS-CoV-2 infection and related deaths under different transmission intensities.

2.2. Data sources

Three main sources of evidence were considered: published literature, publicly available government data/reports and professional expert advisory groups such as Thrombosis and Haematology Society of Australia and New Zealand (THANZ). In Australia, the Commonwealth Government is guided by an expert subcommittee of haematologists from THANZ and vaccine experts to adjudicate on whether cases of AEFI related to the AZ vaccine were classified as TTS. All publicly available data used in this study was sourced from official government websites (Section 3.1).

2.3. Bayesian networks

BN models provide a visual and probabilistic approach to integrating and analysing data [4]. In a BN, the system of interest is depicted as a directed acyclic graph (DAG) in which variables are represented as nodes and parent–child associations between variables are represented as arrows connecting the respective nodes. In a discrete BN, each node is categorised into, or defined by, a set of states that define the classes (e.g., male/female) or ranges (e.g., 1–19, 20–30, 30+ years old) of the corresponding variable. The BN is then quantified by assigning probabilities to these states, conditional on the states of the parent nodes. The probability table for a node without parents is quantified using a prior distribution. The BN structure therefore ascribes a set of conditional independencies on the joint distribution of the variables. This allows a rich model to be designed by encoding various model assumptions about the relationships of variables, while only needing to assign conditional probability tables (CPTs) on a few variables at a time, specifically between a child node and its immediate parent nodes. The simplification of a BN as a connected set of CPTs, in which each node is dependent only on immediate parent nodes is a result of the Markov properties of the underlying DAG, and allows for very flexible model structure, fast computation and the ability to use different information sources to inform different components of the system [6].

The features of a BN are illustrated in the example shown in Fig. 2 , which demonstrates one option for modelling the probability of dying from vaccine-associated TTS based on AZ dose, variant (either no doses, first dose or second dose) and age group. The outcome node, Die from vaccine-associated TTS has one parent node; Vaccine-associated TTS. The Vaccine-associated TTS node is itself a child node of the AZ dose and Age group nodes. The CPT for the outcome node is shown, giving the probability for each state of this node conditional on the state of the parent node (Vaccine-associated TTS).

Fig. 2.

An example Bayesian network modelling the chance of dying from vaccine-associated TTS based on AZ dose and age group. Scenario shown is for the first dose of the AZ vaccine for a female aged between 60 and 69 years old.

2.4. Model design

2.4.1. Model structure

The structure of the BN model described here was developed in four stages. In the first stage, all authors agreed on the overall scope of the model. To accelerate development, it was decided to concurrently design and parameterise the model using three sub-models, based on subject matter. These were 1) risk of developing AZ vaccine-associated TTS, and background risk of developing and dying from atypical severe blood clots (CVST and PVT) over a six-week time-period, i.e. in persons who have not received the AZ vaccine, nor been diagnosed with COVID-19; 2) risk of developing symptomatic COVID-19 over a six-month time-period; and 3) risk of developing and dying from COVID-19 or COVID-19-associated atypical severe blood clots (CVST and PVT) over a six-month time period. These three sub-models were then combined into a single model. The final conceptual model, which defined the structure of the resulting BN, represented the key relationships identified by the experts.

In the second stage of development, for each sub-model, the team of experts compiled a list of relevant variables, agreed on a list of reputable information sources for each variable, and defined the relationships between variables. Questionnaires were developed to formalise the evidence gathering (Supplementary S1 – Example questionnaire for evidence gathering) and prompt experts for relevant information (e.g. ‘Is there evidence to suggest that vaccine effectiveness differs by sex?’). After discussing the evidence with the experts, the answers from these questionnaires were used by the modelling team to design a draft conceptual model. This process was iterated over several weeks until agreement was reached between experts and modellers. The structure of the BN was modified as new evidence was identified or as updates became available. For example, when evidence emerged that the risk of COVID-19-associated CVST and PVT differed by sex [18], links between these nodes were added in the model.

In the third stage, the states of each node were defined. Most nodes in the model were binary, having only two possible states (yes/no; effective/ineffective), with age and community transmission being the only continuous variables requiring categorisation. Age was categorised into ten-year age brackets consistent with those used in the weekly Australian Technical Advisory Group on Immunisation (ATAGI) reports [19]. Community transmission was categorised to be compatible with the ATAGI reported rates for low, medium and high transmission, as well as a baseline state (1000 cases per day) for three Australian states.

In the final stage, the model was critically evaluated to ensure that all variables and relationships could be informed by authoritative, quantitative evidence. Nodes and links in the BN were removed in the absence of evidence, or where the evidence suggested that the parent node had little effect on the outcome. For example, although the experts found evidence that males were at a higher risk of infection [20], the difference between sexes was small and did not appear to have a substantive impact on the risk–benefit analysis; as highlighted above, this can be modified as further supporting information emerges. States of nodes were also examined in a similar manner and the states redefined where necessary according to the available evidence (for example, adding additional transmission rates).

2.4.2. Parameterisation

Once the model structure was finalised, the available evidence was converted into a suitable format to quantify the CPTs. This involved defining the information in terms of conditional probabilities. For example, data reported as number of cases per 100,000 people, or infection rate over a certain time such as 16 weeks or six months, were converted into an equivalent probability and standardised to the same timeframes used in the model (six weeks for background rates of CVST and PVT and six months for all COVID-19-related outcomes, including COVID-19-associated atypical blood clots). CPT values were revised several times during the modelling process as new evidence or updated data became available.

Where evidence was available from more than one source, expert judgement was used to combine information or determine which source was most appropriate. For example, as there were limited Australian data on COVID-19-associated CVST and PVT, this information was sourced from a large international study [18]. Expert judgement was also used if a particular source of evidence did not align with the model structure. For example, the age distribution of cases for the delta variant was sourced from daily reports by New South Wales (NSW) Health for the following age categories: 0–19 years, 5-year age groups from 20 to 69 years, and 70+ years. However, for the BN, the experts agreed that the 0–19 years age category needed to be divided into 0–9 years and 10–19 years. A 40%/60% split for the cases in the respective age groups was assumed, based on the age distribution of cases reported by the National Notifiable Disease Surveillance System (NNDSS) [21].

2.5. Model validation

Four key components of the BN model were inspected and critically evaluated by the modelling team: network structure, node discretisation, CPT parameterisation and overall model behaviour. The evaluation included the assessment of content, face, integral and concurrent validities [22]. Content and face validities were used to assess the model structure, the variables included and the relationships between them, and the discretisation of nodes within the network. All subject matter experts were provided with a step-by-step description of the final structure of the full model. They were then given the opportunity to discuss the assumptions and evaluate whether all relevant evidence from the literature had been included and appropriately represented in the model.

Integral validity was evaluated to confirm that the structure of the model reflected the design assumptions. Two statisticians (MW and KM) were provided with the assumptions and data used to calculate the CPTs. These values were then used to manually calculate the probabilities of various outcomes under different scenarios. These calculations were performed independently of model structure or parameterisation and the outcomes compared against model estimates for these same scenarios. The model structure was assessed for internal consistency with official data sources such as ATAGI, for example the discretisation of age groups and definitions of low/medium/high transmission intensity [19]. As described earlier, in some instances the choice was made to forego concurrent validity in favour of model parsimony based on the available evidence, for example excluding the reported weak relationship between sex and risk of infection as it would have little impact on the outcome but greatly increase the size of the CPT. Evaluation of the external validity of the model requires independent published information about analogous outcomes. Because such information was scarce, the external validity assessment was undertaken by comparing the behaviour of the results of the scenarios generated for the integral validity analysis with the published data listed in Table 1 .

Table 1.

Data sources used in designing and parameterising the model.

Type of data	Variable	Data Source
Government reports	Risk of developing and dying from vaccine- associated TTS	ATAGI weekly updates
	VE against symptomatic infection and death	Australian Government report (delta variant)- Doherty Institute Modelling Report for National Cabinet, Table S2.5 [24] UK Government report (alpha variant) - Vaccine effectiveness table, 16 July 2021. Public Health England [25]
	Risk of developing symptomatic COVID-19 based on age group	Delta variant: NSW COVID-19 cases data [20] Alpha Variant: National Notifiable Diseases Surveillance System public datasets [21]
Peer-reviewed publications	Background chance of developing and dying from atypical severe blood clots (CVST or PVT)	CVST: Incidence and Mortality of Cerebral Venous Thrombosis in a Norwegian Population [17] PVT: Incidence rates and case fatality rates of portal vein thrombosis and Budd-Chiari Syndrome [16]; Survival after splanchnic vein thrombosis: A 20-year nationwide cohort study [15]
	Risk of developing and dying from COVID-19-associated atypical severe blood clots (CVST and PVT)	Cerebral venous thrombosis and portal vein thrombosis: a retrospective cohort study of 537,913 COVID-19 cases [18]
Professional advisory group	Risk of developing and dying from TTS	ATAGI weekly updates [19] Thrombosis and Haematology Society of Australia and New Zealand (THANZ) Vaccine-induced Immune Thrombotic Thrombocytopenia (VITT) advisory group
Calculated	Risk of infection depending on age and variant	Calculated using formula, based on data from parent nodes
	Risk of symptomatic COVID-19 infection under current transmission and vaccination status	Calculated using formula, based on data from parent nodes

2.6. Model Sensitivity

Sensitivity analyses play a crucial role in assessing the robustness of the findings or conclusions of an analysis. They are an important way to assess the impact, effect or influence of key assumptions or variations—such as different methods of analysis, definitions of outcomes, protocol deviations, missing data, and outliers—on the overall conclusions of a model [23]. The sensitivity of the model was examined in three steps.

In the first step, for each intermediate and outcome node (i.e. all child nodes), the strength of influence of each parent node was calculated using the inbuilt function in the GeNie modeller software program [14]. The strength of influence measures the difference in the probabilities of the target CPT based on changes in the probabilities of the parent CPT. A larger difference indicates a stronger influence. Two measures were considered, the average Euclidian difference and the maximum Euclidian difference (Supplementary S2 – Explanation of Euclidian Distance in BNs).

In the second step, this analysis was expanded to identify nodes that were most influential for the chance of developing symptomatic COVID-19 and the chance of dying from COVID-19 within a six-month timeframe. These two variables were selected as key nodes for analysis because both were highly connected (having three or more parents in the final model) and both had multiple indirect links with other nodes, so the route of influence is not easily determined by simply looking at the model structure. The most influential nodes were identified using the ‘Sensitivity Analysis' function in the GeNie modeller software program [14].

In the third step, two separate analyses were carried out for all outcome nodes. The first examined which input nodes were most influential on the outcomes by adjusting the selected states for each node, for example selecting each age group one at a time, and then noting the estimated values of each outcome node. The second analysis for step three calculated and compared the minimum (worst-case) and maximum (best-case) estimates for each outcome node. The best-case and worst-case scenarios were identified by systematically adjusting the scenarios for each of the input nodes. The exception was that low, rather than no transmission was used for best-case scenarios for COVID-19-related outcomes over a six-month timeframe, as it was considered more meaningful. Under a zero-community transmission scenario, all outcome nodes other than Die from background CVST within six weeks (n16) and Die from background PVT within six weeks (n17) have a probability of zero, blocking any influence from the remaining input nodes.

3. Results

3.1. Data sources

The range of data sources used to design and parameterise the model are presented in Table 1. Evidence from Australian data was prioritised, however international studies were used to provide evidence on the rates of CVST and PVT, and vaccine effectiveness.

3.2. Model description

The first sub-model, risks of developing and dying from background atypical severe blood clots (CVST and PVT), is shown in Fig. 3 . The risk of developing TTS associated with the AZ vaccine differs by age group and whether first or second dose of AZ is being considered [26]. In this model, the risk of TTS after the first and second AZ dose (n1) were treated as independent events (risk of TTS after the first dose or after the second dose). For a population-level analysis of total cases and deaths from TTS, a cumulative definition can be used where the risk of TTS after the second dose is the combined risk of both doses. Die from vaccine-associated TTS (n15) represents the proportion of TTS cases who die as a result (case fatality rate). The background rates of CVST and PVT were calculated over six weeks to be comparable to the timeframe in which vaccine-associated TTS is likely to occur after the AZ vaccine.

Fig. 3.

Conceptual model for sub-model 1: Risk of developing and dying from (i) vaccine-associated TTS, and (ii) background atypical severe blood clots (CVST and PVT), i.e. in those who have not received the AZ vaccine and have not been infected with SARS-CoV-2.

The second sub-model (Fig. 4 ) focuses on the risk of developing symptomatic COVID-19 based on age group, SARS-CoV-2 variant and vaccine effectiveness. To enable direct comparison of risks of poor health outcomes versus benefits of the vaccine, all probabilities were calculated for a six-month period to reflect the estimated duration of protection from the AZ vaccine. Data were based on reported cases in different age groups from NSW [20], which provided the best open-source data available in Australia at the time of model development. In line with the dominant variant in Australia, data from June 2021 were used for the delta variant, and data prior to this date were used for the alpha/ancestral variant. Vaccine effectiveness was sourced separately for the delta variant [24] and alpha/ancestral variant[25].

Fig. 4.

Conceptual model for sub-model 2: Risk of developing symptomatic COVID-19 depending on number of AstraZeneca vaccine doses received, SARS-CoV-2 variant, vaccine effectiveness, age group, and level of community transmission.

To simplify the model structure and enable easier updates in the future, the relative risk of symptomatic infection by age and variant was calculated using an intermediate node (n11). The Risk of symptomatic infection under current transmission and vaccination status (n12) over a six-month period was thus calculated for different transmission intensities that reflect various realistic scenarios. The intermediate step of calculating the Risk of symptomatic infection by age and variant (n11) was not necessary from a mathematical perspective, but was included to simplify the conceptual model and resulting CPTs. This reduces the number of different combinations in the resulting CPT and makes it easier for experts to quantify them with unique probabilities.

The final sub-model (Fig. 5 ) collated evidence for the risk of dying from COVID-19, or from COVID-19-associated CVST or PVT. The COVID-19-associated CVST/PVT rates (n19 and n20) represent the total rates in COVID-19 patients, not an increase over the background rates. The probabilities used to parameterise the model for deaths from COVID-19 (n18) are calculated based on case fatality rates for each age/sex category, as reported by Australian Government Department of Health COVID-19 summary statistics (Table S5.3). These figures also include deaths from COVID-19-associated complications, such as rare blood clots (including CVST and PVT). These case fatality rates are entered into the CPT for the scenario where the vaccine is not effective against death (i.e. node VE against death (n10) is set to ‘No’). All scenarios in this CPT where the node VE against death (n10) is ‘Yes’ are set to zero probability of death, as by this definition the vaccine is 100% effective against death. For each scenario in the analysis, the model uses these probabilities to calculate the probability for each age group of dying from COVID-19 based on the values of vaccine effectiveness against death (n10), and the risk of becoming infected (n12).

Fig. 5.

Conceptual model for sub-model 3: Dying from COVID-19 or COVID-19-associated atypical severe blood clots, depending on age, sex, vaccine effectiveness, variant, and level of community transmission.

The overall chance of dying depends on the chance of becoming infected, which in turn depends on the community transmission rate (x% over six months). In the default scenario where infection is unknown, it therefore represents the overall chance of dying from COVD-19 over six months. Once a person is infected, the level of community transmission is no longer relevant, and this node represents a fixed chance of dying from COVID-19 i.e. the case fatality rate based on age and sex, and us such does not have an associated timeframe.

The final BN (Fig. 6 ) combines the three sub-models and integrates the current available evidence regarding the probability of Dying from vaccine-associated TTS (n15), Dying from background CVST (n16) or Dying from background PVT (n17), overall probability of Dying from COVID-19 (n18), and probability of Dying from COVID-19-associated CVST (n19) or Dying from COVID-19-associated PVT (n20). Five input nodes – AZ dose (n1), Age group (n2), SARS-CoV-2 variant (n3), Community transmission at x% over 6 months (n4) and Sex (n5) were used to define the population and transmission scenarios.

Fig. 6.

Bayesian network structure showing relationships between the input, intermediate and outcome nodes.

Several latent nodes were included as intermediate steps when calculating the probability of Dying from COVID-19 (n18). VE against symptomatic infection (n9) and VE against death (n10) were modelled based on the SARS-CoV-2 variant (n3) (either alpha/ancestral or delta) and the AZ dose (n1) received; none, first or second). At the time of this study, there was insufficient evidence to include the effects of the different vaccine schedules (e.g., shorter intervals between first and second doses) on vaccine effectiveness. The Risk of symptomatic infection by age and variant (n11) is an intermediary node used to simplify the calculations required for the child node Risk of symptomatic infection under current transmission and vaccination status (n12). The CPT for n11 represents the probability that an unvaccinated person in a given age group and with a given SARS-CoV-2 variant (n3) would develop symptomatic infection over six months, assuming an overall infection rate of 10% during this time period. For example, assuming the delta variant, an unvaccinated 20–29 year old would have a 15.5% probability of developing a symptomatic infection over six months. In contrast, a 60–69 year old in the same scenario would have a 5.5% probability. The CPT for the child node, Risk of symptomatic infection under current transmission and vaccination status (n12), can then be calculated based on the risk of infection without the vaccine (n11), vaccine effectiveness against symptomatic infection (n9) and the current transmission rate over six months (n4). The final variables used to calculate the risk of Dying from COVID-19 (n18) were Age group (n2), Sex (n5), the Risk of symptomatic infection under current transmission and vaccination status (n12) and VE against death(n10). The remaining two outcome nodes were Die from COVID-19-associatedCVST (n19) and Die from COVID-19-associatedPVT (n20), each dependent on developing CVST from COVID-19 (n13) or PVT from COVID-19 (n14), respectively.

3.3. Parameterisation

Full probability tables for all nodes are given in Supplementary S3 – Values for conditional probability tables. Of the five input nodes, priors for two of these (sex and age group) were set to reflect the approximate distribution of the Australian population. For Sex (n5) a uniform distribution (50% males and 50% females) was adopted. The distribution for the Age groups (n2) was based on data from the Australian Bureau of Statistics [27]. To reflect the COVID-19 situation in Australia at the time the analysis was carried out (August 2021), the priors for the SARS-CoV-2 variant (n3) were set as 95% delta variant and 5% alpha/ancestral variant. AZ dose (n1) was set to 30% unvaccinated, 35% received first dose only, and 35% received second dose. Priors for AZ dose can be updated as vaccine coverage increases. Uniform priors were used for the Community transmission at x% over six months (n4) – where x is the percent transmission at each state, as it is expected that a value will be selected for this state prior to running scenarios.

The default probabilities for the model are shown in Fig. 7 . Scenarios can be generated by selecting a single state for each input node (shown in orange). This evidence then propagates through the network to produce an estimate from each outcome (shown in purple). Additional scenarios can be generated to evaluate the effect of the vaccine once a person becomes infected with SARS-CoV-2 by setting node n12 to ‘Yes’. Probability values for a medium transmission scenario (n4) for the delta variant (n3) and a fully vaccinated population (n1) are given in Supplementary S4 – Probabilities for scenario, as an example of scenario analysis. Of the 15 intermediate and outcome nodes, three (VE against symptomatic infection [n9], VE against death [n10], and relative Risk of symptomatic infection by age and variant [n11]) were calculated from data provided in government reports [20], [24], [25]. The case fatality rate for vaccine-associated TTS (n15) was sourced from ATAGI reports [26].

Fig. 7.

Parameterisation for the delta variant under a medium transmission scenario.

For the Risk of symptomatic infection under current transmission and vaccination status (n12), the CPTs for any scenario in which VE against symptomatic infection (n9) equalled ‘Yes’ (i.e. the vaccine is effective) or relative Risk of symptomatic infection by age and variant (n11) equalled ‘No’ (i.e. there was no risk of infection) were parameterised at 100% probability of no infection (i.e. zero risk). For the remaining scenarios (i.e., the VE against symptomatic infection (n9) was ‘No’ and relative Risk of symptomatic infection by age and variant (n11) was ‘Yes’), the probability for each state of Community transmission at x% over six months (n4) was calculated as a proportion of the 10% baseline in node n11. For example, in node n12 the state ‘Two-percent’ represents a 2% community transmission rate over six months, which is one fifth of the 10% baseline assumed for node n11. Under this scenario, the value for the ‘Yes’ state in the CPT is therefore 0.2, or 20% of the assumed baseline. The remaining nine CPTS were populated directly with data from published literature, and modified for external consistency if necessary. A full explanation of all assumptions used in the calculations is given in Supplementary S5 – Assumptions.

3.4. Validation

All subject matter experts were satisfied that the final conceptual model structure sufficiently represented all relevant variables and relationships within the scope of the model, and that chosen states for each node were consistent with what could be parameterised based on the available evidence. Calculations made independently of the model agreed with model predictions (Supplementary S6 – Manual calculations for validation), confirming the integral validity of the model structure in meeting the design assumptions. The results of the scenario testing (Supplementary S7 – Influence of inputs nodes on outcomes), which were also used for sensitivity analysis, confirmed that the model behaved as expected. For example, increasing the rate of community transmission increased the fatalities from COVID-19 and COVID-19-associated atypical severe blood clots. Similarly, increasing the proportion of the population who were vaccinated decreased these same outcomes. Additional checks based on the scenarios used for the manual calculations further confirmed that outcomes from the model were consistent with the available evidence (Supplementary S8 – External validation checks).

One potential anomaly revealed in this analysis was that for lower age groups, model estimates for dying from COVID-19-associated atypical severe blood clots were higher than estimates of the overall probability of dying from COVID-19. Investigation revealed that this was a result of the data being taken from different sources, with the probability of blood clots being derived from UK data [18] and the COVID-19 case fatality data from Australia [20], where there had been relatively few COVID-19 fatalities at the time of writing. While this should not be considered as an error in the model, it has the potential to cause confusion when end-users are interpreting scenario results, which may reduce trust in the model estimates. Clear communication around the limitations and assumptions of the model are critical in helping to prevent these misunderstandings.

3.5. Sensitivity analysis

The results of the strength-of-influence analysis comparing the relative influence of direct parent nodes on child nodes were measured using the average and maximum Euclidean distance (ED), where a larger ED indicates greater influence. A person’s age group (average ED 0.013) was more influential than their sex (average ED 0.001) in determining their chance of Dying from COVID-19 (n18). Age group (ED 0.043) was also more influential than virus variant (ED 0.031) in determining the Risk of symptomatic infection by age and variant (n11). Full results are shown in Supplementary S9 - strength-of-influence analysis.

Fig. 8 shows the results of the sensitivity analysis for the Risk of symptomatic infection under current transmission and vaccination status (n12). The strength of influence is shown by the shading, with darker red shading representing a stronger influence, i.e. small changes in these nodes will lead to larger changes in node n12 [28]. Nodes shown in grey have no links to, and hence no influence on the target node (n12) because of the network structure. The shading of the target node for this analysis (in this case n12) represents the overall degree to which the value of the connected nodes will influence the target. In Fig. 8, this therefore reflects the range of possible values for the risk of symptomatic infection based on changes in the six parent and grand-parent nodes (n1, n2, n3, n4, n9 and n11).

Fig. 8.

Sensitivity to findings for ‘Risk of symptomatic infection under current transmission and vaccination status (n12)’. Nodes with darker red shading have more influence than lighter shaded nodes. Grey nodes are not connected upstream of the target node (n12) and therefore have no influence. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

As expected, when all nodes were considered (rather than just the direct parents), Community transmission at x% over 6 months (n4) was highly influential on the Risk of symptomatic infection under current transmission and vaccination status (n12) (Fig. 8). Of the input nodes, both Age group (n2) and the AZ dose (n1) had a strong influence on the Risk of symptomatic infection under current transmission and vaccination status (n12).

The equivalent analysis for Die from COVID-19 (n18) (Supplementary S10 - Sensitivity to findings for ‘Die from COVID-19 – n18), showed that Age group (n2) has a large influence on this node. The AZ dose (n1) is shown as influential in both preventing Risk of symptomatic infection under current transmission and vaccination status (n12) and preventing Dying from COVID-19 (n18) . The stronger influence of Age group (n2) compared with AZ dose (n1) on the Die from COVID-19 (n18) node is due to both the input data and the model structure, in that nodes directly linked to the target node will be more influential than variables linked indirectly.

The results of alternating the states of individual input variables are provided in Table 2 . The minimum and maximum values in Table 2 represent the largest and smallest feasible probabilities for each outcome node being ‘Yes’, when the respective input node is used to define a scenario. For example, when no evidence was set for any other input nodes, changing the values of AZ dose (n1) results in a maximum estimated probability of Dying from COVID-19 (n18) equivalent to 224 per million, and can be reduced to at most 8 deaths per million. AZ dose (n1) in these tables, and in Fig. 8 and Supplementary S10 were averaged across all age groups and transmission rates, so it should also be considered that AZ dose (n1) would be expected to have more influence at the higher transmission rates. Analysis of sensitivity to findings can be conducted for specific scenarios, e.g., for a specific age group and under a specific transmission scenario.

Table 2.

Sensitivity of outcome values to changes in inputs (shown as per million cases). Minimum and maximum represent the smallest and largest values for each outcome node when selecting different states of the input nodes.

			Input nodes
			AZ dose (n1)	Age group (n2)	Variant (n3)	Community transmission rate at x% over 6 months (n4)	Sex (n5)
Output nodes	Die from vaccine- associated TTS (n15)	Range	1.18	0.19	–	–	–
		Min	0.00	0.31	–	–	–
		Max	1.18	0.50	–	–	–
	Die from background CVST (n16)	Range	–	0.03	–	–	–
		Min	–	0.03	–	–	–
		Max	–	0.05	–	–	–
	Die from background PVT (n17)	Range	–	0.53	–	–	–
		Min	–	0.00	–	–	–
		Max	–	0.53	–	–	–
	Die from COVID-19 (n18)	Range	215.92	655.53	89.39	269.85	16.24
		Min	7.99	0.00	80.63	0.00	76.98
		Max	223.91	655.53	170.02	269.85	93.22
	Die from COVID-19-associated CVST (n19)	Range	0.08	0.09	0.02	0.28	0.05
		Min	0.05	0.04	0.07	0.00	0.06
		Max	0.13	0.13	0.09	0.28	0.11
	Die from COVID-19-associated PVT (n20)	Range	0.90	1.04	0.23	3.04	0.39
		Min	0.55	0.44	0.74	0.00	0.76
		Max	1.45	1.48	0.97	3.04	1.16

Based on Table 2, it is evident that Age group (n1) is the most influential node for the Die from COVID-19 (n18) node, as indicated by the large range of probabilities associated with this input. In contrast Community transmission at x% over 6 months (n4) was more influential than Age group (n1) for the outcomes relating to dying from COVID-19-associated atypical severe blood clots (n19 & n20). Values in Table 2 are reported in absolute rather than relative changes in cases per million) to emphasise the difference in the number of expected deaths in each scenario.

The results of each outcome under the best-case and worst-case input scenarios (defined in Supplementary S11 – Best-case and worst-case scenario definitions) are given in Supplementary S12 – Best-case and worst-case scenario results. Note that for best-case scenarios, males were used instead of females for the Die from COVID-19 (n18) outcome due to zero deaths in females in the younger age groups and low, rather than no transmission has been used for COVID-19-related outcomes. Delta variant was used in all scenarios to reflect the current situation in Australia. The largest range in values was for the Die from COVID-19 (n18) outcome, comparing the worst-case scenario of unvaccinated males aged greater than 70 years in a high transmission scenario (5391 deaths per million) to the best-case scenario of fully vaccinated females aged 10–19 years in a low transmission scenario (0.008 deaths per million). The outcome nodes that were least sensitive to changes in the input nodes (in absolute numbers) related to the risk of dying from CVST, either background rates (n16) or COVID-19-associated CVST(n19), where the minimum and maximum values varied by less than one death per million (0.024 and 0.445 respectively).

4. Discussion

BNs provided an ideal framework for rapidly prototyping a decision-support tool to consolidate the existing evidence on risks and benefits of the AZ vaccine. The BN for AZ vaccine risk–benefit analysis developed in this study enables users to set scenarios by vaccination status (none, one or two doses of AZ vaccine), age, sex, SARS CoV-2 variant, and community transmission rate. Parameterisation of the model for the Australian population was based on seven separate data sources. By helping the subject matter experts to combine the relevant evidence from different sources, the resulting model can be used to probabilistically estimate and compare risks of adverse outcomes and generate meaningful scenarios for a risk–benefit analysis. The analyses also provide useful information for informing the debate on the risks of vaccine-associated TTS relative to the risks of dying from COVID-19 or associated CVST or PVT.

Using a BN framework to collate and analyse existing evidence highlighted several key messages for informing the risk–benefit analysis for the AZ vaccine. The rate of community transmission was found to be a major moderating influence on the risk–benefit analysis for the AZ vaccine. These results should therefore be considered in the context of the dynamic nature of the COVID-19 pandemic, where transmission rates are likely to change rapidly. It is also worth noting that once someone has symptomatic COVID-19, their risks of dying from COVID-19-associated clots, whilst higher than the risk of dying from vaccine-associated TTS, are orders of magnitude smaller than the risk of dying from COVID-19 itself. Including deaths from COVID-19-associated clots in the Die from COVID-19 (n18) node would therefore not significantly alter the risk–benefit assessment. For example, from Table 2, we can see that the risks of dying from COVID-19 (depending on the scenario) range from 0 to 655.3 deaths per million cases. In comparison, the rates for dying from COVID-19-related CVST and PVT range from 0 to 0.28 and 0 to 3.04 deaths per million cases respectively.

As expected, the model indicated that age group was found to have a higher influence than sex on the risk of dying from COVID-19. While there was large variance in the risk of dying from COVID-19 depending on a person’s age group, sex and vaccination status, there was less variance in the risk of developing and dying from either the background or COVID-19-associated CVST or PVT. Knowing which outcomes were highly variable, as well as which inputs were highly influential on each outcome provides useful information when looking to design a simple decision-support tool for risk–benefit analysis. As an example, these details could be used to customise the inputs required to simplify the tool for the user (e.g. not asking for sex if it isn’t relevant). The same information could also be used to target certain groups, such as focusing efforts on certain age groups, or by those who have only had one vaccine dose.

The objective of the model described here is to provide a risk–benefit analysis to inform the CoRiCal decision-support tool. In results reported previously [29], which specifically compared the population-level risks and benefits of the AZ vaccine based on the BN, vaccination was expected to reduce overall deaths from atypical blood clots in the majority of age groups and transmission scenarios. Specifically, a person was estimated to be 14–28 times more likely to develop COVID-19-associated atypical blood clots if they developed symptomatic infection, than developing TTS after their first dose of AZ vaccine. The difference in the estimated risk of fatalities was even greater. Depending on their age group, the probability of someone with a symptomatic infection dying from COVID-19-associated atypical blood clots was 58–126 higher than the chance of someone dying from TTS after their first dose of the AZ vaccine (although it should be noted that both events are extremely rare).

By adapting the CPTs of the vaccine-associated TTS node (n6) to be the cumulative probability across the two doses, the model can also be used for population-level analysis (Lau et. al. 2021). For example, using scenarios based on this population-level model, it is estimated that in a partially vaccinated population of one million people aged ≥70 years (30% unvaccinated, 35% received only first their dose, 35% received two doses) there would be at most one death expected from vaccine-associated TTS. In the same population, the AZ vaccine would be expected to prevent 25 deaths under a low transmission scenario and more than 3000 deaths under a high transmission scenario.

One key advantage of BNs as modelling tools is the ability to easily adapt the model by editing the CPTs [4]. This feature has proven to be crucial for our study, where new evidence was available weekly and needed to be updated on a regular basis. This feature also leads to a clear, coherent and auditable framework for adapting the model to other contexts. For example, the CPTs can be informed using other country-specific data, particularly for case fatality rates which vary substantially between countries. In Australia, even during the peak of the delta wave in 2021, diagnosed COVID-19 cases were hospitalised and access to mechanical ventilators were generally available where required. Because of this, case fatality rates in Australia are likely to be lower than in other countries where residents have less access to healthcare, or where access to mechanical ventilators became restricted. Updating the model with country-specific case fatality rates would allow more accurate representation of the benefits of vaccination in these areas where the risk of dying from COVID-19 once infected may be different to the Australian scenario.

The transparency of BN models and the model-building process was also critical to building a trusted model. First, having a model that users can interrogate and see why a probability has been estimated for a given scenario allows users to explore and understand the model. Fully documenting the design assumptions and explicitly identifying the sources used for populating the CPTs allows users to understand and evaluate where the probabilities have originated from. Another advantage of BNs was the ability to easily compartmentalise the model into sub-models, and narrow or broaden the scope as required by the addition or removal of nodes. This capability facilitated a prototyping approach that was able to reflect the evolving evidence around the AZ vaccine.

Although the probabilities in the model were derived from empirical studies or publicly available government data, decisions about which data to include (based on availability, robustness and compatibility with other datasets), and how they should be integrated were based on expert judgement. The experts’ role in the process was therefore crucial in both selecting and interpreting the available evidence, such as when choosing between results of different studies. In some cases, it may be more appropriate to include the study with the larger cohort, whereas in other instances, a smaller study that more closely resembles the population being modelled was preferable. In other instances, expert knowledge was required to align the data from different sources. For example, the youngest age group reported for the data on the chance of developing TTS after each vaccine dose was for those under 50 years of age, whereas the model has five ten-year age groups for this same population (0–9 years, 10–19 years, etc.).

An important distinction when designing a model for use in the context of public health is whether it should be interpreted as an individual model or a population model. For example, does 10% chance of infection represent a 10% chance that an individual will become infected, or that 10% of the population will become infected? While in practice these values might be considered to be interchangeable, in this model, the distinction was important in the definition of the node states and the corresponding calculation of the probability of developing TTS. The individual model was designed to represent the probability of TTS after the first and second dose as independent events. To model population-level estimates, e.g., the number of cases of TTS per million people where 35% have had only one dose, and 35% have had two doses, the model should be designed to consider cumulative risk, i.e., those who received two doses were also exposed to the risk associated with the first dose [29]. It is also important to note that individual factors, such as comorbidities and access to healthcare, were not included in the model, so the risk for a particular individual may vary substantially from the population estimate. As with any data model, communicating a clear interpretation of the BN is crucial in a decision-support context where clinicians and the public, as well as policy makers, public health managers and the broader scientific community might misinterpret the model outputs.

Constructing an evidence-based model relying on expert-derived assumptions introduced several challenges for validating the model. Whereas the predictive performance of a data-driven model can be validated using cross-validation, and expert-derived models can be validated based on the opinion of independent experts [6], neither option was suitable for the model presented here. Instead, the logic in the model was validated against independent calculations based on the same evidence. While this did not validate the evidence or the assumptions, it did confirm that the model accurately reflects the information reported in the selected sources.

The modeller-led approach used in this research, although well-suited to a rapid design process, was a deviation from best-practice expert-elicitation processes (for example Wu et. al. [9]; Richards et al. [30]), where the initial conceptual model would ideally be developed by the subject matter experts, facilitated by the modelling experts. Instead, presenting the subject-matter experts with a draft BN structure based on preliminary discussions allowed for an expedited design process. The design process was also facilitated by structured questionnaires used to help the experts collate the evidence into a suitable format that the modelling team could use to populate the CPTs.

A key benefit of BNs is that the interface facilitates model interaction, allowing users to explore different scenarios and develop a deep understanding of how concepts are represented and related in the model. However, a BN interface can be daunting to untrained users and interpreting the probabilities in relatable terms can be difficult. A frontend interface for the model has been created to simplify the inputs and communicate the results in terms of relatable risk [11]. This publicly available decision-support tool can also be updated as the scope of the model is expanded to include other vaccines, adverse events and comorbidities.

While the scope of the present study is currently limited to the AZ vaccine, the process described here can be easily repeated to expand the model to include additional inputs, including other COVID-19 variants or vaccines, demographic variables such as remoteness, or individual variables such as comorbidities. Specific next steps in model development are the inclusion of long COVID as an outcome and the chance of myocarditis from mRNA vaccines. More generally, the process could be extended to consideration of vaccine safety in the context of other health outcomes, particularly in rapidly changing environments.

Although the model has been quantified using data from Australia as well as international data from the UK and US, it has been adapted for the Australian context, and priority has been given to Australian data if available. Using the same process described here, however, the model is easily adaptable to international settings by re-parameterising it with local data where available. Through rapid aggregation, modelling and communication of vaccine-associated risks, the comparative merits of vaccination in target populations can be better understood, leading to improved decision-making by policy makers and public health managers, and increased capability and capacity for clinicians to guide patients to make informed decisions about vaccines.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary material

The following are the Supplementary data to this article: