Abstract
Background
Vaccines were crucial in controlling the Covid-19 pandemic. As more vaccines receive regulatory approval, stakeholders will be faced with several options and must make an appropriate choice for themselves. We proposed a multi-criteria decision analysis (MCDA) framework to guide decision-makers in comparing vaccines for the Indian context.
Methods
We adhered to the ISPOR guidance for the MCDA process. Seven vaccine options were compared under ten criteria. Through three virtual workshops, we obtained opinions and weights from citizens, private-sector hospitals, and public health organisations. Available evidence was rescaled and incorporated into the performance matrix. The final score for each vaccine was calculated for the different groups. We performed different sensitivity analyses to assess the consistency of the rank list.
Results
The cost, efficacy and operational score of the vaccines had the highest weights among the stakeholders. From the six scenario groups, Janssen had the highest score in four. This was driven by the advantage of having a single dose of vaccination. In the probabilistic sensitivity analysis for the overall group, Covaxin, Janssen, and Sputnik were the first three options. The participants expressed that availability, WHO approvals and safety, among others, would be crucial when considering vaccines.
Conclusions
The MCDA process has not been capitalised on in healthcare decision-making in India and LMICs. Considering the available data and stakeholder preference at the time of the study, Covaxin, Janssen, and Sputnik were preferred options. The choice framework with the dynamic performance matrix is a valuable tool that could be adapted to different population groups and extended based on increasing vaccine options and emerging evidence.
*ISPOR – The Professional Society for Health Economics and Outcomes Research.
Keywords: Covid-19, Vaccines, India, Multicriteria decision analysis, MCDA, Decision making, Healthcare
1. Introduction
India’s burden of Covid-19 infections through the pandemic was significant, and attempts to control the disease by non-pharmacological measures such as social distancing and mask-wearing had only been partly successful in controlling the spread of the disease [1], [2]. The initial outbreaks were partially controlled by lockdowns in various states, but this was an extreme measure which had an enormous economic impact on individuals and states. It is noteworthy that in the history of science, there have never been more vaccines developed for a single disease in such a short span of time. Evidence from clinical trials and real-world studies suggested that COVID-19 vaccines were pivotal in modifying the disease course by preventing hospitalisation, morbidity and mortality due to COVID-19 infections [3], [4]. When we initiated this project in September 2021, India had approved seven vaccines for use, and about 21 were under clinical trials [5]. As more vaccines emerge, several vaccine characteristics, such as effectiveness, nature of adverse events, routes of administration, logistical issues and price, will play a role in determining vaccine choices. Stakeholders will have to make informed decisions on the most appropriate vaccine for their populations.
This project aimed to provide a framework of criteria for decision-makers to objectively compare vaccines that may facilitate the selection process. In settings with several competing alternatives with varying characteristics, healthcare decision-making can be complex. Therefore, adopting a structured approach that is explicit, inclusive, transparent, and reproducible approach is imperative. A set of techniques collectively known as Multiple Criteria Decision Analysis is helpful in improving the quality of this decision-making process. This can improve the consistency and legitimacy of decisions. We adopted the good-practice guidelines of the ISPOR task force report for Multiple Criteria Decision Analysis (MCDA) to compare currently approved Covid vaccines in India [5], [6]. The eight steps suggested by the task force were adapted to our context, and we describe them in detail below (Fig. 1 ).
Fig. 1.
Steps in the Mutli-Criteria Decision Analysis (MCDA) process.
2. Defining the Problem
India has a mixed healthcare system with public and private providers and high out-of-pocket expenditure, indicating that the consumer often has a choice in the health services they avail. Although, at the time, vaccines were approved for emergency use, we anticipated that in due course, there would be an abundance of vaccine options. The problem identified was “how will different stakeholders choose the most appropriate vaccine as increasing numbers enter the market”. Hence the populations of interest could be individuals; subgroups such as the elderly, immunocompromised, pregnant women, and children; or even a state or the nation. The initiative was led by public health professionals with expertise in virology, clinical medicine, health economics and epidemiology. Our goal was to develop a performance matrix with qualitative guidance that would function as a choice framework when newer vaccines become available and evidence emerges.
3. Selecting and structuring the criteria
In order to evaluate each vaccine, we considered attributes that were adapted from the SMART Vaccine 2.0 platform [6]. In our study, a domain was defined as a broad theme that housed different criteria; a criterion was a specific standard used for weighting; and attributes were sub-components used in some criteria where composite measures were used (Fig. 2 ).
Fig. 2.
Description of components of the framework.
While examining different domains and criteria, we considered those that would be valuable but finally incorporated the ones where evidence was available. We proposed the following domains - Clinical, Economic, Programmatic, Policy and Societal. These housed various potential criteria. The clinical domain could include criteria such as protection from infection, severe disease, hospitalisation, death frequency and nature of adverse events. The economic domain could involve the cost of the vaccine, the cost of implementing a program, the cost-effectiveness of the vaccine, and the capacity to deliver necessary doses within appropriate timelines. The programmatic domain could have facets that would differ based on the vaccine type, including cold chain requirements, transport convenience, expiry, wastage, training, ease of administration, includability into a national immunisation schedule in the long term, frequency, time interval between doses and route of administration. The Policy domain would include whether the vaccine was indigenously produced, which could influence a public sector procurement, inclusion in the WHO emergency use listing and international acceptance for travel, which were crucial factors to be considered at the time. The social domain would involve the influence of media, medical and scientific fraternity, political bodies, the manufacturer's brand, country of origin and other factors that could influence the perceptions of the stakeholder.
Only criteria for which evidence was available at the time or could be quantified via a Likert rating were selected for incorporation into the performance matrix (Table 2). Ten criteria were finally selected– protection/efficacy, severe adverse events, cost of complete primary vaccination, operational score, time interval between doses, number of doses, route of administration, WHO approval, policy score and social score. During the meetings, the authors presented an initial framework with criteria, and two additional criteria which were WHO approvals and the number of doses, were suggested by stakeholders during the discussions.
Table 2.
The final performance matrix.
|
Domains |
Clinical |
Economic |
Programmatic |
Political |
Social |
References |
|||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Criteria-> | Protection | Severe Adverse events | Cost of complete primary vaccination (INR) | Operational score (range) | Time interval between doses (weeks) | No of doses | Route of administration | WHO approval | Policy score | Social score | |
| Options: | |||||||||||
| 1.Covaxin | 77.8 (95 % CI, 65.2–86.4) | Very low | 2420 | 3.8 (3.7,3.9) | 28 | 2 | IM | 1 | 1 | 0 | [14], [15], [16] |
| 2.Covishield | 67.1 (95 % CI, 52.3–77.3) | low (thrombosis events) | 1520 | 3.35 (3.3,3.4) | 70 | 2 | IM | 1 | 3 | 0 | [17], [18], [19], [20] |
| 3. Sputnik V | 91.1 (95 %, 83.8–95.1) | Very low | 2290 | 3.05(2.8,3.3) | 21 | 2 | IM | 0 | 2 | 0 | [21], [22] |
| 4. Moderna | 94.1 % (95 % CI, 89.3 to 96.8 %) | low (myocarditis) | 3750 | 2.95(2.6,3.3) | 28 | 2 | IM | 1 | 1 | 0 | [23] |
| 5. Pfizer–BioNTech |
87 (95 %, 55–100) | low (myocarditis) |
2930 | 2.6(2.4,2.8) | 21 | 2 | IM | 1 | 1 | 0 | [24], [25] |
| 6. Janssen | 67 (59–73.4) | Very low | 750 | 3.9(3.8,4) | 0 | 1 | IM | 1 | 2 | 0 | [26], [27] |
| 7. ZyCov-D | 66 % | Very low | 1128 | 3.5(3.4,3.6) | 28 | 3 | Needle-free parenteral | 0 | 3 | 0 | [28], [29], [30], [31] |
INR – Indian Rupees; IM- intramuscular.
Likert scales – Serious Adverse events: -very low= (<1 %), low = 2 (<1 % with concerning conditions), moderate = 3 (1–5 %), high = 4 (>5 %).
Route of administration score. IM-4, Needle-free-3, oral-2, Nasal-1.
WHO approval for emergency use: 1 – approved, 0 – under assessment [32].
Policy score- 1- imported. Developed and manufactured overseas, 2- foreign development and indigenously manufactured, 3 – indigenously developed and manufactured.
4. Measuring the performance of the options against the criteria:
We identified the vaccine options as those approved for emergency use by the regulatory authority, the Drug Control General of India, up to September 2021 (Table 1 ). We extracted available evidence for each vaccine for the different criteria and input the values into the performance matrix (Table 2 ). We used protection from symptomatic infection as the clinical criterion. Although severe morbidities such as hospitalisation or death would have been preferable, this data was not available for all options. From clinical trials, post-authorisation and real-world data, the frequency of serious adverse events following immunisation (SAFI) was identified for each vaccine and scored on a Likert scale [7], [8], [9], [10]. The cost-effectiveness of different vaccine options was not available; thus, the maximum retail price for primary vaccination (not including boosters) in the private sector was used as a surrogate for the cost. Programmatic aspects of the vaccines were extracted from the WHO’s guidance and the manufacturer’s manual and these were incorporated as scorable attributes into the operational score [11] (Table 2). This was scored by two of the authors, who were program implementation experts, and the mean score was used in the final performance matrix (Details of Operational score are in the supplementary material). The criteria of WHO approval and the number of doses were recurring themes raised and hence were added after the initial two (citizens and private) stakeholder meetings. Since the social influences were very heterogenous, the scoring of social factors was not done, instead was qualitatively documented.
Table 1.
Details of the vaccine options.
| Vaccine | Scientific name | Country of origin | Manufactured by |
|---|---|---|---|
| Covaxin | BBV152 | India | Bharat Biotech International, India. |
| Covishield (AstraZeneca) AZD1222 |
ChAdOx1 nCoV-19 | UK | Created at Oxford University, UK and manufactured in Serum Institute of India Ltd. |
| Sputnik V | Gam-COVID-Vac | Russia | Manufactured in India by Dr Reddy’s Laboratory |
| Moderna (Spikevax) | mRNA-1273 | USA | Moderna, NIAID, BARDA |
| Pfizer–BioNTech Comirnaty |
BNT162b2 | USA | Pfizer, Inc., and BioNTech |
| Janssen (Jcovden) | Ad26.COV2-S | Netherlands | Johnson & Johnson's subsidiary - Janssen Pharmaceuticals |
| ZyCov-D | DNA SARS-CoV-2 vaccine | India | Cadila Healthcare |
UK - United Kingdom. USA - United States of America. NIAID - National Institute of Allergy and Infectious Diseases at the National Institutes of Health. BARDA - Biomedical Advanced Research and Development Authority.
Since the vaccine cost was in thousands of rupees and efficacy was in percentages, this needed to be rescaled to comparable scores. We proceeded with a global rescaling for the scores. This assumes that for each criterion score, the lowest and highest scores are not from the choices (vaccines) within the framework but from a realistic expectation of cost and performance expected in the broader spectrum of foreseeable options. This is in contrast to local rescaling, where the remapping of values is relative to the scores of the choices in the current list of options [12].
It was acknowledged that other relevant criteria might emerge, which may be added under an appropriate existing or an ‘others’ domain for future use in the reusable performance matrix. We reviewed the matrix to ensure it was as complete as feasible and there was no overlap, redundancy, or preference dependence, as guided by the ISPOR task force [13].
5. Identifying stakeholders
Considering the heterogeneity of the Indian healthcare landscape, the relevant stakeholders in the context were citizens, private sector hospitals and public health organisations. Since the project was conducted during the pandemic, we had to resort to virtual meetings to engage with participants. We sought informed, opinionated, expressive, and technologically-able representatives who could participate in a focus group discussion. The citizens’ stakeholders were contacted on a convenience basis, and individuals known to the extended network of the authors were invited. Similarly, three private sector administrators were requested to nominate two participants each for the private hospital group. Attendees representing the public health organisations included national and state-level vaccine experts, policymakers, and officials from NGOs such as WHO and UNICEF. Two of the authors who were part of government agencies were able to facilitate this consultation. Considering the diversity of the stakeholders, we conducted separate online meetings that provided details appropriate for each group. The distribution of the participants across professions and organisations is displayed in Table 3.
Table 3.
Distribution of all stakeholders.
| Stakeholder group | Number of participants | Occupations |
|---|---|---|
| Citizens | 8 (2 males) | Teacher, Principal, Housewife, Social worker, Lawyer, Student, and Architect. |
| Private hospitals | 6 (4 males) from 3 hospitals | Infectious Disease physicians, Microbiologists, Pediatrician, Managers |
| Public health organisations. | 6 (3 males) | Health Economist, Clinicians, Epidemiologist, WHO and UNICEF vaccine experts, and State health representatives. |
6. Obtaining the stakeholder weights
There were three separate stakeholder meetings conducted between October 2021 and February 2022. Each session was structured in five phases beginning with introductions and the administration of informed consent, followed by an explanation of the problem, an orientation to the MCDA process, its application in vaccine selection, discussions and finally weighting of the criteria, which was done on an online form. During the meeting, the participants explored a range of factors critical to prioritising vaccines that would influence individual or organisational decision-making, and these contributions were qualitatively documented. The weight represented the quantum of importance rendered to each of the criteria. They were instructed to weigh each criterion (using numbers with one decimal point) such that the sum of all weights would equal 10. The names of the vaccines and scores were not known while weighing.
Recurring themes from the group discussions were that trial efficacy, real-world effectiveness data and safety would be crucial factors in deciding. One expressed view was that vaccines with published data from Indian populations would lend more applicability to their context. The vaccine cost was not perceived as a significant determinant by some since it was assumed that vaccination was a one-time investment, and at the time of the study, it was provided for free in the government program. Some clinicians and epidemiologists expressed that if a vaccine could help prevent infection (rather than disease), this would control spread. From the private-sector and citizens’ perspective, the brand of the vaccine was important since some companies had a better reputation for reliability and transparency. Participants observed that the opinions of thought leaders such as family doctors, elders, religious leaders and celebrities, along with information or misinformation on social and traditional media, could significantly influence choice or even adversely promote hesitancy. Similarly, at a country level, the vaccines selected by other nations could provide valuable experience and precedence for adoption. Government officials also mentioned that vaccine manufacturers would need to demonstrate or ensure the scalability of vaccines to be considered as a choice, and the geography of the target population may lend a preference to the least doses needed. These factors would have featured in the ‘other’ criteria, but their scoring and weighting were not performed due to the high variability in their sources and opinions.
The mean weights for each criterion from the three stakeholder meetings are represented in Table 4 . The overall group’s score was the mean of weights from all stakeholders in the three groups. The first sensitivity analysis was performed with an equal distribution of scores among the ten criteria. For the second sensitivity analysis, half the total score of 5 was equally divided among six favourable criteria (where higher scores had a positive association), and the other five was among the four unfavourable criteria (higher scores had a negative association).
Table 4.
Weights assigned to the criteria by stakeholders.
| Criteria | Clinical |
Economic | Programmatic | Political |
Social |
||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Stakeholder groups: | Protection | Severe Adverse events | Cost of complete primary vaccination | Operational score | The time interval between doses (weeks) | No of doses | Route of administration | WHO approval | Policy score | Social score | Total score |
| Citizens | 2.7 | 1 | 2.8 | 0.9 | 0.7 | 0.9 | 0.6 | 0.4 | 10 | ||
| Private hospitals | 2.5 | 2.1 | 0.8 | 1.4 | 1 | 0.9 | 0.5 | 0.8 | 10 | ||
| Public organisations | 1.3 | 1.4 | 0.9 | 1.6 | 0.4 | 1.3 | 1 | 0.9 | 0.5 | 0.7 | 10 |
| Overall | 1.9 | 1.3 | 1.3 | 1.1 | 0.6 | 1.1 | 0.8 | 0.8 | 0.5 | 0.6 | 10 |
| Sens 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 10 |
| Sens 2 | 0.8 | 1.3 | 1.3 | 0.8 | 1.3 | 1.3 | 0.8 | 0.8 | 0.8 | 0.8 | 10 |
Sens 1 – sensitivity 1, where weights were equally distributed between the criteria; Sens 2 – sensitivity 2, where weightage of 5 was equally divided among the positive criteria, and negative implications between the italicised negative criteria. WHO approval and the number of doses were not considered initially and hence not scored by Citizens and private hospital groups.
As seen from Table 4, the highest weights for the citizen’s group were assigned to the cost of vaccination and efficacy/protection; for the private hospital group, they were efficacy and adverse events; and for the public health organisations, they were spread between operational score, adverse events and efficacy.
7. Calculating the aggregate scores for the vaccines
The final total score for a vaccine option was calculated by summing the products of the criteria’s performance score and the mean weights (e.g. explained in supplementary material). This was then sorted in descending order to create a rank list according to each stakeholder group (Table 5 ). The vaccines have also been scored for the overall group and according to sensitivity analyses 1 and 2, as described in the previous section.
Table 5.
The rank order of options for different stakeholders and sensitivity groups.
| Order of Pref. |
Public |
Private |
Citizen |
Overall |
Sens 1 |
Sens 2 |
||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Vaccine | Score | Vaccine | Score | Vaccine | Score | Vaccine | TOTAL SCORE | Vaccine | TOTAL SCORE | Sputnik | TOTAL SCORE | |
| 1 | Janssen | 7.47 | Sputnik | 6.8 | Sputnik | 7.0 | Janssen | 7.3 | Janssen | 7.3 | Janssen | 7.78 |
| 2 | Covaxin | 6.87 | Janssen | 6.8 | Covaxin | 6.8 | Covaxin | 6.75 | Covaxin | 6.86 | Covaxin | 6.84 |
| 3 | Covishield | 5.86 | Covaxin | 6.5 | Janssen | 6.7 | Sputnik | 6.01 | Covishield | 5.87 | Sputnik | 5.78 |
| 4 | Sputnik | 5.72 | Zydus | 6.2 | Zydus | 6.0 | Moderna | 5.78 | Sputnik | 5.48 | Covishield | 5.74 |
| 5 | Moderna | 5.63 | Covishield | 5.6 | Moderna | 5.9 | Covishield | 5.7 | Moderna | 5.3 | Pfizer | 5.4 |
| 6 | Pfizer | 5.44 | Pfizer | 5.4 | Pfizer | 5.5 | Pfizer | 5.64 | Pfizer | 5.28 | Moderna | 5.29 |
| 7 | Zydus | 4.75 | Moderna | 5.4 | Covishield | 5.1 | Zydus | 4.91 | Zydus | 4.94 | Zydus | 5.19 |
As seen in Table 5, Sputnik emerged as the preferred choice for private hospitals and the citizens' group. Janssen was the first choice for the governmental agencies and the sensitivity scenarios and was among the top 3 in the other two groups. Covaxin appeared as one of the top 3 options in all the scenarios. The main drivers of the scores for the overall group are demonstrated in Fig. 3 . Fewer doses and a lower cost of vaccination were contributory for Janssen to score higher than Covaxin.
Fig. 3.
Drivers of the options for the Overall group.
(The drivers for other groups are in the Supplementary material).
8. Dealing with uncertainty
Besides the three stakeholder groups, we had an overall group and two sensitivity analyses to ascertain the variability of the results. We also performed a deterministic sensitivity analysis for the first choice in the overall group by varying the inputs for the upper and lower limits of cost (20 % variation), efficacy and the operational score of Janssen, maintaining the inputs of all others unchanged. This was conducted to assess the validity of the result by changing inputs one at a time. Janssen retained the first position in all cases but was replaced by Covaxin in a hypothetical scenario where the number of doses was increased to two. In the probabilistic sensitivity analysis (PSA), stochastic probability was introduced, and parameters were sampled from their respective distribution (Table 2). Efficacy, within the confidence intervals; operational score, within the range; and cost with 20 % variation, all with a normal distribution, were subjected to a Monte Carlo Simulation of 10,000 iterations, and from this, the percentage (proportion) of being the first choice was obtained (Fig. 4 ). Covaxin was the preferred choice in 34 % of the iterations, followed by Sputnik (27 %) and Janssen (30 %).
Fig. 4.
Probability sensitivity analysis output for All-stakeholders. In the PSA Covaxin was the preferred choice in 34 % of the iterations, followed by Sputnik (27 %) and Janssen (30 %).
9. Discussing the findings
This study synthesises evidence and weighted opinions of three stakeholder groups reflecting their preferences in prioritising a Covid-19 vaccine. The public organisations’ stakeholders prioritised operational scores highly, considering the importance of distribution and training for delivering the vaccines, whereas the private hospitals and citizen groups prioritised cost and efficacy higher. Regarding the final vaccine score, even though it did not have the highest protection rate, the reasons for Janssen having the highest score were the single dose and the lowest cost of primary vaccination. Covaxin scored high due to the policy score, comparatively high protection rate and WHO approval which was absent for Sputnik. Despite being indigenously produced, the reason for the Covishield vaccine to have a lower score was the serious adverse events score, which was also present for the mRNA vaccines (Moderna and Pfizer). Although the mRNA vaccines had among the highest protection rates, the higher cost was another reason that led to lower scores. Zydus had the lowest scores due to the unaccustomed, needle-free route, higher number of doses and lack of WHO approval at the time of assessment.
The rank list of the vaccines and the drivers for each stakeholder group demonstrate the key factors which explain the vaccine's performance. It also illustrates how the scores may change if some factors are modified, for example, changing the cost, obtaining WHO approval or manufacturing the vaccine within the country to boost the policy score. It was evident that there may be a dependence between cost, frequency and duration of vaccination. However, since these were all aspects considered crucial during the discussions, it was decided to place them as individual criteria.
There has been considerable work using the MCDA process in the covid-19 setting to guide decision-makers in prioritising patients who receive vaccinations, ensuring equitable healthcare access and screening innovations to respond more efficiently during the pandemic [33], [34], [35]. In the field of vaccines, a study by Son et al. used an MCDA approach to study the benefit-risk profile of four Covid vaccines in Korea and found that the mRNA vaccines had a good balance [36]. There were also similar projects which used the MCDA approach to develop a framework to guide vaccine selection for Latin America, Egypt and Saudi Arabia [37], [38], [39]. These studies used different MCDA methods but have identified similar criteria and involved expert inputs. Our project has engaged with more stakeholders, incorporated more criteria within domains and demonstrated its application for seven vaccine candidates.
In the Indian context during the usual settings, before the introduction of a vaccine into the Indian immunisation programme, evidence on its efficacy, safety, long-term protection, disease burden, vaccine acceptance, post-marketing surveillance data, feasibility in the program, cold-chain space requirement, target age group, supply chain logistics and cost-effectiveness of the vaccine would be assessed by the National taskforce against immunisation (NTAGI) [40]. However, during the pandemic, none of the products had received full licensure at the time of introduction, although they were approved for restricted use in an emergency setting. Available data was reviewed based on safety, short-term efficacy and immunogenicity. During the pandemic, the whole vaccination program was regulated by the government, and booster doses were provided by the private sector [41], [42]. Hence Covishield (82 %) was most utilised, followed by Covaxin (17 %), both Indian manufactured vaccines, and Sputnik (<1 %) was provided in the private sector [43]. This program was operationalised in a period of scarcity where indigenous vaccines were prioritised and hence didn’t reflect the list in this project. However, in a state of steady supply post-pandemic, the principles used here could be beneficial.
There were several limitations to our study. The convenience sampling of participants who could engage in a virtual meeting may not represent the priorities of the wider stakeholder populations, and this would ideally need broader systematic sampling strategies. There were constraints to the MCDA process where some of the participants found the concept of weighting unclear and were critical of the complexity of the process. This could have been resolved by a second round of interactions where results, drivers and sensitivities could have been explained to stakeholders. We did not elicit stakeholder preference for the change of scores within a criterion, and a linear approximation assumption was employed. We added two criteria after initial stakeholder meetings with the citizens and private-hospitals groups and were unable to obtain their revised score. There were also generic limitations to the MCDA process where vital factors may not have been considered or are differently valued at the time of the meeting in comparison to real-life decision-making, leading to discordance between explicit weighing and implicit decision-making. In one session, outspoken participants may have overtly influenced the discussion, preference, and weighting, and this may have been mitigated by patient moderation, ensuring participation from all members. Although we had these limitations, we believe this framework is a versatile tool for flexible decision-making with inputs from diverse stakeholders whose inputs are otherwise not sought.
This choice framework has several policy implications. In situations where rapid decisions are to be made when data are scarce, the MCDA adopts a transparent and explicit process. In this case, after clinical considerations and approval by the Drug Controller General of India, a screening phase would be crucial to assess if there is an adequate vaccine supply for the population of interest. Each stakeholder group would need to use the performance matrix as a template and consider whether there are additional criteria that need to be considered. For instance, from the public sector perspective, the operational factor of delivering the vaccine, including the geography, access to a region, and equity, could be incorporated. The weights for each stakeholder group would be context-specific and change as more information, such as social influence, the inclusion of boosters etc., emerge into the criteria. Hence the performance matrix would be dynamic. This can also be adapted to emergency settings where there could be a supply–demand mismatch and could be modified for specific subgroups such as the elderly, pregnant women or children. The sensitivity analysis demonstrates how the ranking order changes in different scenarios, facilitating policymakers to adopt strategies fitting to changing scenarios. Thus, the final purpose of the framework would be to function as a guide to consider and expand criteria, incorporate evolving evidence and proceed with appropriate decision-making, which can enable reproducibility and communication to respective stakeholders.
10. Conclusions
The MCDA process is a collaborative interdisciplinary tool utilising available evidence for explicit and accountable decision-making, which has not been adequately capitalised for healthcare decision-making in India. Considering the available data and stakeholder preference at the time of the study, Janssen, Sputnik, and Covaxin were preferred options. The dynamic performance matrix that was developed is a versatile tool that could be adapted to different population groups and extended based on expanding vaccine options and emerging evidence.
CRediT authorship contribution statement
Tarun K. George: Conceptualisation, Study design, Literature search, Data collection, Interpretation, Writing and revisions of manuscript. Nayana P. Nair: Study design, Literature search, Data collection, Interpretation, Writing original manuscript. Awnish Kumar Singh: Literature search, Data collection, Interpretation, Writing original manuscript. A. Dilesh Kumar: Literature search, Data collection, Interpretation, Revision of manuscript. Arup Deb Roy: Conceptualization, Literature search, Data collection, Writing of original manuscript. Varshini Neethi Mohan: Investigation, Validation, Interpretation, Manuscript writing and revision. Gagandeep Kang: Conceptualisation, Data collection, Interpretation, Review of manuscript, Supervision.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
We would like to acknowledge and thank Praveen Thokala, Anna Vassal, Saudamini Dabak and Hwee Lin for their support and advise during the course of the project. Also, our sincere gratitude to the anonymous reviewers, who's input considerably strengthened the final manuscript.
Footnotes
Supplementary data to this article can be found online at https://doi.org/10.1016/j.vaccine.2023.04.062.
Appendix A. Supplementary material
The following are the Supplementary data to this article:
Data availability
No data was used for the research described in the article.
References
- 1.Talic S., Shah S., Wild H., Gasevic D., Maharaj A., Ademi Z., et al. Effectiveness of public health measures in reducing the incidence of covid-19, SARS-CoV-2 transmission, and covid-19 mortality: systematic review and meta-analysis. BMJ. 2021 Nov;18(375) doi: 10.1136/bmj-2021-068302. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Ayouni I., Maatoug J., Dhouib W., Zammit N., Fredj S.B., Ghammam R., et al. Effective public health measures to mitigate the spread of COVID-19: a systematic review. BMC Public Health. 2021 May 29;21(1):1015. doi: 10.1186/s12889-021-11111-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Suthar A.B., Wang J., Seffren V., Wiegand R.E., Griffing S., Zell E. Public health impact of covid-19 vaccines in the US: observational study. BMJ. 2022 Apr;27(377) doi: 10.1136/bmj-2021-069317. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Watson O.J., Barnsley G., Toor J., Hogan A.B., Winskill P., Ghani A.C. Global impact of the first year of COVID-19 vaccination: a mathematical modelling study. Lancet Infect Dis. 2022 Sep 1;22(9):1293–1302. doi: 10.1016/S1473-3099(22)00320-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.India – COVID19 Vaccine Tracker [Internet]. [cited 2023 Mar 3]. Available from: https://covid19.trackvaccines.org/country/india/.
- 6.McCormick B.J.J., Waiswa P., Nalwadda C., Sewankambo N.K., Knobler S.L. SMART Vaccines 2.0 decision-support platform: a tool to facilitate and promote priority setting for sustainable vaccination in resource-limited settings. BMJ Glob. Health 1. 2020 Nov;5(11) doi: 10.1136/bmjgh-2020-003587. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Beatty A.L., Peyser N.D., Butcher X.E., Cocohoba J.M., Lin F., Olgin J.E., et al. Analysis of COVID-19 vaccine type and adverse effects following vaccination. JAMA Netw Open. 2021 Dec 22;4(12) doi: 10.1001/jamanetworkopen.2021.40364. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Lane S., Yeomans A., Shakir S. Reports of myocarditis and pericarditis following mRNA COVID-19 vaccination: a systematic review of spontaneously reported data from the UK, Europe and the USA and of the scientific literature. BMJ Open. 2022 May 1;12(5) doi: 10.1136/bmjopen-2021-059223. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Fraiman J., Erviti J., Jones M., Greenland S., Whelan P., Kaplan R.M., et al. Serious adverse events of special interest following mRNA COVID-19 vaccination in randomised trials in adults. Vaccine. 2022 Sep 22;40(40):5798–5805. doi: 10.1016/j.vaccine.2022.08.036. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Kant A., Jansen J., van Balveren L., van Hunsel F. Description of frequencies of reported adverse events following immunisation among four different COVID-19 vaccine brands. Drug Saf. 2022;45(4):319–331. doi: 10.1007/s40264-022-01151-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.COVID-19 vaccination: supply and logistics guidance [Internet]. [cited 2022 May 7]. Available from: https://www.who.int/publications-detail-redirect/who-2019-ncov-vaccine-deployment-logistics-2021-1.
- 12.Monat J.P. The benefits of global scaling in multi-criteria decision analysis. Judgm Decis Mak. 2009;4(6):17. [Google Scholar]
- 13.Marsh K, IJzerman M, Thokala P, Baltussen R, Boysen M, Kaló Z, et al. Multiple Criteria Decision Analysis for Health Care Decision Making—Emerging Good Practices: Report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016 Mar 1;19(2):125–37. [DOI] [PubMed]
- 14.Zare H., Rezapour H., Mahmoodzadeh S., Fereidouni M. Prevalence of COVID-19 vaccines (Sputnik V, AZD-1222, and Covaxin) side effects among healthcare workers in Birjand city. Iran Int Immunopharmacol. 2021 Dec;101(Pt B) doi: 10.1016/j.intimp.2021.108351. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Ella R., Reddy S., Blackwelder W., Potdar V., Yadav P., Sarangi V., et al. Efficacy, safety, and lot-to-lot immunogenicity of an inactivated SARS-CoV-2 vaccine (BBV152): interim results of a randomised, double-blind, controlled, phase 3 trial. Lancet. 2021 Dec 11;398(10317):2173–2184. doi: 10.1016/S0140-6736(21)02000-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.India’s “Covaxin” vaccine shows high efficacy against COVID-19 infections in phase 3 trial [Internet]. [cited 2022 Jun 10]. Available from: https://www.gavi.org/vaccineswork/indias-covaxin-vaccine-shows-high-efficacy-against-covid-19-infections-phase-3.
- 17.Wise J. Covid-19: New data on Oxford AstraZeneca vaccine backs 12 week dosing interval. BMJ. 2021 Feb;3(372) doi: 10.1136/bmj.n326. [DOI] [PubMed] [Google Scholar]
- 18.Voysey M., Clemens S.A.C., Madhi S.A., Weckx L.Y., Folegatti P.M., Aley P.K., et al. Safety and efficacy of the ChAdOx1 nCoV-19 vaccine (AZD1222) against SARS-CoV-2: an interim analysis of four randomised controlled trials in Brazil, South Africa, and the UK. Lancet. 2021 Jan 9;397(10269):99–111. doi: 10.1016/S0140-6736(20)32661-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Knoll M.D., Wonodi C. Oxford–AstraZeneca COVID-19 vaccine efficacy. Lancet. 2021 Jan 9;397(10269):72–74. doi: 10.1016/S0140-6736(20)32623-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Marathe S. Covishield - Fact sheet Eng-Hindi insert Ver 1.cdr. Fact Sheet. :2.
- 21.Logunov D.Y., Dolzhikova I.V., Shcheblyakov D.V., Tukhvatulin A.I., Zubkova O.V., Dzharullaeva A.S., et al. Safety and efficacy of an rAd26 and rAd5 vector-based heterologous prime-boost COVID-19 vaccine: an interim analysis of a randomised controlled phase 3 trial in Russia. Lancet Lond Engl. 2021 Feb 20;397(10275):671–681. doi: 10.1016/S0140-6736(21)00234-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Jones I., Roy P. Sputnik V COVID-19 vaccine candidate appears safe and effective. Lancet. 2021 Feb 20;397(10275):642–643. doi: 10.1016/S0140-6736(21)00191-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Baden L.R., El Sahly H.M., Essink B., Kotloff K., Frey S., Novak R., et al. Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine. N Engl J Med. 2021 Feb 4;384(5):403–416. doi: 10.1056/NEJMoa2035389. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Polack F.P., Thomas S.J., Kitchin N., Absalon J., Gurtman A., Lockhart S., et al. Safety and efficacy of the BNT162b2 mRNA Covid-19 Vaccine. N Engl J Med. 2020 Dec 31;383(27):2603–2615. doi: 10.1056/NEJMoa2034577. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Tartof S.Y., Slezak J.M., Fischer H., Hong V., Ackerson B.K., Ranasinghe O.N., et al. Effectiveness of mRNA BNT162b2 COVID-19 vaccine up to 6 months in a large integrated health system in the USA: a retrospective cohort study. Lancet. 2021 Oct 16;398(10309):1407–1416. doi: 10.1016/S0140-6736(21)02183-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Safety and Efficacy of Single-Dose Ad26.COV2.S Vaccine against Covid-19 | NEJM [Internet]. [cited 2022 May 7]. Available from: https://www.nejm.org/doi/full/10.1056/NEJMoa2101544.
- 27.Sadoff J., Gray G., Vandebosch A., Cárdenas V., Shukarev G., Grinsztejn B., et al. Safety and efficacy of single-dose Ad26.COV2.S vaccine against Covid-19. N Engl J Med. 2021 Jun 10;384(23):2187–2201. doi: 10.1056/NEJMoa2101544. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Zydus reports positive Phase III data of first DNA Covid-19 vaccine [Internet]. Clinical Trials Arena. 2021 [cited 2022 May 7]. Available from: https://www.clinicaltrialsarena.com/news/zydus-covid-vaccine/.
- 29.Momin T, Kansagra K, Patel H, Sharma S, Sharma B, Patel J, et al. Safety and immunogenicity of a DNA SARS-CoV-2 vaccine (ZyCoV-D): Results of an open-label, non-randomised phase I part of phase I/II clinical study by intradermal route in healthy subjects in India. eClinicalMedicine [Internet]. 2021 Aug 1 [cited 2022 May 7];38. Available from: https://www.thelancet.com/journals/eclinm/article/PIIS2589-5370(21)00300-X/fulltext. [DOI] [PMC free article] [PubMed]
- 30.ZyCoV-D COVID-19 Vaccine — Precision Vaccinations [Internet]. [cited 2022 Jun 10]. Available from: https://www.precisionvaccinations.com/vaccines/zycov-d-covid-19-vaccine.
- 31.Efficacy, safety, and immunogenicity of the DNA SARS-CoV-2 vaccine (ZyCoV-D): the interim efficacy results of a phase 3, randomised, double-blind, placebo-controlled study in India - ScienceDirect [Internet]. [cited 2022 Jun 10]. Available from: https://www.sciencedirect.com/science/article/pii/S0140673622001519. [DOI] [PMC free article] [PubMed]
- 32.Covid-19 Vaccines [Internet]. [cited 2023 Mar 9]. Available from: https://www.who.int/teams/regulation-prequalification/eul/covid-19.
- 33.Roy A., Kar B. A multi-criteria decision analysis framework to measure equitable healthcare access during COVID-19. J Transp Health. 2022 Mar;1(24) doi: 10.1016/j.jth.2022.101331. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Ruggeri M., Cadeddu C., Roazzi P., Mandolini D., Grigioni M., Marchetti M. Multi–criteria–decision–analysis (MCDA) for the horizon scanning of health innovations an application to COVID 19 emergency. Int J Environ Res Public Health. 2020 Jan;17(21):7823. doi: 10.3390/ijerph17217823. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Chaker Masmoudi H, Rhili A, Zamali I, Ben Hmid A, Ben Ahmed M, Khrouf MR. Multi-Criteria Decision Analysis to Prioritise People for COVID-19 Vaccination When Vaccines Are in Short Supply. Front Health Serv [Internet]. 2022 [cited 2022 Jun 28];2. Available from: https://www.frontiersin.org/article/10.3389/frhs.2022.760626. [DOI] [PMC free article] [PubMed]
- 36.Son K.H., Kwon S.H., Na H.J., Baek Y., Kim I., Lee E.K. Quantitative benefit-risk assessment of COVID-19 vaccines using the multi-criteria decision analysis. Vaccines. 2022 Dec;10(12):2029. doi: 10.3390/vaccines10122029. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Mohamed R. Hybrid Multi-Criteria Decision Making Approach to Ranking COVID-19 Vaccines. Int J Comput Appl. 183(34):7.
- 38.Espinoza M.A., Guzman J., Soto J., Hernandez G., Guarin D., Boers Trilles V., et al. PIN63 choosing the right COVID-19 vaccine: a multiple stakeholder multicriteria decision analysis for the assessment of available vaccines in Latin America. Value Health. 2021 Jun;24:S117. [Google Scholar]
- 39.Abdelwahab S.F., Issa U.H., Ashour H.M. A novel vaccine selection decision-making model (VSDMM) for COVID-19. Vaccines. 2021 Jul;9(7):718. doi: 10.3390/vaccines9070718. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.John T.J. India’s National Technical Advisory Group on Immunisation. Vaccine. 2010 Apr;28:A88–A90. doi: 10.1016/j.vaccine.2010.02.041. [DOI] [PubMed] [Google Scholar]
- 41.notice15april21.pdf [Internet]. [cited 2023 Mar 17]. Available from: https://cdsco.gov.in/opencms/export/sites/CDSCO_WEB/Pdf-documents/notice15april21.pdf.
- 42.Singh K., Verma A., Lakshminarayan M. India’s efforts to achieve 1.5 billion COVID-19 vaccinations: a narrative review. Osong Public Health Res Perspect. 2022 Oct;13(5):316–327. doi: 10.24171/j.phrp.2022.0104. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.India COVID-19 Vaccine Tracker [Internet]. [cited 2022 Jun 27]. Available from: https://geographicinsights.iq.harvard.edu/IndiaVaccine.
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
No data was used for the research described in the article.




