Applications of discrete choice experiments in COVID-19 research: Disparity in survey qualities between health and transport fields

Abstract

Published choice experiments linked to various aspects of the COVID-19 pandemic are analysed in a rapid review. The aim is to (i) document the diversity of topics as well as their temporal and geographical patterns of emergence, (ii) compare various elements of design quality across different sectors of applied economics, and (iii) identify potential signs of convergent validity across findings of comparable experiments. Of the N = 43 published choice experiments during the first two years of the pandemic, the majority identifies with health applications (n = 30), followed by transport-related applications (n = 10). Nearly 100,000 people across the world responded to pandemic-related discrete choice surveys. Within health applications, while the dominant theme, up until June 2020, was lockdown relaxation and tracing measures, the focus shifted abruptly to vaccine preference since then. Geographical origins of the health surveys were not diverse. Nearly 50% of all health surveys were conducted in only three countries, namely US, China and The Netherlands. Health applications exhibited stronger pre-testing and larger sample sizes compared to transport applications. Limited signs of convergent validity were identifiable. Within some applications, issues of temporal instability as well as hypothetical bias attributable to social desirability, protest response or policy consequentiality seemed likely to have affected the findings. Nevertheless, very few of the experiments implemented measures of hypothetical bias mitigation and those were limited to health studies. Our main conclusion is that swift administration of pandemic-related choice experiments has overall resulted in certain degrees of compromise in study quality, but this has been more so the case in relation to transport topics than health topics.

1. Introduction

COVID-19 has had a profound impact on the lives of everyone around the world since early 2020. Due to the rarity of global pandemics, researchers have been keen to analyse the behavioural impact of COVID-19 using choice experiments and choice modelling in various domains. COVID-19 has significant consequences for public health and has strained hospitals and people working in health care. To slow infection rates, many governments have imposed restrictions and lockdowns to reduce mobility. It is, therefore, no surprise that choice modellers have mostly analysed health-related and travel-related choice behaviour. What is surprising is the speed with which choice experiments were designed and implemented, with the first surveys containing choice experiments sent out mere weeks after the start of the pandemic.

In this review, we summarise choice experiments related to COVID-19 that have appeared in the literature with the aim to document the diversity of topics and their emergence, to compare experimental design quality across different applied economics areas, and to identify whether similar choice experiments exhibit convergent validity.

2. Methods

The focus of the analysis is on peer-reviewed discrete choice experiments published in the scholarly literature that are related to any aspect of the COVID-19 pandemic. In order to obtain this set of references, we modified the previously established search query of Haghani et al. (2021a), formulated to source and track the overall literature of discrete choice modelling. One application of such query string is that it can be modified to produce specific subsets of interest within the literature. We did so, in this case, by combining the query string with a combination of terms that characterise pandemic-related studies. In specific terms, we combined the said query, using Boolean operator AND, with the string (“Coronavirus” OR “COVID-19” OR “SARS-COV-2”). See details of this search query in Appendix.

The search was conducted at the end of 2021 in the Web of Science. An initial set of slightly more than 100 peer reviewed articles that were detected by this search were screened to filter those that specifically report on a discrete choice experiment. Studies that used non-experimental choice data, i.e., revealed choice, were excluded. The screening was carried out by the first author based on the main inclusion criterion of identifying studies disseminated as peer reviewed journal articles and reporting on a discrete choice experiment in a context related to the COVID-19 pandemic. After screening, a total of N = 43 articles met the inclusion criteria. Consistent with Haghani et al. (2021b) and Haghani et al. (2021c), the articles were further classified to the subdomains of health, transport, environmental and marketing/consumer studies. This determination was made based on the nature and topic of the choice experiment reported in the paper. This processing identified n = 30 experiments related to health, n = 10 experiments related to transport, and n = 2 and n = 1 experiments respectively related to business and environmental topics.

The text of each paper in the dataset was fully examined and the following information was extracted for each: (1) the country or countries were the experiment was conducted, (2) the topic of the choice experiment, (3) the list of attributes, (4) the period of time where the experiment was undertaken, (5) the sample size, whether the sample represented of a generic population or a specific population within that country, (6) whether any polit study or focus group interview was conducted during the design process, (7) the type of design and number of choice sets, (8) whether the experiment design implemented any measure of testing internal validity, (9) whether the experiment took any explicit measure to mitigate potential hypothetical bias, and (10) the key findings. Information on item (10) was subsequently used as the basis for assessing convergent validity of studies that were conducted on same/comparable topics. Since the number of studies in contexts of marketing or environment was only a few, most of our comparisons will be between health and transport-related experiments.

3. Results

3.1. Geographical distribution of choice experiments

Pandemic-related choice experiments were reported in a total of 25 countries around the world. The distribution of the origin of these experiments in health and transport has been visualised in Fig. 1 . The distribution of these experiments across geographical regions is rather distinctly different across health and transport. Sixty percent of transport choice experiments were conducted in Asian countries, whereas USA as well as Australia and European countries had a more noticeable representation in health experiments. In transport, only India and Italy reported on multiple choice experiments (n = 2 for each) and the rest of the countries (Iran, USA, South Korea, China, UK, Pakistan, Bangladesh) only reported one experiment each. This makes a total of 9 countries. In comparison, there were twenty countries involved in conducting health-related experiments. Nearly fifty percent of health-related experiments, however, were concentrated in three countries, USA (n = 10), China (n = 8) and The Netherlands (n = 4), followed by Australia, France and UK with n = 3 reported experiment in each. It should also be noted that, one single experiment reported by Hess et al. (2022) was administered in 18 countries. If we exclude that study, then the percentage of experiments conducted in the top three countries in that list would be nearly 60% as opposed to 50%. The vast majority of health experiments targeted the population of only a single country. Other than the experiment of Hess et al. (2022), only one experiment in health, that of Liu et al. (2021a), has been extended across more than one country (USA and China). This is also the case within the transport experiments that were mostly limited to the population of a single country, except for the survey of Zannat et al. (2021) that drew samples from both Bangladesh and Pakistan. Both marketing-related experiments were conducted in the USA (Grashuis et al., 2020; Park and Lehto, 2021), whereas the sample for the single reported environmental study was drawn from the population of Canada, Norway and Scotland.

Fig. 1.

The number of health-related and transport-related discrete choice surveys in the context of COVID-19 conducted across the world during the first two years of the pandemic. The temporal month-by-month emergence of these experiments have been visualised and are accessible via the online supplementary material of this article.

The vast majority of the reported surveys composed samples that were representative of generic populations as opposed to specific cohorts. Exceptions to these are a few studies that targeted specific segments of the population, namely Huang et al. (2021) approached clinicians, Li et al. (2021b) and Ceccato et al. (2021) approached university students, Luevano et al. (2021) approached health-care workers, Manca et al. (2021) approached frequent flyers, and Park and Lehto (2021) approached hotel guests.

3.2. Common themes in health-related choice experiments

Examination of the topics of health-related experiments revealed that the topic of these experiments could each identify with one of these five major themes. This includes, in the order of their frequency, preferences for (1) vaccine attributes (Borriello et al., 2021; Craig, 2021; Dong et al., 2020; Eshun-Wilson et al., 2021b; Hess et al., 2022; Huang et al., 2021; Kreps et al., 2020; Leng et al., 2021; Li et al., 2021b; Liu et al., 2021a; Luevano et al., 2021; McPhedran and Toombs, 2021; Schwarzinger et al., 2021) (2) non-pharmaceutical preventative measures such as stay at home, social distancing, surveillance and contact tracing policies (Degeling et al., 2020; Eshun-Wilson et al., 2021a; Genie et al., 2020; Jonker et al., 2020; Li et al., 2021a; Mouter et al., 2021; Rad et al., 2021), (3) restriction lifting/relaxation policies and exit strategies (Chorus et al., 2020; Krauth et al., 2021; Reed et al., 2020), (4) allocation of limited medical resources and the associated dilemmas (e.g., ICU capacities, vaccines) (Gijsbers et al., 2021; Luyten et al., 2021; Michailidou, 2021) and (5) diagnostic and testing methods (Katare et al., 2022; Liu et al., 2021b). Fig. 2 shows the frequency of each of these themes within our dataset of references. Studies on preferences for and uptake of vaccines constituted the dominant theme in health, followed by issues related to public preferences for non-pharmaceutical preventative policies, i.e., stay-at-home, surveillance and contact tracing policies.

Fig. 2.

Frequency of major common themes in health-related discrete choice experiments in the context of COVID-19 pandemic.

https://protect-au.mimecast.com/s/_d4rCgZ0N1iAowvKXs2E0BN?domain=unsw-my.sharepoint.com

https://protect-au.mimecast.com/s/oh3KCjZ1N7inBGzDLH74ulj?domain=unsw-my.sharepoint.com.

3.3. Common themes in transport-related choice experiments

Within the transport domain, only two recurring themes were identifiable. This includes preferences for commute mode choice during the pandemic (Ceccato et al., 2021; Luan et al., 2021; Scorrano and Danielis, 2021; Xu et al., 2021) as well as attitudes towards public transport use and passenger crowding perception in public transport vehicles (Aaditya and Rahul, 2021; Aghabayk et al., 2021; Park and Lehto, 2021). Three individual studies were also detected that did not identify with either of the two major categories. This includes the experimental survey of Cherry et al. (2021) on willingness to pay for travel time saving and reliability, the survey of Manca et al. (2021) on attitude towards air travel, and, the survey of Zannat et al. (2021) on shopping trip behaviour. Fig. 3 visualises these relative frequencies.

Fig. 3.

Frequency of major common themes in transport-related discrete choice experiments in the context of COVID-19 pandemic.

3.4. Topics choice experiments in marketing and environmental sciences

Two pandemic-related choice experiments were attributable to topics that are typically studied in marketing and consumer choice domain. This included the survey of Grashuis et al. (2020) on grocery shopping preferences and that of Park and Lehto (2021) on hotel choice during the pandemic. Both surveys were conducted in the USA and in the early stages of the pandemic (i.e., May and March 2020, respectively).

Only one study was attributable to the environmental domain and that is the work of Hynes et al. (2021). The topic of the survey, per se, is not related to the pandemic, and in fact, is a survey that was conducted during the years prior to the pandemic.1 However, authors repeated the survey during the early stages of the pandemic in order to test potential effects of COVID-19 and the temporal stability of preferences and willingness to pay for environmental benefits.

Readers can see the list of all qualified studies along with their major category and the theme within their category in Fig. 4 . Two supplementary videos of this article also demonstrate the temporal sequence and emergence of these choice experiments within health and transport domains and across various geographical regions. The general theme of each survey, their time and location of implementation, their sample size, as well as their key finding(s) are the information provided by the supplementary videos.

Fig. 4.

Pandemic-related discrete choice experiments across various divisions of applied economics.

3.5. Temporal trends in pandemic-related choice experiments

Fig. 5 shows a timeline of all health- and transport-related choice experiments based on the month in which the data collection started. The colour-coding is consistent with those of Fig. 2, Fig. 3, Fig. 4 and demonstrates the theme of each experiment. This analysis shows that, in the health domain, during the early stages of the pandemic, i.e., the first half of 2020, issues surrounding non-pharmaceutical preventative measures as well as restriction relaxation constituted the dominant foci of the experiments. Of the nine experiments conducted before June 2020, seven were related to these two themes. This is understandable, in that, in the absence of pharmaceutical solutions to the pandemic during that period of time, the focus of policy makers around the world was on implementation of policies such as lockdowns, travel restriction, physical distancing, contact tracing as well as planning for exit strategies from such measures. With the prospect of COVID-19 vaccines becoming a reality, however, health economists shifted their focus towards issues related to vaccine preference and uptake since June 2020. An exception to this trend is the experiment of Borriello et al. (2021) that investigated preferences of Australians for vaccine attributes as early as March 2020. The longest duration of data collection for these experiments are reported in Hess et al. (2022) whose overall data collection process on the topic of vaccine uptake, conducted across 18 nations, was ongoing between July 2020 and March 2021. Towards the end of 2020 and early 2021, vaccines became the dominant topic in choice experiments. Health-related choice experiments conducted since November 2020 are exclusively on the topic of vaccines.

Fig. 5.

Temporal sequence of discrete choice surveys conducted on health and transport-related topics in the context of COVID-19 pandemic.

Choice experiments in the transport sector do not exhibit a distinct temporal pattern, as observed with health studies.

3.6. Comparison of sample sizes across health and transport-related choice experiments

Stark contrasts were observable between the health and transport domains in terms of the sample sizes of their choice experiments. The average and median sample size of health-related experiments are respectively 2 889 and 1 915. Whereas the corresponding figures for transport experiments are 1 823 and 509. Pandemic-related choice experiments in the domain of health have overall been conducted using noticeably larger samples. This difference is visually observable in the form of a box-and-whisker plot in Fig. 6 . This difference may be explained by differences in available budget in the two disciplines.

Fig. 6.

Distribution of sample sizes of choice experiments conducted in the health and transport domains (on the left) and visualisation of individual studies (replica of Fig. 4) proportional to their sample sizes (on the right).

3.7. Comparison of piloting and use of focus-groups across health and transport-related choice experiments

In addition to the sample size, the higher prevalence of piloting and the use of focus groups in experiment design was far more notable in health-related surveys. Studies in health contexts conducted as early as March 2020 reported on conducting focus-group interviews during their design process (Rad et al., 2021). Other examples are the survey of Leng et al. (2021) (July 2020) and that of Dong et al. (2020). Seven experiments in total reported on conducting a pilot study, and those were all exclusively studies in the health domain (i.e., 23% of the health studies reported on piloting their survey and 0% in transport). Li et al. (2021a) also reported on piloting the design with faculty members, graduate and undergraduate students, survey design specialists and members of public. Mouter et al. (2021) collected a pilot sample of 80 respondents and also interviewed six experts (for a survey conducted in May 2020). Jonker et al. (2020) reported on two rounds of pilot testing (with 238 and 260 respondents) while their main survey was conducted in April 2020. Huang et al. (2021) pre-tested the survey on a pilot sample of 20 participants (main survey conducted in March 2021). The survey of Eshun-Wilson et al. (2021b) was piloted with a sample of 100 anonymous participants. Eshun-Wilson et al. (2021b); Liu et al. (2021a) also reported on recruiting a pilot sample, though the size of the sample was not reported. Even a number of health-related studies that skipped the piloting phase have acknowledged that explicitly. For example, Reed et al. (2020) conducted their survey in May 2020 and have mentioned in their paper that they prioritised expediency at the cost of some of the standard procedures of choice experiment design, and that they did not pilot the survey and rather relied on informal feedback from a sample of colleagues and friends. Similarly, Li et al. (2021a) who conducted their experiment on August 2020 acknowledged that they did not seek any focus group feedback due to time restriction. The elements of pre-testing, focus groups and piloting, however, appear to be missing from most transport surveys. The more rigorous pre-testing and piloting of health-related studies may be driven by the various guidelines and standard procedures that exist for conducting a choice experiment in health.

3.8. Comparison of design methods across health and transport choice experiments

Of the 30 choice experiments in health, half used an experimental design that was optimised for efficiency, orthogonality was a criterion that was also frequently used (n = 10). A similar finding is observed in the transport context, where five out the ten studies reported the use of an efficient experimental design, while others mostly reported orthogonality as the main design criterion. Noteworthy is that, in three choice experiments in transport, the attribute levels are pivoted around individual reference levels (Cherry et al., 2021; Manca et al., 2021; Scorrano and Danielis, 2021). Four studies do not provide information about how the choice sets were created, two amongst health studies (Katare et al., 2022; Michailidou, 2021) and two amongst transport studies (Aaditya and Rahul, 2021; Scorrano and Danielis, 2021).

3.9. Comparison of internal validity measures across health and transport choice experiments

Two of the surveyed studies in total reported on implementing measures of internal validity. Both studies were in health contexts. In the experiment of Jonker et al. (2020), one out of the fifteen choice sets presented to participants was a duplicate as a test of internal validity. In the experiment of Dong et al. (2020), a trap question, a choice set that includes an alternative with dominant attributes, was presented as a test for internal validity.

3.10. Signs of convergent validity in pandemic-related choice experiments

The multitude of choice experiments that were conducted on the same topic during the pandemic presents a unique opportunity to investigate convergent validity of these experiments, that is, whether comparable experiments conducted independently have resulted in consistent findings. Also, by a stretch, given that temporal distribution of experiments on the topic of vaccines in particular stretched over various stages of the pandemic and also over various geographical regions, there could be opportunities for investigating temporal stability and population validity of choice experiments within this specific context.

The context of vaccine preference is the one in which choice experiments show the highest degree of consistency in design and attribute definitions, and therefore, are most comparable in that regard. The most common finding of convergence across these studies has been the importance of vaccine efficiency, as reflected in the relative magnitude of the coefficient estimated for this attribute (Craig, 2021; Dong et al., 2020; Hess et al., 2022; Leng et al., 2021; Li et al., 2021b; Liu et al., 2021a; McPhedran and Toombs, 2021) and vaccine safety (side effects) (Borriello et al., 2021; Craig, 2021; Hess et al., 2022; Huang et al., 2021; Leng et al., 2021; Li et al., 2021b; Liu et al., 2021a) in the uptake. Two studies also pointed out the importance of the country/origin of manufacturing of the vaccine as an attribute that they found significant (Li et al., 2021b; Schwarzinger et al., 2021). The association between the likelihood of acceptance and respondents’ education level was independently found by three studies (Craig, 2021; Leng et al., 2021; Schwarzinger et al., 2021). A wide range of predicted uptake was reported by these studies: 86% (Australia, March 2020) (Borriello et al., 2021), 29% (France, June 2020) (Schwarzinger et al., 2021), 85% (China, June 2020) (Leng et al., 2021), 69% (USA, Nov 2020).

We consider the three studies whose experiment was in the context of lockdown strategies (Chorus et al., 2020; Krauth et al., 2021; Reed et al., 2020). While the designs of the three surveys are not entirely consistent in terms of the attributes and framing of questions, they were experiments in a comparable context and all conducted around the same time (during early stages of the pandemic, in April and March 2020) in culturally comparable countries (Germany, Netherlands and USA). The identification of preference heterogeneity for lockdown policies in the form of multiple classes of people was a common finding of Chorus et al. (2020) and Reed et al. (2020), although the former survey suggested three classes and the latter suggested four classes. Another common observation was the moderating effect of individual characteristics on preferences (Krauth et al., 2021; Reed et al., 2020). People's willingness to make individual/societal financial sacrifices in favour of saving lives, as a dominant preference, was another repeated observation (Chorus et al., 2020; Reed et al., 2020).

In the context of contact tracing apps, two surveys estimated the uptake, both in the Netherlands and both during early stages of the pandemic (Jonker et al., 2020; Mouter et al., 2021). The difference between their predicted uptake, however, is stark (50%–65% versus 24%–78%). A plausible explanation for this discrepancy is that the study of Jonker et al. (2020) took place in April just before the peak of infections was reached and when individuals did not know yet where the peak of infection rate will be and when it will end), whereas, the study of Mouter et al. (2021)et al. took place in May (i.e. individuals knew that the peak was over; Namely the peak was somewhere in April 2020).

We did not detect any notable sign of convergent validity across studies concerning resource allocation. This is not, per se, a sign of invalidity of their findings, but rather stems from differences of their designs and attribute definitions. The same holds for the surveys conducted in transport contexts.

3.11. Hypothetical bias mitigation in pandemic-related choice experiments

The potential presence of hypothetical bias and its effect on findings and predictions of discrete choice experiments have been documented in previous work (Haghani et al., 2021b). In Haghani et al. (2021c), ten major factors were discussed that could potentially engender such bias in choice experiments. Their role varies across contexts and applications of choice experiments. In other words, in certain applications of choice experiments the potential role of some sources of hypothetical bias is more prominent than others. For example, “lack of familiarity and contextual tangibility” has been listed as a potential factor. But for surveys that were conducted in the middle of the pandemic and in relation to issues such as lockdown and social distancing, this probably has not been a major source of bias. People around the world all lived through this pandemic and the disease prevention restrictions, the language around these policies were dominating the news and people were familiar with them, and as such, the contexts would resonate with them, and they would not find the context of such surveys intangible.2 We believe, of these ten factors, there are three that are most likely a major source of any potential hypothetical bias in pandemic-related choice experiments: (1) lack of (perceived) policy (societal) consequentiality, (2) strategic behaviour, protest response and deceit, and (3) warm glow and social desirability. These three factors are discussed below.

During the time when these surveys were conducted, governments around the world were making and revising policies dynamically and depending on the condition of disease spread and based on the medical knowledge that was emerging about its nature. With time being a pressing issue in relation to these policies, it is plausible that respondents of some of pandemic-related choice surveys have had doubts about whether results obtained from their responses will make it to actual policy and that their expressed preferences will translate to the decision-making at the highest levels of governance. Among the studies that published pandemic-related choice surveys, only one addressed this issue. Li et al. (2021a) carried out their choice experiment to study people's willingness to follow stay-at-home orders in USA. They report on implementing a policy consequentiality script as well as a perceived consequentiality questionnaire, the two mitigation measures that specifically target the abovementioned source of hypothetical bias.

With respect to surveys on vaccine preference and uptake, it would be understandable for people that the survey would mainly be for predicting people's willingness to be vaccinated, while pharmaceutical companies were developing and trialling the vaccines. The majority of vaccine surveys were conducted in 2020 while pharmaceutical companies only made vaccines accessible to governments since early 2021. Therefore, it is plausible that people did not rule out the potential impact of their responses on policies surrounding vaccine rollouts and mandates. This could potentially be a reason for people with some degrees of vaccine hesitancy to, for example, protest in their responses against any hypothetical mandate.3 Craig (2021) reported on administering an honesty oath script to respondents prior to the actual vaccine preference survey as a way of mitigating potential hypothetical bias that could stem from this source.

The hypothetical choice questions put to respondents in certain pandemic-related surveys poses certain social dilemmas to them (Chorus et al., 2020; Gijsbers et al., 2021; Luyten et al., 2021; Michailidou, 2021). In such circumstances it is plausible to assume that respondents may engage in type of responses that depicts a more socially desirable picture of them. They may have personal desires, for example, for lockdowns or social distancing restrictions to be lifted, but may mask that preference during the survey in favour of options that they perceive more socially acceptable, such as those that indicate that they are willing to make financial sacrifices in order to save lives (Chorus et al., 2020). Same goes with experiments that pose trade-offs in terms of the allocation of vital but limited medical resources, such as vaccine or ICU bed prioritisation (Gijsbers et al., 2021; Michailidou, 2021). The survey of Michailidou (2021), for example, found that participants’ response often violated optimal allocation of resources to benefit female patients with respect to hospital bed allocation or that respondents were less likely to allocate resources to higher income groups, while also showing no signs of racial bias. The nature of such questioning makes it possible that the findings be affected by hypothetical bias caused by the warm glow (or social desirability) phenomenon. In order to mitigate this effect, Michailidou (2021) implemented the method of indirect questioning (or third-person response) that is one of the solutions for hypothetical bias caused by social desirability effect4 (see Fig. 3 in Haghani et al. (2021c)). They contrasted that with responses to direct questioning and found a mismatch, implying the abovementioned theory about hypothetical bias. They justify this discrepancy in following words: “This mismatch between choices and beliefs might be due to participants overestimating the extent to which minorities experience discrimination or, due to participants showing less discrimination because of social desirability bias, yet projecting their racial biases when asked about the choices of others” (p. 5).

Transport-related surveys, on the other hand, did not report on implementing distinct measures of hypothetical bias mitigation. Cherry et al. (2021) mentioned that they asked additional opinion and attitudinal questions following the main survey to identify strategic bias. Zannat et al. (2021) also reported on combining their hypothetical choice response data with (self-reported, survey based) revealed preference data. In addition, three studies implemented pivot designs that are essentially to create familiarity and reduce potential hypothetical bias (Cherry et al., 2021; Manca et al., 2021; Scorrano and Danielis, 2021).

4. Concluding remarks

The COVID-19 pandemic created many unprecedented problems that required policy makers to know about preferences of people in novel contexts that had not been studied prior to the pandemic. This includes problems related to the preferences of people for accepting or adhering to a range of pharmaceutical and non-pharmaceutical disease control measures, as well as problems related to their mobility and travel behaviour. These areas, i.e., health and transport, are two domains where choice modeller are typically active and present (Haghani et al., 2021a). As a result, and prompted by these urgent societal needs imposed by the pandemic, choice modellers mobilised their efforts to address these problems and conducted more than forty experiments during the first two years of the pandemic. Some of these experiments began merely weeks after the official declaration of the global pandemic.

It is understandable that the urgency of some of the problems created by the pandemic might have justified that the researchers fast-track their design and execution of the study. The pressing nature of many of these unprecedented problems and the urgent need to obtain knowledge that can guide policy making have, in many cases, reflected in elements of design quality in pandemic-related choice experiments. In designing a typical choice experiment, researchers often have adequate time to conduct focus-group interviews, run pre-test and pilot experiments and utilise such feedback to enhance the quality and rigour of the main experiment, a practice which is assumed to eventually reflect in more accurate estimates and predictions. This process may often take several months or even years. In the context of COVID-19 experiments, however, the luxury of time was not present, and many studies had to compromise on these fronts, although to varying degrees.5

After carefully analysing these experiments, it became evident that experiments in the health domain applied more elements of quality control and hypothetical bias mitigation than the transport counterparts. While health-related experiments were conducted on average earlier than transport surveys, they reported on elements of pre-testing and piloting far more often than transport surveys.6 Moreover, health experiments collected substantially larger samples and, to less noticeable degrees, paid closer attention to pre-empting or mitigating the issue of hypothetical bias in their design.

We argue that the abovementioned observation could potentially be a reflection of a broader problem where choice modellers in the health sector follow more unified guidelines in their designs than choice modellers in transport. In other words, this observation could be flagging a potential broader issue, and if further future studies confirm that this is the case (i.e., that this is not specific to pandemic-related studies per se), then it will be recommendable that transport researchers need to adopt existing guidelines that are common in health or that they develop specific guidelines tailored to specific choice problems typically studied in transport. In a recent study looking at effectiveness of hypothetical bias mitigation methods, it was observed that health economics significantly pay more attention to the issue of hypothetical bias mitigation compared transport researchers (Haghani et al., 2021c). Therefore, we cannot rule out the possibility that the observations of the current study are also reflective of a broader difference in experimental design culture across these two sectors, a problem that warrants attention. Therefore, a conclusion could be that transport researchers need to pay more attention to pre-testing phase and qualitative research in their choice experiment design.

What was further noticeable in terms of the contrast between health and transport-related experiments was the relative popularity of pivot designs in transport and the absence of this method in health. Transport researchers have essentially developed and adopted this method as a mean of creating contextual familiarity for survey respondents and thereby reducing hypothetical bias. A previous study also noted the unique popularity of this technique in transport, compared to all other areas of applied economics (Haghani et al., 2021c). The said observation was also reflected in the current study in pandemic-related choice experiments where three out of ten transport surveys adopted this technique, whereas, similar to a broader trend, this method was absent in health. We do not, however, necessarily see this as a reason to encourage choice modellers in health to adopt the pivoting method. The nature of choice surveys in transport and travel behaviour is such that individual respondents are often familiar with different choice sets, different attributes or different attribute levels. Therefore, for a typical commuter to be able to relate to the survey, it is often useful that we pivot the attributes around their experienced levels. This is not often the case in health-related contexts. In surveys of vaccine preference or restriction policies, for example, there are little variations as to what individual respondents may be experiencing in terms of options (e.g., policies, vaccines) or characteristics of those options. Hence, the absence of this method in health, in our assessment, stems mainly from fundamental differences in the contexts of choice that health and transport researchers typically investigate.

The concurrent and independent investigation of similar choice problems in the context of COVID-19 across the world also offered a unique opportunity to investigate the question of convergent validity in choice experiments. While no two choice experiments on any topic were conducted identically, and despite differences in design, we were able to detect noteworthy signs of convergent validity, i.e., consistency in findings of independent studies using independent samples and surveys. This could be a promising indication and further evidence in support of the assumption that what choice experiment capture could be reliable reflections of people's preferences and reasonable proxies for true behaviour. There were even signs of convergent validity across experiments conducted in different countries and using samples from populations with major cultural differences. At the same time, we should take note of the fact that the issue of temporal stability was also flagged in the context of COVID-19 experiments. We observed that the timing of surveys often made differences in terms of their predictions. We suggest that this issue, i.e., temporal stability of findings of choice experiments, is one that warrants further attention from future studies.

An important dimension that was not particularly analysed in this review was the type of survey instrument used by studies. A large number of studies (particularly in health) (Craig, 2021; Eshun-Wilson et al., 2021b; Genie et al., 2020; Gijsbers et al., 2021; Katare et al., 2022; Li et al., 2021a) reported on administering their surveys through major survey platforms such as Qualtrics, sampling from the panel of respondents of those companies. Web-based survey tools often provide flexibility to randomize question order and cost-effectively select targeted populations. This, however, contrasts with the use of social media platforms such as Facebook, LinkedIn, Twitter, WhatsApp or Instagram for participant recruitment, particularly in some transport studies (Aaditya and Rahul, 2021; Xu et al., 2021; Zannat et al., 2021). It is unclear how the use of unofficial recruitment platforms such as social media might have impacted on the perceived consequentiality of the surveys (on respondents’ part), and thereby, validity of the results. Given the topical nature of COVID-19, it is possible that some respondents might have simply engaged with such surveys on social media out of curiosity and merely to explore what is being questioned. Whether or to what extent the use of these different participant recruitment instruments could impact on the accuracy of results is unclear, but the disparity between health and transport studies in this area is also striking (in line with other elements of survey quality documented by this work). This constitutes another issue that could be taken into consideration by future discrete-choice experiments that will perhaps keep emerging on COVID-19 related topics (Buchanan et al., 2021; van den Broek-Altenburg et al., 2021).

CRediT authorship contribution statement

Milad Haghani: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Visualization, Funding acquisition, Writing – original draft. Michiel C.J. Bliemer: Investigation, Visualization, Writing – review & editing. Esther W. de Bekker-Grob: Conceptualization, Writing – review & editing.

Declaration of competing interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Appendix A—Studies of discrete choice experiments in the context of COVID-19 pandemic

reference	topic	population	country	time of survey	sample size	design	variables	summary of findings
health
Borriello et al. (2021)	vaccine attribute preference & WTP	generic	Australia	Mar 2020	2 136	efficient 8 sets	-mild side effects -major side effects -effectiveness -administration mode -administration location -immediacy of availability -price	-preference for mild side effects, mode and location of administration & price were heterogenous -preferences for immediacy & severe side effects were homogenous -86% uptake predicted
Chorus et al. (2020)	Preferences for lockdown relaxation policies	generic	Netherlands	Apr 2020	1 009	efficient 9 sets	-increase in no. deaths -increase in no. people with lasting physical injuries -increase in no. people with lasting mental injuries -increase in no. children with lasting educational disadvantage -increase in no. households with income loss -work pressure in health sector	-education lag & income loss found acceptable for saving lives -elderly people reluctant to sacrifice economic pain for educational benefit of younger -multiple classes identified
Craig (2021)	willingness to be vaccinated	generic	US	Nov 2020	1 153	efficient 8 sets	-proof of vaccination -vaccination setting -vaccine effectiveness -duration of immunity -risk of sever side effect	-individual demographics were unrelated to their willingness except for education level -less educated were less willing -effectiveness and safety most important attributes -precited uptake 68.8%
Degeling et al. (2020)	preference for covid surveillance technology	generic	Australia	Feb 2020	2 008	efficient 12 sets	-respect for personal autonomy -privacy/confidentiality -data certainty/confidence -mortality prevention -data security -infectious disease mortality prevention -attribution of responsibility	-greater preference for a system that prevents deaths -personal autonomy was not a big concern -data security was a concern
Dong et al. (2020)	vaccine preference	generic	China	June 2020	1 236	efficient 10 sets	-effectiveness -duration of protection -adverse event -total number of injections -price -origin of product	-strong preference for high effectiveness -price was the least important attribute
Eshun-Wilson et al. (2021a)	preference for social distancing policy measures	generic	US	June 2020	2 428	near orthogonal 6 sets	-duration of policy -income loss -closure of educational facilities -closure of outdoor activity venues -closure of large gatherings -closure of social and lifestyle venues -risk of infection	-strongest preference for prohibition of large gatherings -indifference to closure of social & lifestyle venues -four classes identified: risk averse, conflicted, prosocial, back to normal -men twice more represented in back-to-normal class
Eshun-Wilson et al. (2021b)	preferences for vaccine distribution strategies	generic	US	Mar 2021	2 895	near orthogonal 10 sets	-vaccination location -waiting time at vaccination sites -appointment scheduling -number of doses required -vaccination enforcement -peers vaccinated -vaccination frequency	-public preferred single dose to two dose vaccine -public preferred one-off vaccine rather than annually -public preferred lesser wait times at vaccination sites -vaccine mandate had a negative effect on acceptance -four classes of respondents were identified
Genie et al. (2020)	preferences for pandemic response	generic	UK	Aug 2020	4 000	efficient 8 sets	-lockdown type -lockdown length -postponement of non-urgent medical care -number of excess deaths -number of infections -impact on household spending job losses	n.a.
Gijsbers et al. (2021)	preference for ICU priority	generic	Netherlands	Oct 2020	243	orthogonal efficient 9 sets	-patient age -patient profession -patient guardianship -risk-conscious behaviour on a societal level -health-conscious behaviour -expected length of stay	-first-come first-serve strategy was not supported -risk-conscious behaviour was the most important factor -length of stay was the least important factor
Hess et al. (2022)	vaccine uptake	generic	18 countries	Aug-2020	13,128	efficient 6 sets	-risk of infection -risk of serious illness -estimated protection duration -risk of mild side effects -risk of severe side effects -population coverage -exemption from international travel restrictions	-Higher efficacy increases uptake -Longer protection increases uptake -Sever side effects reduces uptake
Huang et al. (2021)	vaccine preference	clinicians	China	Mar 2021	11,951	fractional factorial 16 sets	-vaccine effectiveness -duration of immunity -risk of adverse reactions -whether peers have been vaccinated (peer influence)	-safety was the most important determining factor
Jonker et al. (2020)	contact tracing app uptake	generic	Netherlands	Apr 2020	900	near orthogonal 13 sets	-group size allowed to the user -warning type -warning recipient -testing requirement -testing results upload responsibility -monetary incentive	-predicted uptake ranged from 59.3% to 65.7% -uptake predicted 64% for most realistic scenario -uptake rate changed significantly with age -other factors correlated with uptake: education level, underlying health issues, perceived covid-19 infection risk
Katare et al. (2022)	preferences for diagnostic testing features	generic	US	Jul 2020	1 505	not reported 9 sets	-Testing method -Testing location -Monetary incentive	-four classes of respondents identified -monetary incentive increased willingness to test
Krauth et al. (2021)	preferences for lockdown exit strategies	generic	Germany	Apr 2020	1 020	efficient 16 sets	-re-opening schools -re-opening restaurants and bars -tracing app -quarantine for elderly -available ICU capacity -unemployment rate	-one dominant attribute was avoiding tracing apps -second dominant was provision of ICU capacity -preferences varied across demographics
Kreps et al. (2020)	vaccine acceptance	generic	US	Jul 2020	1 971	orthogonal 5 sets	-efficacy -protection duration -risk of severe side effects -risk of mild side effects -government authorisation -vaccine origin -source of endorsement	-critical thresholds for attributes identified
Leng et al. (2021)	vaccine preference	generic	China	Jul 2020	1 888	efficient partial profiles 8 sets	-vaccine effectiveness -side effects -accessibility -number of doses -vaccination sites -duration of vaccine protection -proportion of acquaintances vaccinated	-most important attributes were vaccine effectiveness, side effects, proportion of acquaintances vaccinated -higher vaccination rate showed an incentive effect rather than free-rider problems -predicted uptake was 85% -preference heterogeneity was substantial -older age, lower education level, lower income, higher trust in vaccine, higher risk of infection associated with higher vaccination acceptance
Li et al. (2021a)	willingness to follow stay-at-home orders	generic	US	Aug 2020	731	efficient 6 sets	-number of weeks to stay at home -mask wearing requirement -increase in number of cases -increase of unemployment -schools reopening	-the estimate of willingness to stay at home was approximately five and half weeks -individuals considered trade-off between case control and economic impacts -age, ability to work from home and employment status were main drivers of heterogeneity of willingness
Li et al. (2021b)	vaccine preference	university students	Hong Kong	Jan 2021	1 941	orthogonal 18 sets	-efficacy -protection duration -risk of non-severe side effects -area of origin -out-of-pocket price -number of injections	-important factors in order of significance: side effects, efficacy, vaccine origin, number of doses, price
Liu et al. (2021b), Liu et al. (2020), Liu et al. (2021c)	preferences for AI diagnosis of covid	generic	China	Aug 2020	428	orthogonal 6 sets	-diagnostic method -outpatient waiting time -diagnosis time -accuracy -follow-up after diagnosis -diagnostic expense	-opt-in rate for AI diagnosis was 55.8% -three classes of respondents were identified -most important attributes were accuracy, expense of diagnosis
Liu et al. (2021a)	vaccine hesitancy	generic	China & US	Feb 2021	9 077	random	-vaccine technology -side effects -efficacy -immediacy of effectiveness -duration of effectiveness -cost	-cultural differences for vaccine preferences -US respondents prioritised efficacy, then cost -Chinese respondents prioritised cost, then efficacy -Chineses more concerned about side effects -perceived risk of infection was lower in China
Luevano et al. (2021)	vaccine preference	healthcare workers	France	Dec 2020	4 346	efficient 8 sets	-Efficacy -Indirect protection -Safety -Protection duration -Recommendation source	-17% expressed outright rejections -The prospect of protecting elderly decreased hesitancy
Luyten et al. (2021)	preference for vaccine prioritisation	generic	Belgium	Oct 2020	2 060	Bayesian efficient partial profiles 10 sets	-age -medically vulnerable -their cost to economy if infected -whether they are essential workers -whether they spread the virus to many if infected	-elderly group was given lower priority -two clusters of respondents identified -bigger cluster gave higher priority to virus spreaders -smaller cluster gave higher priority to those with pre-existing conditions
McPhedran and Toombs (2021)	Vaccine acceptance	generic	UK	Aug 2020	1 501	orthogonal efficient 6 sets	-level of protection -recommender of the vaccine -number of doses needed for full protection -location of vaccine administration -coverage in the media	-efficacy the most important factor in vaccine selection especially for older age groups
Michailidou (2021)	dilemmas in allocation of medical resources to covid patients	generic	US	May 2020	1 842	not reported 2 sets	-patient gender -patient race/ethnicity -patient income -patient parenthood	-participants violate optimal allocation more often for benefit of females -less likely to allocate resources to high-income patients -race had no role
Mouter et al. (2021)	uptake for contact tracing app	generic	Netherlands	May 2020	990	Bayesian efficient 8 sets	-who gets notified -waiting time for testing -shops refusing service to those without app -stopping condition for contact tracing -no. people unjustifiably quarantined -no. deaths prevented -no. household with prevented financial issue	-prevention of deaths & financial loss had a very strong influence on the uptake -uptake was estimated between 24% and 78% -societal effects are a major factor in the uptake
Rad et al. (2021)	willingness to isolate post diagnosis	generic	Iran	March 2020	617	orthogonal 14 sets	-no. days in isolation -isolation payment -treatment payment -service provision (food etc)	-average monetary value of a WTA for 7 days of isolation was US$51.71 -monetary value of a WTA depended on employment status and income -monetary value of accepting isolation increased nonlinearly by number of days
Reed et al. (2020)	preference for restriction lifting	generic	US	May 2020	5 953	orthogonal 10 sets	-nonessential business reopening -no. of contracted cases -time for economic recovery -% falling below poverty line	-four latent classes identified -the largest class was covid risk minimisers -opening supporters constituted smallest class -political affiliation, race, income associated with class membership
Schwarzinger et al. (2021)	vaccine acceptance	generic	France	June 2020	1 942	efficient 8 sets	-vaccine efficacy to reduce infectious risk -risk of serious side effects -location of manufacture -place of administration	-29% opted for vaccination and 71% did not. -vaccine hesitancy/refusal associated with female gender, age, low education, history of poor compliance, no history of chronic condition & lower perceived severity of covid -location of manufacturing affected hesitancy -vaccine hesitancy was lower in group with herd immunity information, recovered patients or their associates
transport
Aaditya and Rahul (2021)	willingness to use public transport	generic	India	June 2020	410	not reported 8 sets	-sanitisation in public transport -crowd management -social distancing -increase in travel time -increase in travel cost	-social distancing and sanitisation found to be the most important factors
Aghabayk et al. (2021)	crowding perception	generic	Iran	Nov 2020	590	efficient 6 sets	-travel time -crowding -standing or sitting position	-tolerance for crowding decreased during pandemic
Ceccato et al. (2021)	mode choice	university students & employees	Italy	Aug 2020	6 598	efficient 15 sets	-cost -in-vehicle time -walking time to reach stop/vehicle -waiting time at the stop -sanitisation -ventilation -crowd management -safety enforcement	-preferences are different between students and employees -available travel alternatives and specific risk mitigation measures on vehicles were found to be most significant
Cherry et al. (2021)	value of travel time saving & reliability	generic	US	Mar 2020	7 743	orthogonal, pivots 8 sets	-travel time -toll cost -time-of-day (occupancy requirement)	-widespread changes in travel behaviour and a reduction in WTP for travel time savings and travel time reliability across all traveller types
Cho and Park (2021)	behaviour change of crowding impedance on public transit	generic	South Korea	Nov 2020	623	orthogonal efficient 9 sets	-in-vehicle time -transfer time -crowding factor -travel mode	-crowding impedances after the covid pandemic are about 1.04–1.23 times higher than before the covid pandemic
Luan et al. (2021)	mode choice	generic	China	June 2020	428	orthogonal efficient 3 sets	-travel time -travel cost -degree of congestion -wait time	-regret aversion psychology was not dominant -for longer trips, the probability of private car decreased, and public transport increased
Manca et al. (2021)	attitude change to air travel	air travellers	UK	June 2020	388	efficient, pivots 3 sets	-fare (round trip per person) -total time at the departure airport -total time at the arrival airport -transfer	-no significant effects observed concerning perceptions of safety arising from wearing a mask, or concerns over the necessity to quarantine -respondents perceive virtual substitutes for business travel, for example video calls and similar software, as only a temporary measure
Scorrano and Danielis (2021)	mode choice	generic	Italy	June 2020	315	not reported, pivots 6 sets	-operational cost -parking cost -travel time -percentage of pedestrian lanes -percentage of cycling lanes -weather (rainy or sunny day)	-more cycling lanes increase bike use but at the expense of walking and bus use -covid pandemic significantly altered mode choices -strong negative impact on bus and shifting bus users towards private modes
Xu et al. (2021)	mode choice	generic	Pakistan	Feb 2021	318	orthogonal 9 sets	-trip time -trip cost -access time -parking time -covid precautionary measures	-economic variables such as trip time and trip cost, were determined to be significant -commuters were more conscious of covid preventive measures
Zannat et al. (2021)	shopping trip behaviour	generic	Bangladesh & India	May 2020 & Apr 2020	815	efficient 2 sets	-no. cases in the country -no. cases in the city -no. deaths in the city -no. affected household members -Type of government restrictions	-different shopping preferences observed for various socio-economic groups
business
Grashuis et al. (2020)	preference for grocery shopping	generic	USA	May 2020	900	efficient 6 sets	-purchase method -time windows -minimum order requirements -fees	-trend in the covid pandemic caused significant differences in grocery shopping preferences
Park and Lehto (2021)	hotel choice	hotel guests	USA	March 2020	422	orthogonal 2 sets	-cleanliness and hygiene -physical distancing -staff and guest requirement	-cleanliness and hygiene aspect are considered especially critical
environment
Hynes et al. (2021)	stability of environmental preferences	generic	Canada, Scotland, Norway	May 2020	1 508	Bayesian efficient	-health -litter -area -jobs -additional costs	-preferences and WTP remain relatively stable in the face of covid

Appendix B—The search query string

(TS=(“choice modelling*" OR “choice modeling*" OR “discrete choice model*" OR “model* of discrete choice” OR “random utility choice model*" OR “discrete choice method*" OR “discrete choice analysis*" OR “discrete choice analyses*" OR “discrete choice theory” OR “discrete choice experiment*" OR “stated choice experiment*" OR “stated choice survey” OR “stated choice method*" OR “discrete choice survey” OR “hypothetical choice experiment*" OR “hypothetical choice survey*" OR “econometric choice model*" OR “model* of econometric choice” OR “econometric choice method” OR “econometric choice analysis*" OR “econometric choice theory*" OR (“conjoint analysis” AND “choice”) OR “choice-based conjoint” OR (“stated preference*" AND “choice*") OR (“revealed preference*" AND “choice*") OR “stated choice data” OR “revealed choice data” OR “stated choice observation*" OR “revealed choice observation*"

OR ((“multinomial logit” OR “random parameter logit” OR “random coefficient logit” OR “mixed logit” OR “error components* logit” OR “latent class logit” OR “nested logit” OR “ordered logit” OR “multinomial probit” OR “mixed probit” OR “random parameter probit” OR “random utility theory” OR “random regret minimization*" OR “random utility maximization*" OR “random regret logit” OR “random utility logit”) AND (“choice*" OR “preference*"))

OR “choice survey design” OR ((“choice survey*" AND “design*") AND (“efficient” OR “orthogonal” OR “D-efficient” OR “D-optimal” OR “D-optimum” OR “E-efficient” OR “E-optimal” OR “E-optimum"))))

AND.

(TI=(“Covid-19″ OR “Coronavirus” OR “SARS-COV-2″) OR AK=(“Covid-19″ OR “Coronavirus” OR “SARS-COV-2″))

Appendix A. Supplementary data

The following are the Supplementary data to this article: