Summary
In this research, we investigated the protection motivators and precautionary behaviors against coronavirus disease 2019 (COVID-19) and the associations between them. To do this, we developed two original scales, collected data (2783 responses) using an online survey, after removing the responses (319), which were filled in incompletely or incorrectly in the questionnaire, we obtained 2464 participants covering the aged 18+ population in Turkey. Based on random sampling, our sample complies with these ratios and generally reflects the aged 18+ population of Turkey. We confirmed the psychometrical validity and reliability of our two scales using the collected data. Herewith, we found that perceived susceptibility of COVID-19 infection is very high, perceived severity of COVID-19 is medium, COVID-19 related information seeking is high, beliefs on precautions’ efficacy is high and also the practice of precautionary behaviors is high. Our research depicts that all protection motivators significantly are related with the practice of precautionary behaviors (routine and leisure). However, with the only exception of perceived severity of COVID-19 is not related with precautionary behaviors (routine). Besides, we saw that females’ average in all variables is significantly higher than males and some variables are sensitive to age, education level, marital status and the number of children. We believe that the findings provide essential inputs for authorities in establishing public health policies against the present pandemic and likely ones in the future.
Keywords: COVID-19, protection motivators, precautionary behaviors, protection motivation theory, Turkey
INTRODUCTION
Since the end of 2019, the coronavirus disease 2019 (COVID-19) crisis is the main global topic for our planet. The measures taken by governments and supra-governmental organizations cannot stop the spreading and the World Health Organization declared it as a pandemic on 12 March 2020. The intensity of the pandemic such as the spread of the virus, total infected people and case fatality rates in various nations show different patterns. Although there are many different variables responsible for these variations, social, cognitive and behavioral factors of nations play an important role in this equation. How a society perceives the risk posed by the pandemic, how it collects information to increase awareness, how it develops a collective and common understanding against the threat and to what extent it responds with precautionary behaviors are critical variables for a nation’s performance on fighting against a pandemic. As theories posit that cognitive processes based on perceptions, as we call them protection motivators here, trigger the behaviors (e.g. Rogers, 1975; Janz and Becker, 1984), understanding the relations between them is crucial for policymakers, strategy developers, communicators and other authorities. However, we encountered no scale measuring either the protective motivators or the precautionary behaviors against the COVID-19 pandemic in the literature during our theorization period (March to April 2020) before data gathering, which is necessary to measure and explore the association between them empirically. In this study, we aim to build psychometrically valid and reliable constructs representing the protection motivators based on beliefs and perceptions, and precautionary behaviors against the COVID-19 pandemic, and to investigate relations between the protection motivators and precautionary behaviors. Additionally, to strengthen the results of the main analysis, we try to examine the sensitivity of these variables to demographic variables.
BACKGROUND AND HYPOTHESES
Since the first days of the epidemic, the lack of effective medical treatment and vaccination has revealed the importance of behavioral measures all over the world (Khosravi, 2020). Some of these measures are: hand hygiene, wearing a facemask, social distancing, not leaving home unless it is essential (Kwok et al., 2020). These behavioral measures are considered critical for Public Health Emergency Preparedness (PHEP) especially during the epidemic (Lee et al., 2020). The role of the public is considered vital in the effective implementation of PHEP (Khosravi, 2020). Past outbreaks have shown that correct and decisive public participation in preventive measures has reduced the spread of the epidemic and facilitated the efforts to contain it (Dryhurst et al., 2020).
During an epidemic or pandemic, increased risk perceptions and awareness of the public can make positive contributions to the implementation of precautionary measures (Van Bavel et al., 2020). As stated in the Protection Motivation Theory: ‘people appraise the severity and likelihood of being exposed to a depicted noxious event, evaluate their ability to cope with the event, and alter their attitudes accordingly’ (Rogers, 1975). The model argues that the change in attitudes is not arbitrated by or the consequence of the emotion of fear. Instead, it is predicted by the level of protective motivation triggered by cognitive assessment processes based on beliefs, perceptions and necessary awareness. The emphasis, therefore, relates not to fear as an emotion but to cognitive processes and protection motivation (Rogers, 1983). If a person recognizes a severe threat and believes she/he has the ability to react effectively to prevent it, then he or she develops self-protecting attitudes or behaviors (Witte, 1993). Besides, the Health Belief Model proposes that if an individual, subjectively assesses the severity of a health problem and personal likelihood of susceptibility, perceives a high risk, then he or she is more motivated in taking necessary actions than others. The model posits that while the individual is encouraged by ‘the efficacy of an advised precautionary action’, discouraged by ‘the constraints to practice it’ (Janz and Becker, 1984; Carpenter, 2014). Similarly, the Extended Parallel Process Model theorizes that when the perceived threat and perceived effectiveness are high, hazard control processes are initiated. When people are aware of and worried about a likely critical threat, and perceive and believe that a particular strategy or response can effectively block it, they are motivated to control it (motivation for protection) by employing the response or strategy which is believed to be useful to prevent the threat (Witte, 1992).
As a summary of the above theories, some mechanisms must be triggered to initiate the processes of taking protective measures against an epidemic or pandemic. First, individuals believe that she/he is personally susceptible to infectious disease. Second, they develop the perception that the threat posed by the contagious disease is severe and the advised preventive measures are functional to stop or at least reduce the threat. Finally, they believe and confirm the efficacy of these practices (Van den Broucke, 2020). In this process, based on the risk communication theory (Otway and Wynne, 1989), the individual follows the course of the outbreak based on the information (infection, mortality and recovery cases, preventive measures and their efficacy, etc.) from various media (Du et al., 2020). This information flow process plays a critical role in the risk assessment of the outbreak (Depoux et al., 2020). An individual’s awareness about the threat and the efficacy of advised precautionary behaviors is often related to the communication and dissemination of necessary information to the public and public intention to seek relevant information from various sources (Holmes, 2008).
Within the scope of the theoretical framework mentioned above, we focused on the relationship between the belief and perception-based protection motivators and the precautionary behaviors of individuals, which are known to be particularly effective in containing COVID-19 pandemic (Bashirian et al., 2020; Costa, 2020; Hotle et al., 2020; Khosravi, 2020). Coherent with their level of necessity and dispensability in life, we examined precautionary behaviors in two titles as routine (e.g. washing hands frequently, wearing masks, not going out, not accepting guests, not using public transportation) and leisure (e.g. not traveling abroad, not going to the cinema, theater, restaurants, sports halls and matches) since we believe that the attitudes toward indispensable daily and optional leisure activities are expected to be different (e.g. Beltran et al., 2021; Isch et al., 2021). Individuals can more easily renounce leisure activities than daily routines. While individuals can practice precautionary behaviors to their daily necessities to reduce the likelihood of catching the disease, at the same time, they can show total avoidance against other individuals (Lee and You, 2020). Within the framework of all these, our hypotheses are presented below and the research framework is presented in Figure 1.
Fig. 1:

Research model.
H1a: Perceived Susceptibility of COVID-19 (Susceptibility) significantly increases Precautionary Behaviors-Routine (Precautions-Routine).
H1b: Perceived Severity of COVID-19 (Severity) significantly increases Precautions-Routine.
H1c: COVID-19 Related Information Seeking (Info-seek) significantly increases Precautions-Routine.
H1d: Beliefs on Precautions’ Efficacy (Efficacy) significantly increases Precautions-Routine.
H2a: Severity significantly increases Precautionary Behaviors-Leisure (Precautions- Leisure).
H2b: Severity significantly increases Precautions-Leisure.
H2c: Info-seek significantly increases Precautions-Leisure.
H2d: Efficacy significantly increases Precautions-Leisure.
DATA AND METHODS
We conducted our research on people using the internet and social media in Turkey, which is among the top 10 countries in the total number of COVID-19 cases in the world for a while as of May 2020. An online questionnaire form prepared by authors was sent to individuals all around the country including all seven geographical regions using social networks such as WhatsApp groups, Facebook fellows and email contacts, and they all are requested to share the link of the online survey instrument with their social networks. The data collection period started on 5 May 2020 and ended on 17 May 2020, ∼9 weeks after the first COVID-19 positive case is confirmed by the government on 11 March 2020 (RTMH, 2020a). We received 2783 responses and after removing the responses with missing value (319), we obtained 2464 participants (aged 18+) who had answered all the questions of the questionnaire, and employed these responses in our analyses. When we scrutinized the demographic characteristics of the dropout sample, we see that there is no significant difference between our main and dropout sample in terms of age, gender, occupation, number of children, marital status, education, region of residency or dwelling type.
Turkey’s population is circa 84 million (Worldometers, 2020) and the population aged 18+ is ∼ 58 million [Turkish Statistical Institute (TSI), 2021], we figured out 1537 as a minimum sample size to represent the population based on the procedure recommended by Krejcie and Morgan (Krejcie and Morgan, 1970) with 95% confidence interval and ±2.5% margin of error. According to the data covering the aged 18+ population gathered by TSI related to 2020, for instance, 50.39% (50.04% in our research) of Turkey’s population is women and 49.61% (49.96% in our study) is men and 62.31% (64.41% in our research) of the population is married and 37.69% (35.59% in our study) is single (TSI, 2021). Therefore, the sample used in this study can be said to generally reflect the population of Turkey based on random sampling. The descriptive statistics of the demographics of our sample are presented in Table 1.
Table 1:
The descriptive statistics of demographics
| Variables | Frequency | % | Variables | Frequency | % |
|---|---|---|---|---|---|
| 1. Gender | 5. Marital status | ||||
| Female | 1233 | 50.04 | Married | 1580 | 64.12 |
| Male | 1231 | 49.96 | Single | 884 | 35.88 |
| 2. Age | 6. Education level | ||||
| 18–29 | 678 | 27.52 | Primary/secondary school | 127 | 4.75 |
| 30–39 | 595 | 24.15 | High school | 298 | 12.09 |
| 40–49 | 742 | 30.11 | Associate degree | 392 | 15.91 |
| 50–59 | 299 | 12.13 | Bachelor’s degree | 1060 | 43.02 |
| 60–69 | 133 | 5.40 | Graduate degree | 587 | 23.82 |
| 70≥ | 17 | 0.69 | |||
| 3. Occupation | 7. Region of residency | ||||
| Public sector | 888 | 36.04 | Marmara | 572 | 23.22 |
| Private sector | 501 | 20.33 | Aegean | 438 | 17.78 |
| Self-employment | 137 | 5.56 | Mediterranean | 229 | 9.29 |
| Student | 371 | 15.06 | Central Anatolia | 701 | 28.45 |
| Retired | 332 | 13.47 | East/South east Anatolia | 365 | 14.81 |
| Unemployed | 235 | 9.54 | Black sea | 159 | 6.45 |
| 4. Children | 8. Dwelling | ||||
| No children | 877 | 35.59 | Flat | 2078 | 84.33 |
| Have children | 1587 | 64.41 | House/villa | 386 | 15.67 |
| (n = 2464) |
We performed Exploratory Factor Analysis (EFA) to check the structural validity of our scales’ data sets, Cronbach’s Alpha Test to evaluate the reliability of the scales and correlation analysis to discover the relations between factors. To investigate the factor structures of the scales, we employed Confirmatory Factor Analysis (CFA). For determining the impacts of independent variables on dependent variables, multiple linear regression analyses were conducted using the factor scores produced during EFA (Johnson and Wichern, 2002) as normalized indicators for dependents/independents variables.
Measures
We developed a scale that consists of four factors and is measured by 20 items. These factors are Susceptibility, Severity, Info-seek and Efficacy based on the concepts emphasized in the Protection Motivation Theory, the Health Belief Model, the Extended Parallel Process Model and the Risk Communication Theory. We named this scale ‘the Protection Motivators for Precautionary Behaviors against COVID-19 (PMPBC)’.
The items for the Precautionary Behaviors against COVID-19 (PBAC) scale are prepared using the precautionary measures against COVID-19 advised and applied by health system officials and governments (WHO, 2020a; RTMH, 2020b). We established a 20 items scale to measure this phenomenon in 2D namely precautionary behaviors toward daily routines (Precautions-Routine) and leisure activities (Precautions-Leisure).
We employed five points Likert scale for participants’ assessment as 1 for ‘strongly disagree’ and 5 for ‘strongly agree’ for all items in our survey instrument. Both scales’ content and face validity are confirmed by four academics and a Turkish literature teacher by rephrasing and eliminating irrelevant or unnecessary items before the data collection phase. Moreover, a pretest is conducted on a small group of participants to confirm the items are well understood.
Statistical analysis employed
Using SPSS 20.0, AMOS 20.0 and EViews 7.1 for conducting our statistical analyses, we reported the results of factor variables as frequency, mean, standard deviation and percentage (the average of the responses for items of relevant factors over the top score), confirmed psychometric validity, which is subjectively specified as the capability of the test to evaluate what it alleges to measure or the capacity of the tool to measure the characteristic of the construct under research (DeVon et al., 2007) and reliability (internal consistency) of our two scales by using factor and reliability analyses. Using frequently used and advised tests in literature, we also confirmed that our data are not subject to common method variance. We conducted correlation analysis to assess inter-variable associations. To examine the influence of independent variables (Efficacy, Info-Seek, Susceptibility and Severity) on dependent variables (Precautions-Routine and Precautions-Leisure), we employed multivariate linear regression analysis considering the basic assumptions of linear regression analysis. As a control analysis, we performed multiple comparison analyses to strengthen the results of the main analysis by evaluating the sensitivity of our dependent and independent variables to demographics.
RESULTS
Before conducting EFA and CFA, the sample is divided into two equal parts based on random sampling. Then, EFA and CFA are separately applied on two samples. To provide the structural validity of the data relevant to our two scales (PBAC and PMPBC), EFA is conducted. Before applying EFA, the samples obtained from the splitting of the main sample are checked to be proper for EFA by utilizing KMO and Bartlett’s tests. The results of these tests (KMO1 = 0.952 and KMO2 = 0.822; p1 = 0.000 and p2 = 0.000 successively) show that the samples are convenient for EFA concerning two scales at marvelous and meritorious status, respectively (Kaiser, 1974). Then, correlation analysis is employed to determine which factors are uncorrelated by rotating component matrixes of these scales with the Varimax method, which is an Orthogonal method providing uncorrelated factor score and widely used in the literature for obtaining interpretive and significant factors.
Using EFA, we excluded five items from the PBAC and eight items from the PMPBC scales because of small correlation values (<0.3), multiple factor loadings and lower communalities (<0.5) (Hair et al., 2014). The results of EFA, reliability analysis of the dependents and independents variables, and removed items during EFA are illustrated in Table 2.
Table 2:
The results of EFA, reliability analysis of the dependents and independents variables and removed items during EFA
| The results of EFA, reliability analysis of the dependents and independents variables | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Scaled | Factor | Items |
Factor descriptive statistics
|
EFA |
Reliability analysis |
|||||||
| SD | %e | Factor loadings | Total variance explained (%) | KMO test | Bartlett’s test (p) | (α) | Total (α) | Corrected item-total correlations | ||||
| Precautionary behaviors against COVID-19 (PBAC) | Precautionary behaviors-routine (Precautions-Routine) | Because of COVID-19 contagion risk, … | 4.240 | 0.712 | 84.792 | 0.586–0.822 | 59.615 | 0.952a | 0.000 | 0.907b | 0.922b | 0.562–0.765c |
| 1. I prefer staying home instead of going out. | ||||||||||||
| 2. I reduce my habit of touching products and money while shopping. | ||||||||||||
| 3. I wash my hands more than usual. | ||||||||||||
| 4. I sanitize the packages that a mail or cargo carrier brings. | ||||||||||||
| 5. I do not invite anyone to my home and postpone all social gatherings. | ||||||||||||
| 6. I do not order food from outside. | ||||||||||||
| 7. I do not travel by public transport. | ||||||||||||
| 8. I always wear a mask when I go out. | ||||||||||||
| 9. I keep away or move away if I see someone who coughs or sneezes. | ||||||||||||
| 10. I do not get in a taxi. | ||||||||||||
| Precautionary behaviors-leisure (Precautions-Leisure) | Because of COVID-19 contagion risk in the near future, … | 3.633 | 0.901 | 72.666 | 0.570–0.815 | 0.850b | 0.529–0.745c | |||||
| 11. I do not travel abroad. | ||||||||||||
| 12. I do not go to places like cinema, theater, and matches. | ||||||||||||
| 13. I do not eat at a restaurant. | ||||||||||||
| 14. I do not go to gyms and swimming pools. | ||||||||||||
| 15. I do not stay in a hotel. | ||||||||||||
| Protection motivators for precautionary behaviors against COVID-19 (PMPBC) | Perceived susceptibility of COVID-19 (Susceptibility) | 1. I am worried that COVID-19 will likely infect my family members. | 4.290 | 0.787 | 85.804 | 0.698–0.884 | 66.766 | 0.822a | 0.000 | 0.813b | 0.810b | 0.568–0.754c |
| 2. I am worried that COVID-19 will likely infect my family elders. | ||||||||||||
| 3. I am worried that COVID-19 will likely infect me and then spread to my family. | ||||||||||||
| Perceived severity of COVID-19 (Severity) | 4. The COVID-19 is a lethal health threat. | 2.743 | 0.866 | 54.854 | 0.685–0.752 | 0.701b | 0.328–0.444c | |||||
| 5. The health system has a hard time to cope with this disease. | ||||||||||||
| 6. COVID-19 pandemic will last longer than expected. | ||||||||||||
| COVID-19 related information seeking (Info-Seek) | To update knowledge and improve my awareness about COVID-19, | 3.269 | 1.025 | 65.379 | 0.767–0.837 | 0.811b | 0.589–0.700c | |||||
| 7. I constantly follow the news about the COVID-19. | ||||||||||||
| 8. I persistently follow the posts about Corona epidemic on social media. | ||||||||||||
| 9. I always talk about Corona with my friends and family members. | ||||||||||||
| Beliefs on precautions’ efficacy (Efficacy) | I believe……………………. against COVID-19 outbreak. | 3.712 | 0.894 | 74.240 | 0.510–0.851 | 0.705b | 0.439–0.539c | |||||
| 10. The recommended hygiene precautions are effective | ||||||||||||
| 11. The recommended social/physical distance rules are effective | ||||||||||||
| 12. All individuals must strictly adhere to the measures implemented | ||||||||||||
| Removed items during EFA | ||
|---|---|---|
| Scaled | Factor | Items |
| Precautionary behaviors against COVID-19 (PBAC) | Precautions-Routine | Because of COVID-19 contagion risk, …
|
| Precautions-Leisure | Because of COVID-19 contagion risk in the near future, …
|
|
| Protection motivators for precautionary behaviors against COVID-19 (PMPBC) | Susceptibility |
|
| Severity |
|
|
| Info-Seek |
|
|
| Efficacy |
|
|
|
| ||
The sample is convenient for EFA at marvelous and meritorious status, respectively (Kaiser, 1974).
The test value (α) is greater than the threshold value (0.7) (Ravichandran and Rai, 1999; Jonsson, 2000; Hair et al., 2014).
The corrected item-total correlations of the scales are also greater than limit value (0.2) (Streiner and Norman, 2003).
Five-point Likert-type scale (from 1 = strongly disagree to 5 = strongly agree) is used for measuring the items.
Percentage values of factors are computed depend on the average of the responses (1–5) for items of relevant factors over the top score (5).
The results express that the most of factor loadings are > 0.60 and the difference among factor loadings taking place in the relevant factors are higher than 0.1. Besides, the results of Reliability Analysis, the test value of Cronbach’s Alpha (α) and the corrected item-total correlations of the scales are higher than their threshold values (0.7 and 0.2, respectively) (Ravichandran and Rai, 1999; Jonsson 2000; Streiner and Norman, 2003; Hair et al., 2014). Thus, the results confirm the structural validity and internal consistency of both scales.
For investigating the factor structures of the scales, we employ CFA depending on the results gained from EFA, via the maximum likelihood estimation method. According to the results obtained from CFA, all CMIN/DF values (CMIN1/DF1=2.654 for dependent scale and CMIN2/DF2=3.567 for independent scale) are lower than the limit level (5) (Marsh and Hocevar, 1985). Additionally, all of the fit index values (GFI1=0.982, GFI2=0.979; AGFI1=0.968, AGFI2=0.964; NFI1=0.981, NFI2=0.967; NNFI1=0.981, NNFI2=0.966; CFI1=0.988, CFI2=0.976, and RMSEA1=0.036, RMSEA2=0.045) are higher/lower than the good fit threshold proposed by Schermelleh-Engel et al. (Schermelleh-Engel et al., 2003). Consequently, we can say that both scales promote structural validity.
Besides, for researching whether there is a Common Method Bias (CMB), which describes the measurement error that is compounded by the sociability of respondents who want to provide positive answers (Chang et al., 2010) and is a potential problem in behavioral research (Podsakoff et al., 2003) in our sample, we conduct Harman’s Single-Factor Test (HSFT), Common Latent Factor (CLF) and Common Marker Variable (CMV) methods, which are commonly opted for scrutinizing CMB (Podsakoff et al., 2003). We discover that HSFT (34.70%), CLF (23.67%) and CMV (18.85%) values are smaller than the limit value (50%) and it refers that our data are not exposed to CMB.
To examine the relationships between our variables, the correlation analysis is shown in Table 3. For checking impacts of independent variables (Severity, Info-Seek, Susceptibility and Efficacy) on dependent variables (Precautions-Routine and Precautions-Leisure) related to our hypotheses, the summary of multiple linear regression analyses conducted between the dependent and independent variables and the results of the hypotheses are presented in Table 3.
Table 3:
The correlation analysis between variables and the summary of multiple linear regression analyses between the dependent and independent variables and the hypotheses’ results
| The correlation analysis between variables | ||||||||
|---|---|---|---|---|---|---|---|---|
| Precautions- Routine | Precautions- Leisure | Susceptibility | Severity | Info-Seek | Efficacy | Age | Number of children | |
| Precautions-Routine | 1.000 | |||||||
| Precautions-Leisure | 0.000 | 1.000 | ||||||
| Susceptibility | 0.427a | 0.188a | 1.000 | |||||
| Severity | –0.015 | 0.305a | 0.000 | 1.000 | ||||
| Info-Seek | 0.169a | 0.099a | 0.000 | 0.000 | 1.000 | |||
| Efficacy | 0.414a | 0.280a | 0.000 | 0.000 | 0.000 | 1.000 | ||
| Age | –0.004 | 0.177a | –0.092a | 0.052a | –0.154a | –0.004 | 1.000 | |
| Number of Children | –0.040a | 0.134a | –0.096a | 0.078a | –0.053a | –0.050a | 0.537a | 1.000 |
| The summary of multiple linear regression analyses between the dependent and independent variables and the hypotheses’ resultsb,c,d | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model ID | Dependent variable | R2 | Adjusted R2 | SE | F | p | Independent variables coeff. | Unstand. coeff. |
t | p | Collinearity statistics |
Hyp. ID | Hypothesis result | |
| B | Tol. | VIF | ||||||||||||
| 1 | Precaution- Routine | 0.298 | 0.297 | 0.882 | 228.441 | 0.000e | (Constant) | 0.000 | –0.016 | 0.987 | – | – | ||
| Susceptibility | 0.373 | 22.033 | 0.000e | 0.986f | 1.014f | 1a | Accepted | |||||||
| Severity | –0.013 | –0.792 | 0.429 | 0.996f | 1.004f | 1b | Rejected | |||||||
| Info-Seek | 0.185 | 11.031 | 0.000e | 0.995f | 1.005f | 1c | Accepted | |||||||
| Efficacy | 0.409 | 24.200 | 0.000e | 0.992f | 1.008f | 1d | Accepted | |||||||
| 2 | Precaution- Leisure | 0.207 | 0.208 | 0.888 | 161.365 | 0.000e | (Constant) | 0.000 | –0.026 | 0.980 | – | – | ||
| Susceptibility | 0.170 | 9.414 | 0.000e | 0.986f | 1.014f | 2a | Accepted | |||||||
| Severity | 0.313 | 17.412 | 0.000e | 0.996f | 1.004f | 2b | Accepted | |||||||
| Info-Seek | 0.114 | 6.378 | 0.000e | 0.995f | 1.005f | 2c | Accepted | |||||||
| Efficacy | 0.295 | 16.375 | 0.000e | 0.992f | 1.008f | 2d | Accepted | |||||||
|
| ||||||||||||||
Correlation is significant at α = 0.05 level.
For ensuring the normality more robust, the Two-Step Approach transformation is applied (Templeton, 2011).
One-Sample Kolmogorov–Smirnov Normality Test indicates that all variables are distributed normally [null hypothesis (H0): The variable is distributed normality], since p > α = 0.05.
White Homoscedasticity Test indicates that there is no heteroscedasticity [the null hypothesis (H0): There is homoscedasticity in the model] in the models (p > 0.05).
The regression models and the coefficients are significant at α = 0.05 level.
Due to the tolerance value > 0.1 and VIF <10, the model is not subject to multicollinearity (Hair et al., 2014).
It is expected that anyone with higher levels of precautions-routine most likely will also show higher levels of precautions-leisure and vice versa from a psychological perspective. Nevertheless, since we apply the Varimax method, which is an Orthogonal Method providing uncorrelated factor score [(Hair et al., 2014), p. 104, 112] to obtain interpretive and significant factors, no correlation is founded between these two variables as expressed in Table 3. Examining the findings in Table 3, the regression models and the coefficients (except the coefficient of perceived Severity in the first regression model) are significant at α = 0.05 level and the signs of these significant coefficients are positive. The independent variables (Susceptibility, Severity, Info-Seek and Efficacy) have more total effect on the Precautions-Routine model (R2Precautions-Routine = 0.297) than the Precautions-Leisure model (R2Precautions-Leisure = 0.208), even though, the perceived Susceptibility is not statistically significant in the first regression model.
To strengthen the results of the main analysis by determining whether there are any statistically significant differences between the means of subgroups of gender and marital status as binary variables, we employ the Independent Samples t-Test for control analysis. We discover that there are statistically significant differences between gender subgroups namely males have lower Precautions-Routine (p < 0.000), Precautions-Leisure (p < 0.000), Susceptibility (p < 0.005), Severity (p < 0.000), Info-Seek (p < 0.004) and Efficacy (p < 0.000) than females and marital status sub-groups; for Precautions-Leisure (p < 0.000) and Severity (p < 0.000), for Susceptibility (p < 0.007) and Info-Seek (p < 0.000). We also find out that the variables in our model are not significantly sensitive to other demographic factors, functioning as control variables, such as geographical region, occupation or dwelling type.
DISCUSSION
To the extent of our knowledge, this research is the first study on the relationship between perceived severity, susceptibility of COVID-19, beliefs on efficacy of preventive actions, information seeking about COVID-19 and precautionary behaviors among Turkish people. Our research, based on widely representative Turkish population data, offers a valuable comprehension of perceptions, beliefs and cognitive processes as protection motivators and practices of precautionary behaviors related to the COVID-19 pandemic. A substantial proportion of the participants reported that their perception of COVID-19 pandemic’s severity is medium (=2.74; %=54.85), likelihood assessment of their (self and family) susceptibility is very high (= 4.29; %=85.80), beliefs about the efficacy of recommended precautions is high (=3.71; %=74.24) and information seeking to increase their knowledge on the health risk posed by COVID-19 is high (=3.27; %= 65.38). These findings indicate that Turkish people take the COVID-19 pandemic as a serious health risk as of June 2020 during our data collection period. Moreover, another research conducted June 2021 reveals that COVID-19 is still perceived as a serious health risk by the majority (%79) of Turkish people (TTB, 2021). Perceived severity at medium level seems coherent with the reported severity of disease by national health authorities based on the fact that the majority of the cases show mild or no symptoms and low Case Fatality Rate, thanks to the Turkish health system seemingly being blanket and effective. Perceived susceptibility at a very high level is in line with COVID-19’s very high contagious characteristic, which is exacerbated by non-symptomatic cases and long incubation periods. While high-level of beliefs about the efficacy of recommended precautions shows that the majority of people believe that the precautionary measures advised by authorities are functional and useful to contain the virus, high-level COVID-19-related information seeking indicates that the majority of people are eager to acquire information about the COVID-19 pandemic through various means available.
Similarly, the results show that the majority of the participants practice advised PBAC ( = 3.94; % = 78.72)—very high for Precautions-Routine is ( = 4.24; % = 84.79) and high for Precautions-Leisure ( = 3.63; % = 72.67). The items in our survey instrument questioning the precautionary behaviors toward leisure activities have future orientation since most of the leisure and hospitality industries were closed by the government due to preventive measures during our data collection effort. Most of the participants might think that life will return to normal after COVID-19 and leisure activities will be safe to practice. However, a very high score of Precautions-Routine might indicate that during the ninth week of the closeout in Turkey, people strictly practice the advised precautionary behaviors toward daily routine activities to protect themselves and their families.
The control analyses exploring the sensitivity of our dependent and independent variables to demographics depict that females have higher scores in all variables than males, meaning that the female population in Turkey is more perceptive and sensitive to both the protection motivators and precautionary behaviors against the COVID-19 pandemic. This finding is in line with the findings of Lee and You (2020) and Dryhurst et al. (2020) and contradicts Qian et al. (Qian et al., 2020). When the statistics of cases and mortality rates are examined worldwide, it is known that males are more vulnerable than females (Jin et al., 2020; Richardson et al., 2020; WHO, 2020b).
Although studies have shown that older people show more cautious behavior than younger ones (e.g. Li et al., 2020; Andryukov and Besednova, 2021; Barber and Kim, 2021) and reported fatality rate for older cases is significantly higher than younger ones (e.g. WHO, 2020c; Undurraga et al., 2021), we surprisingly observe that most variables are negatively and significantly correlated with age. Similarly, we were expecting that individuals with more children have higher family-related responsibility and the aged individuals are much more sensitive to protection motivators and precautionary behaviors. Besides, we saw that most of our factor variables are negatively related to education level which is opposite to our expectation. These findings are quite difficult to interpret. However, Dryhurst et al. (Dryhurst et al., 2020) reports that relations between age, education and risk perception show different directions changing according to country. Additionally, Qian et al. (Qian et al., 2020) state that they find no evidence that age, education level and marital status are significantly related to the psychological and behavioral responses during the COVID-19 outbreak. Even though it is not easy to interpret, the reason behind those unexpected associations could be the fact that older, experienced and educated individuals tend to be calmer and wiser in case of crises and prefer to avoid fluctuated and exaggerated perceptions, cognitions and practices. We also notice that marital status has some significant effects of factor variables such as married individuals have a higher score of Precautions-Leisure and perceived Severity than single, and single individuals have a higher score of perceived Susceptibility and Info-Seek than married. However, the effect sizes of the sensitivity of the factor variables to demographics are mostly very small and should be considered to abstain from wrong conclusions.
As the main findings of our study, we found that Precautions-Routine is significantly sensitive to cognitive aspects such as perceived Susceptibility, beliefs on Efficacy and Info-Seek dimensions of protection motivators construct in our research model. Surprisingly, Precautions-Routine is found to be insensitive to the perceived Severity. The various perplexing information shared by various sources about the severity of COVID-19 might make people confused and most of the COVID-19 positive cases showing no or mild symptoms and lower Turkish Case Fatality Rate than the world average (Coronavirus Resource Center, 2020) might make people assess the severity of the disease is not very high. Thus, in our multiple regression model, this variable does not emerge as a prominent predictor of Precautions-Routine. This finding can also be interpreted as, apart from the perceived Severity of COVID-19, the other three factors motivate the Turkish population to practice Precautions-Routine.
On the other hand, Precautions-Leisure is found to be significantly sensitive to all four variables of our Protection Motivators for the Precautionary Behaviors scale. This finding shows that perceived Susceptibility, perceived Severity, beliefs on Efficacy and Info-Seek are strong indicators of precautionary behaviors toward leisure activities (mostly avoidance oriented).
The present study provides associations between protection motivators and behavioral responses in line with the concepts of the Public Health Emergency Preparedness, the Protection Motivation Theory and the Health Belief Model. The findings of this research are consistent with some previous findings such as Iorfa et al. (Iorfa et al., 2020) claiming positive relations among COVID-19 knowledge, risk perception and precautionary behavior; Li et al. (Li et al., 2020) reported that perceived severity, public knowledge, perceived controllability and precautionary behaviors are positively associated.
Our findings suggest that highlighting the severity, susceptibility and efficacy of preventive actions through strategic communication can be used as leverage points to increase public awareness and agileness, and motivate the public to engage more seriously in practicing precautionary behaviors. This finding is important for authorities to encourage the public to practice precautionary behaviors against COVID-19 today and upcoming any other infectious epidemic or pandemic tomorrow. Embracing the severity of a disease provides insight into the risk posed to an individual’s own and loved ones’ health. To make people acquired on the severity of a disease, governmental or non-governmental organizations can employ Nudge Theory by using strong eye-catching content or nudges to increase the likelihood of an individual’s making a particular choice, or behaving in a desired way, using a triggered automatic cognitive process to favor the desired outcome (Tagliabue and Simon, 2018). The visuals and experiences of infected and hospitalized individuals or the mourning of family members after their COVID-19 related loss can be useful to enlighten or mobilize the community about the severity of a disease.
The perceived Susceptibility of a contagious disease covers how people see the probability of being caught to the disease. We can advise that the infection rate should be articulated with real-life examples and the lessons learned during filiation efforts not only with abstract numbers. This approach is related to cognitive dissonance theory (Festinger et al., 1956) stating that depending on the importance of the issue and the degree of our discomfort, people are motivated to change their beliefs or behaviors. Providing individuals with real-life information about the prevalence of the disease in the vicinity and the victims known in the close community can trigger the mechanism of dissonance and create discomfort in the individual’s cognitive sphere. An individual tends to balance the discomfort by changing his/her perception on the susceptibility of the contagious disease.
Misinformation, uncontrolled and inadequate expert statements disseminated in various media, vicious rumors and malicious gossips produced and globally disseminated by explicit or tacit malevolent hubs, can confuse people and affect their beliefs. To cope with the misinformation effect (Polak et al., 2016), the beliefs on the efficacy of preventive measures advised/dictated by authorities should be based on the evidence or information on the effectiveness of measures to prevent infection. The arguments based on consistent findings of objective and scientifically solid observations or experiments in a clear form supported by open communication are critical for addressing peoples’ beliefs to promote communities’ beliefs on the efficacy of preventive measures against COVID-19.
Information seeking model theorizes that information-seeking is triggered by an individual’s perception of the current state of knowledge is less than that needed to deal with a problem. Any individual recognizing his or her own deficiency in information or knowledge for reaching a decision struggles to gain more information and knowledge until the individual is convinced that he or she has enough knowledge or information. In addition to inadequate knowledge or information, deliberate misinformation efforts using various media pose threat to the decision process of individuals. To cope with misinformation aiming at individuals’ erroneous or biased decisions, the information-seeking behavior of an individual is one of the most critical traits. Individual ready to acquire knowledge and information on any subject provides readiness for learning, and a better decision-making process based on real-world facts. Information seeking is reasonably related to life-long learning, learning to learn, keeping receptors on to observe and appreciate the environment to create a learning community. The communities, governmental and non-governmental organizations, and parents should encourage and support the trait of information seeking and learning supported by reliable information sources. The habits of questioning, comparing, verifying, benchmarking and systems thinking should be promoted by formal education and official state policies.
To begin with, this research has a few limitations. Due to cross-sectional design, it does not strongly support causal relationships proposed between our factor variables, and findings only reflect the data collection period. People without internet access are not represented. The online data collection method through social networks limits us to knowing the percentage of participants who refuse to fill out the survey instrument. However, during the pandemic, online data collection is the most practical and safest option due to measures taken for interpersonal distancing during our data-gathering period. Online data collection method through social networks might be the reason for our biased sample in terms of education (highly educated). The authors of this paper used their own social networks and the social networks of peers in different universities, academic and governmental institutions, and related students, alumni and occupation-based social networking groups. The characteristic of our sample should be considered before any generalization.
In the final analysis, we think that the predicting variables of our research, namely, perceived Susceptibility, perceived Severity, beliefs on Efficacy and Info-Seek are the concepts that can be utilized to increase the prevalence and the intensity of precautionary behaviors either routine or leisure to protect not only individuals but also the societies from contagious diseases such as COVID-19.
Finally, it is worth stating that this study also provides psychometrically confirmed two authentic scales measuring the levels of protection motivators and the prevalence of PBAC based on individuals’ perceptions. We believe that these two scales, developed for this research, are valuable contributions to literature.
Contributor Information
Ufuk Türen, Turkish Military Academy, Ankara 06654, Turkey.
Yunus Gökmen, Faculty of Communication, Baskent University, Ankara 06790, Turkey.
Haluk Erdem, Turkish Military Academy, Ankara 06654, Turkey.
Gökdeniz Kalkin, Muğla Sıtkı Koçman University, Muğla 48680, Turkey.
REFERENCES
- Andryukov B. G., Besednova N. N. (2021) Older adults: panoramic view on the COVID-19 vaccination. AIMS Public Health, 8, 388–415. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Barber S. J., Kim H. (2021) COVID-19 worries and behavior changes in older and younger men and women. The Journals of Gerontology. Series B, Psychological Sciences and Social Sciences, 76, e17–e23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bashirian S., Jenabi E., Khazaei S., Barati M., Karimi-Shahanjarini A., Zareian S. et al. (2020) Factors associated with preventive behaviours of COVID-19 among hospital staff in Iran in 2020: an application of the protection motivation theory. Journal of Hospital Infection, 105, 430–433. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Beltran D. G., Isch C., Ayers J. D., Alcock J., Brinkworth J. F., Cronk L. et al. (2021) Mask wearing behavior across routine and leisure activities during COVID-19. https://psyarxiv.com/2qya8/download?format=pdf (1 July 2021, date last accessed).
- Carpenter C. J. (2014) Health belief model. In Thompson T. L., Golson G. (eds), Encyclopedia of Health Communication. Sage, Los Angeles, CA, pp. 544–546. [Google Scholar]
- Chang S. J., van Witteloostuijn A., Eden L. (2010) From the editors: common method variance in international business research. Journal of International Business Studies, 41, 178–184. [Google Scholar]
- Coronavirus Resource Center (2020). How does mortality differ across countries? https://coronavirus.jhu.edu/data/mortality (30 May 2020, date last accessed).
- Costa M. F. (2020) Health belief model for coronavirus infection risk determinants. Revista de Saude Publica, 54, 47–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Depoux A., Martin S., Karafillakis S., Preet R., Wilder-Smith A., Larson H. (2020) The pandemic of social media panic travels faster than the COVID-19 outbreak. Journal of Travel Medicine, 27, taaa031. [DOI] [PMC free article] [PubMed] [Google Scholar]
- DeVon H. A., Block M. E., Moyle‐Wright P., Ernst D. M., Hayden S. J., Lazzara D. J. et al. (2007) A psychometric toolbox for testing validity and reliability. Journal of Nursing Scholarship, 39, 155–164. [DOI] [PubMed] [Google Scholar]
- Dryhurst S., Schneider C. R., Kerr J., Freeman A. L. J., Recchia G., van der Bles A. M. et al. (2020) Risk perceptions of COVID-19 around the world. Journal of Risk Research, 23, 994–1006. [Google Scholar]
- Du H., Yang J., King R., Yang L., Chi P. (2020) COVID-19 increases online emotional and health-related searches. Applied Psychology: Health and Well‐Being, 12, 1039–1053. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Festinger L., Riecken H. W., Schachter S. (1956) When Prophecy Fails. Harper and Row, New York, NY. [Google Scholar]
- Hair J. F., Black W. C., Babin B. J., Anderson R. E. (2014) Multivariate Data Analysis, Pearson new international edition, 7th edition. Pearson Education Limited, New York, NY. [Google Scholar]
- Holmes B. J. (2008) Communicating about emerging infectious disease: the importance of research. Health, Risk & Society, 10, 349–360. [Google Scholar]
- Hotle S., Murray-Tuite P., Singh K. (2020) Influenza risk perception and travel-related health protection behavior in the US: insights for the aftermath of the COVID-19 outbreak. Transportation Research Interdisciplinary Perspectives, 5, 100127. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Iorfa S. K., Ottu I. F., Oguntayo R., Ayandele O., Kolawole S. O., Gandi J. C. et al. (2020) COVID-19 knowledge, risk perception, and precautionary behavior among Nigerians: a moderated mediation approach. Frontiers in Psychology, 11, 566773. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Isch C., Ayers J. D., Alcock J., Brinkworth J. F., Cronk L., Hurmuz-Sklias H. et al. (2021) Mask wearing behavior across routine and leisure activities during COVID-19. PsyArXiv, doi: 10.31234/osf.io/2qya8 [DOI] [Google Scholar]
- Janz N. K., Becker M. H. (1984) The health belief model: a decade later. Health Education Quarterly, 11, 1–47. [DOI] [PubMed] [Google Scholar]
- Jin J. M., Bai P., He W., Wu F., Liu X. F., Han D. M. et al. (2020) Gender differences in patients with COVID-19: focus on severity and mortality. Frontiers in Public Health, 8, 152–156. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Johnson R. A., Wichern D. W. (2002) Applied Multivariate Statistical Analysis, Vol. 5, No. 8. Pearson Prentice Hall, Upper Saddle River, NJ. [Google Scholar]
- Jonsson P. (2000) An empirical taxonomy of advanced manufacturing technology. International Journal of Operations & Production Management, 20, 1446–1474. [Google Scholar]
- Kaiser H. F. (1974) An index of factorial simplicity. Psychometrika, 39, 31–36. [Google Scholar]
- Khosravi M. (2020) Perceived risk of COVID-19 pandemic: the role of public worry and trust. Electronic Journal of General Medicine, 17, em203. [Google Scholar]
- Krejcie R. V., Morgan D. W. (1970) Determining sample size for research activities. Educational and Psychological Measurement, 30, 607–610. [Google Scholar]
- Kwok K. O., Li K. K., Chan H. H. H., Yi Y. Y., Tang A., Wei W. I. et al. (2020) Community responses during early phase of COVID-19 epidemic, Hong Kong. Emerging Infectious Diseases, 26, 1575–1579. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lee M., You M. (2020) Psychological and behavioral responses in South Korea during the early stages of coronavirus disease 2019 (COVID-19). International Journal of Environmental Research and Public Health, 17, 2977. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lee M., Ju Y., You M. (2020) The effects of social determinants on public health emergency preparedness mediated by health communication: the 2015 MERS outbreak in South Korea. Health Communication, 35, 1396–1311. [DOI] [PubMed] [Google Scholar]
- Li J., Yang A., Dou K., Wang L., Zhang M., Li X. (2020) Chinese public’s knowledge, perceived severity, and perceived controllability of the COVID-19 and their associations with emotional and behavioural reactions, social participation, and precautionary behaviour: a national survey. PsyArXiv, 10.31234/osf.io/5tmsh [DOI] [PMC free article] [PubMed] [Google Scholar]
- Marsh H. W., Hocevar D. (1985) Application of confirmatory factor analysis to the study of self-concept: first-and higher order factor models and their invariance across groups. Psychological Bulletin, 97, 562–582. [Google Scholar]
- Otway H., Wynne B. (1989) Risk communication: paradigm and paradox. Risk Analysis, 9, 141–145. [Google Scholar]
- Podsakoff P. M., MacKenzie S. B., Lee J. Y., Podsakoff N. P. (2003) Common method biases in behavioral research: a critical review of the literature and recommended remedies. The Journal of Applied Psychology, 88, 879–903. [DOI] [PubMed] [Google Scholar]
- Polak M., Dukała K., Szpitalak M., Polczyk R. (2016) Toward a non-memory misinformation effect: accessing the original source does not prevent yielding to misinformation. Current Psychology, 35, 1–12. [Google Scholar]
- Qian M., Wu Q., Wu P., Hou Z., Liang Y., Cowling B. J. et al. (2020) Psychological responses, behavioral changes and public perceptions during the early phase of the COVID-19 outbreak in China: a population based cross-sectional survey. BMJ Open, 10, e040910. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Ravichandran T., Rai A. (1999) Total quality management in information systems development: key constructs and relationships. Journal of Management Information Systems, 16, 119–155. [Google Scholar]
- Republic of Turkey Ministry of Health (RTMH). (2020a) COVID-19 Infection Guide. https://hsgm.saglik.gov.tr/depo/birimler/goc_sagligi/covid19/rehber/COVID-19_Rehberi20200414_eng_v4_002_14.05.2020.pdf (19 May 2020, date last accessed).
- Republic of Turkey Ministry of Health (RTMH). (2020b) Yeni Koronavirüs Hastalığı (COVID-19). https://covid19bilgi.saglik.gov.tr/tr/ (23 May 2020, date last accessed).
- Richardson S., Hirsch J. S., Narasimhan M., Crawford J. M., McGinn T., Davidson K. W. et al. ; the Northwell COVID-19 Research Consortium. (2020) Presenting characteristics, comorbidities, and outcomes among 5700 patients hospitalized with COVID-19 in the New York City Area. JAMA, 323, 2052–2059. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Rogers R. W. (1975) A protection motivation theory of fear appeals and attitude change. The Journal of Psychology, 91, 93–114. [DOI] [PubMed] [Google Scholar]
- Rogers R. W. (1983) Cognitive and physiological processes in fear appeals and attitude change: a revised theory of protection motivation. In Cacioppo J. T., Petty R. (eds), Social Psychophysiology: A Sourcebook. Guilford Press, New York, NY, pp. 153–176. [Google Scholar]
- Schermelleh-Engel K., Moosbrugger H., Müller H. (2003) Evaluating the fit of structural equation models: tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8, 23–74. [Google Scholar]
- Streiner D. L., Norman G. R. (2003) Health Measurement Scales: A Practical Guide to Their Development and Use. Oxford University Press, Oxford, UK. [Google Scholar]
- Tagliabue M., Simon C. (2018) Feeding the behavioral revolution: contributions of behavior analysis to nudging and vice versa. Journal of Behavioral Economics for Policy, 2, 91–97. [Google Scholar]
- Templeton G. F. (2011) A two-step approach for transforming continuous variables to normal: implications and recommendations for IS research. Communications of the Association for Information Systems, 28, 41–58. [Google Scholar]
- Turkish Medical Association (TTB). (2021) Covid-19 Pandemic Evaluation Reports. https://www.ttb.org.tr/userfiles/files/ttb_ covid_18ay_rapor.pdf (18 February 2022, date last accessed).
- Turkish Statistical Institute (TSI). (2021) Data for Statistics. https://data.tuik.gov.tr/Kategori/GetKategori?p=nufus-ve-demografi-109&dil=1 (22 May 2021, date last accessed).
- Undurraga E. A., Chowell G., Mizumoto K. (2021) COVID-19 case fatality risk by age and gender in a high testing setting in Latin America: Chile, March–August 2020. Infectious Diseases of Poverty, 10, 11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Van Bavel J. J., Baicker K., Boggio P. S., Capraro V., Cichocka A., Cikara M. et al. (2020) Using social and behavioural science to support COVID-19 pandemic response. Nature Human Behaviour, 4, 460–471. [DOI] [PubMed] [Google Scholar]
- Van den Broucke S. (2020) Why health promotion matters to the COVID-19 pandemic, and vice versa. Health Promotion International, 35, 181–186. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Witte K. (1992) Putting the fear back into fear appeals: the extended parallel process model. Communication Monographs, 59, 329–349. [Google Scholar]
- Witte K. (1993) A theory of cognition and negative affect: extending Gudykunst and Hammer’s theory of uncertainty and anxiety reduction. International Journal of Intercultural Relations, 17, 197–215. [Google Scholar]
- World Health Organization (WHO). (2020a) Coronavirus Disease (COVID-19) Advice for the Public. https://www.who.int/emergencies/diseases/novel-coronavirus-2019/advice-for-public (23 May 2020, date last accessed).
- World Health Organization (WHO). (2020b) COVID-19 Weekly Surveillance Report (18–24 May 2020, Epi week 21). http://www.euro.who.int/en/health-topics/health-emergencies/coronavirus-covid-19/weekly-surveillance-report (29 May 2020, date last accessed).
- World Health Organization (WHO). (2020c) WHO Delivers Advice and Support for Older People DURING COVID-19. https://www.who.int/news-room/feature-stories/detail/who-delivers-advice-and-support-for-older-people-during-covid-19 (15 November 2021, date last accessed).
- Worldometers (2020). Countries in the World by Population. https://www.worldometers.info/world-population/population-by-country/ (23 May 2020, date last accessed).
