Development & validation of scales to assess stigma related to COVID-19 in India

Abstract

Background & objectives:

COVID-19 pandemic has triggered social stigma towards individuals affected and their families. This study describes the process undertaken for the development and validation of scales to assess stigmatizing attitudes and experiences among COVID-19 and non-COVID-19 participants from the community.

Methods:

COVID-19 Stigma Scale and Community COVID-19 Stigma Scale constituting 13 and six items, respectively, were developed based on review of literature and news reports, expert committee evaluation and participants’ interviews through telephone for a multicentric study in India. For content validity, 61 (30 COVID-19-recovered and 31 non-COVID-19 participants from the community) were recruited. Test–retest reliability of the scales was assessed among 99 participants (41 COVID-19 recovered and 58 non-COVID-19). Participants were administered the scale at two-time points after a gap of 7-12 days. Cronbach’s alpha, overall percentage agreement and kappa statistics were used to assess internal consistency and test–retest reliability.

Results:

Items in the scales were relevant and comprehensible. Both the scales had Cronbach’s α above 0.6 indicating moderate-to-good internal consistency. Test–retest reliability assessed using kappa statistics indicated that for the COVID-19 Stigma Scale, seven items had a moderate agreement (0.4-0.6). For the Community COVID-19 Stigma Scale, four items had a moderate agreement.

Interpretation & conclusions:

Validity and reliability of the two stigma scales indicated that the scales were comprehensible and had moderate internal consistency. These scales could be used to assess COVID-19 stigma and help in the development of appropriate stigma reduction interventions for COVID-19 infected, and mitigation of stigmatizing attitudes in the community.

The unprecedented outbreak of the novel coronavirus (COVID-19) in early 2020 demanded a coordinated response to contain the disease and address its psychosocial impact1. COVID-19 pandemic created an environment for social stigma and discrimination towards individuals perceived to be infected or vulnerable to the infection2. Stigma was further exacerbated by infection control techniques and restrictive practices such as quarantine, isolation and lockdown, deployed to protect global health3. Similar experiences of stigma in the past have been reported during the outbreak of severe acute respiratory syndrome (and Ebola virus)4,5.

Misinformation driven by erroneous news reports, media stories and public’s misunderstanding of health messages4,6 resulted in stigmatization of individuals or communities7. Studies from India indicated the existence of stigma towards COVID-19 patients, healthcare workers, certain religious groups, people from particular regions, migrant workers and people having an international travel history1,2,8,9. Discriminatory and abusive behaviours towards healthcare workers, frontline workers and COVID-19 patients1,2,10,11 and their families, denial of essential services and resultant mental health distress have been documented8,12,13,14.

Stigma is a major social determinant of health that drives morbidity, mortality and health disparities15,16. Assessing and addressing stigma is necessary, along with clinical and epidemiological management for effective prevention17. Stigma associated with pandemic is a well-documented phenomenon18,19, and several guidelines have been issued by the governmental bodies to address stigma associated with COVID-1920,21.

There are a few studies conducted in Egypt, Turkey and India which have assessed and described stigma towards healthcare and frontline workers during COVID9,10,22,23. However, there is a dearth of literature on validated scales that assess experiences of stigma in COVID-19-infected individuals or stigma attitudes among community members towards COVID-19 infected/suspected24,25,26. Such scales may be valuable to measure stigma related to COVID-19 and contribute to developing appropriate mitigation strategies.

Hence, a study was conceptualized to develop and validate scales to assess (i) stigma experienced by the COVID-19-recovered individuals, and (ii) the prevailing stigmatizing attitudes in the community towards COVID-19 infected. Here we describe the process undertaken in a multicentric study to develop and validate scales for assessing stigma related to COVID-19 and present the validity and reliability of these scales.

Material & Methods

For the development of the scales, the following steps were undertaken: (i) review of literature and selection of scale items; (ii) review of selected items by experts; and (iii) pilot testing of the scales for assessing content validity and test–retest reliability24,27,28.

Review of literature and item development for the scale: Given the evidence on the measurement of HIV stigma, literature related to HIV Stigma Framework29,30 and the HIV Stigma Scale31 was reviewed. The HIV Stigma Framework integrates societal- and individual-level conceptualizations of stigma; acknowledges the power differentials between uninfected and infected individuals and discovers HIV stigma mechanisms for the both. The model proposes that for those who are HIV infected, stigma mechanisms manifest through enacted, anticipated and internalized stigma; while for those who are uninfected, these manifest through display of prejudice, stereotyping and discrimination towards HIV infected, leading to negative outcomes for both. Utilizing this framework, two different scales were developed to assess actual experiences of stigma among COVID-19-recovered individuals and prevailing attitudes of stigma among communities towards COVID-19-affected individuals.

The research team additionally reviewed the HIV Stigma Scale31 and its 12-item short version32 comprising four subscales measuring domains related to personalized stigma (perceived consequences of others knowing about the person’s HIV status such as loss of relationships or avoidance by others), disclosure concerns (regulating information, keeping secrecy and worrying about breach of information), concerns with public attitudes (public perception of people with HIV and its consequences) and negative self-image (shame, guilt and feelings of uncleanness due to HIV status). Being aware of the differences in epidemiology, transmission patterns and route of transmission between HIV and COVID-19, three of the four domains, namely personalized stigma, disclosure concerns and concerns with public attitudes were found to be relevant in the context of COVID-19 stigma.

Apart from this, news items, media stories and reports released by various organizations were also reviewed. This review during the initial stages of the outbreak showed that people infected with COVID-19 were labelled as ignorant and careless33. Patients and their family members were discriminated and denied essential services and facilities34 and treatment by medical authorities35. Asserting that the denial of treatment to patients is a gross violation of human rights, the National Human Rights Commission issued an advisory on ‘Right to health in the context of COVID-19’36.

The aforementioned sources were employed to construct two different scales – one measuring stigma experiences of those who had recovered from COVID-19 (COVID-19 Stigma Scale) and the other measuring community attitudes towards those diagnosed/suspected with COVID-19 (Community COVID-19 Stigma Scale). For the study, COVID-19 recovered individuals were defined as persons who had been COVID-19 positive and had recovered and completed isolation/hospitalization period, while non-COVID-19 participants from the community included persons who were not infected with COVID-19 till the time of the interview. Given the challenges related to conducting field-based face-to-face interviews during the pandemic, it was decided to test the scales through telephonic interviews. For the ease of eliciting responses over the phone, it was decided to use three-point response categories ‘Agree (2)’, ‘Cannot say (1)’ and ‘Disagree (0)’.

Experts panel review: Following the scale development, an expert panel comprising a psychologist, public health practitioner, epidemiologist, sociologist, demographer, statistician and social worker critically reviewed the selected items. The panel suggested the addition of four items to measure stigma experiences of COVID-19 recovered. Thus, COVID-19 Stigma Scale consisted of 13 items and Community COVID-19 Stigma Scale of six items (Table I).

Table I.

Items included in the scales

Scales and Items	Agree	Can’t say	Disagree
COVID-19 Stigma Scale
Items (n=13)
People I cared for stopped calling or interacting after learning that I was infected with/suspected for COVID-19
I have lost friends/relatives after telling them that I was infected with/suspected for COVID-19
My family has gone through a lot of difficulties because of my COVID-19 status
Some people avoid touching me even after my recovery once they know I was infected with/suspected for COVID-19
I have been insulted/discriminated because of my history of being infected with/suspected for COVID-19
The way people treat me makes me feel unwanted
I was denied healthcare services when the doctors found out I was infected with/suspected for COVID-19
I/my family members were denied essential services when the service providers found out I was infected with/suspected for COVID-19
Telling others that I was infected with/suspected for COVID-19 is risky
I work hard to keep my COVID-19 history as a secret
I have stopped socializing with some people because of their reactions towards me due to my COVID-19 status
I am very careful about who I tell that I was infected with/suspected for COVID-19 as I worry that they will disclose it to others
People only refer to me by my COVID-19 status
Community COVID-19 Stigma Scale
Items (n=6)
People infected with COVID-19 are always careless and spread the disease
Most people are uncomfortable around COVID-19-infected people even after their results are negative and after they are discharged from the hospital
People with COVID-19 are treated as an outcast
If a person was infected with COVID-19, it is better to avoid his/her family members
People with the COVID-19 disease got what they deserved
People infected with COVID-19 should continue to be isolated even after their recovery

Pilot testing of the scales: Both COVID-19 Stigma Scale and Community COVID-19 Stigma Scale were pilot tested. The process undertaken, study population and methodology to assess the validity and reliability of the scale are described below.

Study area and population: Given the sociocultural, linguistic and epidemiological diversity of India, it was decided to pilot test both the scales in at least one language of the major geographical zones (North, Central, South, West, East and Northeast) of the country. The study was conducted from May to August 2020 in collaboration with research institutes in seven sites (Gorakhpur, Bhopal, Chennai, Mumbai, Pune, Bhubaneswar and Dibrugarh) from these zones. The interview schedule, participant information sheets and consent forms were translated in local languages, namely Hindi, Tamil, Marathi, Odia and Assamese and back translated to English. The interview schedule additionally included questions on socio-demographic characteristics, i.e. age, gender, education, occupation and marital status of the participants.

Sample size: To assess the internal consistency of the scales, the sample size was calculated using Bonnet formula37 assuming Cronbach’s alpha under the null hypothesis as 0.3, alternate hypothesis as 0.6, five per cent level of significance, 90 per cent power, k=number of items in the tool (6 for Community COVID-19 Stigma Scale and 13 for COVID-19 Stigma Scale) and a 20 per cent non-response. Thus, the sample size calculated was 68 for the non-COVID-19 participants and 63 for the COVID-19-recovered participants.

For assessing test–retest reliability using the Cohen’s kappa coefficient, the desired sample size was 49 for both the scales, assuming kappa under null hypothesis as 0.3 and under alternate hypothesis as 0.6, for a three category Likert scale at five per cent level of significance and 80 per cent power38. Hence, the sample size calculated for internal consistency as mentioned above took care of the minimum required sample size for assessing the test–retest reliability as well.

Recruitment and selection of study participants: COVID-19-recovered and non-COVID-19 participants from the community aged 18 yr and above were recruited from seven study sites. In each of the sites, two investigators trained in social sciences conducted interviews with the participants who provided informed consent orally over the telephone and were included in the study. The Central Ethics Committee for Human Research, ICMR-National Centre for Disease Informatics and Research, Bengaluru, and the Institutional Ethics Committees of the collaborating institutes reviewed and approved all the study documents.

COVID-19-recovered participants: Lists of COVID-19-recovered individuals (and their phone numbers) available with the district health officials or Indian Council of Medical Research (ICMR) COVID-19 testing institutes were obtained during May–July 2020. This list was employed to select participants using circular systematic sampling, separately for men and women, till the desired sample size was achieved

Non-COVID-19 participants from the community: Contact tracing lists from the health department as well as beneficiary data available with community-based organizations were obtained and merged to form a representative sampling frame of non-COVID-19 participants from the community. Participants were selected using circular systematic sampling till the required sample size was achieved.

Assessment of scales: Before administering the two scales, they were assessed for content validity. Content validity was assessed for representation of scale items through review and feedback from two social science experts and for item – comprehensibility through interviews with research participants. For the latter, based on prior studies27, the team administered scales in five languages (Hindi, Tamil, Marathi, Odia and Assamese), to purposively selected six COVID-19-recovered (3 women and 3 men) and six non-COVID-19 participants (3 women and 3 men). In total, from seven sites, 30 participants were recruited.

Following this, to improve the comprehensibility, revision in translation of items was made in two languages (Hindi and Marathi). The COVID-19 Stigma Scale and Community COVID-19 Stigma Scale were pilot tested among a separate set of 58 COVID-19-recovered and 73 non-COVID-19 participants.

Assessment of reliability: Internal consistency across the items and test–retest reliability of responses over time were evaluated to examine reliability of the scales. Of the required sample size of 63 participants, 58 COVID-19-recovered participants were administered the interview schedule at one point in time and re-administered to 41 after a gap of 7-12 days. The remaining participants (17) were unresponsive despite repeated attempts, leading to the final sample size for test–retest reliability to be less than the required size of 49.

For the Community COVID-19 Stigma Scale, although the required sample size was 68, data were collected from 73 community participants at one point in time and re-administered to 58 after a gap of 7-12 days. Fifteen participants either refused or were non-responsive. However, since the required sample was only 49, this was sufficient for assessing test–retest reliability. An analysis of the participants who did not respond for follow up interviews did not indicate any differences in socio-demographic profile compared with those who participated.

Data analysis:

Content validity: Translated documents containing participants, responses to both scales were reviewed with an objective to understand participants’ perspectives on language, comprehensibility and relevance of items.

Reliability of the stigma items: Cronbach’s alpha was used for assessing reliability in terms of internal consistency. Values >0.7 indicate good internal consistency, while those between 0.6 and 0.7 are considered acceptable39. For test–retest reliability, an agreement between responses of the participants at two-time intervals was examined using overall percentage agreement and Cohen’s kappa statistics. Significant kappa value (P<0.05) between 0.41 and 0.60 indicates moderate agreement and above 0.61 indicates substantial agreement40.

Results

Table II describes the socio-demographic characteristics of the COVID-19-recovered and non-COVID-19 participants from the community. The mean age of the COVID-19-recovered participants was 36.3 yr (n=58) and 69 per cent of participants were male. More than half (51.7%) of the participants were literate with higher secondary and above education level. Most of the participants (88.4%) were from urban areas. The mean age of the community participants was 41.7 yr (n=73) and 52 per cent of the participants were male. Majority (68.5%) were literate with higher secondary and above education level. Sixty three per cent of the participants were from urban area.

Table II.

Percentage distribution of participants by socio-demographic characteristics

Socio-demographic characteristic	COVID-19-recovered participants	Non-COVID-19 participants from the community
Number of participants	58	73
Age
Mean age (yr)	36.33	41.66
SD	12.09	16.48
Range	19-82	20-80
Age group (yr)
18-29	31.0	27.4
30-44	44.8	39.7
45-59	20.7	11.0
≥60	3.4	21.9
Sex
Male	69.0	52.1
Female	31.0	47.9
Education
Illiterate	6.9	5.5
1-8 std	15.5	9.6
9-11 std	25.9	16.4
12 and above	51.7	68.5
Occupation
Employed in government or private sector	22.4	35.6
Labour	17.2	2.7
Business/self-employed	17.2	18.3
Housewives	24.1	19.2
Students	6.9	6.8
Unemployed	8.6	4.1
Others (retired, farming, priest)	3.4	13.7
Marital status
Never married	25.9	17.8
Currently married	69.0	74.0
Separated	5.2	8.2
Residence
Urban	88.4	63.0
Rural	11.6	37.0

Content validity: Experts reviewed both the scales and found the items to be relevant for eliciting information pertaining to the constructs of COVID-19 stigma. Apart from translation errors in Marathi and Hindi COVID-19 Stigma Scale, the participants did not express any difficulties in comprehension.

Internal consistency: Cronbach’s α was observed to be 0.72 for COVID-19 Stigma Scale indicating good internal consistency and 0.62 for Community COVID-19 Stigma Scale indicating internal consistency in an acceptable range.

Test–retest reliability: The percentage agreement of responses at two-time points ranged from 66 to 81 per cent for the COVID-19 Stigma Scale and 51-79 per cent for the Community COVID-19 Stigma Scale. The test–retest reliability indicated that out of 13 items for the COVID-19 Stigma Scale, seven items had a moderate agreement (0.4-0.6) and for rest of the items, the kappa statistics was less than 0.4. For the Community COVID-19 Stigma Scale, of the six items, four items had a moderate agreement and for the remaining two items, the value was <0.1 (Table III).

Table III.

Test–retest reliability of scales by COVID-19 -recovered and non-COVID-19 participants from the community

COVID-19-recovered participants			Non-COVID-19 participants from the community
Items	Percentage agreement	Kappa (P)	Items	Percentage agreement	Kappa (P)
People I cared for stopped calling or interacting after learning that I was infected with/suspected for COVID-19	70.7	0.45 (0.001)	People infected with COVID-19 are always careless and spread the disease	70.2	0.49 (<0.001)
I have lost friends/relatives after telling them that I was infected with/suspected for COVID-19	78.0	0.51 (<0.001)	Most people are uncomfortable around COVID-19-infected people even after their results are negative and after they are discharged from the hospital	70.2	0.42 (<0.001)
My family has gone through a lot of difficulties because of my COVID-19 status	73.2	0.37 (0.016)	People with COVID-19 are treated as outcast	71.9	0.47 (<0.001)
Some people avoid touching me even after my recovery once they know I was infected with/suspected for COVID-19	78.0	0.56 (<0.001)	If a person was infected with COVID-19, it is better to avoid his/her family members	78.9	0.40 (0.001)
I have been insulted/discriminated because of my history of being infected with/suspected for COVID-19	70.7	0.43 (0.004)	People with the COVID-19 disease got what they deserved	63.2	0.07 (0.550)
The way people treat me makes me feel unwanted	70.7	0.41 (0.009)	People infected with COVID-19 should continue to be isolated even after their recovery	50.9	0.09 (0.432)
I was denied health care services when the doctors found out I was infected with/suspected for COVID-19	75.6	0.18 (0.173)
I/my family members were denied essential services when the service providers found out I was infected with/suspected for COVID-19	65.9	0.23 (0.003)
Telling others that I was infected with/suspected for COVID-19 is risky	80.5	0.47 (0.007)
I work hard to keep my COVID-19 history as a secret	80.5	0.24 (0.072)
I have stopped socializing with some people because of their reactions towards me due to my COVID-19 status	70.7	0.43 (0.002)
I am very careful about who I tell that I was infected with/suspected for COVID-19 as I worry that they will disclose it to others	73.2	0.32 (0.026)
People only refer to me by my COVID-19 status	68.3	0.23 (0.125)

Discussion

The two scales, namely COVID-19 Stigma Scale and Community COVID-19 Stigma Scale developed in this multicentric study conducted in different geographical zones of India presented a unique perspective towards understanding the stigma through the stigmatized and stigmatizer lens. Assessment of content validity of scales demonstrated good comprehension of items by the participants. Similar results for language validity of the scale were found in a study that assessed cancer stigma41. Findings of internal consistency highlighted reliable values of Cronbach’s alpha for COVID-19 Stigma Scale and Community COVID-19 Stigma Scale. The test–retest reliability determined using kappa statistics was mostly >0.4 and significant, indicating that the responses of the participants taken at varied intervals of time for the same items were similar and reliable. The findings were in agreement with the results of studies conducted in Southern Ethiopia28 and Turkey41.

Some items in the COVID-19 Stigma Scale were constructed from the HIV Stigma Scale31. A few earlier studies have also adapted this scale to assess stigma among cancer and hepatitis C patients27,41,42 and COVID-19 recently11. Although stigma related to COVID-19 have been assessed among patients and healthcare workers1,22,43,44, this is one of the few studies that has developed a scale to assess stigmatizing attitudes towards the COVID-19-infected/suspected individuals.

The findings need to be interpreted in light of the following limitations. Data collection using telephonic interviews, the only feasible way to collect data during the pandemic, may have led to selection bias, difficulty in comprehension of items and socially desirable responses. In addition, evolving understanding of the epidemic in different locations during the first wave and less than the required number of participants for assessing COVID-19 Stigma Scale may have resulted in low test–retest reliability of the scale. As the scale was developed during the initial stages of the pandemic, it is possible that some of the stigma dimensions were not included in both the scales. The strength of this study rests in the fact that the scales were developed by a multidisciplinary team of experts who drew from varied sources of literature and culturally contextual perspectives on COVID-19-related stigma in India. The scales may be relevant in detecting stigma related to other infectious diseases in the future and planning interventions in India or countries with similar epidemic settings.