The Coronavirus Impact Scale: Construction, Validation, and Comparisons in Diverse Clinical Samples

Abstract

Objective

This report is of the construction and initial psychometric properties of the Coronavirus Impact Scale in multiple large and diverse samples of families with children and adolescents. The scale was established to capture the impact of the coronavirus pandemic during its first wave. Differences in impact between samples and internal structure within samples were assessed.

Method

A total of 572 caregivers of children and adolescents or expecting mothers in diverse clinical and research settings completed the Coronavirus Impact Scale. Samples differed in regard to developmental stage, background, inpatient/outpatient status, and primary research or clinical setting. Model free methods were used to measure the scale’s internal structure and to determine a scoring method. Differences between samples in specific item responses were measured by multivariate ordinal regression.

Results

The Coronavirus Impact Scale demonstrated good internal consistency in a variety of clinical and research populations. Across the groups studied, single, immigrant, predominantly Latinx mothers of young children reported the greatest impact of the pandemic, with noteworthy effects on food access and finances reported. Individuals receiving outpatient or inpatient care reported greater impacts on health care access. Elevated scores on the Coronavirus Impact Scale were positively associated with measures of caregiver anxiety and both caregiver- and child-reported stress at a moderate effect size.

Conclusion

The Coronavirus Impact Scale is a publicly available scale with adequate psychometric properties for use in measuring the impact of the coronavirus pandemic in diverse populations.

By March 2020, it was clear that coronavirus was resulting in profound disruptions to the lives of children and their families. The authors were all conducting studies and clinic visits that needed a brief measure by caregiver report to capture coronavirus-related impact. In addressing this need, we were guided by 2 principles: brevity and non-redundancy. First, the scale must be brief and a low burden for respondents. Second, the scale should avoid redundancy with previously validated clinical scales. The resultant scale and its initial psychometrics are reported here.

The scale was designed to be completed by an expecting mother or a caregiver of a child in mental health treatment or research. To meet a novel, acute measurement need during the first wave of the pandemic, the scale was constructed rapidly, relying on discourse among measurement experts and clinicians who were directly working with families impacted by the pandemic. At the time of construction, aspects of the impact of epidemics and natural disasters were already established, especially increases in symptoms of stress.¹ The guiding principles of brevity and non-redundancy, prior knowledge about disaster response, and immediate clinical and research experience with families in a novel situation resulted in the inclusion of diverse items with wide coverage of impact indicators broadly covering stress response, economic impact, and access to care. These inclusions would correspond to major foci of research interest in pandemic impact later identified by pediatric mental health practitioners and researchers.²

The study team made the scale immediately available to the public via the United States’ National Institutes of Health (NIH) Disaster Research Response (DR2) program website (with an initial deposit on April 1, 2020).³ The scale was also included as a measure of the Adolescent Brain and Cognitive Development study. The study team completed regulatory requirements and was able to gather data from clinically relevant samples between April 2020 and January 2021. Initial psychometrics were promising and were made available on the NIH DR2 program website with the scale’s manual, and an initial version of this report was published as a preprint on PsyArXiv on May 24, 2021.

Since the construction, distribution, and preliminary report of the scale, a great deal has been learned about children’s and adolescent’s responses to the pandemic as well as its impact on their well-being. In the United States, relative to the corresponding time in 2019, there was an initial decline and then an increase in emergency department evaluations for mental health issues, with adolescents in particular seeking emergent aid.⁴ Indeed, relative to 2019, US adolescents and young adults reported more depressive and anxiety symptoms⁵ and were also more frequently evaluated in the emergency departments for suicide attempts.⁶ As many authors are also US practitioners in intensive mental health services, these reports reflected our contemporary experience at the time, which informed the CIS construction. However, there is much to learn, especially from investigations grounded in evidence. On broad review of the reports of the pandemic on child and adolescent mental health, Cortese et al⁷ have systematically identified a relative need for reports that are data driven, longitudinal, or interventional. To that end, this report provides some novel data in clinical samples and will inform later longitudinal analysis of publicly available longitudinal data of the scale responses (https://nda.nih.gov/abcd).

To provide initial validation, the scale was integrated into ongoing and new studies of children and their families in mental health treatment and research. The authors collaborated to test the scale in 5 diverse samples. These were recruited to represent different clinical situations, geographic locations, and patient backgrounds to test its broad utility. The authors also had the ability to rapidly engage these participants for research. The overall goal was to understand the scale’s internal structure and preliminary concurrent associations.

The target populations of the samples included in this report are as follows:

• •Caregivers of children receiving outpatient mental health care from a large children’s hospital in Colorado
• •Caregivers of children and adolescents who were being admitted to a child and adolescent psychiatric inpatient service in Baltimore, Maryland
• •Mothers of young children involved in parent−child dyadic psychotherapy in Massachusetts; the mothers are primarily Latinx immigrants with a trauma history
• •Caregivers of youth with primary diagnoses of attention-deficit/hyperactivity disorder (ADHD), reflecting severe chronic irritability, and anxiety disorders and youth with no history of DSM-5 diagnosis participating in research at the National Institute of Mental Health (NIMH)
• •Women recruited prenatally who were participating in a longitudinal study of the impact of stress on pregnancy, birth outcomes, and early child development

The goal of this article is to report how the scale was constructed and to test its internal consistency to provide guidance on scoring and interpretation. The main hypothesis tested was that the summative score in the scale would provide a measure of impact across samples by contemporary psychometric standards. The internal structure of the scale, for example, how individual items cluster together, was illustrated in a model-free fashion and was not expected to be the same across all samples. In addition, differences in impact response profiles were tested across groups. Finally, to provide external validation, concurrent associations between overall impact and behavioral measures of anxiety and stress in groups were tested, where we expected positive associations to both.⁵

Method

Participants

Participants were recruited at from 5 locations or cohorts: Children’s Hospital Colorado (CHCO), Johns Hopkins Medical Institutions (JHMI), Early Connections/Conexiones Tempranas (EC/CT), Boston University, National Institute of Mental Health (NIMH) Intramural Research Program, and Prospective Research on Early Determinants of Illness and Children’s Health Trajectories (PREDICT). The study was approved by the assigned institutional review boards, and consent was obtained. Details on participant characteristics and accrual are found in Supplement 1, available online, and in Table 1.

Table 1.

Sample Characteristics

	CHCO (n = 159)	JHMI (n = 112)	EC/CT (n = 41)	NIMH (n = 73)	PREDICT (n = 187)	Comparison statisticsa
Child characteristics
Age, y, mean (SD)	12.0 (3.4)	14.4 (2.1)	2.9 (1.8)	14.0 (2.8)	3.0 (1.3)	F4,567 = 638.5, p < .001 (EC/CT,PREDICT)<CHCO<(JHMI,NIMH)
Natal sex (% Female)	48	72	51	38	50	χ24 = 25.29, p <.001 (CHCO,NIMH)<JHMI
Race, %b
American Indian	4	0	0	3	0	Fisher test p = .011
Asian	10	4	0	4	4	Fisher test p = .059
Black	9	39	8	10	9	χ24 = 64.1, p < .001 (CHCO,EC/CT,NIMH,PREDICT)<JHMI
White	86	52	31	68	84	χ24 = 83.2, p < .001 (EC/CT,JHMI,NIMH)<(CHCO,PREDICT) EC/CT<NIMH
Other	6	5	51	14	3	χ24 = 103.7, p < .001 (CHCO,JHMI,NIMH,PREDICT)<EC/CT PREDICT<NIMH
Ethnicity, %b
Latinx	13	5	83	14	7	χ24 = 170.0, p < .001 (CHCO,JHMI,NIMH,PREDICT)<EC/CT
Caregiver characteristics
Age, y, mean (SD)	45.6 (13.3)	NA	32.0 (5.6)	NA	33.5 (5.2)	F2,384 = 80.6, p < .001 (EC/CT,PREDICT)<CHCO
Natal sex, % female	NA	76	100	NA	100	NA
Marital status, %
Married	79	51	29	NA	73	χ23 = 50.9, p < .001 (EC/CT,JHMI)<(CHCO,PREDICT)
Divorced	9	20	2	NA	2	Fisher test p < .001 PREDICT<(JHMI,CHCO)
Widowed	0	5	0	NA	0	Fisher test p = .001 (CHCO,PREDICT)<JHMI
Separated	4	5	20	NA	0.5	Fisher test p < .001 (CHCO, JHMI, PREDICT)<EC/CT
Never married	6	20	49c	NA	6	χ23 = 37.6, p < .001 (CHCO,JHMI,PREDICT)<EC/CT CHCO<(JHMI,PREDICT)
Unmarried couple	3	0		NA	17
Educational attainment, %
Less than high school	0	3	39	2	0.6	Fisher test p < .001 (CHCO,JHMI,NIMH,PREDICT)<EC/CT
High school or GED	3	19	27	2	11	Fisher test p < .001 (CHCO,NIMH)<(JHMI,EC/CT) CHCO<PREDICT<EC/CT
Some college	11	28	5	8d	26	χ24 = 21.7, p < .001 NIMH<PREDICT
Associate’s degree	11	3	5		11
Bachelor’s degree	38	26	22	14	33	χ24 = 15.1, p = .005 NIMH<CHCO
Master’s degree	25	15	0	75d	13	χ24 = 113.0, p < .001 EC/CT<(JHMI,PREDICT)<CHCO<NIMH
Doctoral degree	12	5	0		6
Employment, %
Working now	59	83	51	NA	65	χ23 = 21.0, p < .001 (CHCO,EC/CT,PREDICT)<JHMI
Temporary leave	3	2	2	NA	0	Fisher test p = .083
Unemployede	2	6	12 34	NA	0.5	Fisher test p < .001 (CHCO,JHMI,PREDICT)<EC/CT PREDICT<JHMI
Retired	3	2	0	NA	2	Fisher test p = .965
Disabled	6	3	0	NA	0	Fisher test p = .001 PREDICT<CHCO
Keeping house	23	4	NA	NA	29	χ22 = 26.0, p < .001 JHMI<(CHCO,PREDICT)
Student	0.6	0	0	NA	0	Fisher test p = .618
Other	4	0	14	NA	3	Fisher test p = .101

Note: Throughout, “NA” is used to indicate that these data were not collected for this sample or a comparison test is inappropriate. All descriptive and comparison statistics are of nonmissing values. Missing values for each group are as follows. CHCO: 1 for education; JHMI: 5 for marital status, 24 for education, 10 for employment; EC/CT: 2 for race; NIMH: 2 for race, 3 for ethnicity; PREDICT: 8 for natal sex, 4 for race, 5 for ethnicity, and 20 for education. CHCO = Children’s Hospital Colorado; EC/CT = Early Connections/Conexiones Tempranas; GED = graduate equivalency degree; JHMI = Johns Hopkins Medical Institute; NIMH = National Institute of Mental Health; PREDICT = Prospective Research on Early Determinants of Illness and Children’s Health Trajectories.

^aComparison statistics by test of mean difference (1-way analysis of variance) or proportion (χ² approximation or Fisher exact test as appropriate) for nonmissing values. A comparison test is not made for caregiver natal sex, where the null hypothesis is invalid because 2 samples recruited only women. Two-tailed, pairwise comparisons are Holm−Bonferroni corrected. All significant tests (p < .05) are indicated by a “<” or “>” character. Multiple group differences may be indicated by parenthetical groupings; for example (CHCO,JHMI,PREDICT)<EC/CT indicates that the value for EC/CT is greater than the values of each of the CHCO, JHMI, and PREDICT groups.

^bFor the EC/CT and PREDICT sample, race and ethnicity report is for the whole family; for CHCO, JHMI, and NIMH, caregivers are prompted to respond about their child.

^cNever married and unmarried couples are not distinguished in the EC/CT sample.

^dNIMH data do not distinguish between “some college” and “associate’s degree” or graduate degrees.

^eUnemployed is accompanied by “looking for work” for the CHCO, JHMI, and NIMH samples. For EC/CT, unemployment has 2 categories. The first is “looking for work (12%), and the second is “not looking for work” (34%). This latter category cannot be distinguished from keeping house, which is not probed for this sample.

Coronavirus Impact Scale: Measure Development

The Coronavirus Impact Scale is a 12-item scale assessing multiple aspects of COVID-19−related effects on children and families (Table 1). The first 8 items are indicators of COVID-19 impact across a broad array of domains to which respondents make a 4-level ordinal response on severity with anchored prompts. Items 9 through 11 reflect direct or familial experience with SARS-CoV-2 infection and symptoms. Item 9 is rated on a 4-level ordinal scale, whereas items 10 and 11 have a fifth level to indicate death due to coronavirus. The 12th item is a free response item that is not part of the scaled scores; it prompts respondents to list potential additional ways that coronavirus has affected their life. In the PREDICT cohort, for continuity with a prior wave of data collection, the question about financial income impact was asked separately, outside of the Coronavirus Impact Scale questions.

The Coronavirus Impact Scale was developed rapidly at the onset of the pandemic in a 2-step process. Drs. Stoddard and Kaufman originally drafted the scale’s first items as a supplement to the clinical interview of the Yale−Vermont Adversity in Childhood Scale.⁸ Therefore, it follows that scale’s structure in providing respondents with well-anchored response options ordered on severity. As depicted in Table 2, respondents are probed to rate the impact of coronavirus on multiple domains of life (eg, routines, medical care access, income, etc) using the detailed ordinal anchors.

Table 2.

The Coronavirus Impact Scale

Rate how much the coronavirus pandemic has changed your life in each of the following ways.

1. Routines: 0. No change. 1. Mild. Change in only one area (eg, work, education, social life, hobbies, religious activities). 2. Moderate. Change in two areas (eg, work, education, social life, hobbies, religious activities). 3. Severe. Change in three or more areas (eg, work, education, social life, hobbies, religious activities).

2. Family Income/Employment: 0. No change. 1. Mild. Small change; able to meet all needs and pay bills. 2. Moderate. Having to make cuts but able to meet basic needs and pay bills. 3. Severe. Unable to meet basic needs and/or pay bills.

3. Food Access: 0. No change. 1. Mild. Enough food but difficulty getting to stores and/or finding needed items. 2. Moderate. Occasionally without enough food and/or good quality (eg, healthy) foods. 3. Severe. Frequently without enough food and/or good quality (eg, healthy) foods.

4. Medical health care access: 0. No change. 1. Mild. Appointments moved to telehealth. 2. Moderate. Delays or cancellations in appointments and/or delays in getting prescriptions; changes have minimal impact on health. 3. Severe. Unable to access needed care resulting in moderate to severe impact on health.

5. Mental health treatment access: 0. No change. 1. Mild. Appointments moved to telehealth. 2. Moderate. Delays or cancellations in appointments and/or delays in getting prescriptions; changes have minimal impact. 3. Severe. Unable to access needed care resulting in severe risk and/or significant impact.

6. Access to extended family and non-family social supports: 0. No change. 1. Mild. Continued visits with social distancing and/or regular phone calls and/or televideo or social media contacts. 2. Moderate. Loss of in person and remote contact with a few people, but not all supports. 3. Severe. Loss of in person and remote contact with all supports.

7. Experiences of stress related to coronavirus pandemic: 0. None. 1. Mild. Occasional worries and/or minor stress-related symptoms (eg, feel a little anxious, sad, and/or angry; mild/rare trouble sleeping). 2. Moderate. Frequent worries and/or moderate stress-related symptoms (eg, feel moderately anxious, sad, and/or angry; moderate/occasional trouble sleeping). 3. Severe. Persistent worries and/or severe stress-related symptoms (eg, feel extremely anxious, sad, and/or angry; severe/frequent trouble sleeping).

8. Stress and discord in the family: 0. None. 1. Mild. Family members occasionally short-tempered with one another; no physical violence. 2. Moderate. Family members frequently short-tempered with one another; and/or children in the home getting in physical fights with one another. 3. Severe. Family members frequently short-tempered with one another and adults in the home throwing things at one another, and/or knocking over furniture, and/or hitting and/or harming one another.

9. Personal diagnosis of coronavirus. 0. None. 1. Mild. Symptoms effectively managed at home. 2. Moderate. Symptoms severe and required brief hospitalization. 3. Severe. Symptoms severe and required ventilation.

10.Number of immediate family members diagnosed with coronavirus: ___ Rate the symptoms of the person who was most sick: 0. None. 1. Mild. Symptoms effectively managed at home. 2. Moderate. Symptoms severe and required brief hospitalization. 3. Severe. Symptoms severe and required ventilation. 4. Immediate family member died from coronavirus.

11.Number of extended family member(s) and/or close friends diagnosed with coronavirus: ____ Rate the symptoms of the person who was most sick: 0. None. 1. Mild. Symptoms effectively managed at home. 2. Moderate. Symptoms severe and required brief hospitalization. 3. Severe. Symptoms severe and required ventilation. 4. Extended family member and/or close friend died of coronavirus.

12.Other. Please tell us about any other ways the coronavirus pandemic has impacted your life:

After determining its layout and several draft items, the second step involved soliciting expert and clinician opinion for item development for indicators of impact of the pandemic on the lives of families who were involved in mental health treatment or research. Over the course of 2 weeks, authors as well as their collaborators and clinical colleagues who were working directly with affected families recommended a pool of topics that was iteratively reviewed and condensed to 21 probes. These were further condensed to 8 conceptually nonoverlapping items to indicate impact across multiple domains that were evident in March to April 2020. Items were selected to survey impact without a specific subscale structure in mind. Three additional items were added to probe direct or familial experience with coronavirus diagnoses. Finally, given the preliminary nature of the scale, an open-ended item was added querying about further potential impacts from the respondent. The scale was finalized on April 15, 2020, and was made available on PhenX and National Library of Medicine Disaster Research Response sites. On April 28, 2020, the scale was forward and backward translated into Spanish by 2 native Spanish speakers who were bilingual, bicultural, and working with Dr. Paris.

There are 2 versions of the scale. The 4.15.2020 version of the scale allows ratings of items 10 and 11 on a scale of 1 to 4 (mild symptoms to death). The more recent, 5.2.2020 version (Table 1) expands the scale of these 2 items to 0 to 4, making explicit a recommendation to give a score of 0 for “none” on these 2 items indicating no symptoms or no person was ill. All respondents in this study completed the 5.2.2020 version of the scale.

Additional Measures

Anxiety and stress measures were available in the CHCO and PREDICT samples. For CHCO, these were 3 Patient-Reported Outcomes Measurement Information System (PROMIS) short forms, parent report for child (ie, “parent proxy”) for anxiety (version 2.0, 8a),⁹ physical stress experiences (version 1.0, 8a),¹⁰ and psychological stress experiences (version 1.0, 8a).¹⁰ Physical experiences of stress include stress-associated symptoms of trouble breathing, shaking, and muscle tension, whereas psychological symptoms include feeling under pressure, concentration difficulties, and difficulties managing life. Concurrent validity measures reported here for the PREDICT sample include the PROMIS short form adult self-report for anxiety (version 2.0, 6a)¹¹ and the Perceived Stress Scale.¹²

Data Analysis

A major analytic goal was to measure internal consistency to determine a scoring method, for which a conservative, model-free method best suited the size of the Coronavirus Impact Scale and our sample sizes. Therefore, our a priori primary measure of interest for internal consistency was the Revelle β, which is the minimum reliability value of all possible split-halves.¹³ The standardized Revelle β represents the proportion of variance in responses that are attributable to a common, that is, general, factor across items.¹⁴ Standardized response values are derived from polychoric correlations. To interpret the sum of items on a scale as adequately representing a unitary construct a common rule has been Revelle β > 0.5.¹³ Scale substructure was determined by an item cluster analysis (iclust) approach, a model-free method of determining the internal structure of a scale that is ideal for short measures.^15,16 Finally, Cronbach α on ordinal items responses converted to integers as commonly done is also reported. This metric, raw Cronbach α, estimates the expected correlation between the Coronavirus Impact Scale and a test just like it in an identical sample.

Analyses were conducted using package psych v 2.0.12¹⁷ in the R v 4.0.3 environment.¹⁸ Nine participants from the JHMI group were missing 5 or more responses because of not completing the second page of the scale presented to them. Their scales were not used for further analysis. Data from the remaining 563 individuals were all complete except 26 participants missing 1 item, 22 missing 2 items, and 2 missing 3 items. Missingness was similar across the groups but was more common for items 10 and 11, which indicated the number of family members who were ill. The item completion rate is detailed in Table S1, available online. Missingness was rare for items 1 through 8, in which only 1.4% of individuals were missing a single response. Imputation for sum scores was done by replacing missing values with median values for the item across all samples. Deleting cases with missing responses instead of imputing missing responses did not change results.

Differences between samples of within-participant item ratings were tested by multivariate ordinal regression using package mvord v 1.1.1 and using multivariate normally distributed errors and a general covariance structure.¹⁹ To improve interpretability, significant estimates of sample deviations from the mean response (across groups) at p < .001 are reported in the text, although full results can be found in Table S2, available online. A preliminary examination of parent−child impact score concordance was assessed via interclass, that is, Pearson, correlation in the NIMH sample.

Results

Participant Characteristics

The samples’ distinctive characteristics are evident in self-report of demographics depicted in Table 1. Additional details about the NIMH concordance sample are in Table S3, available online.

Internal Consistency

Items 1 through 8 were well distributed across all samples, allowing for further analysis of internal consistency (Table S1, available online). The 2 items 9 and 10, probing for direct or immediate family experience of infection, were unlikely to have nonzero responses, yielding distributions too kurtotic and skewed to assess the item. In any of the 5 samples, the greatest percentage of nonzero responses for these items was 11.4%. As may be expected, item 11, a report of extended family or friends having a diagnosis of COVID-19, was also zero-inflated but mixed across samples in terms of acceptable distribution, with nonzero responses ranging from 15.4% to 46.2% (Table S1, available online).

Therefore, items 1 through 8 were carried forward as reasonably well distributed (eg, the absolute value of skew and kurtosis < 2 across all samples), for assessment of internal consistency. Measures of internal consistency and internal structure are displayed in Table 3. Notably, in all 5 samples, the internal consistency of items 1 through 8 yielded acceptable standardized Revelle β = 0.51 to 0.70, with β values greater than 0.5 suggesting that the items on a scale represent a unitary construct. They also yielded acceptable Cronbach α measures of reliability (α = .64-.75). This suggests that the sum of these items are adequately interpretable as an overall measure of COVID-19 impact. Similarly, in support of the measure comprising a single scale, in all samples, the optimally reliable cluster solution was 1, with similar loadings of items onto a single cluster (very simple structure [VSS] fit = 0.90-0.96) (Table 3).

Table 3.

Internal Consistency and Structure of the Coronavirus Impact Scale

	CHCO	JHMI	EC/CT	NIMH	PREDICT
Raw αa	0.75	0.64	0.70	0.74	0.73
Standardized βb	0.67	0.51	0.63	0.65	0.70
Cluster analysis
Optimal no. of clustersc	1	1	1	1	1
VSS fitd	0.94	0.91	0.91	0.90	0.96
Item loadingse
1. Routines	0.60	0.62	0.43	0.65	0.68
2. Income	0.52	0.48	0.55	0.38	0.47
3. Food	0.68	0.55	0.79	0.53	0.60
4. Medical access	0.76	0.60	0.56	0.54	0.57
5. Psych access	0.61	0.38	0.45	0.56	0.60
6. Support	0.61	0.48	0.51	0.67	0.47
7. Personal stress	0.61	0.65	0.73	0.85	0.74
8. Family stress	0.43	0.27	0.46	0.61	0.53

Note: CHCO = Children’s Hospital Colorado; EC/CT = Early Connections/Conexiones Tempranas; JHMI = Johns Hopkins Medical Institute; NIMH = National Institute of Mental Health; PREDICT = Prospective Research on Early Determinants of Illness and Children’s Health Trajectories; VSS = very simple structure.

^aRaw Cronbach α estimates reliability, or the expected correlation between the Coronavirus Impact Scale and a test just like it in an identical sample. Raw values are calculations based on ordinal values converted to integers.

^bStandardized Revelle β, or minimum split half correlation, represents the proportion of variance in responses that are attributable to a common, that is, general, factor across items. Standardized values are derived from polychoric correlations.

^cThe number of clusters optimized on both α and β. Here the optimized number of clusters is one across samples, suggesting a unidimensional scale across samples.

^dVery simple structure (VSS) fit metric used by Revelle for iclust; 0 is poor and 1 is perfect.

^eThese loadings may be interpreted as the loadings onto a single factor solution in factor analysis.

Internal Structure

Despite being adequately unidimensional for interpreting a sum score across items 1 through 8, there is certainly substructure in the scale. The more detailed cluster dendrograms reveal this structure (Figures S1-S5, available online) and demonstrate a remarkably similar structure across all groups.

Impact Scores and Response Profiles by Group

Overall, mean coronavirus impact score (sum of items 1-8) differed between groups (analysis of variance [ANOVA] F_4,558 = 19.2, p < .001). In Holm−Bonferroni corrected pairwise t tests, the mean (SD) total impact score for EC/CT [12.0 (4.2)] and CHCO [11.1 (3.7)] did not significantly differ, but each was significantly greater than those of JHMI [9.4 (3.5)], NIMH [8.7 (3.3)], and PREDICT [8.2 (3.6)], with all corrected p values <.002. The mean impact score for JHMI was greater than that of PREDICT (p = .026). Otherwise, there were no significant pairwise differences.

In addition to differences in total scores, there were some differences in rated items (Figure 1). In multivariate ordinal regression, some samples significantly deviated from the average ratings across all samples for an item. Highly significant deviations (all p values <.001) are reported here, and the full regression table (Table S2, available online) provides complete results. Impact on items 2 (family income and employment) and 3 (food access) were rated higher by those in the EC/CT group and lower by those in the NIMH group. Those in the EC/CT group also rated item 4 (medical care access) at a higher level than the mean rating of all groups. Item 5 (mental health care access) was rated higher by the JHMI and CHCO groups. Items 7 (personal stress) and 8 (family stress/discord) were rated higher by the CHCO group. The PREDICT sample rated impact less for items 1 (routines), 4 (medical care access), 5 (mental health care access), 7 (personal stress), and 8 (family stress/discord), and they rated greater impact for item 2 (income).

Figure 1.

Mean Ratings by Sample for Each of the First Eight Coronavirus Impact Scale Items

Note:Items are in order of presentation in the scale (as inTable 2).chco = Children’s Hospital Colorado; CIS = Coronavirus Impact Scale; ecct = early connections/conexiones tempranas; jhmi = Johns Hopkins Medical Institutions; nimh = National Institute of Mental Health; pred = Prospective Research on Early Determinants of Illness and Children’s Health Trajectories.

Impact Score Associations With Infection

Although items 9 through 11 did not have properties allowing them to participate in a test of internal consistency, we could test their relationship to the impact score, the sum of items 1 through 8, across the sample. Across samples, impact scores are less for those reporting no personal diagnosis or no symptoms vs those reporting any level of symptoms [mean (SD) = 9.3 (3.7) and 12.4 (4.4), respectively, t(49.7) = 4.5, p < .001, d = 0.79, a large standardized difference]. When referencing family members or extended family/friends, no significant difference in average impact score was found for those reporting no diagnosis or symptoms vs any level of symptoms (all p values ≥.29, d ≤ 0.18).

Impact Score Associations With Anxiety and Stress and Caregiver−Child Concordance

In the CHCO sample, the impact score predicted parent-report of index child anxiety [b (SE) = 0.48 (0.23), p = .049, adjusted R² = 2.0%], physical stress [b (SE) = 0.96 (0.23), p < .001, adjusted R² = 9.7%], and psychological stress [b (SE) = 0.71 (0.19), p < .001, adjusted R² = 8.1%] in in a multivariate linear model. Similarly, in the PREDICT sample, total impact score predicted anxiety [b (SE) = 0.39 (0.093), p < .001, adjusted R² = 8.4%] and perceived stress [b (SE) = 0.70 (0.13), p < .001, adjusted R² = 12.4%).

In the NIMH sample, the correlation between caregiver-report and child-report on the same family was r (47) = 0.57, p < .001.

Discussion

This report describes the development and initial validation of the Coronavirus Impact Scale, a rapidly constructed, brief measure of coronavirus impact. Overall, there is evidence to support summing the first 8 items of the Coronavirus Impact Scale to represent a unitary construct of impact in diverse samples. For every sample, this was supported through a model free test of saturation on the common factor and hierarchical clustering by psychometric criteria. External validation is supported by positive associations between the Impact Score and established, general measures of anxiety, stress reactions, and direct experience of infection. The study also provides evidence of which specific types of impact items co-occur and differences in specific experiences of impact across the samples. Altogether, these findings suggest acceptable psychometrics and utility of the scale for diverse samples.

Although there is good support of a common factor of impact across items, there is some inhomogeneity, or lumpiness, in the scale, where certain items cluster together in all samples. This is especially true of the 2 items representing food access and financial impact, which is to be expected, given their correspondence in general and an area in need of more research and intervention.²⁰ Of note, the clustering of stress with family support is consistent with recent work suggesting that family support mitigates COVID-19 impact in families.²¹ Extending the use of the scale to child respondents has preliminary support by the reasonably high correlation of the sum score between caregiver and child respondents. However, users should consider that the scale probes broad experiences of personal stress and family function and not child-specific impacts. Users should also consider that the scale was constructed at the outset of the pandemic, where indicators may most reflect the impact during the first wave. However, its early inclusion in the Adolescent Brain and Cognition and Development (ABCD) study suggests that it will have future utility to learn about early impacts on child development. In addition, it has been included in several preregistration reports,22, 23, 24, 25 suggesting that it will continue to have future utility and opportunity for comparison to later impacts.

We did not expect insufficient support for including direct or familial experiences of COVID-19 diagnosis in the scale. The interpretation should not be that these experiences do not contribute to impact. These experiences may have been relatively infrequent across samples at the time of this study. Indeed, those participants reporting personal or familial diagnosis of COVID-19 infection reported greater Impact Scores. Should data collection have occurred when direct experiences of infections were more prevalent, these items may have been analyzable for their contribution to overall impact. Unfortunately, waves of COVID-19 have continued to occur since the scale construction. Another important consideration is that direct experiences may be underreported by respondents because of privacy concerns.

Like this effort, many scales have been developed to measure aspects of the pandemic’s effects on people’s lives. Although it is beyond the scope of this report to introduce them all, this scale fits into measurements developed and capturing data very early in the pandemic. This scale was included in a US repository of disaster research response tools among more than 125 others, which can be found in the NIH Disaster Research Response COVID-19 collection.²⁶ Specific to adolescents, a collaborative open science framework effort also collected research tools at the same time.²⁷ Finally, a promising international effort has designed measures to address pandemic impact in adolescents.²⁸

Since construction, the scale has been reported in numerous studies in diverse samples both in the United States, where it was constructed, and internationally. Several findings of this work demonstrate its validity and inform the use of the scale. Internal consistency has been acceptable to excellent in peer-reviewed reports across large and diverse samples of adults and families.29, 30, 31, 32, 33 The scale shows convergence with other assessments of impact. In 810 adolescents in the United Kingdom,³⁴ the scale detected impact in its covered domains and demonstrated convergence with qualitative interviews in a subset of 18 individuals, except in representing protective factors and coping. In 550 caregivers in the US foster care system,³⁵ the scale has demonstrated convergence with measures of stress and burnout. The scale’s Impact Score was associated with severity of COVID symptoms persisting for more the 6 months in 299 adults.³⁶ The scale also shows utility in discovering associations between pathology and the pandemic’s impact. In our own work with longitudinal samples, we have found scale-measured impact to associate with parent reports of their children’s anxiety³⁷ and hyperactivity/impulsivity.³⁸ In examples of others’ work, impact as measured by the scale has been shown to moderate the relationship between school support and depression in predominantly Black families in urban areas³³ and to associate with eating disorder severity in Portuguese adults.^32,39

Differences between groups in item responses emerged. A high level of impact with a concerning disparity in financial impact and food access was experienced by Latinx, immigrant, single mothers of young children. This finding provides some quantitative evidence for such an expected disparity relative to other groups.⁴⁰ Based on direct experience of those working with the young mothers in the EC/CT sample, their immigrant status (mostly undocumented), being almost all monolingual Spanish speakers, and loss of employment/low incomes are likely factors that have led to direct impacts (eg, language barriers for services or lack of resources) as well as stress from fears of eviction and food insecurity. This reflects the broader truth that vulnerable and marginalized groups shoulder an unequal burden of the coronavirus pandemic.³¹

Two samples were characterized by seeking treatment in 2 settings outside of primary research; these were outpatients recently entering treatment (CHCO) and psychiatric inpatients assessed at admission (JHMI). In particular, the JHMI sample’s acute distress is likely reflected in the missing responses and lower reliability scores, a phenomenon often encountered in clinical measurement.⁴¹ Both samples reported relatively higher impact for mental health access. In the JHMI sample, 43 individuals provided free text responses to item 12, which asks the respondent to describe how the coronavirus has affected their life. These responses are detailed in Table S4, available online. They evocatively illustrate the ways in which the pandemic has had an impact on these families through economic, social, and psychological effects. These responses suggest that the CIS has good coverage of experienced impact for this group. Although much work is focusing on discovering associations between impact and health outcomes, these results support identifying differential experiences that may be key indicators of specific points of intervention, such as access to care.

The study and scale do have several limitations. The scale was constructed rapidly by clinical and research teams under adjustment to the pandemic themselves to develop a reasonable, publicly accessible, and free measure of impact that could be rapidly implemented in studies. Although this process is not ideal for the construction of novel psychometrics, a quantitative, iterative process of item selection would not have allowed the capture of responses in a final study in the first wave. Target families’ experiences were indirectly and informally assessed; formal qualitative methods and iterative quantitative analysis would have generated a better measure. Despite this limitation, the measure is reliable, represents a unitary construct, and has been found to be useful in diverse samples. Cohorts included in this investigation were samples of convenience. This limits our inferences about specific factors that drive group differences where they are highly confounded with group, for example age of child, which may be better examined in epidemiologic samples, for example, the ABCD study. Missingness was rare but was greater for reporting on family and friends’ illness. The reason cannot be well examined here, but speculatively may be due to privacy concerns or measurement error. An inadequate number of Spanish-speaking participants were included to robustly evaluate the validity of the Spanish version of the scale. Tests of interrater reliability, with different caregivers reporting on the same household, were not conducted.

In conclusion, this study provides psychometric validation for a well-anchored, brief assessment of the impact of the coronavirus on families and their children. In comparisons across 5 samples, it demonstrates clinical utility in measuring overall impact as well as demonstrating specific differences in impact among samples. It is also a useful clinical tool for identifying targets for intervention and support for families.