COVID-19 Rapid Response: How the California Health Interview Survey Adapted During the Global Pandemic

As a large, well-established population survey, the California Health Interview Survey (CHIS), housed at the University of California Los Angeles Center for Health Policy Research, was well poised to adapt to the changing conditions and challenges presented by the COVID-19 pandemic. Our goal was to continue to provide equity-focused data products relevant to public health,^1,2 but with a more rapid data processing timeframe to meet the immediate insights needed during the pandemic.

For nearly 20 years, CHIS has been collecting information on a population-representative sample of California’s adults, adolescents, and children. CHIS annually collects data from approximately 20 000 households. In 2019, CHIS transitioned from a landline and cell phone random-digit-dial methodology to a dual-mode administration that takes place on the Web or by telephone.³ CHIS employs an address-based sample design with mail recruitment inviting sampled individuals to complete a Web survey (mail push to Web). A phone number is provided in each mail invitation to allow interested respondents to alternatively complete the survey with a trained interviewer. This was important to reduce coverage bias because approximately 5% of the state’s population have no computers with Internet access.⁴ Up to six calls are made for nonresponse follow up with an offer of a computer-assisted telephone interview, in which interviewers follow a script provided by a software application.

To ensure coverage of respondents with limited English proficiency, in addition to English, CHIS is administered in Spanish, Cantonese, Mandarin, Korean, Tagalog, and Vietnamese.⁵ The two principal objectives of CHIS are to provide a large enough sample to yield robust health estimates for local jurisdictions (large and medium counties and county groups among smaller counties) and to provide statewide surveillance of racial/ethnic disparities in health status, health care coverage, and health care utilization. Both objectives are strongly aligned with the policy data demands of understanding the unequal toll of the pandemic on California counties and on racial/ethnic groups.

The culmination of decades of survey administration, the recent CHIS redesign and stakeholder engagement¹ provided a solid base in adapting survey operations in 2020. We describe the data collection and data release adaptations CHIS made in response to the COVID-19 pandemic, consider the implications of these changes on data quality, and discuss future implications on data collection for population-based surveys such as the CHIS.

PANDEMIC IMPACTS ON 2020 DATA COLLECTION

CHIS 2020 data collection commenced in early March 2020 contemporaneously with the beginning of stay-at-home orders in California. The first completed surveys were received on March 9, just 10 days before California issued a statewide stay-at-home order on March 19, 2020.⁶ Nearly 96% of all CHIS 2020 data collection occurred after these orders took effect. CHIS 2020 adult data collection concluded on October 31, 2020.

CHIS 2020 data collection was divided into 17 mailing waves with a total sample of 190 428 (Figure 1). More than 85% of the data collection was completed with the first 10 waves of sample mail outs (n = 161 640), spanning the beginning of the stay-at-home orders through the first major surge of COVID-19 infections within California from late June through August. Each of these individual mailing sample waves was selected to independently provide coverage across the state and, when aggregated together, to provide a comprehensive picture of the California population, with a goal of 20 000 total completed adult surveys. However, as the cycle of waves progressed, cooperation in some geographic areas differed from the projections used to design the sample. Therefore, modifications were made to the design of later waves to reduce the sample in areas that were performing better than expected and increase the sample in areas that were underperforming.

FIGURE 1—

California Health Interview Survey 2020 Sample Sizes and Data Collection Periods by Sample Wave and Month

Note. AsNHPI = Asian, Native Hawaiian, and Pacific Islander.

While CHIS internally addressed adaptation of day-to-day operations given the stay-at-home orders (e.g., ensuring remote access to secure data), our data collection partner, SSRS, a full-service survey research firm located in Pennsylvania, continued to operate its mail and printing services as essential services.⁷ This allowed the production and mailing of recruitment materials to proceed unimpeded by widespread pandemic shutdowns. Though telephone data collection only accounts for approximately a tenth of total CHIS interviews, SSRS was able to transition their telephone interviewing operation in a matter of days to a decentralized, home-based model allowing inbound computer-assisted telephone interviewing and outbound nonresponse follow-up to continue.

When we examined the impact of the pandemic on CHIS data collection results, the average completion rate for CHIS 2020 was 11.4%. For comparison, CHIS 2019 only achieved an average 8.7% completion rate. Despite state and local leadership loosening some restrictions over the course of the year,⁸ completion rates were relatively consistent across the waves of sample mail outs. The latter, smaller waves of data collection had slightly lower completion rates (average of 10.4%), but given that many of these waves began in July at the initial peak of infections in California, this decline in completion rates in the latter waves is most likely attributable to increasing proportions of workloads in harder-to-survey counties where we were falling short of annual county-level targets. Even the least effective waves of 2020 data collection outpaced the most effective waves of 2019. There was also a noticeable shift in the timing and mode of completed surveys in CHIS 2020 relative to CHIS 2019. Nearly 50% of all completed adult surveys per wave were completed in the two weeks following the initial invitation letter, a noticeable increase from 33% in 2019. Overall, computer-assisted telephone interview completes accounted for 10.9% of all completed adult surveys in 2019 but only 7.7% in 2020. The revised sample design and data collection strategy allowed CHIS to take advantage of the increased number of Californians reachable at home because of the stay-at-home orders. The strong health focus of CHIS may have been a more salient factor to many respondents given the spotlight on public health during the pandemic.

The CHIS sample design is intended to support annual estimates for California and many individual counties by aggregating all interviews conducted across all weekly sample waves. The need for timely data during the pandemic and the strong response observed during this time warranted examining the feasibility of producing more timely subannual estimates. We employed a strategy that pooled all interviews conducted across sample waves within a given calendar month (Figure 1). The monthly totals were determined to be sufficient for statewide estimates, as well as for large substate regions and demographic subgroups. Given that the sample distribution in later waves was modified to compensate for underperforming areas across the state to reach annual targets, the unweighted sample in these later waves tends to be less representative of the statewide population. To help compensate for the geographic differences in the sample over time, we included substate geographic variables as raking dimensions during weighting. Accounting for the yield of completed surveys each month and the geographic changes in the sample distribution, CHIS decided to release monthly estimates for May, June, July, and August, but not for September and October.

ADAPTING QUESTIONNAIRE TO REFLECT COVID-19 NEEDS

With an increasing number of COVID-19 cases and public health guidelines across the state, CHIS resolved to add COVID-19‒related questions with the planning for a specific COVID-19 module beginning in mid-March 2020. Without ample time to conduct iterative cognitive pretesting, the CHIS team at UCLA consulted various experts involved in monitoring the pandemic. We explored COVID-19‒related questions from surveys in the United States and Canada including those shared on the Centers for Disease Control and Prevention COVID-19 Community Survey Question Bank, which at the time of our instrument development offered a comprehensive set of potential questions for community surveys. Through several revisions, CHIS finalized a nine-item COVID-19 module in April 2020 with indicators on COVID-19 concerns, testing, and diagnoses; financial and social impacts of the pandemic; personal acceptance of the COVID-19 vaccine; and reasons why respondents had no insurance (Appendix A, available as a supplement to the online version of this article at http://www.ajph.org). Simultaneously, the UCLA Asian American Studies Center and CHIS began collaborating on a module intended to capture the impacts of the COVID-19 pandemic among Asian, Native Hawaiian, and Pacific Islander (AsNHPI) communities. Responding to the rise of anti-Asian rhetoric and hate incidents targeting the AsNHPI community, the AsNHPI module measures the prevalence of anti-Asian bias and perceived association between COVID-19 and Asians, the economic and social impacts of COVID-19 specifically on AsNHPI communities, and perceptions among AsNHPI communities on how national, local, and academic institutions handled the pandemic. CHIS began fielding the AsNHPI 12-item module in July 2020 (Appendix B, available as a supplement to the online version of this article at http://www.ajph.org).

These two novel modules underwent multiple rounds of collaborative development with stakeholders at the UCLA Fielding School of Public Health, UCLA Geffen School of Medicine, the UCLA Asian American Studies Center, and the California Department of Public Health. With its data collection partner, SSRS, the CHIS team expeditiously developed and implemented the general COVID-19 module on May 5, 2020, and the COVID-19 AsNHPI module on July 2, 2020. As these questions were fielded, modifications were made to adapt to the changing practices and guidelines surrounding COVID-19, including expanding the universe of the indicator on whether the respondent was ever tested from a subset of adults who saw a clinician related to COVID-19 symptoms to all respondents to reflect the growing availability of testing that did not require clinician recommendation.

NEW DISSEMINATION APPROACHES FOR COVID-19 DATA

Given that CHIS is designed as an annual survey, the COVID-19 module presented two challenges: (1) Could the data be made public almost in real time? and (2) How can data be presented so that they are generally understood and differentiated from the annual data products?

The sample yields from the first months after the May implementation of the COVID-19 module were robust and well-distributed enough across the state to release statewide preliminary monthly estimates. However, there were few protocols in place for producing and releasing preliminary estimates monthly. CHIS worked with SSRS to abbreviate the typical data processing procedures, including expediting the timeline for geocoding and race upcoding needed to produce population survey weights, allowing CHIS to produce monthly population estimates in a publicly accessible data dashboard.

The dashboard on preliminary monthly estimates containing data collected in May was launched in September 2020, with subsequent releases of June data in October 2020, July data in November 2020, and August data in December 2020. Users were able to access preliminary estimates for the majority of the new COVID-19 indicators at the state level and apply demographic filters including racial/ethnic subgroups, insurance status, and household size to illustrate how these sociodemographic indicators affected the prevalence of indicators in the COVID-19 modules, including the types of difficulties experienced because of COVID-19 and whether the respondent had ever been tested for COVID-19. Furthermore, because COVID-19 resources and responses were at the local level, CHIS generated substate estimates by pooling data collected from May through August 2020. CHIS established 18 substate groupings with enough sample size to produce data estimates at a more granular level including data at the county level for seven larger California counties including Los Angeles, Santa Clara, and Alameda. With these substate estimates, users could access data for various indicators in the COVID-19 module and apply health and sociodemographic filters to create a visual illustration of how these covariates are distributed across a map of California (Appendix C, available as a supplement to the online version of this article at http://www.ajph.org). For the COVID-19 AsNHPI module, pooled data from July through October were used to create estimates on these special topics for the AsNHPI population and were added to the dashboard in February 2021 (Appendix D, available as a supplement to the online version of this article at http://www.ajph.org).

FUTURE IMPLICATIONS AND TAKEAWAYS

The urgency of providing health data during the COVID-19 pandemic necessitated a way to share CHIS findings with the public in real time. Although the design of the CHIS remains an annual survey, the 2020 CHIS demonstrated for the first time the viability of pooling completed surveys conducted across weekly sample waves within a calendar month to produce monthly statewide estimates. Fluctuations in sample sizes and changes in the geographical distribution of the sample across the weekly sample waves created some challenges in producing stable monthly estimates or allowing for an accurate comparison or progression of indicators across several months.

In planning for 2021, CHIS has improved the consistency of sample releases across the survey administration year to help ensure adequate sample sizes are available from month to month to produce monthly COVID-19 preliminary estimates through the bulk of 2021. In the future, CHIS will also consider how to implement real-time measures to assess the representativeness of the raw sample yield to improve the quality of monthly estimates with less reliance on geographical and demographic raking factors in the weighting processes to correct for month-to-month variation in the sample design.

The scientific value is largely to inform other surveys that reporting preliminary monthly estimates from an annual population-based survey is possible, albeit limited to a subset of indicators. A monthly data collection sampling approach is necessary to produce monthly estimates, but, to our knowledge, these provisional estimates from other surveys were conducted in English only or in English and Spanish only and, thus, were limited in coverage of racial/ethnic and language diversity.^9,10

COVID-19 exposed longstanding inequities shaped by socioeconomic conditions and opportunities. Our preliminary monthly COVID-19 dashboard was the first time CHIS released monthly data within three months of data collection, compared with a standard release of annual data 10 months after data collection. CHIS has had a two-decade commitment to measuring these socioeconomic conditions to augment public health’s understanding of the inequities in population health. For this reason, we pursued a strategy to release estimates with a large set of sociodemographic filters to inform policies to address these inequities, but the tradeoff with more data granularity was a longer release time. As the annual CHIS 2020 is currently being processed, CHIS will further assess the data quality of the 2020 monthly estimates and streamline processes to optimize the release of monthly estimates. CHIS will also consider other measures that might benefit from subannual estimates. The demonstrated feasibility of the COVID-19 dashboard has established the foundations for potential future use of dashboards as a platform for communicating and democratizing more timely critical health and health-related data.